GBiB: From download to BLAT at assembly hubs
From genomewiki
Revision as of 15:49, 29 January 2020 by David da Silva Pires (talk | contribs) (Adding command to set right permissions to access /usr/local/src/gbib)
Introduction
Genome Browser in a Box (GBiB) has some obvious advantages when compared to other options we have while working with genomic data:
- It is a genome browser with a lot of features and tools that do not exist on other available genome browsers.
- It is much easier to install, configure and maintain when compared with a full mirror of UCSC Genome Browser web site.
- It is a safe way to keep your private data inaccessible to unauthorized users while still collaborating with authorized personal.
Nonetheless, even after choosing GBiB as your genome browser, there is a lot of different choices to do. This wiki page explains how to install, configure and maintain an assembly hub (with a track hub and BLAT) using GBiB on a laptop running Kubuntu 19.10 (Eoan Ermine). Most commands are the same for other GNU/Linux distributions, with the differences probably being only relative to package names and crontab settings. Other architectures should involve the installation of GBiB on a server using only text interface and with specific ports enabled on the firewall to restrict the use of the data for just your network.
GBiB installation
- Create a folder at your machine to place the installation files:
user@host:$> sudo mkdir /usr/local/src/gbib user@host:$> sudo chmod o+x /usr/local/src/gbib
- Download GBiB from UCSC Genome Browser virtual store:
- Go to the Genome Store.
- Click in "Login / Register".
- Check if you agree with the terms and conditions at the box relative to GBiB.
- Check if your hardware and software meet the basic requirements.
- Click in "Add to cart".
- Click in "Cart (1)" on menu.
- Click in "Proceed to checkout".
- Click in "My products" on menu.
- Copy the address of download (let's call it <download_link>).
- Download GBiB to /usr/local/src/gbib, uncompress and delete it.
user@host:$> cd /usr/local/src/gbib user@host:$> sudo wget <download_link> user@host:$> sudo unzip gbib.zip user@host:$> sudo rm gbib.zip
- Give user sufficient access to the three uncompressed files and to the folder:
user@host:$> sudo chmod o+rw /usr/local/src/gbib/* user@host:$> sudo chmod o+w /usr/local/src/gbib
- Install VirtualBox and start it in background:
user@host:$> sudo apt-get install virtualbox user@host:$> virtualbox &
- Add GBiB to VirtualBox and boot it for the first time:
- Machine ---> Add ---> /usr/local/src/gbib/browserbox.vbox ---> Start
- Wait while the first update is done (it can take from 5 minutes to more than 1 hour to finish the update process, depending of your internet connection speed).
- Close GBiB terminal window.
- Select "Send the shutdown signal".
- Confirm by clicking "OK".
GBiB configuration
- Click at "Settings".
- General ---> Description: Ebola virus genome assembly and track hubs.
- System ---> Motherboard ---> Base Memory: 4.096 MB.
- System ---> Processor ---> Processor(s): 2.
- Display ---> Video ---> Video Memory: 32 MB.
- Shared Folders ---> + ---> Folder Path: ~/var/gbib/work ---> Auto-mount ---> OK.
- Boot GBiB virtual machine:
- Select "browserbox" on menu at left.
- Click at "Start".
- Test if everything is working at the following URLs:
- Login using ssh, for a faster access.
- Open a terminal, like "konsole".
- Password: browser
user@host:$> ssh browser@localhost -p 1235
- Install tools that allows file manipulations:
browser@browserbox:$> gbibAddTools
- Turn off every kind of automatic update:
browser@browserbox:$> gbibAutoUpdateOff
- Do not allow users to mirror tracks:
browser@browserbox:$> gbibMirrorTracksOff
- Turn on the offline mode:
browser@browserbox:$> gbibOffline
- Reboot the virtual machine
browser@browserbox:$> sudo shutdown -r now
Assembly hub configuration
- Log in again using ssh:
user@host:$> ssh browser@localhost -p 1235
Downloading the raw data
- Create the directories that will store the assembly hub configuration files:
browser@browserbox:$> mkdir -p /folders/sf_work/virusNetwork/eboVir3/genome browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/genome browser@browserbox:$> rsync -avzP rsync://hgdownload.soe.ucsc.edu/goldenPath/currentGenomes/Ebola_virus/bigZips/KM034562v1.fa.gz . browser@browserbox:$> gunzip KM034562v1.fa.gz browser@browserbox:$> ln -s KM034562v1.fa eboVir3.fasta
Creating a basic hub.txt file
- Fill the contents of hub.txt file:
browser@browserbox:$> cat > /folders/sf_work/virusNetwork/hub.txt << EOI hub virusNetwork email admin@virus.edu shortLabel Virus Network longLabel Virus Network Hub for Ebola virus genomesFile genomes.txt EOI
- The following rules must be obeyed:
- hub: name without spaces.
- shortLabel: limited to 17 characters.
- longLabel: limited to 80 characters.
Creating a basic genomes.txt file
- Fill the contents of genomes.txt:
browser@browserbox:$> cat > /folders/sf_work/virusNetwork/genomes.txt << EOI genome eboVir3 organism Ebola virus defaultPos KM034562v1:1-18,957 twoBitPath eboVir3/genome/eboVir3.2bit trackDb eboVir3/trackDb.txt EOI
Preparing the data
- In the case that the fasta file is written with all nucleotides in lowercase, convert all the uppercase letters such that the genome do not be considered as if it was all masked. We can use the change_case command, from seq_crumbs:
user@host:$> change_case --in_format fasta --outfile ~/var/gbib/work/virusNetwork/eboVir3/genome/eboVir3-uppercase.fasta --processes 4 -a upper ~/var/gbib/work/virusNetwork/eboVir3/genome/eboVir3.fasta
- If the names of the chromosomes are very long, we need to make them shorter:
browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/genome/ browser@browserbox:$> sed s/Ebola_virus/Ev/ eboVir3-uppercase.fasta > eboVir3-uppercase-shortChromNames.fasta
- Make a symbolic link (just to shorten the name) and get the .2bit file from the FASTA file:
browser@browserbox:$> ln -s eboVir3-uppercase-shortChromNames.fasta eboVir3.fasta browser@browserbox:$> faToTwoBit eboVir3.fasta eboVir3.2bit
- Get and sort from the largest to the shortest a file with the name and the size of all chromosomes of the genome of interest:
browser@browserbox:$> twoBitInfo eboVir3.2bit stdout | sort -k2nr > eboVir3-chromSizes-sorted.txt
- Build an AGP file from the fasta file, marking all N's as gaps, using the hgFakeAgp command.
browser@browserbox:$> hgFakeAgp -minContigGap=1 eboVir3.fasta eboVir3.agp
- Check if the new AGP file matches the fasta file:
browser@browserbox:$> sort -k1,1 -k2n,2n eboVir3.agp > eboVir3-sorted.agp browser@browserbox:$> checkAgpAndFa eboVir3-sorted.agp eboVir3.2bit
Track hub configuration
We have to configure at least one track at our track hub in order to have a working assembly hub. The first track that we will configure is the assembly track, which shows a block for every range that contains nucleotides different of 'N'. In other words, the gaps are the holes in the track.
- Create the contents of trackDb.txt (track without spaces or dots and with the first character as a letter, shortLabel <= 17 chars, longLabel <= 80 chars):
browser@browserbox:$> mkdir -p /folders/sf_work/virusNetwork/eboVir3/tracks/map/assembly/{input,output}
browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/tracks/map/assembly/
browser@browserbox:$> cat > trackDb.txt << EOI
track assembly
shortLabel Assembly
longLabel Assembly
type bigBed 6
bigDataUrl tracks/map/assembly/output/assembly.bb
EOI
The name of each track ("track" field) must be unique at the entire file and should not contain the character '.' (dot).
- Construction of the assembly track directly from the AGP file:
browser@browserbox:$> ln -s ../../../../genome/eboVir3-sorted.agp input/
browser@browserbox:$> grep -v "^#" input/eboVir3-sorted.agp | awk '{if ($5 != "N") printf "%s\t%d\t%d\t%s\t0\t%s\n", $1, $2, $3, $6, $9}' | sort -k1,1 -k2,2n > output/assembly.bed
browser@browserbox:$> ln -s ../../../../genome/eboVir3-chromSizes-sorted.txt input/
browser@browserbox:$> bedToBigBed -verbose=1 -type=bed6 -tab output/assembly.bed input/eboVir3-chromSizes-sorted.txt output/assembly.bb
- Edit the main trackDb.txt file to include the assembly track configuration.
browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/ browser@browserbox:$> cat > trackDb.txt << EOI #========================================== # MAPPING AND SEQUENCING. # Assembly. include tracks/map/assembly/trackDb.txt EOI
- Double check the integrity of your hub with the command hubCheck:
browser@browserbox:$> hubCheck /folders/sf_work/virusNetwork/hub.txt browser@browserbox:$> hubCheck http://127.0.0.1:1234/folders/sf_hubs/virusNetwork/hub.txt
Blat configuration
- From the folder that contains the .2bit file, start two gfServer's, specifying the assembly hub ports that will be used to access the DNA sequence and the aminoacids sequence:
browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/genome browser@browserbox:$> mkdir ../log/ browser@browserbox:$> ~browser/bin/blat/gfServer start 127.0.0.1 42420 -stepSize=5 -log=../log/gfServer.log eboVir3.2bit & browser@browserbox:$> ~browser/bin/blat/gfServer start 127.0.0.1 42421 -trans -log=../log/gfServer-trans.log eboVir3.2bit &
- Edit the file genomes.txt of the assembly hub in order to include the lines relatives to blat and transBlat (pay attention to the capital 'B' at "transBlat"):
blat 127.0.0.1 42420 transBlat 127.0.0.1 42421
GBiB maintenance
- Make an update of all softwares and data:
$> gbibOnline $> gbibAutoUpdateOn $> updateBrowser $> gbibAddTools $> gbibAutoUpdateOff $> gbibOffline
Additional configuration
Gap track
It is easy to build a gap track directly from the AGP file. The gap track evidences the genome loci that contains N's, being a kind of complementary track of the assembly track.
browser@browserbox:$> mkdir -p /folders/sf_work/virusNetwork/eboVir3/tracks/map/gap/{input,output}
browser@browserbox:$> cd /folders/sf_work/virusNetwork/eboVir3/tracks/map/gap/
browser@browserbox:$> ln -s ../../../../genome/eboVir3-sorted.agp input/
browser@browserbox:$> grep -v "^#" input/eboVir3-sorted.agp | awk '{ if ($5 == "N") printf "%s\t%d\t%d\t%s\n", $1, $2, $3, $8}' | sort -k1,1 -k2,2n > output/gap.bed
browser@browserbox:$> ln -s ../../../../genome/eboVir3-chromSizes-sorted.txt input/
browser@browserbox:$> bedToBigBed -verbose=1 -type=bed4 -tab output/gap.bed input/eboVir3-chromSizes-sorted.txt output/gap.bb
browser@browserbox:$> cat > trackDb.txt << EOI
track gap
shortLabel Gap
longLabel Gap Locations
type bigBed 4 .
bigDataUrl tracks/map/gap/output/gap.bb
EOI
browser@browserbox:$> cat >> /folders/sf_work/virusNetwork/eboVir3/trackDb.txt << EOI
# Gap Locations.
include tracks/map/gap/trackDb.txt
EOI
Basic HTML page
Let's compose a basic page to our organism of interest:
browser@browserbox:$> cat > /folders/sf_work/virusNetwork/eboVir3/description.html << EOI <p> <i>Ebola</i> virus genome assembly and track hub. <ul> <li><a HREF="http://www.ncbi.nlm.nih.gov/genome/4887" TARGET="_blank"> NCBI genome/4887 (Ebola virus)</a></li> </ul> </p> <p> <b>UCSC Genome Browser assembly ID:</b> eboVir3<br> </p> EOI
For a more complete example, see the code of the following page: http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/araTha1/description.html.
Other tips
At last the hubCheck command will run without any error being identified. But, if you point your browser to http://127.0.0.1:1234 and go to: Genomes ---> group = Virus Network, you will see the following error:
"Couldn't open http://127.0.0.1:1234/folders/sf_hubs/virusNetwork/schMan2/description.html"
- Forcing configuration files to be loaded again every time that the page is reloaded (instead of after at least 300 seconds):
- Insert "udcTimeout=1&" right after http://genome.ucsc.edu/cgi-bin/hgTracks? at URL.
- To disable this feature, click at "clear" on the message that appears at the top of the page.
- Create the HTML page description for the hub:
$> cat > ~/var/gbib/hubs/virusNetwork/description.html << EOI <HEAD><TITLE>Virus Network Hub</TITLE> <BODY> <P> Ebola virus genome assembly and track hub. <UL> <LI><A HREF="http://www.ncbi.nlm.nih.gov/genome/4887" TARGET="_blank"> NCBI genome/4887 (Ebola virus)</A></LI> </UL> </P> <BODY></HTML> EOI
- Include an image of the organism.
- The same substitution have to be done at the bed file of the track:
$> sed s/Schisto_mansoni/Sm/ smps.bed > smps-shortChromNames.bed
- The bed file of the track have to be sorted first by the name of the chromosome and after by the starting coordinate:
$> sort -k1,1 -k2,2n smps-shortChromNames.bed > smps-shortChromNames-sorted.bed
- Convert from bed to bigBed:
$> bedToBigBed -verbose=1 -type=bed12 -tab -extraIndex=name smps-shortChromNames-sorted.bed schMan2-chromSizes-sorted.txt smps.bb
- Construction of the GC content track:
browser@browserbox:$> hgGcPercent -wigOut -doGaps -file=stdout -win=5 -verbose=0 eboVir3 \
eboVir3.2bit | gzip -c > eboVir3-gc5Base-wigVarStep.gz
browser@browserbox:$> wigToBigWig eboVir3-gc5Base-wigVarStep.gz eboVir3-chromSizes.txt eboVir3-gc5Base.bw
- Contents of groups.txt:
$> cat > /usr/local/src/gbib/hubs/virusNetwork/schMan2/groups.txt << EOI name user label Custom priority 1 defaultIsClosed 1 name map label Mapping priority 2 defaultIsClosed 0 name genes label Genes priority 3 defaultIsClosed 0 name mrna label mRNA priority 4 defaultIsClosed 1 name regulation label Regulation priority 5 defaultIsClosed 1 name comparative label Comparative priority 6 defaultIsClosed 1 name varRep label Variation priority 7 defaultIsClosed 0 name x label Experimental priority 8 defaultIsClosed 1 EOI
- Let's compose an HTML page to our track:
$> cat > ~/var/gbib/work/virusNetwork/eboVir3/assembly.html << EOI <H2>Description</H2> <P> Replace this text with a summary describing the concepts or analysis represented by your data. <H2>Methods</H2> <P> Replace this text with a description of the methods used to generate and analyze the data. <H2>Verification (Validation)</H2> <P> Replace this text with a description of the methods used to verify (validate) the data. <H2>Credits</H2> <P> Replace this text with a list of the individuals and/or organizations who contributed to the collection and analysis of the data. <H2>References</H2> <P> Replace this text with a list of relevant literature references and/or websites that provide background or supporting information about the data. Other interesting information: background information, display conventions, and acknowledgments. EOI
visibility full html schMan2-description boxedCfg on colorByStrand 150,100,30 230,170,40 color 150,100,30 altColor 230,170,40 dataVersion Dec. 2011 Sanger 5.2 # directUrl http://verjo-server-01.iq.usp.br/genome/pires/virusNetwork/schMan1/geneView/%s iframeUrl https://www.google.com.br/search?q=$$ iframeOptions height='400' width='640' scrolling='yes' priority 100 url http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?form=4&db=n&term=$$ urlLabel NCBI Details: urls pmid="http://www.ncbi.nlm.nih.gov/pubmed/$$" spId="http://www.uniprot.org/uniprot/$$" track roche454-blat bigDataUrl roche454-blat.bb shortLabel Roche 454 Trinity longLabel Schistosoma mansoni RNA-Seq Roche 454 Trinity contigs mapped by Blat type bigBed 12 searchIndex name visibility full color 64,0,96 altColor 64,32,128 track assembly longLabel Assembly shortLabel Assembly priority 10 visibility pack colorByStrand 150,100,30 230,170,40 color 150,100,30 altColor 230,170,40 bigDataUrl eboVir3-assembly.bb type bigBed 6 html trackDescriptions/assembly url http://www.ncbi.nlm.nih.gov/nuccore/$$ urlLabel NCBI Nucleotide database group map track gap longLabel Gap shortLabel Gap priority 11 visibility dense color 0,0,0 bigDataUrl eboVir3-gap.bb type bigBed 4 group map html trackDescriptions/gap track gc5Base shortLabel GC Percent longLabel GC Percent in 5-Base Windows group map priority 23.5 visibility full autoScale Off maxHeightPixels 128:36:16 graphTypeDefault Bar gridDefault OFF windowingFunction Mean color 0,0,0 altColor 128,128,128 viewLimits 30:70 type bigWig 0 100 bigDataUrl eboVir3-gc5Base.bw html trackDescriptions/gc5Base # For bigWig data, we can use the new trackDb setting, negateValues on, to allow display on the Crick strand. EOI
- If the fasta file that was used to create the .2bit file was masked (i.e., it had aminoacids with lowercase letters), we can use the gfServer flag "-mask":
$> gfServer start 127.0.0.1 42423 -trans -mask schMan2.2bit &
- Add this commands to cron, writing them just before the "exit" command at last line:
$> sudo su - $> vim /etc/rc.local @vim $> @vim $> # Blat and transBlat daemons running against Ebola virus genome at ports 42422 and 42423, respectively. @vim $> cd /folders/sf_hubs/virusNetwork/eboVir3 @vim $> ~browser/bin/blat/gfServer start localhost 42422 -stepSize=5 -log=/var/log/gfserver.eboVir3.log eboVir3.2bit & @vim $> ~browser/bin/blat/gfServer start localhost 42423 -trans -log=/var/log/gfserver.eboVir3-trans.log eboVir3.2bit &
To add a track hub by directly adding the hub's URL to the browser URL. If you add hubUrl=[URL] to your hgTracks URL line, it will add the hub directly into the browser (e.g. http://genome.ucsc.edu/cgi-bin/hgTracks?db=hub_47033_schMan1&hubUrl=http://www.vision.ime.usp.br/~davidsp/hub/hub.txt).
Additional configuration to hub.txt:
descriptionUrl description
Additional configuration to genomes.txt:
groups eboVir3/groups.txt description Ebola virus version 3 orderKey 1 htmlPath eboVir3/description scientificName Ebola
Custom track configuration
browser position chr22:20,100,000-20,100,900 browser hide all track name="Track label" description="Chromossomes coordinates list" type=bigBed visibility=full color=200,50,50 itemRgb=On colorByStrand=0,0,50 0,50,0 useScore=1 altColor=100,200,200 group=x priority=1 db=eboVir3 url="http://verjo-server-01.iq.usp.br/genome/pires/smps.html#$$ htmlUrl="http://verjo-server-01.iq.usp.br/genome/pires/track-description.html" bigDataUrl=http://verjo-server-01.iq.usp.br/genome/pires/Schistosoma_mansoni_v5.2.gff.bed.bb
- The following rules must be obeyed:
- name: can consist of up to 15 characters, and must be enclosed in quotes if the text contains spaces.
- description: can consist of up to 60 characters, and must be enclosed in quotes if the text contains spaces.
- visibility: values include: 0 - hide, 1 - dense, 2 - full, 3 - pack, and 4 - squish.
- group: values include: custom, mapping, genes, mrna, regulation, comparative, variation, and x.
Setting locale
Since Kent's tools (like bedToBigBed) expect to find a system that provides a case-sensitive sort, we have to set the environment variables relative to locale to the value "C".
Put the following lines at the bottom of ~browser/.bashrc at your GBiB:
# Define custom locale settings. export LANG="C" export LANGUAGE="C" export LC_MESSAGES="C" export LC_CTYPE="C" export LC_NUMERIC="C" export LC_TIME="C" export LC_COLLATE="C" export LC_MONETARY="C" export LC_PAPER="C" export LC_NAME="C" export LC_ADDRESS="C" export LC_TELEPHONE="C" export LC_MEASUREMENT="C" export LC_IDENTIFICATION="C" export LC_ALL="C"
After that, load .bashrc again by doing:
$> . ~browser/.bashrc
References
See also:
