Conservation Track QA

Preparing to QA

Please follow the instructions in this wiki page, in addition to the New Track Checklist. The Conservation track will consist of one or more of the following subtracks: multiz#way, phyloP, phastCons and Most Conserved/Conserved Elements. Sometimes if there is only a multiz#way and a Most Conserved/Conserved Elements tracks, they will be in separate tracks. Otherwise, they should all be together.

Make sure that pushQ entry has all gbdb files and downloads needed for the track

For gbdb, there should be either be one big .maf file or one .maf file for each chromosome for the multiz#way track. Both phyloP and phastCons will have one .wib file associated with each. There are no gbdb files for the Most Conserved/Conserved Elements track.

For the downloads there will typically be a directory for multiz#way, phastCons and phyloP. Typically all the contents of these directories are pushed, unless there are special files that the developer does not want to go out. There is no directory for the Most Conserved/Conserved Elements track.

To see the list of assemblies actually used to build the tracks

cat /hive/data/genomes/$db/bed/multiz#way/species.lst

(Note that sometimes the file extension is .list; depends on the developer. Hiram tends to use .list)

Make a list of all organisms in the Conservation track

This will help you make sure that the correct name and date are associated with each assembly

human	Homo sapiens	Mar 2006	hg18
chimpanzee	Pan troglodytes	Mar 2006	panTro2
...	...	...	...

Make a list of all organisms for which there are nets & chains and put them in phylogenetic order

This will provide you with a reference for the phylogenetic order of species. Wikipedia is a good place to look for species relationships to confirm that our tree is correct (e.g., euarchontoglires, Laurasiatheria, primates, carnivores). NCBI's Taxonomy Browser can also be helpful.

• hg18
• panTro2
• ...

QAing the track

Figure out extFile and seq tables

• if they are standard maf files, there will be no entries in the seq table.
• There may be more than one set of entries in the extFile table. Make sure you only push the set that pertains to the actual files you are pushing to hgnfs1 (e.g. /gbdb/ponAbe2/multiz8way/anno/maf/*)
• These are the ones that will need pushing to beta:

mysql> select path from extFile where path like "%anno/maf%";

Do not push extFile and seq tables, you must use this script to copy the rows from dev to beta:

copyExtSeqRows.csh

• If this is a new assembly and this is the first Conservation track, the extFile table probably doesn't exist on beta and likely only has a single entry on dev. If this is the case, it is ok to just push the entire table from dev to beta.
• Include extFile table in push request when you are ready to push to the RR.

Check trackDb.ra file

For the multizway track make sure there is a speciesCodonDefault entry (usually is this species). This is the species that is used as the default for codon translation. Also ask Jim to sign off on the species listed in the speciesDefaultOff entry. This entry determines which species are turned off by default and, therefore, which ones are on by default.

Test the tables

Check multizway#wayFrames table

• check that each species in the description page has entries in the Frames table (select distinct(src) from multiz8wayFrames), including entries for the species that this track is for. If there are species missing, look in the makedoc to see if there was a reason they were excluded (e.g. no good gene set). If there is nothing in the makedoc, ask the developer.

Find out how phastCons was run

The --not-informative flag is used when subsets of a multiple alignment are used to calculate phastCons (i.e. when this track has subgroups/subtrees). If this is the case there will be a *.non-inf file that contains the list of orgs not used in the calculations for that subgroup. An example of a *.non-inf file can be found here:

/cluster/data/ponAbe2/bed/multiz8way/cons/primates/primates.non-inf

it includes: monDom4,ornAna1

These species are not primates and so are not informative when doing the primate-only phastCons. Check in the make doc to see if phastCons was run with the --not-informative flag. If it was, check for a *.non-inf file and see if the species given in that file belong there (i.e. for a 'primates' subgroup, there shouldn't be any primates specified in the *.non-inf file)

Make sure hgPal works with this set of tables

• the hgPal CGI should work with the multiz table for this Conservation track. To do this go to a RefSeq track, click on a gene and then click on the "CDS FASTA" link on the trackUi page. This should take you to a page that links to the MAF file. This can also be reached by clicking on the "Protein FASTA" link for a UCSC Gene, and by selecting "cds FASTA" output for either track (refseq/ucsc genes) in the Table Browser. For both, make sure that all the buttons work.

Read through both description pages

• Conservation track:
  • Check image that displays on conservation details page.
  • Check "Gene tracks used for codon translation" table against make doc.
  • Make sure organisms are listed (in all places, including the gif & track controls (Multiz Alignments Configuration box)) in the correct phylogenetic order. If the check boxes in the track control are not in order, you can reorder them in trackDb.ra next to the sGroup_### line under "track multiz##way".
  • Make sure that this page includes all the extra sections (if the multizs have been annotated). Sections should include: Gap Annotation, Genomic Breaks, Base Level, and a chart for protein/codon translation
  • Make sure there is a tree model available.
• Most Conserved track:
  • Make sure the text refers to the correct species.

Test in the Genome Browser

• Zoom out past 1M bps (this tests the multiz*waySummary table)
• Check out codon translation for a few species.
• make sure that Conserved Elements (a.k.a. Most Conserved) track items have names like "lod=22", not like "chr3.172".

Check downloads

Read all README files in hgdownload directories

Make sure that there are no spelling or grammatical errors. Also check that the information, including the help links, are correct and relevant.

Check one maf file

Verify that it has all the species involved in the track, and that at least one entry in the file is formatted according correctly (more information here: http://genome.ucsc.edu/FAQ/FAQformat.html#format5).

hgwdev: zcat chr#.maf.gz  | head

Modify downloads.html

Modify the Multiple Alignments section for this assembly in downloads.html. If this is the first Conservation track for this assembly, you'll need to create a new Multiple Alignments section underneath the Pairwise Alignments section.

Things to not do

joinerCheck, searching

Things we used to do to QA the conservation track but no longer do

Check annotated maf files for overlapping blocks

(maf files are generated automatically now, so we no longer need to do this) [hgwdev:/gbdb/ponAbe2/multiz8way/anno/maf>

foreach f (*.maf)
  echo -n "${f}: "
  mafFilter -overlap -minRow=1 $f > /dev/null
end

If there are 'rejected blocks', contact the developer.

Check upstream files to make sure that the species name doesn't appear in an "s" line

(the upstream maf files are generated automatically now, so we no longer need to do this)

[hgwdev:~/goldenPath/ponAbe2/multiz8way/maf> zcat upstream*.maf.gz | grep "s ponAbe2" | wc -l
0

If this is not zero, contact the developer.

Check upstream files to make sure gene names haven't been truncated (to 9 chars)

(the upstream maf files are generated automatically now, so we no longer need to do this)

[hgwdev:~/goldenPath/ponAbe2/multiz8way/maf> zcat upstream*.maf.gz | head

##maf version=1 scoring=zero
a score=0.000000
s CG13384-RD_up_1000_chr2L_8383467_f 0 1000 + 1000

If the gene names are short (9 characters) contact the developer.

To generate a list of gene names and pipe to a file called "test": zcat upstream1000.maf.gz | grep "NM_.*" | cut -f2 -d" " > test

To see how many file names are 10 or more characters long: grep "[0-9]\{10,\}" test

Check upstream files to make sure sequence hasn't been reverse-complemented incorrectly

(the upstream maf files are generated automatically now, so we no longer need to do this)

Since reverse-complement is a relative thing, the MAF sequence is supposed to be in the direction of transcription. That is, for a negative strand gene, its reversed-complement of the genome sequence. So it is supposed to be r-c of the genome, and not r-c of the direction of transcription.

From the MAF file documentation:

strand -- Either '+' or '-'. If '-', then the alignment is to the reverse-complemented source.

Search in the MySQL database for a gene on the minus strand. Then find that gene in the upstream1000.maf.gz file then check for correct r-c.

e.g. mysql> select name, chrom, strand from ensGene where strand = "-" limit 1\G
name: ENSORLT00000000020
chrom: chr1
strand: -

[hgwdev:~/goldenPath/oryLat2/multiz5way> zcat ensGene.upstream1000.maf.gz | grep "ENSORLT00000000020"
s ENSORLT00000000020 0 1000 + 1000 gacactgaaggacgtGGACGTTATTTACCAACATCAAAGCACACAAATATAtggcacagaaac [ -clip - ]

Check this sequence with the sequence just upstream from this gene in the browser.

Test in the Genome Browser

• Zoom back in to <50,000 bp and find example areas of all annotation types (check against the maf file for that location):
  • pale yellow bar
  • green square brackets
  • vertical blue bar
  • gaps
• Note that it is much easier to see the yellow, green and blue areas when Multiz Alignments is set to "pack" in the track controls.