Chains and Nets QA

From genomewiki
Revision as of 22:15, 8 March 2010 by Vanessa (talk | contribs)
Jump to navigationJump to search


There are two components of chains/nets [1]: the actual chains/nets themselves and the liftOver directories/files. The liftOver [2] files make it possible to convert coordinates from one assembly to another. The liftOver files also have several associated files that will need to be pushed to hgdownloads at the end of the QA process. Chains and Nets are typically QAed in reciprocal pairs as similar tests are being done for both database-chain/net sets; however, this wiki only documents QAing one track.

Prior to QAing on hgwbeta: Pushing from hgwdev

1) Use sudo mypush to push chain/net tables from hgwdev to hgwbeta. This will include 3 tables: a chain, net and chain link table. Syntax for this command is: sudo mypush [from database] [tableName] mysqlbeta. See the example below:

  • hgwdev: sudo mypush danRer6 chainMm9 mysqlbeta
  • hgwdev: sudo mypush danRer6 netMm9 mysqlbeta
  • hgwdev: sudo mypush danRer6 chainMm9Link mysqlbeta

2) Make strict on beta so that trackDb is updated to include the tables just pushed. Syntax for this command is: make strict DBS=[from database].

  • ex: hgwbeta: make strict DBS=danRer6

3) Email a push request to copy /gbdb/[from database]/liftOver/[from database]To[to database].over.chain.gz from hgwdev to hgnfs1. Note that the liftover file will now be available on both beta and RR as it is on hgnfs1. However, because the hgcentral database on the RR doesn't have an entry pointing to this file, it will only be visible on beta.

  • ex: email: Please push /gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz from hgwdev to hgnfs1.

4) Add a line from hgcentraltest database to hgcentral beta so that file pushed in step 3 is visible on beta (this is similar to the same way that tables are not visible until trackDb is updated). To do this you first find the line on hgcentraltest.liftOverChain, copy it and then insert it into hgcentralbeta.liftOverChain. Note that it is very easy to make a mistake that can bring down hgcentral so make sure that your syntax and mysql statements are correct. See the example below:

  • hgwdev: hgsql hgcentraltest
  • mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;
  • (copy the output into a text editor for later use)
  • (open another window on hgwbeta) hgwbeta: hgsql hgcentralbeta
  • (put the values copied from hgcentral test into this statement. please note that the formatting must be followed exactly)

mysql> insert into liftOverChain values ("danRer6","mm9","/gbdb/danRer6/liftOver/danRer6ToMm9.over.chain.gz","0.1","0","0","Y","1","N");

  • (checking your work) mysql> select * from liftOverChain where fromDb=“danRer6” and toDb=“mm9”;

QAing on hgwbeta

To QA a chain/net file first run this script: chainNetTrio.csh [database] [other database].

  • ex: chainNetTrio.csh danRer6 mm9

This script runs several other scripts and outputs the results into several files. It is recommended that you create a separate directory to run this script in so you don't end up with the random output files in another directory. Read through the results in these three output files: *.chain.* , *.chain2.* , *.net.*. Mainly check that all the scripts tested ok and do a few manual checks that requested in these files.

There are two CGIs that use the liftOver data: hgConvert [3] and hgLiftOver [4]. To test these CGIs first select a top level short net that has very few other chains that match to the same area. To test hgConvert, first go to the genome browser of your "fromDb" database and center the window on the pre-selected net. Click on "Convert" link in the top blue bar. Configure the pull down menu to point to the "toDb" being tested and click "submit". The results should be the same as chain/net track details for that chain.

To test hgLiftOver first click on the "Utilities" link in the side blue bar on the home page and then click the first link to "Batch Conversion". Configure the pull down menus to match the toDB and fromDb being tested, enter the pre-selected test coordinates and click "submit". If the search fails, lower the minMatch (as low as 0.01) until some result for some region is obtained successfully. Check to see if the results match those from hgConvert and the chain/net track. Results in hgConvert may not match hgLiftOver but will still match the net and chains exactly. If results in hgLiftOver are NOT in hgConvert the developer should be contacted.

After QAing on hgwbeta: Pushing to the RR

1) Make strict on beta so that trackDb is completely up-to-date with anyone else's changes.

  • ex: hgwbeta: make strict DBS=danRer6

2) Email a push request to copy the three chain/net (chain, net and chain link) tables from mysqlbeta to mysqlrr and also trackDB.

  • ex: email: Please push the following tables from the danRer6 database: chainMm9, netMm9, chainMm9Link from mysqlbeta to mysqlrr

and

danRer6 database: trackDb from mysqlbeta to mysqlrr.

Note: for steps 1-3 you may have to do some sneaky/selective pushing that is different from the steps above if there is already a chain/net on the RR for an older assembly. You will need to drop tables for previous assemblies on Hgwbeta and then on RR.

  • ex: To find out what tables to drop use this command with the appropriate assembly names: hgsql -e "show tables like '%DanRer4%'" gasAcu1

Create a file with these tables. Add to the front of each line: "drop table" and at the end of each line add ";"

  • To do this easily in vi:

Let's say your file contains the following 4 lines:

  • chr1_chainDanRer4
  • chr2_chainDanRer4
  • chr3_chainDanRer4
  • netDanRer4

And you want to edit it to instead contain these 4 lines:

  • DROP TABLE gasAcu1.chr1_chainDanRer4;
  • DROP TABLE gasAcu1.chr2_chainDanRer4;
  • DROP TABLE gasAcu1.chr3_chainDanRer4;
  • DROP TABLE gasAcu1.netDanRer4;

First do this command:

  • %s/^/DROP TABLE gasAcu1./

Then, do this command:

  • %s/$/;/

3) Test the chain/Net track on the RR to make sure it is working properly.

  • If you need to dropped tables from hgwbeta, all that needs to be done is to copy all of the lines in the file that you created and drop them in the command line. Run this command again: hgsql -e "show tables like '%assembly%'" dbName

4) Add line to hgcentral.liftOverChain from hgcentralbeta.liftOverChain. Use the same steps listed in step 4 under "Prior to QAing on hgwbeta". Note that you do not need to push the liftOver file as the RR and hgwbeta both have access to hgnfs1.

  • To log into hgcentral from hdwdev: hgsql -h genome-centdb hgcentral

5) Test that hgLiftOver and hgConvert to make sure they are working as expected.

6) Email a push request to copy this file: /usr/local/apache/htdocs/goldenPath/[from database]/liftOver/[from database]To[to database].over.chain.gz and this directory: /usr/local/apache/htdocs/goldenPath/[from database]/vs[to database]/* from hgwdev to hgdownloads so that the public can download these files.

  • ex: email: Please push /usr/local/apache/htdocs/goldenPath/danRer6/liftOver/danRer6ToMm9.over.chain.gz and /usr/local/apache/htdocs/goldenPath/danRer6/vsMm9/* from hgwdev to hgdownloads.
  • If tables were dropped on hgwbeta, email pushers to drop the tables.

7) To make a link to these new download files change the downloads.html file.

  • hgwdev (in the browser tree):vi downloads.html
  • (find a link under "Pair-wise alignments" to a similar database. copy it, paste it and change it so that it points to the new vs directory)
  • hgwdev: cvs commit downloads.htm

8) Make alpha in browser tree so that the changes in downloads.html are now in htdocs.

  • ex: hgwdev (inside browser tree): make alpha

9) Email a push request to copy this file: /usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads, thus allowing the public to access the download files.

  • ex: email: Please push usr/local/apache/htdocs/downloads.html from hgwdev to hgdownloads

The end!

Retrieved from "https://genomewiki.ucsc.edu/index.php?title=Chains_and_Nets_QA&oldid=9629"