Requests for liftOver files

Overview

The MLM FAQ on LiftOver files shares some guidelines:

liftOver files for assemblies with no browser

We do not provide liftOver files for assemblies for which we do not have at least a minimal browser. If a user requests such a file, please point them to this wiki page and wish them luck!

liftOver file requests for assemblies with a browser

When a specific liftOver file is requested from a user for an existing browser, we should definitely consider making it. It should go through the usual process (make a redmine, assign to an engineer, place in pushQ, QA and release). QA can do a minimal QA job on it (does it work? does it show up?) before pushing it to hgdownload and enabling it on the RR. There is no need to create companion net/chain tracks (unless the user specifically asked for that). The MLM who received the question is not responsible for QAing the liftOver file (it should just go into the pushQ like all data). After it is released, whoever QAd it should contact the user to let them know that it's available.

Here is a quick review of the process, covered below on this page:

push the files in /gbdb/ on hgwdev to hgnfs1

These files have the same name and live in a similar place as hgdownload-test (/$db/liftOver/$db1To$db2.over.chain.gz/), but are what is used by the CGIs to allow lifting.

edit hgcentral on beta and the RR once gbdb goes through

The lines on dev in hgcentraltest that allow the /gbdb/ files to load in the CGIs need to added to hgcentralbeta and hgcentral. This is probably the most detailed part as it requires careful manipulation of hgcentral on the RR.

push the files from hgdownload-test to hgdownload

To make the files available for download, a push of the files to hgdownload is also needed.

Push the files in /gbdb/ on hgwdev to hgnfs1

Here is an example push email where mm10 <---> danRer10 liftOver files and rn6 <---> danRer10 liftOver files and hg19 <---> danRer10 files are being released.

Hi Pushers,

Please push:

/gbdb/mm10/liftOver/mm10ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToMm10.over.chain.gz
/gbdb/rn6/liftOver/rn6ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToRn6.over.chain.gz
/gbdb/hg19/liftOver/hg19ToDanRer10.over.chain.gz
/gbdb/danRer10/liftOver/danRer10ToHg19.over.chain.gz

      from hgwdev --> hgnfs1/genome-euro/-asia

Note that items that are symlinked on hgwdev should become real files on hgnfs1.

Reason: Releasing LiftOvers as part of MLQ ######

Thanks,

This push will make the files available on both hgwbeta and the RR as hgnfs1 is shared by both of them.

Edit hgcentral on beta and the RR once gbdb push goes through

Note that these examples must be edited to the appropriate databases and also be updated in both directions. In the below example danRer6 <---> mm9 is being released, but first in the danRer6 --> mm9 direction, it will need to be repeated a second time in the mm9 --> danRer6 direction.

Acquire the hgwdev entry

 hgsql -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentraltest > liftOver.dev

This will save the danRer6 --> mm9 lines to a file called liftOver.dev

Check that the lines do not exist on Beta

hgsql  -h mysqlbeta  -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentralbeta

This will query beta and should come back empty indicating there are no lines like the ones. We will next populate the lines using what we took from dev.

Provided lines don't exist, load lines from dev on Beta

hgsql -h mysqlbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta

Check behavior on Beta

At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9

Check that lines do not exit on RR (before loading)

hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='danRer6' and toDb='mm9';" hgcentral

This will query beta and should come back empty indicating there are no lines like the ones. We will next populate the lines using what we took from dev.

Provided lines don't exit, load lines on the RR

hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral

Check behavior on the RR

At this point you should be able to go to hgLiftover and hgConvert and see things in operation on hgwbeta, where in this example danRer6 will have an option to lift coordinates to mm9.

Repeat the process in the reverse direction

Now reverse the names and repeat the process. I suggest removing liftOver.dev before starting.

For reference here are the steps (note that everywhere danRer6 has been replaced by mm9 and vice-versa and the load steps are the same). Acquire:

hgsql -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentraltest > liftOver.dev

Check it doesn't exist on beta

hgsql  -h mysqlbeta  -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentralbeta

Load on Beta (provided it doesn't exist)

hgsql -h mysqlbeta -e "load data local infile 'liftOver.dev' into table liftOverChain" hgcentralbeta

Check it doesn't exist on RR

hgsql  -h genome-centdb -Ne "select * from liftOverChain where fromDb='mm9' and toDb='danRer6';" hgcentral

Load on RR (provided it doesn't exist):

hgsql -h genome-centdb -e "load data local infile 'liftOver.dev' into table liftOverChain;" hgcentral

Push the files from hgdownload-test to hgdownload

Send a push-request to release the files to hgdownload:

Hi Pushers,

Please push the following download file(s):

/usr/local/apache/htdocs-hgdownload/goldenPath/galGal4/liftOver/galGal4ToHg19.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/galGal4/liftOver/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/hg19ToGalGal4.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/md5sum.txt

   from hgwdev --> hgdownload
         (in the path, "htdocs-hgdownload" should become "htdocs")

Reason: Adding some user requested files to hgdownload, refs #####

Thanks,

Note that it is probably good to include the md5sum.txt, but that the md5sum.txt may have additional files only seen on dev, this is alright. If users email about these additional files, we can always point them to genome-preview and hgdownload-test, or move on releasing the liftOver files if browsers exist on the RR.

Note about *Not Required* Pairwise Files

When doing a release for a mailing list request, there isn't a mandate to release the related Pairwise files, unless asked. If you do push those files to hgdownload an added step will be needed to update downloads.html where there are Pairwise Alignments sections under each species and lines like Human/Chicken (galGal3) that point to a hg19/vsGalGal3 directory that needs pushing (and the link needs creation on downloads.html). The gbdb steps would remain unchanged.

If you were pushing these files, here is an example hgdownload push that inlcudes the vs$Db directories, in this example danRer10 <---> hg19. But note we don't need to do this unless requested.

/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToHg19.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToRn6.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/liftOver/danRer10ToMm10.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/*
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/reciprocalBest/*
/usr/local/apache/htdocs-hgdownload/goldenPath/danRer10/vsHg19/reciprocalBest/axtRBestNet/*

/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/md5sum.txt
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/liftOver/hg19ToDanRer10.over.chain.gz
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/axtNet/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/reciprocalBest/*
/usr/local/apache/htdocs-hgdownload/goldenPath/hg19/vsDanRer10/reciprocalBest/axtRBestNet/*'

And here is a reminder to edit the downloads.html page and push that as well:

Hi Pushers,

Please push the following file from hgwdev:
   /usr/local/apache/htdocs-hgdownload/downloads.html

   to same location on hgdownload *except*
   replace "htdocs-hgdownload" with "htdocs" in the directory path

Reason: Adding links to hg19 and danRer10 pairwise directories, refs #####