GRC Patch Release

Introduction

The GRC Patch Release track is different from other tracks in that you are technically also releasing the patch assembly to the RR, though nobody will ever actually see the assembly (ie. it will not be visible from hgGateway). The release of the patch assembly is what allows the links in the hgc page to display the DNA from the patched assembly. Note that in the making of an assembly, there are many tables and files that are automatically created that are not necessary for this track to function, thus we only push certain parts, not the whole assembly.

These are the extra steps you will need to take in addition to the normal Checklist steps for pushing to beta of tables and associated files. The examples below assume you will be on hg19, some push request emails are for mm10.

Stage the track on beta

1. Make the patch database on beta:

hgwbeta> hgsql
hgwbeta> create database hg19Patch#;

2. Push the chromInfo table to hgwbeta

hgwdev> sudo mypush hg19Patch# chromInfo hgwbeta

Note: The patch database has the sole purpose of containing the chromInfo table, which points to the 2bit, so that users can click on a patch, going to hgc, where they can then 'get DNA' for the patch. Be sure to test this for a patch which is unique to your new patch version.

3. Add the necessary lines to hgcentralbeta.dbDb

hgwdev> hgsql -Ne 'select * from dbDb where name like "hg19Patch#"'  hgcentraltest > dbDbHg19Patch

• The above My SQL command will create the file dbDbHg19Patch (-N strips column names, -e executes commands and then quits).
• From the same directory you ran that command you need to edit it, for example hgwbeta> emacs dbDbHg19Patch.
• Change the active column to be 0 rather than 1, this will prevent the database from showing on hgGateway.

  mysql> select * from dbDb where name like "hg19Patch10"\G
  *************************** 1. row ***************************
         name: hg19Patch10
  description: Sep. 2012 (GRCh37.p10/hg19Patch10)
      nibPath: /gbdb/hg19Patch10
     organism: Grch37.p10
   defaultPos: chr1_jh636052:3604121-3614120
       active: 1
     orderKey: 145
       genome: GRCh37.p10
     scientificName: Homo sapiens
     htmlPath: /gbdb/hg19Patch10/html/description.html
     hgNearOk: 0
       hgPbOk: 0
   sourceName: GRCh37 Patch 10 Genome Reference Consortium Human Reference 37
        taxId: 9606
 >>This is just showing you what information is contained in hg19Patch10

Change the 6th column "active:" for the created file dbDbHg19Patch from "1" to 0:

 hg19Patch10	Sep. 2012 (GRCh37.p10/hg19Patch10)	/gbdb/hg19Patch10	Grch37.p10	chr1_jh636052:3604121-3614120	1	
 hg19Patch10	Sep. 2012 (GRCh37.p10/hg19Patch10)	/gbdb/hg19Patch10	Grch37.p10	chr1_jh636052:3604121-3614120	0	
>>Scroll to the right to see change from 1 -> 0, make this change before uploading to hgcentralbeta.

Save the change (AND KEEP the changed file dbDbHg19Patch for RR Release) and use the below command to upload the edited file to hgcentralbeta database into table dbDb:

hgwdev> hgsqlbeta -e "load data local infile 'dbDbHg19Patch' into table dbDb" hgcentralbeta

4. Add the necessary lines to hgcentralbeta.genomeClade

hgwdev> hgsql -Ne 'select * from genomeClade where genome like "GRCh37.p#"' hgcentraltest > genomeCladeHg19Patch

Running the above command on dev will pull GRCh37.p10 information for example and create the file genomeCladeHg19Patch (In the above example "GRCh37.p#" could be named something different on a different assembly).

hgwdev> hgsqlbeta -e "load data local infile 'genomeCladeHg19Patch' into table genomeClade" hgcentralbeta

Run the above from the same directory on beta to upload the information to hgcentralbeta database into table genomeClade (AND KEEP the temporary file genomeCladeHg19Patch for RR Release).

5. Edit lines in trackDb/human/hg19/trackDb.ra NOTE:This step may not be necessary

include altSeqComposite#old.ra public  
include altSeqComposite#new.ra alpha,beta  <== add beta to new patch so it shows in beta NOTE:This step may not be necessary

6. Ask for push of /gbdb/hg19Patch#/hg19Patch#.2bit and /gbdb/hg19/hg19Patch#/* (which should contain the assembly in .fa file format) from hgwdev to hgnfs1.

• See example push request below.
• The following Sunday, the files should be on hgdownload. This can be pushed earlier if needed.

7. Follow Checklist steps for pushing to beta to move track tables to hgwbeta either individually or creating a list of tables with a text editor and using bigPush.csh (skip already done gbdb push).

sudo mypush $db $table hgwbeta

or, for a list of tables:

bigPush.csh $db $tableListFile

8. Do a make beta of your database on hgwbeta (only for your main db, not the patch db)

hgwbeta> make beta DBS=hg19

9. Reminder to run joinerCheck. If the file is edited, be sure to push it also. However, all.joiner is pushed with every CGI release, so you could also wait and let that push go out instead of requesting the push it yourself.

Good output example:

 ★  ~/kent/src/hg/makeDb/schema
joinerCheck -database=hg38 -keys -identifier=chainHg38Patch9Id all.joiner
Checking keys on database hg38
 hg38.chainHg38Patch9.id - hits 20372 of 20372 (100.000%) ok
 hg38.chainHg38Patch9Link.chainId - hits 129018 of 129018 (100.000%) ok
 hg38.chainHg38Patch9Link.chainId - hits 129018 of 129018 (100.000%) ok
 hg38.netHg38Patch9.chainId - hits 86 of 86 (100.000%) ok

Bad output example:

★  ~/kent/src/hg/makeDb/schema
joinerCheck -database=hg38 -keys -identifier=chainHg38Patch11Id all.joiner
Checking keys on database hg38
Identifier chainHg38Patch11Id not found in all.joiner

Push the track to the RR

IMPORTANT!! do this before pushing any of the other tables or files

1. Ask the admins to rsync the hg19Patch# database to the RR from hgwbeta. Add a reminder line to update euroNode, as rsyncs are unscripted updates.

• See rsync push request below.
• The patch db should only contain one table, chromInfo. If others are pushed, no problem, just request that tables other than chromInfo are dropped.

2. Add the necessary lines to hgcentral.dbDb, using the same dbDbHg19Patch file that you used to load the lines into hgcentralbeta.dbDb

hgwdev> hgsql -h genome-centdb -e "load data local infile 'dbDbHg19Patch' into table dbDb" hgcentral

3. Add the necessary lines to hgcentral.genomeClade, using the same genomeCladeHg19Patch file that you used to load the lines into hgcentralbeta.genomeClade

hgwdev> hgsql -h genome-centdb -e "load data local infile 'genomeCladeHg19Patch' into table genomeClade" hgcentral

4. Change trackDb.ra to include public for the new patch and remove public for the old patch, mentioned in steps above, and on hgwbeta> make public $db NOTE: if you make public before requesting the rsync, you'll need to let admins know you don't want to push created _public files, so best to wait.

5. Do a regular push request of tables and trackDb and friends to release to the RR.

Drop old patch# tables from beta and from the rr

• Patches are cumulative, all patch data in a new patch includes previous patches.
• Only the current patch database and the current patch tables should reside on beta and rr.
• Drop ALL old patch tables and patch databases, other than the current patch, from beta and rr.
• Related RM with example sql queries: http://redmine.soe.ucsc.edu/issues/20139
• https://www.ncbi.nlm.nih.gov/grc/help/patches

1. Drop all old patch DATABASES (e.g., hg19Patch9) from beta AND the rr/euro/asia >show databases like "%atch%";

2. Drop all old patch tables for your assembly (e.g., for hg38, drop the old table, "chainHg38Patch3") from beta AND the rr/euro/asia >'show tables like "%atch%";' hg38

A note about defaultDb

Summary: It is not necessary to add new patches to defaultDb. And there where 5 entries removed in April 2019. Also, this whole page may be much less relevant given the change of the system of releasing patches.

Example push requests

Below are example push requests from 2017. Push requests were done in the following order:

*************************************************************
DONE
subject: Push gbdb files to hgnfs1 for GRCh38.p11

Hi Pushers,

Please push the following files:
 
/gbdb/hg38/hg38Patch11/*
/gbdb/hg38Patch11/hg38Patch11.2bit  

from hgwdev --> hgnfs1/euro/asia
(Note: Items that are symlinked on dev should become real files on hgnfs1.)

Reason:  Staging GRCh38.p11 on beta, refs #19665

*************************************************************
DONE
subject: rsync hg38Patch11 database

Hi Pushers,

Please rsync the following database:
 
hg38Patch11
 
from hgwbeta --> mysqlrr/euro/asia

Reason: Staging GRCh38.p11, refs #19665

*************************************************************
REQUESTED BUT WAS NOT NEEDED AND DIDN"T WORK :)
DON"T DO THIS...
NO NEED TO DO MAKES ON THE PATCH DBS.

subject: rsync hg38Patch11 database

Hi Pushers,

Can you also push trackDb and friends for hg38Patch11?

After the push, can you tell me the output of  'show tables' for db hg38Patch11?

Reason: Adding trackDb and hgFindSpec (search capability) for GRCh38.p11 for rr release, refs #19665


*************************************************************
DONE

subject: Push files to hgdownload for GRCh38.p11

Hi Pushers,

Please push the following files:

/usr/local/apache/htdocs-hgdownload/goldenPath/hg38/hg38Patch11/*
/usr/local/apache/htdocs-hgdownload/downloads.html

from hgwdev --> hgdownload
NOTE: In the path, "htdocs-hgdownload" should become "htdocs"

Reason:  Releasing download files for GRCh38.p11, refs #19665


*************************************************************
DONE
subject: Push hg38 tables for GRCh38.p11

Hi Pushers,

Can you please push the following tables,

on database hg38:
 
chainHg38Patch11
chainHg38Patch11Link
extHg38Patch11
hg38Patch11Haplotypes
hg38Patch11Patches
netHg38Patch11
seqHg38Patch11
altSeqLiftOverPslP11

from hgwbeta----->rr/euro/asia

Reason:  Staging release for GRCh38.p11, refs #19665


*************************************************************
DONE
subject: Push hg38 trackDb & friends for GRCh38.p11

Hi Pushers,

Can you please push trackDb and friends for the following database:

hg38

Reason:  Releasing GRCh38.p11 after addition of public .ra release tag on trackDb/human/hg38/trackDb.ra and removal of public tag on the last p.9, refs #19665


*************************************************************
DONE
subject: Drop tables for GRCh38.p9

Hi Pushers,

Please drop the tables below for the following database:

hg38

tables to drop:

altSeqLiftOverPslP9
chainHg38Patch9
chainHg38Patch9Link
extHg38Patch9
hg38Patch9Haplotypes
hg38Patch9Patches
netHg38Patch9
seqHg38Patch9

from mysqlrr/euro/asia

Reason: Dropping GRCh38.p.9 now that GRCh38.p.11 is released, refs #19665

*******************************************************************
DONE
subject: Drop tables for GRCh38.p11

Hi Pushers,

Please drop the tables below for the following database:

hg38Patch11

tables to drop:

hgFindSpec
hgFindSpec_public
trackDb
trackDb_public

from mysqlrr/euro/asia

Reason: Dropping unnecessary tables from hg38Patch11 db, only chromInfo is needed, which points to the 2bit to get p11 sequences refs #19665