ThreeStateTrackDb

This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd. It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.

This addresses the following issues:

• Release tags currently do not prevent modifications to tracks already on the RR from leaking to the RR while in QA. This is caused by the release tag not accurately modeling the process. This is addressed by adding a third state: "public".
• Managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number of release tags in trackDb.ra stanza files. Also, the release tag doesn't currently work on the composite track stanza.

Updated trackDb release process

The three state trackDb (and hgFindSpec) release process is:

• alpha state: make alpha run on hgwdev loads all tracks with release alpha into trackDb and hgFindSpec regardless of the table existing.
• beta state: make beta run on hgwbeta loads all tracks with release beta into trackDb and hgFindSpec only if the table exists.
• public state: make public run on hgwbeta loads all tracks with release public into trackDb_public and hgFindSpec_public only if the table exists.

Having no release tag is equivalent to having all 3 release tags present.

QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR. Then trackDb_public and hgFindSpec_public will be pushed from mysqlbeta to mysqlrr, renaming them to trackDb and hgFindSpec.

Tracks that don't already exist on the RR function as they do now:

• Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
• QA pushes table to mysqlbeta, does a make beta, and QAs track
• When track is ready, QA does a make public, then pushes the tables and trackDb_public to mysqlrr, renaming them to trackDb

If a developer subsequently wants to make changes to the track, it would work like this:

• the old stanza:

track someTrack
shortLabel Mediocre RNAs
visibility hide

• becomes two stanzas:

track someTrack
release alpha
shortLabel Great RNAs
visibility pack

track someTrack
release beta,public
shortLabel Mediocre RNAs
visibility hide

• nothing leaks out to the RR before it is ready. QA looks at the changes on hgwbeta by changing the first stanza to release alpha,beta and the second stanza to release public

• when it is deemed worthy, the trackDb.ra entry can be collapsed back to one stanza, with no release tags (although leaving release alpha,beta,public in there would have the exact same effect):

track someTrack
shortLabel Great RNAs
visibility pack

Managing large composite tracks

To address this issue, each large composite track will be moved to its own file with all of its contained track stanzas. To minimize the amount of editing required, the include directive will be modified to have a release attribute. Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.

For example, if a developer created the file wgEncodeGencode.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):

include wgEncodeGencode.ra alpha

When QA gets it, this will become:

include wgEncodeGencode.ra alpha,beta

And when it is released:

include wgEncodeGencode.ra

(no release restrictions is the same as having: alpha,beta,public)

Updating existing public tracks

When changes need to be made to an already-released composite track, the composite trackDb file is copied to a new file name and it is added to git. So two copies of the entire file will exist (the *.new.ra having all the new changes/updates) and trackDb.ra will look like:

 include wgEncodeGencode.new.ra alpha
 include wgEncodeGencode.ra beta,public

When the track is in QA and staged on hgwbeta, this will become:

 include wgEncodeGencode.new.ra alpha,beta
 include wgEncodeGencode.ra public

To release the new version, QA will:

 cp wgEncodeGencode.new.ra wgEncodeGencode.ra

commit the changes. And change the include line back to:

   include wgEncodeGencode.ra

The changes won't leak to the RR before QA approves them and removes the release restrictions on the include directive (which is equivalent to a release restriction of alpha,beta,public).

Updating a track that is mid-QA

Often, a developer needs to start more work on a track before it is through the QA process and released and make the changes visible on genome-test for data contributors. In this case, a third file can be created and committed to git:

 include wgEncodeGencode.latestForContributor.ra alpha  #name is arbitrary
 include wgEncodeGencode.new.ra beta
 include wgEncodeGencode.ra public

When QA of the original update is complete that change released, the developer is free create a pushQ entry for it. QA can copy the latest version to wgEncodeGencode.new.ra when ready to move it to beta. Once the latest changes have been git committed to wgEncodeGencode.new.ra and there are no include directives for the arbitrarily-named wgEncodeGencode.latestForContributor.ra file, the latter can be git removed.

Existing super-tracks MUST use release tags

When you add new tracks to previously released super-tracks, the old paradigm of needing a table + trackDb entry in order for your track to be anything but completely invisible on the RR *does not apply*. So, if you are adding a new track to a super-track that is already on the RR, even though the tables are not yet on the RR, you must use release tags.

Currently, if you do not use release tags and add a new track to a super-track, users will be able to see your new track listed on the super-track control page. However, when the user tries to turn it on, they won't see anything. When the user tries to go to the hgTrackUi page, they will get an error.

This issue is documented in redmine: http://redmine.soe.ucsc.edu/issues/2046.

Migration Plan

The goal of the migration plan is to minimize any disruption in tracks currently being developed.

Changes that can be made while the current release model continues to function:

• Check for any existing tables were tracks have overlapping tags. For instance, there are composite tracks where some tables have the beta tag and some have no tag. Correct any that are found.
• Add generic release tag changes to the library code and hgTrackDb program
• Modify all instances of release beta to be release beta,public.
• Modify trackDb/makefile where make beta requests the beta,public tags
• Setup hgwbeta-public
• Convert ENCODE composite tracks to one per include.
• edit any trackDb that is staged on hgwbeta but not yet released back to release beta (remove the public)

At this point, all mechanism are in place

• replace make strict with make beta and make public
• switch to new methodology