ThreeStateTrackDb

This is a proposal for modifying the trackDb release process developed by Brian, Brooke, and Markd. It is intended as a straight-forward modification to the existing process to address some of the current problems; not as a major, comprehensive change.

Issue #1: release tag does not prevent modifications to tracks already on the RR from leaking to the RR while in QA. This is caused by the release tag not accurately modeling the process. This is addressed by adding a third state: "public".

The three states would function like so:

alpha state:

'make alpha' run on hgwdev loads all tracks with 'release alpha' into trackDb regardless of the table existing.

beta state:

'make beta' run on hgwbeta loads all tracks with 'release beta' into trackDb only of the table exists.

public state:

'make public' run on hgwbeta loads all tracks with 'release public' into trackDb_public only of the table exists.

Having no release tag is equivalent to having all 3 release tags present.

QA can look at trackDb_public on hgwbeta-public.cse.ucsc.edu as a final check before pushing it to the RR. Then trackDb_public will be pushed from mysqlbeta to mysqlrr, where it will be renamed trackDb.

A normal, "old school" track will function in much the same way as it functions now:

• Developer creates table(s) and adds a corresponding stanza to trackDb.ra, with no release tags
• QA pushes table to mysqlbeta, does a 'make beta', and QAs track
• When track is ready, QA does a 'make public', then pushes table and trackDb_public* to mysqlrr (and trackDb_public is renamed trackDb along the way)

If a developer subsequently wants to make changes to the track, it would work like this:

• the old stanza:

track someTrack
shortLabel Mediocre RNAs
visibility hide

• becomes two stanzas:

track someTrack
release alpha
shortLabel Great RNAs
visibility pack

track someTrack
release beta,public
shortLabel Mediocre RNAs
visibility hide

• nothing leaks out to the RR before it is ready. QA looks at the changes on hgwbeta by changing the first stanza to 'release alpha,beta' and the second stanza to 'release public'

• when it is deemed worthy, QA collapses the trackDb.ra entry back to one stanza, with no release tags (although leaving release=alpha,beta,public in there would have the exact same effect):

track someTrack
shortLabel Great RNAs
visibility pack

Issue #2: managing releases of updated versions of ENCODE composite tracks is complex, requiring editing a potentially large number release tags in trackDb.ra stanza files. The release tag doesn't currently work on the composite track stanza.

To address this issue, each large composite track will be moved to its own file, named in the form: trackDb.compositeName.version.ra, and include all of its contained track stanzas. To minimize the amount of editing required, the include directive will be modified to have a release attribute. Since includes are processed line-per-line, not as part of a stanza, an attribute is an easier approach than adding a release tag.

For example, if a developer created the file trackDb.encGencode.1.ra, the following line could be added to trackDb.ra (or trackDb.wgEncode.ra, or whatever):

include trackDb.encGencode.1.ra alpha

When QA gets it, this will become:

include trackDb.encGencode.1.ra alpha,beta

And when it is released:

include trackDb.encGencode.1.ra alpha,beta,public
(NOTE TO MARKD/BRANEY: should there be no attribute instead??)

When changes are made to an already-released composite track, the file is copied, the version number in the name is incremented, and it is added to cvs. When a developer needs to change something in trackDb, two copies of the entire file will exist, and trackDb.ra will look like:

 include trackDb.encGencode.2.ra alpha
 include trackDb.encGencode.1.ra beta,public

The changes won't leak to the RR until QA approves them and promotes the version 2 file to alpha,beta,public. Once an older version of a composite track is no longer on any systems, the corresponding .ra file can be cvs removed.

Note that the version numbers here are just sequential, and they do not necessarily correspond to the ENCODE "V2" track releases.