HgTracks multi-region changes

This page provides an overview of the changes made to hgTracks and the position cart variable by Galt when implementing multi-region (a.k.a. "exon-mostly") display mode. It began as part of code review #16514 by Angie. Instead of adding these notes to the Redmine ticket, I am putting them in the wiki so that Galt can correct any of my misunderstandings, and hopefully this will serve as a rough guide for any developers who will need to interact with this code in the future.

The big merge of code changes for the initial implementation of multi-region display is 8c908f948b; see also http://genecats.cse.ucsc.edu/git-reports-history/v327/review/user/galt/index.html

hgTracks has always operated on the assumption that the user is viewing a single region of some reference assembly sequence. There are global variables for that region and global variables for layout parameters of a single region image. These global variables are referenced throughout all tracks' drawing code. Track loaders fetch data from a single region. Galt implemented multi-region display in a way that barely touched the bulk of hgTracks code -- quite an accomplishment! -- partly by introducing new structures and loops, and partly by carefully overwriting the pre-existing global variables. Developers working on loading and drawing code in the future can mostly ignore those changes and continue using the old globals in the same way as before. However, there may be cases in which track handlers will have to make use of data from all regions (for example, wiggle auto-scale). Developers who add a new global variable will have to decide whether it needs to be updated every time the current region changes. But hopefully they'll not add any more globals.  :)

Multi-region display UI

There is a new button underneath the browser image, labeled 'multi-region'. Clicking it produces a pop-up with options for multi-region display. In the initial version there is a radio button set as shown below, followed by a checkbox that controls whether the regions are visually separated by thin red lines (the default) or highlighting alternating regions.

(*) Show single-chromosome view (default)

( ) Show exons or ( ) genes using GENCODE v22. Use padding of: [6] bases.

( ) Custom regions from BED URL: [ ]

( ) Show one alternate haplotype, placed on its chromosome, using ID: [ chr1_KI270762v1_alt ]

[ ] Highlight alternating regions in multi-region view

The 'Show exons or genes' line appears only when a suitable gene table can be identified. The 'Show one alternate haplotype' option appears only when there is an altLocations sql table.

Within the four main options, there are additional parameters:

• a fifth mode option to omit regions between genes instead of between exons; default unselected
• the padding option for exon or gene mode; default 6
• the URL for a BED file (with optional multi-region-specific settings) provided by the user; default empty
• an alternate haplotype ID; default is "chr6_cox_hap2" or if not applicable, the first item in the altLocations table

There are also two new keyboard bindings: 'e v' changes the mode to 'Show exons' and 'd v' changes the mode to single-region.

The code includes several additional modes not shown above, mostly "demo" modes for testing during development. Since those are not reachable by users, I won't describe those.

Changes to the position cart variable (and hence other CGIs)

Before multi-region, the position cart variable contained either seq:start-end or a search term that must be resolved into seq:start-end. seq was always a chromosome/scaffold name that could be found in the current db's chromInfo table, and had start and end coordinates relative to that chromosome/scaffold. Several other CGIs (such as the Table Browser, Gene Sorter, DI, VAI) assumed this formatting of position. Now, when viewing custom regions from the user or an alt haplotype, position's value begins with virt and its start and end coordinates are for a "virtual chromosome" constructed by joining all regions.

The virt:start-end form does not (yet) make sense to any other CGI besides hgTracks, so some code has been added to the beginning of each CGI that uses the position variable to detect virt: at the beginning; if found, it swaps in the value of a new cart variable, nonVirtPosition.

Code changes

Needless to say, changing a paradigm in hgTracks involves adding and changing a lot of code. This is a high-level overview, not a comprehensive list.

New cart variables

virtMode

Boolean: true if we are not in single-region mode.

virtModeType

One of {"default", "exonMostly", "geneMostly", "customUrl", "singleAltHaplo" and some others not accessible by UI}. This stores the user's choice of mode in the multi-region pop-up.

multiRegionsBedUrl

The user's URL for custom regions (user sets in pop-up)

singleAltHaploId

The name of the alternate haplotype sequence used in alt-haplo mode (user sets in pop-up)

emAltHighlight

If true, alternating regions are differentiated by highlighting instead of red lines. (checkbox in pop-up)

emPadding

The number of bases to add on each side of each exon/gene when making the region list in exonMostly/geneMostly mode (user sets in pop-up)

emGeneTable

The table used to make regions in exonMostly/geneMostly mode. Currently this is set by the code, not the user.

lastVirtModeType

Stored in cart with the same value as virtModeType, for detecting changes to virtModeType.

lastVirtModeExtraState

CGI-encoded sequence of several mode-specific parameters for detecting changes:

• singleAltHaploId (see above)
• multiRegionsBedUrl: the name and file modification date of the user's URL for custom regions
• singleTransId: ignoring here because the singleTrans mode is not UI-accessible

virtModeShortDescr

Typically one word like "exons" or "genes"; this appears in the ideogram. This is inferred from the mode except when in customUrl mode if the user specifies one.

nonVirtPosition

When in multi-region mode, the start and end of all regions on the first region's chromosome. For use by other CGIs when position begins with "virt".

oldPosition

For detecting changes in position that somehow are not caught by pre-existing cart var lastPosition.

position.db

Not just a position -- this stores pretty much all of the above, CGI-encoded, in case we change db and then return to this db.

New data structures in hgTracks.h

struct virtRegion

This maps a particular region to its place in the reference assembly.

struct virtRegion
/* virtual chromosome structure */
   {
   struct virtRegion *next;
   char *chrom;
   int start;
   int end;
   char strand[2];	/* + or - for strand */
   };

struct virtChromRegionPos

This is one member of an array; the members of the array contain successive offsets within the "virtual chromosome" for each region.

struct virtChromRegionPos
/* virtual chromosome region position*/
   {
   long virtPos;
   struct virtRegion *virtRegion;
   };

struct positionMatch

This is bed3 without the chrom, i.e. a start-end range.

struct positionMatch
/* virtual chroom position that matches or overlaps search query chrom,start,end */
{
struct positionMatch *next;
long virtStart;
long virtEnd;
};

struct window

TODO

struct window  // window in multiwindow image
   {
   struct window *next;   // Next on list.

   // These two were experimental and will be removed soon:
   char *organism;        /* Name of organism */
   char *database;        /* Name of database */

   char *chromName;
   int winStart;           // in bases
   int winEnd;
   int insideX;            // in pixels
   int insideWidth;
   long virtStart;         // in bases on virt chrom
   long virtEnd;

   boolean regionOdd;      // window comes from odd region? or even? for window separator coloring

   struct track *trackList;   // track list for window
   };

struct convertRange

TODO

struct convertRange
   {
   struct convertRange *next;
   char *chrom;
   int start;
   int end;
   long vStart;
   long vEnd;
   boolean found;
   boolean skipIt;
   };

New global variables

Many new global variables were added, following the convention of declaring in hgTracks.h and initializing in simpleTracks.c. However, several of these are used only within hgTracks.c.

boolean virtMode

Cart variable, see above.

char *virtModeType

Cart variable, see above. Default: "default" (i.e. not multi-region)

char *lastVirtModeType

Cart variable, see above.

char *virtModeShortDescr

Cart variable, see above.

char *lastVirtModeExtraState

Cart variable, see above.

char *multiRegionsBedUrl

Cart variable, see above. Default: ""

boolean emAltHighlight

Cart variable, see above.

int emPadding

Cart variable, see above. Default: 6

char *emGeneTable

Cart variable, see above.

struct cart *lastDbPosCart

Not a cart! struct cart is used to CGI-decode the values that were encoded in lastVirtModeExtraState into a hash. Only the hash is used.

char *virtModeExtraState

Not a cart variable, but it becomes the next value of lastVirtModeExtraState, i.e. it encodes mode-specific parameters such as singleAltHaploId and multiRegionsBedUrl (plus mod date)

struct virtRegion *virtRegionList

List of regions that are joined to make the virtual chromosome

struct virtChromRegionPos *virtChrom

Array of successive regions and offsets into the virtual chromosome

int virtRegionCount

Number of regions in virtual chromosome and size of the array virtChrom

long virtSeqBaseCount

Number of bases in virtual chromosome

long virtWinStart, virtWinEnd

Start and end of the portion of the virtual chromosome that is currently displayed.

long virtWinBaseCount

The length of the currently display portion of the virtual chromosome. Set to virtWinEnd - virtWinStart every time virtWinEnd and virtWinStart are changed.

long defaultVirtWinStart, defaultVirtWinEnd

Used only when transitioning from default mode into singleAltHaplo mode: the start/end within the virtual chromosome of the alt haplo plus flanks of equal length on the main chromosome (trimmed to start/end of main chromosome). In singleAltHaplo mode, the virtual chromosome is constructed by concatenating all assembly sequences (!) and then replacing the main chromosome sequence with the part of the main chromosome preceding the alt haplo, then the alt haplo, and then the rest of the chromosome after the alt haplo.

char *virtChromName

When in multi-region mode, virt; otherwise just the good old chromName.

boolean virtChromChanged

True only when changing from multi-region mode into another, or when a parameter change is detected using lastVirtModeExtraState; not used by C code, but is passed forward to JS code.

struct track *emGeneTrack

The struct track for gene table for exonMostly/geneMostly

struct rgbColor vertWindowSeparatorColor

Constant: { 255, 220, 220} (light red for vertical lines between regions)

char *singleAltHaploId

Default: "chr6_cox_hap2"

struct window *windows

// list of windows in image

struct window *currentWindow

// current window

bool trackLoadingInProgress

// flag to delay ss layout until all windows are ready.

int fullInsideX, fullInsideWidth

Full-image insideX and insideWidth, for the few tracks that need to know the offset/width of the whole image, not just the offset/width of the current region's slice.

char *singleTransId

Not described here because singleTrans mode is not UI-accessible

int demo2NumWindows, demo2WindowSize, demo2StepSize

Not described here because demo2 mode is not UI-accesible