View source for Graph data format discussion

The wiggle format designed by UCSC for graphical display in
the browser provides high performance and configurability to
the user.  However, it does not maintain the original data
beyond a few digits of precision.  Although the majority of
data does not have significance beyond these digits, it has
been a point of confusion to the users when the data extracted
from a UCSC wiggle table or file does not match the submitted data.

It has been proposed (e.g. in the ENCODE DCC grant proposal) that
we design a new graphing data format that will preserve the 
submitted data exactly. Lacking this, we have promised to 
provide for download an additional copy of each ENCODE dataset as originally
submitted.  This has the disadvantage of doubling the disk space
required, and also raises the risk of out-of-sync copies on 
the site (already a problem with our existing wiggle downloads).

A few thoughts on what a new data format might look like (from
recent discussions) are included here:

* A simple binary file that represents each data point in 16 bits would capture all the precision necessary.

* Memory loading for this would be efficient, but to assure adequate performance for large regions (e.g >1MB) ,a summary table or file could be precomputed using our existing graph averaging methods (e.g. mean, median, max).

* The existing ascii input wiggle formats could be maintained -- wigEncode could be expanded to generate the simple graph binary format instead of wiggle binary.

* There is already existing code with similar functionality in hgGenome.

* NOTE: Nearly all of the high-density ENCODE graphing data provided so far fits one of two patterns:
   - one value per base across the genome (or ENCODE region)
   - fixed-width regions (e.g. 22mers) tiled across the genome
Variable-width data (e.g. sites) tends to be sparser.

Some design issues are:

* run-length encoding - how to specify that a number applies to N bases
* sparse data
* fully populated, every base, data

And questions to answer:

* Does it participate in hgc click-throughs?
* Can it be a custom track?
* How does it function in the table browser?
* Which correlation functions there?
* How do you get it to be fast enough for a browser display on a whole chrom?
* Are there different graphing options, bar graph, points graph, line through points?

[[Category:ENCODE]]