Genome-preview machine

In early 2011, we created a new machine called genome-preview, which is a mirror of our development machine: genome-test.

It is here: http://genome-preview.soe.ucsc.edu/

It has these warnings on the home page:

WARNING: This is our preview site. This website is a weekly mirror of our internal development server for public access. Data and tools here are under construction, have not been quality reviewed, and are subject to change at any time. We provide this site for early access, with the warning that it is less available and stable than our public site. For high-quality reviewed annotations on our production server, visit http://genome.ucsc.edu.

In general, we do not push updates/files/tables/patches to this machine by hand. It is generally understood that it will sometimes be out of commission, as it is a mirror of a development machine.

The impetus behind this machine was to allow early access to the ENCODE data by ENCODE consortia members.

Automated Updates

The genome-preview machine is updated on a weekly basis like so: On Friday Mornings (5:27 am) a build crontab makes the CGIs for genome preview. On Friday nights (11:33 pm), the following stuff is copied from hgwdev:

• mysql tables (excluding mysql, customTrash, and other things)
• hgcentraltest tables (excluding userDb, sessionDb, namedSessionDb and hubStatus)
• CGIs
• htdocs

Most of the stuff on /gbdb comes from backups of the /hive/data and /hive/groups.

• /hive/data is backed up on Fridays at 5:22 am
• /hive/groups is backed up on Wednesdays at 5:22 am.

Note that this may cause some inconsistencies if new data is added on Thursday to /hive/groups and accessed by new CGIs, since the data will not be available on the backups until the following Wednesday.

Updating Genome-Preview with pushDevToPreview

There is a really easy, automatic way to update Preview that combines all the below steps into one script:

/cluster/home/dschmelt/danScripts/previewUpdate

If there is a need to sync the CGIs on hgwdev to genome-preview the pushDevToPreview script will do it (from the qateam@hgwdev identity).

 pushDevToPreview -h
Usage: pushDevToPreview [-chiqvy] 

  -c | --check    : Don't do the push, just say what would be pushed.
  -h | --help     : Print this message.
  -i              : Interactive mode. Ask me before doing any command.
  -q | --quiet    : Quiet mode. Don't print anything unless necessary.
  -v | --verbose  : Increase verbosity.
  -y | --noprompt : Just do it, don't ask me.

PREPARATION from your hgwdev kent repository clean your tree and update your libraries.

If your libraries are out of date or your repository is not clean you can run into errors when making your CGIs:

cd ~/kent/src
make clean > /dev/null
cd ~/kent/src/hg/lib
git pull
make

NOW from your hgwdev kent repository complete the below steps to update CGIs/js/style files.

Note: you'll first need to update the js files for preview at: ls -l /usr/local/apache/htdocs/js/preview/

cd ~/kent/src/hg/js
git pull
make USER=preview

Note: you'll first need to update the style files for hgwdev at: ls -lrt /usr/local/apache/htdocs/style

cd ~/kent/src/hg/htdocs/style
git pull
make alpha

Note: you'll first need to update the CGIs for preview at: ls -lrt /usr/local/apache/cgi-bin-preview/

cd ~/kent/src
git pull
USER=preview make cgi

There are some redundant git pulls here, they exist to drive home the message to be sure your branch is current. If you are seeing errors on the build process please ask for help. Once all your files are updated, you should run the command with the -c --check option to see what will be updated: pushDevToPreview -c and remember to have accessed the qateam identity ssh qateam@hgwdev

If things look pretty good, then you are ready to run the command, with the -i interactive mode so you can see what commands are being run:

 pushDevToPreview -i

Pushing files from hgwdev to genome-preview

Do you wish to proceed? (y/[n]): y

Doing directory /usr/local/apache/cgi-bin-preview/ between hgwdev and genome-preview


The command to be executed is /usr/bin/rsync -rlptDi --delete --chown=qateam:protein --exclude=hg.conf*  /usr/local/apache/cgi-bin-preview/ genome-preview:/usr/local/apache/cgi-bin-preview/
...
...

Finally you'll check on genome-preview if the changes are updated.