CGI Build Process

This page explains the process we use for building and releasing our CGIs. This is done on a two-week schedule.

The builds are built into sandboxes which are located here:

hgwbeta:/data/tmp/releaseBuild
titan:/scratch/releaseBuild

Setting Up the Environment for the Build

NOTE: This section is a one-time only set up that you must do before starting to do the builds.

Before starting as buildmeister, you will need to set up the following in your log in file:

go to your home directory

hgwdev> cd

edit your .tcshrc file

hgwdev> vi .tcshrc file

add the following lines to set permissions on files and to set the environment variables ($BOX32, $WEEKLYBLD, $BUILDDIR and $BRANCHNN are set in one of the build scripts - buildEnv.csh):

umask 002
source /cluster/bin/build/scripts/buildEnv.csh

To be able to run the Java robot programs, add the following to the top of your path setting:

set path = ( /usr/java/default/bin \
  $path \
     .
     .
     .
  )

optional helper functions:

alias wb 'cd $WEEKLYBLD'

if (( "$HOST" == "$BOX32" ) || ( "$HOST" == "hgwbeta" )) then
    setenv hg $BUILDDIR/v${BRANCHNN}_branch/kent/src/hg
endif

wb gets you to the scripts dir.

cd $hg gets you to the sandbox
on beta when you have to recompile stuff
after branch tag moves, etc.

source your .tcshrc to check the variables:

hgwdev> tcsh

Set up autologin among the general cluster machines

On your local cse box (i.e. screech, pfft, whatever) >ssh-keygen -t dsa (use enter for all defaults) >cd ~/.ssh, add yourself to the authorized keys >cp id_dsa.pub authorized_keys

Also put these in hgwdev:~/.ssh >scp -r .ssh/ hgwdev:

Permissions on .ssh should be 700.

Permissions on files in .ssh/ should be 600 or 640.

Set up autologin to hgdownload by copying your public key to the list of authorized keys on those machines:

hgwdev> vi ~/.ssh/id_dsa.pub
copy the public key into the clipboard and then log into hgdownload as user qateam
hgwdev> ssh qateam@hgdownload
hgdownload> cd ~/.ssh
paste the key to the authorized_keys file
hgdownload> vi authorized_keys

Build Symlinks

These are critical for building 32 bit and 64 bit utilities

Go to ~/bin directory

hgwdev> cd ~/bin

Make sure you have $MACHTYPE directories

hgwdev> mkdir i386
hgwdev> mkdir x86_64

Create a symlink for each $MACHTYPE

hgwdev> ln -s /cluster/bin/i386 i386.cluster
hgwdev> ln -s /cluster/bin/x86_64 x86_64.cluster

The symtrick.csh uses these automatically.
If a script crashes and leaves the symlinks in an incorrect state, use unsymtrick.csh to restore. Build scripts check to see if unsymtrick.csh should be executed.

Preview Day Build : Day 2

This is day 2 in the schedule.

Run CVS Reports

go to /cluster/bin/build/scripts/oldsettings

hgwbeta> cd $WEEKLYBLD

edit buildEnv.csh: change the 5th line then the 4th line

hgwbeta> vi buildEnv.csh

 line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
 line 5 (NEW): // change it to look like the date and comment from line 4

 line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
 line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112

source your .tcshrc

hgwbeta> tcsh

check that the vars are correct

hgwbeta> env

commit the changes to this file to CVS:

hgwbeta> @ NEXTNN = ( $BRANCHNN + 1 ) ; cvs commit -m "v$NEXTNN preview" buildEnv.csh

run doNewReview.csh as a test to see what it will do (review the variablies)

hgwbeta> ./doNewReview.csh

run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev) and direct output to a log file

hgwbeta> ./doNewReview.csh real >& doNewRev.log &
hgwbeta> tail -f doNewRev.log

There was a warning not to background the job in the previous step because it or a sub-script might hang "waiting for input", but this doesnt seem to be the case any more.

This step will block while CVS locks are present. Temporary locks, from someone doing pretty much any command in cvs, will just slow things down. Old abandoned locks will hang the script completely. To look for these locks, go to hgwdev, get your csh ENV vars, and run:

hgwbeta> ssh hgwdev
hgwdev> tcsh
hgwdev> $WEEKLYBLD/findCvsLocks.csh

check any reported lock files and delete them if they are old (i.e. older than a few minutes), then kill and re-run doNewReview.csh.

Check the reports

The reports are automatically built by the script into this location.
Briefly review the reports quickly as a sanity check.

===Generate summary and review pairings=== (Donna takes care of this)

Summarize the code changes that were committed during the past week.
Update this page with the summary.
Assign code-review partners.
Send an email to browser-staff with the summary and code-review pairings.

Test on hgwbeta

Wait to hear from QA about how their CGIs look on hgwbeta. Each member of the QA team has testing responsiblities.

Make changes to code base as necessary

This happens on days 9, 10, and 11 in the schedule.

If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This page explains how to move a Branch Tag and how to Patch a Branch on hgwbeta.

Final Build : Day 9

This is day 9 in the schedule.

Do the Build

go to weekly build dir on beta

hgwbeta> cd $WEEKLYBLD

edit the buildEnv.csh file

hgwbeta> vi buildEnv.csh
setenv BRANCHNN 131
setenv TODAY 2006-04-10     # v131 final
setenv LASTWEEK 2006-03-27  # v130 final

edit file: add one to the BRANCHNN

setenv BRANCHNN 132

edit file: change the 3rd line then the 2nd line

line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2

line 2 (OLD): setenv TODAY 2006-04-10     # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final

re-source buildEnv.csh

hgwbeta> source buildEnv.csh

check that the vars are correct

hgwbeta> env

commit the changes to this file to CVS:

hgwbeta> cvs commit -m "v$BRANCHNN final build" buildEnv.csh

run doNewBranch.csh as a test to see what it will do (review the variables)

hgwbeta> ./doNewBranch.csh

run for real, send the output to a file and review while it is written (takes ~1 hour)

hgwbeta> ./doNewBranch.csh real >& doNew.log &
hgwbeta> tail -f doNew.log

look for files that tell you it was successful (script will report whether these files were created):

hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok  
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok

If you get errors it might be because the script is wrong, rather than there actually be an error. For example, to check for errors, the 'make' log file is greped for 'error|warn' so any new 'C' file with error or warn in its name will show up as an error whether or not it compiled cleanly. You might need to change the script to remove references to files like this, eg edit buildBeta.csh to ignore references to files like gbWarn.c and gbWarn.o in the log:

hgwbeta> vi buildBeta.csh
...
make alpha >& make.alpha.log
# These flags and programs will trip the error detection
sed -i -e "s/-DJK_WARN//g" make.alpha.log
sed -i -e "s/-Werror//g" make.alpha.log
#-- report any compiler warnings, fix any errors (shouldn't be any)
#-- to check for errors:
set res = `/bin/egrep -i "error|warn" make.alpha.log | /bin/grep -v "gbWarn.o -c gbWarn.c" | /bin/grep -v "gbExtFile.o gbWarn.o gbMiscDiff.o"`

What the doNewBranch.csh script does:

edits the versionInfo.h file
makes tags (takes several minutes)
builds CVS reports (takes 5-10 minutes)
does build (takes 5-10 minutes)
1. builds CGIs (most important)
2. builds utils (of secondary importance)

Check the reports

The reports are automatically built by the script into this location.
Briefly review the reports quickly as a sanity check.

Run the Robots

run doRobots.csh, and watch the log if you are interested (most log messages go to the logs/ dir mentioned below)

hgwbeta> nohup ./doRobots.csh >& $BRANCHNN.robots.log &
hgwbeta> tail -f $BRANCHNN.robots.log

What the doRobots.csh script does:

runs robots one at a time
1. hgNear (20 min)
2. hgTables (several hours)
3. TrackCheck (several hours)
4. LiftOverTest (quick)

Review the error logs for the robots:

error logs located here: hgwbeta:/cluster/bin/build/scripts/logs

hgNear -- sends email with results
hgTables -- send email with results
TrackCheck -- must check by hand: grep -i "error" logs/TrackCheck-v$BRANCHNN.log (QA and buildmeister)
LiftOverTest -- must check by hand: cat logs/LiftOverTest-v$BRANCHNN.log (QA and buildmeister)

Fixing problems in the Build

This usually happens between days 9 and 12.

QA advises buildmeister to patch or move the branch tags

see these instructions.

Push the CGIs

This is day 16 in the schedule.

Push to hgwN only

hgwN is one of the RR machines, hgw1-8. Each build, rotate to the next machine in numeric order i.e. hgw1 then hgw2 etc. so that one machine is not being worked more than the others.
QA will send an email to push-request the morning of the push letting the pushers know that today is a CGI push day (this is their notice to be vigilant about pushing quickly).
QA will ask for push of CGIs from hgwbeta to hgwN only. Make sure you add a note to the cluster-admins to make sure they know which RR machine to push to first. If there is a NEW CGI going out this week, be sure to make a prominent note of it in your push request. The admins push from a script, and they will need to add your new CGI to the script. (the build-meister should not be cc'd on this email.)

As of January 2009, here's a list of the CGIs and data files we push. Note: CGIs and data files may have been added since this list was created -- this is meant to be a starting point.

cartDump cartReset das hgc hgEncodeVocab
hgBlat hgConvert hgCustom hgGateway hgGene hgGenome
hgLiftOver hgNear hgPcr hgSession hgTables hgTracks hgTrackUi
hgVisiGene mkEncodeFrameset hgEncodeDataVersions
pbGateway pbGlobal pbTracks phyloGif hgPal

/usr/local/apache/cgi-bin/all.joiner
/usr/local/apache/cgi-bin/galaAvail.tab
/usr/local/apache/cgi-bin/hgNearData/*
/usr/local/apache/cgi-bin/hgGeneData/*
/usr/local/apache/cgi-bin/hgcData/*
/usr/local/apache/cgi-bin/hgCgiData/*
/usr/local/apache/cgi-bin/visiGeneData/*
/usr/local/apache/cgi-bin/loader/*
/usr/local/apache/cgi-bin/encode/cv.ra
/usr/local/apache/cgi-bin/lsSnpPdbChimera.py
/usr/local/apache/htdocs/js/*

Run TrackCheck.java on hgwN. This is the responsibility of the QA person who tests hgTracks.

start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
make a props file which specifies the machine/db to check. If you are in a hurry, set zoomCount=0 and it will only check the default position for each assembly (much quicker!). Example props file for hgw4:

machine hgsqlbeta.cse.ucsc.edu
server hgw4.cse.ucsc.edu
quick false
dbSpec all
table all
zoomCount 3

run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw1.props > & $WEEKLYBLD/logs/TrackCheck-hgw1.07-13-2006
grep the output for 'error' (and 'Error').

Monitor Apache Error Log (QA does this):

hgwN:/usr/local/apache/logs/error_log

To watch the log without line wraps, use less -S. Use "F" to follow incoming errors.

Wait to hear from QA about how their CGIs look on hgwN. Each member of the QA team has testing responsiblities. Check also that TrackCheck ran successfully.

Push to the rest of the RR

QA will ask for push from hgwbeta to the rest of the hgwN machines.
QA will send email to the build-meister to let him/her know that the CGIs are on the RR.

Remember to keep track of new features

Anyone can add to this list at any time, but if no notes for this release have been made on the new features page, now is a good time to add some.

Final Build Wrap-up

This is day 16 in the schedule.

The buildmeister should do these steps once QA has notified you that all RR machines have been updated.

zip the source code

hgwbeta> tcsh
hgwbeta> cd $WEEKLYBLD
hgwbeta> env
hgwbeta> ./doZip.csh  (this is automatically pushed to hgdownload)

build and push hgcentral IF there are any changes

hgwbeta> ssh hgwdev
hgwdev> tcsh
hgwdev> cd $WEEKLYBLD
hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real

IF there are any changes: send email to push-request@cse.ucsc.edu to request push of hgcentral to hgdownload

 Please push from hgwdev --> hgdownload
 /usr/local/apache/htdocs/admin/hgcentral.sql
 This includes settings for the mirror sites.

build 32bit build and push to hgdownload

hgwdev> ssh hgwbeta
hgwbeta> tcsh
hgwbeta> cd $WEEKLYBLD
hgwbeta> ./doBuildCgi32.csh

the push to hgdownload happens automatically to here:

hgdownload:/mirrordata/apache/cgi-bin-i386/

build 'userApps' target (various utilities) on hgwbeta and scp them to hgdownload

hgwbeta> cd $WEEKLYBLD
hgwbeta> ./doHgDownloadUtils.csh

send email to genome-mirror@soe.ucsc.edu. First check that the source file version is right:

http://hgdownload.cse.ucsc.edu/admin/

Include this link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip. Use the last email as a template (see https://www.soe.ucsc.edu/pipermail/genome-mirror). If you push the hgcentral.sql, make sure to mention this has also changed in the email.

CGI Build Process

Contents

Setting Up the Environment for the Build

Build Symlinks

Preview Day Build : Day 2

Run CVS Reports

Check the reports

Test on hgwbeta

Make changes to code base as necessary

Final Build : Day 9

Do the Build

Check the reports

Run the Robots

Fixing problems in the Build

QA advises buildmeister to patch or move the branch tags

Push the CGIs

Push to hgwN only

Push to the rest of the RR

Remember to keep track of new features

Final Build Wrap-up

Navigation menu

Page actions

Page actions

Personal tools

Navigation

Search

related sites

hosted projects

Tools