CGI Build Process

This page explains the process we use for building and releasing our CGIs. This is done on a two-week schedule.

The builds are built into sandboxes which are located here:

hgwbeta:/data/tmp/releaseBuild
hgwdevold:/scratch/releaseBuild

Preview Day Build

This is day 1 in the schedule.

Run CVS Reports

• go to /cluster/bin/build/scripts/oldsettings

hgwbeta> wb
hgwbeta> cd oldsettings

• copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX[+ 1].preview

hgwbeta> cp buildEnv.csh.v111.final buildEnv.csh.v112.preview

• edit this newly-created file: change the 5th line then the 4th line

hgwbeta> vi buildEnv.csh.v112.preview

 line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
 line 5 (NEW): // change it to look like the date and comment from line 4

 line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
 line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112

• copy the newly created file over the file located up one directory

hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.preview buildEnv.csh

• source your .tcshrc

hgwbeta> tcsh

• check that the vars are correct

hgwbeta> env

• run doNewReview.csh as a test to see what it will do (review the variablies)

hgwbeta> ./doNewReview.csh

• run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev)

hgwbeta> ./doNewReview.csh real

Check the reports

• The reports are automatically built by the script into this location.
• Briefly review the reports quickly as a sanity check.

Generate summary and review pairings

• Summarize the code changes that were committed during the past week.
• Update this page with the summary.
• Assign code-review partners.
• Send an email to browser-staff with the summary and code-review pairings.

Make changes to code base as necessary

This happens on days 9, 10, and 11 in the schedule.

• If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This document explains how to move a Branch Tag on hgwbeta.

Final Build

This is day 8 in the schedule.

Do the Build

• go to /cluster/bin/build/scripts/oldsettings

hgwbeta> wb
hgwbeta> cd oldsettings

• copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX.final

hgwbeta> cp buildEnv.csh.v112.preview buildEnv.csh.v112.final

• edit this file

hgwbeta> vi buildEnv.csh.v112.final
setenv BRANCHNN 131
setenv TODAY 2006-04-10     # v131 final
setenv LASTWEEK 2006-03-27  # v130 final

• edit file: add one to the BRANCHNN

setenv BRANCHNN 132

• edit file: change the 3rd line then the 2nd line

line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2

line 2 (OLD): setenv TODAY 2006-04-10     # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final

• copy the newly created file over the file located up one directory

hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.final buildEnv.csh

• source your .tcshrc

hgwbeta> tcsh

• check that the vars are correct

hgwbeta> env

• run doNewBranch.csh as a test to see what it will do (review the variables)

hgwbeta> ./doNewBranch.csh

• run for real and send the output to a file

hgwbeta> ./doNewBranch.cvs real >& doNew.log

• review the output file as it is written to

hgwbeta> tail -f doNew.log

• look for files that tell you it was successful

hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok  
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok

• What the doNewBranch.cvs script does:

1. edits the versionInfo.h file
2. makes tags (takes several minutes)
3. builds CVS reports (takes 5-10 minutes)
4. does build (takes 5-10 minutes)
   1. builds CGIs (most important)
   2. builds utils (of secondary importance)
5. runs robots one at a time
   1. hgNear (20 min)
   2. TB (1 hour)
   3. TrackCheck (several hours)
   4. LiftOverTest (quick)

• Review the error logs for the three robots:

/build/scripts/logs/ (error logs)

hgNear -- sends email with results
hgTables -- send email with results
hgTrackCheck -- must check by hand+
LiftOverTest -- must check by hand+
+ located here: hgwbeta:/cluster/bin/build/scripts/logs 

Push the CGIs

This is day 12 in the schedule.

Push to hgw1 only

• Wait to hear that all pair-reviews are complete.
• Ask for push of CGIs from hgwbeta to hgw1 only.
• Run TrackCheck.java on hgw1.

1. start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
2. make a props file which specifies the machine/db to check.
3. run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw3.props > & hgw3.07-13-2006
4. grep the output for 'error'.

• Check Apache Error Log

hgw1:/usr/local/apache/logs/error_log

• Wait to hear from QA about how their CGIs look on hgw1. Each member of the QA team has testing responsiblities.

Push to the rest of the RR

• Ask for push from hgwbeta to hgw2-8 & MGC.

Final Build Wrap-up

This is day 12 in the schedule.

• zip and push source to hgdownload

hgwbeta> wb
hgwbeta> tcsh
hgwbeta> env
hgwbeta> doZip.csh  (last line of output is push-request)

• build and push hgcentral

hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real

• • request push of hgcentral to hgdownload

 Please push from hgwdev --> hgdownload
 /usr/local/apache/htdocs/admin/hgcentral.sql
 This includes settings for the mirror sites. 

• build 32bit build and push to hgdownload

hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doBuildCgi32.csh

• • after this completes, check the directory for this file: 32bitUtils.ok
  • the push to hgdownload happens automatically to here:

hgdownload:/mirrorsites/apache/cgi-bin-i386/

• zip liftOver utils on hgwbeta and scp them to hgdownload

hgwbeta> wb
hgwbeta> doHgDownloadUtils.csh

• send email to genome-mirror

Link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip