CGI Build Process

From genomewiki
Jump to navigationJump to search

This page explains the process we use for building and releasing our CGIs. This is done on a two-week schedule.

The builds are built into sandboxes which are located here: 

hgwbeta:/data/tmp/releaseBuild
hgwdevold:/scratch/releaseBuild

Preview Day Build

This is day 1 in the schedule.

Run CVS Reports

  • go to /cluster/bin/build/scripts/oldsettings
hgwbeta> wb
hgwbeta> cd oldsettings
  • copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX[+ 1].preview
hgwbeta> cp buildEnv.csh.v111.final buildEnv.csh.v112.preview
  • edit this newly-created file: change the 5th line then the 4th line
hgwbeta> vi buildEnv.csh.v112.preview

 line 5 (OLD): setenv LASTREVIEWDAY 2006-03-20 # preview of v110
 line 5 (NEW): // change it to look like the date and comment from line 4

 line 4 (OLD): setenv REVIEWDAY 2006-04-03  # preview of v111
 line 4 (NEW): setenv REVIEWDAY 2006-today  # preview of v112
  • copy the newly created file over the file located up one directory
hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.preview buildEnv.csh
  • source your .tcshrc
hgwbeta> tcsh
  • check that the vars are correct
hgwbeta> env
  • run doNewReview.csh as a test to see what it will do (review the variablies)
hgwbeta> ./doNewReview.csh
  • run for real (this takes about 20 minutes - it does the run by ssh'ing to hgwdev)
hgwbeta> ./doNewReview.csh real

Check the reports

  • The reports are automatically built by the script into this location.
  • Briefly review the reports quickly as a sanity check.

Generate summary and review pairings

  • Summarize the code changes that were committed during the past week.
  • Update this page with the summary.
  • Assign code-review partners.
  • Send an email to browser-staff with the summary and code-review pairings.


Make changes to code base as necessary

This happens on days 9, 10, and 11 in the schedule.

  • If there are problems with the build a developer will fix the code. This fix needs to be patched into the build on hgwbeta. This document explains how to move a Branch Tag on hgwbeta.

Final Build

This is day 8 in the schedule.

Do the Build

  • go to /cluster/bin/build/scripts/oldsettings
hgwbeta> wb
hgwbeta> cd oldsettings
  • copy the last buildEnv.csh file to a new file called buildEnv.csh.vXXX.final
hgwbeta> cp buildEnv.csh.v112.preview buildEnv.csh.v112.final
  • edit this file
hgwbeta> vi buildEnv.csh.v112.final
setenv BRANCHNN 131
setenv TODAY 2006-04-10     # v131 final
setenv LASTWEEK 2006-03-27  # v130 final
  • edit file: add one to the BRANCHNN
setenv BRANCHNN 132
  • edit file: change the 3rd line then the 2nd line
line 3 (OLD): setenv LASTWEEK 2006-03-27  # v130 final
line 3 (NEW): // change it to look like the date and comment from line 2

line 2 (OLD): setenv TODAY 2006-04-10     # v131 final
line 2 (NEW): setenv TODAY (today)        # v132 final
  • copy the newly created file over the file located up one directory
hgwbeta> cd ../
hgwbeta> cp oldsettings/buildEnv.csh.v112.final buildEnv.csh
  • source your .tcshrc
hgwbeta> tcsh
  • check that the vars are correct
hgwbeta> env
  • run doNewBranch.csh as a test to see what it will do (review the variables)
hgwbeta> ./doNewBranch.csh
  • run for real and send the output to a file
hgwbeta> ./doNewBranch.cvs real >& doNew.log
  • review the output file as it is written to
hgwbeta> tail -f doNew.log
  • look for files that tell you it was successful
hgwbeta> /cluster/bin/build/scripts/32bitUtils.ok  
hgwbeta> /cluster/bin/build/scripts/CvsReports.ok


  • What the doNewBranch.cvs script does:
  1. edits the versionInfo.h file
  2. makes tags (takes several minutes)
  3. builds CVS reports (takes 5-10 minutes)
  4. does build (takes 5-10 minutes)
    1. builds CGIs (most important)
    2. builds utils (of secondary importance)
  5. runs robots one at a time
    1. hgNear (20 min)
    2. TB (1 hour)
    3. TrackCheck (several hours)
    4. LiftOverTest (quick)
  • Review the error logs for the three robots:
/build/scripts/logs/ (error logs)

hgNear -- sends email with results
hgTables -- send email with results
hgTrackCheck -- must check by hand+
LiftOverTest -- must check by hand+

+ located here: hgwbeta:/cluster/bin/build/scripts/logs


Push the CGIs

This is day 12 in the schedule.

Push to hgw1 only

  • Wait to hear that all pair-reviews are complete.
  • Ask for push of CGIs from hgwbeta to hgw1 only.
  • Run TrackCheck.java on hgw1.
  1. start here: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck>
  2. make a props file which specifies the machine/db to check.
  3. run it: [hgwdev:/scratch/javaBuild/src/edu/ucsc/genome/qa/cgiCheck> nohup TrackCheck hgw3.props > & hgw3.07-13-2006
  4. grep the output for 'error'.
  • Check Apache Error Log
hgw1:/usr/local/apache/logs/error_log

Push to the rest of the RR

  • Ask for push from hgwbeta to hgw2-8 & MGC.

Final Build Wrap-up

This is day 12 in the schedule.

  • zip and push source to hgdownload
hgwbeta> wb
hgwbeta> tcsh
hgwbeta> env
hgwbeta> doZip.csh  (last line of output is push-request)
  • build and push hgcentral
hgwdev> ./buildHgCentralSql.csh
hgwdev> ./buildHgCentralSql.csh real
    • request push of hgcentral to hgdownload
 Please push from hgwdev --> hgdownload
 /usr/local/apache/htdocs/admin/hgcentral.sql
 This includes settings for the mirror sites.
  • build 32bit build and push to hgdownload
hgwbeta> cd /cluster/bin/build/scripts
hgwbeta> doBuildCgi32.csh
    • after this completes, check the directory for this file: 32bitUtils.ok
    • the push to hgdownload happens automatically to here:
hgdownload:/mirrorsites/apache/cgi-bin-i386/


  • zip liftOver utils on hgwbeta and scp them to hgdownload
hgwbeta> wb
hgwbeta> doHgDownloadUtils.csh
  • send email to genome-mirror
Link to latest source: http://hgdownload.cse.ucsc.edu/admin/jksrc.zip
Retrieved from "https://genomewiki.ucsc.edu/index.php?title=CGI_Build_Process&oldid=2553"