Make your own virtual space

So you want to have your own copies of the binaries and your own sandbox or hgwdev-user.soe.ucsc.edu browser? Here is a walkthrough:

Request a "virtual space" from the cluster-admins:

This includes, among other things, adding a block to the file /usr/local/apache/conf/httpd.conf

<VirtualHost 128.114.xx.yyy>
 ServerName hgwdev-myusername.sdsc.edu
 ServerAlias hgwdev-myusername hgwdev-myusername.soe.ucsc.edu hgwdev-myusername.gi.ucsc.edu
 ScriptAlias /cgi-bin/ "/usr/local/apache/cgi-bin-myusername/"
</VirtualHost>

Prepare for the making of your CGIs

• If you don't already have one, set up your own personal Kent repository by following the instructions here: http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git#Setting_Up_Your_Own_Personal_Git_Kent_Repository

• Make sure your .bashrc or .tcshrc file (in your home directory) sets the build environment variables. It should include something like the following:

export MACHTYPE=`uname -m`

• Copy an hg.conf file from another user and place it here '/usr/local/apache/cgi-bin-username/'. It should contain at a minimum:

include ../cgi-bin/hg.conf
db.trackDb=trackDb_myusername

# Angie recommends that you do not add these lines unless you are using
# your own central database instead of hgcentraltest.
# Assuming you are using the default central.db (hgcentraltest):
# * If you use the default central.* settings from ../cgi-bin/hg.conf, then you can
#   seamlessly switch between genome-test and hgwdev-$USER -- your cookie session
#   will be the same on both.
# * If you change the central.* options, you will have separate cookies between
#   genome-test and hgwdev-$USER, which means your settings from one won't carry
#   over to the other unless you use the same hgsid when switching hosts.
central.host=localhost
central.domain=hgwdev-myusername.soe.ucsc.edu
central.cookie=hguid.hgwdev-myusername
backupcentral.domain=hgwdev-myusername.ucsc.edu 

Note that you can define your own background image by putting it in ~/public_html/, adding a short .css file (e.g., myusername.css) with the following lines

body.hgTracks {
    background: url("http://genome-test.gi.ucsc.edu/~myusername/yourimage.jpg");
}

to your /usr/local/apache/htdocs-myusername/style/ directory, and adding the following line to your /usr/local/apache/cgi-bin-myusername/hg.conf file

browser.style=/style/myusername.css 

Here is a generic [background image] file you can download and edit.

• Please note, in order for your browser to be fully functional, there needs to be a symlink to trash in your htdocs-myusername:

[myusername@hgwdev ~]$ cd /usr/local/apache/htdocs-myusername
[myusername@hgwdev htdocs-myusername]$ ln -s ../trash

• In order to load database tables and run CGIs on the command line for debugging, you will also need to create your own ~/.hg.conf . It must be readable and writable only by you because it contains a username and password that enables write access to the database. Here is one way to create it:

touch ~/.hg.conf
chmod 600 ~/.hg.conf
cat > ~/.hg.conf <<EOF
include /usr/local/apache/cgi-bin-myusername/hg.conf
db.user=uuuu
db.password=pppp
EOF

Instead of uuuu and pppp, use the actual write-access username and passwords -- ask cluster-admin or a colleague (best not to email them).

Populate my /usr/local/apache/cgi-bin-myusername/ directory for the first time by doing the following:

• Before continuing forward, make sure user account is included in the genecats group. This can be done with '$ groups'. If genecats is not present, message cluster admins with a request at cluster-admin@soe.ucsc.edu.

• From your home directory on hgwdev /cluster/home/myusername type the following things:

[myusername@hgwdev ~]$ cd kent
[myusername@hgwdev ~/kent]$ git pull
[myusername@hgwdev ~/kent]$ cd src
[myusername@hgwdev src]$ make libs
[myusername@hgwdev src]$ cd hg
[myusername@hgwdev hg]$ make cgi
[myusername@hgwdev hg]$ cd htdocs
[myusername@hgwdev htdocs]$ make user
[myusername@hgwdev htdocs]$ cd ../makeDb/trackDb
[myusername@hgwdev trackDb]$ make update

You now have your own compiled version of the CGIs.

Keep it current

Note: When we used CVS for revision control, it made sense to automate a nightly job that merged in everybody else's changes with our own work-in-progress and rebuilt the tree, so that we would keep up with others' changes and discover any conflicts sooner rather than later. Now that we use git, the workflow is a little different because git will refuse to merge if you have made changes but have not yet committed them to your local repository. If you always commit your changes at the end of the day, and never leave any work-in-progress edits overnight, then the script below will work. In Angie's opinion it is better to use a more git-compatible workflow, for example using branches for projects that take longer than a day and being more deliberate about pulling and merging. See http://genomewiki.ucsc.edu/index.php/Getting_Started_With_Git and http://genomewiki.ucsc.edu/index.php/Working_with_branches_in_Git .

Here is a script (makeBinaries.csh) that you can copy and paste into your ~myusername/bin directory to make your binaries and trackDb_myusername every night. You may or may not want to keep the git commands: they will fail if you have made edits but have not yet committed them to your local repository. Make sure to change the permissions (chmod 775 ~/bin/makeBinaries.csh) and if the script doesn't run check that the export PATH is set in your .bashrc file:

#!/bin/sh
#	Script to rebuild the current user's sandbox

#	Set environment 
umask 002
MACHTYPE=`uname -m`
PATH=/usr/local/bin:/bin:/usr/bin:/cluster/bin/${MACHTYPE}:${HOME}/bin/${MACHTPE}:/cluster/bin:/cluster/bin/scripts:${HOME}/bin:${HOME}/bin/scripts/
export MACHTYPE PATH
if [ -z "${USER}" ]; then
	USER=${LOGNAME}
	export USER
fi

#	Clean up the source tree
cd $HOME/kent/src && make clean > /dev/null 2> /dev/null && rm -f tags

#	Update the source tree
cd $HOME/kent 
git pull > $HOME/git.pull.output 2>&1
if [ $? -ne 0 ]; then
	echo "git pull failed, see $HOME/git.pull.output" | \
		mail -s 'Git pull error on kent tree' ${LOGNAME}
fi

#	Build the CGIs
cd ~/kent/src
make -j 40 cgi > ~/daily.out 2>&1
#	If there are any errors, they will come via separate email from cron
egrep -y "erro|warn" ~/daily.out | grep -v "\-Werror" | \
	grep -v "gbWarn.o" | grep -v "disabling jobserver mode"
make tags-all >/dev/null 2>&1

#	Update htdocs
cd ~/kent/src/hg/htdocs
make user >/dev/null 2>&1

#	Update trackDb
#	Errors with context will be reported by cron
cd ~/kent/src/hg/makeDb/trackDb/
make update |& grep -i -B20 error

Automate it:

Create a crontab file. Note that you may have numerous crontab files. Each machine that you want cron jobs running on will need its own crontab file (assuming the jobs are different). Example, name your crontab file: hgwdev.cron for your hgwdev crontab. With your editor, add a line to your crontab file to run the command at the time you desire. For example, to run your daily build at 04:42 every weekday, the line would read:

42 04 * * 1-5 bin/makeBinaries.csh

See also: 'man 5 crontab' for a description of the fields used in these crontab lines.

When your crontab file is set correctly, hand it off to the cron system with the command:

$ crontab <yourCrontabFile>

Example: 'crontab hgwdev.cron'

To see what you have submitted to the cron system, ask it to display what it knows about your cron jobs with the -l argument:

$ crontab -l

Cron job tips: It is better to not run your jobs exactly at the top of the hour. They would be competing with system cron jobs that are running at times like the top of the hour. Better to run your job at odd minutes during the hour. Don't use the example time listed here, otherwise everyone's job will be running at 04:42 every day and competing with each other.

Any output to stdout or stderr by your cron job will be emailed to you via the cron system as it runs your job. If you would rather not see that output in email, construct your cron job shell script in such a way that it takes care of all of its stdout and stderr output. For example, it could create an organized by date hierarchy of log files for its output.

A quicker list of the steps as of 2024

1. Ask Erich to create browser sandbox:
2. hgwdev-USERNAME.gi.ucsc.edu loads from:
3. hgwdev:/usr/local/apache/htdocs-USERNAME
4. hgwdev:/usr/local/apache/cgi-bin-USERNAME

1. Create ~/.hg.conf (get from someone in the group, e.g. max); replace "max" with your username in in ~/.hg.conf (and remove all commented lines)
2. create /usr/local/apache/cgi-bin-USERNAME/hg.conf (get from Max). Replace "max" with your username in this file! (optionally remove all commented lines)

1. Run the following commands:

chmod 600 ~/.hg.conf cd ~ && git clone USERNAME@hgwdev.gi.ucsc.edu:/data/git/kent.git export PATH=$PATH:/cluster/home/USERNAME/kent/src/utils/ # Need gitFiles from this directory cd ~/kent/src && make cgi -j 30 cd ~/kent/src && make doc && make doc-install # This needs the gitFiles script ~/kent/src/hg/makeDb/trackDb && make DBS=hg38 cd /usr/local/apache/htdocs-USERNAME && ln -s ../trash trash

1. Then your browser should be ready at https://hgwdev-USERNAME.gi.ucsc.edu/cgi-bin/hgTracks?db=hg38