Browser Installation

Installing a Browser Mirror on Red Hat and its Derivatives

The following How-to provides a step-by-step procedure for installing the Genome Browser on a Mandriva 2006/2007 system. It should also work on any other Red Hat derived Linux OS (and probably most distros for that matter).

These instructions are based on the procedures outlined in the general Browser Mirror Procedures as well as various HOWTOs found in jksrc. I have tried to combine the various steps into an easy to follow procedure which does will not require a lot of specialized knowledge to successfully complete the installation.

The procedure described below will create a full mirror of the Genome Browser. Most users are probably only interested in creating a partial mirror. See the (TODO: links)general Browser Mirror Procedures and the Partial Mirror document for more information on how to create and maintain a partial mirror.

---

Prerequisites:

• It is assumed that you have both Apache 2 web-server and MySql 4.1.x installed. The Genome Browser is currently running under MySql 4 but a move to MySql 5 is being investigated.
• A full mirror will require a lot of disk space. Currently (Dec 2006) a clean install of the Browser consumed close to 2T of storage. Additional space will also be required for future database expansion. A partial mirror will need less (perhaps far less). Plan accordingly. (TODO: link for discussion of minimum configuration).
• Note the rsync commands listed below must be executed with sufficient write permissions to create their assigned directories and files (which probably means root).

---

1. Get Executables

More details for these methods can be found in the official mirror docs and README.building.source. Download the released zipped version of the source files from here. Follow the instructions for how to compile the files.

• Option 1: Use rsync to get a copied of the compiled binaries.

  rsync -avzP rsync://hgdownload.cse.ucsc.edu/cgi-bin/ /var/www/cgi-bin

This command will grab the AMD Opteron x86_64 binaries. These binaries do not seem to be Opteron specific and apparently work with any of the AMD Athlon processors. If the binaries work for you they represent the easiest way forward.

• Option 2: Get the jksrc files and compile the executables yourself. Compiling the source can be a challenge if things don't work out of the box. However by compiling the jksrc tree you will get many useful tools and scripts which will be of value for bioinformatic and admin tasks.

• Option 3: (Suggested method) Download jksrc. Attempt to compile the source files. If this initially proves to be problematic, use rsync to get the pre-compiled executables and compile the source later. You will find several useful scripts and some of the browser documentation in /src/product (in the jksrc archive). The scripts and docs in the source tree are available whether or not you successfully compile the entire tree. (TODO: link to compile howto)

2.Configure Apache server

In order to support SSI it is necessary to set the XBitHack. Add the following somewhere in /etc/httpd/conf/httpd.conf

     XBitHack on
     <Directory /var/www/html>
     Options +Includes
     </Directory>

Find the location of your web pages. This should be /var/www/html by default. Set the enviromental variable if desired.

     export WEBROOT="/var/www/html"

Find the location of your cgi-bin directory. This should be /var/www/cgi-bin. Set the enviromental variable if desired.

     export CGI-BIN="/var/www/cgi-bin"

Next, find the location of your MySQL data. This should be located in /var/lib/mysql. Set the enviromental variable if desired.

     export MYSQLDATA="/var/lib/mysql"

Note: These variables can be set in /etc/profile so they will be available globally to all users. Also they can be skipped entirely if absolute paths are used instead.

3. Get all the html files

Test the rysnc connection:

   rsync -navz --progress rsync://hgdownload.cse.ucsc.edu

Determine the destination of the copy ($WEBROOT) and fire off the production copy. The trailing slash is important!

   rsync -avzP rsync://hgdownload.cse.ucsc.edu/htdocs/ /var/www/html/

4. Optional: Get the data for each individual genome assembly

(TODO: verify this entire section!!! Is it even necessary!!)

The individual assemblies are only required if the mysql tables are going to be build from scratch or if direct access to the assemblies is needed for research purposes (apparently unusual). It is generally preferable to download the mysql tables directly (covered later in this document) and so this step can probably be skipped.

Here is an example of how to rsync a single database assembly (Human)

Get Human March 2006 full data set (hg18: 13.0 Gb) by doing:

     mkdir -p $WEBROOT/goldenPath/hg18/database/
     rsync -avzP --delete --max-delete=20 \
     rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/hg18/database/ $WEBROOT/goldenPath/hg18/database/

This process must be repeated for each of the desired databases.

A list of all possible assemblies found on browser can be created by issuing the command

     rsync -v rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/ > goldenPath.dat

A list of all mysql data tables can be created by issuing the command

     rsync -v rsync://hgdownload.cse.ucsc.edu/mysql/ > gb_tables.dat

The lists retrieved by these two commands will be quite similar. In both cases it will be necessary to edit the directory names returned to be certain only data tables are being included.

5. Obtain the /gbdb data file area

You will need the portions of /gbdb used by the browser. (This is a large download):

     rsync -avzP --delete --max-delete=20 rsync://hgdownload.cse.ucsc.edu/gbdb/ /gbdb/

You can restrict this to only individual assemblies, like hg18 or mm8. In any case, you should always download the /gbdb/genbank directory, as many assemblies link into it.

6. Set up database tables:

These instructions should be followed in conjunction with /src/product/README.mysql.setup.There are two ways to install the tables.The first involves building the tables from the assembly dumps (optionally downloaded above).The second and preferable method involves rsyncing the binary tables themselves. This second method is preferable since the mysql table build process using the assemblies is quite computationally intensive.

Caveats for direct syncing:

• Your MySql version must be compatible with the table version (currently 4.0.x) (comment max: I'm using mysql5 and that works fine with direct syncing)
• The hgcentral (and others?) table which is found in /var/lib/mysql/ must recieve special handling (covered later).
• The actual download size of the tables is more than simply downloading the assemblies. This is because of the extensive use of indexes in the tables.
• The method for installing the tables using the assemblies is covered in the official mirror docs and is not covered here

To proceed with syncing the tables directly issue the following command:

         rsync -avzP --delete --max-delete=20 rsync://hgdownload.cse.ucsc.edu/mysql/XXX/ /var/lib/mysql/XXX/

where XXX is the name of each database to be mirrored. You will need to generate a list of tables to be mirrored. Note you can NOT simply sync with hgdownload.cse.ucsc.edu/mysql since the mysql directory contains a number of files and sub directories which are specific to each instance of the mysql database.

If you're mirroring mouse or human (or many other vertebrates) you will need other databases in addition to just the genome assemly databases: proteome, genbank, proteins040315, proteins060115 and uniProt.

An unedited list of potential tables to be mirrored can be found by issuing the command:

         rsync -v --dry-run rsync://hgdownload.cse.ucsc.edu/mysql

This list will then have to be edited so that only the correct tables are mirrored. The script (TODO:sync_tables) can be used to download a complete list of eligable tables and to automatically sync with each.

After the tables have been created it is necessary to add the required users along with their associated permissions. The entire process of MySQL configuration is described in /src/product/README.mysql.setup as found in jksrc. In brief 3 users are required. These users are readonly, readwrite, browser. These users are configured as follows:

User	MySql Permission	Databases	Used by
browser	SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER	All except hgcentral	developers
readonly	SELECT	All except hgcentral	CGI scripts
readwrite	SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER	hgcentral	browser(?)

Do not forget the grant rights for the databases genbank, proteome, uniProt and proteinsxxxx as well (See also Example Mysql Grants).

Each database must have these 3 users added with the associated permissions. The easiest way to accomplish this is to use the script ex.MySQLUserPerms.sh which can be found in src/products in jksrc. The script sets the permissions on each database listed by name. NOTE:This script must be edited before use!. The script handles each database explicitly by name. It is likely that the script does not contain the latest set of database names. A current list of database names must be generated and any which are missing will need to be added. Also future updates to the database may require additional changes to the script. As an alternative , it is possible at the cost of a small amount of security to set the permissions globally using *.* edits. An example of the required edit to the script so that permissions are added globally is:

   ${MYSQL} -e "GRANT SELECT, UPDATE on *.* to browser@localhost IDENTIFIED BY 'password';"mysql

After the edits are made the script will add these 3 users to all of the databases found used by the browser. These permissions are limited to localhost for security reasons. ex.MySQLUserPerms.sh is heavily documented and should be read to make sure that the changes discussed above are understood.

After adding the MySql Users it is necessary to add hg.conf to the cgi-bin directory. Optionally add hg.conf to any developers' home directories.hg.conf contains username/password information and is required by various scripts which access the the databases, in particular the cgi-bin scripts. A sample hg.conf can be found [1]. More discussion of this script can be in README.mysql.install which is located in /src/products in jksrc. The default user/password combinations and permissions can be changed, however doing so will require editing of other scripts which have the user/passwords hardcoded in them (notably ex.MySQLUserPerms.sh). It is probably best to keep the defaults at least until one knows what one is doing.

In hg.conf you will need to set the document root:

   browser.documentRoot=/var/www/html

The actual path could be different depending on your actual root directory. After an appropriate hg.conf has been created, it should be installed in /var/html/cgi-bin and the permissions set to 600 (my setup has the file owner/group of apache/apache).

7. Set up the "hgcentral" tables

Download the schema for the hgcentral database (hgcentral.sql)(TODO:here).

Create a hgcentral database

     mysql> create database hgcentral

Add the hgcentral tables

      mysql -youraccountoptions hgcentral < hgcentral.sql

Create a user/password with the ability to update and insert. This user is currently "readwrite". The script ex.MySQLUserPerms.sh will add this usercreate this user to hgcentral.

(optional) Set correct location of Blat Servers. By design the location of the Blat Servers is incomplete in hgCentral. This prevents their over use or abuse. In order to implement Blat you will either need to connect to the UCSC servers or setup your own. You will need permission to connect to the UCSC Blat Servers. Please see a discussion of the requirements and restrictions in the official docs (TODO: find correct link). The following sql command will update the table to point at the ucsc servers

    mysql> USE hgcentral;

    mysql> UPDATE blatServers SET host=CONCAT(host,".cse.ucsc.edu");

Please get permission before using Blat Servers!

8. Create a "trash" directory

The cgi programs use a temporary area to create and store images used by the browser. This directory is by default looked for in /var/www/trash. You should make this directory and allow the user that runs the web server write access to it. As a point of maintenance this directory will need to be cleaned out from time to time.

9. Create the hgFixed database

There are two ways of creating and populating the hgFixed database. The first method (and easiest!) is to sync with the appropriate MySql directory containing the database. Simply make sure that "hgFixed" is being included in the rsync process as described in step 6.

If you decide to build the tables directly you can use the procedure described below.

Download a copy of the dumped hgFixed tables.

     rsync -avzP --delete --max-delete=20 \
     rsync://hgdownload.cse.ucsc.edu/genome/goldenPath/hgFixed/ \
     $WEBROOT/goldenPath/hgFixed/

$WEBROOT/goldenPath/hgFixed contains both raw dumps of the sql tables contained in the hgFixed database as well as the .sql scripts which will load the tables directly into the database.

Create a hgFixed database

     mysql> create database hgFixed

Load each of the hgFixed tables using the method described above. The general syntax for each tables is:

    mysql -u myname -p mypassword hgFixed < table.sql

Since there are quite a few tables so it will be necessary to write a short script to automate this process.