GBiC Testing

This page describes how to get a Ubuntu or CentOS (or whatever Linux machine) up and running in order to test the Genome Browser in the Cloud/mirror install script. For more information about the GBiC script itself see the README:

Installing vagrant

The first thing we will want to do in order to test the mirror install script is to get vagrant installed. Vagrant is a command line tool used to quickly start, stop (and destroy!) virtual machines, which we will then run the mirror script on. Vagrant technically works with any virtualization software, but since QA should already have VirtualBox installed for GBiB testing, we can just use that. For more reading about vagrant, see the following:

https://www.vagrantup.com/
https://www.vagrantup.com/docs/getting-started/

If you are using a Mac, we can use homebrew and brew commands to install vagrant. Enter the following commands on your machines terminal, not on dev! (One step will require your laptop's password).

brew tap caskroom/cask #if not already installed
brew install brew-cask #if not already installed
brew cask install vagrant
brew tap homebrew/completions
brew install vagrant-completion

If you don't want to install vagrant through the command line, they have an installer here for Mac, Windows, and some Linuxes. I haven't used the installer, although I imagine it works since for some reason vagrant isn't in yum's repo.

Once vagrant is installed, we are ready to create our first virtual machine. You can check it is installed by running $ vagrant and seeing the list of options. You can always run the above commands with uninstall or untap to remove brew additions if something does not seem correct to repeat the steps. You might also occasionally have to update vagrant, one option is downloading it from the earlier link.

Creating your VM hierarchy

You will want to create a directory called Vagrant, probably in your home directory, in order to keep track of all your virtual machines. Inside ~/Vagrant, create a separate directory for each operating system you are planning to test, or for each instance of your tests. For example you may want to run several different tests on CentOS 7, so you make 3 different directories in your ~/Vagrant directory. During the initial QA of the GBiC script, I had the following structure:

~/Vagrant
~/Vagrant/vagrant-centos6.5
~/Vagrant/vagrant-centos6.7
~/Vagrant/vagrant-centos7
~/Vagrant/vagrant-ubuntu14

This structure is important. Every time you install a VM, vagrant creates a Vagrantfile, which contains instructions on how to format the machine. This Vagrantfile is placed in the current directory of wherever you run the init command, so make sure you keep Vagrantfiles separate. Later on we will modify these Vagrantfiles in order to forward ports and check whether the GBiC script has installed a mirror correctly, so keep your Vagrantfiles safe! Don't spin up multiple machines in the same directory as Vagrant will force you to delete Vagrantfiles and you will lose your settings. There is no reason to reinstall operating systems once you've done it once, so just come up with your directory hierarchy and stick to it.

Installing and Uninstalling a VM

Once you've decided on what operating systems you will be testing and have created a directory structure you like you can finally install a VM.

I recommend testing the mirror install script on both Ubuntu and CentOS, with a focus on CentOS, since that is the favored server OS these days. Vagrant refers to virtual machines as "boxes", and you will use vagrant to download a "box" from their list of publicly available boxes. The general process of installing a box is:

vagrant init box/name     # this installs the vm and makes a Vagrantfile
vim Vagrantfile           # if need be
vagrant up                # starts the VM

For instance, to install a Ubuntu 14.04 LTS box:

vagrant init ubuntu/trusty64
vagrant up

or to install a CentOS 6.7 box:

vagrant init bento/centos-6.7
vagrant up

The vagrant init command creates an empty Vagrantfile, here are the steps and important lines from the Vagrantfile:

Note: An alternative to modifying the vagrantfile is to specify the ports on the ssh command line later (see below).

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant init debian/wheezy64
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vim Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "debian/wheezy64"
  # config.vm.network "forwarded_port", guest: 80, host: 8080

What we need to do is uncomment (but keep indented) the "# config.vm.network "forwarded_port", guest: 80, host: 8080" section. Also you may want to change the port number "host: 8080" to something like 8081 or something. If you are going to have mulitple machines up and running at the same time, give each a different port number so we don't have any collisions (I used 8080-8083 for each of my four machines). What this does is send the VM's port 80 (which is where http listens) to your machines port 8081. Therefore, once we use the GBiC script to install Apache and setup an http server, we can access the mirror from our laptop on localhost:8081 or http://127.0.0.1:8081.

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vim Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "debian/wheezy64"
    config.vm.network "forwarded_port", guest: 80, host: 8081

Laptop's aren't usually setup to work as servers, so unfortunately you won't be able to open up your mirror to the world, but at least you won't need to put GNOME and Firefox on your VM and run Firefox through the command line just to check your mirror. If you do install GNOME and Firefox (which is kind of fun) you will be able to access the mirror from plain localhost. If you've ran the above init and up commands already, then you will need to restart your virtual machine after editing the Vagrantfile (vagrant reload). If you use the ssh command (below), there is no need to restart it.

Technically we are now supposed to use vagrant box add box/name to add the box, and then use vagrant up to start up the machine, but the up command add's the box anyways, so we can just skip the add step and use up:

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant up

There should be messages sent to stdout indicating success, but just to check that everything worked, use the ssh command to actually get into your VM:

An alternative to modifying the vagrantfile as mentioned above is to specify the ports on the ssh command line below:

vagrant ssh -- -L 8888:localhost:80

Here is the SSH command without specifying the port:

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant ssh
Linux wheezy 3.2.0-4-amd64 #1 SMP Debian 3.2.81-1 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
vagrant@wheezy:~$ 

I have been successfully dropped into a shell on a Debian machine. You can also open up your VirtualBox app and you will notice you have a machine running, in this case mine was called vagrant-debian_default_someNumber_someNumber.

Make sure you are running the correct commands from the correct directories, especially if you have multiple machines running at once, since all the commands are relative to your current directory. Notice how in the above ssh example I was in ~/Vagrant/vagrant-debian, if I instead was in ~/Vagrant/vagrant-ubuntu14, and the ubuntu VM was running, I would have ssh'd into the ubuntu machine!

If you're ready to turn your machine off, just run an exit command to exit the VM, and the run either vagrant halt or vagrant destroy to turn off or delete the VM. The halt command simply turns the machine off. Thus if you run vagrant up again it will be in the same state you left it. If you want to erase everything on the machine, you can run vagrant destroy, which erases all of your settings! Running vagrant up after a destroy will start up a new VM with the default state, so the destroy command is useful if you've really messed something up and want to start again, or if the version of the script you're testing is problematic and you need to test it fresh each time.

Hopefully you will only have to do the above steps once, as once you've added a box, it sticks around forever, and all you will be doing is vagrant up, ssh, halt, or destroy. Some example commands to restart a VM are vagrant global-status to see your machine's id (like 1a2b3c4d), and then vagrant up 1a2b3c4d to launch it, and vagrant ssh 1a2b3c4d to enter it (remember that your ~/Vagrant/vagrant-systemName/Vagrantfile defined the port where your machine will appear).

Testing the GBiC Script

Once you're in your virtual machine (vagrant ssh) installing a mirror is pretty easy. First you'll want to grab some command line tools if your VM doesn't already come with them, things like vim and wget and rsync, and then you should be ready to go.

If Fedora/RHEL/CentOS based:

sudo yum install -y wget vim rsync

Debian/Ubuntu based:

sudo apt-get install -y wget vim rsync

Now we're ready to get the script from the store at https://genome-store.ucsc.edu/ where it is free for non-commercial use. For QA or dev purposes, you can check https://hgwdev.gi.ucsc.edu/gbic/browserSetup.sh for a wget'able location to download the script to your Virtual Machine. Make sure to update the script by running a 'make alpha' from:

~/kent/src/product/installer

Finally we can actually run the script! When you run the script the first time on a new machine you must pass it the install parameter:

sudo bash browserSetup.sh -b install

The -b option tells the script to use batch mode so you don't have to sit there and watch it while it downloads everything, and install does all kinds of stuff outlined in the README, which you can read here

After the install command finishes, point the browser on your host machine to http://localhost:8080, where 8080 is whatever port number you specified earlier. You should now have a functioning Genome Browser that loads all data from UCSC! You will probably want to go to hgTracks and play around a little bit, click into hgGene, get some hgTables output and some BLAT output. As far as general testing goes that should be it. You can try downloading some tables or doing a minimal install and entering offline mode to see if there are any errors.

Summary

christopherLee@Christophers-MacBook-Pro:~$ brew tap caskroom/cask #if not already installed
christopherLee@Christophers-MacBook-Pro:~$ brew install brew-cask #if not already installed
christopherLee@Christophers-MacBook-Pro:~$ brew cask install vagrant
christopherLee@Christophers-MacBook-Pro:~$ brew tap homebrew/completions
christopherLee@Christophers-MacBook-Pro:~$ brew install vagrant-completion
christopherLee@Christophers-MacBook-Pro:~$ mkdir -p Vagrant/vagrant-centos6.7
christopherLee@Christophers-MacBook-Pro:~$ cd Vagrant/vagrant-centos6.7
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant init bento/centos-6.7
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant up
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant ssh
vagrant@centos:~$ sudo -i
vagrant@centos:~$ yum install -y wget vim rsync
vagrant@centos:~$ wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh
vagrant@centos:~$ bash browserSetup.sh -b install
vagrant@centos:~$ # play around in browser a little bit
vagrant@centos:~$ exit
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant destroy     # halts and removes machine, keeps vagrantfile for future use

Notes

Offline/Online Status

If you are in offline mode, the lines below in /usr/local/apache/cgi-bin/hg.conf will be commented out:

vagrant@vagrant-ubuntu-trusty-64:~$ cat /usr/local/apache/cgi-bin/hg.conf | grep gbdb
gbdbLoc1=/gbdb/
gbdbLoc2=http://hgdownload.soe.ucsc.edu/gbdb/

Change the defaultDb

Documentation for these changes are shown in the following lines from hg.conf:

# if you want a different default species selection on the Gateway
# page, change this default Human to one of the genomes from the
# defaultDb table in hgcentral:
# hgsql -e "select genome from defaultDb;" hgcentral
# If you need a different version of that specific genome, change
# the defaultDb table entry, for example, a different mouse genome
# version as default:
# hgsql -e 'update defaultDb set name="mm8" where genome="Mouse"
# then this defaultGenome would read: defaultGenome=Mouse
defaultGenome=Human

1. Change the default organism, e.g., from human to mouse. You only need to edit hg.conf

Edit /usr/local/apache/cgi-bin/hg.conf

• Apache owns this, so you need to use "sudo vi hg.conf" to write to it, otherwise it will be read-only.

sudo vi hg.conf

e.g., change:

defaultGenome=Human

to

defaultGenome=Mouse

Now do a cart reset, and your default organism on Gateway will be mouse instead of human. The default assembly version of mouse is now shown (as of 2/2017 this is mm10), but you can do a database edit to make a different assembly version the default, such as mm9.

2. If you are already using the default organism, human, and you haven't changed hg.conf, you can change the human assembly from the current default (as of 2/2017 this is hg38) to a different assembly (such as hg19) with a sql table update.

To run MySQL in your installation, you will either need the appropriate password as defined in the GBiC script, or you can go to your home directory (type "cd") and make your own .hg.conf file:

Note: Don't do this as root. Note the " . " before "hg.conf"

vi .hg.conf

Paste in the following contents from this template.

Next, change the permissions:

chmod 600 .hg.conf

Once your .hg.conf file is in place, you will be able to use the command below (note the "kent" use of hgsql, not "mysql").

hgsql -e 'update defaultDb set name="mm9" where genome="Mouse"'

Drop a database

If you have permissions errors while trying to drop a database, try this workaround:

• Example below to drop database ornAna1

vagrant@vagrant-ubuntu-trusty-64:~$ sudo rm -rf /var/lib/mysql/ornAna1

Permission Errors

If you have an old version of the .hg.conf file into your home dir, you may only have read access to sql. As of Feb 1, 2017, the .hg.conf template on gitHub was edited to include full sql permissions by default.

The fastest way to fix sql permissions issues for an old file is to simply copy in the new template.

There are a few ways to fix this, as you will inevitably want to add, drop, or update MySQL tables. 1. You can submit the following command, which will move you to the root dir, as root. Once you are root, you should be able to run sql commands. To get out of this mode, type, "exit".

sudo -i

2. Change your local .hg.conf (in your home dir). Change this:

# Credentials to access the local mysql server
db.host=localhost
db.user=readonly
db.password=access

to this:

db.host=localhost
db.user=browser
db.password=genome

3. Root has its own permissions for sql, and this root-specific username and pw could also be used. This step is probably not necessary, but is being noted for documentation purposes. You need to be root to run this:

 cat /root/.my.cnf

mysql -ubrowser -pgenome -e "show databases"

Or, you can do this:

sudo -i cat /root/.my.cnf

Example:

root@vagrant-ubuntu-trusty-64:~# cat .my.cnf
[client]
user=root
password=cbIkUmqt