GBiC Testing

This page describes how to get a Ubuntu or CentOS (or whatever Linux machine) up and running in order to test the Genome Browser in the Cloud/mirror install script. For more information about the GBiC script itself see the README:

Installing vagrant

The first thing we will want to do in order to test the mirror install script is to get vagrant installed. Vagrant is a command line tool used to quickly start, stop (and destroy!) virtual machines, which we will then run the mirror script on. Vagrant technically works with any virtualization software, but since QA should already have VirtualBox installed for GBiB testing, we can just use that. For more reading about vagrant, see the following:

https://www.vagrantup.com/
https://www.vagrantup.com/docs/getting-started/

If you are using a Mac, we can use homebrew and brew commands to install vagrant. Enter the following commands on your machines terminal, not on dev! (One step will require your laptop's password).

brew tap caskroom/cask #if not already installed
brew install brew-cask #if not already installed
brew cask install vagrant
brew tap homebrew/completions
brew install vagrant-completion

If you don't want to install vagrant through the command line, they have an installer here for Mac, Windows, and some Linuxes. I haven't used the installer, although I imagine it works since for some reason vagrant isn't in yum's repo.

Once vagrant is installed, we are ready to create our first virtual machine. You can check it is installed by running $ vagrant and seeing the list of options. You can always run the above commands with uninstall or untap to remove brew additions if something does not seem correct to repeat the steps.

Creating your VM hierarchy

You will want to create a directory called Vagrant, probably in your home directory, in order to keep track of all your virtual machines. Inside ~/Vagrant, create a separate directory for each operating system you are planning to test, or for each instance of your tests. For example you may want to run several different tests on CentOS 7, so you make 3 different directories in your ~/Vagrant directory. During the initial QA of the GBiC script, I had the following structure:

~/Vagrant
~/Vagrant/vagrant-centos6.5
~/Vagrant/vagrant-centos6.7
~/Vagrant/vagrant-centos7
~/Vagrant/vagrant-ubuntu14

This structure is important. Every time you install a VM, vagrant creates a Vagrantfile, which contains instructions on how to format the machine. This Vagrantfile is placed in the current directory of wherever you run the init command, so make sure you keep Vagrantfiles separate. Later on we will modify these Vagrantfiles in order to forward ports and check whether the GBiC script has installed a mirror correctly, so keep your Vagrantfiles safe! Don't spin up multiple machines in the same directory as Vagrant will force you to delete Vagrantfiles and you will lose your settings. There is no reason to reinstall operating systems once you've done it once, so just come up with your directory hierarchy and stick to it.

Installing and Uninstalling a VM

Once you've decided on what operating systems you will be testing and have created a directory structure you like you can finally install a VM.

I recommend testing the mirror install script on both Ubuntu and CentOS, with a focus on CentOS, since that is the favored server OS these days. Vagrant refers to virtual machines as "boxes", and you will use vagrant to download a "box" from their list of publicly available boxes. The general process of installing a box is:

vagrant init box/name     # this installs the vm and makes a Vagrantfile
vim Vagrantfile           # if need be
vagrant up                # starts the VM

For instance, to install a Ubuntu 14.04 LTS box:

vagrant init ubuntu/trusty64
vagrant up

or to install a CentOS 6.7 box:

vagrant init bento/centos-6.7
vagrant up

The vagrant init command creates an empty Vagrantfile, here are the steps and important lines from the Vagrantfile:

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant init debian/wheezy64
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vim Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "debian/wheezy64"
  # config.vm.network "forwarded_port", guest: 80, host: 8080

What we need to do is uncomment (but keep indented) the "# config.vm.network "forwarded_port", guest: 80, host: 8080" section. Also you may want to change the port number "host: 8080" to something like 8081 or something. If you are going to have mulitple machines up and running at the same time, give each a different port number so we don't have any collisions (I used 8080-8083 for each of my four machines). What this does is send the VM's port 80 (which is where http listens) to your machines port 8081. Therefore, once we use the GBiC script to install Apache and setup an http server, we can access the mirror from our laptop on localhost:8081 or http://127.0.0.1:8081. Laptop's aren't usually setup to work as servers, so unfortunately you won't be able to open up your mirror to the world, but at least you won't need to put GNOME and Firefox on your VM and run Firefox through the command line just to check your mirror. If you do install GNOME and Firefox (which is kind of fun) you will be able to access the mirror from plain localhost. If you've ran the above init and up commands already, then you will need to restart your virtual machine after editing the Vagrantfile.

Technically we are now supposed to use vagrant box add box/name to add the box, and then use vagrant up to start up the machine, but the up command add's the box anyways, so we can just skip the add step and use up:

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant up

There should be messages sent to stdout indicating success, but just to check that everything worked, use the ssh command to actually get into your VM:

christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-debian$ vagrant ssh
Linux wheezy 3.2.0-4-amd64 #1 SMP Debian 3.2.81-1 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
vagrant@wheezy:~$ 

I have been successfully dropped into a shell on a Debian machine. You can also open up your VirtualBox app and you will notice you have a machine running, in this case mine was called vagrant-debian_default_someNumber_someNumber.

Make sure you are running the correct commands from the correct directories, especially if you have multiple machines running at once, since all the commands are relative to your current directory. Notice how in the above ssh example I was in ~/Vagrant/vagrant-debian, if I instead was in ~/Vagrant/vagrant-ubuntu14, and the ubuntu VM was running, I would have ssh'd into the ubuntu machine!

If you're ready to turn your machine off, just run an exit command to exit the VM, and the run either vagrant halt or vagrant destroy to turn off or delete the VM. The halt command simply turns the machine off. Thus if you run vagrant up again it will be in the same state you left it. If you want to erase everything on the machine, you can run vagrant destroy, which erases all of your settings! Running vagrant up after a destroy will start up a new VM with the default state, so the destroy command is useful if you've really messed something up and want to start again, or if the version of the script you're testing is problematic and you need to test it fresh each time.

Hopefully you will only have to do the above steps once, as once you've added a box, it sticks around forever, and all you will be doing is vagrant up, ssh, halt, or destroy. Some example commands to restart a VM are vagrant global-status to see your machine's id (like 1a2b3c4d), and then vagrant up 1a2b3c4d to launch it, and vagrant ssh 1a2b3c4d to enter it (remember that your ~/Vagrant/vagrant-systemName/Vagrantfile defined the port where your machine will appear).

Testing the GBiC Script

Once you're in your virtual machine (vagrant ssh) installing a mirror is pretty easy. First you'll want to grab some command line tools if your VM doesn't already come with them, things like vim and wget and rsync, and then you should be ready to go.

If Fedora/RHEL/CentOS based:

sudo yum install -y wget vim rsync

Debian/Ubuntu based:

sudo apt-get install -y wget vim rsync

Now we're ready to get the script from the store at https://genome-store.ucsc.edu/ where it is free for non-commercial use. For QA or dev purposes, you can check /cluster/home/chmalee/gbicLocation for a wget'able location to download the script to your Virtual Machine.

Finally we can actually run the script! When you run the script the first time on a new machine you must pass it the install parameter:

sudo bash browserSetup.sh -b install

The -b option tells the script to use batch mode so you don't have to sit there and watch it while it downloads everything, and install does all kinds of stuff outlined in the README, which you can read here

After the install command finishes, point the browser on your host machine to http://localhost:8080, where 8080 is whatever port number you specified earlier. You should now have a functioning Genome Browser that loads all data from UCSC! You will probably want to go to hgTracks and play around a little bit, click into hgGene, get some hgTables output and some BLAT output. As far as general testing goes that should be it. You can try downloading some tables or doing a minimal install and entering offline mode to see if there are any errors.

Summary

christopherLee@Christophers-MacBook-Pro:~$ brew tap caskroom/cask #if not already installed
christopherLee@Christophers-MacBook-Pro:~$ brew install brew-cask #if not already installed
christopherLee@Christophers-MacBook-Pro:~$ brew cask install vagrant
christopherLee@Christophers-MacBook-Pro:~$ brew tap homebrew/completions
christopherLee@Christophers-MacBook-Pro:~$ brew install vagrant-completion
christopherLee@Christophers-MacBook-Pro:~$ mkdir -p Vagrant/vagrant-centos6.7
christopherLee@Christophers-MacBook-Pro:~$ cd Vagrant/vagrant-centos6.7
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant init bento/centos-6.7
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant up
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant ssh
vagrant@centos:~$ sudo -i
vagrant@centos:~$ yum install -y wget vim rsync
vagrant@centos:~$ wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/browserSetup.sh
vagrant@centos:~$ bash browserInstall.sh -y install
vagrant@centos:~$ # play around in browser a little bit
vagrant@centos:~$ exit
christopherLee@Christophers-MacBook-Pro:~/Vagrant/vagrant-centos$ vagrant destroy     # halts and removes machine, keeps vagrantfile for future use