Amazon EC2 cloud setup
Initial Set Up
Following the instructions with Amazon documentation: Getting Started Guide
You have your account setup, and credit card submitted.
Going to use the Java toolset, lookup where java is installed on hgwdev, set environment:
JAVA_HOME=/usr/java/jre1.6.0_11
From your Amazon account management screen, save the cert-*.pem and pk-*.pem files and save in your home directory ~/.ec2/ directory, chmod 700 ~/.ec2/ for security, and set the files at chmod 600 to be read-only for yourself.
Set environment:
EC2_PRIVATE_KEY=$HOME/.ec2/pk-*.pem EC2_CERT=$HOME/.ec2/cert-*.pem
Record your Amazon account number for later reference, it looks like: 1234-5678-9012
Pick up Java toolset from: Java Toolset
Unpack in some directory, and set environment:
EC2_HOME=/some/directory/ec2-api-tools-1.3-34128
Add $EC2_HOME/bin to your PATH
And to protect Java from the huge memory on hgwdev, set environment:
EC2_JVM_ARGS=-Xmx256M
Verify regions available with the toolset command:
$ ec2-describe-regions REGION eu-west-1 eu-west-1.ec2.amazonaws.com REGION us-east-1 us-east-1.ec2.amazonaws.com
Set environment:
EC2_URL=https://us-east-1.ec2.amazonaws.com
Create SSH keys:
$ cd $HOME/.ssh $ ec2-add-keypair gsg-keypair > id_rsa.gsg-keypair $ chmod 600 id_rsa-gsg-keypair
Sample AMI Images
Initial sample images are listed with:
$ ec2-describe-images -o self -o amazon > describe-images.txt
And, we want to use this one for a test first:
$ grep getting-started.manifest describe-images.txt IMAGE ami-3c47a355 ec2-public-images/getting-started.manifest.xml ... etc...
Note the identification ami-3c47a355, used in this command:
$ ec2-run-instances ami-3c47a355 -k gsg-keypair RESERVATION r-cee26ea7 891631566226 default INSTANCE i-32a7c65b ami-3c47a355 pending gsg-keypair 0 m1.small 2009-04-02T22:47:17+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc
Means it is running, Note the instance identification: i-32a7c65b
Use in this command to display the status of the instance:
$ ec2-describe-instances i-32a7c65b RESERVATION r-cee26ea7 891631566226 default INSTANCE i-32a7c65b ami-3c47a355 ec2-72-44-35-57.compute-1.amazonaws.com domU-12-31-39-03-46-52.compute-1.internal running gsg-keypair 0m1.small 2009-04-02T22:47:17+0000 us-east-1c aki-a71cf9ce ari-a51cf9cc
Note the name of the machine in that output: ec2-72-44-35-57.compute-1.amazonaws.com
Open SSH and HTTP ports:
$ ec2-authorize default -p 22 GROUP default PERMISSION default ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 $ ec2-authorize default -p 80 GROUP default PERMISSION default ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0
You can now see the Apache WEB server at: http://ec2-72-44-35-57.compute-1.amazonaws.com/
To see SSH keys:
$ ec2-get-console-output i-32a7c65b ec2: 2048 92:81:4f:19:e6:5b:23:e6:78:b7:e2:5d:be:ea:70:1d /etc/ssh/ssh_host_key.pub ec2: 2048 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc /etc/ssh/ssh_host_rsa_key.pub ec2: 1024 4f:18:83:90:7b:11:be:c3:b4:c5:7a:a3:11:9d:26:76 /etc/ssh/ssh_host_dsa_key.pub
Login via ssh to this system:
$ ssh -i $HOME/.ssh/id_rsa.gsg-keypair root@ec2-72-44-35-57.compute-1.amazonaws.com
Verify the displayed RSH key is one of those from the console-output command:
RSA key fingerprint is 20:9d:15:98:f6:71:fc:8d:4f:82:b6:e7:3f:76:e0:cc. Are you sure you want to continue connecting (yes/no)? yes
__| __|_ ) Fedora 8
_| ( / 32-bit
___|\___|___|
Welcome to an EC2 Public Image
:-)
Getting Started
--[ see /etc/ec2/release-notes ]--
[root@domU-12-31-39-03-46-52 ~]#
There is your login prompt. You are now a Linux system administrator.
Mounting snapshots
Creating a snapshot based on the Ensembl image, identification found at: Public Datasets
$ ec2-create-volume --snapshot snap-c78360ae -z us-east-1b VOLUME vol-399d7d50 550 snap-c78360ae us-east-1b creating 2009-04-03T17:19:02+0000
Attach that volume to a running image, as device /dev/sdh:
$ ec2-attach-volume vol-399d7d50 -i i-4bc1a022 -d /dev/sdh ATTACHMENT vol-399d7d50 i-4bc1a022 /dev/sdh attaching 2009-04-03T17:25:04+0000
Taking a look at those:
$ ec2-describe-volumes VOLUME vol-399d7d50 550 snap-c78360ae us-east-1b in-use 2009-04-03T17:19:02+0000 ATTACHMENT vol-399d7d50 i-4bc1a022 /dev/sdh attached 2009-04-03T17:25:04+0000
Examining instances:
$ ec2-describe-instances | grep running INSTANCE i-4bc1a022 ami-3c47a355 ec2-75-101-203-203.compute-1.amazonaws.com ip-10-251-126-22.ec2.internal running gsg-keypair 0 m1.small 2009-04-03T17:03:17+0000 us-east-1b aki-a71cf9ce ari-a51cf9cc
Logged into the instance, mounting that volume:
# mkdir /mnt/ensembl # mount /dev/sdh /mnt/ensembl # cd /mnt/ensembl # df -h . /dev/sdh 546G 535G 11G 99% /mnt/ensembl
After it was mounted:
$ ec2-describe-volumes VOLUME vol-b49d7ddd 550 snap-c78360ae us-east-1c in-use 2009-04-03T17:57:54+0000
ATTACHMENT vol-b49d7ddd i-bac4a5d3 /dev/sdh attached 2009-04-03T17:59:17+0000
Creating a snapshot
Create a 1000 Gb volume:
$ ec2-create-volume -s 1000 -z us-east-1c VOLUME vol-5e9e7e37 1000 us-east-1c creating 2009-04-03T18:09:45+0000
Attach that volume to a running image, as device /dev/sdi:
$ ec2-attach-volume vol-5e9e7e37 -i i-bac4a5d3 -d /dev/sdi
Make a new filesystem on this volume, logged into the instance:
# yes | mkfs -t ext3 /dev/sdi This filesystem will be automatically checked every 34 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
That took about 15 minutes, 8000 inode tables was the longest bit of the process. Create a mount point and mount it:
# mkdir /mnt/ucsc # mount /dev/sdi /mnt/ucsc
Load your data into that filesystem. It is going into the S3 storage system as that takes place. When it is complete, create a snapshot:
$ ec2-create-snapshot vol-5e9e7e3 SNAPSHOT snap-a8a352c1 vol-5e9e7e37 pending 2009-04-03T23:48:55+0000
Turning Off an Instance
You need to make sure you turn these instances off, they consume accounting charges if they are left on:
$ ec2-terminate-instances i-32a7c65b INSTANCE i-32a7c65b running shutting-down
