Unix environment

Working in the UNIX™ environment

Editor

The most important tool will most likely be your editor. It doesn't matter what you want to use, but whatever it is, learn it well. vi and emacs are the most common editors used in the unix environment. Your choice of editor will become critical when you use your shell command line in its editing mode. There are very good tutorials on the internet for your editor. There is a VI quick start command listing in genomewiki. See also: Editor War. <embedurl>http://www.cse.iitm.ac.in/~osslab/joomla/images/stories/vim-editor_logo.png{width=100}{height=100}</embedurl><embedurl>http://macin.files.wordpress.com/2008/10/carbon-emacs-icon.png{width=100}{height=100}</embedurl>

Shell

There are two shells in common use on unix: bash and tcsh. Next to your editor, your shell command line is going to be a critical element of your efficiency using unix. You will want the command line editing features turned on for your command line to recognize your favorite editor commands. Learn how to use your command line editing feature.

Understand what stdout, stderr and stdin are and how to control their input and output in compound shell commands. There are very good bash and tcsh tutorials on the internet. You will never again compose a long command line just to find out it has a typo error in it, and then have to type the whole thing in again. Use your command line editor to rapidly fix the typo to repeat the corrected command.

Also, verify that you can easily cut and paste between your shell command line and other applications on your desktop. This function depends upon what kind of desktop you operate. Each desktop may have different mechanisms for this function.

Customize your login shell with a good .rc file. For example, note customizations in Hiram's file:

~hiram/.bashrc.hiram

Do not place passwords or other sensitive information in these files. chmod them to 644 and allow your work to inspire others.

See also: [Bash vs. Csh]

Regular Expressions

You will be using regular expressions in your editor, your shell and in other commands. Just about everything. You will need to know how to use them. You can get pretty far with a minimal familiarity of the basics. Keep a reference handy for the odd cases where you need to use the more extensive operations.

Commands

Finding good commands

You can use the apropos command to find a command related to some function you would like to perform. The apropos command performs a simple string search, of a given word, through the documentation pages to output a single line header of commands matching that string. The command may be related to what you want to accomplish.

<embedurl>http://farm1.static.flickr.com/186/451709341_0930c677e0.jpg?v=0{width=510}{height=330}</embedurl>

Use the man someCommand command to view the manual page for someCommand

[hiram@hgwdev /tmp] apropos apropos
apropos              (1)  - search the whatis database for strings
man                 (rpm) - A set of documentation tools: man, apropos and whatis.
[hiram@hgwdev /tmp] man apropos
apropos(1)                                                          apropos(1)
NAME
      apropos - search the whatis database for strings
... etc ...

Fun tricks with man. Try this:

man -Pcat man | cat -A | less

Then:

man -Pcat man | col -b | cat -A | less

The (1) in apropos(1) indicates which section of the manuals the command can be found in. The first set of books were labeled with the integers 1 2 3 4 ... Therefore section (1) indicates the first book. This numbering system broke down after the manuals became electronic documents without any real instantiation in an actual book.

grep

The grep command is used to find lines in text files, or in streaming output from previous pipeline commands. Given a string, any line matching that string is printed out. The output can be the inverse, printing out any lines that do not match the string.

Example: select only the bed format lines from a custom track file so they can be used with hgLoadBed. Removing any lines that begin with track or browser, an example of the inverse function with the -v argument:

grep -v "^track|^browser" customTrack.txt > file.bed

Alternatively, if it is known that all chromosome names start with "chr", select only those lines:

grep "^chr" customTrack.txt > file.bed

Both these examples assume there is only one track defined in customTrack.txt

If you want your string to instead be an actual regular expression, use grep with the -E argument, or use the egrep command which is equivalent to grep -E.

To efficiently scan an entire directory hierarchy of files, use the following find | xargs grep pipeline:

find . -type f -print0 | xargs --null grep "<your string>"

The -print0 argument to find combined with the --null argument on xargs makes this pipeline work properly even if the file names include blanks. If you know all your file names have no blanks, omit the -print0 and --null. Blanks in unix file names are discouraged since they always need to be taken care of in special ways when working with a command line which interprets whitespace as the delimiter for separate strings. The usual work-around is to use underscore in place of_blanks_in_file_names.

To grep the contents of a manual page. For example, find the string apropos in the man manual page:

man -Pcat man | col -b | grep -i apropos

See also: Sieve

sed

The sed command is used to perform batch text editing in files. The simple format:

sed -e "s/search string/replace string/g" someFile.txt > result.txt

Will replace all occurrences of "search string" with "replace string" in someFile.txt with the result going to result.txt. "g" means global -- will replace every instance of the string on a line, not just the first. If this looks somewhat familiar to the perl syntax:

result =~ s/search string/replace string/g;

it is because perl copied this syntax from sed. The search string can be a regular expression. For example, given a bed file from Ensembl with Ensembl chromosome names (numbers), change them to UCSC chromosome names, prefixed with "chr":

sed -e "s/^\([0-9XY][0-9]*\)/chr\1/; s/^MT/chrM/; s/^Un/chrUn/" ensembl.bed > ucsc.bed

The backslash paren syntax \(...\) captures that match in a sequentially numbered buffer which can later be used in the replace string as \1 where 1 is the sequential buffer count. The carat ^ anchors the match to the beginning of the line. Multiple operations are separated by the semicolon; Know your regular expressions well to get the best results from sed. It is a very powerful and useful tool.

awk

<embedurl>http://upload.wikimedia.org/wikipedia/commons/0/0b/Riesenalk.JPG{width=100}{height=181}</embedurl> The awk command is a simple programming language useful in processing text files or pipeline streams of text.

From the man page for awk, the fundamental definition is pretty simple:

An  AWK program consists of a sequence of pattern-action statements and optional function definitions.
  pattern   { action statements }
  function name(parameter list) { statements }

If pattern is omitted, it matches all input lines. If the { action statements } are omitted, the default action is { print }.

pattern can be a regular expression, or a variety of other matching expressions. The man page tells all. I'm guessing a google search for awk tutorial would lead to some useful exercises.

For example, given a bed file, select out lines where the score is above 650:

awk '$5 > 650' file.bed > result.bed

By default, awk separates input lines into fields by white space. To work with tab separated files with tab as the field separator, set the FS and OFS variables to tab for awk. For example, convert an Ensembl gff file into a UCSC bed6 file, fixing the chromosome names and using the frame offset as the bed score column:

zcat Homo_sapiens.GRCh37.57.gtf.gz \
       | awk -F'\t' -v 'OFS=\t' '{
sub(/^MT/,"M",$1)
sub(/^/,"chr",$1)
sub(/\./,0,$8)
print $1,$4,$5,$3,$8,$7
}'

The chromosome names are converted with the first two sub() substitutions, the third sub() converts the frame "." to "0" to print out a number for the bed score column as a frame number.

Example, compute a running sum of a column of numbers in a file, for example a chrom.sizes file:

awk -v 'OFS=\t' '{sum += $2; print sum,$0}' chrom.sizes

Example, find the longest line in a file:

awk '{print length($0),$0}' chrom.sizes | sort -rn | head -1
28 chr19_gl000209_random        159169

Screen

The screen command is a useful virtual terminal to allow you to take your terminal session from one location to another.

See also

Unix-Haters Handbook