QA scripts

This is a list of the most frequently-used programs and scripts that QAers use. It does not include track-specific programs, such as chainNetTrio.csh, or these. Some are devised by QA and live in the source tree at kent/src/utils/qa (these mostly end in ".csh"). Some are programs that the engineers or system administrators have written.

Running Scripts on Multiple Tables at Once

For the scripts that do not accept a list of tables in a file, here's a little bash script that you can use to run any of these scripts on multiple tables. Save the list of tables in a file (in your hive directory) called 'tableList_temp', and then type (note: you can press return after each line):

for i in $(cat tableList_temp)
   do
   [script_name] [database/assembly] $i
   echo $i
done

Can also be run as one line for easy copy/paste. Example:

for i in $(cat tableList); do runJoiner.csh danRer5 $i; echo $i; done

The following scripts do not accept a file, so are good candidates for the above:

• runJoiner.csh
• countPerChrom.csh
• runBits.csh
• (more coming...)

Must know about!

bigPush.csh

(also see: mypush)

Pushes tables in list to mysqlbeta and records size.
  Requires sudo access to mypush to run.
  
  Do not redirect output or run in the background,
  as it will require you to type your password in.
  Program will ask you for your password again after
  large tables. If you take too long to re-type in
  the table the script stalled on might not get
  pushed. Double-check that all tables have been
  pushed!
  
  Will report total size of push and write two files:
  db.tables.push -> output for all tables from mypush
  db.tables.pushSize -> size of push

commTrio.csh

 Sorts and compares two files.  
 Counts unique and common records.

     usage:  leftFileName rightFileName [rm]
             optional [rm]: remove the three output files when finished

compareWholeColumn.csh

  gets a column from a table on dev and beta and checks diffs.
  reports numbers of rows unique to each and common.
  can compare to older database.
  writes files of everything.

    usage:  database table column [db2] 

compareWholeTable.csh

  gets an entire table from two machines and checks diffs.
  reports numbers of rows unique to each and common.
  writes files of everything.
  not real-time on RR -- uses genome-mysql.

    usage:  database table [machine1] [machine2]
      (defaults to dev and beta)

countPerChrom.csh

  check to see if there are annotations on all chroms.
  will check to see if chrom field is named tName or genoName.

    usage:  database1 table [database2] [RR] [histogram]

      checks database1 on dev
      database2 will be checked on beta by default
        if RR is specified, will use genome-mysql
      histogram option prints bar graph, not values

featureBits

(also see: getYield.csh)

 featureBits - Correlate tables via bitmap projections. 
 usage:
   featureBits database table(s)

(truncated for brevity)

findLevel

  searches trackDb hierarchy for your table and corresponding .html file
  also returns the value of the priority and visibility entries
  and the .ra file location for each

    usage:  database tableName

findOrg.csh

 Finds the organism name given the assembly name
  usage: assemblyName [date]
         will accept name with or without digit
         use 'date' to also retrieve assembly date
         (e.g. 'ornAna2' or 'ornAna')

getAssemblies.csh

  gets the names of all databases that contain a given table.
  will accept the MySQL wildcard, %, but not on RR machines
  note: not real-time on RR.  uses nightly TABLE STATUS dump.

    usage:  tablename [machine] [verbose] - defaults to beta
              "verbose" prints list of assemblies checked

getTrackName.csh

 Returns the short label and group of the track for this table.
 In the case of a composite track, it returns the short label
 for both the sub track and the parent track.

    usage:  database tableName

joinerCheck

(also see: runJoiner.csh)

 joinerCheck - Parse and check joiner file
 usage:
   joinerCheck file.joiner
 options:
   -fields - Check fields in joiner file exist, faster with -fieldListIn
      -fieldListOut=file - List all fields in all databases to file.
      -fieldListIn=file - Get list of fields from file rather than mysql.
   -keys - Validate (foreign) keys.  Takes about an hour.
   -tableCoverage - Check that all tables are mentioned in joiner file
   -dbCoverage - Check that all databases are mentioned in joiner file
   -times - Check update times of tables are after tables they depend on
   -all - Do all tests: -fields -keys -tableCoverage -dbCoverage -times
   -identifier=name - Just validate given identifier.
                    Note only applies to keys and fields checks.
   -database=name - Just validate given database.
                    Note only applies to keys and times checks.
   -verbose=N - use verbose to diagnose difficulties. N = 2, 3 or 4 to
              - show increasing level of detail for some functions.

mypush

used with sudo, generally like so, from hgwdev:

• sudo mypush $db $table mysqlbeta

Usage: mypush database table-pattern [hostlist]
 NOTE: use single quotes around table-pattern  
       if it contains shell special chars like * or ? 

realTime.csh

(also see: updateTimes.csh)

  gets update times from all machines in real time for tables in list.

    usage:  database tablelist (will accept single table)

runBits.csh

  runs featureBits and checks for overlap with gaps.

    usage:  database trackname [checkUnbridged]
              where overlap with unbridged gaps can be turned on

updateTimes.csh

(also see: realTime.csh)

  gets update times for three machines for tables in list.
  if table is trackDb, trackDb_public will also be checked.
  warning:  not in real time for RR.  uses overnight dump.

    usage:  database tablelist 

            reports on dev, beta and RR
            tablelist will accept single table

Might also like!

checkBOT.csh

This is useful for answering mailing list questions. Also see the "bottleneck" command.

  wrapper around bottleneck check.
  gives delay stats for IP address(es).

      usage:  ipAddress [terse]
              (terse gives only data)

      (use ipAddress = "all" to get all IPs having delays)

compareTrackDbAll.csh

  checks all fields in trackDb

    usage: database [machine1] [machine2] [mode] 

      (defaults to hgw1 and hgwbeta)
      mode = (fast | verbose | fastVerbose) 
       - verbose is for html field - defaults to terse
       - fast = (genome-mysql) - defaults to realTime (WGET)

checkPushedFiles.csh

 checks to see if files are in place, after a push

 usage: website files(s)

 website should include the path of the directory where
 the files reside, such as:
   http://hgdownload.cse.ucsc.edu/goldenPath/hg19/liftOver/ 

 file(s) is either a single name or a list of names, and can
 include items with additional directory structure, like so:
   filename
   dir/filename
   dir/dir/dir/filename

 any output other than '200 OK' indicates an error.

compareTableToFile.csh

 Ensures that a table correlates with its associated file.
 Only prints results if there is a diff between table and file.
 Works for these file types: narrowPeak, broadPeak, gappedPeak,
                             bedGraph, NRE, BiP, gcf
 For wiggle files, you must specify [wig] parameter.

  usage:  database tableName fileName [wig] [verbose]
   fileName includes path of download file 
   e.g. /goldenPath/<db>/fileName.gz

   use verbose for more details

copyExtSeqRows.csh

 Automatically copies appropriate rows from the extFile and seq tables
 from hgwdev to hgwbeta.

(truncated for brevity)

countRows.csh

  gets the rowcount for a list of tables from dev, beta and RR.

    usage:  database tablelist [genome-mysql]
      tablelist can be just name of single table

    RR results not in real time, but from dumps
    genome-mysql option adds results from public mysql server

findBlatServer.csh

 gets info about which blat server hosts which genome(s)

 usage:  db|host|all  [db|host]  [machine]
   first parameter required: one specific db or host or all dbs
   second parameter optional: order by db or by host (blatServer)
     defaults to order by db
   third parameter optional: specify machine
     defaults to RR

findColumn.csh

  searches database for all tables containing a specified column name.

    usage:  database, field, [machine = hgwdev|hgwbeta] 
      (defaults to beta)

findPushQLocks.csh

  find all locks in the pushQ on hgwbeta

    usage: go|real
     run with 'go' to see a list of locks
     run with 'real' to unlock all the locks

getChainLines.csh

(also see: getMatrixLines.csh)

  Searches the README.txt files to find the correct parameters for the 
  $chainMinScore and $chainLinearGap variables. 

    usage:  fromDb toDb (these can be in either order)

getChromlist.csh

  prints the chrom names for an assembly.

    usage:  database [norandom]

getMatrixLines.csh

(aslo see: getChainLines.csh)

  Searches the README.txt files to find the correct parameters for the 
  $matrix variable.  This is the q-parameter from the blastz run. 

    usage:  fromDb toDb (these can be in either order)

getYield.csh

(also see: featureBits)

  uses featureBits to get yield and enrichment.

    usage:  database trackname [reference track]
              refTrack defaults to refGene

runJoiner.csh

  runs joinerCheck -keys, finding all identifiers for a table.
  runs joinerCheck -times (use "noTimes" to disable).
  set database to "all" for global.
  for chains/nets, use tablename format: chainDb.

    usage:  database table [all.joiner file to use] [noTimes]

tdbQuery

tdbQuery - Query the trackDb system using SQL syntax.
Usage:
    tdbQuery sqlStatement
Where the SQL statement is enclosed in quotations to avoid the shell interpreting it.
Only a very restricted subset of a single SQL statement (select) is supported.   Examples:
    tdbQuery "select count(*) from hg18"
counts all of the tracks in hg18 and prints the results to stdout
   tdbQuery "select count(*) from *"
counts all tracks in all databases.
   tdbQuery "select  track,shortLabel from hg18 where type like 'bigWig%'"
prints to stdout a a two field .ra file containing just the track and shortLabels of bigWig 
type tracks in the hg18 version of trackDb.
   tdbQuery "select * from hg18 where track='knownGene' or track='ensGene'"
prints the hg18 knownGene and ensGene track's information to stdout.
   tdbQuery "select *Label from mm9"
prints all fields that end in 'Label' from the mm9 trackDb.
OPTIONS: ...

(truncated for brevity)

ENCODE QA scripts

These are scripts that the ENCODE QA team frequently uses.

encodeEmail.pl

usage: /cluster/bin/scripts/encodeEmail.pl <email addresses>
	The email addresses will be encoded and sent to stdout

encodeQaCheckHgdownloadFiles

usage: encodeQaCheckRRFiles [-h] [-d] [-s SERVER] database composite files

Compares files on the RR against a list of files

positional arguments:
  database              The database, typically hg19 or mm9
  composite             The composite name, wgEncodeCshlLongRnaSeq for
                        instance
  files                 The list of files

optional arguments:
  -h, --help            show this help message and exit
  -d, --dev             Check for files missing on dev that are present on the
                        RR
  -s SERVER, --server SERVER
                        The server to use, like hgdownload.

Example:
encodeQaCheckHgdownloadFiles hg19 wgEncodeSydhTfbs files.list
encodeQaCheckHgdownloadFiles hg18 wgEncodeHudsonalphaChipSeq checkPushFilesList

encodeQaInit

For more specific info about the other scripts this script runs and the files it creates see the ENCODE QA wiki.

usage: encodeQaInit [-h] [-t] [-m MDB] database composite release redmine

Initializes QA directory for claiming a release

positional arguments:
  database           The database, typically hg19 or mm9
  composite          The composite name, wgEncodeCshlLongRnaSeq for instance
  release            The new release to be released
  redmine            The Redmine issue number

optional arguments:
  -h, --help         show this help message and exit
  -t, --test         Test mode:doesn't change status to reviewing, outputs to
                     test qa Directory
  -m MDB, --mdb MDB  use a different mdb composite name

Example:
encodeQaInit hg19 wgEncodeSydhTfbs 1 69
encodeQaInit hg18 wgEncodeHudsonalphaChipSeq 3 504

encodeQaPrepareRelease

usage: encodeQaPrepareRelease [-h] database composite stage

Stages a track either to beta or to public

positional arguments:
  database    The database you're using
  composite   The composite you're using
  stage       The stage you are staging to

optional arguments:
  -h, --help  show this help message and exit

Examples:
           encodeQaPrepareRelease hg19 wgEncodeSydhTfbs beta
           encodeQaPrepareRelease hg19 wgEncodeHaibTfbs public

encodeQaSqlRelease

Creates a pushQ entry directly in the L queue of the Main pushQ so the ENCODE track will have an entry in the release log.

usage: encodeQaSqlRelease <release.sql> <sponsor>
example: encodeQaSqlRelease release.sql wong

encodeStatus.pl

Sets the ENCODE status of subIds inputted (QA usually only uses after releasing a track, setting the subIds' ENCODE status to 'released').

usage: encodeStatus [-instance=instanceName] [-force] project-id|project-name [status]

valid statuses: loaded, displayed, approved, reviewing, released

-instance	Default instance is 'prod'
-force		Use if you want to set a status that is not normally allowed (e.g. to reset
		to an earlier status).

mdbPrint

Useful for checking the experiments (expIds) have been done correctly, see ENCODE QA wiki.

mdbPrint - Prints metadata objects, variables and values from 'metaDb' table.
usage:
   mdbPrint {db} [-table=] [-byVar] [-line/-count]
                 [-all]
                 [-vars="var1=val1 var2=val2..."]
                 [-obj= [-var= [-val=]]]
                 [-var= [-val=]]
                 [-specialHelp]
Options: ...

(truncated for brevity)

qaAddTrackReferences

From list of pubMed Ids, provides html output of references in CBSE citation format.

usage qaAddTrackReferences <pubmed_id1> <pubmed_id2> ... <pubmed_idn>

qaEncodeTracks.csh

Runs test suite for ENCODE tracks (this script is run by encodeQaInit, which puts the output in the script.out file, but can still be run on its own).

 Runs test suite for ENCODE tracks
 (it's best to direct output and errors to a file: '>&')
 In general, this script only prints output if there are problems

    usage: db tableList

raDiff

Used on metaDb .ra files and cv.ra files; may be expanded to trackDb .ra files.

usage: qaRaDiff [-h] RaFileOne RaFileTwo

Describes the differences between the two .ra files

positional arguments:
  RaFileOne   The .ra file
  RaFileTwo   The .ra file to compare to

optional arguments:
  -h, --help  show this help message and exit

example: qaRaDiff alpha/wgEncodeUwTfbs.ra beta/wgEncodeUwTfbs.ra

raMerge

Used on metaDb .ra files and cv.ra files; may be expanded to trackDb .ra files.

usage: raMerge [-h] [-t] RaFileOne RaFileTwo

Merges two .ra files in a way that you would expect

positional arguments:
  RaFileOne      The .ra file
  RaFileTwo      The .ra file to merge with

optional arguments:
  -h, --help     show this help message and exit
  -t, --trackDb  Print as trackDb

example: raMerge alpha/wgEncodeUwTfbs.ra beta/wgEncodeUwTfbs.ra