Monitoring Tasks

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task	frequency	who Mar 17' to Aug '18	who April '16 -Feb '17	_	_	NOW
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be.	daily	Jairo	Cath, Matt			Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.	weekly	Jairo	Cath			Conner
receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day)	daily	Cath	ChrisV			Lou
receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day)	daily	Cath	Cath, Matt			Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2	daily, problems every 3-6 months	Jairo	Cath, Luvina			Jairo
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here	weekly	Cath	Brian, Jonathan			Lou
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)	monthly	Brian	Matt, Brian			Daniel
receive an email on additions and subtractions to files in /gbdb on hgnfs1	daily	Brian	ChrisL, Luvina			Conner
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate	daily	ChrisL, Brian	Matt, Brian			Jairo Brian Lou
email request to build doPublicCrawl public.ix and public.ixx hub search files and push them.	runs quarterly	Jairo	ChrisL			Daniel
increment databaseAday, /cluster/bin/scripts/databaseAday.csh	problems only	ChrisL, Cath, Brian	Brian, Jonathan			Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go	problems only	ChrisL, Brian, Cath	Brian, Jonathan			Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go	problems only	ChrisL, Cath, Brian	Brian, Jonathan			Jairo, Brian, Lou
get output from checkGenomeMysql	daily	Brian	ChrisL, Matt			Brian, Daniel
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match.	runs weekly	Jairo, Cath	Brian, Cath, Jonathan			Jairo,Conner,Daniel
Check for unigene update for human SAGE track	daily	Cath, Braney	Luvina, Braney			?
look for broken symlinks in /gbdb:	daily	Browser Staff	Browser Staff
look for broken symlinks in in (very large) visiGene directory	twice yearly	ChrisL	Cath, Galt			Conner
look for broken symlinks in htdocs-hgdownload/goldenpath	daily	Browser Staff	Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git.	daily	Browser Staff	Browser Staff
Check to see what kind of browsers and o/s people are using to access the public website	weekly	Ann, Jairo, ChrisL	Ann, ChrisL, Brian			Ann, Jairo, Daniel
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK.	monthly	Ann [was: Matt, Brian]	Brian, Luvina			Ann,Lou,Hiram
run /cluster/bin/scripts/sessionStats.csh	monthly	Angie, Brian, Bob	Angie, Brian, Bob			Angie, Brian, Bob
ask Galt/Jim about blat updates	3x per year	ChrisL,Cath	Cath, Brian			Daniel, Conner
check that the ga4gh beacon server is running. Email Max if there is an issue.	daily	Jairo	Cath			Conner
check that blat servers are running ok	ongoing	Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath	Ann, Brian, Donna, Galt, Luvina, Matt, Gary,			Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems	as needed	many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo	many: cath, chmalee, kuhn, brianlee, braney, luvina, gary, ann,, jcasper, matt			Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo
check that beta builds alright	3x week	Brian, ChrisL	Brian, ChrisL,Braney			Brian, Braney, Jairo, Lou
hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh).	runs daily, complains moderately	ChrisL	Matt			Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.	hourly	Brian	Brian			Conner
qaAutoTrack -script that does some auto-QA on the autopushed tracks.	daily	Jario				Jairo
checkAllStaticLinks.csh script that checks static pages.	weekly					Daniel,Conner,Brian
from build cron
receive cron output for tableDescriptions build errors	rare	Cath	Luvina			?
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository)	daily	Galt	Galt			Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible.	weekly	Cath	Luvina			Lou
Monitor & report UCSC Genome Browser outages. See wiki procedures	hopefully, not much!	Cath	Cath, Luvina			Conner
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates)	as needed	Brian	Brian			Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address	rarely	ChrisL,Matt	Matt			Daniel
Check genome-preview machine is still running well	weekly	Jairo	Brian			Conner
Updated Facebook with Announcements	As needed	Team	Brian/Alexa/Team			Tea,
Update Twitter with Announcements	As needed	Team	Brian/Alexa/Team			Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so	runs daily, complains bi-weekly	jcasper	jcasper			jcasper
omimUploadWrapper.sh uploads the OMIM tables to OMIM	runs daily, complains rarely	jcasper	jcasper			jcasper
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so	runs daily, complains monthly	jcasper	jcasper			jcasper
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so	runs daily, complains monthly	jcasper	jcasper			jcasper
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so	runs weekly, complains quarterly	jcasper	jcasper			jcasper
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so	runs weekly, complains quarterly	jcasper	jcasper			jcasper
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree	runs weekly	jcasper	jcasper			jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/)	runs twice a week	jcasper	jcasper			jcasper