Monitoring Tasks

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task	frequency	Until Aug '18	Until Sep '19	Now
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be.	daily	Jairo	Jairo	Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.	weekly	Jairo	Conner	Lou
Retired (See #23998) receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day)	daily	Cath	Lou	Retired
Retired (See #23998) receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day)	daily	Cath	Lou	Retired
Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See #23998 for more info.	weekly		Lou, Conner	Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2	daily, problems every 3-6 months	Jairo	Jairo	Daniel
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here	weekly	Cath	Lou	Gerardo
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)	monthly	Brian	Daniel	Gerardo
receive an email on additions and subtractions to files in /gbdb on hgnfs1	daily	Brian	Conner	Lou
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate	daily	ChrisL, Brian	Jairo Brian Lou	Jairo, Brian, Lou
email request to check Genomes section of the Wikipedia Page and make other changes or updates if needed.	runs quarterly		Bob,Conner	Bob,Gerardo
email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required	runs monthly			BrianL
Retired (See #27425) email request to check doPublicCrawl public.ix and public.ixx hub search files and push them.	runs quarterly	Jairo	Daniel	Gerardo,Brian
increment databaseAday, /cluster/bin/scripts/databaseAday.csh	problems only	ChrisL, Cath, Brian	Jairo, Brian, Lou	Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go	problems only	ChrisL, Brian, Cath	Jairo, Brian, Lou	Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go	problems only	ChrisL, Cath, Brian	Jairo, Brian, Lou	Jairo, Brian, Lou
get output from checkGenomeMysql (discovers corrupt tables)	daily	Brian	Brian, Daniel	Daniel, Gerardo
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match.	runs weekly	Jairo, Cath	Jairo,Conner,Daniel	Lou,Gerardo,Daniel
look for broken symlinks in /gbdb:	daily	Browser Staff	Browser Staff	Browser Staff
look for broken symlinks in in (very large) visiGene directory	twice yearly	ChrisL	Conner	Brian
look for broken symlinks in htdocs-hgdownload/goldenpath	daily	Browser Staff	Browser Staff	Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git.	daily	Browser Staff	Browser Staff	Browser Staff
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK.	monthly	Ann [was: Matt, Brian]	Ann,Lou,Hiram	Ann,Jairo,Hiram
Monthly reminder to check for stale New pennantIcon tags	monthly		Lou	Lou
run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia	monthly	Angie, Brian, Bob	Angie, Brian, Bob	Angie, Lou, Bob
ask Galt/Jim about blat updates	3x per year	ChrisL,Cath	Daniel, Conner	Daniel, Gerardo
check that the ga4gh beacon server is running. Email Max if there is an issue.	daily	Jairo	Conner	Daniel
check that blat servers are running ok link to error log checking	ongoing	Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath	Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo	Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems	as needed	many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo	Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo	Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
check that beta builds alright	3x week	Brian, ChrisL	Brian, Braney, Jairo, Lou	Brian, Braney, Jairo, Lou
Retired (See #27779) hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh).	runs daily, complains moderately	ChrisL	Daniel, Braney, Christopher	Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.	hourly	Brian	Conner	Daniel
qaAutoTrack -script that does some auto-QA on the autopushed tracks.	daily	Jario	Jairo	Jairo
checkAllStaticLinks.csh script that checks static pages.	weekly		Daniel,Conner,Brian	Daniel,Gerardo,Brian
assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors.	monthly		QA Team	QA Team
HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates	weekly on Tuesday			QA Team, cluster admin
Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found here.	daily			Daniel, Gerardo
from build cron
receive cron output for tableDescriptions build errors, AutoPush completed	rare	Cath	QA Team	QA Team
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository)	daily	Galt	Galt	Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible.	weekly	Cath	Lou	Daniel
Monitor & report UCSC Genome Browser outages (SLA). See wiki procedures	hopefully, not much!	Cath	Conner	Gerardo
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates)	as needed	Brian	Brian	Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address	rarely	ChrisL,Matt	Daniel	Gerardo
Check genome-preview machine is still running well	weekly	Jairo	Conner	Daniel
Updated Facebook with Announcements	As needed	Team	Team	Team
Update Twitter with Announcements	As needed	Team	Team	Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so	runs daily, complains bi-weekly	jcasper	ChrisL	ChrisL
omimUploadWrapper.sh uploads the OMIM tables to OMIM	runs daily, complains rarely	jcasper	ChrisL	ChrisL
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so	runs daily, complains monthly	jcasper	ChrisL	ChrisL
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so	runs daily, complains monthly	jcasper	ChrisL	ChrisL
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so	runs weekly, complains quarterly	jcasper	ChrisL	ChrisL
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so	runs weekly, complains quarterly	jcasper	ChrisL	ChrisL
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree	runs weekly	jcasper	jcasper	jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/)	runs twice a week	jcasper	jcasper	jcasper