Monitoring Tasks: Difference between revisions

From Genecats
Jump to navigationJump to search
No edit summary
(Adding the Hub Public Coordinator role to Monitoring Tasks)
 
(144 intermediate revisions by 11 users not shown)
Line 1: Line 1:
Please add to the documentation page, [[Monitoring_Tasks_Notes|Monitoring Tasks Notes]] to add procedures and notes for specific jobs, then link back to the job in the list below. 
<table border="1">
<table border="1">


Line 4: Line 5:
<th>task</th>
<th>task</th>
<th>frequency</th>
<th>frequency</th>
<th>who (starting Aug '13)</th>
<th> Until Aug '18</th>
<th>who (through July '13)</th>
<th>Until Sep '19</th>
<th>Now</th>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>from qateam cron</b></td>
<td colspan="7" style="background-color:gray"><b>from [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs hgwdev] qateam cron</b></td>
</tr>
 
<tr>
<td>ask Galt/Jim about blat updates</td>
<td>3x per year</td>
<td>Pauline, Steve, Jonathan</td>
<td>Pauline, Steve, Jonathan</td>
</tr>
</tr>


Line 22: Line 17:
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>Respond to items on the broken links report: Check to make sure that broken links reported are actually broken.  If so, fix the link if it is clear what the new link should be and push to the RR.  If not, contact the appropriate person at the web site in question to find out what the new link should be.</td>
<td>daily</td>
<td>daily</td>
<td>Matt</td>
<td>Jairo</td>
<td>Steve</td>
<td>Jairo</td>
<td>Jairo</td>
</tr>
</tr>


Line 29: Line 25:
<td>Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.</td>
<td>Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report.</td>
<td>weekly</td>
<td>weekly</td>
<td>Luvina</td>
<td>Jairo</td>
<td>Steve</td>
<td>Conner</td>
<td>Lou</td>
 
</tr>
 
 
<tr>
<td>'''Retired (See #23998)''' receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) </td>
<td>daily</td>
<td>Cath</td>
<td>Lou</td>
<td>Retired</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>'''Retired (See #23998)''' receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkMetaAday.csh checkMetaAday.csh]; alert;  if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) </td>
<td>as needed</td>
<td>daily</td>
<td>many: kuhn, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper, matt</td>
<td>Cath</td>
<td>many: kuhn, rhead, pauline, katrina, brianlee, braney, luvina, gary, ann, steve, jcasper</td>
<td>Lou</td>
<td>Retired</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from gbLoaded cron job and alert braney if anything seems very outdated</td>
<td> Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See [http://redmine.soe.ucsc.edu/issues/23998 #23998] for more info. </td>
<td>daily, problems every 3-6 months</td>
<td>weekly</td>
<td>Steve, Jonathan</td>
<td></td>
<td>Brooke, Steve, Jonathan</td>
<td>Lou, Conner</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from showProcesslist.csh</td>
<td>receive output from [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_realTime.csh_.28previously_known_as_gbLoaded.29 realTime.csh] (previously gbLoaded) cron job and alert braney if anything seems very outdated <br> No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2</td>
<td>daily, no action</td>
<td>daily, problems every 3-6 months</td>
<td>Steve, Jonathan</td>
<td>Jairo</td>
<td>Brooke, Steve, Jonathan</td>
<td>Jairo</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...if we could get rid of our dependence on those, we could get rid of this cron job)</td>
<td>get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkTableStatus.csh_.22_TABLE_STATUS_dump.22_emails here]</td>
<td>weekly</td>
<td>weekly</td>
<td>Steve, Jonathan</td>
<td>Cath</td>
<td>Brooke, Steve, Jonathan</td>
<td>Lou</td>
<td>Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td>get an email from backupCentral.csh each month (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834)</td>
<td>monthly</td>
<td>monthly</td>
<td>Steve, Jonathan</td>
<td>Brian</td>
<td>Brooke, Steve, Jonathan</td>
<td>Daniel</td>
<td>Gerardo</td>
</tr>
</tr>


Line 71: Line 83:
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>receive an email on additions and subtractions to files in /gbdb on hgnfs1</td>
<td>daily</td>
<td>daily</td>
<td>Steve, Jonathan</td>
<td>Brian</td>
<td>Brooke, Steve, Jonathan</td>
<td>Conner</td>
<td>Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>receive an email from cron if qateam DailySandboxUpdate fails</td>
<td>receive an email from cron if qateam DailySandboxUpdate fails, '''note:''' emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate</td>
<td>daily</td>
<td>daily</td>
<td>Steve, Jonathan</td>
<td>ChrisL, Brian</td>
<td>Brooke, Steve, Jonathan</td>
<td>Jairo Brian Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
 
<tr>
<td>email request to check [https://en.wikipedia.org/wiki/UCSC_Genome_Browser#Genomes Genomes section of the Wikipedia Page] and make other changes or updates if needed.</td>
<td>runs quarterly</td>
<td></td>
<td>Bob,Conner</td>
<td>Bob,Gerardo</td>
</tr>
 
<tr>
<td>email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required</td>
<td>runs monthly</td>
<td></td>
<td></td>
<td>BrianL</td>
</tr>
 
<tr>
<td>'''Retired (See #27425)''' email request to check [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Push_to_the_RR doPublicCrawl] public.ix and public.ixx hub search files and push them.</td>
<td>runs quarterly</td>
<td>Jairo</td>
<td>Daniel</td>
<td>Gerardo,Brian</td>
</tr>
 
 
<tr>
<td> increment databaseAday, /cluster/bin/scripts/databaseAday.csh </td>
<td>problems only</td>
<td>ChrisL, Cath, Brian</td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from checkMetaAday.csh; alert (steve, jcasper) if anything is out of sync (checks that tables are in sync between beta and RR for database of the day)</td>
<td>checkHgFindSpec -checkTermRegex,  /cluster/bin/scripts/runCheckHgFindSpec go</td>
<td>problems only</td>
<td>ChrisL, Brian, Cath</td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
 
<tr>
<td>check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go</td>
<td>problems only</td>
<td>ChrisL, Cath, Brian</td>
<td>Jairo, Brian, Lou</td>
<td>Jairo, Brian, Lou</td>
</tr>
 
 
<tr>
<td>get output from checkGenomeMysql (discovers corrupt tables)</td>
<td>daily</td>
<td>daily</td>
<td>Matt</td>
<td>Brian</td>
<td>Luvina</td>
<td>Brian, Daniel</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td>receive output from syncAday.csh; alert (steve, jcasper) if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day)</td>
<td>[[http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#Cronjob:_Results_from_checkGbibMd5.sh checkGbibMd5.sh]] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match.</td>
<td>runs weekly</td>
<td>Jairo, Cath</td>
<td>Jairo,Conner,Daniel</td>
<td>Lou,Gerardo,Daniel</td>
</tr>
 
<tr>
<td>look for broken symlinks in /gbdb:</td>
<td>daily</td>
<td>daily</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
<tr>
<td>look for broken symlinks in in (very large) visiGene directory</td>
<td>twice yearly</td>
<td>ChrisL</td>
<td>Conner</td>
<td>Brian</td>
<td>Brian</td>
<td>Luvina</td>
</tr>
</tr>


<tr>
<tr>
<td>get output from checkGenomeMysql</td>
<td>look for broken symlinks in htdocs-hgdownload/goldenpath</td>
<td>daily</td>
<td>daily</td>
<td>Steve, Jonathan</td>
<td>Browser Staff</td>
<td>Brooke, Steve, Jonathan</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>from build cron</b></td>
<td>Check the source tree for duplicate case-independent filenames. These wreck havoc with git.</td>
<td>daily</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
<td>Browser Staff</td>
</tr>
</tr>


<tr>
<tr>
<td>receive cron output for tableDescriptions build errors</td>
<td>sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK.</td>
<td>rare</td>
<td>monthly</td>
<td>Pauline, Matt</td>
<td>Ann [was: Matt, Brian]</td>
<td>Brooke, Pauline</td>
<td>Ann,Lou,Hiram</td>
<td>Ann,Jairo,Hiram</td>
</tr>
 
<tr>
<td>Monthly reminder to check for stale New pennantIcon tags</td>
<td>monthly</td>
<td></td>
<td>Lou</td>
<td>Lou</td>
</tr>
 
 
<tr>
<td>run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia</td>
<td>monthly</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Brian, Bob</td>
<td>Angie, Lou, Bob</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>from galt's cron</b></td>
<td>ask Galt/Jim about blat updates</td>
<td>3x per year</td>
<td>ChrisL,Cath</td>
<td>Daniel, Conner</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td>run ReleaseLog script (this requires a copy of the kent repository)</td>
<td>check that the [https://genomicsandhealth.org/work-products-demonstration-projects/beacon-project-0 ga4gh] [http://genome.ucsc.edu/cgi-bin/hgBeacon beacon server] is running. Email Max if there is an issue. </td>
<td>daily</td>
<td>daily</td>
<td>Galt (if QA takes this over, Pauline will do it)</td>
<td>Jairo</td>
<td>Galt</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>back up the qapushq</td>
<td>check that blat servers are running ok [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#check_that_blat_servers_are_running_ok link to error log checking]</td>
<td>every 30 minutes during the work day/week</td>
<td>ongoing</td>
<td>Galt</td>
<td>Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath</td>
<td>Galt</td>
<td>Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>from braney's cron</b></td>
<td>receive output from hgTracksRandom cron; alert admins to problems</td>
<td>as needed</td>
<td>many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo</td>
<td>Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo</td>
<td>Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo</td>
</tr>
</tr>


<tr>
<tr>
<td>braney has a cron that runs /cluster/bin/x86_64/hubPublicCheck and complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. This should be claimed by a QAer and transferred to the qateam cron.</td>
<td>check that beta builds alright</td>
<td>runs daily, complains rarely?</td>
<td>3x week</td>
<td>Brian Lee</td>
<td>Brian, ChrisL</td>
<td>braney</td>
<td>Brian, Braney, Jairo, Lou</td>
<td>Brian, Braney, Jairo, Lou</td>
</tr>
</tr>
<tr>
<tr>
<td>'''Retired (See #27779)''' hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. '' (NOTE: To run only on Beta hubs requires these declarations: '''export HGDB_CONF=$HOME/.hg.conf.beta'''  and '''SHELL=/bin/sh''')''.</td>
<td>runs daily, complains moderately</td>
<td>ChrisL</td>
<td>Daniel, Braney, Christopher</td>
<td>Daniel, Braney, Christopher</td>
</tr>


<td>braney has a cron that runs /hive/data/outside/otto/omim/omimWrapper.sh that checks for new OMIM data and downloads and loads it into hgwdev if so</td>
<tr>
<td>runs daily, complains bi-weekly</td>
<td>Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response.</td>
<td></td>
<td>hourly</td>
<td>braney</td>
<td>Brian</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>qaAutoTrack -script that does some auto-QA on the autopushed tracks.</td>
<td>daily</td>
<td>Jario</td>
<td>Jairo</td>
<td>Jairo</td>
</tr>


<td>braney has a cron that runs /hive/data/outside/otto/omim/omimUploadWrapper.sh that uploads the OMIM tables to OMIM</td>
<tr>
<td>checkAllStaticLinks.csh script that checks static pages.</td>
<td>weekly</td>
<td></td>
<td></td>
<td>braney</td>
<td>Daniel,Conner,Brian</td>
<td>Daniel,Gerardo,Brian</td>
</tr>
</tr>


<tr>
<tr>
<td>braney has a cron that runs /hive/data/outside/otto/decipher/decipherWrapper.sh that checks for new DECIPHER data and downloads and loads it into hgwdev if so</td>
<td>assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors.</td>
<td>runs daily, complains monthly</td>
<td>monthly</td>
<td></td>
<td></td>
<td>braney</td>
<td>QA Team</td>
<td>QA Team</td>
</tr>
</tr>


<tr>
<tr>
<td>braney has a cron that runs /hive/data/outside/otto/isca/iscoWrapper.sh that checks for new ISCA data and downloads and loads it into hgwdev if so</td>
<td>HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates</td>
<td>runs daily, complains monthly</td>
<td>weekly on Tuesday</td>
<td></td>
<td></td>
<td></td>
<td>braney</td>
<td>QA Team, cluster admin</td>
</tr>
</tr>


<tr>
<tr>
<td>braney has a cron that runs /hive/data/outside/otto/gwas/gwasWrapper.sh that checks for new GWAS data and downloads and loads it into hgwdev if so</td>
<td>Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found [http://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA#Hub_Public_Coordinator_Role here].</td>
<td>runs daily, complains quarterly</td>
<td>daily</td>
<td></td>
<td></td>
<td></td>
<td>braney</td>
<td>Daniel, Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>from ann's cron</b></td>
<td colspan="7" style="background-color:gray"><b>from build cron </b></td>
</tr>
</tr>


<tr>
<tr>
<td>Ann has a cron that sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.cse.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs. Sometimes companies set up mirrors, which is not OK.</td>
<td>receive cron output for tableDescriptions build errors, AutoPush completed</td>
<td>monthly</td>
<td>rare</td>
<td>Pauline</td>
<td>Cath</td>
<td>Ann</td>
<td>QA Team</td>
<td>QA Team</td>
</tr>
 
<tr>
<td colspan="7" style="background-color:gray"><b>from galt's cron</b></td>
</tr>
 
<tr>
<td>run ReleaseLog script (this requires a copy of the kent repository)</td>
<td>daily</td>
<td>Galt</td>
<td>Galt</td>
<td>Galt</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>non-cron jobs</b></td>
<td colspan="7" style="background-color:gray"><b>non-cron jobs</b></td>
</tr>
</tr>


<tr>
<tr>
<td>Check old forum (http://redmine.soe.ucsc.edu/forum/index.php) for MLQs that may have slipped through the cracks: Check to make sure that the last response for every ticket came from one of us.  If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to.  If not, notify the appropriate MLM with as much information about the MLQ as possible.</td>
<td>Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us.  If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to.  If not, notify the appropriate MLM with as much information about the MLQ as possible.</td>
<td>weekly</td>
<td>weekly</td>
<td>Luvina</td>
<td>Cath</td>
<td>Steve</td>
<td>Lou</td>
<td>Daniel</td>
</tr>
</tr>


<tr>
<tr>
<td>update sla.html whenever we have RR downtime</td>
<td>Monitor & report UCSC Genome Browser outages (SLA). [http://genomewiki.ucsc.edu/genecats/index.php/Monitoring_Tasks_Notes#SLA_Monitoring_.26_Reporting See wiki procedures]</td>
<td>hopefully, not much!</td>
<td>hopefully, not much!</td>
<td>Steve, Jonathan</td>
<td>Cath</td>
<td>Brooke, Pauline</td>
<td>Conner</td>
<td>Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td>update the two crontabs in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates)</td>
<td>update crontab in genecats/qa/crontabs/*.crontab (and [http://genomewiki.cse.ucsc.edu/genecats/index.php/Updating_QA_crontabs log in as qateam and restart the qateam crontab] with updates)</td>
<td>as needed</td>
<td>as needed</td>
<td>Pauline, Jonathan</td>
<td>Brian</td>
<td>Brooke, Pauline</td>
<td>Brian</td>
<td>Brian</td>
</tr>
</tr>


<tr>
<tr>
<td colspan="4" style="background-color:gray"><b>ENCODE</b></td>
<td>update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address</td>
<td>rarely</td>
<td>ChrisL,Matt</td>
<td>Daniel</td>
<td>Gerardo</td>
</tr>
</tr>


<tr>
<tr>
<td>Check genome-preview with Selenium test, ensures various big ENCODE tracks are accessible, and that machine is still running</td>
<td>Check genome-preview machine is still running well</td>
<td>weekly</td>
<td>weekly</td>
<td>Brian</td>
<td>Jairo</td>
<td>Brian</td>
<td>Conner</td>
<td>Daniel</td>
</tr>
 
<tr>
<td>Updated Facebook with Announcements</td>   
<td>As needed</td>
<td>Team</td>
<td>Team</td>
<td>Team</td>
</tr>
 
<tr>
<td>Update Twitter with Announcements</td>     
<td>As needed</td>
<td>Team</td>
<td>Team</td>
<td>Team</td>
</tr>
 
 
<td colspan="7" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>
 
<tr>
<td>omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so</td>
<td>runs daily, complains bi-weekly</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
 
<tr>
<td>omimUploadWrapper.sh uploads the OMIM tables to OMIM</td>
<td>runs daily, complains rarely</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
 
<tr>
<td>decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
 
<tr>
<td>iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so</td>
<td>runs daily, complains monthly</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
 
<tr>
<td>gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
 
<tr>
<td>geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so</td>
<td>runs weekly, complains quarterly</td>
<td>jcasper</td>
<td>ChrisL</td>
<td>ChrisL</td>
</tr>
</tr>
<td colspan="7" style="background-color:gray"><b>otto cron jobs (/hive/data/outside/otto) </b></td>


<tr>
<tr>
<td>Receive validation checks on ENCODE cv.ra metadata, but won't change unless cv.ra changes</td>
<td>~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree</td>
<td>rare/never</td>
<td>runs weekly</td>
<td>Brian</td>
<td>jcasper</td>
<td>Brian</td>
<td>jcasper</td>
<td>jcasper</td>
</tr>
</tr>


<tr>
<tr>
<td>Update broken links from cron on hgEncodeVocab/cv.ra checkLinks</td>
<td>~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/)</td>
<td>rare, fixes needed about once a month</td>
<td>runs twice a week</td>
<td>Brian</td>
<td>jcasper</td>
<td>Brian</td>
<td>jcasper</td>
<td>jcasper</td>
</tr>
</tr>


Line 242: Line 472:


[[Category:Browser QA]]
[[Category:Browser QA]]
[[Category: Browser QA Monitoring Tasks]]

Latest revision as of 20:59, 25 March 2022

Please add to the documentation page, Monitoring Tasks Notes to add procedures and notes for specific jobs, then link back to the job in the list below.

task frequency Until Aug '18 Until Sep '19 Now
from hgwdev qateam cron
Respond to items on the broken links report: Check to make sure that broken links reported are actually broken. If so, fix the link if it is clear what the new link should be and push to the RR. If not, contact the appropriate person at the web site in question to find out what the new link should be. daily Jairo Jairo Jairo
Respond to the trackDbGlobal report: Make public and request a push of trackDb and friends for any assemblies that show up as changed in the trackDbGlobal report. weekly Jairo Conner Lou
Retired (See #23998) receive output from syncAday.csh; alert; if anything is out of sync (checks that tables are in sync between beta and RR for database of the day) daily Cath Lou Retired
Retired (See #23998) receive output from checkMetaAday.csh; alert; if anything is out of sync (compares hgcentralbeta to hgcentral for db of the day) daily Cath Lou Retired
Receive output of checkSyncAndMetaDataAll.py which is a combination of the two previous cronjobs syncAday.csh and checkMetaAday.csh. This cron refines the hgcentralbeta/hgcentral sync output as well as the hgwbeta/hgw1/hgw2 table syncs, and runs on all databases. See #23998 for more info. weekly Lou, Conner Lou
receive output from realTime.csh (previously gbLoaded) cron job and alert braney if anything seems very outdated
No output for xenTro3, fr1, fr2, fr3, eboVir3, dm2		 daily, problems every 3-6 months Jairo Jairo Daniel
get output from checkTableStatus.csh on Mondays (this is a report on whether the genbank table dumps are up-to-date...{Read how this cronjob is important for updateTimes.csh here weekly Cath Lou Gerardo
get an email from backupCentral.csh each month that keeps a dump of hgcentral tables (there's a redmine ticket on whether we should change this one: http://redmine.soe.ucsc.edu/issues/7834) monthly Brian Daniel Gerardo
receive an email on additions and subtractions to files in /gbdb on hgnfs1 daily Brian Conner Lou
receive an email from cron if qateam DailySandboxUpdate fails, note: emails defined in /cluster/home/qateam/bin/scripts/DailySandboxUpdate daily ChrisL, Brian Jairo Brian Lou Jairo, Brian, Lou
email request to check Genomes section of the Wikipedia Page and make other changes or updates if needed. runs quarterly Bob,Conner Bob,Gerardo
email request to check trackDbLibrary.shtml and trackDbHub and run hubCheck -settings -level=required runs monthly BrianL
Retired (See #27425) email request to check doPublicCrawl public.ix and public.ixx hub search files and push them. runs quarterly Jairo Daniel Gerardo,Brian
increment databaseAday, /cluster/bin/scripts/databaseAday.csh problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
checkHgFindSpec -checkTermRegex, /cluster/bin/scripts/runCheckHgFindSpec go problems only ChrisL, Brian, Cath Jairo, Brian, Lou Jairo, Brian, Lou
check to see if there are new files to go with the CGI push /cluster/home/qateam/bin/scripts/findCgiFiles.csh go problems only ChrisL, Cath, Brian Jairo, Brian, Lou Jairo, Brian, Lou
get output from checkGenomeMysql (discovers corrupt tables) daily Brian Brian, Daniel Daniel, Gerardo
[checkGbibMd5.sh] compares the md5sum of gbibBeta.zip on dev with the sum for gbib.zip on genome-store, complaining if they don't match. runs weekly Jairo, Cath Jairo,Conner,Daniel Lou,Gerardo,Daniel
look for broken symlinks in /gbdb: daily Browser Staff Browser Staff Browser Staff
look for broken symlinks in in (very large) visiGene directory twice yearly ChrisL Conner Brian
look for broken symlinks in htdocs-hgdownload/goldenpath daily Browser Staff Browser Staff Browser Staff
Check the source tree for duplicate case-independent filenames. These wreck havoc with git. daily Browser Staff Browser Staff Browser Staff
sends a reminder to check the list of folks who have mirror sites. This list is created by a script of Hiram's and located here: http://genome-test.gi.ucsc.edu/admin/tools/useCount.html. The information comes from the phoneHome program in the CGIs (see #22593 for login/password and discussion about how an initial 'registration.txt' file is created then moved to trash, where presence of that file prevents the request being repeated so only an initial phone call is made). Sometimes companies set up mirrors, which is not OK. monthly Ann [was: Matt, Brian] Ann,Lou,Hiram Ann,Jairo,Hiram
Monthly reminder to check for stale New pennantIcon tags monthly Lou Lou
run /cluster/bin/scripts/sessionStats.csh NOTE: this is separate to each machine (genome-asia & genome-euro) thus cronjobs are specific to each machine and jonathan owns asia monthly Angie, Brian, Bob Angie, Brian, Bob Angie, Lou, Bob
ask Galt/Jim about blat updates 3x per year ChrisL,Cath Daniel, Conner Daniel, Gerardo
check that the ga4gh beacon server is running. Email Max if there is an issue. daily Jairo Conner Daniel
check that blat servers are running ok link to error log checking ongoing Ann, Brian, Galt, Matt, Erich, ChrisL, Jairo, Cath Ann, Galt, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Galt, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
receive output from hgTracksRandom cron; alert admins to problems as needed many: cath, chmalee, kuhn, brianlee, braney, ann,, jcasper, matt, jairo Bob,Ann, Galt, Braney, Jonathan, Christopher, Erich, Matt, Brian, Conner, Daniel, Lou, Jairo Clay, Bob, Braney, Jonathan, Erich, Matt, Brian, Gerardo, Daniel, Lou, Jairo
check that beta builds alright 3x week Brian, ChrisL Brian, Braney, Jairo, Lou Brian, Braney, Jairo, Lou
Retired (See #27779) hubPublicCheck complains when the hubPublic table doesn't have the same strings as the actual hub.txt file. (NOTE: To run only on Beta hubs requires these declarations: export HGDB_CONF=$HOME/.hg.conf.beta and SHELL=/bin/sh). runs daily, complains moderately ChrisL Daniel, Braney, Christopher Daniel, Braney, Christopher
Curl of http://genomewiki.ucsc.edu/index.php/Assembly_Hubs to test Public Wiki is accessible, output only if no response. hourly Brian Conner Daniel
qaAutoTrack -script that does some auto-QA on the autopushed tracks. daily Jario Jairo Jairo
checkAllStaticLinks.csh script that checks static pages. weekly Daniel,Conner,Brian Daniel,Gerardo,Brian
assemblyStatsCron.py pulls and reports GB usage stats over the last month from the RR+mirrors. monthly QA Team QA Team
HTTPs expiration check runs genecats/qa/testTools/checkHTTPSdir/checkHTTPS to report any expiring certificates weekly on Tuesday QA Team, cluster admin
Hub Public Coordinator maintains error-free Public Hubs hubs and keeps the hubs updated. Fully automated to update labels/titles, the search index, and contact hub provider when hub is down. Monitor the three cronjobs: hubPublicAutoUpdate, hubSearchUpdate, and hubPublicMail. More details can be found here. daily Daniel, Gerardo
from build cron
receive cron output for tableDescriptions build errors, AutoPush completed rare Cath QA Team QA Team
from galt's cron
run ReleaseLog script (this requires a copy of the kent repository) daily Galt Galt Galt
non-cron jobs
Check for MLQs that may have slipped through the cracks with Google Groups to check for unanswered Q's. Check to make sure that the last response for every ticket came from one of us. If not, look it up in redmine to ensure that A) there is indeed a redmine ticket for it, and B) it has been responded to. If not, notify the appropriate MLM with as much information about the MLQ as possible. weekly Cath Lou Daniel
Monitor & report UCSC Genome Browser outages (SLA). See wiki procedures hopefully, not much! Cath Conner Gerardo
update crontab in genecats/qa/crontabs/*.crontab (and log in as qateam and restart the qateam crontab with updates) as needed Brian Brian Brian
update UCSC Entrez LinkOut when necessary. Entrez LinkOut sends requests for changes (and statistics) to the browser-qa email address rarely ChrisL,Matt Daniel Gerardo
Check genome-preview machine is still running well weekly Jairo Conner Daniel
Updated Facebook with Announcements As needed Team Team Team
Update Twitter with Announcements As needed Team Team Team
otto cron jobs (/hive/data/outside/otto)
omimWrapper.sh checks for new OMIM data and downloads and loads it into hgwdev if so runs daily, complains bi-weekly jcasper ChrisL ChrisL
omimUploadWrapper.sh uploads the OMIM tables to OMIM runs daily, complains rarely jcasper ChrisL ChrisL
decipherWrapper.sh checks for new DECIPHER data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
iscaWrapper.sh checks for new ISCA data and downloads and loads it into hgwdev if so runs daily, complains monthly jcasper ChrisL ChrisL
gwasWrapper.sh checks for new GWAS data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
geneReviewsWrapper.sh checks for new GeneReviews data and downloads and loads it into hgwdev if so runs weekly, complains quarterly jcasper ChrisL ChrisL
otto cron jobs (/hive/data/outside/otto)
~jcasper/bin/scripts/updateHgConf.sh - checks for changes to hg.conf files and checks them in to the kent tree runs weekly jcasper jcasper jcasper
~jcasper/bin/backup_hgconfprivate.sh - checks most recent versions of hg.conf.private files into a private repository (~jcasper/hgconf_private/) runs twice a week jcasper jcasper jcasper
Retrieved from "https://genomewiki.ucsc.edu/genecats/index.php?title=Monitoring_Tasks&oldid=5860"