Emergency Backup BLAT Servers

Overview

In case certain blat servers go down, there is a a backup blat server ("blatx") that has ports pre-configured to host the "top 5" blatted assemblies (hg38, hg19, hg18, mm10, mm9). If the current blat server(s) hosting any of these 5 assemblies goes down, manually switch to the specified fail-over server in the list below by updating the hgcentral.blatServers table. A common solution is for the admins to restart the blat server that has gone down.

List of current backup "blatx" server ports

hg38

host: blatx

trans port: 17786 isTrans=1, canPcr=0

untrans port: 17787 isTrans=0, canPcr=1

hg19

host: blatx

trans port: 17778 isTrans=1, canPcr=0

untrans port: 17779 isTrans=0, canPcr=1

hg18

host: blatx

trans port: 17782 isTrans=1, canPcr=0

untrans port: 17783 isTrans=0, canPcr=1

mm10

host: blatx

trans port: 17780 isTrans=1, canPcr=0

untrans port: 17781 isTrans=0, canPcr=1

mm9

host: blatx

trans port: 17784 isTrans=1, canPcr=0

untrans port: 17785 isTrans=0, canPcr=1

dm6

host: blatx

trans port: 17860 isTrans=1, canPcr=0

untrans port: 17861 isTrans=0, canPcr=1'

dm3

host: blatx

trans port: 17866 isTrans=1, canPcr=0

untrans port: 17867 isTrans=0, canPcr=1

rn6

host: blatx

trans port: 17854 isTrans=1, canPcr=0

untrans port: 17855 isTrans=0, canPcr=1'

danRer11

host: blatx

trans port: 17846 isTrans=1, canPcr=0

untrans port: 17847 isTrans=0, canPcr=1

danRer10

host: blatx

trans port: 17858 isTrans=1, canPcr=0

untrans port: 17859 isTrans=0, canPcr=1

danRer7

host: blatx

trans port: 17864 isTrans=1, canPcr=0

untrans port: 17865 isTrans=0, canPcr=1

"mm10 blat is down!" Example for QA to follow when moving a down-blat-server assembly to emergency blatx

How is mm10 blat currently configured?

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+--------+-------+---------+--------+
| db   | host   | port  | isTrans | canPcr |
+------+--------+-------+---------+--------+
| mm10 | blat1d | 17779 |       0 |      1 |
| mm10 | blat1d | 17778 |       1 |      0 |
+------+--------+-------+---------+--------+

Update the blatServers table to point mm10 to the backup "blatx" server:

hgsql -h genome-centdb -e "update blatServers set host='blatx' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17780' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17781' where db='mm10' and isTrans='0'" hgcentral

Check the change:

hgsql -h genome-centdb -e "select * from blatServers where db='mm10'" hgcentral
+------+-------+-------+---------+--------+
| db   | host  | port  | isTrans | canPcr |
+------+-------+-------+---------+--------+
| mm10 | blatx | 17781 |       0 |      1 |
| mm10 | blatx | 17780 |       1 |      0 |
+------+-------+-------+---------+--------+

Mm10 blat should now be fixed (is on the emergency server), go to the RR and do a blat test to be sure.

When the failed blat server has been fixed, roll back to it. In this example, move mm10 from blatx back to blat1d:

hgsql -h genome-centdb -e "update blatServers set host='blat1d' where db='mm10'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17778' where db='mm10' and isTrans='1'" hgcentral
hgsql -h genome-centdb -e "update blatServers set port='17779' where db='mm10' and isTrans='0'" hgcentral

mm10 blat should now be fixed on the original blat server, be sure to go to the RR and do another blat test.

Test with gfServer

A quick command-line test example:

gfClient blatx.soe.ucsc.edu 17845 /gbdb/strPur4/  test.fa dnaTestOut.psl

About gfServer logs

As of 1/25/18: Haifang says: I just upgraded gfServer on blatx and added two new options, -debugLog -seqLog. See more about revieing gfServer.log here.

>ssh qateam@blatx.soe.ucsc.edu

Chris Lee says:

debugLog and seqLog are options on gfServer, not gfClient. You can see the additional lines in the gfServer logs on the blatx machine:

[qateam@blatx.soe.ucsc.edu /home/qateam] tail -15 /scratch/strPur4/gfServer.log 
>query
gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaatatg
taaaaaccgactgaagacagcttagttagaaacgtgtgtgggatcaacta
tggcgttgccgttgactagatctagatccctacttttttttgttctcaat
acggtatgtagcttggtatccgcctctgacaatgatcccagccctagctt
ctgatcatgatctggcacaaaatcaatgtaggtctagatctagatctatc
cgacttcgtaaacaacatagcctatttctttatcgttgaactatgagttt
atgactaaaacattcttcttcttcttcttctt
2018/01/25 09:58:52: debug: 1711519 101 clumps, 36593 hits
2018/01/25 09:59:18: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 09:59:18: warn: Zero sized query
2018/01/25 10:00:12: info: gfServer version 36x2 on host blat4d, port 17845 connection from 132.249.245.79
2018/01/25 10:00:12: debug: 0ddf270562684f29pcr cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 4000
2018/01/25 10:00:12: debug: 1791616 PCR cctacattgattttgtgccagatcatgatcagaagctagggctgggatca gcccagctaaaaccaatcaccttcaagggataaaggttcatccaaat 63 clumps

Compared to a gfServer running without those options, like on blat1a:
[qateam@blat1a.soe.ucsc.edu /scratch/bosTau7] tail gfServer.log 
2018/01/25 07:24:13: info: gfServer version 34x12 on host blat1a, port 17845 connection from 128.114.119.132
2018/01/25 08:00:02: info: gfServer version 34x12 on host blat1a, port 17845 connection from 132.249.245.79
.... # more of the same

(Note: 128.114.198.32 is the new hgwdev, in the above example 132.249.245.79 was the old hgwdev address)

List of current original "blat1*" server ports

| db       | host   | port  | isTrans | canPcr |
------------------------------------------------
| hg38     | blat1d | 17902 |       1 |      0 |
| hg38     | blat1d | 17903 |       0 |      1 |
------------------------------------------------
| hg19     | blat1a | 17778 |       1 |      0 |
| hg19     | blat1a | 17779 |       0 |      1 |
------------------------------------------------
| hg18     | blat1b | 17778 |       1 |      0 |
| hg18     | blat1b | 17779 |       0 |      1 |
------------------------------------------------
| mm10     | blat1d | 17778 |       1 |      0 |
| mm10     | blat1d | 17779 |       0 |      1 |
------------------------------------------------
| mm9      | blat1c | 17778 |       1 |      0 |
| mm9      | blat1c | 17779 |       0 |      1 |
------------------------------------------------
| dm6      | blat1c | 17858 |       1 |      0 |
| dm6      | blat1c | 17859 |       0 |      1 |
------------------------------------------------
| dm3      | blat1d | 17790 |       1 |      0 |
| dm3      | blat1d | 17791 |       0 |      1 |
------------------------------------------------
| rn6      | blat1b | 17784 |       1 |      0 |
| rn6      | blat1b | 17785 |       0 |      1 |
------------------------------------------------
| danRer11 | blat1a | 17874 |       1 |      0 |
| danRer11 | blat1a | 17875 |       0 |      1 |
------------------------------------------------
| danRer10 | blat1c | 17862 |       1 |      0 |
| danRer10 | blat1c | 17863 |       0 |      1 |
------------------------------------------------
| danRer7  | blat1d | 17788 |       1 |      0 |
| danRer7  | blat1d | 17789 |       0 |      1 |
------------------------------------------------