Cell Browser data flow and architecture: Difference between revisions
| (3 intermediate revisions by the same user not shown) | |||
| Line 18: | Line 18: | ||
</pre> | </pre> | ||
2. The output files from <code>cbBuild</code> are placed inside <code>/usr/local/apache/htdocs-cells</code>. Note that the original configuration files and expression matrices inside the dataset directory are converted into either JSON or binary files (BIN). These files are used by the Cell Browser to . | 2. The output files from <code>cbBuild</code> are placed inside <code>/usr/local/apache/htdocs-cells</code>. Note that the original configuration files and expression matrices inside the dataset directory are converted into either JSON or binary files (BIN). These files are used by the Cell Browser website to display the visualization. The original files are human readable; whereas, the ones used by the browser are for faster access. | ||
3. Once the dataset is on [https://cells-test.gi.ucsc.edu/ cells-test], the next destination is [https://cells-beta.gi.ucsc.edu/ cells-beta]. You will push the directory and files from <code>htdocs-cells</code> onto <code>/usr/local/apache/htdocs-cells-beta</code> using the command: | 3. Once the dataset is on [https://cells-test.gi.ucsc.edu/ cells-test], the next destination is [https://cells-beta.gi.ucsc.edu/ cells-beta]. You will push the directory and files from <code>htdocs-cells</code> onto <code>/usr/local/apache/htdocs-cells-beta</code> using the command: | ||
<pre> | <pre> | ||
cbPush dir-name-1 dir-name-2 | cbPush dir-name | ||
cbPush "dir-name-1 dir-name-2 dir-name-3" * Use this if you would like to push multiple datasets at once to cells-beta | |||
</pre> | </pre> | ||
Note that cbPush requires you to input a directory name | Note that cbPush requires you to input a directory name. | ||
A good alias to have in your .bashrc that pushes the current directory you are in onto beta: | A good alias to have in your .bashrc that pushes the current directory you are in onto beta: | ||
| Line 33: | Line 34: | ||
alias cbPushDir='cbPush "${PWD##*/}"' | alias cbPushDir='cbPush "${PWD##*/}"' | ||
</pre> | </pre> | ||
You could name this alias whatever you prefer. | |||
4. Once your dataset is on beta, you are almost there! Once the dataset is checked over for potential bugs, you will use the command: | 4. Once your dataset is on beta, you are almost there! Once the dataset is checked over for potential bugs, you will use the command: | ||
| Line 41: | Line 44: | ||
You will be prompted to type in a password, use your <code>hgwdev</code> password. Once you do that, the datasets will be built onto the hgw0, hgw1, and hgw2 machines! Voila! | You will be prompted to type in a password, use your <code>hgwdev</code> password. Once you do that, the datasets will be built onto the hgw0, hgw1, and hgw2 machines! Voila! | ||
Important to note that <code>sudo cellsPush</code> pushes out '''ALL''' of the changes that are on beta, so make sure everything is ready to be pushed out. You can use <code>datasetDiffs -r</code> to double check if there are any additional changes that might get pushed out along with your new dataset. | |||
== '''System Architecture Map''' == | == '''System Architecture Map''' == | ||
Latest revision as of 22:58, 10 August 2023
How does data flow between the different machines?
How does building a cell browser work?
What files are copied over? Which ones are transformed into another format?
1. Data is first deposited in a dataset directory inside /hive/data/inside/cells on hgwdev and then gets built onto cells-test using the command:
# For datasets with no additional subsets cbBuild -o alpha # For dataset collections you will use the recursive option "-r" cbBuild -r -o alpha
2. The output files from cbBuild are placed inside /usr/local/apache/htdocs-cells. Note that the original configuration files and expression matrices inside the dataset directory are converted into either JSON or binary files (BIN). These files are used by the Cell Browser website to display the visualization. The original files are human readable; whereas, the ones used by the browser are for faster access.
3. Once the dataset is on cells-test, the next destination is cells-beta. You will push the directory and files from htdocs-cells onto /usr/local/apache/htdocs-cells-beta using the command:
cbPush dir-name cbPush "dir-name-1 dir-name-2 dir-name-3" * Use this if you would like to push multiple datasets at once to cells-beta
Note that cbPush requires you to input a directory name.
A good alias to have in your .bashrc that pushes the current directory you are in onto beta:
alias cbPushDir='cbPush "${PWD##*/}"'
You could name this alias whatever you prefer.
4. Once your dataset is on beta, you are almost there! Once the dataset is checked over for potential bugs, you will use the command:
sudo cellsPush
You will be prompted to type in a password, use your hgwdev password. Once you do that, the datasets will be built onto the hgw0, hgw1, and hgw2 machines! Voila!
Important to note that sudo cellsPush pushes out ALL of the changes that are on beta, so make sure everything is ready to be pushed out. You can use datasetDiffs -r to double check if there are any additional changes that might get pushed out along with your new dataset.
System Architecture Map
