Bandwidth testing

Units

For these measurements:

• 1 kB/s = 1,000 bytes per second
• 1 mB/s = 1,000,000 bytes per second
• 1 gB/s = 1,000,000,000 bytes per second

Memory:

• 1 Kb = 1024^2 bytes = 1,048,576 bytes
• 1 Gb = 1024^3 bytes = 1,073,741,824 bytes
• 1 Tb = 1024^4 bytes = 1,099,511,627,776 bytes

Machines tested

machine name	processor speed	memSize	10 files of size	dd /dev/zero to /dev/shm	cp /dev/shm to /tmp	cp /dev/shm to hive	rsync /dev/shm to /dev/shm	rsync /dev/shm via rsh	UDT via UPD
hgwdev	Opteron 1150 Mhz	32 Gb	1 gB	5278 mB/s 10X dd	480 mB/s 2X cp	58 mB/s	1638 mB/s 20X rsync	n/a	4070 mB/s 10X send
kolossus	Intel 2261 Mhz	1 Tb	1 gB	6700 mB/s 10X dd	600 mB/s 2X cp	58 mB/s	120 mB/s 4X rsync	115 mB/s	1135 mB/s
okazaki	Intel 1600 Mhz	2 Gb	75 mB	700 mB/s	540 mB/s 6X cp	11.3 mB/s	11.5 mB/s	11.6 mB/s	10.7 mB/s 4X send
mitzie	Intel 400 Mhz	512 Kb	18 mB	113 mB/s	25 mB/s	n/a	54.4 kB/s up 316 kB/s down	n/a
cloud0	Opteron 2000 Mhz	7.5 Gb	288 mB	940 mB/s 4X dd	78 mB/s	n/a	16 mB/s 2X rsync	43 mB/s 3X rsync	46 mB/s 3X send
altair	Opteron 2300 Mhz	32 Gb	1 gB	5554 mB/s 8X dd	600 mB/s 2X cp	100 mB/s 2X cp	200 to 1000 kB/s big variation	n/a	400 mB/s 3X send

Procedure

Sizing files based on available /dev/shm space. Largest file size limited to 1 gB.

The /dev/zero to /dev/shm write is done a dd command:

 dd if=/dev/zero of=/dev/shm/memToMem.$I.zero bs=200000 count=$ddCount

Running multiple copies of that simultaneously until the best speed was obtained.

The /dev/shm copy to /tmp is done the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously until the best speed was obtained. It only took several (2 to 3) multiple instances before they began to interfere with each other and slow down the effective rate.

The /dev/shm copy to the hive filesystem was done with the cp command:

cp -p /dev/shm/memToMem.$I.zero ./memToFile.$I.zero

Running multiple copies of that simultaneously did not improve the speed on the machine where the copies were running. However, running copies from multiple machines at the same time to the same hive directory did not interfere with each other. They were independent of the other activity.

The altair /dev/shm to hive filesystem copy was performed to the /genome/ filesystem.

The rsync /dev/shm to /dev/shm was to and from hgwdev. Multiple instances of rsync from kolossus to hgwdev seemed to add together for a total bandwidth higher than a single instance of rsync.

The machine mitzie is at the end of a Santa Cruz DSL line and over a WiFi network connection across the driveway to the garage. The WiFi link is faster than the DSL line.

The UDT via UPD transfers are with the udp transport software from: UDT.