Profile cover photo
Profile photo
James Cuff
934 followers -
molecular biophysicist | performance hunter | nerd
molecular biophysicist | performance hunter | nerd

934 followers
About
James's posts

Post has attachment
Aiming for 4,096 miles in 2016!
It's been a while since I've written anything here at all. Anyway, I wanted to share why I've not put fingers to keyboard in a long while. It's mostly because of this crazy running thing... it sort of escalated... 2014 (5 million steps, 2,400 miles): 201...

Post has attachment
12,290,000 V.92 dial-up modems...
It's that time of year for a new fast file system. Here's a single box teaser with a pair of disks that we are building... echo 1 > /proc/sys/vm/swappiness

mdadm -C /dev/md0 --level=raid0 --raid-devices=2 /dev/disk0 /dev/disk1

mke2fs -E nodiscard /dev/md0...

Post has attachment
IT organizational health...
Scene One: A doctor's office on main campus Enter stage left: IT leader (ITL) Doctor: "Hello, ITL please do take a seat..." ITL: "Hello Doctor, I can't thank you enough for taking time to see me for my annual organizational health and performance review!" D...

Post has attachment
How does your cluster sound?
So I was on the airplane coming back from XSEDE15 in St. Louis, and got to thinking about all the amazing visualizations that were on display. I wondered. What would a cluster sound like? On our HPC cluster we have millions of jobs running each month, an...

Post has attachment
James Cuff commented on a post on Blogger.
If you have this issue... where the first connect fails because the ssh keys are not valid yet...

[root@sa01 noss-obm-names]# ssh root@chassis2a152-cmc-obm
The authenticity of host 'chassis2a152-cmc-obm (10.31.33.222)' can't be established.
RSA key fingerprint is SHA256:hosULTUpzukZlcno/l7nIn32UpIMfeZhCoAJ1rt5RZ0.
Are you sure you want to continue connecting (yes/no)

(this makes sshpass just fail)

you can get round it with this magic:

[root@sa01 noss-obm-names]# sshpass -p calvin ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@chassis2a152-cmc-obm getversion

And you are good to go!

Post has attachment
James Cuff commented on a post on Blogger.
For those that care about the actual algo if you want it to go super fast use a named pipe and pigz.  Native non compressed output 

real 3m23.022s
user 6m22.666s
sys 0m38.528s

with the built in compression:

real 15m15.894s
user 38m12.299s
sys 0m33.233s

With a single named pipe and gzip:

409  mkfifo outr1.dat outr1un.dat outr2.dat outr2un.dat

412  gzip -c < outr1.dat > outr1.dat.gz &
413  gzip -c < outr2.dat > outr2.dat.gz &
414  gzip -c < outr2un.dat > outr2un.dat.gz &
415  gzip -c < outr1un.dat > outr1un.dat.gz &

Then run it:

real 13m27.782s
user 7m40.987s
sys 0m25.483s

So faster than native by a couple of minutes...

But with pigz!  Which we have talked about before!

http://blog.jcuff.net/2012/05/never-underestimate-power-of-genome-pig.html

we get it down to:

real 1m55.487s
user 8m51.995s
sys 0m23.691s

which means writing the pair of 7G files is the real rate determining step once we also fixed huge pages.  so we have gone from

57m47.317s

to 

1m55.487s

so close to 30x in the end...  

Not a couple of hours work :-)

Post has attachment
Of huge pages and huge performance hits, are we alone?
We do a fair amount of sequence analysis here. One thing we do a lot of is trimming sequence data. The files are somewhat large. I'm not allowed to call this "big data" :-) There's a neat trimming code called "trimmomatic" (awesome name eh?). It's a si...

Post has attachment
Please, purchase my storage solution....
CUE : Story opens, In a small office at a research computing department: Endearing Storage Vendor: ".... so, now you have seen our technology, if you wish to purchase our one of a kind "storage solution" you will be all set... forever, we will love to partn...

Post has attachment
Of big microscopes and even bigger data...
We recently installed one of these awesome electron microscopes ... In the center I help PI, w e are imaging brains , but more about that another time. Right now this is all about getting this thing running, and running at speed, and some lovely UNIX geeke...

Post has attachment
OdyBot And Pointy Haired Alerting! AKA: Grumpy old man shakes fist at web two dot oh!
Hi all, It's been ages since I last posted. We have been super busy in the day job running our monster high performance computing infrastructure and keeping lots of petabytes spinning and many scientists and researchers happy . I wanted to quickly note th...
Wait while more posts are being loaded