Game changer for HPC: IBM and NVidia novel architecture

The big annoucnement at SC'13, the International Supercomputing conference sponsored by IEEE and ACM that is in its 25th year and that this year is in Denver, came from an IBM VP, speaking at the Nvidia booth. I believe the VP was Dave Turek.

The annoucement did make the press, even the Wall Street Journal, but the press is not reporting the magnitude of the annoucement, possibly because they were working off press releases rather than the technical details relayed by the IBM VP late in the evening during tonight's opening night gala portion of the conference.

It's more than just marrying Nvidia GPUs to IBM's forthcoming POWER8 processor. POWER7 is what powers Watson, and although the POWER series and the Xeon series leapfrog each other in raw chip benchmarks, IBM engineers its HPCs holistically, with its own POWER CPUs, its own processor boards, and most importantly, its own architecture to maximize throughput. Real-world applications run twice as fast on POWER systems than they do on Xeon systems.

No, what the surprising, and welcome, proclamation from the IBM VP is was "the end of the server in HPC" -- I haven't seen that quote yet in any press covering the general announcement. Anyone who has seen a modern-day "supercomputer" walks away disappointed -- the racks of commodity-like PCs simply strung together with Infiniband. This has led to the modern HPC mantra lament of "We've got compute out the wazoo -- it's I/O we need more of."

That's one reason why Hadoop was a stunning upset to the HPC community. HPC usually completely separates compute from storage, with storage relegated to a system like Lustre. Oh, the file systems are so-called "parallel file systems," but all that really means is that the pipe is made fatter by having multiple parallel pipes each of standard size. At a 30,000 foot PowerPoint view, it's still just two bubbles, one for compute and one for storage, connected by one line.

Hadoop introduced the novel idea of bringing the compute to the storage. (BTW, the other reason for Hadoop's popularity is that the Map/Reduce API and its even higher abstractions of Pig and Hive are orders of magnitude easier to program for than the marriage of MPI/OpenMP that has become the standard in the HPC world. But the easier Map/Reduce API actually comes with a performance penalty, compared to hand-tuned MPI/OpenMP software.)

When the IBM VP said "end of the server," he went on to explain that IBM intends to incorporate GPUs throughout the entire architecture and "workflow" as he put it, including directly into storage devices. He wouldn't elaborate on exactly where else in their architectures that IBM would incorporate GPUs, but he said something to the effect of, "if you study our past architectures and see the direction we were going in and project out into the future, that probably won't be too far off."

This is quite a change from 2009, when Dave Turek said:

One of the big names in HPC, IBM, is taking a more reserved attitude towards GPUs. Dave Turek, VP of Deep Computing, notes that the company currently has no GPUs in its portfolio. 'The use of GPUs is very embryonic and we are proceeding at an appropriate pace.' He believes the industry has entered a period of evaluation that will last between 18 to 24 months and there will be a gradual dissemination into more conventional segments.

Putting GPU in the storage is taking the Hadoop idea of bringing compute to storage to the next level.

It's a whole new paradigm in HPC. The chapter of the past two decades of "lots of servers + Infiniband" is about to be closed, and a new one opened by IBM and Nvidia.