HPC on Wall Street
By Josh Simons on Sep 23, 2008
I attended the High Performance on Wall Street Conference yesterday in New York City and found the experience odd in several respects.
First, there were my personal logistics. Due to a personal commitment this past weekend in Upstate New York, I had to kidnap my wife and drive into NYC rather than either fly or take the train from Boston as I normally would. As it turned out, the conference was held near the UN, the UN was in general session, and Ahmadinejad was speaking. The protests and the blockades and the police details at every intersection in the area caused incredible traffic jams for hours during both our arrival and departure.
Second, the conference attendees. I had wondered how the recent and continuing turmoil on Wall Street might affect attendance. There were several last minute speaker changes-- so-and-so from Lehman Brothers unable to attend, so-and-so no longer with Merrill Lynch, etc., but the main conference room was reasonably full so I assumed there wasn't much impact. Until someone took a poll during one of the sessions. By a rough hand count, he estimated that 75% of the people in the room were vendors of some sort, while only 25% were associated with customers. I think he was very generous in that estimation. I would have guessed only 10-15% were customers based on the hands I saw. Very disappointing for an attendee like myself who was primarily interested in understanding the customer perspectives of HPC in this veritical market.
The third odd aspect of the conference was the content and there were two specific weirdnesses. First, there was little or no sharing of real information by customers, which I suppose is not surprising since much of what is done with HPC on Wall Street is used to establish a strategic advantage against competing companies. More surprising, however, was the strong sense I got that the problems discussed (primary by vendor representatives) mostly sounded familiar to me as a long-time HPC person. The domain was different, but the problems were the same. And yet the problems were presented as industry specific and novel with seemingly no acknowledgement of the earlier work done in these areas within other more traditional HPC segments. Here are two examples.
- Market Data Distribution. As near as I can tell (and I should state now the caveat that much of my HPC background is in developer tools, not customer application architectures) the challenge with MDD is to distribute and deliver a real-time stream of market data with low latency and jitter and with some amount of in-band data filtering to some number of consumer endoints, which might be human traders at workstations or electronic trading programs. To me, that problem sounds a lot like what the intelligence community has been dealing with for years, though I would imagine the intel community deals with much larger data volumes, whether from satellite feeds or from other sources. I'd be very surprised if there is not a large body of knowledge held at numerous government contractor organizations on how to architect and deploy systems capable of dealing with these kinds of infeed and distribution issues.
- Low overhead messaging. One speaker at the conference, who appeared on several panels, seemed to be evangelizing the idea of getting software out of the communication pathway to increase messaging performance. It's a good idea. In HPC circles, we've called that "OS bypass" or something similar for over a decade. It's a well-known and widely used concept in current HPC systems.
I'm very excited to see HPC techniques being adopted and exploited by a new and important vertical like Financial Services. There is much in HPC that can be of value to customers in markets like these. But a knowledge of the work done earlier in the field is essential for the rapid and effective adoption of HPC as a differentiating advantage. Time spent analyzing old problems and rediscovering proven solutions is time and money lost. Perhaps it is time to apply the considerable experience and expertise of some of those crusty old HPCers out there to new application areas.