J.P. Morgan has spent billions of dollars in an effort to transform its computerized systems, most recently via Athena, 35 million lines of Python code. Before this disappointing effort that came with a price tag of over a billion dollars, J.P. Morgan had followed a different path with its acquisition of Highbridge, a hedge fund. These two paths of software development are worthy of examining. Let’s call the Highbridge option the “special forces” option, and Athena the “Army option.”
The Special Forces Option: Highbridge Statistical Arbitrage
The genesis of Highbridge was a Fund of Funds run by two childhood friends, Glenn Dubin and Henry Sweica, not surprisingly called Dubin and Swieca. During the time that the friends managed their Fund of Funds, they became investors in many top hedge funds, including DE Shaw, a quantitative fund run by former Columbia University computer science professor David Shaw. Dubin and Swieca followed DE Shaw, but even more they followed their top employees as they left. Some like Jeff Bezos started technology companies, while others like Evan Dick moved to the beach in California and started an open source investment technology project.
Evan had started his career coding for NASA in college, getting picked off by DE Shaw to code and trade. He was not a normal programmer, but excellent and in his departure from DE Shaw, Dubin and Swieca saw an opportunity to not only transform their own business, but also expand into a new one. Glenn Dubin took the initiative and flew to California to see Evan Dick and made him an offer to help Dubin and Swieca transform their Fund fo Funds business and simultaneously build a hedge fund platform, a platform on which Evan and his team would then run all of the quantitative operations and funds. Evan agreed, and a contract was signed on a napkin at dinner and sealed with a handshake. As Evan told me, it was “all about trust.”
The Hedge Fund Investment Platform
You see, many of the same things that were needed by a Fund of Funds from a risk perspective could be utilized by Evan and his team in building the risk infrastructure for a multi-strategy hedge fund, so he was really multi-tasking. After risk systems, they built trading and reconciliation systems and at the end had only a handful of people running some of the most sophisticated integrated investment systems in the world. All this was done faster, cheaper, and better than it would have been if Dubin and Sweica had hired an institutional infrastructure provider like SS&C (which incidentally would have provided an antiquated, slow as molasses, but stable solution).
After all of the infrastructure was built, Dubin and Swieca continued to operate as a Fund of Funds, while Evan and his team began trading statistical arbitrate strategies for the new hedge fund Highbridge Capital Management. Oher inhouse strategies such as fundamental long-short equity, convertible arbitrage, and event-driven were consistently rolled out at Highbridge as new groups were recruited. The one constant was Evan’s Stat Arb team’s technology infrastructure. In essence, what Dubin and Swieca did with Highbridge was build infrastructure so they could take the investments that Dubin and Swieca had done via a Fund of Fund and bit by bit bring it all in-house into a multi-strategy hedge fund, cutting out the “middle men” (and making tons more money in the process).
By the time Evan and I met in New York in 2004, Highbridge was the talk of the town. I had met Glenn via a friend who worked at Dubin and Swieca and Glenn told me the architect of the systems had a name, and passed the phone to Evan Dick. Evan’s systems were so automatic, he was sailing in the Atlantic talking to me on a satellite phone (of course there were other members of his team, but the point was that the systems were almost faultless and took almost no maintenance, allowing the team to constantly improve). He talked to me about improving the systems to handle hundreds of billions of dollars in AUM and he wanted me to join. I turned him down (Evan told me in 2017 that I was the only person that ever turned him down) because I was already committed, but I never doubted him or his systems after a couple of hours of discussions. To this day he is one of the best natural systems architects I have ever met.
That infrastructure, built by a group of five people came to be considered one of the best in the world. Eventually, J.P. Morgan made an offer to purchase Highbridge, not only to buy the hedge fund, but more importantly to utilize their technology to transform their business. A little known fact was that when this happened, the group’s systems were almost immediately used to manage over $120 billion of additional assets. While this was a success, in the end the bureaucratic nature of the large bank won out and the engineering talent went for the doors, being replaced by people who could only maintain the infrastructure with decreasing effectiveness. As for Evan, he moved to Florida and prepared to go to space (which he’s done twice on his old friend Jeff Bezos’ spaceships).
The Army Option: Athena and 35 Million Lines of Python
After Evan and his team left J.P. Morgan, the next big effort to rework the systems of J.P. Morgan was a project called Athena. I have always viewed Athena as an institutional reaction to the departure of the elite Highbridge talent. In essence, if they could not have the special forces, they’d throw an army at the problem! This army then proudly proclaimed after many years that they produced 35 million lines Python code! This in of itself was suspect (and dangerous, as numerous articles online now attest).
You see, for us engineers is that we do not count our productivity by the lines of code, but by its parsimony, speed and lack of errors and on this count, Athena is a failure of monumental proportions. Over the years I had students who worked on the project and came back to get a breather and learn proper software engineering. The main point to us familiar with banking software that demonstrates that Athena is a failure is that J.P. Morgan offered Athena to their external clients.
This might seem strange, but all experienced systems architects in finance know that the banks always offer their cruddy systems to clients only after it does not make money or provide a strategic advantage internally. The other famous example like this was Goldman Sachs Slang/SecDB systems–off limits till the strategic advantage was gone! In other words, only after it has proven to not work well will it be rolled out to the clients. Now, while we could discuss failed systems for a long time, I mention this whole situation because it illustrates a powerful point. That point — more is most often less as far as productivity in software engineering, a concept articulated by Fred Brooks in his classic book, The Mythical Man-Month.
The More is Less of Software Engineering
The Mythical Man-Month is an indispensable tome in software engineering and lays out that counting lines of code, hours, or man-months in software projects is absurd because what matters is productivity and not all engineers’ work is equal. Some are average (call them 1x engineers), fewer are good (10x engineers) and an elite few are great (100x or greater). In this sense, engineers are more like elite athletes, and should be managed as such. The danger is that if we treat all engineers as equal, 100x engineers will be pulled into helping to manage 1x engineers, which will destroy their productivity. That’s like getting Joe Montana (at his prime) and then relieving him in the middle of a game with a good high school QB–not just a bad idea, but a way to lose terribly.
An example suffices to illustrate. Say we have one 100x engineer on our team who spends 50% of her time helping 10 average engineers via meetings, code review and other activities. In that one action, you just lost almost half your group’s productivity, but its even worse than that, because large groups often hire till all productivity ceases. It’s actually much worse than that even because average coders introduce so many bugs and kill elite systems and make elite engineers leave companies. The lack of understanding this simple concept is precisely why the financial industry is littered with projects that cost hundreds of millions of dollars (or much more) and produce no discernable improvement–financial firms most often hire average to good engineers for some very difficult problems.
This very problem of how software engineering does not work well in large groups is why small teams like Evan and his special forces engineers will always beat armies of hundreds of mediocre programmers from places like J.P. Morgan.