Defense Against the DARQ Arts, Optimization: Part 3- My Optimization Odyssey, Columbia and MINOS

Despite the fact that I turned down Columbia three times and Umberto Eco jumped ship, I felt safe in New York City (thanks Angus Young) and enjoyed myself almost immediately. As far as what I learned, I can pretty much say, as in all software projects, I only learned by doing. In other words, even in Ph.D. programs what you get out of it is largerly determined by what you bring into it.

A Cross-Discipline, Custom-built Ph.D. Program

I arrived at Columbia in the summer before school started and began coding on a project for the NYSE within days. I was the only person accepted into my program that year as they had guaranteed me that it would be “cross-discipline” and customized for me personally that year. You see, most business school Ph.D. programs are like MBA programs, pretty well structured. While most students go through a rigorous application process and equally rigorous course schedule where only 1-2% are accepted, occassionally students are recruited to the programs so they can help push the bounds of their disciplines and those students selectively choose the classes they need. I was like a kid in a candy store, where I was getting paid more than other masters students graduating from BYU and I had the added benefit that Columbia University was paying my tuition for whatever courses I took.

That’s exactly what also what happened with Nobel Prize winner Milton Friedman, who is also a Columbia Ph.D. Friedman had started working in math and economics early, thinking he would be an actuary and he was recruited to work with Columbia professor Harold Hotelling (who later went to UNC where he worked with Dennis Tolley, Hal’s dad). Because of how Friedman entered the program at Columbia, he could do pretty much whatever he wanted as long as he got his work done. His dissertation? He just presented some of his work, pretty much the same thing I did after I pulled out what was already making money at BGI (my first employer). I already had one model that was being published and copied all over the investment industy and I didn’t want to give anyone any more hints (the key is you “publish” your dissertation for the world to see).

One of my heroes, Milton Friedman, the son of poor Hungarian immigrants, and one of the greatest minds in economics was one of my deciding factors in accepting Columbia’s offer. I had religiously covered all his work in Oregon as a kid. Columbia told me that they would give me the same “Free(dom) to Choose” that Friedman had when he worked with Harold Hotelling in getting his Ph.D. They stayed true to their promise.

Climate Models and the Columbia Earth Institute

Those cross-disciplinary Ph.D’s are the rare Ph.D.’s I like to work with, people like Dennis Tolley, Barr Rosenberg, my colleague Mark Parker and the head of games program at the University of Utah, Mike Young. While I was hoping for more accounting, semiotics and computer science, I quickly lost the semiotics (thanks Umberto!) and I substituted in quantitative finance, taking even more finance courses than the finance Ph.D.’s. Then I got even a bit more “esoteric” like taking a one-on-one class on advanced finance and modeling with the head of the Columbia Earth Institute, Geoff Heal (another cross-discipline Ph.D.) where we just did projects together.

It was in fact when working with Geoff Heal that I began to understand climate models and methodologies, something that Geoff was the top economics expert in the world on at the time. In short, the flexibilty I came into Columbia with coupled with my programming skills opened tons of doors for me that were simply not available to the other Ph.D.s. and I learned to think creatively about models rather than just trying to memorize what was presented to me and blindly accept the assumptions. I have to say, that I have Geoff to thank more for that than any of the other economics or finance professors at Columbia. If you are ever interested in learning just how much people’s assumptions affect our best economics models I would suggest reading Mary Morgan’s The World in the Model: How Economists Work and Think. I guarantee you’ll never listen to an economist give their opinion in the same way again after you read it.

At Columbia Geoff Heal recruited me to a special one-on-one course of directed research with him where we covered supercatastrophe insurance models and related weather models. My family’s environmental activism made it fascinating and a lifelong interest which is borne out in the ULISSES Project.

Columbia sorted itself out very quickly, with me naturally spending more time with the interesting people who vied for my time (like Geoff) or naturally just spending more time with people who could code, like my Department Chair (Indian Institute of Technology degree in Computer Science, and cross discipline Ph.D. from Michigan) and a Ph.D. in Computer Science who had been brought into Columbia Business School from the Fu School of Engineering and Applied Science to “infuse it with math and computer science” thanks to an Alfred P. Sloan Foundation grant. As it turned out, this last guy was the very guy who built that neat little analysis and optimization plug-in I used my first two years at BYU. It was because of an almost random comment that he made that helped my optimization work move forward.

Stanford and MINOS

For all my time at BYU, I had tried to develop my own optimization software and I thought I would now learn how to do it better at Columbia. As well intentioned as the OR people are in business schools, they typically are not accomplished programmers and keep their research and classes to “toy problems” and that simply wasn’t going to work for me. I realized that almost immedaitely when I sat in a business school OR class from their top experts, an OR Ph.D. from Carnegie Mellon, to see what he knew about optimization. Very quickly I realized that I probably knew more than the instructor because I’d been coding from scratch on Mathematica and SAS IML. I walked out of the class and went to vent to the CS Ph.D. from Fu (who was incidentially one of my dissertation committee members). I asked if he knew anyone better in the Computer Science at Columbia. He said, “yes, but they’ll just be using Stanford’s SOL programs.” This comment was all I needed.

I had of course known about Dantzig for years, but I had never thought I could get a copy of his software, as it was “legend” to me. Within minutes I was looking up Stanford’s SOL and MINOS, which was George Dantzig’s group and the associated optimization software. Luckily for me, the exploding Internet facilitated “legend” being distilled to fact in one afternoon and then I just needed to put it into action. What I learned from Stanford’s engineering is that the applications I envisioned would require powerful machines, in fact more powerful ones than at the business school. Luckily, within days my BYU NeXT Computer Lab work started to pay unanticipated dividends that allowed me to move forward.

Dividends from the NeXT Computer Lab

One day as I was musing on how I could build my own servers, I got a call from one of Steve Jobs’ investors in NeXT, who had invested following Ross Perot‘s famous investment. He was a Texan who had followed Perot into a lot of things, and Perot was now cashing out, and this man called me because we knew people in common and he was told “James, knows a lot about NeXT Computers.” After pleasantries, he got right to the point and asked me what I thought he should do…”follow Ross and cash out or keep my chips on the table?” His investment in NeXT would transfer into Apple Stock with the Apple acuistion and he just didn’t know what to do.

He had grown rich as a Texas wildcatter and in advertising and was nervous about all this “technology stuff.” I calmed him down and first described NeXTSTEP, based on Unix Research developed at UC Berkeley, and the changes it could bring to Apple with Jobs also coming back. He said, “I don’t know what any of that technology stuff means, but what I need to know is should I stay at the table or cash in all my chips.” His second gambling comment helped me change tack and we started to talk about gambling and technology stocks.

Tim Cook, a supply-chain guy, had something to succeed Jobs in running becuase the NeXTSTEP with its BSD-Research Unix foundation allowed Apple to develop OS X (as in UniX or LinuX), a software foundation that allowed for Apple to respond to new tech challenges. It was a dividend that not only Cook harvests to this day, but I also harvested myself that allowed me to push the limits of MINOS.

As it turned out he was a big time gambler in Las Vegas (they call guys like this “whales” in Vegas), so I explained how he bet with his gut was formally called Kelly-betting, and then I explained it. I said the best tech investors in history like Claude Shannon and Philip Fisher Kelly-bet because betting on the right company at the right time could make more money than all their other bets of their lives combined. I said Shannon did this on HP, and Fisher did this on Texas Instruments. I think it was the Texas Instruments example that really drove home my point and he said he’d follow “my advice.” I said, “it’s not advice, it’s just a conversation between friends” (I wasn’t a registered investment advisor so I didn’t want to run afoul of any rules.) He said “just like at a blackjack table or oil wildcatting,” if it worked out he’d figure out a way to “pay me back.”

The Payback from a Texas Wildcatter

This short phone call I took as a courtesy turned into a multi-year contract where I became his “business advisor” and that lasted my whole grad school years with the added benefit of being so lucrative that I made enough to fund a build-out of my own server room in the second bedroom of my New York co-op (oh, I also made enough consulting to fund my own place near Columbia rather than live in dismal student housing). I always told him that I wasn’t an investment advisor, to which he said, and “I’m just your good friend that wants to make sure you’ve got what you need to keep informed, so you can help keep me informed.” He had great connections in Texas and sent me computers and parts from Compaq so I could assemble my own servers. He also paid for every information source I told him I could conceivably use. That coupled with my consulting for groups like I/B/E/S meant I had way better information resources than Columbia University.

This one consulting contract allowed me run more sophisticated systems than I worked on at Columbia during my “day job,” but even more imporant than that I had the freedom to build models how I thought they needed to be built, not how I was told to by the professors in my day job. It was at this time that I started to use MINOS on my own models, using the tool more creatively and flexibly than professsors at Columbia who were only using opitmization on simple problems like mean-variance optimization. All of this evolved in a big way the year before I graduated, when I got a call from the the largest investor in the world, Barclays Global Investors (BGI), but that will have to wait for next week.