The AGORA Initiative (and my big Hail Mary)

This is the final entry on the foundations of the ULISSES Project, addressing the AGORA Initiative, our teaching material that was encoded in a series of libraries.   Those libraries began as a set of mathematical programming exercises, and all of that started with one of the biggest “Hail Mary’s” of my academic life.  A little bit of explanation is in order. 

The Cautionary Tale of the FRI Failure of my Friend

Two years before I arrived at Columbia University, a friend of mine had gone to Stanford and pursued his Ph.D. at Food Research Institute (FRI), at the time the most elite mathematical programming group in the world (especially as it related to commodity options and trading).  One year after starting at Stanford, he was back at BYU to “re-tool” after what he described as an “epic failure.”  Not wanting to face the same circumstance, I asked for advice.   Without hesitation he told me that his biggest mistake was that before grad school he cared too much about getting into grad school.  What he told me was that he should have been more concerned about getting out of grad school, or in other words graduating. 

He then told me that there was one class, the capstone class that everyone at BYU who was pursuing a Ph.D. in economics or finance took, Econ 588, Advanced Econometrics, and he’d come back to BYU just for that course because he was going to try to do a Ph.D. again, but this time the right way.  We took that course together and he was right…it was the single most important course at my time at BYU.

Now let me put this in context.  When I finally got to Columbia, my preparation was so thorough that I only had to take one econometrics class from the famous Japanese econometrician Fumio Hayashi.  I then tested out of all my other econometrics requirements.  As good as Fumio was, I have to give the credit to my econometrics professor at BYU, Jim McDonald, for preparing me so well.  That class had more than a dozen people already accepted into top Ph.D. programs at places like Stanford, MIT, and the University of Chicago.   In short, it was a tougher cohort than any group at Columbia University.

My Hail Mary

I worked as hard as I could in Jim’s class, putting in more time in that class than all my other classes combined. At the end I was told I was the high score…not just of that class, but of all time (the course had been taught for over twenty years at the time).  Now the funny thing is I wasn’t even sure I had an “A,” let alone the high score. I asked how Jim could know that I’d done better than anyone else.  He said, “I’ve had a question on that test for years that no one could answer correctly, so I just left it on the test, unchanged.   This year, you gave the first correct response.”  

When I thought about what he’d told me, at first, I didn’t know what question he was talking about, but then I got it…it was the question that I couldn’t figure out how to start, so I just looked at it and wrote down a number that came to me.   It turns out that was the answer.  I thought about that “Hail Mary” many times over the years, specifically what the probability was that a random answer like that would be correct.  It became an obsession, so much so that I even calculated the probability that one student (any student, not necessarily me), would be correct by pure chance.  

 So, what were the odds?  At best it was less than 1 in 225, or in other words less than 0.50% chance that one student over the years would answer that answer correctly by just guessing. At worst it was much lower, less than 1 in 10,000. It depends on one’s assumptions, like if one knows before hand that it was a two digit answer with three trailing digits. Either way it was very unlikely. Ever since then I started saying that “I specialize in the impossible,” meaning if there is a 5% chance then I don’t think something will happen, but if it’s less than 1% then it becomes almost certain. There are again many interpretations of my own statement there, and I’ll leave it for you to puzzle on that.

The Legacy of the Hail Mary

Regardless of how it happened, answering that one question had a significant legacy. For over 30 years Jim and a group at BYU attempted to build a functional code library around his class. He had this idea it could change the world. Then years later, as Jim was preparing to retire, I came into town, myself retiring from my first career in investment management.

In that meeting Jim asked me to return to BYU to teach his class.  He in essence said that my answer to the question was a sign that I was supposed to be the person to continue his class.  That discussion led to many others and before long I was agreeing to teach at the University of Utah (BYU’s in-state rival) but doing so with Jim’s class notes in hand and over 30 years of his work files and the agreement that I would “finish the work.”

The first thing that I did was translate the math in those files into computer code.  For over 30 years Jim had polished it with assistant after assistant.  My original thought was that there would be no errors in the math.  Amazingly, I found over 200 errors in the equations when I translated the pure math into computer code.  That set of computer code became the beginning of the AGORA Initiative, and I expanded from there.

Questioning the Pope and the Bible

Now, I want to be clear what this code was.  You see, Jim’s course itself was a mathematical economics course, but it didn’t use code for anything but the exercises.  As much as I loved Jim, I had a deep sense of skepticism baked into me because at my first job in industry my boss had written the “Bible of the Investment Industry,” which was really just a bunch of mathematical notes that Barr Rosenberg had given in lectures over the years.   When I converted that book to code, I found that many of the problems just couldn’t be solved.  I confronted the author about the problem, and he admitted that he had never seen Barr’s answers, so he just presented the questions as he’d recorded them, and assumed the math was solvable.

This was a massive red flag to me because what the author of this work was saying was that he wrote down the words of Barr Rosenberg, but he never solved the problems himself.  OK…huge problem of methodology here.  I don’t question that he wrote down what he thought he heard, but there was a huge assumption that he wrote it down correctly and an even bigger assumption that Barr was correct in the first place.  In those days, people used to call Barr “the Pope” because his words carried so much weight.   So, here’s this guy writing down what the “the Pope” says (presumably from God?) and everyone else just assumed that he got the transcription right, and that Barr was right in the first place. 

Years later after my encounter with his book, I got to know Barr well, and I can confidently say we disagreed on many things. In a good number of those cases, after a vigorous debate, we would agree that I was indeed correct, or at least more correct.   In so many words, I disagreed with the Pope and, sometimes I was right, and sometimes he was.   The experience with this first book instilled in me an obsession to verify, and like I said, in Jim’s notes I found over 200 mathematical errors when I did so.

Textual Analysis and Sources

It turned out when I did textual analysis (yep, like real Biblical analysis again), I found that Jim had copied down most of the equations with errors from other sources, and he operated under the assumption that because the equations came from published papers or books, that they were correct.   In some some cases they were correct in the sources, just not correct in the context that they were placed in the notes (meaning there had been a contextual transcription error).  This made me very concerned, and it made me adopt the foundational philosophical principle of AGORA Initiative.  If you haven’t confirmed the math in code, it has not been confirmed.  

This is what I always did in industry.   In my first job, I started with Barr Roseberg’s Fortran code library ported to SAS.  I then ported this code to R and later Python.   Each time I confirmed. This was where I left the AGORA Initiative when I departed the University of Utah, a massive library of verified code for financial and economic modeling.  While it started with Jim McDonald’s and Barr Rosenberg’s libraries, it grew to include more hardened and verified code, especially when Hal Tolley took it over as the guardian of the codebase.   This code library was really the central value proposition for my students in the Computational Finance Lab at the University of Utah.  It was mastery of this code library that allowed my students to get jobs in industry right after my course.  It was decidedly something very different from what students normally encounter in academic programming courses.

Theory Through Practice

An example serves to illustrate.  One of my students, who came to me when he was on his second master’s degree and after working for a major broker and pension fund, asked why he could finally understand risk modeling after one session with coding material in the Computational Finance Lab.   I simply said, “The others teach you theory without practice, I teach you theory through practice.”  He went on after that class to become the Data Czar for the State of Utah.  He’s since moved on to other things, but he credited that class with him finally being able to understand issues that he’d spent years trying to master.

That’s what those training files from the AGORA Initiative did, they were thirty years of applied programming at top finance firms and hedge funds (with 30 years of academic coding) put into verified files.  It was those very files that found a new life as the foundational files for a new set of tools. In our company Flex Axion we ported these files into Rust and we called it FlexCode.   Now, FlexCode has grown into so much more than a set of mathematical programming libraries, but it all started with those AGORA libraries.

So, a logical question to ask at this point is how different is FlexCode than various AI Coding tools?  The answer is that it isn’t different, it in fact relies on the underlying LLMs (any of which can be “plugged into” FlexCode), but each of these LLM coding tools in turn then rely on our hardened, optimized, and secure routines.  This might not sound important but let me give you an example of how important it can be.

One of the core packages of FlexCode is the module that allows us to map compute supply chains.  This is a very difficult thing to do that requires adherence to over 30 core axioms that we have meticulously compiled and programmed over 20 years.   That code is not only verified, but efficient.   Recently, I wanted to see if I used the top LLMs and gave them the same axioms and specifications, could they accomplish the same result?  The answer was a resounding “yes” (this should not be surprising, because if I could do it myself, then I could also describe how an LLM should do it).  Now, while the LLM generated code did work, on average it expanded it by twenty times.   What this meant is that it would take more processing and time to complete the job.  Furthermore, the LLM code was not verified as safe, mitigating much of the benefit of coding it in Rust.

Possibilities and Vulnerabilities

This last point is important and something that I will return to in the future, but I believe we’re just starting see problems when AI programming tools are used with unseasoned programmers that do not adhere to basic software engineering principles.  To avoid ambiguity, I strongly believe that AI tools hold the promise of great benefit with the ability to unleash amazing productivity gains, but those gains will not be evenly distributed.   Bad coding and bad ideas will persist and while AI tools can help good programmers and ideas become even better, they can also allow bad programmers to get by, make bad ideas seem good and insert massive vulnerabilities into systems.  I will return to these ideas in the future.