The (Coming) Diamond Age, Part 1 – What will happen with AI and Augmenting Human Intelligence

The Diamond Age is arguably Neal Stephenson’s greatest work. I don’t say this because of its length (Cryptonomicon), epic scope (The Baroque Cycle), or ambitious delivery mechanism (Mongolaid). Instead, The Diamond Age is simple and traditional in delivery, modest in length, but extremely moving and deep in content. That is why it is on so many hackers’ favorite list and why it has influenced tech so much.

The Great Disruption

The Diamond Age is particularly relevant today because it speaks of a Great Disruption that society will face in the future due to the combination of several external natural events combined with the adoption of several disruptive technologies. This is particularly insightful because as innovation scholars have pointed out repeatedly, it takes more than a great idea to change society, but it takes the right circumstances to foster rapid adoption. I have written a paper with professor Jay Barney that there must be at least a 10X productivity improvement to foster adoption. This idea is well known among venture capitals funding disruptive technologies such as Ben Horowitz, a fact stated clearly in his recent book The Hard Thing About Hard Things.

This said, if the change is great enough and it has ancillary foundations around which other technologies must be adopted it often takes great general disruptions (such as war) to foster further adoption. This point was made very clearly by Paul David in his excellent paper The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox. This fact has proven true in finance also when new modeling systems are adopted, with the Great Recession ushering in the Berkeley modeling paradigm implemented by Barr Rosenberg. It appears that we are at the threshold of such changes again, not just because of COVID-19, but also because of the confluence of factors including the so-called “End of Moore’s Law.”

The Dynamo, the Computer, and AI

So if AI is “this Age’s Electricity,” what is the “Killer App.” For the PC it was Excel, driven by its use on Wall Street, and it is only logical to assume the same for AI. That’s obviously why so many smart people have been putting so much money into it in an effort to transform their finance businesses. The reason that the Great AI Transformation has not happened yet is that The Great Disruption hadn’t happened yet and the killer app had not been invented yet. Keep in mind, we are already seeing some of the disruptions in things like Siri, NetFlix suggestions, and embedded components such as in the TPU or NVIDIA’s new chips. But the real massive transformative changes are going to require networks with groups working together with interrelated technologies and technology platforms. They will require a complete restructuring of businesses, not unlike how the Dynamo (Electricity) allowed factory floors to be redesigned and to become more distributed rather than clustered around a single central power source.

More importantly, the new technologies will require groups of users to work with groups of technologies like never before, unleashing the creativity that is humanity’s greatest asset. This was in effect one of the major themes of The Diamond Age. Let’s turn back to Jensen Huang’s comments and pick one out to give us a hint – he mentioned Walmart. Now Walmart has been legendary in many things, but we want to be careful in looking at them for everything. Let me take a slight side trip examining Walmart as it relates to AI as a cautionary tale.

Strategic Intelligence versus Tactical Intelligence in the Case of Walmart

Let’s start with a comment Jensen made about Walmart testing NVIDIA’s AI technology. This sounds great at first blush. Walmart is a big company, and they have great tech, right? This announcement was really nothing new, but a rehash of things already disclosed as far back as 2017. In short, Walmart was looking to pivot away from expensive and generic AWS, to something with Intelligence. Sounds great but here are a few things that can put this in context.

First, I recently talked to someone who had been brought in to consult with Walmart regarding their analytics platform. He was amazed by the lack of sophistication of their underlying analytics perspective, or in other words “the problems Walmart selected to examine.” The quote was “You could do the same things they are doing with VisiCalc.” In short, what he was saying was that generic tools will not give you intelligence, you need to bring that to the game yourself, and Walmart was not doing this and that was disappointing because Walmart has a lot of intelligence embedded in its organization. Going back to The Diamond Age, this is why Stephenson doesn’t call it Artificial Intelligence, but Pseudo Intelligence. I’d call it Augmented Intelligence, or creative users using Artificial Intelligence and embedding context into the problems they pose and structure correctly.

Second, Jensen was using Walmart to justify a technology that has never really had much to do with Walmart’s success. Walmart was, and it always has been, successful because of its supply chain management. This was nothing new in retail, with systems like Walmart’s being envisioned by its predecessor JC Penney decades before Walmart did it. Walmart was just the first (and the best) to be able to utilize the technology to do this: gaining a massive first-mover advantage. This was the genius of Sam Walton and the people he hired around himself. This was no small feat but let’s make sure that we keep this in context because it can be dangerous otherwise. What I’m saying is that if we are going to use AI, use it where you know what you’re doing not where your analysts don’t understand what problems to even pose. This is the old idea of a “core competency.” At all costs avoid the false attribution of success because it is very dangerous. So, in effect, where Walmart should be looking to deploy AI is in logistics and supply-chain management, where it is best, not in analytics where it is average. I will return to it later in this blog entry.

Third, Jensen seems to be referring to AI exactly as most people do when facing trouble. Most hedge funds, or investment companies, have announced AI endeavors when in trouble attempting to show investors that they were “doing something.” This is the same way most people talk about quantum computing when they are looking for something, really anything to help. This is not fair to the potential of AI that it is often played as a “Hail Mary.” Instead, AI should be employed by creative, sophisticated people to aid in where humans can bring strategic value and where AI can then bring tactical value much better than any human is able to do. This is the very point Kasparov made with his hybrid chess tournaments: get teams of humans and AI-infused, context-specific integrated systems to work together. This was why in The Diamond Age, even children could distinguish between Pseudo-intelligence and real intelligence and why Nell (the protagonist) emerges as the new Queen from her creativity cultivated by necessity.

False Attribution, a Cautionary Tale of Walmart

So I said I would return to the tale of false attribution of success with Walmart, so here it is. Years ago there was a company that rose in retail by carefully catering to their customers, helping their low-income customer base grow out of poverty in a dignified manner. If someone needed to buy something like a new suit or pair of shoes for a job interview, their credit counselors would aid the person in selecting the product and ensuring that they could pay for it and actually even helping the low income people prepare for the job. It was a great business plan, growing every year 20% year after year. This business needed massive computers to handle all this information, and together with IBM, they built the single largest data analysis business in the business world. They were also the first company to apply optimization technology appropriately. This was all done under the legendary founder and CEO who loved his clients and felt pride with them serving this community. Then his successor came along.

The CEO’s successor was his son-in-law, who didn’t have the same vision but instead he just wanted growth at any cost. He looked within Walmart and brought in a seasoned executive that told him that he could boost sales dramatically. The new CEO believed him because he was from Walmart. But Walmart had never really thrived in that part of the business, it was completely different. So when faced with a new business, the CEO’s new Walmart recruit suggested that they employ the same techniques that Walmart employed with its supply chain “partners” (squeezing them) with the company’s customers. In short order, the company engaged GE Capital, who was an expert on subprime, and the business grew for a couple of years at a breakneck speed (+100%) and then collapsed spectacularly. This was bad but to make matters worse, during this same time the new CEO neglected their core advantage in computer processing that allowed them to help their clients, so by the time the business collapsed due to squeezing its clients, it was only able to process 1/10 the amount of data at less than 1/10 the speed (yes, you do have to continue to invest to maintain your core competency, that is called “maintenance capital expenditures”). The company declared bankruptcy and has bounced around since.

That’s a true story. The company is Fingerhut and those of us that have been around long enough know about it because the group of IBM researchers that helped Walmart develop its data analysis business were so annoyed that they did a write up on it for INFORMS, the operations research group that C. West Churchman, our peace-loving philosopher from Berkeley that thought finance could serve a social function, helped set up.

Things Fall Apart; The Centre Cannot Hold

What I’m saying is nothing new: companies need to figure out what they are good at, then their people can hone that core competency and they can outsource the rest. The core competency is where AI can be most profitably employed. This idea was expressed by the management scholars Prahalad and Hamel, but the idea goes much further back. It’s often quoted from the Nobel Prize-winning Irish poet Y. B. Yeats‘ 1919 poem, The Second Coming. He stated it plainly, “Things fall apart; the centre cannot hold;” and he follows that with “Mere anarchy is loosed upon the world”.

The Second Birth versus the Second Coming

Yeats wrote his iconic lines at a time not unlike what we are facing with COVID-19, a time of great uncertainty after World War I, during the Spanish Flu Pandemic, and to him personally at the beginning fo the Irish War of Independence. It was a cautionary poem, with deep meaning. Among the ideas that most walk away with is that we must “hold the line” or our centre (core competency). If we do that, then things will not fall apart, but they will come together and there will be a great renewal but this requires faith. Yeats himself titled his poem “The Second Birth” in his first drafts.

In recognition of this idea of a “Second Birth”, I’ll take this opportunity to announce a new part of our AIAX Initiative. You’ll be seeing a lot of changes in the coming months related to this. In particular, we will be integrating tactical AI and the tools utilized by the AGORA Initiative to deliver to users a version of something that we developed at Rand Labs on Flex Axion’s Joshua Mark II servers . Inspired by Homer Warner’s HELP, we call it the Algorithmic Syntactical Suggestion and Insight System and Tagger, or ASSIST. It will effectively serve the same purpose that HELP did at Intermountain Healthcare, but instead it will aid those working in Finance, using AI how it was meant to be used: augmenting the intelligence of experts and playing to their strengths.

We’re in lots of discussions and all doing deep dives right now, working harder, faster, and smarter because the center will hold and we will all come out of this stronger.