Do AI's Dream of Carbon Humans?
The curse of dimensionality is real. "Exponential" improvements will not be enough.
Finally, there’s a topic in the news that I think I can write about. And it’s Good Friday so I have the time to write about it. The stars align.
ChatGPT and its friends have been the talk of the tech world. They are super cool even if it is hilarious the gap between the best screenshots people send and the typical response you get from a normal conversation with the product. (Its grasp of facts is very bad. Even basic textbook definition questions will give you laughably wrong answers compared to the same question typed into Google. But still: automatically generating coherent English to basically any question is amazing.)
But something has been lost in the AI conversation. Very few of the people talking about AI as a munition, a nuclear weapon — perhaps, even worse — mention the central point that gave rise to “machine learning” and “AI” — I always feel more comfortable putting them in scare quotes because often they are “statistics with a much better marketing department”.
I am about to simplify the problem a lot, but seriously, this is what most of what is labeled Artificial Intelligence really is.
Model: Y = F(X) + E, X and E are independent (usually, just mean-independent: E[E|X] = 0).
The goal is to find “F”. The problem is that F becomes very hard to find as the number of variables included in X increases. Note that when I say “very hard”, I do not mean “you need billions of observations”. I mean, very hard. You need an enormous amount of data to fit a model with even 20 continuous variables in general. See, Stone (1982).
Most of AI/ML work is really a collection of assumptions to remove the “in general” part of that condition. To say that the model is sparse in some way. I.e., I don’t know which of the variables in X matter, but I do know that most of them are not too important.
This is a great, powerful strategy for fitting a lot of very flexible models that have been very useful at predicting many real-world phenomenon because these sorts of assumptions are often true in standard business contexts (think: models for which customers are likely to convert or at risk of not renewing, etc).
But many of the problems people are predicting AI will solve “soon” are not such problems. Artificial General Intelligence (AGI), for example, is fundamentally not a low-dimensional problem. Stone (1982) is still true, and it will not go away.
To solve these kinds of problems will require real improvements in mathematics, new ways of modeling problems, etc. I’ve seen no evidence of any improvement in the underlying mathematics. We’re all still repackaging stuff from the 1980’s but with more data.
The world our current AI’s see is too plain, too low-dimensional to lead to a revolution.
You can see this by noticing the difference in quality between a session with ChatGPT which is usually frustrating if you are talking about a subject that you know something about (because it gets basic logic wrong) and the AI image generators which are very good. The dimensionality of images is less than the dimensionality of general thoughts. You don’t have to be perfect when creating images to create very good images. A small discoloration here and there will be imperceptible to the human eye. A misplaced word is the difference between the lightning and the lightning bug, in the words of Mr. Twain.
All of which is to say, we shouldn’t fear that the AI will enslave us or exterminate us, at least not for a while and without some humans getting to do some cool math along the way.
In the meantime, AI will target you for ads, find new customers, help you find your next great read, and create boobs. Lots of boobs, I’m sure.