After playing with state of the art AI models (i.e. GPT4), and having somewhat understood the ways they train such models, I have now come to the conclusion that it is impossible to create super-human intelligence based on existing human data.
As a side note, it is probably possible to make an AI that has attributes that almost reach the level of world class experts, but preparing such training data would be very difficult. And the process would probably be the equivalent of employing a world class expert to train the model as if rearing a child. You need to "teach" it things. Preferably interactively, with thoughtful feedback (RLHF). It would be labor intensive, and there's still no guarantee process would be good enough. At some point one would wonder whether raising a real child instead of an AI child is better use of resources.
But no, the problem with super-human intelligence is not only the quality of the training we give it, but that we don't really have a way to define the optimizing function. Unlike games like chess or go, where there is a well defined ordering of intelligence, in "general intelligence" we have no idea what to look for, which is why the best we can do today is to ask the language model to do "something like this" -- and feed it the best data that humanity can offer. But that would just result in human-level intelligence at best, not superhuman.
Of course, we can always take the simulated game approach -- for example, if we want the best politician, we can create elaborate simulations of the world, and use the same simulated self-play strategies as chess or go to make an AI that is better than what existing human data allows. But to create a simulated world is going to take a unimaginably huge amount of resources. And even worse, we often don't really have an objective "intelligence" function to optimize -- we don't really know what intelligence is.
In the path to so called "Artificial General Intelligence (AGI)", we've solved the problem of computation through Moore's law, we've solved the problem of gathering data through the internet. We've proved that once we solved these problems, we can create human-alike AIs. But to create super-human AIs, we need to *recognize* them as better. We have no way to do so. In fact, we *know* for a fact that we haven't even fully solved the problem of recognizing "human intelligence". The software industry is still debating how to evaluate a person's programming abilities. Schools are still struggling to define objective criteria for assessing academic performance. Smart people aren't recognized by fools to be smart. The bestest AI might sound like a madman to most people. Even if we magically created a "superhuman" AI, it's a virtual certainty that many people won't recognize it as such -- in which case you'd ask, is it really superhuman if nobody thinks it is?
We can't imagine inventing cars. We're all actually just asking for "faster horses". AI will definitely get faster, and maybe that is "superhuman" in some sense. But if you wanted some mind-blowing insights on deep philosophical questions? First, you'd have to be smart enough to recognize the best answer.
Intelligence is not a "NP" problem, there's no general "easy" way to give a text an intelligence score or rank. The Turing completeness of natural language makes generation and validation generally as hard. Intuitively it feels to me that the "human life" subsets of language generation and validation are no different.
