Reactions to Situational Awareness

My first glance reactions to Situational Awareness: The Decade Ahead by Leopold Aschenbrenner.

I enjoyed reading it a lot.

The most persuasive part of his argument to me is the relationship between compute and intelligence. This is sort of like the New England Patriots to me; I'm going to believe in it until it stops working. I see reasons why it might stop (run out of data, limited by energy / computing power available), but I don't know when or if we'll actually hit those constraints. People are pretty good at avoiding constraints!

I think he underrates the likelihood of a bottleneck somewhere that keeps us from getting to the AGI he imagines. Any individual bottleneck might be unlikely, but as long as one exists, the entire system is constrained.

Something I see Leopold do at points is assume a super AI, in his case, an automated AI researcher that is 100x as competent as today's top AI researcher. With this assumed, any AI research problem is solvable because you can scale up infinite 100x AI researchers to get around the problem. Once any AI research problem is solvable, then any problem is solvable.

What I think will ultimately happen is something like this:

• An AI will exist that is super human on many dimensions. It will be able to do many things way better than humans and will be inarguably smarter than most humans. [0] Most of todays knowledge work will be offloaded to the AIs. This will be similar to the way that a lot of the production work of 1750 has been moved to machines in factories.
• That AI will also have limitations. There will be some things that it can't do as well as humans or where humans will have the ability to reliably trip it up, despite it's intelligence. To extend the factory analogy, you'll still have humans pressing buttons for reasons other than just keeping the humans in control.
• This will be really destabilizing. Society is going to change more between 2020 and 2040 than it did between 1950 and 2020.

Somewhat off topic: earlier this year, I read Meet You in Hell, which is the story of Henry Clay Frick and Andrew Carnegie. The dynamics of that era, with the railroad leading to a spike in demand for steel and steel leading to a spike in demand for coke were very recognizable in today's AI race.

[0]: I think GPT-4 is already this! Do you know a single person who knows as much stuff about as many things as it does? I don't. And yet it still has limitations!

2024-06-14