The implications of AI as a skill leveler

Ethan Mollick's fantastic One Useful Thing newsletter has an overview of a recent paper he did studying the impact of AI tools on BCG consultants.

One observation in particular stood out to me:

We also found something else interesting, an effect that is increasingly apparent in other studies of AI: it works as a skill leveler. The consultants who scored the worst when we assessed them at the start of the experiment had the biggest jump in their performance, 43%, when they got to use AI. The top consultants still got a boost, but less of one. Looking at these results, I do not think enough people are considering what it means when a technology raises all workers to the top tiers of performance. It may be like how it used to matter whether miners were good or bad at digging through rock… until the steam shovel was invented and now differences in digging ability do not matter anymore. AI is not quite at that level of change, but skill levelling is going to have a big impact.

This data is invaluable, but I think the framing of it (through no fault of the author's) obscures how individuals should be using LLMs. From the perspective of studying the impact of LLMs on a population of BCG consultants, there are low performing consultants and high performing consultants. But as individuals, we are a mix of low performers and high performers depending on the task.

Therefore the implication is that we should be much less afraid of our weaknesses, especially in areas that are complementary to our strengths. The quality (skill?) of being willing to learn by doing is going to be increasingly important, since the LLM will help cover the flaws. Then the way to maximize one's impact is to pick projects where you have a relative strength (beyond the jagged frontier of AI, in Ethan's framing) and pair it with tasks where the LLM can provide complementary, replacement level support.

I can't claim to have mastered this, but over the past 6 months, I've experienced this first hand across a number of domains: programming and in my day job, about material science, chemistry, and technical writing about cosmetics.

My friend Kenneth quipped recently: "What will you do with infinite junior software engineers?", but it's even broader than that. You have infinite access to baseline expertise in basically anything. What will you do with it?

2023-09-18