Interviews stopped testing code. The AI era asks something else

"We can't expect engineers to work with AI and then interview them under a no-AI rule." Coinbase's VP of engineering left that line in a report, then spent a full year redesigning the company's engineering interviews. The reason was blunt: nearly 100% of the code being merged now has AI involved in generating it. Yet the interview was still 45 minutes of whiteboard writing by hand, as if AI didn't exist. They analyzed the signals from two interview rounds and found an 84% overlap in predictive power. They had been measuring the same thing twice, with two methods, and that thing is something AI finishes in two minutes. So they turned the question around: in today's working environment, what abilities can AI not supply? The answers came out concrete. Can you catch AI making a subtle mistake? Given three AI-generated solutions, can you judge which fits this situation? When should you trust the output, and when should you doubt it? These questions never used to appear in interviews, because that kind of judgment was hard to test effectively. Now it can be. Running the new design, they found a group of candidates the old process had been missing all along. Their judgment was fine; they just weren't fast at memorizing syntax. What the old test happened to measure was exactly the thing least related to actual job performance. This week, OpenAI published a usage study sampling 7.2 million ChatGPT messages across two years. The most common thing people use ChatGPT for is seeking advice, at 32.1%. Getting advice used to require the right person: someone you knew, willing to spend the time, who happened to understand your problem, and you had to dare to ask. Now a billion people can tell an AI, "here's my situation, what would you do?" These two stories are the same story underneath. AI opened doors that used to be shut. In interviews, what shut you out was memorizing syntax. In advice, it was not knowing the right person. With the doors open, judgment, communication, and the ability to think a problem through get their chance to be seen. Knowing how to use AI is becoming as basic as knowing how to type. The question that surfaces after that: when AI hands you many answers, how good are you at telling which one is right? Coinbase put it calmly: they're hiring people who can get work done together with AI, and that ability is built in the course of using it. Some things you really do have to use to find out.