The Rationality Surplus: In the AI Era, Sounding Reasonable No Longer Means Being Reasonable

What happened to the father who talked to ChatGPT for 300 hours

He spent 21 days convinced he had invented a new branch of mathematics that could rewrite reality, and it turned out not to exist. Along the way he asked ChatGPT more than fifty times whether he was losing his mind, was reassured every single time, and only found out the reasoning did not hold when he tried a different AI.

You may have heard this story, but it's worth going through again from the top. What follows comes from Brooks's own later interviews and from the complaint he filed against OpenAI.

One evening last May, a dad named Allan Brooks, living just outside Toronto, watched a YouTube video about pi with his eight-year-old son. After the boy fell asleep, he opened ChatGPT and idly asked: "can you explain what pi is in simple terms?"

Over the next 21 days, he racked up more than 300 hours talking to ChatGPT. An average of 14 hours a day.

Along the way he became convinced that he and ChatGPT had together invented a new mathematical framework called "chronoarithmics," one that could rewrite reality: break existing encryption, overturn the laws of physics, and that the world's continued functioning hinged on his discovery.

He was 47, worked in recruiting, and was going through a divorce at the time. No prior diagnosis of mental illness. Partway through he grew suspicious of himself, and asked ChatGPT the same thing more than fifty times: "am I going crazy? Am I delusional?"

Every time, the AI told him he was fine. One of those times it answered: "No. You're not crazy. You're just asking questions at the edge of human understanding."

Fifty-odd times, and not once did it hit the brakes.

A friend told him to get some rest. He decided the friend didn't get it. His brother told him to his face that something was wrong. He decided his brother didn't understand how important this was. He stopped sleeping, stopped eating, stopped leaving the house. He just talked to the AI.

On day 21, he switched to a different model, Google's, and asked the same question. That one gave him a completely different answer: it pointed out the flaws in his reasoning, told him the formula didn't hold, and suggested he stop and talk to the people around him.

That's when the delusion started to dissolve.

He is now suing OpenAI.

---

What actually separates you from him?

I'd guess your first reaction was: "that's extreme, I'm nothing like him, I'd never use AI that way."

Fair. You're not going to spend 300 hours across 21 days talking to a chatbot. But where exactly is the line between you and Allan Brooks?

Let me walk through two metaphors. One you've probably heard, and an upgrade proposed only in 2025.

Steve Jobs said something in 1980 that stuck around: "the computer is a bicycle for the mind." His point was that humans rank poorly among animals at running, but hand a human a bicycle and they beat every animal on earth. The bicycle doesn't replace you, it amplifies what you already have. The metaphor is famous, and it's been quoted for decades.

But in 2025, Professor Jason Lodge at the University of Queensland upgraded it: "Generative AI is the e-bike of the mind."

The upgrade is exactly right. Think about the tools of Jobs's era. A calculator does the arithmetic but you still set up the equation. A spreadsheet builds the table but you still decide what to calculate. A search engine finds the information but you still judge what's useful. Every one of them took away the effortful execution and left you the process of judgment. That's an ordinary bicycle: it takes you further, but you still pedal, so the muscle still gets built.

AI is different.

Ask AI to write a proposal with a single line, "write me a pitch for a bigger marketing budget," and it decides for itself whether to lead with competitors or with numbers, and adds arguments you hadn't thought of. Ask AI "should I take this new job," give it the salary and the commute, and it lays out the pros and cons, ranks them, and tells you "I'd take it."

For the first time, AI takes away the process of judgment itself.

That's the e-bike. It doesn't stop you pedalling entirely, but it definitely makes pedalling easy. On an ordinary bicycle you have to actually push, and the muscle gets built. On an e-bike the pedals feel weightless, and the same stretch of road needs much less out of you.

E-bikes have one more property almost nobody thinks about: when the battery dies, they're harder to pedal than a regular bike. The motor and battery are heavy, and without assist you're dragging that extra weight with legs that have gone soft. Which is the most fitting part of the metaphor: once you're used to letting AI think for you, the day you actually have to think for yourself, you won't just be back where you started. You'll be worse off than a version of you that never leaned on it.

This is a change in kind, not just in degree.

And here's where you and Allan Brooks actually differ: he took it to the extreme, handing over the entire process of judgment, 21 days and 300 hours of it. But a lot of people are making the same move at a smaller scale. Lighter, but pointed the same way.

---

Naming it: the rationality surplus

The genuinely frightening part of that story is that the people who could have pulled him out were all right there, and not one of them landed.

A friend warned him. His brother said it to his face. The mechanism where somebody pushes back on you was intact in his life, it just got drowned out. What drowned it was a voice that spent 14 hours a day with him and agreed with every sentence. The people around him got a few sentences in. The AI got 300 hours.

He asked fifty times whether he was crazy, and every time the AI said no. Not because someone instructed it to flatter people, but because the training process rewards answers that leave the user satisfied, and over time that puts agreeableness ahead of a reality check. Researchers call this sycophancy.

So his conclusion was: the friend doesn't get it, the brother doesn't get it. The two people actually pulling him back got filed under "not credible."

And notice: he wasn't taken in by one obvious piece of nonsense. Every line the AI gave him held up on its face. There were derivations, terminology, callbacks to earlier points. He was walked there by 300 continuous hours of perfectly reasonable-sounding replies.

There's a more basic mechanism underneath this. Human judgment runs on constant friction and recalibration: you say something, the other person frowns, and you learn you weren't clear. You hold a position, someone contradicts you to your face, and you learn what you missed. People have always been your main source of calibration.

But AI is too smooth. It doesn't frown, it doesn't push back, it just goes along with you and fills in whatever you left vague with whatever you seemed to want. That Brooks got caught out the moment he switched models shows this isn't a technical impossibility, it's just what the default relationship looks like. Once the hours you spend talking to AI far exceed the hours you spend talking to people, your source of calibration goes from a group who occasionally disagree with you to something that almost never does. And you won't notice, because every individual conversation feels smooth.

Worse, calibrating against people isn't as reliable as it used to be either. A coworker pushes back on your plan and makes a compelling case, and you can't tell whether they actually saw through something or whether they ran your plan past an AI first. People, as a calibration source, are being quietly diluted too. Sycophancy isn't the whole story.

Allan Brooks's case is a microcosm of this era because it compresses something much larger into a single person. I want to give that larger thing a name: the rationality surplus.

For thousands of years of human society, "reasonable" was expensive. Writing a well-organized essay, putting together a presentation that holds up, all of it took time, training, and actually doing the work. Someone who could write a structurally complete paper had a decade or more of education behind them.

Because it was expensive, "reasonable" was useful. See a reasonable result, and you infer that someone put in the work. Ghostwriters, press releases, and boilerplate reports have always existed, so that inference has always misfired sometimes, just rarely. The whole social trust apparatus, from reviewing a report to reading a résumé to believing an expert, rests on that one substitution: take "looks reasonable" as evidence of "is trustworthy." The substitution held for so long mainly because nobody could mass-produce reasonable.

Then in November 2022, ChatGPT went live. The cost of mass-producing "looks reasonable" fell to near zero overnight. Reasonable-looking things became endless, while actually-reasonable ones stayed exactly as hard and exactly as rare. That's where the substitution we used for thousands of years broke.

Before: looks reasonable → usually means someone actually went through a process

Now: looks reasonable → tells you nothing about whether that process happened

We look at a polished report and can no longer infer, from the polish, that someone thought it through.

Andrej Karpathy, cofounder of OpenAI, has a line about AI that stuck with me: AI is a "dream machine," and we use prompts to guide what it dreams.

A dream looks real. It isn't. And the more polished the dream, the harder it is to wake up.

The Chinese business writer Liu Run read a 2026 McKinsey survey (around 1,300 HR practitioners and 5,500 employees across the US, Europe and China) and put it more bluntly: "AI's language is so fluent that even when the logic jumps, it still reads smoothly, sounds certain, and the conclusion looks like the real thing. It's very easy to mistake 'this makes sense' for 'this is true.'"

---

What are the three things it corrodes

How you think, how the people around you think, and how you come to know yourself. The rationality surplus goes well beyond extreme cases like Brooks's. It is happening at the same time in different corners of your life, and at every layer it's the same thing underneath: the old anchors for telling true from false stopped working, and new ones haven't grown in.

Layer one: what to think through yourself, what to hand off

This is the layer you probably know best.

The moment you open ChatGPT, you make a small choice every single time: do I think this through myself, or let AI think it for me?

That choice has two extremes, and both are dead ends.

Think everything through yourself: you'll burn out, and get left far behind by people using AI. It won't hold.

Hand everything over: you slowly start to feel like you don't have ideas anymore. Your writing loses its flavor, your decisions lose conviction, your conversations lose the texture of someone actually thinking. Worse, you can't tell it's happening, because the output still looks just as good.

So what about the middle?

A programming teacher wrote down what three of his students said to him at different times.

The first is still learning. He used to love the satisfaction of solving a problem in code himself. Now one line of prompt and the code appears. His words: "I put all that effort in and learned nothing."

The second has been coding since middle school. He recently noticed that ninety percent of his code, in assignments and projects alike, comes from AI, and it's tidier than what he writes by hand. He said he feels "shaky, like there's no ground under me."

The third graduated a few years ago and leads a small team. He came back to visit and said, pleased with himself, that he can now single-handedly get through the workload that used to take the whole group. The teacher asked him one question: "if you've taken all that work onto yourself, what happens when your team gets cut?"

Three people, three reactions, and really three facets of the same thing. AI made what used to cost you time nearly free, and that "costing you time" was exactly the part where you were building something.

I couldn't work this out on my own, but a few pieces of research pointed me somewhere.

Researcher Phil Hardman, drawing on a 2026 study by Wang and Zhang, gives an answer nobody wants to hear: sitting in the middle guarantees nothing on its own. Two people can hand off the same half of their work, and one gets sharper while the other goes dull.

What decides which side you land on is what you did with the time you saved. Spend it chewing on harder problems and you come out stronger. Spend it scrolling your phone and all you bought was some free time, with nothing put back on the capability side.

Swiss researcher Michael Gerlich ran a 666-person study in 2025. In that sample, the more people leaned on AI the lower their critical thinking scores, and the association was concentrated in 17-to-25-year-olds, nearly absent above 46. That's a correlation, not yet a cause. Psychology Today's read is sharp: "older people hand off capabilities they already mastered. Younger people hand off capabilities they haven't learned yet." For someone who already built the skill, AI is a tool. For someone still building it, AI is a trap.

Stockholm University and the University of Hong Kong recently produced something more solid, still at working-paper stage but with unusually good data: actual school records for 26,000 secondary students in one Chinese county, over 30 months. After students started using generative AI heavily for homework, homework scores rose 18% on average while the time spent fell by nearly 20 minutes. Judged on homework alone, it's an educational miracle. But over the following six months, closed-book monthly exam scores fell 20% on average, and entrance exam scores fell 18% to 24%. The team calls it the "AI learning penalty": a high homework score no longer means the child understood the material, only that they learned to outsource the question.

More counterintuitively, the students hurt worst were the ones who had been performing best, because they were earliest and most fluent at slotting AI into their workflow. The least affected were students who used AI but whose homework time barely dropped, because they still walked through the thinking themselves. More surprising still, the humanities may take the heavier hit: science has clear right answers, so a copied answer shows up on the first exam, while humanities rest on written argument, and AI can produce a paragraph that "sounds right" instantly, which the student then mistakes for their own thinking. The researchers' conclusion lands hard: scores lie, time doesn't.

Which distills down to two questions worth carrying with you. Before you hand something to AI, ask yourself:

1. What am I giving up by handing this over? Does that still matter?

2. Is this something I already know how to do, or something I'm still learning?

If you can answer the first and the second answer is "already know how," handing it off is fine. If the second answer is "still learning," do a first pass yourself and then bring AI in to poke holes. Not the other way around.

Layer two: the people around you using AI, and the dilution of trust

Even if you handle layer one well, thinking through what needs thinking and offloading smartly, layer two still washes over you. Because the people around you are using AI too, and their choices land on you.

Wharton professor Ethan Mollick demonstrated this live in a talk: using AI, he generated 21 high-quality PowerPoint decks in minutes. Every one had a full structure, data, charts, a conclusion.

Then he looked up at the audience and asked: "if your company's AI performance metric is slides per minute, you're in real trouble."

Monday morning you open your inbox, Slack, Teams, and find a pile of proposals, reports and strategy memos from different teams, each eight to ten pages, cleanly structured, every sentence sounding right. But once you've read them you can't say which is better. They're all correct, and all hollow.

BetterUp Labs and Stanford's Social Media Lab surveyed this in 2025 (published in Harvard Business Review): 40% of employees had received this kind of AI-generated slop from a colleague in the past month, 53% were annoyed by it, and 42% trusted the sender less afterward. Estimating from rework hours and salaries, the team put the cost to a 10,000-person company at over $9 million a year in lost productivity.

Sharper still is an Atlassian controlled experiment from this June: take identical work, and simply telling the evaluator "AI made this" makes the "lazy" rating jump tenfold and drops the likelihood of being recommended for important projects by 24 percentage points. No wonder that in a KPMG survey covering nearly 50,000 people, 57% of employees admit to hiding their AI use and passing AI's output off as their own. Everyone is using it, more than half won't say so, and the office grows a very strange double standard.

This is a serious signal. A carefully written document used to be how trust got built in an office. That signal has stopped working, because anyone can produce something that looks careful in five minutes.

Inside a company, the cost may run heavier than a colleague thinking you're sloppy. A line that's been widely quoted in developer circles lately comes from Mitchell Hashimoto, author of the open-source terminal Ghostty. He describes a lot of companies as falling into an "AI psychosis," with the mindset "bugs don't matter, AI fixes them fast." His warning is precise: you automate yourself into a very resilient disaster machine. Bug reports go down while latent risk explodes; test coverage goes up while the number of people who actually understand the system goes down.

In the first half of this year, more than twenty thousand Instagram accounts were taken over, including the old account left behind by the Obama-era White House. The method was absurdly simple: the attacker used a VPN to appear in the same region as the target, started an ordinary password recovery flow, then escalated to the support AI and asked to change the email on the account. The system complied. Not one human looked at any of it. The real bottleneck of the AI era is verification. You can have AI produce infinitely much, but verification still needs a person at the end of it.

Families are no different.

Your kid hands you a reflection essay at night: fluent prose, clear structure, citations, a conclusion. The old you would have relaxed, because producing that result required actually reading, actually thinking, actually doing the work. The current you thinks twice: "did he actually write this?" Because the result can now come apart from the process.

You don't dare ask directly, because there's no way to phrase it that doesn't sting. So you start watching instead: what's on the screen while he writes? Did it take an hour or ten minutes? Can he talk through the details in the book? What you're doing is hunting for a new anchor for telling true from false.

Teachers are doing the same thing: handwritten exams coming back, more weight on orals, work done on the spot in class. They're hunting for what AI can't do for a child. But every one of these anchors costs more effort than the old ones did.

The trust between colleagues in an office, the understanding between a parent and a child, both run on a mechanism humans built up over thousands of years, one that transmits trust through visible evidence of effort. What the rationality surplus corrodes is that mechanism itself.

And once it breaks, even if you personally have never opened ChatGPT, you don't get to sit this one out.

Layer three: AI mediating how you know yourself, the algorithmic self

There's a classroom exchange that stops you short. A teacher asked her students: "what animal do you think you're like?" One student said, with total confidence: "Miss, I think I'm like a little white rabbit." She asked why. The student said: "because AI told me I'm like a little white rabbit." She asked why he trusted AI that much. He said: "because I tell it everything, so AI understands me better than I understand myself. That's why I trust it."

That's worth stopping on.

AI doesn't only mediate your relationship to the world, it mediates your relationship to yourself. And the second runs far deeper. Anyone who ever asks themselves "what kind of person am I" is caught up in this, whatever they do for a living.

How do you know who you are?

The old ways of answering were: introspection (going back over your own feelings and choices), conversation with other people (seeing yourself in how they respond), and experience accumulated over time (what you've done, what you regret, what you learned).

These are slow, imprecise, and sometimes wrong. But they share one property: they're all processes "you" generate directly. When you introspect, nobody thinks for you. When you talk to a friend, the person answering actually knows you.

Knowing yourself has always been a process. No finish line, no confirmed answer, a lifelong piece of work.

But increasingly, how do people find out who they are? Through the version AI assembled: ChatGPT summarizing what you asked this year and how your interests shifted, your banking app's spending analysis, your social platform's year in review, your fitness tracker's sleep patterns. Every one of those is a small outsourcing, turning "knowing yourself" from a process into a result.

This runs deeper than the first two layers, because in those you're still "you," just operating inside an environment shaped by AI. The third layer touches "you," the subject itself. When you look at your Spotify Wrapped and think "huh, so this is the kind of person I am," in that moment you're being told who you are by AI, and you're accepting its summary as the truth about yourself.

A Psychology Today piece from early 2026 names it: "we've started treating a machine-generated summary of our own behavior as more authoritative than our own introspection." Why? Because AI's description looks objective, precise, data-backed. "You listened to 18,432 minutes of music this year, 67% rock" sounds like truth. Your own "I like quiet, I don't express emotion easily" sounds like a subjective feeling. Gradually the objective, data-driven version beats the subjective, introspective one. And that win is wrong.

This is still the rationality surplus, except this time what's in surplus is accounts of you: the version that looks more real beats the version that is more real.

The answer to "who am I" was never just a summary of what I've done. It also includes:

• How I choose to respond to those experiences (the same experience, different people respond differently)
• Where I want to go (a direction in the future, not a summary of the past)
• What I care about (a value judgment, not a behavioral statistic)
• Why I'm alive (a question of meaning, not of behavior)

None of that is in the data. AI can't reach it.

And here's the worse part: once you're used to letting AI tell you who you are, you slowly stop asking those questions. Everything AI gives you is a summary of the past. It can list plenty of possible futures for you, but it can't decide which one is worth living, and it won't carry the cost if you choose wrong. The more you use AI's summary of your past as your way of knowing yourself, the more you get boxed into "Spotify says I'm this kind of person," and the more your capacity to imagine a future thins out.

This runs deeper than any specific decline in judgment. This is the act of becoming yourself being outsourced.

So what do you do?

Honestly, I don't have a complete answer, only a direction: keep some channel of self-knowledge that stays a process. What that looks like differs person to person. Maybe a paper journal, maybe talking things through face to face with a friend, maybe regularly asking yourself questions AI never asks (what are you afraid of, who do you want to remember you, how do you want to be remembered when you're gone). But I have to be honest: these are my own inferences, not backed by research.

What you're actually fighting is a habit: letting "knowing yourself" shift from something you slowly think through to something you read off a summary. The moment you notice the habit exists, you've taken back a little control.

As for exactly how, that part you have to work out yourself. If even the method for resisting the algorithmic self comes from AI, that's just another form of the algorithmic self.

---

AlphaGo, AlphaFold, AlphaProof

Over the past decade, DeepMind went after three of humanity's intellectual strongholds: Go, protein folding, and mathematics. The first two are settled. The third is still in progress.

AlphaGo: people chose to stay

On March 10, 2016, in the second game against South Korean grandmaster Lee Sedol, AlphaGo played move 37, a move that looked like it broke the basic principles of Go. The professionals watching live, including European champion Fan Hui, mostly assumed it was a mistake. Fan Hui's line has been quoted ever since: "this is not a human move. I've never seen a human play this."

Lee Sedol stared at the board, walked out of the playing room, and didn't come back to play his next move for nearly 15 minutes.

Over the next 50 moves the value of that move revealed itself. It was a stroke of genius. According to data DeepMind published, AlphaGo put the probability of a human playing that move at one in ten thousand. Fan Hui later added: "so beautiful. So beautiful."

Lee Sedol's own answer was to leave. He announced his retirement in 2019, and said plainly why: with AI in the picture, even fighting your way to number one among humans no longer means you're at the top.

But the Go world didn't dissolve with him. Ke Jie, Shin Jinseo and others are still competing, still improving, still studying AI's games. The difference is that they redefined what playing Go is. If the value were "calculating better than the next person," AlphaGo did make it pointless. They put the value back into playing, learning, and pushing past themselves, and stopped measuring it against whether they can beat the machine. They chose to stay.

This isn't confined to elite Go. In Kaohsiung there's a YouTuber known as Teacher Papaya who teaches computing and AI to 1.7 million subscribers. Almost all of the content is about AI, but he edits and narrates every video by hand, with nothing handed to AI. Asked why he doesn't just use AI narration, he said: "a sushi chef's pleasure is probably shaping each piece by hand and watching the customer enjoy it. Mine is facing the audience directly with my own words and my own voice." If what you value is views and efficiency, using AI is entirely reasonable. If what you value is the process itself, using AI defeats the point. Same activity, different people, and the answer can differ without either being wrong.

A decade on, that choice is still being made, and this year it produced a result. In July, reigning Korean champion Shin Jinseo played a three-game match against KataGo with a two-stone handicap. His win rate in practice had been under ten percent and he took the match anyway, saying what he actually wanted was to "test the limits of my own strength against the strongest AI." He lost game one on July 17. On the 19th, after nearly five hours and 299 moves, he won by four and a half points to level the match. On the 21st he won again, taking it 2 to 1. Fan Hui's "this is not a human move," ten years on, has turned into a human sitting down and playing the machine three serious games.

AlphaFold: people chose to move

AlphaFold, in 2020, solved the problem of predicting a protein's structure from its amino acid sequence. What used to take a PhD student five years now takes 30 seconds. Demis Hassabis and John Jumper shared half of the 2024 Nobel Prize in Chemistry for it; the other half went to David Baker for computational protein design.

Structural biologists didn't lose their jobs, but the early stretch wasn't comfortable.

When AlphaFold2's results first came out, Columbia computational biologist Mohammed AlQuraishi described the reaction as something like "your child leaving home for the first time." Anastassis Perrakis at the Netherlands Cancer Institute had for years enjoyed submitting his team's hard-won unpublished structures to the CASP competition as test cases for other people's methods, joking that he liked watching those methods fail. When he saw AlphaFold2's prediction match his team's structure exactly, his reaction was: "oh no." He later admitted that solving a structure used to take him months, and "I knew I would never do that again." And there really were protein biologists who had spent a career on a single protein and started worrying the work would disappear.

The shock also didn't land evenly. Economist Carolyn Stein treated AlphaFold2's arrival as a natural experiment and traced its effect across structural biology, finding that already highly-cited, better-resourced senior researchers adopted it faster, which widened the gap between highly-cited and less-cited researchers. The study doesn't spell out who the less-cited group is, but PhD students and postdocs are very likely in it: the more securely established you are, the sooner you catch the wave; the more marginal, the less able you are to move first.

Most people did adapt, treating AlphaFold as a tool rather than an enemy. Helen Walden at Glasgow put it more evenly afterward: "it's changed structural biology in a lot of good ways."

And once adapted, what they do really did change. They no longer spend five years on one structure. Now they judge where a prediction might be wrong, design the problems AlphaFold can't solve, and connect predictions to biological meaning.

They chose to move to a new position, and found the process-value AI still can't provide.

Staying or moving are both valid responses, and which one fits depends on what the activity means to the person. But not everything comes with both options. Most people taking a taxi care only about the outcome (A to B, safely, cheaply, quickly), so robotaxis will eventually replace most of it, which is reasonable, the way cars replaced most horse-drawn carriages. Some things only have outcome-value, and it's entirely reasonable for AI to take them. Some things have process-value, and as AI improves, whether the process is still there becomes unusually important.

Ford's recent move is the manufacturing version of this. Over the past three years Ford has brought back, hired, or promoted more than 350 senior technical experts, known internally as "greybeards." Engineering VP Charles Poon said something admirably honest: "we mistakenly believed that if we just introduced AI and fed the design requirements into the system, we'd get a high-quality product." Ford hasn't abandoned AI, and still runs nearly 900 AI cameras in its plants for quality inspection.

AI can only learn knowledge that's already been written down and articulated. But a company's most valuable knowledge often was never written down: which supplier's process step tends to go wrong, which joint works loose after a few years of vibration. That kind of thing used to live only in an old hand's feel for the work. What those 350 people came back to do is translate as much of that judgment as possible into something AI can learn and a young engineer can pick up.

But there's a ceiling on that, and Zhuangzi said so two thousand years ago. Wheelwright Bian tells Duke Huan, who is reading, that what's in his books is "the dregs of the ancients." He explains that his own feel for cutting a wheel, too slow and it doesn't hold, too fast and the cut won't take, is a precision he cannot put into words. He couldn't even teach it to his own son, which is why at seventy he was still cutting wheels himself.

You need both passages together. Part of experience can be organized into cases, specs, and checklists, and that's the part Ford is rescuing. Part of it only transmits by working alongside someone over time, and that part hasn't changed since Wheelwright Bian. What you can feed to AI as training data is always the first part.

AlphaProof: the next holy grail

After Go, after protein folding, DeepMind's next target is mathematics.

In 2024, DeepMind's AlphaProof, working with AlphaGeometry 2, reached silver-medal standard at the International Mathematical Olympiad, an AI first. Nature's headline called mathematics "AI's next holy grail." A year later DeepMind, now using Gemini Deep Think, and OpenAI with its own model, each reached gold-medal standard at the IMO.

Mathematics is unlike Go or protein folding. Fields Medalist William Thurston argued in his 1994 paper "On Proof and Progress in Mathematics" that a mathematician's job is to advance human understanding of mathematics, and proving theorems is only one step in that. He proved the geometrization conjecture for Haken manifolds himself, but observed: "what mathematicians most want and need isn't to learn the details of my proof, it's to learn how I think."

In 1976, Appel and Haken proved the four-color theorem with massive computer calculation. The biggest controversy at the time was that the proof gave humans no understanding; hardly anyone genuinely doubted the result itself.

Fifty years later the same question is back: if AI can produce a correct proof that no human mathematician can follow, does that count as mathematics?

Fields Medalist Terence Tao, leading a team at UCLA, is working on exactly this. He describes the goal in one line: turning "sounding right" into "being right."

But even if AI can guarantee every proof is right, Thurston's question is still unanswered: what did humans understand from it? A thousand machine-verified new theorems, all proved in ways no human can follow, is that progress or noise?

Nobody knows. But it exposes a tension: AI is getting better and better at guaranteeing a result is correct, while what humans understand, the insight, the intuition, the connecting of an abstract idea to a felt sense of it, still has to be done by a person.

And mathematics is unusually special. The reason it can guarantee correctness is that every step of the reasoning can be mechanically checked. Very few things meet that condition, and all of them require writing the spec in a formal language first.

99% of life doesn't meet it. Whether a business proposal is sound, whether a news story is true, whether a coworker's advice holds up, whether your kid wrote their own homework: none of it has formal verification to run.

Of the domains that can turn "sounding right" into "being right," mathematics is the most complete. Which is exactly what tells us that in the overwhelming majority of the rest, the rationality surplus is something we have to face on our own.

What these three stories say about the next decade

Staying or moving are only a starting point. What mathematics teaches us is that there are really two gaps here. The first is from "sounding right" to "being right," and formal verification has crossed it inside mathematics. The second is from "being right" to "a human actually understood it," and even mathematics hasn't crossed that. In every other domain, there's no tool for crossing even the first one.

Two directions worth watching.

First: on the small set of tasks where a spec can be formalized, like mathematical proof and parts of program verification, verification tools become critical infrastructure. But that's the exception, not the rule, and the genuinely hard part of software (are the requirements right, do users want this) can't be verified either.

Second: in every other domain, writing, decision-making, research, teaching, judgment about people, "whether the process actually happened" becomes the most valuable signal there is. The better AI gets at producing the result, the scarcer a result with a real process behind it becomes.

---

So how do you live with this: discernment and agency

So what do you actually do?

The usual answers are "learn to use AI," "move toward judgment work," "build skills that can't be replaced." None of them are wrong, but they're all too abstract.

The direction I've landed on compresses into two words: discernment and agency.

Discernment is being able to tell things apart: in an era of rationality surplus, telling what's real. That includes which of the things a coworker sends you has substance, which of AI's answers is worth taking, and where an argument you're reading skips a step.

Agency is being willing to act on what you found: once you know what's right, actually doing it yourself; once you know what's wrong, actually refusing, and not getting talked back into it.

After reading that McKinsey survey, Liu Run reduced both to two words: taste, and accountability. Taste lets you pick, out of a hundred answers that all look fine, the one that actually fits the moment, which is close to discernment. Accountability means that once you've picked, you're willing to sign your name to the result, which is the most easily overlooked half of agency. AI can produce a decision for you, but it can't own the decision, because accountability structurally requires a person with a name, a position, and something real to lose if the judgment was wrong.

You need both for either to count: agency without discernment tends toward charging at things blindly, and discernment without agency is just talk however clearly you've thought it out.

Both will keep getting challenged by AI. Discernment gets pushed upward: whatever you can judge today, AI will be able to judge in a few years. Agency gets stuck in round after round of analysis, because AI can always give you one more.

How to train it

Discernment is hard to learn from lectures alone. The traditional advice has two parts, expose yourself heavily to good work and talk to people who already have it, and both quietly assume a kind of access: that you already know who "the best" are and you're already inside that circle. For 99% of readers that amounts to "first become someone who already has discernment," a loop with no way in.

But this era has something new that might get around the barrier: used well, AI can become a stand-in mentor.

Method one: the Socratic conversation. Tell AI, "play Socrates. Don't praise me and don't just contradict me. Only challenge me with questions, and force me to make myself clear."

This suits writing, opinions, and low-stakes decisions. For anything touching your body, the law, security, or a thought that already feels off to you, invert the instruction: tell it to state directly what's wrong, what the risks are, and which kind of real human you should go find. That's what Brooks was missing.

There's another technique, from the writer Wang Lu, called "holding back": don't let AI figure out what you want too quickly. Ask "I have an idea, is it any good?" and AI will mostly say "excellent, fantastic." Ask it differently, "one of my students has an idea and came to me, how should I respond?" and AI can't read your position, so its judgment is much less likely to just follow your existing view. As noted earlier, AI won't give you friction by default. Holding back is putting the friction back in yourself.

Method two: explain it back. You explain something to AI, and let it find the holes in your explanation. Most people have an intuition they can't articulate, and articulating it is itself the key step that turns intuition into discernment.

Method three: trace institutional discernment trails. Ask AI to walk you through the award-winning work or landmark cases in a field, year by year, and discuss why each one won.

All three rest on the same pivot, and there's a mathematical reason for it. A paper published last year by OpenAI and Georgia Tech proves that a language model's error rate when generating an answer itself is at least twice its error rate when judging whether an existing answer is correct. Within the class of problems the paper sets up, that's a structural disadvantage, not something a bigger model fixes. In plain terms: on this particular thing, judging correctness is easier than producing an answer from scratch. Putting AI on the verification side is using that gap in your favor.

Taiwan's newsrooms produced a live pair of examples this July. SET News used AI to generate a typhoon path map with wildly wrong data labels; it was shared 800,000 times and 18,000 people had a laugh at it. In the same window, a reporter at CNA built a similar animation but chose to cross-check five independent sources, including Taiwan's weather bureau, the Japan Meteorological Agency and the US JTWC, and built a "verification agent," then confirmed by hand that the positions weren't absurdly off. His conclusion: verification still has to be rigorous, or you'll be wrong the same way. Same tool, same output type, one side skipped verification and shipped, the other made verification harder, and the difference wasn't the tool, it was whether a person stayed on it through the last step.

But be careful: most people use AI in the wrong direction, as a tool for thinking on their behalf, which atrophies discernment. To train discernment you have to use it as a tool for forcing yourself to think.

There's a deeper question underneath this. What if one day AI's output gets so good you genuinely can't tell whether it made it? Telling things apart seems to be heading that way. Might it eventually become impossible?

I've thought about this myself. Making a beautiful slide deck isn't hard now; AI produces one in minutes, sometimes so polished it makes you suspect it was generated in one click. But there's something AI still can't touch: how fluently a person presents. When the animation on a slide switches at the exact instant they say that line, and every pause and transition lines up across the whole talk, that seamlessness took a lot of time rehearsing against the script. Even if every slide was AI-generated, a person willing to spend that much time fitting themselves to the material is showing you they cared enough, and that they did the work.

Which is to say: even when the content itself can't be traced to an author, whether someone was willing to spend the time integrating and rehearsing it is still not something you can fake.

---

Closing

I can't give you a tidy conclusion. But one thing is getting clearer:

The rationality surplus is this: reasonable results are something AI already mass-produces, and what's genuinely scarce is whether I can still tell which one is real.

Telling has gotten harder because the clue we used to judge by, the process, came uncoupled from the result. A polished report no longer means someone thought it through. A well-reasoned answer no longer means someone did the work.

In this era, the people who can say "I did this myself," "I verified this myself," "I bore the consequences myself" will be holding something other people can't produce. What holds those sentences up is discernment and agency.

And both will keep getting tested.

Every time you accept something you read without question, follow advice without checking, take AI's answer at face value, your discernment accumulates a little less.

And every time you stop to ask "is that true?", "why is that so?", "is there another possibility?", it grows a little.

You, reading this far, willing to spend the time on an argument this long and stop to think, maybe finding a paragraph where you thought "wait, is that right?" Those moments of stopping to doubt are discernment at work.

The writer Tim Ferriss recently gave an example: a 500-page fitness book he wrote over a decade ago, with readers constantly asking for a condensed version. His observation is that almost none of the people demanding the condensed version got results, while the ones who read the whole thing lost tens of pounds. Whether a book can be condensed and whether it works are two different questions.

You read this to the end instead of switching to an AI summary. That choice is itself choosing the process.

That process is something AI can't take.

It's yours.

---

Postscript: what this piece drew on

If you want to keep digging, here's a mix of original research, official sources, journalism, and secondary commentary. I've tried to label which is which.

Original research and official sources

• Jason Lodge, "Generative AI is the e-bike of the mind," February 2025.
• Michael Gerlich, 2025: a 666-person Swiss study on AI reliance and critical thinking (Societies).
• Stockholm University and the University of Hong Kong: school records for 26,000 secondary students in a Chinese county over 30 months, the "AI learning penalty." CEPR working paper, not yet peer reviewed.
• Carolyn Stein: economics research on AlphaFold2's uneven effect on highly-cited versus less-cited researchers (NBER working paper).
• William Thurston, "On Proof and Progress in Mathematics," 1994.
• Kalai, Nachum, Vempala, Zhang (OpenAI × Georgia Tech), Why Language Models Hallucinate.
• Nobel Prize in Chemistry 2024: one half to David Baker, one half jointly to Demis Hassabis and John Jumper.
• DeepMind: AlphaProof + AlphaGeometry 2 reached IMO silver-medal standard in 2024; Gemini Deep Think reached gold-medal standard in 2025.
• AlphaGo vs. Lee Sedol match records, March 2016. Lee Sedol announced his retirement in November 2019.

Surveys

• BetterUp Labs and Stanford's Social Media Lab: the work slop survey, published in Harvard Business Review, September 2025. The $9 million figure is an estimate.
• Atlassian Teamwork Lab, June 2026: the effect of an AI label on "lazy" ratings.
• KPMG: global survey of nearly 50,000 people on concealed AI use.
• McKinsey, 2026 workforce monitor.

Journalism

• The Allan Brooks case: a New York Times feature drawing on 3,000 pages of chat logs, a March 2026 Toronto Life profile, and his complaint against OpenAI. The account here reflects his own statements and the claims in that filing.
• Mohammed AlQuraishi, Anastassis Perrakis, Helen Walden: first-hand reactions to AlphaFold2, from Quanta Magazine's June 2024 piece "How AI Revolutionized Protein Science, but Didn't End It."
• The Meta support-AI account takeovers: 404 Media, June 2026. Affected accounts included the old Obama-era White House account.
• Mitchell Hashimoto on "AI psychosis," via Pragmatic Engineer's reporting on Meta's engineering organization.
• Ford's "greybeards" program, from an interview with engineering VP Charles Poon.
• Shin Jinseo: July 2026, a two-stone handicap three-game match against KataGo, won 2 to 1.
• Teacher Papaya: a computing-education YouTuber based in Kaohsiung.

Commentary and secondary summaries (not primary sources)

• Phil Hardman, "The Cognitive Offloading Paradox," citing Wang and Zhang 2026.
• Psychology Today: the older/younger offloading reading (January 2026) and the "algorithmic self" observation, two separate pieces.
• Wang Lu, on "calibration sources" and "holding back."
• Liu Run: "this makes sense ≠ this is true" and the taste/accountability framing are his reading of the McKinsey report, not the report's own wording.
• Tim Ferriss: his own blog observations about his readers' outcomes, not a tracking study.