The wrong question, beautifully answered

Ask a model to make the onboarding shorter and it will make the onboarding shorter. Cut three screens to one, rewrite the copy, A/B the button, hand you a version that converts four points higher, all of it without once asking the only question that mattered, which was whether anyone should have to onboard at all. That question was never on the table. You put "shorten this" on the table. The machine answered the thing in front of it, flawlessly, and the flawlessness is the problem, because a four-point lift on a flow that shouldn't exist is still a flow that shouldn't exist, now with better numbers to defend it.

This is not the failure everyone worries about. Everyone worries the model will be wrong: hallucinate a case, invent a statistic, cite a paper that was never written. Fair worry. Wrong worry. The expensive failure is the model being completely right about a question you should never have asked, and so right about it that nobody goes back to check the question.

This one predates the machine by seventy years. In 1957 a statistician at Oak Ridge, Allyn Kimball, needed a word for a mistake that Type I and Type II errors didn't cover. Type I is a false alarm. Type II is a miss. Kimball's was neither. He called it the error of the third kind, and defined it as "the error committed by giving the right answer to the wrong problem." He set it deliberately beside the other two, because getting the answer right and getting the question right are different skills, and being good at the first buys you nothing on the second. Run the statistics perfectly on a question that should have been thrown out and the perfection is real. It is also worth nothing.

A model is a Type III engine by construction. Not a bug someone patches next release, by construction. It is trained to maximise the quality of its answer to the prompt it was handed, and the prompt arrives from outside, treated as fixed, the one thing not up for evaluation. So the single failure it cannot catch is the one where the prompt itself was wrong, because catching that would mean standing outside the task it was set. Ask it whether you asked the right thing and it will gamely tell you, in the same confident register it uses for everything, which is the register it would use if you had.

Sixty years of "define your problem first" and none of it is new. The old worry assumed you could still see the garbage. A badly-framed question used to come back as a bad answer, and the bad was visible, it had a texture, you could feel the friction of it. A confusing, hedged, poorly-organised answer carried its own warning label. You reread it, you frowned, you went back. The machine strips the label off. It produces the well-ordered, confident version of the wrong answer, and it turns out humans read fluency itself as a proxy for truth.

In 1999 Rolf Reber and Norbert Schwarz showed people the same factual statements, some in high colour contrast so they were easy to read, some in low contrast so they were hard. Identical claims. The only thing that changed was how easily the sentence went down. People rated the easy-to-read ones more likely to be true. Not more pleasant. More true. We take "went down smooth" as evidence of "is correct," and the machine makes went-down-smooth on demand, cleanly separated from correct. So a fluent answer to the wrong question does worse than fail to warn you. It manufactures the feeling of having gotten it right, and that feeling is the one that closes the question.

Then fluency does the second job, the one that really costs you. A clunky answer invites a second look. You reread it, you push back, you catch the seam where the logic slipped. A frictionless one gets accepted and filed. Which inverts the whole safeguard: the better-executed the answer, the fewer people reopen the question behind it, so the response most likely to get built on is the one least likely to have had its premise checked. You save your suspicion for the answers that stumble, and the smooth ones walk straight past the guard.

I nearly kept a set of these myself. I had asked an AI to raise the register on an essay, and it read that as "be clever" and handed back a run of polished, quotable little epigrams. They went down smooth, so my first instinct was to keep them. The piece did not need clever. It needed plain, and the polish was the exact thing hiding that the machine had answered the wrong question. A few of the lines were phrases already on my own do-not-use list. That is how good smooth is at getting past you.

A colder version of this ran for real, at scale, with a body count. Through the 1960s the American command in Vietnam needed a number to prove the war was being won, and the number they reached for was enemy dead. Easy to count. Countable every day. So the count became the question. Daniel Yankelovich later named the slide: measure what is easy, ignore what can't be measured, assume the unmeasured doesn't matter, then assume it doesn't exist. Every day the body count got answered precisely, with real rigour applied to real figures, and every day it was the wrong question, because a war is not won by the ratio of corpses. The precision was exactly what kept anyone from reopening the question under it. A clean number is the most persuasive wrong question there is. The machine hands you clean numbers all day.

None of this gets fixed by "prompt better." "Just ask good questions" is the reasonable objection, and it is not enough, because you cannot fully audit your own question from inside the same head that framed it, for the same reason the model can't: the framing is invisible to the thing doing the framing. What you can do is refuse to let the answer's quality stand in for the question's. A small procedure, then. Run it before you accept any fluent answer, precisely because fluency is what tempts you to skip it.

Say out loud the exact question the answer is answering. Not the topic. The question, in one sentence. Half the time the model quietly narrowed it, answered something adjacent and easier, and the polish covered the swap. Then ask the thing it structurally cannot ask for you, because it needs your goals and your goals are not in the prompt: if this answer were perfect, what would it get me? If the honest reply is "a better version of something that shouldn't exist," the answer's quality is irrelevant. You are done.

Then name the question you didn't ask. The should-we-at-all, the what-happens-if-this-works, the who-is-this-even-for. That is where the wrong-problem error hides. Then flip your instinct on the smooth ones: the more effortless the answer, the harder you re-audit the question, not the softer, because smoothness is the cue that's lying to you. And do not ask the model to improve the answer. It will do that forever, and each pass buries the wrong question deeper. Ask it to attack the question instead: what would make this the wrong thing to ask, and what should I ask instead. It cannot hand you your right question. It can list framings faster than you could alone, and which one is right stays yours to decide, which was the only part that was ever the point.

One count tells you where you stand. Of the last five things you had a machine help you build or decide, how many times did you interrogate the question, and how many times did you just grade the answer? If you graded five answers and audited zero questions, the trouble was never answer quality. The answers were fine. The answers are always fine now. You have a question problem, and the machine cannot help you with it, because the machine is the thing making it invisible.