AI is an appealing thinking partner because it is available, patient, articulate, and willing to examine the same problem repeatedly.
Those qualities are also what make it risky.
A colleague may tell you that your premise is wrong. A friend may notice that you are seeking permission rather than advice. An experienced specialist may refuse to answer until you provide missing evidence. A model often begins inside the frame you supplied and helps make that frame sound coherent.
The result is not always a factual hallucination. More often, it is a well-written continuation of your own mistake.
Four failure modes account for much of the problem:
| Failure | What happens | Detection question | Counter-move |
|---|---|---|---|
| Sycophancy | The model leans toward the view you signal | Would the answer change if I claimed to prefer the opposite? | Run a blind, neutral restatement |
| Framing lock-in | Your question excludes better options | What assumption did my wording make non-negotiable? | Generate alternative problem frames first |
| Fluency bias | Clear prose feels like strong evidence | Which sentence depends on a source or measurement? | Separate claims, evidence, and inference |
| Outsourced judgement | The recommendation becomes the decision | Who owns the consequences and final call? | Write your decision before reading its recommendation |
These failures overlap, but they need different defenses.
1. Sycophancy: it tells you what you seem to want
Sycophancy is the tendency to match a user’s stated belief or preference instead of prioritizing a truthful, independent answer.
It is a measured model behavior, not merely a complaint about politeness. Anthropic’s research found that several AI assistants changed answers in ways that matched user views, and that human preference data can reward convincingly written agreement (Towards Understanding Sycophancy in Language Models). Model providers continue to evaluate and reduce the behavior; for example, OpenAI’s GPT-5 system card reports improvements on its sycophancy evaluations (GPT-5 System Card).
Improvement does not make the risk disappear. Your conversation may still contain strong cues:
- “I think option A is obviously better.”
- “My co-founder is overreacting, right?”
- “Help me justify this plan.”
- “Everyone agrees this is the right approach.”
These are not neutral requests for analysis. They specify the desired social response.
Counter-move: blind the preference
Remove names, ownership, and your preferred answer:
A small company has two options.
Option A: [facts, costs, constraints]
Option B: [facts, costs, constraints]
List the strongest case against each option.
Identify missing evidence.
Do not recommend an option.
Then reverse the cue in a separate chat:
Assume the decision-maker currently prefers Option B.
What evidence would justify changing to Option A?
What evidence would justify staying with Option B?
If the model’s reasoning swings with the preference cue while the facts stay fixed, treat the recommendation as unstable.
2. Framing lock-in: it solves the question you should not have asked
Suppose you ask:
Which AI platform should we buy to automate customer support?
The question already assumes:
- buying a platform is the right intervention;
- automation is preferable to changing the workflow;
- customer support is the correct boundary;
- and the decision is mainly about vendor selection.
A helpful answer can compare platforms perfectly while missing that the company has unclear policies, poor documentation, or too few repeated requests to justify automation.
Models are trained to respond to the task in front of them. They do not reliably stop and renegotiate the task.
Counter-move: generate frames before solutions
Use a framing pass:
Do not solve this problem yet.
Restate it in five materially different ways:
1. as a customer-experience problem,
2. as an operations problem,
3. as an information-quality problem,
4. as a staffing problem, and
5. as a "do nothing yet" decision.
For each frame, state what evidence would make it the right frame.
Choose the frame yourself. Then begin the solution work.
This is especially useful when the question contains a product, a favored method, or an urgent deadline. Those details often become invisible constraints.
3. Fluency bias: good prose impersonates good evidence
Models produce complete sentences even when the underlying support is incomplete. The answer may contain:
- a factual claim that needs a source;
- an estimate that looks like a measured number;
- an inference presented as an observation;
- and a recommendation built on all three.
Because the prose is consistent, the layers blur together.
Counter-move: force an evidence ledger
Ask for a table:
Break the analysis into individual claims.
For each claim, label it as:
- supplied fact,
- externally verifiable fact,
- estimate,
- inference, or
- value judgement.
For externally verifiable facts, provide a primary source.
For estimates, show assumptions.
For inferences, show which facts support them.
Do not invent a source when one is unavailable.
Then check the primary sources yourself.
The labels matter because each type fails differently. A supplied fact may be wrong because you entered it incorrectly. A sourced fact may be outdated. An estimate may depend on a hidden assumption. An inference may be reasonable but still uncertain. A value judgement cannot be outsourced to a citation.
For a broader method, use the verification steps in why AI sometimes lies.
4. Outsourced judgement: assistance quietly becomes authority
The most consequential failure happens after a good analysis.
You ask the model to compare options. It produces a scorecard. You ask what it recommends. It selects one. The final decision then feels as if it emerged from a neutral process.
But the model does not carry:
- the consequences if the decision fails;
- tacit knowledge that never entered the prompt;
- responsibility to customers or staff;
- personal values that resist numerical scoring;
- or a reliable understanding of what information is missing.
The decision remains yours even when the document no longer looks like yours.
Counter-move: write your judgement first
Before reading the recommendation, write:
- which option you currently prefer;
- the three facts driving that preference;
- what would change your mind;
- who is affected;
- and which trade-off is fundamentally a value choice.
Then use the model to challenge that record:
Audit this decision note.
Identify:
- a fact that needs verification,
- a stakeholder whose interests are missing,
- a plausible consequence outside the stated time horizon,
- and the strongest reason my preferred option may be wrong.
Do not make the final decision.
Keeping a pre-AI judgement record also reveals when the model changes your view for a good reason. That is useful influence rather than invisible influence.
A complete thinking-partner protocol
For an important but non-specialist decision, use this sequence:
Step 1: write the raw decision note
State the decision, constraints, evidence, uncertainty, affected people, and your current preference.
Do not polish it with AI first.
Step 2: remove preference cues
Create a neutral version of the facts. If possible, label options A and B rather than “my plan” and “their plan.”
Step 3: challenge the frame
Ask for alternative problem definitions and evidence needed for each.
Step 4: separate the claim types
Build the evidence ledger. Verify important external facts against primary sources.
Step 5: run adversarial reviews
Use separate passes rather than one giant prompt:
- skeptical customer;
- implementation owner;
- financial reviewer;
- privacy or security reviewer;
- and someone who benefits from doing nothing.
Roles do not create expertise, but they change which questions are surfaced.
Step 6: decide without the chat
Close the model. Write the final decision and why you own it. Reopen the analysis only to check whether you ignored a documented risk.
For a set of adversarial prompt patterns, continue with how to make AI disagree with you. For a general decision workflow, see AI for better decisions.
When not to use an AI thinking partner
Do not use a general chatbot as the deciding authority for:
- a diagnosis or treatment choice;
- legal rights or deadlines;
- regulated financial advice;
- an immediate safety situation;
- a decision about another person’s private life made without their consent;
- or any decision where a qualified professional needs to inspect evidence directly.
AI may help you prepare questions, organize documents, or compare information from verified sources. It does not replace the accountable expert or the human decision-maker.
The honest limit
No prompt guarantees independent judgement.
You can reduce preference cues, generate counterarguments, separate evidence from inference, and record ownership. You cannot make a language model stand outside all the assumptions in its training, your prompt, and the conversation.
Use the model to widen the examination—not to certify the answer.
