“Challenge my thinking” sounds like a strong prompt.
It often produces a weak result: three polite caveats, a compliment about your careful reasoning, and the same recommendation you started with.
The problem is structural. The model has already seen your framing, your preferred answer, and often the argument you want it to criticize. Asking it to “be critical” does not remove those signals.
Useful disagreement comes from changing the information and sequence available to the model.
The five patterns below are practical ways to do that. None turns a chatbot into an independent expert. Each makes one failure easier to see.
Pattern 1: blind the preference
Preference cues invite agreement. Remove them.
Weak prompt:
I want to move our team from Tool A to Tool B.
Help me stress-test the plan.
The model knows which conclusion would be socially helpful.
Stronger prompt:
A ten-person team is comparing two options.
Option A
- [current facts]
- [costs]
- [constraints]
Option B
- [current facts]
- [costs]
- [constraints]
Do not recommend an option.
For each one, identify:
1. the strongest reason it could fail,
2. the missing evidence that matters most,
3. the condition under which it becomes the better option.
Do not write “our current system” or “the modern replacement.” Even small labels carry a verdict.
When this helps
- vendor choices;
- hiring versus outsourcing;
- build versus buy;
- keeping versus replacing a process;
- or any choice where you already have a favorite.
What it does not solve
You selected the facts. A blind comparison can still be biased by what you omitted.
Pattern 2: separate advocate and critic
A single response tends to compromise: one argument, one caveat, a balanced conclusion.
Use separate passes with incompatible jobs.
Pass A:
Build the strongest evidence-based case for Option A.
Do not discuss Option B.
State every assumption and identify where evidence is missing.
Pass B, in a new chat:
Build the strongest evidence-based case for Option B.
Do not discuss Option A.
State every assumption and identify where evidence is missing.
Pass C:
Compare these two arguments.
Do not reward eloquence or length.
List:
- claims supported by supplied facts,
- claims requiring external verification,
- assumptions that conflict,
- and one experiment that would reduce the largest uncertainty.
[paste both arguments]
Separate chats reduce conversational anchoring. They do not create genuine independence—the passes may still use similar training patterns—but they stop the second role from merely reacting to the first role’s wording.
Pattern 3: require evidence before argument
If you request a persuasive case first, the model may generate claims that fit the case and attach support afterwards.
Reverse the order:
Before forming an opinion, create an evidence table.
For each decision criterion, record:
- supplied facts,
- primary sources needed,
- estimates and their assumptions,
- unknowns,
- and evidence that would falsify each option.
Do not recommend anything until I verify and return the evidence table.
After verification, provide only the corrected table and ask for analysis.
This is slower than asking for an instant recommendation. That is a feature when the answer matters.
A useful stop rule
If the largest uncertainty could reverse the decision, do not ask for a final recommendation. Design a test, collect a measurement, or consult the person who owns the missing information.
Pattern 4: appoint a hostile reviewer with a narrow target
“Be a devil’s advocate” is vague. Give the reviewer a concrete failure surface.
Examples:
Review this plan only for operational failure.
Assume the strategy is reasonable.
Find where ownership, monitoring, rollback, or maintenance is missing.
Review this proposal only from the affected employee's perspective.
Identify which claimed benefits are benefits to management rather than to the employee.
Flag promises the company cannot verify or keep.
Review this data flow only for privacy exposure.
List each data class, destination, retention point, and person with access.
Do not provide legal conclusions.
A narrow hostile review is more useful than theatrical aggression. “Brutally honest” changes tone; a defined review surface changes coverage.
Use several narrow reviews when the stakes justify it. Keep their findings separate until you decide which risks are real.
Pattern 5: use an independent second pass
Run the same neutral brief through:
- a new conversation with no history;
- a second model family;
- or a human reviewer who has not seen the preferred answer.
Then compare disagreements:
Two reviewers analyzed the same decision.
Reviewer 1:
[paste]
Reviewer 2:
[paste]
Create a disagreement register:
- issue,
- each reviewer's claim,
- evidence each relies on,
- missing evidence,
- and the owner who can resolve it.
Do not choose a winner based on confidence or writing quality.
Agreement between two models is not confirmation. Models can share sources, conventions, and blind spots. The useful output is the disagreement register and the evidence it tells you to collect.
Worked example: automating invoice handling
Suppose the initial plan is:
We should use an AI agent to read supplier invoices, enter them into accounting software, and route them for payment.
A generic challenge prompt may return predictable caveats about accuracy and human oversight.
Use the five patterns instead.
1. Neutralize the options
- Option A: keep manual entry and improve the checklist.
- Option B: use deterministic extraction plus human review.
- Option C: use an agent that extracts, enters, and routes.
- Option D: delay the project and first standardize supplier formats.
2. Separate advocates
Each option gets its own strongest case and explicit assumptions.
3. Build the evidence table
Collect:
- monthly invoice volume;
- current handling time;
- correction rate;
- percentage of standard formats;
- value and frequency of duplicate or fraudulent invoices;
- integration permissions;
- and who owns an exception.
4. Run narrow hostile reviews
- finance control;
- operational recovery;
- supplier privacy;
- and employee workflow.
5. Compare independent passes
One reviewer may focus on labor savings. Another may notice that the current process lacks a reliable purchase-order match, making automation premature.
The best result may not be a stronger case for or against AI. It may be a two-week measurement that makes the decision obvious.
Three ways disagreement becomes fake
The model invents objections
Not every counterargument is plausible. Require the objection to identify the assumption or evidence it challenges.
The model optimizes for symmetry
Some choices are not balanced. One option may have much stronger evidence. Do not demand an equal number of pros and cons; demand material issues.
The model becomes confidently contrarian
A prompt can make the model oppose everything. Contrarian output is still generated output. It needs the same verification as agreeable output.
Research has documented sycophancy across AI assistants and shown how preference signals can influence model responses (Anthropic’s sycophancy research). The defense is not a magic phrase. It is a process that reduces preference cues and makes evidence visible.
A copy-ready disagreement sequence
Use this for an important decision:
Phase 1 — frame
Restate the problem in three materially different ways.
Do not solve it.
Phase 2 — evidence
Separate supplied facts, verifiable facts, estimates, inferences, and values.
List the evidence that could reverse the decision.
Phase 3 — cases
Build the strongest case for each option separately.
State assumptions and failure conditions.
Phase 4 — review
Run narrow reviews for operations, affected people, security/privacy, and cost.
Phase 5 — decision support
Create a disagreement register and propose the smallest real-world test.
Do not make the final decision.
For the failure modes behind this sequence, read the four ways an AI thinking partner can distort a decision. For the broader workflow, see AI for better decisions.
The honest limit
You can make an AI produce disagreement. You cannot make it know that you are wrong.
The model may generate the same flawed assumption from a different angle. It may cite an outdated source. It may sound independent because you changed the role label.
Treat disagreement as a search tool: it surfaces assumptions, missing stakeholders, and evidence to collect. The final judgement still belongs to the person who understands the context and accepts the consequences.
