Everyone knows the rule in medicine: big diagnosis, get a second opinion. AI needs a stricter rule. Get four opinions. Then come back in three months and do it again.
That isn’t paranoia. It’s how these tools actually behave.
Same Data, Different Answer
I asked an AI a simple question: has black market activity gone up, down, or sideways over the last 50 years?
The first answer said it wasn’t growing. Pressed, it gave a different answer: it had gone up, substantially. Same AI, same data, opposite headlines.
What happened? The first answer measured the black market as a share of the economy. By that measure, an IMF study of 158 countries found the average shadow economy fell from about 34.5% of GDP in 1991 to about 27.8% in 2015. But I asked about activity. The world economy roughly tripled over that span, so a slightly smaller slice of a much bigger pie means more money underground, not less.
Neither answer was a lie. The first one answered a question I didn’t ask, and led with it confidently. That’s the trap.
The Numbers Underneath Are Shakier Than They Sound
Dig further and it gets worse. Those shadow economy figures are modeled, not counted, and largely from one research team. Methods disagree wildly: for South Korea in 2015, the IMF study estimated nearly 20% of GDP, while Korea’s own public finance institute put it at 8%.
An AI will hand you either number in a clean sentence. It won’t always tell you the other one exists.
The Four-Opinion Method
First opinion: ask one AI. Note the answer and how confident it sounds.
Second opinion: ask again, differently. New chat, reworded question. Ask it to define its terms. If the answer flips, you’ve learned the question was ambiguous, or the AI was.
Third opinion: a different AI. Different company, different training. Agreement here means more, though not as much as you’d think: most AIs learned from the same internet, so they can share the same blind spots.
Fourth opinion: outside the machine. A primary source, a professional, or someone with direct experience. This is the only truly independent check.
If all four agree, you have something to act on. If they split, the honest answer is “unknown,” and you decide accordingly.
Then Put an Expiration Date on It
Here’s what makes AI different from a doctor’s second opinion: the ground underneath changes fast.
Take cybersecurity. CrowdStrike’s 2026 threat report found AI-enabled attacks up 89%. Check Point reports that AI has moved from assisting attackers to actually running attacks, and that the window between a security flaw going public and being exploited has shrunk from days to hours. Advice that was right in spring can be stale by fall.
So whatever you conclude, date it. Redo the check in 90 days. If the answer still holds, your confidence goes up. If it changed, you caught it before it cost you.
Watch Who’s Talking
One more filter. Many of those cybersecurity numbers come from companies that sell security. They profit when you’re worried. That doesn’t make them wrong, but it means you trust the direction and discount the decimals.
The same goes for any AI company telling you its tool is reliable, any vendor quoting a study, and any government report. Everyone has a stake. Your job is to find where sources with different stakes land on the same answer.
The Bottom Line
AI is fast, cheap, and getting better. It’s also confidently wrong often enough to cost you. Red team your own decisions the way companies red team their systems: try to break the answer before you act on it. Treat a single AI answer as a rumor. Four that agree is a lead. And every lead has an expiration date.
Sources
Medina & Schneider, IMF Working Paper: Shadow Economies Around the World (2018)
https://www.imf.org/en/Publications/WP/Issues/2018/01/25/Shadow-Economies-Around-the-World-What-Did-We-Learn-Over-the-Last-20-Years-45583
Korea Times: IMF study and Korea’s own estimate
https://www.koreatimes.co.kr/amp/economy/20180212/ratio-of-shadow-economy-to-gdp-falls-below-20
StationX summary of CrowdStrike 2026 Global Threat Report
https://app.stationx.net/articles/ai-cybersecurity-threats
Check Point AI Security Report 2026
https://devopsdigest.com/node/14566