The Vendor BS Detector

What this is for

You're evaluating an AI tool or vendor for your team and the sales deck looks great, which tells you nothing. This is for the meeting itself — the questions that separate a vendor who's built something real from one reselling someone else's model with a markup and a slide template.

The 9 questions

1. Where does our data go, and is it used for training? Good answer: names the specific model provider, states in writing whether your data trains their model (it shouldn't, by default), and can show you the data processing agreement on request. Bad answer: "It's all secure and private" with no specifics, or hesitation when you ask for it in writing.

2. What happens on your worst-case error? Good answer: a real story — a specific failure, what it cost, what changed afterward. Every vendor has one. Bad answer: "We haven't really had issues" or "Our error rate is extremely low." No tool with real usage has zero failures.

3. Is this a real model or a wrapper on someone else's? Good answer: straightforward — "We use OpenAI's/Anthropic's API and our value is in [specific thing: workflow, data, interface]." Being a wrapper isn't automatically bad if the wrapper adds real value. Bad answer: vague deflection, or claiming proprietary AI they clearly didn't build, with no technical detail when pressed.

4. What's the actual accuracy on our data, not your demo data? Good answer: offers a pilot on your real, messy data before you sign anything, and is upfront that accuracy on your data may differ from the demo. Bad answer: only shows you their curated demo and resists letting you test with your own inputs.

5. Who owns the outputs? Good answer: clear, in writing — you own what the tool produces for you, no ambiguous shared-rights language. Bad answer: "We'd have to check the terms" or ownership language buried in a EULA nobody read.

6. What does month 13 cost? Good answer: gives you the renewal price now, including likely increases, and explains what triggers a price change. Bad answer: "Let's cross that bridge later" or a year-one price with no mention of what happens after.

7. What's the exit and data-export path? Good answer: a documented process to export your data in a usable format and a stated timeline for deleting it from their systems after you leave. Bad answer: no clear answer, or export only in a proprietary format that locks you into rebuilding elsewhere.

8. What does implementation actually require from my team? Good answer: a specific list — hours, roles, systems to connect, a realistic timeline with a named point of failure if it slips. Bad answer: "It's plug and play" with no detail. Nothing enterprise-relevant is truly plug and play.

9. Who else in my industry is live on this, and can I talk to them? Good answer: a live reference you can actually call, not just a logo on a slide. Bad answer: logos with no offer to connect you, or references who turn out to be a two-week pilot, not a real deployment.

Run the demo yourself, with your own data

Before the contract, not after: bring your own messy input — a real client email, a real spreadsheet with your actual formatting problems, a real transcript with crosstalk. Ask the vendor to run it live, unscripted, in the room. Watch what breaks. A vendor who resists this, or insists on using only their prepared examples, is telling you the tool doesn't hold up outside the demo script.

Where this goes wrong

• People accept "we take security seriously" as an answer to question 1. It isn't an answer — it's a sentence with no content. Ask for the specific data policy in writing.
• People skip the reference call in question 9 because it feels like extra work. It's the single fastest way to find out if month 13 support looks like month 1 support.
• People let the vendor pick the demo data. If you don't bring your own, you're evaluating their best case, not your real one.

The 2-minute version

Before your next AI vendor call, pick three questions from this list that matter most to your risk (usually 1, 4, and 7) and ask them directly. Bring your own messy data to the demo. If a vendor won't let you test with it, that's the answer.

---

Where you stand with AI, in 10 minutes. The free AI Readiness Quiz gives you a personalized roadmap instead of another tool list: genxcelerate.com

Weekly, on Tuesdays. The Gen Xcelerator — practical AI intelligence for people who already know how to do the job. Subscribe free

Experience Is the API. — GenXcelerate