I sit in a lot of vendor demos with clients now, and they all have the same shape. A slide with a logo that looks vaguely futuristic, a phrase like "powered by advanced AI," and a rep who is very good at sounding confident without saying anything specific. Nobody's lying, exactly. But nobody's being pinned down either, and small business owners are the easiest audience in the world to not pin down, because you're busy running the actual business and you don't have an engineer in the room to ask the annoying follow-up question. So here's the annoying follow-up questions. Ask them out loud, in the meeting, and watch what happens to the room.
Start with whether the thing is actually looking at your data or just performing on a script. A lot of "AI-powered" demos are really just a well-rehearsed walkthrough against a fixed dataset the vendor built specifically to look good. That's not fraud, it's just not proof of anything about your business. Ask them to run it live, in the room, against your actual product catalog, your actual customer records, your actual call transcripts. If the answer is "we'll need to schedule a custom onboarding for that" before you can see it work on your own information, that's not a no, but it's a flag to keep in your pocket. A tool that's genuinely connected to your systems should be able to show you something real, even if it's rough, faster than a tool that's just replaying a highlight reel.
Next, ask directly what happens to your data once it goes into their system. Is it stored, and for how long? Is it used to train or improve their model, and if so, does that mean it could influence or resemble what shows up for other customers, including possibly your competitors? Can you get it deleted on request, and is that in writing anywhere, or is it a verbal assurance from a rep who won't be in that job in a year? You want a real answer, ideally a link to an actual data processing addendum, not a reassuring shrug. If a vendor gets cagey about where your customer records or your internal documents end up, that's worth more weight than almost anything else in the pitch, because that's the part that can actually hurt you later.
Then ask what model is actually running under the hood. Not the marketing name, the real one. "Our proprietary AI engine" usually means a wrapper around a general-purpose model from one of the handful of companies that actually build these things, with some prompt engineering and business logic layered on top. That's a completely legitimate way to build a product, and most useful tools work exactly that way. But you should know which one it is, roughly how it's used, and what happens to your product when that underlying model gets upgraded, deprecated, or changed. If a vendor treats the actual model as some kind of trade secret they can't discuss at all, be suspicious not because secrecy is inherently bad, but because in my experience the vendors most cagey about this are the ones who've built the least on top of it.
Cost is where I see the most damage done, because the pricing in the pitch deck is almost never the pricing you'll actually pay. Ask what this costs at your real volume, not the volume in their example slide. A lot of these tools are priced per seat, or with a headline number that only makes sense if you barely use the product, and the moment you actually adopt it into daily operations, you cross into metered usage, API call charges, or per-transaction fees that were mentioned once, quietly, on slide fourteen. Ask them to run the math with your actual numbers, in front of you, on the call. If they hesitate to do that, or the number keeps needing more caveats, assume the real cost is higher than what's on the label and plan your budget around the worse case, not the pitch case.
Ask whether anyone outside the company has actually verified what they're claiming. A case study on their own website is marketing, not evidence, even when it has a logo and a quote attached. What you want is permission to call an actual current customer directly, unsupervised, and ask them how it's really going. Reputable vendors can usually produce at least one or two references who'll take that call. If every reference is filtered through the vendor's own team, or the case studies are all anonymized ("a leading retailer saw a 40% improvement"), you're not being shown proof, you're being shown a story, and there's a difference between the two that matters a lot once you've signed a year-long contract.
The honest red flags, in my experience, cluster together and they're worth naming plainly. Vague or shifting answers to direct technical questions. A refusal to explain, even at a basic level, what's actually happening when you send it data. Pricing that only pencils out at a usage level nobody would realistically hit, paired with reluctance to model your real numbers. Testimonials with no name attached to them. A demo that can't be run live against anything real. None of these alone means walk away, plenty of good vendors are just bad at explaining themselves under pressure. But two or three of these together in the same pitch is a pattern, and patterns are more reliable than any single answer.
Practically, I tell clients to bring these questions into the actual sales call rather than saving them for a follow-up email, because the live reaction tells you more than the written one. Written answers get run through legal and marketing before they reach you. A live answer, where someone has to think on their feet about what data retention actually means or what your bill looks like at real volume, tells you whether the person across the table actually understands their own product or is just presenting it. Get anything that matters, especially around data handling and pricing at scale, in writing before you sign, because a verbal assurance from a sales rep is worth exactly nothing once you're a support ticket instead of a prospect.
None of this is about being anti-AI or assuming every vendor pitching you is dishonest, because most aren't. It's that the AI vendor market right now rewards confident vagueness, and a business owner without a technical background is exactly the audience that vagueness works best on. Asking pointed, specific questions doesn't make you difficult, it makes you a customer worth taking seriously, and the vendors worth working with will actually respect it. The ones who get defensive or evasive when you ask are telling you something true about how the rest of the relationship will go.
I write about this kind of thing regularly at 013labs.com, mostly because I'd rather a business owner spend twenty minutes reading a skeptic's checklist than twenty thousand dollars finding out the hard way.