In a June 2025 press release, Gartner published an estimate that should reframe how every small business shops for AI help: of the thousands of vendors marketing “agentic AI,” only about 130 actually have agentic capabilities. The rest are doing what Gartner calls “agent washing”: rebranding scripted chatbots, RPA scripts, and rule-based automation as autonomous AI. The same release predicted that more than 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value, and inadequate risk controls.
We run an AI automation agency, so that estimate is about our industry, and pretending otherwise would make this guide useless. The honest conclusion the data forces is this: shopping for AI help is not comparison-shopping between polished websites. It is interrogation. The supply side of this market grew faster than any accountability mechanism, so the accountability mechanism has to be you.
This guide is the interrogation script: 12 questions to ask any AI automation agency (including us) with what a good answer sounds like versus a bad one, a red-flag list you can run in 30 seconds, a 10-minute scorecard, and an honest section on when you should not hire an agency at all. At the end, we answer all 12 questions ourselves, because a buyer’s guide written by a vendor is worthless unless the vendor takes its own test.
Why Is It So Hard to Tell a Real AI Agency From a Fake One?
Because the supply side exploded faster than the evidence. Through 2025, seemingly every marketing shop, web studio, and solo consultant relabeled itself an “AI agency,” and modern AI tooling makes a five-minute demo cheap enough that anyone can look credible on a sales call. Gartner’s June 2025 estimate, roughly 130 genuinely agentic vendors among thousands claiming the label, means the default vendor you meet is statistically not the real thing.
The project data shows what happens after the contract is signed. MIT NANDA’s 2025 “GenAI Divide” report, which analyzed 300 enterprise deployments, found that 95% of generative-AI pilots produced no measurable P&L impact, and that the 5% that succeeded embedded AI deep inside real workflows instead of bolting on generic tools. A 2024 RAND Corporation study, built on interviews with 65 data scientists and engineers, found that more than 80% of AI projects fail (twice the failure rate of ordinary IT projects) and that the root causes are organizational, not technical: misaligned objectives, underestimated data work, and chasing technology instead of business outcomes.
Read RAND’s list again, because it is secretly a list of vendor behaviors. Misaligned objectives happen when an agency proposes a tool before understanding your workflow. Underestimated data work happens when nobody audits your systems before quoting. Technology-chasing happens when the pitch leads with the model instead of your missed calls. An agency that skips diagnosis is not just being sloppy. It is reproducing the documented failure pattern at your expense.
The stakes cut both ways, which is why “just wait a few years” is also bad advice. In Salesforce’s Small & Medium Business Trends Report (December 2024, 3,350 SMB leaders surveyed), 55% of SMBs reported using AI (up from 39% a year earlier) and 91% of SMBs using AI said it boosts revenue. The technology works when the implementation is honest. Your problem is sorting the roughly 130 from the thousands, and the 12 questions below are the sorting mechanism.
Four Definitions Before You Shop
Agent washing. Gartner’s term for marketing ordinary software (scripted chatbots, workflow templates, robotic process automation) as autonomous “AI agents.” The software may still be useful; the deception is in the price and the promise.
Agentic AI. Software that can plan multi-step actions and operate tools (calendars, CRMs, phone systems) to complete a goal without a human scripting every branch. A true AI agent decides what to do next; a chatbot recites what it was told.
AI audit. A paid diagnostic performed before anything is built: workflow inventory, call and lead baseline, systems and data check, prioritized plan. Aries Consulting Group’s 2026 pricing guide puts a properly scoped SMB version at $2,000-$8,000, and notes that “free assessments” are typically sales funnels.
Workflow automation. Deterministic plumbing (if a form is submitted, create the CRM record and send the confirmation) built in tools like n8n, Make, or Zapier. Most good “AI” deployments are mostly this, with a language model placed only at the decision points that need judgment. That is not a scandal; it is good engineering.
What Should You Ask an AI Automation Agency Before You Sign?
Ask the 12 questions below, in order, on a discovery call. They take about 45 minutes. Grade the shape of each answer, not just its content: real operators answer with specifics (named clients, named staff, numbers with dates, admitted failures), while agent-washers answer with adjectives. An agency that gets defensive by question four just saved you months of discovery.
Every question comes with what a good answer sounds like and what a bad one sounds like. The bad answers are not caricatures: each is something we have heard a competitor say, or heard from a client describing the agency they hired before us.
Question 1: “Can I Call a Live System You Built, Right Now?”
Demos prove almost nothing; production proves almost everything. MIT NANDA’s 95% figure describes exactly this gap: pilots that dazzled in a controlled environment and then produced no P&L impact in the real world. A system that has survived months of real callers, real edge cases, and a real integration is the single strongest piece of evidence an agency can offer, and it costs them nothing to show you if it exists.
A good answer hands you a phone number during the meeting. “Here is the after-hours line of a business we serve. Call it now, and then I’ll connect you with the owner as a reference.” Live systems, named clients (with permission), and numbers with dates attached.
A bad answer offers a demo environment instead. “We can set up a sandbox for you” is fine as an addition and disqualifying as a substitute. So is an NDA excuse that conveniently covers every single client, or case studies with no names, no dates, and suspiciously round numbers.
Question 2: “What’s Your Diagnosis Process Before You Propose Anything?”
RAND’s 2024 root causes (misaligned objectives and underestimated data work) are both diagnosis failures, which is why this question predicts project outcomes better than any technology question. An agency that quotes before measuring is guessing with your money. The market rate for a real SMB diagnostic is $2,000-$8,000 per Aries Consulting Group’s 2026 guide, and the deliverable should be worth keeping even if you never hire the auditor.
A good answer describes a structured audit with a deliverable you own. A workflow inventory, a baseline of your call and lead numbers, a systems and data check, and a prioritized plan with expected payback per item (the structure our own AI audit follows). The agency should be willing to sell the audit alone, with no build attached.
A bad answer proposes a tool on the first call. If someone recommends a voice agent before asking your call volume, your booking rate, or which CRM you run, the “diagnosis” is a formality on the way to a predetermined invoice. “We’ll figure out the details during onboarding” means after you have signed.
Question 3: “What Exactly Do I Own If We Part Ways: Accounts, Prompts, Workflows, Data?”
Exit ownership is where agent-washing hardens into hostage-taking. If the phone number, the platform accounts, the prompts, and the workflows all live inside the agency’s master account, you do not own an automation. You rent one, and the rent goes up the day leaving would cost you your phone line. Ask this before you ask about price, because a cheap monthly with no portability is the most expensive option on the table.
A good answer lists transferable artifacts, in writing. Platform accounts opened in your name or contractually transferable, phone numbers ported to you, prompt documents and workflow exports delivered on exit, and your call recordings, transcripts, and customer data explicitly yours. All of it in the master service agreement, not in a verbal reassurance.
A bad answer hides behind “our proprietary platform.” Proprietary is fine; non-exportable is not. If termination means everything switches off and nothing transfers, the recurring fee is a ransom schedule.
Question 4: “Who Specifically Works My Account, and What’s Their Seniority?”
Agencies scale by selling with seniors and delivering with juniors, or with white-labeled subcontractors you never meet. In AI builds this matters more than it does in web design, because prompt design, integration architecture, and escalation logic are judgment calls, and bad judgment ships silently. You find out in month three, through your customers.
A good answer uses names. The person selling either builds the system or introduces you to the person who will, on the same call. Team size stated honestly; subcontracting disclosed if it exists.
A bad answer is “our team of experts.” If they will not name the builder before you sign, you will not get the builder after you sign.
Question 5: “How Do You Price, and What Triggers Overages?”
You cannot judge a quote without market bands. Digital Agency Network’s 2026 agency pricing guide puts pilot-scale projects at $5,000-$15,000, SMB retainers at $500-$2,000 a month, and mid-market retainers at $3,000-$8,000 a month. Voice AI adds a usage layer on top: Softcery’s 2026 analysis of 14 voice platforms found all-in costs of $0.05-$0.25+ per minute once telephony, speech-to-text, the language model, text-to-speech, and platform fees stack, and advertised “base rates” of $0.05-$0.09 usually exclude several of those components.
A good answer decomposes the number. Setup fee, monthly retainer, and usage pass-through stated separately; per-minute or per-conversation costs named; alerts or caps offered on usage so one busy week cannot produce a surprise invoice.
A bad answer is a single bundled figure with no usage line. Watch for “unlimited” plans with fair-use fine print, quotes far below the market bands (the missing money is usually hiding in your minutes), and pricing that only exists after a mandatory “strategy session.”
Question 6: “What Will You NOT Automate in My Business, and Why?”
An agency willing to automate anything is optimizing its invoice, not your outcomes. Real operators carry a refusal list, because they have watched where automation breaks: judgment-heavy conversations, high-liability actions, and processes that were broken before automation, which automation only helps fail faster.
A good answer is immediate and concrete. Something like: “We won’t let the AI give clinical or legal advice, touch refunds or payments without a human approval step, or replace your front desk outright. It takes overflow and after-hours first.” The speed of the answer tells you whether the list actually exists.
A bad answer is “AI can handle almost anything now.” That sentence is how businesses end up inside Gartner’s predicted 40% cancellation cohort.
Question 7: “How Do You Measure Success: Which Numbers, From What Baseline?”
The failed cohorts in the MIT and RAND studies share a trait: nobody agreed what success meant before launch, so nobody could prove or disprove it after. Without a pre-launch baseline there is no ROI claim: only vibes and screenshots.
A good answer names the metrics and insists on a baseline. Answered-call rate, speed-to-lead, booked appointments per 100 inquiries, hours of admin reclaimed, and cost per resolution, measured for 30 days before launch, then tracked against that baseline. We publish the full formulas and two worked examples in our guide to measuring AI ROI in a small business.
A bad answer shows you a dashboard of “conversations handled.” Activity metrics without a baseline are decoration. So are ROI calculators whose default assumptions are rigged in the vendor’s favor.
Question 8: “What Happens When the AI Is Wrong: Escalation, Guardrails, Liability?”
The AI will be wrong sometimes: a wrong answer, a wrong slot, a wrong price. The question is whether failure is designed or merely discovered. A vendor who treats this question as pessimism has never run a production system; a real one has an incident story ready and a mechanism to match.
A good answer describes the machinery. Escalation to a human by warm transfer with a transcript attached, confidence thresholds below which the AI hands off instead of guessing, human review of 100% of conversations in week one, error monitoring after that, and contract language that says who is responsible for what.
A bad answer is an accuracy percentage with no source and no mechanism. “Our AI is 99% accurate” answers a question you did not ask. You asked about the other 1%.
Question 9: “How Do You Handle My Industry’s Compliance: HIPAA, BAA, TCPA, Recording Consent?”
If you run a medical or dental practice, a voice vendor touching patient information is a HIPAA business associate and must sign a business associate agreement (BAA). If anyone proposes outbound calls or texts, TCPA consent rules apply, and call-recording consent laws vary by state. The liability lands on your business, not the vendor’s, which is why the vendor’s fluency here is a screening question, not a legal formality.
A good answer knows the acronyms before you say them. They volunteer whether they and their underlying platform vendors will sign a BAA, how recording disclosure works in your state, and how outbound consent and quiet hours are enforced, without you having to raise any of it.
A bad answer is “the platform takes care of that.” Platforms provide compliance features; they do not configure them for you. A compliance feature left unconfigured is a liability with good marketing.
Question 10: “What’s the Contract Structure: Checkpoints and Exit Ramps, or Four Phases of Lock-In?”
Gartner’s June 2025 prediction that over 40% of agentic AI projects will be canceled by the end of 2027 is a base rate you should price into the paperwork. A fair contract assumes the project might not work and defines what happens then. An unfair one assumes it will work and charges you either way.
A good answer has gates. A paid pilot with acceptance criteria agreed before launch, an exit window after the pilot, month-to-month terms after the initial period, and the ownership terms from question 3 written into the exit clause.
A bad answer is a 12-month minimum before anything goes live. Multi-phase “transformation roadmaps” payable up front, and early-termination penalties larger than the remaining contract value, are lock-in with a project plan stapled to it.
Question 11: “What Breaks When an API or Model Changes, and Who Fixes It, on Whose Dime?”
Models get deprecated, APIs get versioned, integrations drift, and a prompt tuned for one model can behave differently on its successor. After launch, maintenance is the actual product. RAND’s 2024 finding that teams systematically underestimate data and infrastructure work applies doubly to year two, when the novelty budget is gone and the automation is load-bearing.
A good answer includes maintenance in the retainer, with a response-time commitment. Monitoring that catches failures before your customers do, model updates tested before rollout, and a named number of included maintenance hours.
A bad answer bills maintenance hourly, as a surprise. “It shouldn’t break” is not an operations plan, and “we’ll fix issues as they come up” usually means you discover the outage from an angry customer.
Question 12: “What Results Have You Failed to Achieve?”
This is the honesty test, and the base rates make it powerful. With RAND putting AI project failure above 80% and MIT putting pilot success at 5%, a vendor claiming a spotless record is either lying or has too few clients to have failed yet. Neither is the partner you want.
A good answer is a specific story with a lesson attached. A deployment killed at pilot because the volume never justified it, a client churned over a mis-scoped integration, a use case they no longer sell, and what changed in their process because of it.
A bad answer is “all our clients see great results.” Follow up once (“none, ever?”) and watch whether you get a story or a subject change.
What Are the Red Flags of a Fake AI Agency?
The fastest tells: the pitch leads with technology instead of your workflow, the price only exists inside a sales call, and the statistics come with no source and no year. Any one of these is a caution flag; two or more means walk. Here is the 30-second version to run against any website or first call.
Leads with the model, not your workflow. “Powered by the latest AI” is a fact about their supplier, not about your missed calls. Real pitches start with questions about your volume, your booking rate, and your systems.
Only sells big packages. No audit-only option and no pilot means their unit economics depend on lock-in, not on results.
A “free audit” that is actually a pitch deck. Aries Consulting Group’s 2026 guide is blunt: free assessments are typically sales funnels. The test is the deliverable: does it contain your numbers, and is it useful if you never hire them? By that definition our own free audit is a sales funnel too; we ask to be judged by the same test.
Guarantees specific revenue. No vendor controls your close rate, your pricing, or your market. Guaranteed-outcome pitches survive on the clients who never measure.
Quotes statistics with no primary source. This industry recycles numbers that trace back to no study at all. Ask “what’s the primary source, and what year?” of every stat in the pitch, including the ones in this article, all of which are named and dated for exactly that reason.
Sells “AI employees.” Vocabulary like “digital workforce” and “hire your AI employee” layered over what is visibly a chatbot template is agent-washing’s native accent. Per Gartner’s June 2025 estimate, the odds that the vendor behind it is genuinely agentic are roughly 130 in thousands.
When Do You NOT Need an AI Automation Agency?
You do not need an agency when the workflow is simple enough to build in an afternoon, when a technical owner already exists in-house, or when your volume is too low to pay back even a cheap build. An honest buyer’s guide has to draw this line, so here is ours.
Skip the agency for two- or three-step deterministic workflows. Form submission to CRM record to notification email; invoice reminders on a schedule; review requests after a closed job. Zapier, Make, or n8n templates plus one focused weekend get you there, and paying agency rates for it is paying someone to type.
Skip the agency if you have a real in-house owner. Not “someone technical-ish,” but a person who lives in your CRM, will own an automation platform, and will still be maintaining the workflows in month eight. Ownership, not talent, is the scarce input: an automation nobody owns breaks silently and poisons trust in the whole idea.
Skip everything if you lack volume. Below roughly ten calls or leads a day (our working threshold, not a law of physics), most automation math gets thin, and RAND’s root-cause list warns exactly against chasing technology over business outcomes. Buying AI to fix a demand problem is the canonical version of that mistake. Fix the offer and the marketing first.
Hire the agency when integration surface and stakes are real. Voice systems that must book into a practice-management or field-service platform, HIPAA or TCPA exposure, multi-system writeback, guardrail and escalation design, and ongoing maintenance. This is where specialists earn their fee, and where DIY costs quietly exceed a retainer.
How Do You Score an AI Agency in 10 Minutes?
Score each of the 12 questions 0, 1, or 2: two points for a specific, verifiable answer; one point for plausible but vague; zero for deflection. Under 12 total, walk away. From 12 to 17, proceed only with a paid pilot and an exit ramp. At 18 or above, you have a shortlist candidate. The table below compresses the whole interview into one screen. Screenshot it and bring it to every call.
| Question | 2 points sounds like | 0 points sounds like |
|---|---|---|
| 1. Live system | A number you can call today, plus a reference | Demo sandbox, NDA wall, nameless case studies |
| 2. Diagnosis | Structured audit with a deliverable you keep | Tool proposed on the first call |
| 3. Exit ownership | Accounts, prompts, exports, data transfer, in the MSA | “It lives in our proprietary platform” |
| 4. Staffing | Named builder, met before signing | “Our team of experts” |
| 5. Pricing | Setup + monthly + usage decomposed, caps offered | One bundled number, usage hidden |
| 6. Refusal list | Immediate, concrete list of won’t-automate items | “AI can do almost anything” |
| 7. Measurement | Named metrics against a 30-day baseline | “Conversations handled” dashboards |
| 8. Failure handling | Warm transfer, thresholds, week-one review, liability terms | Unsourced accuracy percentage |
| 9. Compliance | Volunteers BAA / TCPA / consent detail unprompted | “The platform handles that” |
| 10. Contract | Pilot gate, exit window, month-to-month after | 12-month lock-in, phases prepaid |
| 11. Maintenance | In the retainer, SLA, proactive monitoring | Hourly surprises, “it shouldn’t break” |
| 12. Admitted failures | A specific story with a lesson | “All our clients see great results” |
Two scoring notes from running this on real vendor calls: a zero on question 3 (ownership) or question 9 (compliance, where it applies to you) should veto an otherwise high score: those are the two failure modes that get expensive. And treat charm as noise. In our experience the correlation between a smooth sales call and a good build is roughly zero.
How Does Benian Answer Its Own 12 Questions?
Compressed and factual, so you can hold us to the same standard and cross-examine us on a call. What follows is our operating reality as of August 2026, labeled as operating experience, not as a study, which is exactly the label a vendor’s claims about itself should carry.
Live systems (Q1). Production deployments include a Miami dental practice and a Texas HVAC company; we published the month-one numbers, with client permission, in our deployment data write-up, and prospects can hear a live line during a discovery call.
Diagnosis (Q2). We work audit-first: the paid AI audit follows the structure described above, and the deliverable is yours whether or not a build follows. Our free audit is the scoped-down version and, yes, a sales funnel: judge it by whether the deliverable contains your numbers and stands on its own.
Ownership and contract (Q3, Q10). Accounts and phone numbers are opened in your name or contractually transferable; prompts, workflow exports, documentation, recordings, and data transfer on exit. Engagements run pilot-first with acceptance criteria agreed before launch, an exit window after the pilot, and month-to-month terms after the initial period. Benian OS is built on the same principle: it is your operating layer and your data, not a rented dashboard.
Staffing (Q4). Small team by design, and the founder is in every build. You meet the person doing the work before you sign, because there is nobody else to hide behind.
Pricing (Q5). Proposals decompose setup, monthly, and usage pass-through, with voice minutes billed against the real per-minute stack rather than a teaser base rate. Our bands sit inside the market ranges cited above; our production voice AI deployments have typically run $350-$750 a month plus pass-through telephony and model usage, consistent with the case-study data we publish.
Refusals, guardrails, and compliance (Q6, Q8, Q9). We do not let AI give clinical or legal advice, touch payments or refunds without a human approval step, run fully autonomous outbound sales, or replace a front desk outright. It takes overflow and after-hours first. Escalation is warm transfer with a transcript; week one is 100% human review of calls. For healthcare deployments we sign BAAs and configure recording disclosures, and anything outbound requires documented consent.
Measurement (Q7). A 30-day baseline before launch, then five numbers: answered-call rate, speed-to-lead, bookings, hours reclaimed, cost per resolution, reported weekly through month one and monthly after.
Maintenance (Q11). Monitoring and maintenance hours are included in the retainer with a response-time commitment, and we test model updates on our own systems before they touch a client’s.
Failures (Q12). We have told prospects their call volume was too low to justify voice AI, we have ended pilots that did not clear their gates, and we under-scoped integration hours on early builds and ate the difference. An agency that has never done any of those things has not been an agency very long.
The Next Step
Print the 12 questions and take them into every sales call (including ours). If you want to see how we handle question 2 in practice, the free audit is where our diagnosis process starts: we baseline your calls, leads, and workflows and hand you the findings. By the rule in this guide, judge it the same way you would judge anyone else’s: by whether what you receive is useful even if we never speak again.