Six questions to ask an AI helpdesk vendor that they'll struggle to answer

Vendor pitches in this category blur together. Here are six questions that surface the difference between a polished demo and a system you can operate for two years.

For each, I've included what a good answer sounds like and what a bad answer sounds like. The vendors worth your time will recognize the questions and respond cleanly. The ones who don't will pivot to "let me have a sales engineer follow up."

1. What happens when my custom ticket statuses don't match the defaults?

Bad answer: "We're highly configurable."

Good answer: "At runtime, we read your tenant's actual status list via the API and route based on what's there. If we encounter a status we don't recognize, we mark the ticket for human review rather than guessing. There's a config screen where you can map custom statuses to behavior categories. Here, let me show you."

If they have to think about this question, they haven't run their system against a real MSP tenant.

2. Show me your audit trail for a specific failed action from six months ago.

Bad answer: "We have full compliance logging." (Vague compliance words = no logs.)

Good answer: "Every action by every agent is logged with the agent name, the input it received, the tools it called, the result of each call, and a hash of the LLM prompt used. Retention is 12 months by default, extendable. Here's a query you can run yourself."

The vendors who actually operate this in production have specific answers about log structure and retention. The ones who don't are pre-incident.

3. What's your rollback path if a deployed specialist starts making bad decisions?

Bad answer: "We monitor for that."

Good answer: "Each specialist is a separately deployed unit with its own kill switch. From the admin UI, you can pause a specialist with one click. It stops accepting new work but completes in-flight items cleanly. Median rollback time from 'we noticed something is off' to 'specialist is paused' is under 30 seconds."

Notice the specific time. Vendors who've handled real incidents have a number. Vendors who haven't talk about monitoring abstractly.

4. How do you handle tenant-specific custom fields that don't exist in your demo?

Bad answer: "It learns from your data over time."

Good answer: "We introspect each tenant's schema at runtime. If the tenant has a custom field that's in scope for the work the specialist is doing, we read/write it based on a mapping table. If there's a field we don't know about, we skip it rather than guess."

"Learns over time" is what vendors say when they don't actually handle schema variance and are hoping you won't notice.

5. What's your incident-response process when your AI is the cause of the incident?

Bad answer: "We follow standard SOC 2 incident procedures."

Good answer: "Specifically for AI-caused incidents, we have a runbook that includes: pulling the LLM prompt and the model version active at the time, the input that triggered the bad behavior, the tool calls made, and the downstream effects we can identify. We notify affected customers within 4 hours with a preliminary writeup. We've had to do this twice. Happy to walk you through one."

The honest answer involves them having had at least one incident and knowing the shape of it. Vendors who haven't operated in production yet don't have this answer.

6. Who else is running this in production at MSP scale? Can I talk to two of them?

Bad answer: "We have many customers." Or: "We can introduce you, but they're in a pilot phase."

Good answer: "Yes. Here are two MSPs who've been on the platform 6+ months. They've agreed to take reference calls. The first is a 25-tech shop running it across 40 client tenants; the second is a 60-tech shop running it across 200+ tenants. Both will give you their unvarnished take. I'd recommend you talk to them before deciding."

The vendors with real references happily make the intro. The ones without references will explain why references aren't relevant yet.

The pattern under the pattern

Each question above tests the same thing: has this vendor's product operated in real MSP production, or only in demos?

Demo-mode answers are abstract. Operating-mode answers are specific. They involve numbers, named runbooks, real incidents, real customers. The shift from one to the other happens around customer #5 or #10 in production, and you can't fake your way through it in a sales call if the answers are concrete enough.

---

What's the question that's exposed the most vendors for you?