Back to Insights Educational article

AI for Customer Support: A Practitioner's Playbook (2026)

The model is the cheapest decision you make. What kills support bots in month three is the eval set you skipped, the refund tool with no spending cap and the multilingual launch you treated as a model problem.

Last updated: May 24, 2026 Customer Support

~9 min read Reading time 7 Sections Guide Long-form guide

Most AI support bots that get fired in month three were doomed in week one, when somebody picked a vector database before writing fifty representative test conversations.

I keep getting asked which model to use for customer support. It is almost never the question that matters. The question that matters is what you let the bot do, what you make it refuse and how you find out it is lying to customers before they tell you. This is a piece for an operator who already knows what an LLM is and wants to ship one without joining the wall of post-mortems. From the few of these we have deployed and the ones I have read autopsies of, the same handful of decisions show up every time.

01 The Klarna lesson

Why the headline number was never the point

In February 2024 Klarna announced its OpenAI assistant was handling two thirds of all chat support, doing the work of 700 agents. The press release was everywhere. Eighteen months later the same company was hiring humans back, the CEO publicly saying quality had slipped. Both things are true. The bot did handle that volume. It also did not handle it well enough.

The takeaway most operators drew was the wrong one. It was not that LLMs cannot do support. It was that resolution rate is a vanity metric the moment a customer's choice is auto-resolve or escalate-and-suffer. If you measure only the first column you ship a system optimised for closing tickets, not solving problems. Klarna's number was real. The thing that number measured was not what the board thought they were buying.

02 Model choice

Pick the model last, not first

The model is the cheapest decision you make. Claude Sonnet 4.5 at $3 in and $15 out per million tokens, GPT-5 via Azure OpenAI for compliance-bound buyers, Gemini 2.5 Pro if you are already a Google Workspace shop. They all clear the bar. None of them will save you from a bad retrieval layer or a refund tool with no spending cap. I have watched two teams spend six weeks A/B testing Claude against GPT and ship a worse product than the team that picked Sonnet on day one and spent those six weeks writing eval cases.

The same logic applies to vector stores. Pinecone is fine. pgvector inside the Postgres you already run is also fine and cheaper. Cohere Rerank in front of either one buys you more accuracy than swapping the embedding model. The interesting work is in tool definitions and the JSON schemas you hand the model, not in benchmark battles. Read Anthropic's tool-use docs once, then write your refund tool five times before you write it well.

03 Vendor reality

Intercom Fin, Zendesk AI and the build-vs-buy trap

Intercom Fin is good. It is good in a narrow way: B2B SaaS with clean help-centre docs, low SKU complexity and a CSAT bar that tolerates the occasional shrug. The pricing per resolved conversation looks reasonable until you model it against your real ticket mix and realise you are paying premium for the easy ones the bot was going to win anyway. For consumer brands with messy product catalogues, multilingual returns flows or anything regulated, Fin is the wrong shape of tool. Zendesk AI agents are in roughly the same bucket, with the additional friction that you are now married to Zendesk's data model.

In January 2024 Intercom reported that customers of its Fin AI agent resolve an average of 41 percent of conversations end to end. That figure is the honest baseline to plan around, whatever a louder deck claims.

Build only when the off-the-shelf path forces you to bend your business around its limits. The honest middle ground for most teams I talk to is a thin custom orchestration layer over a managed model and a managed vector store, talking to your existing helpdesk through its API. You write maybe four thousand lines of code. You own the bits that matter, which are the tool definitions, the eval harness and the escalation routing. You rent the bits that do not, which are the model weights and the storage.

FIG. 01 – WHERE EACH PATH FITS Three honest paths, with their failure modes

Off-the-shelf (Fin, Zendesk AI) – Thin custom layer – Full custom build
Fits when – Clean docs, low SKU complexity, English-first – Messy data, real tool calls, multilingual – Regulated industry or unique workflow
Typical all-in cost – $2K–$10K / mo – $60K–$180K build, $4K–$15K / mo – $200K+ build, $20K+ / mo
How it fails – You bend the business to fit the tool – Owner team leaves and nobody maintains it – Six month delay then you ship Fin anyway

04 The actual work

Retrieval, tools and the eval set you keep skipping

The work that decides whether your bot survives contact with real traffic is the part nobody puts in the proposal deck. None of it is the model.

Where to spend the weeks

Fifty real conversations, hand-graded, run nightly against every model version
Tool schemas with explicit refusal cases, not just happy paths
Knowledge that updates the same day the product changes, automated from a single source
Confidence threshold tuned per intent, not one global number

Where the dead bots spent their weeks

A/B comparing three models on a deck instead of writing the eval set
Ingesting every PDF in the SharePoint without grading any of them
A refund tool with no spending cap and no audit trail
Quarterly knowledge refreshes for a product that ships weekly

The eval set is the unglamorous part. Fifty conversations is the floor, not the ceiling. They have to be real, with the spelling mistakes and the conjoined questions and the customer who started in Polish and switched to English halfway through. OpenAI's cookbook has decent starting templates if you have never built one. The point of the eval is not to score the bot. The point is to notice when a model update or a knowledge change quietly breaks something that used to work.

05 Action tiers

What you let the bot actually do

The capability shift that matters in 2026 is tool calling that mutates state. The bot is not answering questions, it is issuing refunds and changing addresses and pausing subscriptions. This is where the money is and also where the lawsuits live. The right design is three tiers, the wrong design is one big bucket called assistant permissions.

FIG. 02 – ACTION PERMISSION TIERS Reversibility decides which tier each action lives in

Auto, no friction – One-tap confirm – Human required
Examples – Order status, tracking, contact preferences – Refunds under your threshold, subscription pauses, address changes – Account closure, identity change, refunds above threshold
Reversibility – Fully reversible – Reversible with effort – Effectively permanent
What you must log – Action, timestamp, customer id – All of the above plus the confirmation token – Everything plus the human's id and reason

If your refund tool does not have a per-call spending cap and a daily aggregate cap, you do not have a refund tool, you have an incident waiting for a Hacker News thread. The cap is not there because the model is dumb. It is there because a determined customer with a working browser and a willingness to retry will find a phrasing that gets past your guardrails, and the cap is what limits the blast radius when they do.

06 The unpopular take

Multilingual support is mostly a knowledge problem

I keep seeing teams treat multilingual as a model problem. It is not. Sonnet 4.5 and GPT-5 both speak twenty plus languages well enough to fool a casual listener. The problem is that your help articles exist in English, your returns policy exists in English with a half-translated German version from 2022 and your Polish escalation team uses a different ticket schema than your Spanish one. The bot speaks the language fluently and gives confidently wrong answers because the underlying knowledge is wrong or missing.

“ The bot is as smart as the worst-translated paragraph in your knowledge base. Fix that paragraph before you change the model.

Launch in two languages. Get your CSAT, escalation rate and refund-disputed rate roughly equal across both. Only then add the third. Teams that launch in seven on day one are choosing the same dashboard screenshot Klarna chose.

07 How we would approach this

What we would actually ship in twelve weeks

For a mid-market business doing this for the first time, twelve weeks is honest. The first three are knowledge audit and the eval set, which is the part everyone wants to skip. Weeks four through six are the tool layer and the orchestration, picking a model and writing the refund and lookup tools with their caps and their audit trails. Weeks seven and eight are a closed pilot with internal traffic only. Weeks nine and ten run five percent of real traffic with a human reviewing every output, which is annoying and necessary. Weeks eleven and twelve expand to forty or fifty percent if and only if the eval set has stayed green for two consecutive weeks. Anyone selling you four weeks for the same scope is selling you the demo, not the deployment. The ninety day roadmap covers the sequencing in more depth, and the implementation cost breakdown covers what each stage actually runs.

If you want a second pair of eyes

This is the kind of project that benefits from someone who has watched a few of them fail in the same way twice. Our services page covers what that looks like in practice, or you can send us the scope and we will tell you honestly whether you should build, buy or wait six months.

Frequently Asked Questions

Which model should we actually pick?+ Claude Sonnet 4.5 if you are starting fresh and want the best tool-calling discipline today. GPT-5 via Azure OpenAI if your buyer needs an Azure tenancy for compliance reasons. Gemini 2.5 Pro if you are already on Google Workspace and want the billing in one place. The differences are real but smaller than the gap between any of these and a missing eval set. Is Intercom Fin worth it?+ For a B2B SaaS with clean docs and an English-first audience, yes. For a consumer brand with a messy product catalogue or anything multilingual and regulated, no. The pricing per resolved conversation looks attractive until you realise you are paying premium rates for the easy tickets your bot was always going to win. How big should the eval set be at launch?+ Fifty real conversations is the floor. A hundred and fifty is comfortable. Two hundred plus is where most teams stop adding net new value. The set has to grow as you discover new failure modes in production, so plan to add ten to twenty per month for the first six months of operation. How do we stop the bot from issuing fraudulent refunds?+ Per-call spending cap, daily aggregate cap, identity verification step for amounts above your threshold and a full audit trail with the confirmation token logged. Treat the refund tool the same way you would treat a junior agent on day one: scoped permissions, logged actions and a supervisor reviewing the edge cases weekly. What about voice support, not just chat?+ The latency budget for voice is brutal: under eight hundred milliseconds round trip or it feels uncanny. The model layer can hit that. The retrieval layer usually cannot without a separate cache for the top intents. Treat voice as a second project once the chat version has been stable for two quarters, not as a launch-day feature. Will this replace our human agents?+ No. Routine volume drops, complex case volume becomes a larger share of what humans see and the humans who stay need to be better. Companies that promised their boards a headcount cut and got one usually walked it back within a year, Klarna being the most public example. Plan for flat to slightly down headcount and a meaningful shift in the work mix instead.

ANM SOLUTIONS / CONTACT US

Need help applying this to your business?

We turn AI insights into measurable business outcomes. Tell us about your workflow and we'll show you the highest-impact place to start.

Implementation

The 90-Day AI Implementation Roadmap: From Scope to Production

A practitioner's view of the first 90 days of an AI implementation. Boring stack, eval-first, kill switch by day 90. Where most plans go wrong and what to do instead.

Read article

Vendor Selection

How to Choose an AI Implementation Partner Without Overpaying

An honest look at the five categories of AI implementation partner, where each one fits and how to tell a useful vendor from an expensive deck factory in a single technical conversation.

Published: 2026-05-05 · Author: A&M Flow