Mid-market SaaS company
Designed and shipped an AI triage layer that resolved 40% of inbound tickets without human handling.
The Challenge
A mid-market SaaS company's four-person support team was drowning. Ticket volume had doubled in 18 months as the product grew, but headcount had not. Average first-response time hit 4.2 hours, CSAT dropped to 3.8 out of 5, and the team was showing signs of burnout.
An audit of 6 months of tickets revealed that 42% were near-identical: password resets, billing enquiries, a handful of recurring feature questions. Every one of those tickets consumed the same 8–15 minutes of a human agent's time as a genuinely novel problem. The opportunity was clear — automate the repeatable, free the team for the complex.
AI Ticket-Triage Flow
Our Approach
We started with a data audit rather than a model. Six months of resolved tickets were categorised, and 40 distinct repeatable query types were identified. Only then did we design a two-stage AI pipeline: a classifier to label intent and confidence, followed by a RAG (retrieval-augmented generation) layer to draft a verified answer from the company's curated knowledge base.
Shadow mode was essential. For the first two weeks in production, the AI processed every ticket and drafted responses, but humans reviewed 100% of them before anything reached a customer. This produced a labelled dataset of corrections that we used to fine-tune the confidence threshold and catch edge-case misclassifications safely.
The escalation path was designed before the automation. Any ticket the AI flagged as low confidence was routed to the human queue with the AI's draft and reasoning attached — reducing, not eliminating, agent effort even on the tickets the AI didn't auto-resolve.
Results
40%
Tickets auto-resolved
without human handling
8 min
Avg. response time
down from 4.2 hours
4.6/5
CSAT score
up from 3.8
−35%
Agent workload
more time for complex cases
Engagement Phases
Ticket Audit
Week 1Categorise 6 months of resolved tickets, quantify repeatable query types, and validate that automation is viable without degrading response quality.
Knowledge Base Curation
Weeks 2–3Audit, clean, and chunk 200+ help-centre articles. Embed with text-embedding-3-small and load into Pinecone for semantic retrieval.
Pipeline Build
Weeks 3–5Build the GPT-4o classifier + RAG pipeline. Define confidence thresholds, response templates, escalation triggers, and the Zendesk integration via API.
Shadow Mode
Week 6Run AI in parallel on all live tickets. Humans review 100% of AI drafts, corrections are logged, threshold calibration is done from real data.
Staged Rollout
Week 7Enable auto-resolve for the 10 highest-confidence query types only. Monitor CSAT, false-positive rate, and escalation volume daily.
Full Rollout & Handover
Weeks 8–9Expand to all 40 automatable query types. Hand over Langfuse monitoring dashboard and runbooks to the support team lead.
Technology Used
AI / LLM
Vector Store
Backend
Integrations
Infra
Observability
Key Lessons
Shadow mode before live auto-resolve is non-negotiable. Two weeks of human review produced calibration data that would have taken months to gather from production incidents.
The classifier confidence threshold is your most important hyperparameter. We started at 90% and found false-negative rates too high; 85% gave the right precision/recall balance for this domain.
Auto-resolve only works if the knowledge base is accurate and current. We built a lightweight quarterly audit process into the handover so the client's team owns the quality of future answers.
The metric that mattered most to the team was not CSAT — it was workload reduction. Reducing the monotonous ticket queue gave agents energy for the complex cases that actually need human empathy.
Questions & Answers
The confidence threshold is set conservatively. For borderline cases, the AI drafts a response for agent review rather than sending it. Langfuse traces every decision, so any reported wrong answer can be traced back to its retrieval context within seconds.
We built a quarterly audit runbook into the handover: the support team lead reviews the 20 most-retrieved chunks and flags any that reference deprecated features. The embedding pipeline re-runs in under an hour.
Yes, but the classifier needs more fine-tuning data and the confidence threshold should start higher. We would also recommend a stronger human-in-the-loop for any query touching billing or account security, regardless of confidence score.