SAV-LLM: Building a Zendesk AI Assistant

modified
length
11 min read

TL;DR: Freelance contract for pieces2mobile.com, a Zendesk AI assistant powered by Qwen3.5 9B on 2x RTX 5070 via llama.cpp, orchestrated with NestJS and LangGraph.js. Dedicated on-prem server, 24/7, 60-90s per ticket, 70% auto-answered, ~30% escalation. GitLab CI/CD, no Telegram, all escalation via Zendesk agents. Tests are the only contract the LLM cannot cheat.

The Gig

pieces2mobile.com is a French phone repair parts distributor. They had no AI assistant and no viable trial path. Cloud LLM pricing was too obscure to predict, and more importantly, they wanted zero risk of business data hitting third-party servers. A local LLM on dedicated hardware was the answer: predictable cost, full data sovereignty, no API fees.

I built them an on-prem LLM server: two RTX 5070s in a dedicated workstation, running llama.cpp server-cuda. The GPUs belong to this pipeline, no sharing, no scheduling windows. It processes tickets 24/7.

80-90% of LLM projects using frontier models never make it to production. This one did, with a local 14b parameter model running on consumer GPUs. It was built from scratch with NestJS and LangGraph.js. That's what I'm most proud of.

Architecture

The stack is boring on purpose. NestJS for the API layer, TypeScript throughout, Docker Compose for deployment. multi-stage Dockerfile (builder → prod_deps → production, non-root user). No Kubernetes. No service mesh. No event bus. Just a Bull queue consumer that picks up Zendesk webhooks and feeds them through a LangGraph.js pipeline.

[Zendesk Webhook]
     |
     v
[NestJS Queue] --> [LangGraph Pipeline]
                       |
            +----------+----------+
            |                     |
     [Domain Identifier]    [Answer Generator]
            |                     |
            v                     v
     [Vector Store]         [llama.cpp x2]
                              |    |
                          [Qwen3.5] [Validation]

The pipeline has 18+ LangGraph nodes across 5 terminal paths:

  • Security scan: every ticket is scanned before any processing. Unsafe content escalates immediately.
  • Domain identification + classification: identifies pro/particulier context, classifies intent into 9 domains (account, buyback, commercial, general, orders, payment, shipping, technical, warranty)
  • Retrieval: context retrieval from a 2286-line FAQ knowledge base via BGE-M3 embeddings in a vector store (HNSWLib index, ~670KB)
  • Pre-generation abstention gate: if retrieval grade is irrelevant after query rewrite, the system escalates rather than generating a "no information available" response (production fix, observed in April 2026)
  • Evidence synthesis + policy routing: two dedicated nodes that build customer evidence and route to the correct response policy before generation
  • Business validation: validates the generated answer against business rules before sending

Full end-to-end: 60-90 seconds per ticket.

Queue, Don't Stream

Tickets arrive as Zendesk webhooks and land in a Bull queue with Redis persistence. Configurable concurrency of 2, exponential backoff on failure.

The pipeline does not need more parallelism. Reliability matters more. A Zendesk support team needs reliability, every ticket processed, nothing lost on crash. The queue has three states:

State Meaning
WAITING Ticket received, pending processing
PROCESSING LangGraph pipeline active
COMPLETED Answer generated and posted back

If processing fails (schema validation error, LLM timeout), the ticket goes to a dead-letter queue with retry. A GpuHealthService monitors GPU health every minute and pauses the queue if the backend drops, no silent failures.

CI/CD: GitLab Pipeline

The full pipeline runs in GitLab CI:

Stage Runner Description
lint cpu ESLint + Prettier
unit-tests cpu Jest unit tests
integration-tests cpu Jest integration tests
e2e-tests gpu Full pipeline tests on GPU runner (12 regression .qa2.json fixtures), 3h timeout
deploy dev gpu Auto-deploy on dev branch, health check with 24 retries
deploy production gpu Manual deploy on main branch with rollback capability

Cleanup scripts remove root-owned Docker artifacts between stages. Staging (duplicate env) runs alongside dev for CI validation without blocking.

Validation Engine: The Anti-Hallucination Guardrail

Tests are non-negotiable. An LLM can lie to you, it can produce confident, fluent nonsense. But it cannot cheat your tests.

The validation layer is a strategy pattern orchestrator with 20+ registered strategies running in parallel with p-limit(3) concurrency:

  • P0.3 critical strategies: 6 strategies that force escalation regardless of pass rate: dont-know-phrase-detection, customer-status-policy-validation, contact-info-allowlist, pro-retractation-guard, no-compensation-guard, irrelevant-but-replied-guard
  • FAQ mode: 75% confidence threshold: Transaction mode: 100% (returns/refunds)
  • Two-stage pipeline: Claim extraction → claim verification against FAQ corpus

An output guardrail runs deterministic post-processing after LLM generation, pure string operations, <10ms:

  • Replaces hallucinated emails (catches @pieces2mobile.com → sav@partse.com)
  • Enforces scenario-specific phone numbers from a whitelist of 3
  • Normalizes phone formats, detects garbled numbers via sorted-digit matching
  • Corrects SAV address truncation and wrong postal codes
  • Corrects hallucinated Saturday delivery times (Chronopost cutoff: 20h00, not 17h/18h/23h59)
  • Escalation triggers for non-recoverable errors: wrong retraction delay (must be 14 jours), unauthorized commercial gestures

QA Dashboard: Watching the Pipeline Think

A black box that occasionally answers tickets is not a support system. You need visibility into every decision. That's what the QA dashboard is for.

It's a terminal UI built with blessed and blessed-contrib, real-time TUI, no web app, no Grafana. It connects directly to the test database and refreshes every second. Four quadrants:

  • Ticket list (left panel): every processed ticket with status icons: ✅ replied, ⛔ escalated, ⏳ processing, ❌ error. Click any ticket to drill in.
  • Ticket details (top right): full routing trace: every LangGraph node the ticket passed through (🧠 classifyQuery, 📚 gradeRetrieval, ⚖️ validateBusinessRules, ➡️ routing decisions), the LLM answer (truncated), classification confidence, extracted entities, and a per-strategy validation breakdown.
  • Progress gauge: how many tickets processed against the QA2 corpus (174 questions, later expanded to 186). Green fill, hard number.
  • Status donut: replied vs escalated vs processing vs error, at a glance.
  • Live activity log: streaming log of routing decisions as tickets flow through the pipeline.

The dashboard runs against a dedicated test database (assistant_ia_test) with its own MongoDB instance on port 27018, isolated from dev/staging/prod.

Export Pipeline

The same data feeds a CSV export script (export-qa-results.js) that dumps 60+ columns across 9 telemetry categories into a structured CSV:

Category Fields
Core ticketId, question, llmAnswer, status
Classification scenario, confidence, reasoning, applicableRules, extractedEntities
Customer customerType (pro/particulier), customerStatusConfidence
Validation overallValid, passRate, totalStrategies, strategiesPassed/Failed, violations
Strategy (7) customerStatusPolicy, externalPolicySpeculation, faqEchoNeutralizer, commercialGesture, placeholderDetection, warrantyDenial, professionalRetraction
Retrieval strategy, docsRetrieved, avgScore, maxScore, minScore, timeMs
Grading retrievalGrade, relevantRatio, relevantOnlyConfidence, fastRejectCount, llmGradeCount
Workflow actions, routingLogic
QA Annotation status, flags, notes, annotatedBy, dev, review

The QA annotation fields are the feedback loop: each ticket can be reviewed (status, flags, notes, reviewer). Annotated edge cases inform the golden dataset. What the human corrects today becomes a regression test tomorrow.

Analysis Scripts

Beyond the dashboard and export, a family of analysis scripts runs against the exported CSV:

  • analyze-qa-accuracy.js: heuristic keyword matching against FAQ topics (21 topic maps). Produces an accuracy report per ticket, flagging fallback patterns ("désolé", "je ne suis pas sûr") and template quality.
  • analyze-validation-failures.js: digs into which validation strategies fail most often, so you fix the right bottleneck.
  • confusion-matrix.js: breaks down replied vs escalated by domain (account, shipping, technical, warranty...), showing where the pipeline over-escalates or under-escalates.
  • analyze-customer-gate.js: tracks professional vs individual customer handling to ensure policy parity.
  • analyze-faq-failures.js: correlates specific FAQ sections with answer failures.

Continuous Regression

The CI pipeline (e2e stage on GPU runner) runs the full QA suite against 12 .qa2.json regression fixtures, multi-turn conversations with assertion-based testing (not exact string match, but includes_any assertions):

simple-support-flow        → basic acknowledgment flow
ticket-92-free-returns     → return policy edge case
wrong-item-customer-fault  → attribution-sensitive escalation
wrong-item-accusation      → divergence with fault ambiguity
wrong-item-ambiguous       → unclear fault, safe escalation
greeting-order-status      → multi-turn, context preservation
status-clarification       → follow-up with incomplete info
product-info-detail        → technical spec retrieval
order-cancellation-valid   → order lifecycle
followup-rma-details       → RMA submission completeness
human-escalation           → explicit agent handoff
pro-warranty-flow          → professional customer warranty

Each fixture is a multi-turn conversation with assertions at every turn. The pipeline gets a 3-hour timeout for the full suite. A staging environment runs alongside dev for CI validation without blocking deployments.

The golden dataset itself lives in two text files: QA50.txt (50 core questions) and QA119.txt (119 extended), both recently merged into QA2-Production-186.md, 186 questions covering the full product catalog, customer types, and escalation scenarios.

Safe Escalation by Design

The pipeline is built to escalate when it should. Eight distinct escalation paths:

Path Trigger
Security escalation Unsafe content detected
Domain-based escalation Wrong-item divergence with ambiguous fault attribution
Abstention gate Irrelevant retrieval grade after query rewrite
Guided escalation Low-confidence classification
RMA completeness gate Incomplete RMA submission
Macro fallback Macro application failure
Template reply Pre-generation routing for template_reply classification
Post-generation Business rule validation failure

All escalation is handled through Zendesk agents, no Telegram, no external HITL. The LLM handles the straightforward 70%. The remaining ~30% are escalated with context, a guided escalation reason, and a customer-facing message so the agent picks up a warm ticket, not a cold one. Over time, corrected edge cases become test cases, shrinking the 30%.

Numbers That Matter

  • Model: Qwen3.5 9B Q8_0 GGUF (via llama.cpp, not Ollama)
  • GPUs: 2x RTX 5070 (12GB VRAM each), dedicated
  • Runtime: llama.cpp server-cuda on both GPUs, flash-attention, 7168 context
  • Pipeline: 18+ nodes in LangGraph.js, 5 terminal paths
  • End-to-end: 60-90s per ticket
  • Auto-answered: ~70%
  • Escalation rate: ~30%
  • Knowledge base: 2286-line FAQ.md, BGE-M3 embeddings
  • Concurrency: 2 (configurable), 24/7 operation
  • CI/CD: GitLab, 5 stages, GPU runner for e2e

The ROI of On-Premise LLM

Cloud LLM pricing is opaque, per-token costs that compound unpredictably with context windows, retrieval chains, and multi-turn conversations. A single ticket going through a 3-turn LangGraph pipeline with retrieval context can burn 4,000+ tokens just on the input side. Multiply by hundreds of tickets per day and you're looking at a four-figure monthly bill before you've answered a single customer.

The cloud alternatives break down like this:

Model Input (per 1M tokens) Output (per 1M tokens)
GPT-4o $2.50 $10.00
Claude Sonnet 4.6 $3.00 $15.00
DeepSeek V3 $0.27 $1.10

At 500 tickets/month, averaging 4k input + 500 output tokens per ticket, that's 2.25M tokens, roughly $15-45/month in API costs alone. Add webhook infrastructure, vector store hosting, and the hidden cost of integrating yet another vendor's auth and rate limits.

Against that, the on-premise model:

  • Hardware: 2x RTX 5070 in a dedicated workstation: one-time cost, owned by the client
  • Inference: zero per-token cost. The GPUs are already paid for.
  • Latency: 60-90s per ticket, 24/7. No rate limits, no queue contention, no token bucket exhaustion.
  • Data sovereignty: zero customer data leaves the server. No third-party inference API sees a single ticket.
  • Scaling: volume goes up? The GPUs keep running. No new seats, no tier upgrades, no surprise invoices.

The other cost that doesn't show up on a bill: integration risk. Every cloud LLM vendor has a different API contract, different authentication, different rate limiting, different SLAs. By standardizing on llama.cpp server-cuda with an OpenAI-compatible API, the pipeline speaks one protocol. Swap the model file, restart the server, done.

A practical advantage is that the test suite doesn't need API credits. Every CI run, lint, unit, integration, e2e, runs against the local GPUs. No mock servers, no API stubs, no "we can't run tests because the vendor is rate-limiting us."

The real ROI, though, isn't infrastructure savings. It's that a support team that was spending 100% of their time on repetitive tickets can now focus on the 30% that actually need human judgment. The pipeline doesn't sleep, doesn't take vacation, and doesn't get tired of answering "Quel est le délai de livraison ?" for the 400th time.

Source References

These links anchor the claims to the codebase on Vindaloop:

Go Build

SAV-LLM is a reminder that you do not need a cloud subscription, a dedicated GPU cluster, or a team of MLOps engineers to build production LLM applications. Two workstation GPUs in a warehouse, a 14b parameter model running on llama.cpp, a clear domain model, tests that enforce contracts, and safe escalation through tools you already use.

The hardest part was not the LLM integration. It was the test harness that catches hallucinations, the abstention gate that prevents no-info responses from reaching customers, and the validation engine that enforces business rules on every single reply.

The LLM is just another component. Treat it like one.

═══════════════════════════════════════

Related projects
Marklar Distributed System
Source ↗
SAV-LLM
Source ↗