Production Hardening

A prototype has one user, patient expectations, and nobody watching the bill. Production has none of those. Hardening is the work that closes the gap.

Cost

• Per-request cost accounting from traces, broken down by feature and prompt, so you know where the spend actually goes.
• Caching where it is safe: exact-match response caching, prompt-prefix caching, embedding caches for repeated content.
• Model routing — smaller models for classification and routing steps, larger ones only where the output quality demands it, with evals proving the cheaper path holds up.

Reliability

• Fallback chains for provider errors, rate limits, and timeout budgets, with explicit degraded modes instead of blank failures.
• Retry policies that distinguish transient faults from systematic ones, so a bad prompt does not get retried into a large bill.
• Output validation and repair for structured responses, with dead-letter handling for the cases that cannot be repaired.

Experience

• Streaming end to end, so users see progress instead of a spinner.
• Latency budgets per step, measured at p95 — including the retrieval and rerank stages that quietly dominate response time.

Change management

• Versioned prompt management: prompts as reviewed, diffable artifacts with rollout and rollback, not strings edited in production.
• Eval gates on every change, so hardening work never quietly regresses quality.

Framework-exit refactors

Sometimes the hardening problem is the framework. Where LangChain abstractions add indirection without earning it — a simple pipeline wrapped in five layers of callbacks — we refactor to plain SDK calls, keeping the pieces that pull their weight (often LangGraph for stateful workflows and LangSmith for tracing). Your evals make the refactor safe: same suite, same scores, less machinery.