What is the right way to test and evaluate LLM application quality before shipping?

Treat LLM evaluation like unit testing: automated, versioned, and run on every change — not a one-time benchmark. The layers: 1. Offline test suites: DeepEval gives you pytest-style assertions for correctness, hallucination, and safety; if you run RAG, Ragas adds retrieval-specific metrics (faithfulness, context precision). 2. Experiment tracking: Braintrust logs eval runs across prompt and model versions so regressions are visible, not anecdotal. 3. Automated scoring in production: Patronus AI scores and monitors live outputs for failures after launch. 4. Adversarial evals: security testing is evaluation too — Promptfoo (now part of OpenAI, still open source) merges red-team probes into the same eval-driven workflow, and Giskard scans models for quality and vulnerability issues in one pass. Minimum viable setup: 50–100 golden test cases from real user queries, one hallucination metric, one safety metric, wired into CI. Expand from what actually fails.