Playbook 08 · Feedback

Every failure becomes a permanent test

Goal: a closed loop where a production failure, an escalation, a thumbs-down, a wrong answer someone reported, becomes a permanent regression case that gates every future release. Fix it once, and the bug can never silently come back, because the test that would catch it now runs in CI and on a weekly drift job forever.

The path is one direction: production signal → failure queue → human triage → permanent eval case → gated fix → weekly re-verify. Every arrow is a Zolva primitive; nothing here is glue you have to maintain.

Prerequisites #

• A running AgentApp and a place to store the queue (a SQLite file such as failures.db).
• An eval judge provider for the regression cohorts you will create.
• Python ≥ 3.11 and pip install zolva.

Step 1 · Capture the signal #

Attach the queue to the app once. Escalations and guardrail violations are captured automatically; record explicit signals (a thumbs-down from your UI, an agent-review note) with one call.

from zolva import AgentApp, FeedbackQueue

app = AgentApp.from_config("agents/")
q = FeedbackQueue("failures.db")
q.attach(app)          # escalations + violations auto-captured

# from your product surface, when a customer or reviewer flags a reply:
await q.record(session_id, "collections-agent", "thumbs_down",
               note="quoted the wrong due date")

Synthetic and eval sessions are skipped by the capture, so the queue holds real production failures, not test traffic.

Step 2 · Triage into a regression cohort #

A human decides what a good answer looks like. Triaging promotes a captured failure into a permanent eval case, writing the expected behaviour a judge will grade against. Do it in code or from the CLI.

q.accept(failure_id=42,
         cohort_path="evals/regressions.yaml",
         expect="states the correct due date from the ledger, not a guess")

# list what's pending, then promote or reject by id
zolva triage failures.db
zolva triage failures.db --accept 42 \
  --cohort evals/regressions.yaml \
  --expect "states the correct due date from the ledger, not a guess"
zolva triage failures.db --reject 43      # not a real defect

The captured transcript carries the exact input that failed, so the regression case reproduces the real conversation, not a paraphrase of it.

Step 3 · Gate the fix, then re-verify weekly #

Now the regression cohort is an ordinary gated eval. Run it in CI so no change ships without passing it, and on a weekly cron so provider drift can't quietly reintroduce the bug.

zolva eval evals/ --app app:app --gate \
  --judge-provider anthropic --judge-model claude-neutral

A great average never rescues a failing cohort: the gate is on the worst cohort, so a single unfixed regression fails the release. Wire this into the pipeline in the CI-gated releases playbook.

Step 4 · Export the dataset for fine-tuning #

Accepted failures are also a labelled dataset. Export them as JSONL for supervised or preference fine-tuning, with PII masked on the way out.

zolva export-dataset failures.db dataset.jsonl \
  --redaction policies/redaction.yaml

The training itself lives in your stack; Zolva gives you the clean, de-identified on-ramp. See the redaction playbook for the pattern file.

Why the loop closes #

• Auto-capture: escalations and guardrail blocks enter the queue with no extra code; you only hand-record softer signals like a thumbs-down.
• Human-in-the-loop promotion: a person writes the expected behaviour, so the regression encodes judgement, not just "don't do that again".
• Gated forever: once promoted, the case is a normal cohort under the worst-cohort gate; the fix is protected on every PR and every weekly run.
• De-identified export: the dataset on-ramp masks PII, so training data never carries raw customer identifiers out of your systems.

← All playbooks