Module 4 — Evaluations and Optimization

Lesson notes

Evals turn agent development from intuition into engineering. A useful evaluation system contains representative cases, recorded traces, success criteria, graders, and a repeatable comparison between versions.

Evaluation stack

| Layer | Question | Examples |

|---|---|---|

| Outcome | Did the user goal succeed? | Resolution rate, answer correctness |

| Workflow | Was the path acceptable? | Steps, loops, escalation, policy adherence |

| Component | Which stage failed? | Retrieval recall, routing accuracy, SQL validity |

| Operational | Is it viable? | Latency, token/tool cost, error rate |

Use deterministic graders for schemas, exact computations, tests, and policy rules. Use rubric-based model graders for open-ended qualities, calibrated against human labels. Keep humans in the loop for high-impact or genuinely subjective decisions.

Error analysis

Do not optimize only an average score. Inspect failed traces, assign a failure taxonomy, and count patterns. Examples: misunderstood intent, missing context, bad retrieval, wrong tool, invalid arguments, poor synthesis, unsupported claim, policy violation, and premature stopping.

Prioritize with an explicit score such as:

priority = frequency × impact × confidence-in-diagnosis ÷ effort

Fix the largest reliable bottleneck, rerun the full suite, and check for regressions.

Component-level evaluation

End-to-end failure does not reveal the weak stage. In a research workflow, evaluate query generation, source quality, evidence extraction, citation alignment, synthesis, and final instruction following separately. Component tests are faster and make remedies more specific.

Choosing the remedy

Missing instruction → improve prompt or contract.
Missing knowledge → retrieve better context.
Wrong or stale fact → use a trusted tool/source.
Inconsistent format → schema plus validator.
Weak reasoning → decompose, reflect, or use a stronger model selectively.
Tool failure → improve schema, examples, error messages, or runtime.
Policy risk → reduce permissions or add approval gates.
Unsolvable ambiguity → ask the user.

Latency and cost

Measure before optimizing. Reduce unnecessary calls, parallelize independent reads, cache stable results, shorten context, route easy cases to smaller models, use early exits, batch requests, and cap loops. Never lower correctness or safety silently; compare on the same eval set.

Development loop

flowchart LR A[Define task] --> B[Build baseline] B --> C[Run evals and traces] C --> D[Classify failures] D --> E[Fix top bottleneck] E --> F[Regression and cost check] F --> C

This is the measurement layer for Spec Map Build Prove Learn.

Module 4 — Evaluations and Optimization