Module 2 — Reflection Design Pattern

Lesson notes

Reflection adds a feedback loop to generation. A generator produces an artifact; a critic checks it against explicit criteria or external evidence; the generator revises it. The critic may be the same model in a separate call, another model, a test suite, a code executor, a database, or a human.

flowchart LR I[Task and rubric] --> G[Generate] G --> C[Critique or execute] C --> D{Pass?} D -- no --> R[Revise] R --> C D -- yes --> O[Return artifact]

Why direct generation is insufficient

One-pass generation cannot react to its mistakes. Reflection helps when defects are detectable and revisions can repair them. It adds little when the critic has no stronger signal than the generator, the task is subjective without a rubric, or iteration cost exceeds the value.

Chart-generation workflow

Translate the request into chart requirements.
Generate data-transformation and plotting code.
Execute in a sandbox.
Inspect errors plus chart properties: labels, scales, units, missing data, readability.
Revise and repeat within a fixed budget.

The execution result is external feedback. A successful process exit is not enough; semantic and visual checks still matter.

Measuring reflection

Compare a direct baseline with the reflective workflow on the same dataset. Record success rate, severity-weighted defects, iterations, latency, and cost. Use blind human comparison for subjective output, and deterministic tests where possible. Keep cases where reflection makes the answer worse.

SQL-generation workflow

Use a read-only database role and supply only relevant schema context. Generate SQL, validate it, execute with a timeout and row limit, feed syntax or semantic errors back to the model, then inspect whether the result answers the question. Never treat executable SQL from a model as trusted input.

Stop conditions

Rubric or tests pass.
No material improvement between iterations.
Maximum iterations, tokens, time, or money is reached.
The same failure repeats.
Human approval is required.

Real-world patterns

Code assistant: generate patch → run tests/linter → diagnose → revise.
Document writer: draft → check rubric/facts/tone → revise.
Data analyst: generate query/chart → execute → inspect → revise.
Form processor: extract fields → validate against schema → repair or escalate.

Failure modes

A vague critic produces generic advice.
Generator and critic share the same blind spot.
Endless polishing increases cost without changing usefulness.
Revision fixes one criterion while breaking another.
Self-critique invents facts instead of obtaining evidence.

Mitigate with concrete rubrics, external validators, regression checks, iteration caps, and provenance for factual claims.

Module 2 — Reflection Design Pattern