Feedback Alignment Optimizer

An automated procedure that iteratively improves score configurations through hypothesis-driven experimentation, dual-metric evaluation, and cross-cycle learning.

What Does It Do?

The Feedback Alignment Optimizer takes a score that isn't performing well and systematically improves it. It works by:

  1. Establishing a baseline measurement of current performance
  2. Analyzing why the score is making mistakes (root cause analysis)
  3. Proposing targeted hypotheses to fix specific error patterns
  4. Implementing and testing each hypothesis
  5. Accepting improvements and rejecting regressions
  6. Repeating until performance converges or the cycle limit is reached

At the end, it produces three outputs: an Executive Summary (plain English for any audience), a Lab Report (technical action list for operators), and an SME Agenda (decision-formatted meeting agenda for domain experts).

The Optimization Cycle

Each cycle has four phases:

Phase 1: Planning

The optimizer reviews all prior cycle results, the current root cause analysis, known contradictions, and item recurrence patterns. It then proposes multiple hypotheses targeting different error families. The number of hypotheses scales down after consecutive failures (4 → 2 → 1) to become more conservative.

Phase 2: Implementation

Each hypothesis is implemented by editing the score's YAML configuration. The optimizer uses a ReAct loop (view → edit → submit → verify) with up to 10 steps per hypothesis. After submission, a smoke test catches runtime errors immediately.

Phase 3: Evaluation

All submitted versions are evaluated on two independent datasets: a fixed regression dataset (same items every cycle) and the latest human feedback. If no hypothesis succeeds individually, the optimizer attempts to synthesize the best ideas from multiple hypotheses into a combined version.

Phase 4: Review

The winning version is checked against regression thresholds. If it doesn't regress either metric beyond safety limits, it becomes the new baseline for the next cycle. The decision is metric-based, not subjective.

Dual Metrics

The optimizer tracks two metrics simultaneously to prevent “whack-a-mole” improvements where fixing one set of items breaks another:

Accuracy (Regression Dataset)

A fixed dataset built at baseline. The same items are tested every cycle. This prevents regression — if a change helps some items but hurts these known-good items, it's rejected.

Feedback (Alignment)

The latest human feedback items. This measures real-world improvement — are we getting closer to what human reviewers expect?

Both use AC1 (Gwet's agreement coefficient) as the primary metric. A version is accepted only if neither metric regresses beyond safety thresholds.

When It Stops

The optimizer stops for one of these reasons:

  • Target reached — AC1 hit the target accuracy (default 0.95)
  • Max iterations — Completed all requested cycles
  • Improvement plateau — 2 consecutive cycles with minimal improvement
  • Early stop — 5+ consecutive failed cycles (nothing works)
  • User stop — Human sent a stop signal via the dashboard

Continuing and Branching

After a run completes, you have two options for extending the work:

Continue

Resume from where the optimizer left off with additional cycles. All accumulated state is preserved — no re-baselining, no lost context. Use this when the improvement trend is still positive but the run hit its cycle limit.

Access via the ellipsis menu (⋯) on the procedure card in the dashboard.

Branch

Fork from a specific earlier cycle to try a completely different approach. Creates a new procedure with state truncated to that cycle. The original procedure is unchanged. Use this to A/B test strategies or recover from a dead-end path.

Access via the Branch button in the expanded cycle details panel.

Understanding the Information Collected

Beyond improving accuracy, the optimizer collects rich diagnostic information about why a score can't improve further. This is often more valuable than the accuracy gains themselves.

Contradictions

Items where human reviewers gave conflicting labels for similar content. These create an optimization ceiling — the AI cannot achieve agreement higher than what humans achieve with each other. When the optimizer reports a contradiction ceiling, the path forward is to reconcile reviewer standards, not to tune the prompt further.

Item Recurrence Patterns

The optimizer tracks individual items across cycles to identify systemic issues:

  • OSCILLATING (wrong → correct → wrong) — Classic sign of contradictory norms. Fixing one interpretation breaks another. Often indicates a label quality issue.
  • PERSISTENT (wrong in same way 3+ cycles) — Genuine policy gap or an item that's genuinely ambiguous and may need human review.
  • FLIP_FLOP (wrong in different ways) — Unstable classification boundary. The score is uncertain about this item.

Feedback Landscape Diagnostic

Every 2 cycles (after cycle 3), the optimizer generates a multi-level analysis of the error landscape: per-item diagnoses (why each item resists), cross-item patterns (anti-correlated groups where fixing one breaks another), temporal analysis (are errors shifting over time?), and systemic diagnosis (is this a prompt problem or a process problem?).

End-of-Run Outputs

When the optimizer finishes, it produces three outputs written for different audiences:

  • Executive Summary: 4-6 sentences of plain English prose. Covers what improved, the main blocker, and what decisions are needed. Copy-pasteable into an email or Slack message.
  • Lab Report: Technical analysis for operators and the next optimizer run. Includes what happened, why it stalled, error patterns, ceiling analysis, next lab actions (prompt/model/architecture), and suspected mislabeled items. Feeds into the next run as prior context.
  • SME Agenda: Meeting agenda for domain experts. Each item is a decision to make — phrased as a plain-English question with examples showing what the AI said vs. what reviewers expected, concrete options to choose from, and how many disputed reviews it would resolve. No jargon. No AC1 numbers.

Taking Action on Findings

Each of the three end-of-run outputs drives a different follow-up workflow:

From the Lab Report (operators)

Review flagged labels

The lab report identifies specific items suspected of having incorrect or contradictory labels. Review these with domain experts and invalidate or correct as appropriate, then re-run the optimizer.

Try structural changes

If the lab report identifies a structural limit (model capability, transcript format), consider input source changes, model upgrades, or architectural modifications that go beyond prompt tuning.

Update guidelines

When the lab report reveals genuine policy ambiguity (not just prompt wording issues), update the score's guidelines document and re-run.

From the SME Agenda (domain experts)

Run your next meeting from it

Forward the SME Agenda directly to your domain expert or team lead. Each agenda item is already formatted as a decision question with concrete examples and options — no translation needed.

Feed decisions back as hints

After the meeting, feed the decisions back into the next optimizer run using the hint parameter or by updating the score guidelines. This closes the feedback loop.

Address contradictions

If the contradiction ceiling is binding, the SME Agenda will surface the specific policy questions causing it. Resolving them unblocks further optimization and may require reviewer calibration.

Thoughts on Improvement

The optimizer is evolving. Current areas of development:

  • Structured output — Making the diagnostic and prescription machine-parseable so downstream workflows can trigger automatically
  • Cross-run learning — Automatically chaining one run's findings into the next run's context, without manual intervention
  • Contradiction resolution workflow — Auto-creating review tasks for flagged items instead of just reporting them
  • Long-term pattern tracking — Persisting item recurrence data across runs to detect norm drift over weeks and months
  • Smarter convergence — Distinguishing “truly stuck” from “slow improvement on hard problems” to avoid premature stopping

Running the Optimizer

The optimizer can be triggered from the CLI, MCP tools, or the dashboard:

# CLI
plexus procedure run -y plexus/procedures/feedback_alignment_optimizer.yaml \
  -s scorecard="My Scorecard" \
  -s score="My Score" \
  -s max_iterations=10 \
  -s max_samples=200 \
  -s days=730

# Key parameters:
#   days          - Feedback lookback window (default: 90)
#   max_iterations - Cycle limit (default: 3)
#   max_samples   - Items per feedback evaluation (default: 200)
#   dry_run       - Test without promoting champion (default: false)
#   hint          - Expert guidance for the optimizer