Evaluations

Evaluations in Plexus are how you validate and assess your scorecards to ensure they align with your policies and stakeholder needs. They help you measure the effectiveness and accuracy of your scoring criteria before deploying them to production.

What are Evaluations?

An evaluation is like a machine learning evaluation process - it's how you test and validate your scorecards against known correct answers. This helps ensure your scoring criteria are properly calibrated and will produce reliable results when deployed.

Evaluation Components

Each evaluation consists of:

  • Test Dataset: A set of content with known correct answers
  • Scorecard: The scoring criteria being evaluated
  • Results: How well the scorecard's predictions match the known correct answers
  • Metrics: Performance indicators like accuracy, precision, and recall

Evaluation Process

When you run an evaluation:

  • Your scorecard is applied to a test dataset with known correct answers
  • The scorecard's predictions are compared against the ground truth
  • Performance metrics are calculated to measure accuracy and reliability
  • A comprehensive report helps you identify areas for improvement

Example Evaluation

Here's an example of what a scorecard evaluation looks like in Plexus:

Evaluation
scorecard-123
score-123
Elapsed:
$0.5000 total ยท $0.005000/item
Processed 100 of 100 items
Complete
100 / 100100%
Labels: Binary
Balanced distribution
Positive
Negative
Predicted classes
Positive
Negative
Metrics
Goal:

This evaluation shows strong performance across key metrics.

achieve 90% accuracy
010060809095
85%
Accuracy
Confusion matrix
Positive
45
5
Positive
Negative
10
Negative
40
Predicted
Actual
Score Results
1%
Positive
High confidence prediction
95.0%

Understanding Results

Evaluation results help you understand how well your scorecard performs and where it needs improvement:

Score Results

For each score in your scorecard, you get:

  • How well the predictions match known correct answers
  • Detailed explanations of where and why mismatches occurred
  • Confidence levels to identify uncertain predictions
  • Insights for improving score accuracy

Performance Metrics

Overall evaluation metrics help you assess scorecard quality:

  • Accuracy, precision, and recall statistics
  • Performance trends as you refine your scorecard
  • Comparison with baseline benchmarks
  • Quality indicators to guide improvements

Misclassification Triage

Score-Configuration RCA includes a misclassification triage layer that classifies each incorrect item into one operator-facing category: score configuration problem, information gap, guideline gap requiring SME clarification, or mechanical malfunction.

Dataset-backed accuracy evaluations also run this RCA automatically. Coverage is reported as full/partial/none based on how many incorrect items retain feedback_item_id linkage.

Dataset-backed accuracy requires a materialized associated dataset: DataSet.file must point to parquet/csv. Preflight rejects non-materialized datasets early with explicit reason codes so optimizer agents can rebuild before dispatching evaluations.

Evaluation detail views also surface category summaries, mechanical subtype breakdown, evaluation-level red flags, and one primary next action recommendation so you can quickly decide whether to continue score optimization or route work to data remediation, SME clarification, or system debugging.

Open a topic in the RCA section to drill into item-level category, rationale, confidence, and evidence snippets, then select the corresponding score result for full item context.

Reliable Feedback Runner

For feedback optimization loops, use plexus evaluate feedback-runner instead of calling plexus evaluate feedback directly. The runner captures evaluation ID by runner task ID, waits for terminal backend status, and summarizes final metrics + RCA from the evaluation record.

plexus evaluate feedback-runner \
  --scorecard 1039 \
  --score 45425 \
  --days 180 \
  --max-items 50 \
  --sampling-mode random \
  --kanbus-issue-id plx-9aa370

When --kanbus-issue-id is provided, the runner writes a standardized run summary comment with evaluation ID, processed/total counts, AC1/accuracy/precision/recall, RCA presence, and next-action context.

Using Evaluations

Evaluations are essential tools for developing reliable scorecards. Use them to:

  • Validate that scorecards align with your policies and requirements
  • Identify and fix biases or gaps in scoring criteria
  • Track scorecard improvement over time
  • Build confidence in your scoring system before deployment