Evaluate a Score

Quickstart: Evaluate Latest Associated Dataset

# Build/update dataset for the score
plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100 \
  --days 180

# Evaluate against the latest associated dataset
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset

# Optional machine-friendly output
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --json-only

Dataset-backed evaluate accuracy runs now include RCA automatically. RCA is persisted under parameters.root_cause using the same pipeline as feedback-backed runs.

Materialized dataset requirement: dataset-backed accuracy preflight requires DataSet.file to point to a parquet/csv object. If missing, evaluation fails immediately with missing_file_pointer or unsupported_file_type and a rebuild hint.

Operator Loop: Build -> Check -> Evaluate

# 1) Build associated dataset from feedback
plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 200 \
  --days 180

# 2) Evaluate baseline/candidate against dataset
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --baseline <evaluation-id>

In MCP workflows, call plexus_dataset_check_associated before dispatching optimizer runs. If readiness fails, rebuild the dataset first.

Running an Evaluation

You can evaluate content using individual scores or entire scorecards. The evaluation process analyzes your content against the defined criteria and provides detailed results.

from plexus import Plexus

plexus = Plexus(api_key="your-api-key")

# Evaluate using a specific score (accepts ID, name, key, or external ID)
evaluation = plexus.evaluations.create(
    source_id="source-id",
    score="Grammar Check"  # Can use name, key, ID, or external ID
)

# Or evaluate using an entire scorecard (accepts ID, name, key, or external ID)
evaluation = plexus.evaluations.create(
    source_id="source-id",
    scorecard="Content Quality"  # Can use name, key, ID, or external ID
)

# Get evaluation results
results = evaluation.get_results()

# Print score values
for score in results.scores:
    print(f"{score.name}: {score.value}")

# Evaluate using a scorecard
plexus evaluate accuracy --scorecard "Content Quality" --number-of-samples 100

# List evaluation results
plexus evaluations list

# View detailed results for a specific evaluation
plexus evaluations list-results --evaluation evaluation-id

Understanding Results

Misclassification Triage Layer

Feedback-backed evaluations include a misclassification triage layer so operators can quickly decide whether to optimize score logic, escalate guideline questions, fix upstream data quality, or investigate system issues.

Per-item categories

• score_configuration_problem: likely fixable in score YAML/prompt logic.
• information_gap: primary input/context evidence is insufficient or degraded.
• guideline_gap_requires_sme: rubric ambiguity that needs SME clarification.
• mechanical_malfunction: execution/system failure patterns.

Evaluation-level red flags

• prediction_mode_collapse: misclassifications all predicted as one class.
• mechanical_failures_present: one or more items classified as mechanical malfunction.
• low_primary_input_coverage: at least half of analyzed items lacked primary-input context.

How to use it in the dashboard

1. Open evaluation detail and review the segmented category breakdown plus red flags.
2. Expand RCA topics to inspect item-level category, confidence, rationale, and evidence snippets.
3. Select a score result to see the same misclassification triage context in item detail view.

Agent-facing payload contract

• misclassification_analysis.item_classifications_all is the authoritative per-item triage table for coding agents.
• misclassification_analysis.analysis_scope explicitly reports coverage and assignment scope.
• analysis_scope.topic_assignment_scope is currently exemplar_only; topic IDs are only guaranteed for exemplars exposed by semantic memory.
• misclassification_analysis.category_diagnostics.information_gap explains missing/degraded primary-input and missing-required-context signals when information-gap dominates.
• Use item_classifications_all for optimization decisions and next-action logic; use topics for semantic drill-down and narrative context.

How to inspect specifics

1. Use category summary cards to review representative evidence rows (item, source, quote).
2. Click View items in score results on a category card to filter score results to that category.
3. Use Clear category filter to return to the full score-results list.

Batch Evaluations

You can evaluate multiple sources at once using batch processing:

# Create a batch evaluation
batch = plexus.evaluations.create_batch(
    source_ids=["source-1", "source-2", "source-3"],
    scorecard="Quality Assurance"  # Can use name, key, ID, or external ID
)

# Monitor batch progress
status = batch.get_status()

# Get results when complete
results = batch.get_results()

Like individual evaluations, batch evaluations also support the flexible identifier system for scorecards and scores.

Deterministic Associated Dataset Workflow

For repeatable regression checks, build deterministic associated datasets and run evaluation directly against score-associated datasets.

# Build a deterministic associated dataset
plexus dataset load \
  --source <data_source_identifier> \
  --deterministic-order

# Or curate from qualifying feedback labels (newest-first)
plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100

# Disable balancing when pure recency sampling is preferred
plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100 \
  --no-balance

# Or run vetted+balanced curation with report evidence
plexus score dataset-curate-vetted \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --days 180 \
  --max-items 100

# Evaluate against latest associated dataset for this score
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --number-of-samples 200

# Same run with machine-friendly output
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --number-of-samples 200 \
  --json-only

# Evaluate across all associated datasets for this score
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --all-score-associated-datasets \
  --number-of-samples 200

Evaluation Provenance Links

Evaluation detail views now surface direct provenance links for the exact score version and dataset used by the run.

• Score Version links directly to the tested version when scoreVersionId is present.
• Dataset links directly to /lab/datasets/<id> when parameters.dataset_id is present.
• Legacy evaluations without stored provenance show Unavailable instead of inferred links.

The Data navigation lands on /lab/datasets by default; browse sources at /lab/data/sources.

On /lab/datasets, use the explicit filter modes (All, By Score,By Source Version) and standard selectors for score version and source version selection.

Optimization Workflow Runbook (50/200 Cadence)

Standard optimization workflow uses a fast random loop and a hard confirmation gate: n=50 for iteration speed, then n=200 before final accept decisions.

# Stage A: deterministic associated-dataset check
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --number-of-samples 200

# Stage B: fast random loop
plexus evaluate feedback-runner \
  --scorecard 1039 \
  --score 45425 \
  --days 180 \
  --max-items 50 \
  --sampling-mode random \
  --kanbus-issue-id plx-9aa370

# Stage C: hard random gate before accept
plexus evaluate feedback-runner \
  --scorecard 1039 \
  --score 45425 \
  --days 180 \
  --max-items 200 \
  --sampling-mode random \
  --kanbus-issue-id plx-9aa370

Use feedback-runner for optimization loops. It captures evaluation ID by runner task ID, waits on backend evaluation status, and writes a standardized run summary comment to Kanbus when --kanbus-issue-id is provided.

This avoids trusting local process exit and treats the evaluation record as the source of truth for completion, metrics, and RCA availability.

Workflow assessments are recorded as canonical bundles (candidate_assessment_bundle.v1) with:

• Identity (scorecard, score, baseline version, candidate version)
• Stage runs (evaluation IDs, sample protocol, status, deltas)
• Malfunction context (category shares, red flags, primary next action)
• Generalization metrics (gap and stability signals)
• Decision outputs (policy decision + workflow routing decision)

Full bundle payload is persisted as an attachment; compact summary fields are persisted for fast dashboard queries.

Interpreting Workflow KPIs and Decisions

Evaluation detail now includes a Candidate assessment panel with compact workflow evidence. Use it to quickly decide whether to keep optimizing score configuration or reroute to data/SME/system work.

How to read stage evidence

• Deterministic reference: associated-dataset check of baseline vs candidate.
• Random iteration (n=50): fast loop signal for iterative tuning.
• Random gate (n=200): required confirmation stage before accept decisions.
• Review per-stage baseline/candidate status and delta values before trusting the decision.

How to read generalization KPIs

• Baseline gap and Candidate gap: reference AC1 minus random-mean AC1 for each version.
• Gap delta: candidate gap minus baseline gap. Positive means candidate generalized worse.
• Random delta mean/stddev: average and stability of candidate-vs-baseline random deltas across random stages.
• Use these with policy thresholds (for example min_reference_delta=0.01, max_generalization_drop=0.02) for accept/reject outcomes.

Routing guidance from malfunction context

• score_configuration_optimization: continue YAML/code optimization loop.
• data_remediation: information gaps dominate; improve input artifact quality or coverage first.
• sme_guideline_clarification: guideline ambiguity dominates; resolve rubric with SMEs.
• bug_investigation: mechanical failures dominate; investigate runtime/system issues.

Keep deterministic associated-dataset runs and misclassification/RCA outputs together in review. Deterministic stages answer reproducibility, random stages answer generalization, and triage/RCA answers what to do next.

Coming Soon

Detailed documentation about evaluations is currently being developed. Check back soon for:

• Advanced evaluation options
• Custom result formatting
• Evaluation performance optimization
• Result analysis techniques