Deterministic Associated Datasets

Quickstart

Canonical two-step flow: build an associated dataset, then evaluate against the latest one.

# 1) Build associated dataset from recent feedback
plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100 \
  --days 180

# 2) Evaluate against latest associated dataset for that score
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset

# Optional machine-friendly output
plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --json-only

What They Are

• Single-score datasets with fixed labels.
• Built from curated inputs (for example, scorecard examples and linked feedback/score results).
• Stored as regular DataSet records with deterministic provenance metadata.
• Associated to a score by scoreId; any score-linked dataset is an associated dataset.

Why They Matter

• Eliminate drift from changing upstream records during optimization loops.
• Make before/after comparisons auditable and repeatable.
• Enable stable acceptance policy decisions when combined with random-sample generalization runs.

Label Resolution Policy

Two deterministic build paths are available:

• dataset load path: uses deterministic label-source priority and reports skipped IDs.
• score dataset-curate path: scans qualifying feedback newest-first, uses FeedbackItem.finalAnswerValue labels, and balances class coverage by default.
• score dataset-curate-vetted path: runs aligned guideline-vetting report evidence and curates from vetted-good feedback with balancing.

Build and Assign

Build from a DataSource configuration:

plexus dataset load \
  --source <data_source_identifier> \
  --deterministic-order

Or curate directly from qualifying feedback labels:

plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100 \
  --days 180

Balancing is enabled by default. Use --no-balance to keep pure recency sampling.

plexus score dataset-curate \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --max-items 100 \
  --no-balance

Or run the canonical vetted workflow (report evidence first, then dataset build):

plexus score dataset-curate-vetted \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --days 180 \
  --max-items 100

This command auto-runs an aligned guideline-vetting report, uses unanimously non-contradicting items in newest-first order, applies balancing, and returns both report and dataset IDs.

Use in Evaluations

Run accuracy evaluation directly against score-associated deterministic datasets:

plexus evaluate accuracy \
  --scorecard "Example Scorecard" \
  --score "Identify Objections" \
  --use-score-associated-dataset \
  --all-score-associated-datasets \
  --number-of-samples 200

Without --all-score-associated-datasets, the latest associated dataset is used.

Add --json-only for machine-friendly output payloads.

Each new dataset-backed evaluation persists dataset provenance and exposes direct links in the evaluation detail view.

How Associated Datasets Fit Optimization Workflow

Associated datasets are the deterministic stage in the optimization workflow and should be paired with random-sample checks to measure generalization.

• Stage A: deterministic associated-dataset evaluation (reference signal).
• Stage B: random n=50 evaluation for fast iteration feedback.
• Stage C: random n=200 hard gate before final accept.

Workflow outputs persist a canonical assessment bundle with stage IDs, deltas, malfunction routing context, and generalization-gap/stability metrics.

Data Routes

Sources and datasets are exposed as separate routes for clarity.

• Default Data entrypoint: /lab/datasets
• Sources: /lab/data/sources
• Datasets inventory: /lab/datasets
• Dataset detail: /lab/datasets/<datasetId>

Datasets Filter Modes

The datasets inventory at /lab/datasets uses explicit filter modes with standard selectors and index-backed queries only.

• All: account-wide newest-first dataset list.
• By Score: scorecard/score selectors with optional score-version selector.
• By Source Version: data source selector, then source-version selector.
• No raw ID text entry is required in the normal filter flow.

Score Page Visibility

The score detail page includes an Associated Datasets panel that lists datasets linked by scoreId and shows:

• Dataset creation recency
• Row count from persisted build metadata
• Label distribution as a segmented bar

Older datasets without persisted stats are shown with an explicit “stats unavailable” message.

Related