Skip to main content

Evaluate a prompt against a dataset

What you'll build: a dataset made from real feedback or from a test set you already keep, an experiment that runs one or more (version × model) combinations across it, and a report comparing them — so "did this prompt change actually help?" has a data-backed answer.

Requires the worker

Experiment runs are processed asynchronously by the AcruxCore worker process. The API calls below return immediately with a queued status; the worker performs the actual gateway calls and scoring in the background.

1. Collect feedback​

If you already have a test set in a CSV or JSON file, skip to step 2, from a file. Otherwise, start from signal on real traffic. On any trace (or span), leave a thumbs up/down and an optional comment — that comment becomes the example's grading criteria later. Feedback is only eligible for a dataset if its source trace captured payloads (so the prompt variables were stored). If your app renders the prompt itself and sends prompt_version_id with its own messages, send the values it rendered with as variables as well — those variables are what an example replays.

Feedback page

Your app can report feedback too, so a thumbs-down in your own UI lands here without anyone opening the dashboard. The trace id comes back from the call you are rating: the SDK returns it as result.gateway.traceId (result.trace_id from a tool loop), and a raw HTTP call reads it from the x-gateway-trace-id response header — see chat, stream, and collect feedback with the SDK for the full round trip. source records who the rating came from — end_user for your app's users, developer for your own team; it defaults to user.

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const feedback = await hub.traces.submitFeedback({
  traceId: '<trace-id>',
  rating: -1,
  comment: 'Too formal — be warmer',
  source: 'end_user',
});
console.log(feedback.id);

2. Build a dataset​

A dataset is a frozen set of example inputs, each with its own criteria: the prompt variables to render, plus what a judge should check in the reply. Evaluations → Datasets lists them, and a new workspace starts empty.

Empty Evaluations / Datasets page

A dataset can be filled in three ways. Steps 3 to 5 work the same for all three.

  • From feedback — select the feedback rows from step 1.
  • From a file — import a test set you already keep in a CSV or JSON file.
  • By hand — type each row in the dashboard.

From feedback​

Select feedback rows and turn them into a dataset. Each row brings the variables the call was made with, and its comment becomes the example's criteria.

import AcruxCore from '@acruxcoreai/sdk';

const hub = new AcruxCore();

const dataset = await hub.datasets.buildFromFeedback({
  name: 'unhappy-support-replies',
  overallFeedback: 'Warm, concise, always end with a next step',
  feedbackIds: ['<feedback-id-1>', '<feedback-id-2>'],
});
console.log(dataset.id, dataset.exampleCount);

From a file​

In the dashboard, New dataset creates an empty dataset. On that dataset, open Add example and choose the Import file tab. In a CSV, each column becomes a prompt variable, and a column named criteria holds the judge criteria:

question,customer_tier,criteria
How long do I have to return an item?,free,"Says 30 days from delivery, and that items must be unused."
Can I get a refund on an annual plan?,pro,Quotes the 14-day refund window for annual plans.

A JSON file is an array of objects in the same shape, and it may also carry a history of earlier chat turns. The tab shows a preview and lists any row with a problem by its row number. It imports up to 500 rows at a time.

From code, create the dataset and send the rows in one call. The call is all-or-nothing: if one row is invalid, nothing is written and the error names the row.

const dataset = await hub.datasets.create({ name: 'support-test-set' });

const result = await hub.datasets.addExamples(dataset.id, [
  { input: { question: 'How long do I have to return an item?' }, criteria: 'Says 30 days from delivery, and that items must be unused.' },
  { input: { question: 'Can I get a refund on an annual plan?' }, criteria: 'Quotes the 14-day refund window for annual plans.' },
]);
console.log(result); // { added: 2, exampleCount: 2 }

The full request and error shapes are in the API reference.

By hand​

New dataset creates an empty dataset, and Add example on it writes one row: the prompt variables, plus the criteria a judge should check. Use this path for a prompt that has no traffic yet and no test file.

3. Create and run an experiment​

An experiment pairs a dataset with a prompt under test and an explicit grid to sweep: one or more version_ids × one or more models. Creating it is one call; starting a run is another (it returns 202 queued and hands off to the worker).

const experiment = await hub.experiments.create({
  datasetId: '<dataset-id>',
  promptId: '<prompt-id>',
  name: 'support-reply v2 vs v1',
  versionIds: ['<v2-version-id>'],
  models: ['support-model'],
});

const run = await hub.experiments.startRun(experiment.id);
console.log(run.runId, run.status); // -> queued

When a prompt is under test, AcruxCore also adds an automatic baseline cell, so you always compare your candidate against something rather than nothing — by default that's the prompt's production alias, falling back to its latest committed version if there's no production alias yet. Pick a different alias in the dashboard's Baseline alias field (or pass "alias": "staging" over the API) if you want the comparison to be against staging, dev, or any other alias instead.

If the dataset's examples were captured against a different prompt than the one under test here, the run still starts — AcruxCore just adds a warning to the response and the run page so you know the comparison might not be apples-to-apples. It never blocks the run.

4. Read the report​

Poll the run until it finishes, then read the report — per-cell scores across the dataset, so you can see whether v2 beat v1 (or the baseline).

const detail = await hub.runs.get('<run-id>'); // status: queued → running → completed

const report = await hub.runs.getReport('<run-id>'); // per-cell scores

In the web app, Evaluations → Runs lists every run your team has made — newest first, with each run's score, best variant and duration — and each row opens the same report as a comparison grid. The same history is available over the API:

const { data: recentRuns, total } = await hub.runs.list({ limit: 20 });

const datasetRuns = await hub.runs.list({ datasetId: '<dataset-id>' });

5. Use this as a gate before you promote​

The automatic baseline cell from step 3 is what makes this a regression gate, not just a comparison: every run already scores your candidate version and whatever the baseline alias currently serves (production by default), side by side, on the same dataset. Read the report before promoting a version, not after:

  • Candidate clearly beats baseline → promote it, per Version a prompt.
  • Candidate ties or loses on one or more examples → that's a regression the dataset caught before a real user did. Don't promote; either fix the version or revisit the dataset's criteria if the loss looks like a scoring artifact rather than a real quality drop.

Because the dataset is frozen (built once in step 2), the comparison is stable — re-running the same experiment against a later candidate version still checks it against the same examples and the same criteria, so results are comparable across runs over time.

What's next​