Skip to main content
Once a workflow is running, the next question is is it getting better or worse as I change it? Answering that reliably needs a fixed set of documents you know the correct answers for, and a repeatable way to score the workflow against them. anyformat gives you both, in the Health area of each workflow. This page defines the four terms you’ll see there. For how to use them, see the Datasets guide and the Evaluations guide.
The short version: a dataset is a pool of documents with known-correct answers (ground truth). An evaluation runs one workflow version over that dataset and scores it. Tag files into sub-datasets to see accuracy on slices like “hard cases” separately from the overall number.

​
Dataset

A dataset is a fixed collection of documents you use to measure a workflow’s quality. Each workflow has one dataset. The key property is that a dataset is independent from production:
  • Files enter a dataset two ways: promote a processed document with Add to dataset (which duplicates it into the dataset), or upload documents and their ground truth directly. Either way the dataset entry is its own independent entity.
  • Editing ground truth, removing a file from the dataset, or running an evaluation never affects your production data, and production runs never change the dataset.
This separation is what makes an evaluation reproducible: the documents being scored don’t move underneath you.
Datasets cover Extract and Classify workflows. Parse-only workflows aren’t evaluated.

​
Ground truth

Ground truth is the set of expected, correct values for a dataset file — the reference an evaluation scores the workflow’s output against.
  • Ground truth comes from one of two places: validating a processed file’s datapoints (the Add to dataset path), or supplying it directly as a .json file when you upload. Either way it’s editable in Health → Datasets.
  • On the Add to dataset path, a file must have every datapoint validated first — you’re establishing the correct answer, so it has to be complete. On the upload path ground truth is optional: supply it in the .json file, or leave it blank and fill it in later by editing the dataset table.
Accuracy is only ever measured against ground truth. A field the dataset has no ground truth for is not scored.

​
Sub-dataset (slice)

A sub-dataset is a slice of the dataset defined by tags. Tag files by anything meaningful — provider, document type, “hard cases” — and you can read results overall or per sub-dataset. This lets one dataset answer questions like “95% overall, but only 60% on hard cases” without splitting into separate datasets. Files with no tag form the Untagged slice.

​
Evaluation

An evaluation (or eval) is one scored run of one workflow version over a dataset scope — the full dataset, or a single sub-dataset. Running an evaluation:
  1. Runs an extraction on every in-scope dataset file.
  2. Scores each result against that file’s ground truth.
  3. Records the run with a status and an accuracy.
Evaluations are presented as a numbered run list — #1, #2, … — each with its scope, workflow version, date, status, accuracy, and file count.
An evaluation is an immutable historical record. Editing ground truth after Eval #1 does not change Eval #1 — you run Eval #2 to measure the effect. This is what makes version-to-version comparison trustworthy.

​
Accuracy

For an evaluation, accuracy is the share of graded fields that matched ground truth across the in-scope documents. Accuracy and confidence work the same way here as everywhere else in anyformat — confidence guides where to look, accuracy tells you how often the workflow is actually right. See Confidence & accuracy for the full explanation of both numbers.

​
How these fit together

Build a dataset

Add documents, edit ground truth, and tag files into sub-datasets.

Run an evaluation

Score a workflow version and compare accuracy across versions.