Datasets & evaluations

Once a workflow runs, you ask: is it getting better or worse as I change it? To answer, you need a fixed set of documents with known-correct answers, and a repeatable way to score the workflow against them. anyformat gives you both in the Health area of each workflow. This page defines the four terms you meet there. The Datasets guide and the Evaluations guide show how to use them. The short version: a dataset is a pool of documents with known-correct answers, called ground truth. An evaluation runs one workflow version over that dataset and scores it. Tag files into sub-datasets to read accuracy on a slice, such as “hard cases”, apart from the overall number.

---

​

Dataset

A dataset is a fixed collection of documents you use to measure a workflow’s quality. Each workflow has one dataset. The key property is that a dataset is independent from production:

• Files enter a dataset two ways. Promote a processed document with Add to dataset, which duplicates it into the dataset. Or upload documents and their ground truth directly. Either way the dataset entry is its own independent entity.
• Editing ground truth, removing a file from the dataset, or running an evaluation never affects your production data, and production runs never change the dataset.

This separation makes an evaluation reproducible. The scored documents do not move underneath you.

---

​

Ground truth

Ground truth is the set of expected, correct values for a dataset file. An evaluation scores the workflow’s output against it.

• Ground truth comes from one of two places. On the Add to dataset path you validate a processed file’s datapoints. On the upload path you supply it directly as a .json file. Either way you edit it in Health → Datasets.
• On the Add to dataset path, a file needs every datapoint validated first. You are establishing the correct answer, so it has to be complete. On the upload path ground truth is optional. Supply it in the .json file, or leave it blank and fill it in later in the dataset table.

Accuracy is only ever measured against ground truth. A field the dataset has no ground truth for is not scored.

---

​

Sub-dataset (slice)

A sub-dataset is a slice of the dataset defined by tags. Tag files by provider, document type, or “hard cases”. You then read results overall or per sub-dataset. One dataset then answers “95% overall, but only 60% on hard cases” without splitting into separate datasets. Files with no tag form the Untagged slice.

---

​

Evaluation

An evaluation, or eval, is one scored run of one workflow version over a dataset scope. The scope is the full dataset or a single sub-dataset. Running an evaluation:

1. Runs an extraction on every in-scope dataset file.
2. Scores each result against that file’s ground truth.
3. Records the run with a status and an accuracy.

anyformat lists evaluations as a numbered run list: #1, #2, and so on. Each row shows its scope, workflow version, date, status, accuracy, and file count.

---

​

Accuracy

For an evaluation, accuracy is the share of graded fields that matched ground truth across the in-scope documents. Accuracy and confidence work here as they do everywhere else in anyformat. Confidence guides where to look. Accuracy tells you how often the workflow is right. Confidence & accuracy explains both numbers.

---

​

How these fit together