Building a dataset

A dataset is the fixed set of documents an evaluation scores against. Each workflow has one, managed under Health → Datasets. This guide covers filling it, keeping its ground truth correct, and slicing it with tags. Datasets & evaluations defines the terms.

---

​

Adding documents

Two ways put documents in a dataset. Use the one that matches where your labelled data already lives.

​

From your processed results

Promote a document you’ve already run through the workflow:

1. Open its results and confirm every value is correct. Validate all of its datapoints. The correct values become the file’s ground truth.
2. Choose Add to dataset.

This duplicates the file into the dataset as an independent entity. From that point on, the dataset copy and the production file are separate. Editing one never changes the other.

​

By direct upload

If you already have labelled data, upload the documents and their expected answers straight into the dataset. You do not run them through the workflow first.

1. On the Datasets subtab, choose Upload files.
2. Drop your documents. To import expected values, include a .json ground-truth file named after each document. invoice.pdf pairs with invoice.json.
3. Optionally tag the uploaded files to slot them straight into a sub-dataset.
4. Review the staged list, then choose Upload.

Ground truth is optional on upload. Add documents alone and fill their values in later in the dataset table. anyformat pairs each .json to its document by filename, and checks its values against the workflow’s current fields. The review step catches problems before anything is uploaded. It flags any file with a duplicate name, with invalid JSON, or that is an orphan ground-truth file matching no document. Dataset names must be unique, and anyformat never renames a file silently. A freshly added file, copied or uploaded, has ground truth but no score yet. It reads as ungraded until its first evaluation produces the prediction its ground truth is scored against.

---

​

Editing ground truth

Every evaluation measures against the dataset’s ground truth. Keeping it correct matters more than anywhere else. Edit expected values directly in Health → Datasets. You do not go back to the original run. The Ground truth grid shows the expected values for each file. An evaluation compares the workflow’s output to them.

---

​

Sub-datasets: slicing with tags

One overall accuracy number hides where a workflow struggles. Tag files to create sub-datasets, and read accuracy for each slice separately. Tag by whatever distinction matters for your documents:

• Provider: bank A against bank B
• Document type: invoices against receipts
• Difficulty: a “hard cases” tag for the documents that trip the workflow up

Read results overall or per sub-dataset, such as 95% overall but 60% on hard cases, and target the slice that needs work. Files with no tag make up the Untagged slice. When you run an evaluation, the scope you pick is either the full dataset or a single sub-dataset.

---

​

Removing documents

Select files in the Datasets grid and choose Remove to take them out of the dataset. Removal affects the dataset only. The production file is untouched. Removal is final. Add the file to the dataset again and you get a brand-new dataset entry with a clean slate. It is not linked to the removed one, and it carries none of the earlier evaluation history or ground-truth edits. Run an evaluation over the new entry to measure it.

---

​

What’s next?