Evaluating a Workflow

An eval is one graded run of a workflow against a target version: it re-extracts every document in scope, compares each result to that document’s ground truth, and freezes an accuracy headline you can trust to compare against other runs. The Health → Evaluations screen drives this from the app, where a run can be scoped to the whole dataset or a single sub-dataset. This recipe drives the same loop over the v3 API — which always evaluates the whole dataset — so you can wire it into CI, a nightly job, or a regression gate. The loop is four calls:

Upload files + ground truth into the workflow’s dataset — POST /v3/workflows/{workflow_id}/dataset/upload/.
Launch the eval — POST /v3/workflows/{workflow_id}/evals/ — and keep the returned eval_id.
Poll the eval until its status leaves in_progress — GET /v3/workflows/{workflow_id}/evals/{eval_id}/.
Read the headline (accuracy, matched / mismatched / ungraded), or list prior runs to compare over time.

When to use this

Regression gate in CI — run an eval on every workflow change and fail the build if accuracy drops below a threshold.
Nightly accuracy tracking — launch a run on a cron and chart accuracy per run_number over time.
Version comparison — evaluate two versions on the same dataset and diff their headlines.

Throughout, set ANYFORMAT_API_KEY and a WORKFLOW_ID (the target workflow’s UUID):

export ANYFORMAT_API_KEY=af_...
export WORKFLOW_ID=0686bb97-8c30-70f0-8000-97669e000eb8

Step 1 — Upload files + ground truth

Seed the dataset one document at a time. Each call uploads 1–10 files as a single document packet plus an optional ground_truth — the expected values for that document. Ground-truth keys are the workflow schema’s field persistent_ids; a scalar field maps to string | null, a table/object field to an array of row objects. Creation is all-or-nothing, so a rejected file or a bad ground-truth payload stores nothing.

curl -X POST "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/dataset/upload/" \
  -H "Authorization: Bearer $ANYFORMAT_API_KEY" \
  -H 'Idempotency-Key: 5f2b6c3e-8a1d-4f7e-9c0b-1a2b3c4d5e6f' \
  -F 'files=@/path/to/invoice_001.pdf' \
  -F 'ground_truth={"invoice_no": "INV-1", "total": "90.00"}'

from anyformat.sdk import Client, DatasetDocument

client = Client(api_key="YOUR_API_KEY")
workflow_id = "0686bb97-8c30-70f0-8000-97669e000eb8"

# One atomic call per document — there is no batch endpoint.
uploads = client.upload_documents_to_dataset(
    workflow_id,
    [
        DatasetDocument(file="invoice_001.pdf", ground_truth={"invoice_no": "INV-1", "total": "90.00"}),
        DatasetDocument(file="invoice_002.pdf", ground_truth={"invoice_no": "INV-2", "total": "120.00"}),
    ],
)
for u in uploads:
    print(u.document_packet_id, u.ground_truth_saved)

interface DatasetUpload {
  document_packet_id: string;
  files: { id: string; name: string; original_name: string | null }[];
  ground_truth_saved: boolean;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const formData = new FormData();
formData.append('files', fileInput.files[0]);
formData.append('ground_truth', JSON.stringify({ invoice_no: 'INV-1', total: '90.00' }));

const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/dataset/upload/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Idempotency-Key': crypto.randomUUID(),
    },
    body: formData,
  }
);
if (!response.ok) throw new Error(`API error: ${response.status}`);

const upload: DatasetUpload = await response.json();
console.log(upload.document_packet_id, upload.ground_truth_saved);

Ground truth attaches to the workflow’s current version. Fields the dataset has no ground truth for aren’t scored — they land as ungraded, never a failure. See Upload to Dataset for conflict handling (on_conflict) and multi-file packets.

Step 2 — Launch the eval

Launching is async and fire-and-forget: it enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns 202 immediately with an eval_id and run_number. Grading finalizes on a worker — you’ll poll for it in the next step. Omit version_id to evaluate the current version; pass one to pin a specific version.

The public API always evaluates the whole dataset — there is no way to narrow the scope from the endpoint. The app can additionally scope a run to a single sub-dataset, but that narrowing isn’t exposed on the API yet, so an API-launched eval always covers every dataset document.

curl -X POST "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/" \
  -H "Authorization: Bearer $ANYFORMAT_API_KEY" \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

launched = client.launch_eval(
    workflow_id,
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number, launched.enqueued_count)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Omit the body to evaluate the current version.
    body: JSON.stringify({}),
  }
);
if (!response.ok) throw new Error(`API error: ${response.status}`);

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

The response carries counts, never internal ids: enqueued_count documents were queued, failed_count failed to enqueue. If the dataset is empty (or every document fails to enqueue) the call returns 422 EMPTY_EVAL and no eval is created — so upload at least one document in Step 1 first.

Step 3 — Poll until grading finishes

GET .../evals/{eval_id}/ always returns 200 while the eval exists — there are no precondition errors while grading is in flight. Poll it until status is terminal:

`status`	Meaning	Headline
`in_progress`	Extractions running / grading not finalized	`null`
`processed`	Graded — terminal	populated
`error`	Run failed — terminal	`null`

# Repeat until "status" is "processed" or "error".
curl -X GET "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/$EVAL_ID/" \
  -H "Authorization: Bearer $ANYFORMAT_API_KEY"

import time

for _ in range(100):
    an_eval = client.get_eval(workflow_id, launched.eval_id)
    if an_eval.status == "processed":
        print(f"accuracy={an_eval.accuracy} "
              f"({an_eval.matched}/{an_eval.mismatched}/{an_eval.ungraded})")
        break
    if an_eval.status == "error":
        raise RuntimeError("Eval failed")
    time.sleep(5)

interface EvalDetail {
  id: string;
  run_number: number;
  version_id: string;
  status: 'in_progress' | 'processed' | 'error';
  accuracy: number | null;
  matched: number | null;
  mismatched: number | null;
  ungraded: number | null;
  file_count: number;
  failed_count: number;
  created_at: string | null;
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function pollEval(evalId: string): Promise<EvalDetail> {
  for (let i = 0; i < 100; i++) {
    const res = await fetch(
      `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/${evalId}/`,
      { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } }
    );
    if (!res.ok) throw new Error(`API error: ${res.status}`);
    const detail: EvalDetail = await res.json();
    if (detail.status !== 'in_progress') return detail;
    await sleep(5000);
  }
  throw new Error('Eval still in progress after timeout');
}

Polling is fine for a script or CI gate. For a production integration, prefer webhooks over a poll loop.

Step 4 — Read the headline

On a processed eval, accuracy is the matched / (matched + mismatched) fraction. It stays null when the graded denominator is 0 — nothing was gradable, e.g. every field was ungraded for lack of ground truth — so treat null accuracy as “no signal”, not “zero”.

Response (200 OK — processed)

{
  "id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "version_id": "abcdef1234",
  "status": "processed",
  "accuracy": 0.94,
  "matched": 47,
  "mismatched": 3,
  "ungraded": 1,
  "file_count": 42,
  "failed_count": 0,
  "created_at": "2026-07-05T10:00:00.000Z"
}

To compare over time, list the workflow’s evals newest-first — each item is the same headline, so a fresh page can mix graded and still-running runs:

curl -X GET "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/?limit=20" \
  -H "Authorization: Bearer $ANYFORMAT_API_KEY"

# iter_evals follows next_cursor across pages for you.
for an_eval in client.iter_evals(workflow_id):
    print(an_eval.run_number, an_eval.status, an_eval.accuracy)

interface EvalPage {
  items: EvalDetail[];
  next_cursor: string | null;
}

let cursor: string | null = null;
do {
  const base = `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`;
  const url = cursor ? `${base}?cursor=${cursor}` : base;
  const res = await fetch(url, { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } });
  if (!res.ok) throw new Error(`API error: ${res.status}`);
  const page: EvalPage = await res.json();
  for (const e of page.items) console.log(e.run_number, e.status, e.accuracy);
  cursor = page.next_cursor;
} while (cursor);

End-to-end

The whole loop as one runnable script — upload a folder of documents with their ground truth, launch, poll, and print the headline:

"""Evaluate a workflow against ground truth: upload -> launch -> poll -> read."""
import os
import time
import uuid

from anyformat.sdk import Client, DatasetDocument

client = Client(api_key=os.environ["ANYFORMAT_API_KEY"])
workflow_id = os.environ["WORKFLOW_ID"]

# A stable id for this run keeps both steps rerunnable: a retried CI job (same
# commit) replays the original upload packets and eval instead of duplicating
# them; an ad-hoc local run gets a fresh id, so it seeds and evaluates anew.
run_key = os.environ.get("GITHUB_SHA") or uuid.uuid4().hex

# 1. Upload documents + ground truth — one atomic call per document. A stable
#    per-document idempotency key makes a rerun replay the original packet
#    rather than 409 on the duplicate filename.
documents = [
    DatasetDocument(file="invoice_001.pdf", ground_truth={"invoice_no": "INV-1", "total": "90.00"},
                    idempotency_key=f"{run_key}:invoice_001"),
    DatasetDocument(file="invoice_002.pdf", ground_truth={"invoice_no": "INV-2", "total": "120.00"},
                    idempotency_key=f"{run_key}:invoice_002"),
]
uploads = client.upload_documents_to_dataset(workflow_id, documents)
print(f"uploaded {len(uploads)} documents")

# 2. Launch an eval over the whole dataset on the current version. Keying the
#    launch to the same run id makes a retried job replay the original eval
#    instead of billing a second full-dataset run.
launched = client.launch_eval(workflow_id, idempotency_key=f"{run_key}:eval")
print(f"launched eval {launched.eval_id} (run #{launched.run_number}), "
      f"{launched.enqueued_count} enqueued")

# 3. Poll until grading is terminal.
while True:
    an_eval = client.get_eval(workflow_id, launched.eval_id)
    if an_eval.status != "in_progress":
        break
    time.sleep(5)

# 4. Read the headline.
if an_eval.status == "error":
    raise SystemExit("eval failed")
print(f"accuracy={an_eval.accuracy} "
      f"matched={an_eval.matched} mismatched={an_eval.mismatched} "
      f"ungraded={an_eval.ungraded} over {an_eval.file_count} documents")

# CI gate: fail the build on a regression. `accuracy` is None when nothing
# was gradable (no ground truth) — treat that as no signal, not a pass.
THRESHOLD = 0.90
if an_eval.accuracy is None or an_eval.accuracy < THRESHOLD:
    raise SystemExit(f"accuracy {an_eval.accuracy} below {THRESHOLD}")

Notes & gotchas

Every launch costs credits. Launching runs a fresh extraction per document; re-launching runs the whole cohort again. Metering follows How credits work.
Idempotency replays, it doesn’t dedupe by content. Without an Idempotency-Key every launch creates a new eval. Supplying the header replays: a retry with the same key returns the original eval instead of a second run. See Idempotency.
Pin a version for stable replays. When you omit version_id, a replayed key sent after a new version was published resolves to a different version and returns 422. Pass an explicit version_id when you need the replay to survive a version change.
Empty datasets fail fast. A launch with nothing to enqueue returns 422 EMPTY_EVAL and creates no eval — upload at least one document first.
Each run is immutable. Editing ground truth or refining the workflow never changes a past eval — always launch a fresh run to measure a change. That’s what makes the delta between two runs real signal.

Next steps

Launch Eval

Full launch contract — target version, idempotency, error codes.

Get Eval

The status model and frozen-headline response in detail.

List Evals

Keyset pagination over a workflow’s run history.

Evaluations in the app

Drive the same loop from Health → Evaluations, with per-file and per-field drill-down.

Get started

Build workflows

Studio

Coding assistant

Health

Recipes

Evaluating a Workflow

When to use this

Step 1 — Upload files + ground truth

Step 2 — Launch the eval

Step 3 — Poll until grading finishes

Step 4 — Read the headline

End-to-end

Notes & gotchas

Next steps

Launch Eval

Get Eval

List Evals

Evaluations in the app

​When to use this

​Step 1 — Upload files + ground truth

​Step 2 — Launch the eval

​Step 3 — Poll until grading finishes

​Step 4 — Read the headline

​End-to-end

​Notes & gotchas

​Next steps

Launch Eval

Get Eval

List Evals

Evaluations in the app

When to use this

Step 1 — Upload files + ground truth

Step 2 — Launch the eval

Step 3 — Poll until grading finishes

Step 4 — Read the headline

End-to-end

Notes & gotchas

Next steps