> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyformat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating a Workflow

> Score a workflow against ground truth over the v3 API — upload ground truth, launch an eval, poll its status, and read the accuracy headline

An [eval](/concepts/evals) is one graded run of a workflow against a target version: it re-extracts every document in scope, compares each result to that document's ground truth, and freezes an accuracy headline you can trust to compare against other runs. The [Health → Evaluations](/guides/health/evaluations) screen drives this from the app, where a run can be scoped to the whole dataset or a single sub-dataset. This recipe drives the same loop over the v3 API — which always evaluates the **whole dataset** — so you can wire it into CI, a nightly job, or a regression gate.

The loop is four calls:

1. **Upload** files + ground truth into the workflow's dataset — [`POST /v3/workflows/{workflow_id}/dataset/upload/`](/api-reference-v3/datasets/upload).
2. **Launch** the eval — [`POST /v3/workflows/{workflow_id}/evals/`](/api-reference-v3/evals/launch) — and keep the returned `eval_id`.
3. **Poll** the eval until its `status` leaves `in_progress` — [`GET /v3/workflows/{workflow_id}/evals/{eval_id}/`](/api-reference-v3/evals/get).
4. **Read** the headline (`accuracy`, `matched` / `mismatched` / `ungraded`), or [list prior runs](/api-reference-v3/evals/list) to compare over time.

## When to use this

* **Regression gate in CI** — run an eval on every workflow change and fail the build if accuracy drops below a threshold.
* **Nightly accuracy tracking** — launch a run on a cron and chart `accuracy` per `run_number` over time.
* **Version comparison** — evaluate two versions on the same dataset and diff their headlines.

Throughout, set `ANYFORMAT_API_KEY` and a `WORKFLOW_ID` (the target workflow's UUID):

```bash theme={null}
export ANYFORMAT_API_KEY=af_...
export WORKFLOW_ID=0686bb97-8c30-70f0-8000-97669e000eb8
```

## Step 1 — Upload files + ground truth

Seed the dataset one document at a time. Each call uploads 1–10 files as a single [document packet](/concepts/document-packets) plus an optional `ground_truth` — the expected values for that document. **Ground-truth keys are the workflow schema's field `persistent_id`s**; a scalar field maps to `string | null`, a table/object field to an array of row objects. Creation is all-or-nothing, so a rejected file or a bad ground-truth payload stores nothing.

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/dataset/upload/" \
    -H "Authorization: Bearer $ANYFORMAT_API_KEY" \
    -H 'Idempotency-Key: 5f2b6c3e-8a1d-4f7e-9c0b-1a2b3c4d5e6f' \
    -F 'files=@/path/to/invoice_001.pdf' \
    -F 'ground_truth={"invoice_no": "INV-1", "total": "90.00"}'
  ```

  ```python Python (SDK) theme={null}
  from anyformat.sdk import Client, DatasetDocument

  client = Client(api_key="YOUR_API_KEY")
  workflow_id = "0686bb97-8c30-70f0-8000-97669e000eb8"

  # One atomic call per document — there is no batch endpoint.
  uploads = client.upload_documents_to_dataset(
      workflow_id,
      [
          DatasetDocument(file="invoice_001.pdf", ground_truth={"invoice_no": "INV-1", "total": "90.00"}),
          DatasetDocument(file="invoice_002.pdf", ground_truth={"invoice_no": "INV-2", "total": "120.00"}),
      ],
  )
  for u in uploads:
      print(u.document_packet_id, u.ground_truth_saved)
  ```

  ```typescript TypeScript theme={null}
  interface DatasetUpload {
    document_packet_id: string;
    files: { id: string; name: string; original_name: string | null }[];
    ground_truth_saved: boolean;
  }

  const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
  const formData = new FormData();
  formData.append('files', fileInput.files[0]);
  formData.append('ground_truth', JSON.stringify({ invoice_no: 'INV-1', total: '90.00' }));

  const response = await fetch(
    `https://api.anyformat.ai/v3/workflows/${workflowId}/dataset/upload/`,
    {
      method: 'POST',
      headers: {
        'Authorization': 'Bearer YOUR_API_KEY',
        'Idempotency-Key': crypto.randomUUID(),
      },
      body: formData,
    }
  );
  if (!response.ok) throw new Error(`API error: ${response.status}`);

  const upload: DatasetUpload = await response.json();
  console.log(upload.document_packet_id, upload.ground_truth_saved);
  ```
</CodeGroup>

<Note>
  Ground truth attaches to the workflow's **current version**. Fields the dataset has no ground truth for aren't scored — they land as `ungraded`, never a failure. See [Upload to Dataset](/api-reference-v3/datasets/upload) for conflict handling (`on_conflict`) and multi-file packets.
</Note>

## Step 2 — Launch the eval

Launching is **async and fire-and-forget**: it enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns `202` immediately with an `eval_id` and `run_number`. Grading finalizes on a worker — you'll poll for it in the next step. Omit `version_id` to evaluate the **current version**; pass one to pin a specific version.

<Note>
  The public API always evaluates the **whole dataset** — there is no way to narrow the scope from the endpoint. The [app](/guides/health/evaluations) can additionally scope a run to a single **sub-dataset**, but that narrowing isn't exposed on the API yet, so an API-launched eval always covers every dataset document.
</Note>

<CodeGroup>
  ```bash curl theme={null}
  curl -X POST "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/" \
    -H "Authorization: Bearer $ANYFORMAT_API_KEY" \
    -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'
  ```

  ```python Python (SDK) theme={null}
  launched = client.launch_eval(
      workflow_id,
      idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
  )
  print(launched.eval_id, launched.run_number, launched.enqueued_count)
  ```

  ```typescript TypeScript theme={null}
  interface EvalLaunched {
    eval_id: string;
    run_number: number;
    enqueued_count: number;
    failed_count: number;
  }

  const response = await fetch(
    `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
    {
      method: 'POST',
      headers: {
        'Authorization': 'Bearer YOUR_API_KEY',
        'Content-Type': 'application/json',
        'Idempotency-Key': crypto.randomUUID(),
      },
      // Omit the body to evaluate the current version.
      body: JSON.stringify({}),
    }
  );
  if (!response.ok) throw new Error(`API error: ${response.status}`);

  const launched: EvalLaunched = await response.json();
  console.log(launched.eval_id, launched.run_number);
  ```
</CodeGroup>

The response carries **counts, never internal ids**: `enqueued_count` documents were queued, `failed_count` failed to enqueue. If the dataset is empty (or every document fails to enqueue) the call returns `422 EMPTY_EVAL` and **no eval is created** — so upload at least one document in Step 1 first.

## Step 3 — Poll until grading finishes

`GET .../evals/{eval_id}/` **always returns `200` while the eval exists** — there are no precondition errors while grading is in flight. Poll it until `status` is terminal:

| `status`      | Meaning                                     | Headline  |
| ------------- | ------------------------------------------- | --------- |
| `in_progress` | Extractions running / grading not finalized | `null`    |
| `processed`   | Graded — terminal                           | populated |
| `error`       | Run failed — terminal                       | `null`    |

<CodeGroup>
  ```bash curl theme={null}
  # Repeat until "status" is "processed" or "error".
  curl -X GET "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/$EVAL_ID/" \
    -H "Authorization: Bearer $ANYFORMAT_API_KEY"
  ```

  ```python Python (SDK) theme={null}
  import time

  for _ in range(100):
      an_eval = client.get_eval(workflow_id, launched.eval_id)
      if an_eval.status == "processed":
          print(f"accuracy={an_eval.accuracy} "
                f"({an_eval.matched}/{an_eval.mismatched}/{an_eval.ungraded})")
          break
      if an_eval.status == "error":
          raise RuntimeError("Eval failed")
      time.sleep(5)
  ```

  ```typescript TypeScript theme={null}
  interface EvalDetail {
    id: string;
    run_number: number;
    version_id: string;
    status: 'in_progress' | 'processed' | 'error';
    accuracy: number | null;
    matched: number | null;
    mismatched: number | null;
    ungraded: number | null;
    file_count: number;
    failed_count: number;
    created_at: string | null;
  }

  const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

  async function pollEval(evalId: string): Promise<EvalDetail> {
    for (let i = 0; i < 100; i++) {
      const res = await fetch(
        `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/${evalId}/`,
        { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } }
      );
      if (!res.ok) throw new Error(`API error: ${res.status}`);
      const detail: EvalDetail = await res.json();
      if (detail.status !== 'in_progress') return detail;
      await sleep(5000);
    }
    throw new Error('Eval still in progress after timeout');
  }
  ```
</CodeGroup>

<Tip>
  Polling is fine for a script or CI gate. For a production integration, prefer [webhooks](/api-reference/webhooks/overview) over a poll loop.
</Tip>

## Step 4 — Read the headline

On a `processed` eval, `accuracy` is the `matched / (matched + mismatched)` fraction. It stays `null` when the graded denominator is 0 — nothing was gradable, e.g. every field was `ungraded` for lack of ground truth — so treat `null` accuracy as "no signal", not "zero".

```json Response (200 OK — processed) theme={null}
{
  "id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "version_id": "abcdef1234",
  "status": "processed",
  "accuracy": 0.94,
  "matched": 47,
  "mismatched": 3,
  "ungraded": 1,
  "file_count": 42,
  "failed_count": 0,
  "created_at": "2026-07-05T10:00:00.000Z"
}
```

To compare over time, [list the workflow's evals](/api-reference-v3/evals/list) newest-first — each item is the same headline, so a fresh page can mix graded and still-running runs:

<CodeGroup>
  ```bash curl theme={null}
  curl -X GET "https://api.anyformat.ai/v3/workflows/$WORKFLOW_ID/evals/?limit=20" \
    -H "Authorization: Bearer $ANYFORMAT_API_KEY"
  ```

  ```python Python (SDK) theme={null}
  # iter_evals follows next_cursor across pages for you.
  for an_eval in client.iter_evals(workflow_id):
      print(an_eval.run_number, an_eval.status, an_eval.accuracy)
  ```

  ```typescript TypeScript theme={null}
  interface EvalPage {
    items: EvalDetail[];
    next_cursor: string | null;
  }

  let cursor: string | null = null;
  do {
    const base = `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`;
    const url = cursor ? `${base}?cursor=${cursor}` : base;
    const res = await fetch(url, { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } });
    if (!res.ok) throw new Error(`API error: ${res.status}`);
    const page: EvalPage = await res.json();
    for (const e of page.items) console.log(e.run_number, e.status, e.accuracy);
    cursor = page.next_cursor;
  } while (cursor);
  ```
</CodeGroup>

## End-to-end

The whole loop as one runnable script — upload a folder of documents with their ground truth, launch, poll, and print the headline:

```python theme={null}
"""Evaluate a workflow against ground truth: upload -> launch -> poll -> read."""
import os
import time
import uuid

from anyformat.sdk import Client, DatasetDocument

client = Client(api_key=os.environ["ANYFORMAT_API_KEY"])
workflow_id = os.environ["WORKFLOW_ID"]

# A stable id for this run keeps both steps rerunnable: a retried CI job (same
# commit) replays the original upload packets and eval instead of duplicating
# them; an ad-hoc local run gets a fresh id, so it seeds and evaluates anew.
run_key = os.environ.get("GITHUB_SHA") or uuid.uuid4().hex

# 1. Upload documents + ground truth — one atomic call per document. A stable
#    per-document idempotency key makes a rerun replay the original packet
#    rather than 409 on the duplicate filename.
documents = [
    DatasetDocument(file="invoice_001.pdf", ground_truth={"invoice_no": "INV-1", "total": "90.00"},
                    idempotency_key=f"{run_key}:invoice_001"),
    DatasetDocument(file="invoice_002.pdf", ground_truth={"invoice_no": "INV-2", "total": "120.00"},
                    idempotency_key=f"{run_key}:invoice_002"),
]
uploads = client.upload_documents_to_dataset(workflow_id, documents)
print(f"uploaded {len(uploads)} documents")

# 2. Launch an eval over the whole dataset on the current version. Keying the
#    launch to the same run id makes a retried job replay the original eval
#    instead of billing a second full-dataset run.
launched = client.launch_eval(workflow_id, idempotency_key=f"{run_key}:eval")
print(f"launched eval {launched.eval_id} (run #{launched.run_number}), "
      f"{launched.enqueued_count} enqueued")

# 3. Poll until grading is terminal.
while True:
    an_eval = client.get_eval(workflow_id, launched.eval_id)
    if an_eval.status != "in_progress":
        break
    time.sleep(5)

# 4. Read the headline.
if an_eval.status == "error":
    raise SystemExit("eval failed")
print(f"accuracy={an_eval.accuracy} "
      f"matched={an_eval.matched} mismatched={an_eval.mismatched} "
      f"ungraded={an_eval.ungraded} over {an_eval.file_count} documents")

# CI gate: fail the build on a regression. `accuracy` is None when nothing
# was gradable (no ground truth) — treat that as no signal, not a pass.
THRESHOLD = 0.90
if an_eval.accuracy is None or an_eval.accuracy < THRESHOLD:
    raise SystemExit(f"accuracy {an_eval.accuracy} below {THRESHOLD}")
```

## Notes & gotchas

* **Every launch costs credits.** Launching runs a fresh extraction per document; re-launching runs the whole cohort again. Metering follows [How credits work](/concepts/how-credits-work).
* **Idempotency replays, it doesn't dedupe by content.** Without an `Idempotency-Key` every launch creates a **new** eval. Supplying the header replays: a retry with the same key returns the **original** eval instead of a second run. See [Idempotency](/api-reference-v3/introduction#idempotency).
* **Pin a version for stable replays.** When you omit `version_id`, a replayed key sent *after a new version was published* resolves to a different version and returns `422`. Pass an explicit `version_id` when you need the replay to survive a version change.
* **Empty datasets fail fast.** A launch with nothing to enqueue returns `422 EMPTY_EVAL` and creates no eval — upload at least one document first.
* **Each run is immutable.** Editing ground truth or refining the workflow never changes a past eval — always launch a fresh run to measure a change. That's what makes the delta between two runs real signal.

## Next steps

<CardGroup cols={2}>
  <Card title="Launch Eval" icon="rocket" href="/api-reference-v3/evals/launch">
    Full launch contract — target version, idempotency, error codes.
  </Card>

  <Card title="Get Eval" icon="magnifying-glass" href="/api-reference-v3/evals/get">
    The status model and frozen-headline response in detail.
  </Card>

  <Card title="List Evals" icon="list" href="/api-reference-v3/evals/list">
    Keyset pagination over a workflow's run history.
  </Card>

  <Card title="Evaluations in the app" icon="chart-line" href="/guides/health/evaluations">
    Drive the same loop from Health → Evaluations, with per-file and per-field drill-down.
  </Card>
</CardGroup>