Get Eval Accuracy

curl 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/069dcc2c-e14c-7606-8000-2ee4fb17b4f9/accuracy/' \
  -H 'Authorization: Bearer YOUR_API_KEY'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

accuracy = client.get_eval_accuracy(
    "0686bb97-8c30-70f0-8000-97669e000eb8", "069dcc2c-e14c-7606-8000-2ee4fb17b4f9"
)
for row in accuracy.by_field:
    graded = row.matched + row.mismatched
    if graded:
        print(row.persistent_id, row.matched / graded)

import { Anyformat } from "@anyformat/sdk";

const af = new Anyformat({ apiKey: "YOUR_API_KEY" });

const accuracy = await af.getEvalAccuracy(
  "0686bb97-8c30-70f0-8000-97669e000eb8",
  "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
);
const worst = accuracy.by_document_packet.filter((p) => p.mismatched > 0);

{
  "matched": 47,
  "mismatched": 3,
  "ungraded": 1,
  "by_field": [
    { "persistent_id": "0686bb978c3070f0800097669e00aaaa", "matched": 40, "mismatched": 2, "ungraded": 0 },
    { "persistent_id": "0686bb978c3070f0800097669e00bbbb", "matched": 7, "mismatched": 1, "ungraded": 1 }
  ],
  "by_document_packet": [
    {
      "document_packet_id": "069dcc2c-e14c-7606-8000-2ee4fb17cccc",
      "first_file_name": "invoice.pdf",
      "file_count": 1,
      "matched": 47,
      "mismatched": 3,
      "ungraded": 1,
      "has_extraction": true,
      "failed_persistent_ids": ["0686bb978c3070f0800097669e00aaaa"]
    }
  ],
  "coverage": {
    "graded_persistent_ids": ["0686bb978c3070f0800097669e00aaaa", "0686bb978c3070f0800097669e00bbbb"],
    "ungraded_persistent_ids": []
  },
  "by_dataset_tag": [
    { "dataset_tag_id": "069dcc2c-e14c-7606-8000-2ee4fb17aaaa", "matched": 47, "mismatched": 3 }
  ]
}

Rate limit tier: general, 600 requests/min. See Rate limits. Breaks one eval down: the totals, then the counts per field (by_field), per dataset document (by_document_packet) and per dataset tag (by_dataset_tag), plus coverage, the fields that have ground truth to grade against. Every count is a number of values. A value is matched when it equals its ground truth, mismatched when it does not, and ungraded when there is no ground truth for it. Accuracy is matched / (matched + mismatched); ungraded values stay out of it. The eval grades the documents it ran, so a later eval, or a later run on the same version, never changes these numbers.

While the eval runs

The numbers are partial while the eval’s status is in_progress: they grade the documents finished so far, and documents still running count nothing yet. Read them once GET …/evals/{eval_id}/ shows a terminal status.

Reading the breakdown

by_field names each field by its persistent_id, the identifier in the workflow’s schema (see GET /v3/workflows/{workflow_id}/).
by_document_packet names each document by the dataset copy’s document_packet_id. Adding a production packet to the dataset makes a copy with its own id; this is that id, not the production packet’s. Pass it to GET …/document-packets/{document_packet_id}/fields/ to see the values that differ. failed_persistent_ids lists the fields with at least one mismatched value; name is present only when the packet has a name of its own.
by_dataset_tag gives the counts of the documents carrying each tag. A document counts under every tag it carries, so the rows do not add up to the totals.

An unknown eval, or one on another workflow or organization, returns 404 NOT_FOUND.

curl 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/069dcc2c-e14c-7606-8000-2ee4fb17b4f9/accuracy/' \
  -H 'Authorization: Bearer YOUR_API_KEY'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

accuracy = client.get_eval_accuracy(
    "0686bb97-8c30-70f0-8000-97669e000eb8", "069dcc2c-e14c-7606-8000-2ee4fb17b4f9"
)
for row in accuracy.by_field:
    graded = row.matched + row.mismatched
    if graded:
        print(row.persistent_id, row.matched / graded)

import { Anyformat } from "@anyformat/sdk";

const af = new Anyformat({ apiKey: "YOUR_API_KEY" });

const accuracy = await af.getEvalAccuracy(
  "0686bb97-8c30-70f0-8000-97669e000eb8",
  "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
);
const worst = accuracy.by_document_packet.filter((p) => p.mismatched > 0);

{
  "matched": 47,
  "mismatched": 3,
  "ungraded": 1,
  "by_field": [
    { "persistent_id": "0686bb978c3070f0800097669e00aaaa", "matched": 40, "mismatched": 2, "ungraded": 0 },
    { "persistent_id": "0686bb978c3070f0800097669e00bbbb", "matched": 7, "mismatched": 1, "ungraded": 1 }
  ],
  "by_document_packet": [
    {
      "document_packet_id": "069dcc2c-e14c-7606-8000-2ee4fb17cccc",
      "first_file_name": "invoice.pdf",
      "file_count": 1,
      "matched": 47,
      "mismatched": 3,
      "ungraded": 1,
      "has_extraction": true,
      "failed_persistent_ids": ["0686bb978c3070f0800097669e00aaaa"]
    }
  ],
  "coverage": {
    "graded_persistent_ids": ["0686bb978c3070f0800097669e00aaaa", "0686bb978c3070f0800097669e00bbbb"],
    "ungraded_persistent_ids": []
  },
  "by_dataset_tag": [
    { "dataset_tag_id": "069dcc2c-e14c-7606-8000-2ee4fb17aaaa", "matched": 47, "mismatched": 3 }
  ]
}

Overview

Workflows

Upload

Document packets

Runs

Splits

Parse

Knowledge

Evals

SDKs and tools

API v2 (retired)

While the eval runs

Reading the breakdown

​While the eval runs

​Reading the breakdown

While the eval runs

Reading the breakdown