> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyformat.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Get Eval Accuracy
> An eval's accuracy per field, per document and per dataset tag
*Rate limit tier: **general**, 600 requests/min. See [Rate limits](/api-reference-v3/introduction#rate-limits).*
Breaks one [eval](/concepts/evals) down: the totals, then the counts per field (`by_field`), per dataset document (`by_document_packet`) and per dataset tag (`by_dataset_tag`), plus `coverage`, the fields that have ground truth to grade against.
Every count is a number of values. A value is `matched` when it equals its ground truth, `mismatched` when it does not, and `ungraded` when there is no ground truth for it. Accuracy is `matched / (matched + mismatched)`; `ungraded` values stay out of it.
The eval grades the documents **it ran**, so a later eval, or a later run on the same version, never changes these numbers.
## While the eval runs
The numbers are **partial** while the eval's `status` is `in_progress`: they grade the documents finished so far, and documents still running count nothing yet. Read them once [`GET …/evals/{eval_id}/`](/api-reference-v3/evals/get) shows a terminal `status`.
## Reading the breakdown
* `by_field` names each field by its `persistent_id`, the identifier in the workflow's schema (see [`GET /v3/workflows/{workflow_id}/`](/api-reference-v3/workflows/get)).
* `by_document_packet` names each document by the **dataset copy's** `document_packet_id`. Adding a production packet to the dataset makes a copy with its own id; this is that id, not the production packet's. Pass it to [`GET …/document-packets/{document_packet_id}/fields/`](/api-reference-v3/evals/document-packet-fields) to see the values that differ. `failed_persistent_ids` lists the fields with at least one mismatched value; `name` is present only when the packet has a name of its own.
* `by_dataset_tag` gives the counts of the documents carrying each tag. A document counts under every tag it carries, so the rows do not add up to the totals.
An unknown eval, or one on another workflow or organization, returns `404 NOT_FOUND`.
```bash curl theme={null}
curl 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/069dcc2c-e14c-7606-8000-2ee4fb17b4f9/accuracy/' \
-H 'Authorization: Bearer YOUR_API_KEY'
```
```python Python (SDK) theme={null}
from anyformat.sdk import Client
client = Client(api_key="YOUR_API_KEY")
accuracy = client.get_eval_accuracy(
"0686bb97-8c30-70f0-8000-97669e000eb8", "069dcc2c-e14c-7606-8000-2ee4fb17b4f9"
)
for row in accuracy.by_field:
graded = row.matched + row.mismatched
if graded:
print(row.persistent_id, row.matched / graded)
```
```typescript TypeScript (SDK) theme={null}
import { Anyformat } from "@anyformat/sdk";
const af = new Anyformat({ apiKey: "YOUR_API_KEY" });
const accuracy = await af.getEvalAccuracy(
"0686bb97-8c30-70f0-8000-97669e000eb8",
"069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
);
const worst = accuracy.by_document_packet.filter((p) => p.mismatched > 0);
```
```json Response (200 OK) theme={null}
{
"matched": 47,
"mismatched": 3,
"ungraded": 1,
"by_field": [
{ "persistent_id": "0686bb978c3070f0800097669e00aaaa", "matched": 40, "mismatched": 2, "ungraded": 0 },
{ "persistent_id": "0686bb978c3070f0800097669e00bbbb", "matched": 7, "mismatched": 1, "ungraded": 1 }
],
"by_document_packet": [
{
"document_packet_id": "069dcc2c-e14c-7606-8000-2ee4fb17cccc",
"first_file_name": "invoice.pdf",
"file_count": 1,
"matched": 47,
"mismatched": 3,
"ungraded": 1,
"has_extraction": true,
"failed_persistent_ids": ["0686bb978c3070f0800097669e00aaaa"]
}
],
"coverage": {
"graded_persistent_ids": ["0686bb978c3070f0800097669e00aaaa", "0686bb978c3070f0800097669e00bbbb"],
"ungraded_persistent_ids": []
},
"by_dataset_tag": [
{ "dataset_tag_id": "069dcc2c-e14c-7606-8000-2ee4fb17aaaa", "matched": 47, "mismatched": 3 }
]
}
```
## OpenAPI
````yaml GET /v3/workflows/{workflow_id}/evals/{eval_id}/accuracy/
openapi: 3.1.0
info:
title: anyformat API
description: >
Document extraction and workflow automation API.
AnyFormat lets you define extraction workflows that pull structured data
from any document — PDFs, images, scanned files, or plain text. Upload a
file, run it through a workflow, and get back structured fields with
confidence scores and source evidence.
## Quick start
See the [Quick start
guide](https://docs.anyformat.ai/api-reference-v3/introduction) for a
walkthrough.
## Authentication
All endpoints (except `/health/`) require a Bearer token in the
`Authorization` header:
```
Authorization: Bearer
```
Get your API key from
[app.anyformat.ai/api-key](https://app.anyformat.ai/api-key).
## Versioning
Endpoints are versioned by path prefix (`/v3/`). Every response under
a versioned prefix includes `X-API-Version`.
version: 3.0.0
servers:
- url: https://api.anyformat.ai
description: API server
security: []
tags:
- name: workflows
description: >-
Workflows define extraction templates — what fields to extract from
documents. Create workflows, upload files, run extractions, and fetch
results.
- name: document-packets
description: >-
Document packets group uploaded documents for a workflow. Address a packet
directly to (re-)run its workflow.
- name: runs
description: >-
Extraction runs — the first-class result entity. Each run is a single
execution of a workflow against a document packet; fetch its status and
results flat by run id.
- name: splits
description: >-
Splits — the sub-documents a splitter node carves out of a packet.
Download any one of them as a PDF, built from the source documents on
request and never stored.
- name: parse
description: >-
Parse a single document with the platform's fast lite parse — a one-call
shortcut over the workflow upload + run primitives, backed by your
organization's system parse workflow.
- name: health
description: Health check endpoints to verify API availability.
paths:
/v3/workflows/{workflow_id}/evals/{eval_id}/accuracy/:
get:
tags:
- evals
summary: Get eval accuracy
description: >-
Retrieve an eval's accuracy per field, per document and per dataset tag.
Grades the documents the eval itself ran, so a later eval or run on the
same
version never changes these numbers. While the eval's `status` is
`in_progress` the numbers are partial: they grade the documents finished
so
far. Read them once `status` leaves `in_progress`.
`by_document_packet` names each document by the dataset copy's
`document_packet_id`, the id `GET
…/evals/{eval_id}/document-packets/{document_packet_id}/fields/`
takes. An unknown eval, or one on another workflow or organization,
returns 404.
operationId: v3_get_eval_accuracy
parameters:
- name: workflow_id
in: path
required: true
schema:
type: string
title: Workflow Id
- name: eval_id
in: path
required: true
schema:
type: string
title: Eval Id
responses:
'200':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/AccuracyV3'
headers:
X-RateLimit-Limit:
description: Requests allowed per window (60s).
schema:
type: integer
X-RateLimit-Remaining:
description: Requests remaining in the current window.
schema:
type: integer
X-RateLimit-Reset:
description: Seconds until the current window resets.
schema:
type: integer
'400':
description: >-
Validation failed — the request body or query parameters are
invalid.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
VALIDATION_ERROR:
value:
error: The request failed validation.
detail: The request failed validation.
error_code: VALIDATION_ERROR
retryable: false
request_id: req_abc123
'401':
description: >-
Authentication failed — the API key is not valid (`INVALID_API_KEY`)
or the request carries none (`MISSING_API_KEY`).
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
INVALID_API_KEY:
value:
error: The API key is not valid.
detail: The API key is not valid.
error_code: INVALID_API_KEY
retryable: false
request_id: req_abc123
MISSING_API_KEY:
value:
error: The request carries no API key.
detail: The request carries no API key.
error_code: MISSING_API_KEY
retryable: false
request_id: req_abc123
'403':
description: >-
Access denied — the API key lacks access to this resource or the
scope this call needs (`ACCESS_DENIED`,
`insufficient_api_key_scope`).
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
ACCESS_DENIED:
value:
error: You do not have access to this resource.
detail: You do not have access to this resource.
error_code: ACCESS_DENIED
retryable: false
request_id: req_abc123
insufficient_api_key_scope:
value:
error: The API key does not carry the scope this call needs.
detail: The API key does not carry the scope this call needs.
error_code: insufficient_api_key_scope
retryable: false
request_id: req_abc123
'404':
description: >-
Resource not found — the workflow, document packet, or run ID does
not exist or is not visible to this API key.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
NOT_FOUND:
value:
error: The resource does not exist.
detail: The resource does not exist.
error_code: NOT_FOUND
retryable: false
request_id: req_abc123
'429':
description: >-
Rate limited — retry after the `Retry-After` header. Default
windows: 60 requests/minute for extraction submission, 600
requests/minute for general endpoints.
headers:
Retry-After:
description: Seconds until the next request is allowed.
schema:
type: integer
X-RateLimit-Limit:
description: Requests allowed per window (60s).
schema:
type: integer
X-RateLimit-Remaining:
description: Requests remaining in the current window.
schema:
type: integer
X-RateLimit-Reset:
description: Seconds until the current window resets.
schema:
type: integer
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
RATE_LIMITED:
value:
error: Too many requests.
detail: Too many requests.
error_code: RATE_LIMITED
retryable: true
request_id: req_abc123
'500':
description: >-
An internal error stopped the request — `retryable=true`. Retry with
backoff.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
INTERNAL_ERROR:
value:
error: An internal error stopped the request.
detail: An internal error stopped the request.
error_code: INTERNAL_ERROR
retryable: true
request_id: req_abc123
'502':
description: >-
An upstream service failed. Retry with backoff when `retryable` is
true.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
INTERNAL_ERROR:
value:
error: An internal error stopped the request.
detail: An internal error stopped the request.
error_code: INTERNAL_ERROR
retryable: true
request_id: req_abc123
'503':
description: >-
An upstream service is unavailable. Retry with backoff when
`retryable` is true.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
INTERNAL_ERROR:
value:
error: An internal error stopped the request.
detail: An internal error stopped the request.
error_code: INTERNAL_ERROR
retryable: true
request_id: req_abc123
'504':
description: An upstream service did not answer in time. Retry with backoff.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
GATEWAY_TIMEOUT:
value:
error: The upstream service did not answer in time.
detail: The upstream service did not answer in time.
error_code: GATEWAY_TIMEOUT
retryable: true
request_id: req_abc123
security:
- ApiKeyAuth: []
components:
schemas:
AccuracyV3:
properties:
matched:
type: integer
title: Matched
description: Values that matched ground truth.
mismatched:
type: integer
title: Mismatched
description: Values that did not match ground truth.
ungraded:
type: integer
title: Ungraded
description: Values with no ground truth to grade against.
by_field:
items:
$ref: '#/components/schemas/FieldAccuracyV3'
type: array
title: By Field
description: Counts per field.
by_document_packet:
items:
$ref: '#/components/schemas/DocumentPacketAccuracyV3'
type: array
title: By Document Packet
description: Counts per document.
coverage:
$ref: '#/components/schemas/AccuracyCoverageV3'
by_dataset_tag:
items:
$ref: '#/components/schemas/DatasetTagAccuracyV3'
type: array
title: By Dataset Tag
description: Counts per dataset tag, ordered by tag name.
type: object
required:
- matched
- mismatched
- ungraded
- by_field
- by_document_packet
- coverage
- by_dataset_tag
title: AccuracyV3
description: >-
An accuracy breakdown: totals, per field, per document and per dataset
tag.
Accuracy is `matched / (matched + mismatched)`; `ungraded` values are
left
out of it.
ErrorEnvelope:
type: object
title: ErrorEnvelope
description: >-
Uniform error response. `retryable=true` means the same request may
succeed on retry (with backoff); `retryable=false` means the caller must
change the request before retrying.
required:
- error
- error_code
- retryable
- request_id
properties:
error:
type: string
description: Short human-readable summary.
detail:
description: >-
Machine-readable detail, any JSON value. A string for most errors, a
list of field errors for validation failures, and an object for a
structured error such as `FILENAME_CONFLICT` (`detail.conflicts`).
error_code:
type: string
enum:
- ACCESS_DENIED
- AGENTIC_MULTIFILE_UNSUPPORTED
- AUTH_FAILED
- CONCURRENT_EDIT
- DATASET_DOCUMENT_REJECTED
- DATASET_GROUND_TRUTH_SAVE_FAILED
- DATASET_INVALID_REQUEST
- EMPTY_EVAL
- ENDPOINT_RETIRED
- EXTRACT_MODE_FORBIDDEN
- EXTRACT_NODE_UNRESOLVED
- FILENAME_CONFLICT
- FILE_TOO_LARGE
- GATEWAY_TIMEOUT
- GRAPH_READBACK_FAILED
- GT_INVALID
- IDEMPOTENCY_KEY_REUSED
- IMMUTABLE_WORKFLOW
- INSUFFICIENT_CREDIT
- INTERNAL_ERROR
- INTERNAL_INCONSISTENCY
- INVALID_API_KEY
- INVALID_JSON
- INVALID_LOOKUP_FILE_ENCODING
- INVALID_VALUE
- INVALID_WORKFLOW_CONFIG
- KNOWLEDGE_NOT_ENABLED
- KNOWLEDGE_NOT_READY
- KNOWLEDGE_THREAD_MISMATCH
- KNOWLEDGE_UNSUPPORTED_SNAPSHOT_FORMAT
- LOOKUP_FIELD_WITHOUT_FILE
- METHOD_NOT_ALLOWED
- MISSING_API_KEY
- NOT_ACCEPTABLE
- NOT_FOUND
- NOT_IMPLEMENTED
- OBJECT_STORAGE_UNAVAILABLE
- OPERATOR_DEPRECATED
- OPERATOR_NOT_AVAILABLE
- PARSE_MODE_FORBIDDEN
- PAYMENT_REQUIRED
- PRECONDITION_FAILED
- RATE_LIMITED
- REQUEST_TOO_LARGE
- RUN_IN_PROGRESS
- SPLIT_HAS_NO_PAGES
- SPLIT_SOURCE_NOT_PDF
- SPLIT_WORKFLOW_DATASET_UNSUPPORTED
- TOPOLOGY_INVALID
- UNSUPPORTED_FILE_TYPE
- UNSUPPORTED_LOOKUP_FILE
- UNSUPPORTED_MEDIA_TYPE
- UPLOAD_SNIFF_BUDGET_EXCEEDED
- VALIDATION_ERROR
- WORKFLOW_CONCURRENT_EDIT
- WORKFLOW_NAME_CONFLICT
- ZERO_DATA_RETENTION_REFUSES_DATASET
- ZERO_DATA_RETENTION_REFUSES_KNOWLEDGE
- deprecated_operator_ack_required
- insufficient_api_key_scope
- terms_acceptance_required
description: >-
Stable identifier for programmatic handling. New values can be added
at any time, so a client must accept a code it does not know.
examples:
- VALIDATION_ERROR
- AUTH_FAILED
- NOT_FOUND
- PRECONDITION_FAILED
- RATE_LIMITED
- INTERNAL_ERROR
- GATEWAY_TIMEOUT
retryable:
type: boolean
description: True when the caller should retry (with backoff for 429/412/5xx).
request_id:
type: string
description: Correlates this response with server logs.
params:
type: object
additionalProperties: true
description: >-
Values specific to this occurrence of the error that a caller needs
to recover, such as `node_ids` or `required_scope`, keyed by name.
Absent when the code relays none.
doc_url:
type: string
format: uri
description: Link to the documentation of this `error_code`.
FieldAccuracyV3:
properties:
persistent_id:
type: string
title: Persistent Id
description: The field's `persistent_id` in the workflow schema.
matched:
type: integer
title: Matched
description: Values of this field that matched ground truth.
mismatched:
type: integer
title: Mismatched
description: Values of this field that did not match ground truth.
ungraded:
type: integer
title: Ungraded
description: Values of this field with no ground truth to grade against.
type: object
required:
- persistent_id
- matched
- mismatched
- ungraded
title: FieldAccuracyV3
description: Grading counts for one field across the graded documents.
DocumentPacketAccuracyV3:
properties:
document_packet_id:
type: string
title: Document Packet Id
description: >-
The dataset copy's document packet id (hyphenated UUID), not the id
of the production packet it was added from.
examples:
- 069dcc2c-e14c-7606-8000-2ee4fb17b4f9
first_file_name:
type: string
title: First File Name
description: Name of the document's first file.
name:
anyOf:
- type: string
- type: 'null'
title: Name
description: The document packet's own name. Absent when it has none.
file_count:
type: integer
title: File Count
description: Number of files in the document packet.
matched:
type: integer
title: Matched
description: Values that matched ground truth.
mismatched:
type: integer
title: Mismatched
description: Values that did not match ground truth.
ungraded:
type: integer
title: Ungraded
description: Values with no ground truth to grade against.
has_extraction:
type: boolean
title: Has Extraction
description: '`false` when the eval produced no extraction for this document.'
failed_persistent_ids:
items:
type: string
type: array
title: Failed Persistent Ids
description: '`persistent_id` of every field with at least one mismatched value.'
type: object
required:
- document_packet_id
- first_file_name
- file_count
- matched
- mismatched
- ungraded
- has_extraction
- failed_persistent_ids
title: DocumentPacketAccuracyV3
description: Grading counts for one dataset document.
AccuracyCoverageV3:
properties:
graded_persistent_ids:
items:
type: string
type: array
title: Graded Persistent Ids
description: Fields graded on at least one document.
ungraded_persistent_ids:
items:
type: string
type: array
title: Ungraded Persistent Ids
description: Fields graded on no document.
type: object
required:
- graded_persistent_ids
- ungraded_persistent_ids
title: AccuracyCoverageV3
description: Which fields have ground truth to grade against.
DatasetTagAccuracyV3:
properties:
dataset_tag_id:
type: string
title: Dataset Tag Id
description: The dataset tag's id (hyphenated UUID).
matched:
type: integer
title: Matched
description: Values that matched ground truth.
mismatched:
type: integer
title: Mismatched
description: Values that did not match ground truth.
type: object
required:
- dataset_tag_id
- matched
- mismatched
title: DatasetTagAccuracyV3
description: >-
Grading counts for the documents carrying one dataset tag.
A document counts under every tag it carries, so the rows do not add up
to
the totals.
securitySchemes:
ApiKeyAuth:
type: http
description: >-
API key issued from app.anyformat.ai/api-key. Send as `Authorization:
Bearer `.
scheme: bearer
````
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.