> ## Documentation Index
> Fetch the complete documentation index at: https://docs.anyformat.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Launch Eval
> Launch a graded eval over a workflow's whole dataset on a target version
*Rate limit tier: **submission** (60 req/min) — see [Rate limits](/api-reference-v3/introduction#rate-limits).*
Launches an [eval](/concepts/evals) — one graded run of a workflow's **whole dataset** against a target version. The call is **async and fire-and-forget**: it enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns `202` immediately with an `eval_id` and `run_number`. Grading finalizes on a worker — poll the eval by its `eval_id` until its `status` leaves `in_progress`.
The response carries **counts, never internal ids**: `enqueued_count` (documents whose extraction was queued) and `failed_count` (documents that failed to enqueue). Read the eval by `eval_id` for the per-document breakdown.
## Target version
Omit `version_id` to evaluate the workflow's **current version**; pass one to override. A workflow with no version yet returns `404 NOT_FOUND`.
## Idempotency
Without an `Idempotency-Key`, **every launch creates a new eval** — re-launching runs a fresh cohort and is metered accordingly (see [How credits work](/concepts/how-credits-work)). Supplying the header **replays**: a retry with the same key returns the **original** eval instead of launching a second run. See [Idempotency](/api-reference-v3/introduction#idempotency).
When you omit `version_id`, a replayed key sent **after a new version was published** resolves to a different version than the first call, so the request no longer matches and returns `422`. Pass an explicit `version_id` when you need a stable replay across version changes.
## Errors
| Status | `error_code` | When |
| ------ | --------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `422` | `EMPTY_EVAL` | No dataset documents were enqueued — the dataset is empty, or every document failed to enqueue. No eval is created. |
| `400` | `OPERATOR_DEPRECATED` | The target version contains a deprecated operator and cannot run. |
| `404` | `NOT_FOUND` | Unknown workflow (including one in another organization), or the workflow has no version to evaluate. |
| `422` | `VALIDATION_ERROR` | An `Idempotency-Key` was reused with a different request. |
```bash curl theme={null}
# Current version — no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'
```
```python Python (SDK) theme={null}
from anyformat.sdk import Client
client = Client(api_key="YOUR_API_KEY")
launched = client.launch_eval(
"0686bb97-8c30-70f0-8000-97669e000eb8",
idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)
```
```typescript TypeScript theme={null}
interface EvalLaunched {
eval_id: string;
run_number: number;
enqueued_count: number;
failed_count: number;
}
const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
`https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
{
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
'Idempotency-Key': crypto.randomUUID(),
},
// Optional — pin a version. Omit the body to use the current version.
body: JSON.stringify({ version_id: 'abcdef1234' }),
}
);
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);
```
```json Response (202 Accepted) theme={null}
{
"eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
"run_number": 3,
"enqueued_count": 42,
"failed_count": 0
}
```
## OpenAPI
````yaml POST /v3/workflows/{workflow_id}/evals/
openapi: 3.1.0
info:
title: anyformat API
description: >
Document extraction and workflow automation API.
AnyFormat lets you define extraction workflows that pull structured data
from any document — PDFs, images, scanned files, or plain text. Upload a
file, run it through a workflow, and get back structured fields with
confidence scores and source evidence.
## Quick start
See the [Quick start
guide](https://docs.anyformat.ai/api-reference/introduction) for a
walkthrough.
## Authentication
All endpoints (except `/health/`) require a Bearer token in the
`Authorization` header:
```
Authorization: Bearer
```
Get your API key from
[app.anyformat.ai/api-key](https://app.anyformat.ai/api-key).
## Versioning
Endpoints are versioned by path prefix (`/v2/`, `/v3/`). Every response
under
a versioned prefix includes `X-API-Version`.
version: 2.0.0
servers:
- url: https://api.anyformat.ai
description: API server
security: []
tags:
- name: workflows
description: >-
Workflows define extraction templates — what fields to extract from
documents. Create workflows, upload files, run extractions, and fetch
results.
- name: files
description: >-
File collections group uploaded documents and track their extraction
progress. Upload files, check status, and retrieve extraction results.
- name: suggestions
description: >-
AI-powered schema and workflow suggestion endpoints for the AnyFormat
webapp. Authenticated with a Cognito JWT rather than an API key.
- name: document-packets
description: >-
Document packets group uploaded documents for a workflow (v3). Address a
packet directly to (re-)run its workflow.
- name: runs
description: >-
Extraction runs — the v3 first-class result entity. Each run is a single
execution of a workflow against a document packet; fetch its status and
results flat by run id.
- name: health
description: Health check endpoints to verify API availability.
paths:
/v3/workflows/{workflow_id}/evals/:
post:
tags:
- evals
summary: Launch an eval
description: >-
Launch an eval over a workflow's whole dataset (async, fire-and-forget).
Enqueues one fresh extraction per dataset document on the target version
and
pins the realized cohort into a new eval, then returns `202` immediately
with
`eval_id` and `run_number` to poll. Grading finalizes on a worker path —
read
the eval by `eval_id` for the headline once it leaves `in_progress`.
Without an `Idempotency-Key` every launch creates a NEW eval —
re-launching
runs a fresh cohort. Supplying the header replays: a retry with the same
key
returns the original eval instead of launching a second run. Omit
`version_id`
to evaluate the workflow's current version; pass one to override. (With
an
omitted version, a replayed key sent after a new version was published
targets
a different version than the first call, so the backend answers `422`
for the
key reuse — pass an explicit `version_id` for a stable replay.)
A run that enqueues zero documents (empty dataset, or every document
failed
to enqueue) creates no eval and returns `422 EMPTY_EVAL`.
operationId: v3_launch_eval
parameters:
- name: workflow_id
in: path
required: true
schema:
type: string
title: Workflow Id
- name: Idempotency-Key
in: header
required: false
schema:
anyOf:
- type: string
- type: 'null'
description: >-
Optional caller-supplied key (Stripe convention). Retrying with
the same key returns the original eval instead of launching (and
billing) a second full-dataset run.
title: Idempotency-Key
description: >-
Optional caller-supplied key (Stripe convention). Retrying with the
same key returns the original eval instead of launching (and
billing) a second full-dataset run.
requestBody:
content:
application/json:
schema:
anyOf:
- $ref: '#/components/schemas/EvalLaunchRequest'
- type: 'null'
title: Body
responses:
'202':
description: Successful Response
content:
application/json:
schema:
$ref: '#/components/schemas/EvalLaunchedV3'
headers:
X-RateLimit-Limit:
description: Requests allowed per window (60s).
schema:
type: integer
X-RateLimit-Remaining:
description: Requests remaining in the current window.
schema:
type: integer
X-RateLimit-Reset:
description: Seconds until the current window resets.
schema:
type: integer
'401':
description: Authentication failed — missing, invalid, or expired API key.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
AUTH_FAILED:
value:
error: Authentication failed.
detail: Authentication failed.
error_code: AUTH_FAILED
retryable: false
request_id: req_abc123
'404':
description: >-
Resource not found — the workflow, document packet, or run ID does
not exist or is not visible to this API key.
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
NOT_FOUND:
value:
error: Resource not found.
detail: Resource not found.
error_code: NOT_FOUND
retryable: false
request_id: req_abc123
'422':
description: Validation Error
content:
application/json:
schema:
$ref: '#/components/schemas/HTTPValidationError'
'429':
description: >-
Rate limited — retry after the `Retry-After` header. Default
windows: 60 requests/minute for extraction submission, 600
requests/minute for general endpoints.
headers:
Retry-After:
description: Seconds until the next request is allowed.
schema:
type: integer
X-RateLimit-Limit:
description: Requests allowed per window (60s).
schema:
type: integer
X-RateLimit-Remaining:
description: Requests remaining in the current window.
schema:
type: integer
X-RateLimit-Reset:
description: Seconds until the current window resets.
schema:
type: integer
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorEnvelope'
examples:
RATE_LIMITED:
value:
error: Rate limit exceeded — retry after backoff.
detail: Rate limit exceeded — retry after backoff.
error_code: RATE_LIMITED
retryable: true
request_id: req_abc123
security:
- ApiKeyAuth: []
components:
schemas:
EvalLaunchRequest:
properties:
version_id:
anyOf:
- type: string
- type: 'null'
title: Version Id
description: >-
Workflow version to evaluate. Omit to run against the workflow's
current version; pass a value to override.
examples:
- abcdef1234
type: object
title: EvalLaunchRequest
description: |-
Body for ``POST /v3/workflows/{workflow_id}/evals/`` (JSON, optional).
Send ``{}`` (or no body) to evaluate the workflow's current version.
EvalLaunchedV3:
properties:
eval_id:
type: string
title: Eval Id
description: >-
Unique identifier of the new eval (hyphenated UUID); the poll
handle.
examples:
- 069dcc2c-e14c-7606-8000-2ee4fb17b4f9
run_number:
type: integer
title: Run Number
description: 'Per-workflow run counter (#1, #2, …) for this eval.'
examples:
- 3
enqueued_count:
type: integer
title: Enqueued Count
description: >-
Number of dataset documents whose extraction was enqueued for this
run.
examples:
- 42
failed_count:
type: integer
title: Failed Count
description: >-
Number of dataset documents that failed to enqueue. Read the eval
for the failed document ids.
examples:
- 0
type: object
required:
- eval_id
- run_number
- enqueued_count
- failed_count
title: EvalLaunchedV3
description: |-
202 response: the eval was created and its cohort enqueued.
Exposes counts, never the internal extraction/file ids. Read the eval by
``eval_id`` for the per-document breakdown (including which documents
failed to enqueue).
ErrorEnvelope:
type: object
title: ErrorEnvelope
description: >-
Uniform error response. `retryable=true` means the same request may
succeed on retry (with backoff); `retryable=false` means the caller must
change the request before retrying.
required:
- error
- detail
- error_code
- retryable
- request_id
properties:
error:
type: string
description: Short human-readable summary.
detail:
description: >-
Machine-readable detail. String for most errors; list of field
errors for validation failures.
oneOf:
- type: string
- type: array
items:
type: object
error_code:
type: string
description: Stable identifier for programmatic handling.
examples:
- VALIDATION_ERROR
- AUTH_FAILED
- NOT_FOUND
- PRECONDITION_FAILED
- RATE_LIMITED
- INTERNAL_ERROR
- GATEWAY_TIMEOUT
retryable:
type: boolean
description: True when the caller should retry (with backoff for 429/412/5xx).
request_id:
type: string
description: Correlates this response with server logs.
HTTPValidationError:
properties:
detail:
items:
$ref: '#/components/schemas/ValidationError'
type: array
title: Detail
type: object
title: HTTPValidationError
ValidationError:
properties:
loc:
items:
anyOf:
- type: string
- type: integer
type: array
title: Location
msg:
type: string
title: Message
type:
type: string
title: Error Type
input:
title: Input
ctx:
type: object
title: Context
type: object
required:
- loc
- msg
- type
title: ValidationError
securitySchemes:
ApiKeyAuth:
type: http
description: >-
API key issued from app.anyformat.ai/api-key. Send as `Authorization:
Bearer `.
scheme: bearer
````