> ## Documentation Index > Fetch the complete documentation index at: https://docs.anyformat.ai/llms.txt > Use this file to discover all available pages before exploring further. # Launch Eval > Launch a graded eval over a workflow's whole dataset on a target version *Rate limit tier: **submission**, 60 requests/min. See [Rate limits](/api-reference-v3/introduction#rate-limits).* Launches an [eval](/concepts/evals): one graded run of a workflow's **whole dataset** against a target version. The call is **async and fire-and-forget**. It enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns `202` at once with an `eval_id` and a `run_number`. Grading finalizes on a worker, so poll the eval by its `eval_id` until `status` leaves `in_progress`. The response carries **counts, never internal ids**. `enqueued_count` counts the documents whose extraction was queued, and `failed_count` counts those that failed to enqueue. For the per-document breakdown, read the eval by `eval_id`. ## Target version Omit `version_id` to evaluate the workflow's **current version**; pass one to override. A workflow with no version yet returns `404 NOT_FOUND`. ## Idempotency Without an `Idempotency-Key`, **every launch creates a new eval**. Re-launching runs a fresh cohort, and the platform meters it accordingly. See [How credits work](/concepts/how-credits-work). Supplying the header **replays** instead: a retry with the same key returns the **original** eval, and launches no second run. See [Idempotency](/api-reference-v3/introduction#idempotency). When you omit `version_id`, a replayed key sent **after a new version was published** resolves to a different version than the first call, so the request no longer matches and returns `422`. Pass an explicit `version_id` when you need a stable replay across version changes. ## Errors | Status | `error_code` | When | | - | - | - | | `422` | `EMPTY_EVAL` | No dataset document was enqueued, because the dataset is empty or every document failed to enqueue. The API creates no eval. | | `400` | `OPERATOR_DEPRECATED` | The target version contains a deprecated operator and cannot run. | | `404` | `NOT_FOUND` | Unknown workflow, including one in another organization. Also raised when the workflow has no version to evaluate. | | `422` | `IDEMPOTENCY_KEY_REUSED` | An `Idempotency-Key` was reused with a different request. | ```bash curl theme={null} # Current version: no body needed curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4' ``` ```python Python (SDK) theme={null} from anyformat.sdk import Client client = Client(api_key="YOUR_API_KEY") launched = client.launch_eval( "0686bb97-8c30-70f0-8000-97669e000eb8", idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4", ) print(launched.eval_id, launched.run_number) ``` ```typescript TypeScript theme={null} interface EvalLaunched { eval_id: string; run_number: number; enqueued_count: number; failed_count: number; } const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8'; const response = await fetch( `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`, { method: 'POST', headers: { 'Authorization': 'Bearer YOUR_API_KEY', 'Content-Type': 'application/json', 'Idempotency-Key': crypto.randomUUID(), }, // Optional: pin a version. Omit the body to use the current version. body: JSON.stringify({ version_id: 'abcdef1234' }), } ); if (!response.ok) { throw new Error(`API error: ${response.status}`); } const launched: EvalLaunched = await response.json(); console.log(launched.eval_id, launched.run_number); ``` ```json Response (202 Accepted) theme={null} { "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9", "run_number": 3, "enqueued_count": 42, "failed_count": 0 } ``` ## OpenAPI ````yaml POST /v3/workflows/{workflow_id}/evals/ openapi: 3.1.0 info: title: anyformat API description: > Document extraction and workflow automation API. AnyFormat lets you define extraction workflows that pull structured data from any document — PDFs, images, scanned files, or plain text. Upload a file, run it through a workflow, and get back structured fields with confidence scores and source evidence. ## Quick start See the [Quick start guide](https://docs.anyformat.ai/api-reference-v3/introduction) for a walkthrough. ## Authentication All endpoints (except `/health/`) require a Bearer token in the `Authorization` header: ``` Authorization: Bearer ``` Get your API key from [app.anyformat.ai/api-key](https://app.anyformat.ai/api-key). ## Versioning Endpoints are versioned by path prefix (`/v3/`). Every response under a versioned prefix includes `X-API-Version`. version: 3.0.0 servers: - url: https://api.anyformat.ai description: API server security: [] tags: - name: workflows description: >- Workflows define extraction templates — what fields to extract from documents. Create workflows, upload files, run extractions, and fetch results. - name: document-packets description: >- Document packets group uploaded documents for a workflow. Address a packet directly to (re-)run its workflow. - name: runs description: >- Extraction runs — the first-class result entity. Each run is a single execution of a workflow against a document packet; fetch its status and results flat by run id. - name: splits description: >- Splits — the sub-documents a splitter node carves out of a packet. Download any one of them as a PDF, built from the source documents on request and never stored. - name: parse description: >- Parse a single document with the platform's fast lite parse — a one-call shortcut over the workflow upload + run primitives, backed by your organization's system parse workflow. - name: health description: Health check endpoints to verify API availability. paths: /v3/workflows/{workflow_id}/evals/: post: tags: - evals summary: Launch an eval description: >- Launch an eval over a workflow's whole dataset (async, fire-and-forget). Enqueues one fresh extraction per dataset document on the target version and pins the realized cohort into a new eval, then returns `202` immediately with `eval_id` and `run_number` to poll. Grading finalizes on a worker path — read the eval by `eval_id` for the headline once it leaves `in_progress`. Without an `Idempotency-Key` every launch creates a NEW eval — re-launching runs a fresh cohort. Supplying the header replays: a retry with the same key returns the original eval instead of launching a second run. Omit `version_id` to evaluate the workflow's current version; pass one to override. (With an omitted version, a replayed key sent after a new version was published targets a different version than the first call, so the backend answers `422` for the key reuse — pass an explicit `version_id` for a stable replay.) A run that enqueues zero documents (empty dataset, or every document failed to enqueue) creates no eval and returns `422 EMPTY_EVAL`. operationId: v3_launch_eval parameters: - name: workflow_id in: path required: true schema: type: string title: Workflow Id - name: Idempotency-Key in: header required: false schema: anyOf: - type: string - type: 'null' description: >- Optional caller-supplied key (Stripe convention). Retrying with the same key returns the original eval instead of launching (and billing) a second full-dataset run. title: Idempotency-Key description: >- Optional caller-supplied key (Stripe convention). Retrying with the same key returns the original eval instead of launching (and billing) a second full-dataset run. requestBody: content: application/json: schema: anyOf: - $ref: '#/components/schemas/EvalLaunchRequest' - type: 'null' title: Body responses: '202': description: Successful Response content: application/json: schema: $ref: '#/components/schemas/EvalLaunchedV3' headers: X-RateLimit-Limit: description: Requests allowed per window (60s). schema: type: integer X-RateLimit-Remaining: description: Requests remaining in the current window. schema: type: integer X-RateLimit-Reset: description: Seconds until the current window resets. schema: type: integer '400': description: >- `VALIDATION_ERROR` — The request failed validation. `OPERATOR_DEPRECATED` — This workflow version uses a deprecated operator and cannot run. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: VALIDATION_ERROR: value: error: The request failed validation. detail: The request failed validation. error_code: VALIDATION_ERROR retryable: false request_id: req_abc123 OPERATOR_DEPRECATED: value: error: >- This workflow version uses a deprecated operator and cannot run. detail: >- This workflow version uses a deprecated operator and cannot run. error_code: OPERATOR_DEPRECATED retryable: false request_id: req_abc123 '401': description: Authentication failed — missing, invalid, or expired API key. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: AUTH_FAILED: value: error: Authentication failed. detail: Authentication failed. error_code: AUTH_FAILED retryable: false request_id: req_abc123 '403': description: >- Access denied — the API key lacks access to this resource or the scope this call needs (`ACCESS_DENIED`, `INSUFFICIENT_API_KEY_SCOPE`). content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: ACCESS_DENIED: value: error: You do not have access to this resource. detail: You do not have access to this resource. error_code: ACCESS_DENIED retryable: false request_id: req_abc123 '404': description: >- Resource not found — the workflow, document packet, or run ID does not exist or is not visible to this API key. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: NOT_FOUND: value: error: Resource not found. detail: Resource not found. error_code: NOT_FOUND retryable: false request_id: req_abc123 '422': description: >- `EMPTY_EVAL` — The eval has no rows to run. `IDEMPOTENCY_KEY_REUSED` — The idempotency key was already used with a different request. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: EMPTY_EVAL: value: error: The eval has no rows to run. detail: The eval has no rows to run. error_code: EMPTY_EVAL retryable: false request_id: req_abc123 IDEMPOTENCY_KEY_REUSED: value: error: >- The idempotency key was already used with a different request. detail: >- The idempotency key was already used with a different request. error_code: IDEMPOTENCY_KEY_REUSED retryable: false request_id: req_abc123 '429': description: >- Rate limited — retry after the `Retry-After` header. Default windows: 60 requests/minute for extraction submission, 600 requests/minute for general endpoints. headers: Retry-After: description: Seconds until the next request is allowed. schema: type: integer X-RateLimit-Limit: description: Requests allowed per window (60s). schema: type: integer X-RateLimit-Remaining: description: Requests remaining in the current window. schema: type: integer X-RateLimit-Reset: description: Seconds until the current window resets. schema: type: integer content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: RATE_LIMITED: value: error: Rate limit exceeded — retry after backoff. detail: Rate limit exceeded — retry after backoff. error_code: RATE_LIMITED retryable: true request_id: req_abc123 '500': description: >- An internal error stopped the request — `retryable=true`. Retry with backoff. content: application/json: schema: $ref: '#/components/schemas/ErrorEnvelope' examples: INTERNAL_ERROR: value: error: An internal error stopped the request. detail: An internal error stopped the request. error_code: INTERNAL_ERROR retryable: true request_id: req_abc123 security: - ApiKeyAuth: [] components: schemas: EvalLaunchRequest: properties: version_id: anyOf: - type: string - type: 'null' title: Version Id description: >- Workflow version to evaluate. Omit to run against the workflow's current version; pass a value to override. examples: - abcdef1234 type: object title: EvalLaunchRequest description: |- Body for ``POST /v3/workflows/{workflow_id}/evals/`` (JSON, optional). Send ``{}`` (or no body) to evaluate the workflow's current version. EvalLaunchedV3: properties: eval_id: type: string title: Eval Id description: >- Unique identifier of the new eval (hyphenated UUID); the poll handle. examples: - 069dcc2c-e14c-7606-8000-2ee4fb17b4f9 run_number: type: integer title: Run Number description: 'Per-workflow run counter (#1, #2, …) for this eval.' examples: - 3 enqueued_count: type: integer title: Enqueued Count description: >- Number of dataset documents whose extraction was enqueued for this run. examples: - 42 failed_count: type: integer title: Failed Count description: >- Number of dataset documents that failed to enqueue. Read the eval for the failed document ids. examples: - 0 type: object required: - eval_id - run_number - enqueued_count - failed_count title: EvalLaunchedV3 description: |- 202 response: the eval was created and its cohort enqueued. Exposes counts, never the internal extraction/file ids. Read the eval by ``eval_id`` for the per-document breakdown (including which documents failed to enqueue). ErrorEnvelope: type: object title: ErrorEnvelope description: >- Uniform error response. `retryable=true` means the same request may succeed on retry (with backoff); `retryable=false` means the caller must change the request before retrying. required: - error - error_code - retryable - request_id properties: error: type: string description: Short human-readable summary. detail: description: >- Machine-readable detail, any JSON value. A string for most errors, a list of field errors for validation failures, and an object for a structured error such as `FILENAME_CONFLICT` (`detail.conflicts`). error_code: type: string description: Stable identifier for programmatic handling. examples: - VALIDATION_ERROR - AUTH_FAILED - NOT_FOUND - PRECONDITION_FAILED - RATE_LIMITED - INTERNAL_ERROR - GATEWAY_TIMEOUT retryable: type: boolean description: True when the caller should retry (with backoff for 429/412/5xx). request_id: type: string description: Correlates this response with server logs. securitySchemes: ApiKeyAuth: type: http description: >- API key issued from app.anyformat.ai/api-key. Send as `Authorization: Bearer `. scheme: bearer ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.