Launch Eval

# Current version — no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional — pin a version. Omit the body to use the current version.
    body: JSON.stringify({ version_id: 'abcdef1234' }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

POST

workflows

{workflow_id}

evals

# Current version — no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional — pin a version. Omit the body to use the current version.
    body: JSON.stringify({ version_id: 'abcdef1234' }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

Rate limit tier: submission (60 req/min) — see Rate limits. Launches an eval — one graded run of a workflow’s whole dataset against a target version. The call is async and fire-and-forget: it enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns 202 immediately with an eval_id and run_number. Grading finalizes on a worker — poll the eval by its eval_id until its status leaves in_progress. The response carries counts, never internal ids: enqueued_count (documents whose extraction was queued) and failed_count (documents that failed to enqueue). Read the eval by eval_id for the per-document breakdown.

Target version

Omit version_id to evaluate the workflow’s current version; pass one to override. A workflow with no version yet returns 404 NOT_FOUND.

Idempotency

Without an Idempotency-Key, every launch creates a new eval — re-launching runs a fresh cohort and is metered accordingly (see How credits work). Supplying the header replays: a retry with the same key returns the original eval instead of launching a second run. See Idempotency.

When you omit version_id, a replayed key sent after a new version was published resolves to a different version than the first call, so the request no longer matches and returns 422. Pass an explicit version_id when you need a stable replay across version changes.

Errors

Status	`error_code`	When
`422`	`EMPTY_EVAL`	No dataset documents were enqueued — the dataset is empty, or every document failed to enqueue. No eval is created.
`400`	`OPERATOR_DEPRECATED`	The target version contains a deprecated operator and cannot run.
`404`	`NOT_FOUND`	Unknown workflow (including one in another organization), or the workflow has no version to evaluate.
`422`	`VALIDATION_ERROR`	An `Idempotency-Key` was reused with a different request.

# Current version — no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional — pin a version. Omit the body to use the current version.
    body: JSON.stringify({ version_id: 'abcdef1234' }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

Authorizations

Authorization

string

header

required

API key issued from app.anyformat.ai/api-key. Send as Authorization: Bearer <key>.

Headers

Idempotency-Key

string | null

Optional caller-supplied key (Stripe convention). Retrying with the same key returns the original eval instead of launching (and billing) a second full-dataset run.

Path Parameters

workflow_id

string

required

Body

application/json

Body for POST /v3/workflows/{workflow_id}/evals/ (JSON, optional).

Send {} (or no body) to evaluate the workflow's current version.

version_id

string | null

Workflow version to evaluate. Omit to run against the workflow's current version; pass a value to override.

Example:

"abcdef1234"

Response

Successful Response

202 response: the eval was created and its cohort enqueued.

Exposes counts, never the internal extraction/file ids. Read the eval by eval_id for the per-document breakdown (including which documents failed to enqueue).

eval_id

string

required

Unique identifier of the new eval (hyphenated UUID); the poll handle.

Example:

"069dcc2c-e14c-7606-8000-2ee4fb17b4f9"

run_number

integer

required

Per-workflow run counter (#1, #2, …) for this eval.

Example:

3

enqueued_count

integer

required

Number of dataset documents whose extraction was enqueued for this run.

Example:

42

failed_count

integer

required

Number of dataset documents that failed to enqueue. Read the eval for the failed document ids.

Example:

0

Overview

Workflows

Upload

Document packets

Runs

Evals

Target version

Idempotency

Errors

Authorizations

Headers

Path Parameters

Body

Response

​Target version

​Idempotency

​Errors

Authorizations

Headers

Path Parameters

Body

Response

Target version

Idempotency

Errors