Launch Eval

# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    version_id="abcdef1234",  # optional: omit for the current version
    dataset_tag_ids=["069dcc2c-e14c-7606-8000-2ee4fb17aaaa"],  # optional: a tagged subset
    untagged=True,  # optional: also the documents with no tag
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional: pin a version and a tag subset. Omit the body to run the
    // whole dataset on the current version.
    body: JSON.stringify({
      version_id: 'abcdef1234',
      dataset_tag_ids: ['069dcc2c-e14c-7606-8000-2ee4fb17aaaa'],
    }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

POST

workflows

{workflow_id}

evals

# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    version_id="abcdef1234",  # optional: omit for the current version
    dataset_tag_ids=["069dcc2c-e14c-7606-8000-2ee4fb17aaaa"],  # optional: a tagged subset
    untagged=True,  # optional: also the documents with no tag
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional: pin a version and a tag subset. Omit the body to run the
    // whole dataset on the current version.
    body: JSON.stringify({
      version_id: 'abcdef1234',
      dataset_tag_ids: ['069dcc2c-e14c-7606-8000-2ee4fb17aaaa'],
    }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

Rate limit tier: submission, 60 requests/min. See Rate limits. Launches an eval: one graded run of a workflow’s dataset, or of a tagged subset of it, against a target version. The call is async and fire-and-forget. It enqueues a fresh extraction for every dataset document, pins the realized cohort into a new eval, and returns 202 at once with an eval_id and a run_number. Grading finalizes on a worker, so poll the eval by its eval_id until status leaves in_progress. The response carries counts, never internal ids. enqueued_count counts the documents whose extraction was queued, and failed_count counts those that failed to enqueue. For the per-document breakdown, read the eval by eval_id.

Target version

Omit version_id to evaluate the workflow’s current version; pass one to override. A workflow with no version yet returns 404 NOT_FOUND.

Run a tagged subset

Pass dataset_tag_ids to run only the dataset documents that carry at least one of those dataset tags (GET …/dataset/tags/ lists them), and untagged: true to run the documents that carry no tag. Send both to run their union. Omit both to run the whole dataset. An empty dataset_tag_ids: [] is not the whole dataset: without untagged: true it selects no document and returns 400 VALIDATION_ERROR, and the SDKs raise before they send it. The eval read back with GET …/evals/{eval_id}/ shows the selection. A tag that is not one of this workflow’s dataset tags returns 422 DATASET_INVALID_REQUEST, and the API creates no eval. A valid selection that matches no document returns 422 EMPTY_EVAL.

Credits

An eval is billed like runs, one extraction per document. Running out of credit does not answer 402: the documents that cannot run fail, and the eval ends with failures, or empty.

Idempotency

Without an Idempotency-Key, every launch creates a new eval. Re-launching runs a fresh cohort, and the platform meters it accordingly. See How credits work. Supplying the header replays instead: a retry with the same key returns the original eval, and launches no second run. See Idempotency.

When you omit version_id, a replayed key sent after a new version was published resolves to a different version than the first call, so the request no longer matches and returns 422 IDEMPOTENCY_KEY_REUSED. Pass an explicit version_id when you need a stable replay across version changes.

On IDEMPOTENCY_KEY_REUSED, params.resource_id is the original eval’s eval_id: read it with GET …/evals/{eval_id}/. Do not launch again under a new key, which runs and bills a second eval.

Errors

Status	`error_code`	When
`422`	`EMPTY_EVAL`	No dataset document was enqueued, because the dataset or the tag selection is empty, or every document failed to enqueue. The API creates no eval.
`400`	`VALIDATION_ERROR`	`dataset_tag_ids` is `[]` and `untagged` is not `true`, so the selection is empty. Omit `dataset_tag_ids` to run the whole dataset. The API creates no eval.
`422`	`DATASET_INVALID_REQUEST`	A `dataset_tag_ids` value is not one of this workflow’s dataset tags. The API creates no eval.
`400`	`OPERATOR_DEPRECATED`	The target version contains a deprecated operator and cannot run.
`404`	`NOT_FOUND`	Unknown workflow, including one in another organization. Also raised when the workflow has no version to evaluate.
`422`	`IDEMPOTENCY_KEY_REUSED`	An `Idempotency-Key` was reused with a different request. `params.resource_id` is the original eval’s `eval_id`.

# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'

from anyformat.sdk import Client

client = Client(api_key="YOUR_API_KEY")

launched = client.launch_eval(
    "0686bb97-8c30-70f0-8000-97669e000eb8",
    version_id="abcdef1234",  # optional: omit for the current version
    dataset_tag_ids=["069dcc2c-e14c-7606-8000-2ee4fb17aaaa"],  # optional: a tagged subset
    untagged=True,  # optional: also the documents with no tag
    idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)

interface EvalLaunched {
  eval_id: string;
  run_number: number;
  enqueued_count: number;
  failed_count: number;
}

const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
  `https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
  {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
      'Idempotency-Key': crypto.randomUUID(),
    },
    // Optional: pin a version and a tag subset. Omit the body to run the
    // whole dataset on the current version.
    body: JSON.stringify({
      version_id: 'abcdef1234',
      dataset_tag_ids: ['069dcc2c-e14c-7606-8000-2ee4fb17aaaa'],
    }),
  }
);

if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);

{
  "eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
  "run_number": 3,
  "enqueued_count": 42,
  "failed_count": 0
}

Authorizations

Authorization

string

header

required

API key issued from app.anyformat.ai/api-key. Send as Authorization: Bearer <key>.

Headers

Idempotency-Key

string | null

Optional caller-supplied key (Stripe convention). Retrying with the same key returns the original eval instead of launching (and billing) a second full-dataset run.

Path Parameters

workflow_id

string

required

Body

application/json

Body for POST /v3/workflows/{workflow_id}/evals/ (JSON, optional).

Send {} (or no body) to evaluate the workflow's current version.

version_id

string | null

Workflow version to evaluate. Omit to run against the workflow's current version; pass a value to override.

Example:

"abcdef1234"

dataset_tag_ids

string[] | null

Run only the dataset documents carrying at least one of these dataset tags. Omit both this and untagged to run the whole dataset. An empty list selects no document, so it is valid only with untagged: true.

Example:

["069dcc2c-e14c-7606-8000-2ee4fb17b4f9"]

untagged

boolean

default:false

Also run the dataset documents that carry no dataset tag.

Response

Successful Response

202 response: the eval was created and its cohort enqueued.

Exposes counts, never the internal extraction/file ids. Read the eval by eval_id for the per-document breakdown (including which documents failed to enqueue).

eval_id

string

required

Unique identifier of the new eval (hyphenated UUID); the poll handle.

Example:

"069dcc2c-e14c-7606-8000-2ee4fb17b4f9"

run_number

integer

required

Per-workflow run counter (#1, #2, …) for this eval.

Example:

3

enqueued_count

integer

required

Number of dataset documents whose extraction was enqueued for this run.

Example:

42

failed_count

integer

required

Number of dataset documents that failed to enqueue. Read the eval for the failed document ids.

Example:

0

Overview

Workflows

Upload

Document packets

Runs

Corrections

Splits

Webhooks

Deletion certificates

Parse

Knowledge

Evals

Optimizations

SDKs and tools

API v2 (retired)

Target version

Run a tagged subset

Credits

Idempotency

Errors

Authorizations

Headers

Path Parameters

Body

Response