# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'
from anyformat.sdk import Client
client = Client(api_key="YOUR_API_KEY")
launched = client.launch_eval(
"0686bb97-8c30-70f0-8000-97669e000eb8",
idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)
interface EvalLaunched {
eval_id: string;
run_number: number;
enqueued_count: number;
failed_count: number;
}
const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
`https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
{
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
'Idempotency-Key': crypto.randomUUID(),
},
// Optional: pin a version. Omit the body to use the current version.
body: JSON.stringify({ version_id: 'abcdef1234' }),
}
);
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);
{
"eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
"run_number": 3,
"enqueued_count": 42,
"failed_count": 0
}
Launch Eval
Launch a graded eval over a workflow’s whole dataset on a target version
# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'
from anyformat.sdk import Client
client = Client(api_key="YOUR_API_KEY")
launched = client.launch_eval(
"0686bb97-8c30-70f0-8000-97669e000eb8",
idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)
interface EvalLaunched {
eval_id: string;
run_number: number;
enqueued_count: number;
failed_count: number;
}
const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
`https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
{
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
'Idempotency-Key': crypto.randomUUID(),
},
// Optional: pin a version. Omit the body to use the current version.
body: JSON.stringify({ version_id: 'abcdef1234' }),
}
);
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);
{
"eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
"run_number": 3,
"enqueued_count": 42,
"failed_count": 0
}
202 at once with an eval_id and a run_number. Grading finalizes on a worker, so poll the eval by its eval_id until status leaves in_progress.
The response carries counts, never internal ids. enqueued_count counts the documents whose extraction was queued, and failed_count counts those that failed to enqueue. For the per-document breakdown, read the eval by eval_id.
Target version
Omitversion_id to evaluate the workflow’s current version; pass one to override. A workflow with no version yet returns 404 NOT_FOUND.
Idempotency
Without anIdempotency-Key, every launch creates a new eval. Re-launching runs a fresh cohort, and the platform meters it accordingly. See How credits work. Supplying the header replays instead: a retry with the same key returns the original eval, and launches no second run. See Idempotency.
version_id, a replayed key sent after a new version was published resolves to a different version than the first call, so the request no longer matches and returns 422. Pass an explicit version_id when you need a stable replay across version changes.Errors
| Status | error_code | When |
|---|---|---|
422 | EMPTY_EVAL | No dataset document was enqueued, because the dataset is empty or every document failed to enqueue. The API creates no eval. |
400 | OPERATOR_DEPRECATED | The target version contains a deprecated operator and cannot run. |
404 | NOT_FOUND | Unknown workflow, including one in another organization. Also raised when the workflow has no version to evaluate. |
422 | VALIDATION_ERROR | An Idempotency-Key was reused with a different request. |
# Current version: no body needed
curl -X POST 'https://api.anyformat.ai/v3/workflows/0686bb97-8c30-70f0-8000-97669e000eb8/evals/' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Idempotency-Key: 9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4'
from anyformat.sdk import Client
client = Client(api_key="YOUR_API_KEY")
launched = client.launch_eval(
"0686bb97-8c30-70f0-8000-97669e000eb8",
idempotency_key="9e8d7c6b-5a4f-3e2d-1c0b-a9b8c7d6e5f4",
)
print(launched.eval_id, launched.run_number)
interface EvalLaunched {
eval_id: string;
run_number: number;
enqueued_count: number;
failed_count: number;
}
const workflowId = '0686bb97-8c30-70f0-8000-97669e000eb8';
const response = await fetch(
`https://api.anyformat.ai/v3/workflows/${workflowId}/evals/`,
{
method: 'POST',
headers: {
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json',
'Idempotency-Key': crypto.randomUUID(),
},
// Optional: pin a version. Omit the body to use the current version.
body: JSON.stringify({ version_id: 'abcdef1234' }),
}
);
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const launched: EvalLaunched = await response.json();
console.log(launched.eval_id, launched.run_number);
{
"eval_id": "069dcc2c-e14c-7606-8000-2ee4fb17b4f9",
"run_number": 3,
"enqueued_count": 42,
"failed_count": 0
}
Authorizations
API key issued from app.anyformat.ai/api-key. Send as Authorization: Bearer <key>.
Headers
Optional caller-supplied key (Stripe convention). Retrying with the same key returns the original eval instead of launching (and billing) a second full-dataset run.
Path Parameters
Body
Body for POST /v3/workflows/{workflow_id}/evals/ (JSON, optional).
Send {} (or no body) to evaluate the workflow's current version.
Workflow version to evaluate. Omit to run against the workflow's current version; pass a value to override.
"abcdef1234"
Response
Successful Response
202 response: the eval was created and its cohort enqueued.
Exposes counts, never the internal extraction/file ids. Read the eval by
eval_id for the per-document breakdown (including which documents
failed to enqueue).
Unique identifier of the new eval (hyphenated UUID); the poll handle.
"069dcc2c-e14c-7606-8000-2ee4fb17b4f9"
Per-workflow run counter (#1, #2, …) for this eval.
3
Number of dataset documents whose extraction was enqueued for this run.
42
Number of dataset documents that failed to enqueue. Read the eval for the failed document ids.
0

