Post evals

Create the structure of an evaluation that can be used to test a model’s performance.

An evaluation is a set of testing criteria and a datasource. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources.

For more information, see the Evals guide.

NOTE: This Azure OpenAI API is in preview and subject to change.

POST /evals

Error

A valid request URL is required to generate request examples

200

{
  "object": "eval",
  "id": "<string>",
  "name": "<string>",
  "data_source_config": {
    "type": "custom"
  },
  "testing_criteria": null,
  "created_at": 123,
  "metadata": {}
}

Authorizations

ApiKeyAuthApiKeyAuth_OAuth2Auth

api-key · string · header · required

Headers

aoai-evals · enum<string> · required

Enables access to AOAI Evals, a preview feature. This feature requires the 'aoai-evals' header to be set to 'preview'.

Available options: preview

Query Parameters

api-version · enum<string> · default: v1

The explicit Azure AI Foundry Models API version to use for this request. v1 if not otherwise specified.

Available options: v1, preview

Body

application/json

statusCode · enum<number> · required

Available options: 201

data_source_config · object · required

The configuration for the data source used for the evaluation runs. Dictates the schema of the data used in the evaluation.

Show child attributes

testing_criteria · object[] · required

A list of graders for all eval runs in this group. Graders can reference variables in the data source using double curly braces notation, like {{item.variable_name}}. To reference the model's output, use the sample namespace (ie, {{sample.output_text}}).

Show child attributes

name · string

The name of the evaluation.

metadata · object

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Show child attributes

Response

200

application/json

The request has succeeded.

An Eval object with a data source config and testing criteria. An Eval represents a task to be done for your LLM integration. Like:

• Improve the quality of my chatbot
• See how well my chatbot handles customer support
• Check if o4-mini is better at my usecase than gpt-4o

object · enum<string> · default: eval · required

The object type.

Available options: eval

id · string · required

Unique identifier for the evaluation.

name · string · required

The name of the evaluation.

data_source_config · object · required

Configuration of data sources used in runs of the evaluation.

Show child attributes

testing_criteria · object[] | null · required

A list of testing criteria.

Show child attributes

created_at · integer<unixtime> · required

The Unix timestamp (in seconds) for when the eval was created.

metadata · object · required

Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

Show child attributes