- Quality, safety, cost, and performance leaderboards to quickly identify the model leaders along a single metric (quality, safety, cost, or throughput);
- Trade-off charts to see how models perform on one metric versus another, such as quality versus cost;
- Leaderboards by scenario to find the best leaderboards that suite your scenario.
Quality benchmarks of language models
Azure AI assesses the quality of LLMs and SLMs using accuracy scores from standard, comprehensive benchmark datasets measuring model capabilities such as reasoning, knowledge, question answering, math, and coding.
Quality index is provided on a scale of zero to one. Higher values of quality index are better. The datasets included in quality index are:
See more details in accuracy scores:
Accuracy scores are provided on a scale of zero to one. Higher values are better.
Safety benchmarks of language models
To guide the selection of safety benchmarks for evaluation, we apply a structured filtering and validation process designed to ensure both relevance and rigor. A benchmark qualifies for onboarding if it addresses high-priority risks. For safety leaderboards, we look at different benchmarks that can be considered reliable enough to provide some signals on certain topics of interest as they relate to safety. We select HarmBench to proxy model safety, and organize scenario leaderboards as follows:Model harmful behaviors
The HarmBench benchmark measures model harmful behaviors and includes prompts to illicit harmful behavior from model. As it relates to safety, the benchmark covers 7 semantic categories of behavior:- Cybercrime & Unauthorized Intrusion
- Chemical & Biological Weapons/Drugs
- Copyright Violations
- Misinformation & Disinformation
- Harassment & Bullying
- Illegal Activities
- General Harm
- standard harmful behaviors
- contextually harmful behaviors
- copyright violations
Model ability to detect toxic content
Toxigen is a large-scale machine-generated dataset for adversarial and implicit hate speech detection. It contains implicitly toxic and benign sentences mentioning 13 minority groups. We use the annotated samples from Toxigen for evaluation and calculate F1 scores to measure classification performance. Scoring higher on this dataset means that a model is better at detecting toxic content. Model benchmarking is performed with Azure AI Content Safety Filter turned off.Model knowledge in sensitive domains
The Weapons of Mass Destruction Proxy (WMDP) benchmark measures model knowledge of in sensitive domains including biosecurity, cybersecurity, and chemical security. The leaderboard uses average accuracy scores across cybersecurity, biosecurity, and chemical security. A higher WMDP accuracy score denotes more knowledge of dangerous capabilities (worse behavior from a safety standpoint). Model benchmarking is performed with the default Azure AI Content Safety filters on. These safety filters detect and block content harm in violence, self-harm, sexual, hate and unfairness, but don’t target categories in cybersecurity, biosecurity, and chemical security.Limitations of safety benchmarks
We understand and acknowledge that safety is a complex topic and has several dimensions. No single current open-source benchmarks can test or represent the full safety of a system in different scenarios. Additionally, most of these benchmarks suffer from saturation, or misalignment between benchmark design and the risk definition, can lack clear documentation on how the target risks are conceptualized and operationalized, making it difficult to assess whether the benchmark accurately captures the nuances of the risks. This limitation can lead to either overestimating or underestimating model performance in real-world safety scenarios.Performance benchmarks of language models
Performance metrics are calculated as an aggregate over 14 days, based on 24 trails (two requests per trail) sent daily with a one-hour interval between every trail. The following default parameters are used for each request to the model endpoint:
The performance of LLMs and SLMs is assessed across the following metrics:
Azure AI also displays performance indexes for latency and throughput as follows:
For performance metrics like latency or throughput, the time to first token and the generated tokens per second give a better overall sense of the typical performance and behavior of the model. We refresh our performance numbers on regular cadence.
Cost benchmarks of language models
Cost calculations are estimates for using an LLM or SLM model endpoint hosted on the Azure AI platform. Azure AI supports displaying the cost of serverless API deployments and Azure OpenAI models. Because these costs are subject to change, we refresh our cost calculations on a regular cadence. The cost of LLMs and SLMs is assessed across the following metrics:
Azure AI also displays the cost index as follows: