How to measure AI capabilities, assess benchmark validity and understand where results generalise.
Paper · 2026
We combine AI capability profiles with expert assessments of workplace task requirements to identify promising candidates for AI pilots.
Paper · 2026
We argue for comparing human and AI workers through shared profiles of capabilities and propensities when assessing suitability for workplace tasks.
Paper · 2026
We evaluate LLMs’ physical common-sense by giving them control of agents in the Animal-AI environment and comparing their performance with other agents and human children.
Paper · 2025
We evaluate the general cooperative intelligence of LLM-based agents in zero-shot, mixed-motive environments using a natural language multi-agent simulation, showing limited generalisation to novel social situations.
Paper · 2025
We use a Bayesian approach called ‘Measurement layouts’ to infer prosocial capability profiles of multi-agent systems in the Melting Pot testbed, showing that high performance does not always reflect strong cooperation abilities.
Paper · 2025
We analyse 125+ evaluation studies and identify six major paradigms of AI evaluation, each shaped by distinct goals, methodologies, and research cultures.
Paper · 2025
I lay out key benchmark design criteria for evaluating mathematical reasoning in LLMs.
Tutorial · 2024
A tutorial on evaluating large language model capabilities, presented at the European Association for Data Science’s Summer School on Generative AI.
Tutorial · 2024
An AAAI-24 tutorial on using Bayesian methods to build capability profiles for large language models and reinforcement learning agents.
Paper · 2024
We explore whether benchmarks can be solved using simple n-gram patterns and whether LLMs exploit these patterns to solve benchmark tasks.
Paper · 2024
An analysis of the design and outcomes of the Melting Pot competition, which measures agents’ ability to cooperate with others. We developed cognitive profiles for the agents submitted to the competition.