Bergur Thormundsson
Research expert covering technology in the Nordics, AI and smart cities
Artificial intelligence (AI) benchmark and training - Statistics & Facts
There are a multitude of generative AI programs or large language tools (LLMs) circulating in the open market. One question that often arises among users—and even among those simply curious about this new technology—is: which AI program is the best? This is not an easy question to answer, as there are a wide variety of use-cases to evaluate generative AI tools, usually termed benchmarks. For a rapidly growing segment of the world market and a technology rapidly evolving and growing more vital across multiple industries, it is worth examining these benchmarks and what they mean.
MMLU – the baseline benchmark
Created in 2020 by a team from several renowned universities, the Massive Multitask Language Understanding benchmark (MMLU), received an important facelift with the MMLU-Pro version in June 2024. This benchmark has emerged as the premier tool for assessing LLM capabilities. Most of the advanced LLMs, such as OpenAI’s o1, Anthropic’s Claude, and Google’s Gemini, have all been evaluated against MMLU. It assesses model performance across 57 subjects where they have had little or no prior interaction before. All major LLMs have a score of 75% or higher as of May 2025, meaning it is up to the user to gauge which one best fits their style.Broad availability of benchmarks
While MMLU is the baseline, there are a range of alternatives for those wanting to test LLMs in specific skills. A few examples, as of May 2025, include the Multilingual Index Benchmark, where o1 is particularly strong, and the Coding Index Benchmark, where China’s DeepSeek R1 is a close contender to o1. Another particularly fascinating benchmark is Humanity’s Last Exam (HLE), a questionnaire of 2700 questions across a vast array of topics. Once again, o1 is in the lead, though other models are close behind in this difficult benchmark.The most recent model is often the strongest
As can be seen from these benchmarks, as of May 2025, OpenAI’s o1 is an exceptionally capable LLM. This follows a trend of the most recent AI models from major developers being the most capable. Gemini 2.5, which was released publicly in March 2025, had demonstrated considerable capabilities in mid-2025, even beyond o1. If there is a weakness to o1’s considerable capabilities it is the fact that it is the most expensive model to run, a cost users may not feel directly, but one its developers certainly do. Microsoft and other major funders of OpenAI certainly have the financial clout to continue development for now, but at a steep price.The right tool for the right user at the right time
It is difficult to comprehensively evaluate which AI model is the best at anything, as they are updated frequently and each developer races against the others to show off a more capable model. For general users of AI tools, it means that any of the major ones, ChatGPT, Gemini, Claude, to name a few, are all worth using. For the enterprise users with specific needs, it means keeping up with the lightning-fast AI arms race and choosing the right tool to use at any given time. Doing so will give leading early adopters a competitive edge in AI usage, allowing them to stand out from their peers.Bergur Thormundsson
,
Jul 2, 2025
Bergur Thormundsson
Research expert covering technology in the Nordics, AI and smart cities
