We evaluated 9 LLMs as agents playing dialogue games across all 24 official EU languages plus six others — 108,000 episodes in total. In every single EU language, both commercial models beat every open-weight model.
We benchmarked the latest released models — GPT-4.1, Llama-4 Maverick and Scout, and Olmo-2-32B — and compared them with existing ones.
We’ve put online the latest version of our LLM leaderboard, now based on version 2.0 of clembench — the longest-running game-based evaluation of agentic capabilities of LLMs.
Our paper on benchmarking multimodal LLMs in an interactive setting that doesn’t require any annotated test samples has been accepted at COLING 2025.
We benchmarked the most recent Multimodal Language Models using our clembench framework across five dialogue games.
We benchmarked the latest LLMs — Llama 3.1, Mistral-Large, and GPT-4o-mini. Open-source has caught up to commercial models.
We benchmarked the latest commercial and open LLMs for their instruction following and self-play abilities without any human annotation — what we call clembench.