Clembench April 2025 Update
Published:
We benchmarked the latest released models — GPT-4.1, Llama-4 Maverick and Scout, and Olmo-2-32B — and compared them with existing ones.
Blog Posts
It is finally here! Clembench Leaderboard v2.0 is live 🎉
Published:
We’ve put online the latest version of our LLM leaderboard, now based on version 2.0 of clembench — the longest-running game-based evaluation of agentic capabilities of LLMs.
Using Game Play to Investigate Multimodal and Conversational Grounding — accepted at COLING 2025
Published:
Our paper on benchmarking multimodal LLMs in an interactive setting that doesn’t require any annotated test samples has been accepted at COLING 2025.
Benchmarking the Most Recent Multimodal Language Models
Published:
We benchmarked the most recent Multimodal Language Models using our clembench framework across five dialogue games.
Benchmarking the Latest LLMs — July 2024
Published:
We benchmarked the latest LLMs — Llama 3.1, Mistral-Large, and GPT-4o-mini. Open-source has caught up to commercial models.
March 2024: Benchmarking the Latest LLMs with clembench
Published:
We benchmarked the latest commercial and open LLMs for their instruction following and self-play abilities without any human annotation — what we call clembench.