Clembench April 2025 Update
Published:
We benchmarked the latest released models and compared them with existing ones:
- GPT-4.1 (released on April 14th)
- Llama-4 Maverick and Scout (released on April 5th)
- Olmo-2-32B (released on March 20th) — fully open source model (code & data)
Findings 📌
🔷 GPT-4.1 matches the performance of GPT-4o (🔵 blue box) while the pricing 💸 dropped 20%. So you’re getting the same ✅ quality but slightly 💸 cheaper.
🔻 Llama-4 models lack the performance of Llama-3.1 and 3.3 (🟥 red boxes). The models got worse both in 📋 instruction following and 🧩 task solving.
🟢📉 Olmo-2-32B is one of the few truly open-source models, but it lags in performance compared to even smaller Llama-8B (🟩 green box).
As usual, all results and source code are openly shared here: github.com/clembench/clembench
