Page Not Found
That page doesn’t exist.
Sitemap
A list of all the posts and pages found on the site. For you robots out there is an XML version available for digesting as well.
Pages
blogposts
March 2024: Benchmarking the Latest LLMs with clembench
Published:
We benchmarked the latest commercial and open LLMs for their instruction following and self-play abilities without any human annotation — what we call clembench.
Benchmarking the Latest LLMs — July 2024
Published:
We benchmarked the latest LLMs — Llama 3.1, Mistral-Large, and GPT-4o-mini. Open-source has caught up to commercial models.
Benchmarking the Most Recent Multimodal Language Models
Published:
We benchmarked the most recent Multimodal Language Models using our clembench framework across five dialogue games.
Using Game Play to Investigate Multimodal and Conversational Grounding — accepted at COLING 2025
Published:
Our paper on benchmarking multimodal LLMs in an interactive setting that doesn’t require any annotated test samples has been accepted at COLING 2025.
It is finally here! Clembench Leaderboard v2.0 is live 🎉
Published:
We’ve put online the latest version of our LLM leaderboard, now based on version 2.0 of clembench — the longest-running game-based evaluation of agentic capabilities of LLMs.
Clembench April 2025 Update
Published:
We benchmarked the latest released models — GPT-4.1, Llama-4 Maverick and Scout, and Olmo-2-32B — and compared them with existing ones.
Digital Sovereignty in EU — The AI Landscape
Published:
We evaluated 9 LLMs as agents playing dialogue games across all 24 official EU languages plus six others — 108,000 episodes in total. In every single EU language, both commercial models beat every open-weight model.
research
Organizing/Programme committee or invited reviewer
Organizing committee
Supervision
Supervised Theses
talks
Applying Semantic Parsing to QuestionAnswering over Linked Data: Addressing theLexical Gap
Published:
Applying Semantic Parsing to QuestionAnswering over Linked Data: Addressing theLexical Gap
Combining Textual and Graph-Based Features for Named Entity Disambiguation using Undirected Probabilistic Graphical Models
Published:
NERFGUN: A named entity disambiguation tool for linking named entities to Wikipedia
Towards a Large Corpus of Richly Annotated Web Tables for Knowledge Base Population
Published:
Annotation pipeline for extracting information from Web Tables.
AMUSE: Multilingual Semantic Parsing for Question Answering over Linked Data
Published:
Building multilingual semantic parsers using undirected graphical models
Learning Multilingual Semantic Parsers for Question Answering over Linked Data
Published:
PhD Dissertation Talk, Semantic Computing Group, CITEC, Bielefeld University, Germany
TIB-VA at SemEval-2022 Task 5: A Multimodal Architecture for the Detection and Classification of Misogynous Memes
Published:
Multimodal architecture to detect misogynous memes. It is evaluated on the challenge SemEval-2022 Task 5: A Multimodal Architecture for the Detection and Classification of Misogynous Memes, co-located with the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) 2022.
Images in Language Space: Exploring the Suitability of Large Language Models for Vision & Language Tasks
Published:
Findings of the Association for Computational Linguistics: ACL 2023
Evaluating Modular Dialogue System for Form Filling Using Large Language Models
Published:
Workshop on Simulation of Conversational Intelligence in Chat (SCI-CHAT) at EACL 2024
Unveiling Global Narratives: A Multilingual Twitter Dataset of News Media on the Russo-Ukrainian Conflict
Published:
Presented at International Conference on Multimedia Retrieval (ICMR) 2024
Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Models
Published:
Presented at 31st International Conference on Computational Linguistics (COLING 2025), Abu Dhabi, UAE
Benchmarking Incremental Learning of Situation and Language Model using a Text-Simulated Situated Environment
Published:
Presented at ResearchTrend - LLM Agents Track, Online
Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming
Published:
Presented at Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming paper at 1st Workshop on Interactive Task Learning in Human-Robot co-construction (ITL4HRI) at RO-MAN 2025
The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models
Published:
Presented the The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models paper at The 19th Conference of the European Chapter of the Association for Computational Linguistics
teaching
Information Extraction from Web Tables
Graduate course, Bielefeld University, 2018
Project course on table understanding: implementing extraction tasks over real web tables, extending a knowledge base, and evaluating the results.
Statistical Natural Language Processing
Graduate course, Bielefeld University, 2018
Building scalable text-processing systems with statistical methods, integrating recent research results into the students’ own solutions. Taught over 8 semesters, 2014-2018.
Proseminar Visual Analytics
Undergraduate course, Hannover University, 2020
Introduction to visual analytics and computational multimodal systems; each student presents a publication and writes a technical report.
Current Highlights in NLP
Graduate course, University of Potsdam, 2022
MSc seminar on recent developments in NLP, run as a role-playing paper-reading seminar.
Advanced Natural Language Processing
Graduate course, University of Potsdam, 2022
MSc lecture covering text classification, neural language models, syntactic and semantic parsing, and ethics.
Formal Languages and Automata
Undergraduate course, University of Potsdam, 2023
Automata, the Chomsky hierarchy, the Pumping Lemma and Turing machines.
Project Seminar: Models That Explain Themselves
Graduate course, University of Potsdam, 2023
MSc project module implementing model architectures that can explain their own predictions, applied to joint image-text tasks.
Current Highlights in NLP
Graduate course, University of Potsdam, 2023
MSc seminar on recent developments in NLP, run as a role-playing paper-reading seminar.
Methods in Natural Language Processing
Undergraduate course, University of Potsdam, 2023
BSc seminar on methods used across NLP, run as a role-playing paper-reading seminar.
Formal Languages and Automata
Undergraduate course, University of Potsdam, 2024
Automata, the Chomsky hierarchy, the Pumping Lemma and Turing machines.
Project Seminar: Evaluating Chat-Optimized Language Models
Graduate course, University of Potsdam, 2024
MSc project module adapting an existing game to the clembench framework in order to evaluate chat-optimized LLMs.
Current Highlights in NLP
Graduate course, University of Potsdam, 2024
MSc seminar on recent developments in NLP, run as a role-playing paper-reading seminar.
Programming 1
Undergraduate course, University of Potsdam, 2024
BSc course on programming fundamentals: object-oriented design and data structures.
Formal Languages and Automata
Undergraduate course, University of Potsdam, 2025
Automata, the Chomsky hierarchy, the Pumping Lemma and Turing machines.
Project Seminar: Evaluating Chat-Optimized Language Models
Graduate course, University of Potsdam, 2025
MSc project module adapting an existing game to the clembench framework in order to evaluate chat-optimized LLMs.
Current Highlights in NLP
Graduate course, University of Potsdam, 2025
MSc seminar on recent developments in NLP, run as a role-playing paper-reading seminar.
Programming 2
Undergraduate course, University of Potsdam, 2025
BSc course on programming fundamentals: object-oriented design, documentation and code review, and data structures.
Formal Languages and Automata
Undergraduate course, University of Potsdam, 2026
Automata, the Chomsky hierarchy, the Pumping Lemma and Turing machines.
Project Seminar: Learning in Interaction
Graduate course, University of Potsdam, 2026
MSc project module on post-training LLMs from dialogue-game feedback using DPO, PPO, GRPO and SFT with the Playpen framework.