How Well Does AI Speak Luxembourgish? LIST Put 54 Models To The Test

Researchers in Luxembourg have ranked the top-performing AI language models for Luxembourgish—but warn that performance drops as language proficiency levels increase.

Source : siliconluxembourg.lu
Date de publication : 09/05/2025

With an estimated 400,000 speakers, Luxembourgish is considered a small language but one which is essential for integration in the grand duchy. 

And, while it can take years to master, generative AI offers a shortcut for anyone looking to write a quick email or do their homework. 

However, they should beware that some AI models perform better in Luxembourgish than others, according to a team at the Luxembourg Institute of Science and Technology (LIST). 

They benchmarked 54 models using official language exams developed by the Institut National des Langues Luxembourg (INLL) spanning levels from A1 (beginner) to C1 (advanced professional fluency).

LLM Luxembourgish leaderboard 

The research found that the LLMs’ abilities to handle Luxembourgish declined as the level of complexity rose, with top LLMs scoring 92.31 at A1 and only 77.14 at C1.

“Only a handful of large models passed at the higher levels—models that many European SMEs can’t embed due to cost or infrastructure constraints,” said Cédric Lothritz, Research & Technology Associate at LIST.

Recurring results

Researchers found many models made the same kinds of mistakes: misunderstanding

context, failing grammar rules, and even making simple logical errors. According to LIST, it shows current AI systems still struggle with languages that haven’t been central in model training.

Why it matters

The research, which is part of the €40m pan-European LLMs4EU project, was driven by the need to avoid Luxembourgish being digitally sidelined, while helping it to evolve. 

Luxembourg’s contribution to the LLMs4EU initiative also includes the University of Luxembourg and the Zenter fir de Lëtzebuerger Sprooch (ZLS), who are working together to ensure that AI development reflects the country’s cultural and linguistic identity.

Serving linguistic diversity

“LIST’s work shows that technology can serve linguistic diversity, not erase it.” Alexandre Ecker, Director of ZLS, shares this assessment, adding that “benchmarks evaluating how well AI models handle Luxembourgish are an important step toward improving the performance of future AI technologies with regard to the language.”

As AI becomes central to business, education, and communication, ensuring it understands Luxembourgish is crucial not just for cultural preservation, but also for inclusion and innovation.

The leaderboard is an additional feature of the LIST AI Sandbox, a platform allowing researchers to experiment with models in a safe, controlled environment.

Find out more about the Luxembourgish LLM leaderboard.

Jess Bauldry

www.siliconluxembourg.lu/how-well-does-ai-speak-luxembourgish-list-put-54-models-to-the-test/