A new tool designed to evaluate artificial intelligence (AI) models against the European Union's forthcoming AI Act has uncovered significant compliance gaps among some of the industry's most prominent players.
Developed by Swiss startup LatticeFlow AI in collaboration with ETH Zurich and Bulgaria's INSAIT, the \"Large Language Model (LLM) Checker\" assesses generative AI models across dozens of categories, including cybersecurity resilience and discriminatory output. The framework assigns each model a score between 0 and 1, reflecting its alignment with the EU's rigorous AI regulations set to come into effect over the next two years.
According to a leaderboard published by LatticeFlow, AI models from companies such as Anthropic, OpenAI, Meta, and Mistral received average scores of 0.75 or above. However, the LLM Checker highlighted areas where these models fall short, signaling the need for further refinement to meet regulatory standards.
One notable finding was in the category of \"prompt hijacking,\" a cyberattack where malicious prompts are disguised as legitimate queries to extract sensitive information. Meta's \"Llama 2 13B Chat\" model scored 0.42 in this area, while Mistral's \"8x7B Instruct\" model received a score of 0.38. In contrast, \"Claude 3 Opus,\" developed by Google-backed Anthropic, achieved the highest average score of 0.89, showcasing stronger compliance across evaluated categories.
LatticeFlow's CEO and co-founder, Petar Tsankov, expressed optimism about the results, noting that the tool offers companies a roadmap to fine-tune their models in line with the AI Act. \"The test results are positive overall,\" Tsankov told Reuters. \"They provide actionable insights for developers to enhance their models' compliance.\"
The European Commission has welcomed the initiative. A spokesperson stated, \"The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements.\"
The LLM Checker is available online for developers to test their AI models' compliance, offering a valuable resource as companies navigate the new regulatory landscape. With potential fines of up to 35 million euros ($38 million) or 7 percent of global annual turnover for non-compliance, the pressure is on for tech giants to address identified gaps.
The AI Act represents one of the world's most comprehensive attempts to regulate AI technologies, aiming to ensure safety, transparency, and accountability in AI deployment. The emergence of tools like LatticeFlow's LLM Checker marks a significant step toward operationalizing these standards and fostering industry-wide compliance.
Reference(s):
European Union AI Act checker reveals Big Tech's compliance pitfalls
cgtn.com
