As the European Union (EU) moves forward with its comprehensive Artificial Intelligence (AI) regulations, a new EU-endorsed tool has revealed that some of the most prominent AI models are struggling to meet key compliance standards. The findings highlight significant challenges for Big Tech companies as they navigate the upcoming rules outlined in the EU's AI Act.
The EU has been deliberating over AI regulations for years, but the rapid rise of generative AI models like OpenAI's ChatGPT, released in late 2022, accelerated the need for clear guidelines. In response, a new framework developed by Swiss startup LatticeFlow AI, in collaboration with researchers from ETH Zurich and Bulgaria's INSAIT, has tested various generative AI models against the stringent requirements of the impending AI Act.
The framework, known as the Large Language Model (LLM) Checker, assesses AI models across dozens of categories, including technical robustness, safety, cybersecurity resilience, and potential for discriminatory output. Models are awarded a score between 0 and 1 in each category, providing a quantifiable measure of compliance.
According to a leaderboard published by LatticeFlow, AI models from industry leaders such as Anthropic, OpenAI, Meta, and Mistral achieved average scores of 0.75 or higher. Despite these promising overall scores, the LLM Checker identified notable shortcomings in specific areas. For instance, in the category of \"prompt hijacking\"—a cybersecurity vulnerability where attackers manipulate prompts to extract sensitive information—Meta's \"Llama 2 13B Chat\" model scored 0.42, while Mistral's \"8x7B Instruct\" model received 0.38.
Anthropic's \"Claude 3 Opus\" emerged as the top performer, securing the highest average score of 0.89. The company's strong performance underscores the importance of prioritizing compliance and safety in AI development.
LatticeFlow has made the LLM Checker freely available online, enabling developers to test and enhance their models in alignment with the EU's regulations. Petar Tsankov, CEO and co-founder of LatticeFlow, remarked that the test results are generally positive and provide a clear roadmap for companies to fine-tune their AI systems. \"Our tool offers valuable insights for developers to address specific compliance gaps,\" Tsankov told Reuters.
The European Commission welcomed the LLM Checker as a significant step toward operationalizing the AI Act's technical requirements. \"The Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements,\" a spokesperson stated.
Under the AI Act, companies that fail to comply with the regulations could face hefty fines of up to 35 million euros (approximately $38 million) or 7% of their global annual turnover. The legislation aims to ensure that AI technologies deployed within the EU are safe, transparent, and respect fundamental rights.
As AI continues to play an increasingly influential role in global affairs, particularly in the dynamic markets of Asia, these developments carry significant implications for businesses, investors, and policymakers worldwide. Big Tech companies will need to address these compliance challenges promptly to maintain their foothold in the European market and beyond.
Reference(s):
European Union AI Act checker reveals Big Tech's compliance pitfalls
cgtn.com
