LatticeFlow introduces the first comprehensive assessment framework for compliance with the EU AI law

The site has so far evaluated models from OpenAI, Meta, Mistral, Anthropic and Google on more than two dozen technical specifications. Other modelers are also encouraged to request compliance assessments of their models.

“We uncover deficiencies in existing models and benchmarks, particularly in areas such as robustness, security, diversity and fairness,” write researchers from LatticeFlow, INSAIT and ETH Zurich in a specialist article. “Compl-AI shows for the first time the possibilities and difficulties of bringing the obligations of the law to a more concrete, technical level.”

Most models struggle with diversity and non-discrimination

Under the EU AI law, models and systems are marked as unacceptable, high, limited and minimal risk. In particular, an unacceptable label would prohibit the development and deployment of a model. Model makers also face high fines if they do not adhere to the regulations.