Blog post part of our EU AI Act series, written by Bill Marino, a PhD Student in computer science at the University of Cambridge and a Student Fellow at CFI.
The EU AI Act was recently endorsed by EU countries and is expected to start going into effect within the next year. In this post, I’ll describe a couple of scenarios where compliance with this law may be very difficult due to existing gaps in the technical state of the art. I call these technical ‘tension areas”:
The closed-box conundrum
The AI Act subjects the models used in high-risk AI systems –as well as their training data– to a number of rules. For example, the models must be tested for their intended use (Art. 9, s. 5-7), must display appropriate levels of accuracy (Art. 15), and more. Meanwhile, their training data must, among other things, be subject to data governance (Art. 10, s. 2) and possess certain qualities such as representativity (Art. 10, s. 3).
But one problem is that it is increasingly common for AI providers to incorporate, into their AI systems, closed ingredients that are inaccessible and often opaque to them. This includes:
- Closed models, often accessed via API (e.g., Open AI’s GPT series)
- Models trained on closed data (e.g., Meta’s Llama 3)
- Closed data –e.g., where training data remains inaccessible to protect its privacy (Woisetschläger et al. 2024).
The lack of direct access to these ingredients could make it very hard for AI providers, who are largely tasked with ensuring a high-risk AI system complies with the Act (Art. 16, s. a), to know whether their system satisfies the Act’s rules on models and datasets. Even if the suppliers of these closed ingredients are willing –or compelled (e.g., Art. 53)– to share compliance-related information about them, how can AI providers verify that information (and thus the compliance of their overall system) without direct access?
What I believe are needed here are better methods of exchanging and verifying compliance-related information about closed models and datasets. Some colleagues and I are working on a system that supports this, which we call Compliance Cards. In addition, some very interesting new work uses zk-SNARK zero-knowledge proofs to verify ML model accuracy and fairness without direct model access (South et al. 2024). We need AI researchers to invest more in methods like these if we ever hope to resolve the AI Act’s “closed-box conundrum.”
“Open problems” in AI research
In some cases, the Act requires that which AI researchers haven’t yet figured out how to provide –at least, not without serious trade-offs. I.e., it requires the solving of an “open problem.”
For example, the Act requires high-risk AI systems to be designed in such a way that humans can effectively oversee them, which includes being able to “correctly interpret” their outputs (Art. 14, sec. 1-4). Putting aside the fact that humans, thanks to various cognitive biases, are underachievers when it comes to overseeing AI (Green 2022; Laux 2023), the explainability that seems to be mandated here (Gyevnar et al. 2023; Górski and Ramakrishna 2023) is widely seen as one of AI’s biggest “open problem[s]” (Porcedda 2023). There’s no agreement on how it (or the closely-related concept of interpretability) should be defined (Schneeberger et al. 2023; Marcinkevičs and Vogt 2023), let alone evaluated (Adadi and Berrada 2018; Panigutti et al. 2023). Nor are there solutions for their accuracy, security, and privacy trade-offs (Hamon et al. 2021; Navas 2023; Milli et al. 2019; Shokri et al. 2021; Burt 2019). Also worrisome is the fact that state-of-the-art explainability and interpretability tools, despite some very recent progress, simply do not work yet for today’s blockbuster LLMs (Zhao et al. 2023; Levy 2024).
Similar pitfalls plague the Act’s requirements around the cybersecurity of high-risk AI systems (Art. 15). Per the Act, such systems must be “resilient against attempts by unauthorized third parties to alter their use, outputs or performance by exploiting system vulnerabilities” (Art. 15, s. 5). But the fact is that AI remains vulnerable to a number of threats that we simply cannot parry. Principal among these are adversarial input attacks (which leverage inputs specifically designed to cause a model to err) that remain indomitable “despite a decade of study” (Qi et al. 2023; NIST 2024; Hendrycks 2022; Cavallaro and Cristofaro 2023). Meanwhile, data poisoning (Tian et al. 2023), backdoor (Raissi 2023), and, in the case of LLMs, jailbreak attacks (Chao et al. 2023; Liu et al. 2024) are all proving stubbornly resistant to mitigations while previously unknown attacks –e.g., slowdown (Varma et al. 2024) – are being discovered all the time.
The open problems implicated by the Act don’t end there. Also on the list are achieving seamless monitoring (Art. 14), robustness (Art. 15), bias detection (Art. 10), evaluation of safety risks (Art. 9), and more (Raissi 2023; EPRS 2022; MLCommons 2024). While AI researchers should do their part by trying to make inroads into these open problems, the EU’s standard-setting entities must also take action. In particular, as they continue to translate the Act into concrete technical requirements that AI providers can follow in order to stay compliant, these standard-setters should avoid setting bars that even our most talented AI researchers cannot yet foresee how to vault.
Other technical “tension areas”
The duplet above is just a small sampling of the Act’s technical “tension areas.” Ones that I didn’t cover in in this post include:
a) Article 10’s arguably-unattainable error-removal requirements for datasets (Ebers et al. 2021; Heikkilä 2022);
b) Article 74’s requirement that market surveillance authorities be able to access to training data –which seems “unworkable” in scenarios like federated learn (Ebers et al. 2021);
c) A number of provisions that are bedeviled by a striking lack of agreed-upon technical standards within the field, including Article 10’s rules around data governance, Article 15’s rules around accuracy and robustness, and more. (Pouget 2023; van der Sloot and Keymolen 2022; Johnson et al. 2024; Feffer et al. 2024; Rotman 2018; Cabitza et al. 2020; Clemmensen and Kjærsgaard 2023).
These (and others like them) warrant more attention from both AI researchers and policymakers in the coming months, as the Act goes into force and its associated standards are finalized. And until we bridge this divide between what the Act demands and what the technical state of the art provides, the entire AI community is advised, like Tube passengers, to mind the gap.
