AI Surveillance and EU AI Act

Introduction 

The new frontier and integration of Artificial Intelligence (AI) into surveilling technologies have brought uncertainty in both domestic and international policing, leaving countries largely to their own devices to decide how to handle AI surveillance. Josh Chin and Liza Lin, two Wall Street Journal reporters, share multiple Uyghur people’s stories on Chinese AI surveillance within the Xinjiang Autonomous Region.[1] They report that Xinjiang police took Uyghur people’s “blood, iris, fingerprints, and voice recordings… to be input into a biometric database.”[2] China now uses its biometric databases, present in 25 of 31 regions and provinces, in tandem with AI surveillance to monitor, surveil, and police certain groups like the Uyghurs, a primarily Muslim Turkic ethnic group.[3] Chinese AI surveillance uses technology from U.S. companies like Nivida to track blacklisted individuals, using a “risk” scoring system compiled from Chinese biometric data.[4] Further, prior to 2019, China also worked with U.S. companies like IBM and Dell specifically marketing their use of racial recognition in surveillance.[5] AI technology paired with vast biometric databases allows for real time tracking or movement control of people flagged under prompted labels like counterterrorism.[6] While China argues that this decreases terrorism and other crime, critics argue that misuse of such surveillance can lead to predictive policing that stifles protesting, violates privacy, and forces assimilation.[7]

To avoid situations that risk human rights, the European Union (EU) plans to bring some form of regulation to their member states through the EU AI Act.[8] The EU AI Act attempts to prevent predictive policing and rights violations by regulating and even prohibiting certain AI surveillance, but its lack of definitional framework and potential for loopholes jeopardizes the effect it intends to have on a new rapidly changing technology.

AI Surveillance

The introduction of AI into surveillance technologies like CCTVs, body cameras, GPS tracking, cell-site simulators, biometric scanners, drones, motion sensors, and social media monitoring will allow surveillance to be more capable, efficient, and cost effective.[9] AI has developed from language models, where the user must prompt millions of data points for the AI to assess future data, to vision language models, where AI recognizes visual data points within their context and without laborious prompt training.[10] Prompts like, ‘notify me when someone in a hat crosses this street,’ cost a fraction of the price because the visual model doesn’t need to run millions of examples to assess the prompt and it can better account for any variation in the video its assessing.[11] When given vast compiled data, like biometric data, governments can autopilot the control, management, or monitoring of people.[12] As previously stated, China’s unchecked use of AI surveillance with biometric data to target specific ethnic groups like the Uyghurs has left the world stage with ethical questions regarding human rights, arbitrary predictive policing, and liability for AI’s mistakes.[13]

Pakistan has a network of AI surveillance which uses biometric databases like the Chinese system.[14] While its use is similarly questioned by critics, there is also an additional issue that Pakistan’s AI surveillance system is not as internally controlled as China’s and is primarily sourced out of country to a variety of AI companies, including German, French, Canadian, U.S., and Chinese companies.[15] China, only sourcing hardware and software externally, has total internal control over the operation and data in their system, where Pakistan’s surveillance system is much more open to foreign AI companies and their respective systems.[16] This has sparked additional concern over fundamental questions on who’s accountable when the AI makes a mistake or if it discriminates in its assessments.[17] Additionally, there are increased privacy concerns of Pakistan’s biometric data being available to external companies and nations.[18]

In the western world, U.S. cities like Denver, Colorado already use AI to scan mug shot photos for matches of an unidentified subject.[19] Denver law enforcement insists that AI only searches mug shots, shortening work that would have been done less efficiently by another officer.[20] Law enforcement also requires all meaningful decisions using AI to undergo “human review.”[21] However, the U.S. is not without its own abuses in AI surveillance. In Williams v. City of Detroit, a black man was wrongfully arrested due to faulty AI facial recognition leading to a $300,000 pay out by the city.[22] Additionally, the U.S. has used Palantir, a U.S. AI surveillance company, technology in its controversial immigration enforcement.[23] This has raised concerns that AI systems, which are programed to use or disregard specific data, inherently involve policy decisions that are not easily accessible or transparent to the public.[24]

EU AI Act

The European Union plans to address the problems that arise in AI surveillance through the EU AI Act, formally adopted on May 21st, 2024.[25] Its proponents argue that it shifts away from reactive policy and it is the first proactive international legislation on AI.[26] The European Commission views AI not just as a tool, but as a more powerful technology which requires remedies for its mistakes.[27] The Act’s critics argue on the same reasoning that powerful tools like AI are not safeguarded by basic practice regulations, and that the legislation will soon become obsolete as AI quickly evolves.[28]

Regardless, Chapter II Article 5 of the Act generally prohibits law enforcement from using AI systems like: biometric categorization systems, emotional inference, social scoring, untargeted facial recognition databases, real time biometric tracking, and individual criminal risk assessments.[29] There are some exceptions allowing for its use, specifically allowing emotional inference uses in medical situations and real time biometric tracking when looking for missing persons, preventing foreseeable attacks, or to find suspects in serious crimes.[30] These exceptions are only granted when the member state explicitly does so with prior authorization from judicial authority over that member state.[31] To do so, the member state must include appropriate conditions to which their “temporal, geographic and personal limitations” require the use of the prohibited AI system.[32] All this tries to prevent systems that predictive police using people’s risk assessments while allowing some biometric identification collection which, as a “high-risk” system, is only practicable with strict safeguards.[33] The Act pushes liability to the deployer, or user, of “all shortcomings” including those of the provider when using any high-risk system.[34] While this allows biometric data collection, use by law enforcement is still generally prohibited, and the Act attempts to add responsibility to the deployer when using it and other high-risk usages.[35]

The prohibited systems regulation took effect February of 2025.[36] With law enforcement’s established use of AI before the Act, the requirements on transparency in AI’s decision making, and the exceptions in its surveillance use, law enforcement may be forced to fully cease its AI use, making them less effective, or may risk revealing sensitive information in its disclosure.[37] Strangely, the Act states its scope “shall not… affect the competences of the Member States concerning national security, regardless of the type of entity entrusted.”[38] There is no definitional section on what differentiates a law enforcement use and a national security use, in fact it allows room for any entity entrusted with the latter to be exempted.[39] This makes critics question the effectiveness of the prohibited uses.[40] Depending on the Act’s future interpretation, private entities could use prohibited practices in the name of something like counterterrorism.

Conclusion

The Uyghur people in Xinjiang have had AI surveillance monitoring and further policing their cities often in violation of their privacy. If the EU AI Act really wants to prevent predictive policing and rights violations like that in Xinjiang, it must address its lack of definitional framework and potential for loopholes on this new, powerful, and quickly changing technology. Specifically, the EU should consider additional definitive provisions distinguishing terms “national security” and “law enforcement” if they want to mitigate the risk of faulty interpretations that ultimately limit its scope.


[1] Josh Chin & Liza Lin, Surveillance State: Inside China’s Quest to Launch a New Era of Social Control, St. Martin’s Press, 1st ed. 2022).

[2] Emly Feng, ‘Surveillance State’ Explores China’s Tech and Social Media Control Systems, NPR (Sept. 7, 2022), https://www.npr.org/2022/09/07/1118105165/surveillance-state-explores-chinas-tech-and-social-media-control-systems#:~:text=However%2C%20such%20control%20masks%20a,algorithms%20to%20indicate%20religious%20extremism.

[3] Dake Kang & Yael Grauer, Silicon Valley Enabled Brutal Mass Detention and Surveillance in China, Internal Documents Show, AP News (Sept. 8, 2025) https://apnews.com/article/chinese-surveillance-silicon-valley-uyghurs-tech-xinjiang-8e000601dadb6aea230f18170ed54e88 (reporting predictive policing in ethnic provinces); Isabella Qian et al., Four Takeaways From a Times Investigation Into China’s Expanding Surveillance State, N.Y. Times (July 26, 2022), https://www.nytimes.com/2022/06/21/world/asia/china-surveillance-investigation.html# (on China’s use of biometric data bases for surveillance).

[4] Kang, supra note 4.

[5] Id.

[6] Id.

[7] Id.

[8] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence and Amending Regulations (EcEC No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828, 2024 O.J. (L) 1 [hereinafter AI Act].

[9] Jay Stanley, Machine Surveillance Is Being Super-Charged by Large AI Models, ACLU (Mar. 21, 2025), https://www.aclu.org/news/privacy-technology/machine-surveillance-is-being-super-charged-by-large-ai-models.

[10] Id.

[11] Id.

[12] Matther Tokson, The Authoritarian Risks of AI Surveillance, Lawfare (May 1, 2025), https://www.lawfaremedia.org/article/the-authoritarian-risks-of-ai-surveillance.

[13] Kang, supra note 4.

[14] Georgia Green, Investigation Ties Foregin Companies to Pakistan Mass Surveillance System, JURIST (Sept. 10, 2025), https://www.jurist.org/news/2025/09/investigation-ties-foreign-companies-to-pakistans-mass-surveillance-system/.

[15] Id.

[16] Id.

[17] Saeed A. Soomro, Hadi B. Kalhoro & Mahwish Gujjar, AI-Driven Policing in Pakistan: Potential Pitfalls, and Privacy Concerns, 9 Pakistan Soc. Sci. Rev. 3 (July 30, 2025), https://doi.org/10.35484/pssr.2025(9-III)28.

[18] Id.

[19] Allison Sherry, Colorado Law Enforcement Welcomes Speedy AI Facial Recognition Technology Along with Rules, Some Advocates Worry About Privacy and Misuse, CPR News (July 25, 2024), https://www.cpr.org/2024/07/25/colorado-law-enforcement-police-fast-ai-facial-recognition-technology/.

[20] Id.

[21] Colo. Rev. Stat. § 24-18-303 (2025).

[22] Stipulated Order of Voluntary Dismissal with Prejudice, Williams v. City of Detroit, No. 21-cv-10827 (E.D. Mich. 2022).

[23] Steven Hubbard, ICE to Use ImmigrationOS by Palantir, a New AI System, to Track Immigrants’ Movements, American Immigration Council (Aug. 21, 2025), https://www.americanimmigrationcouncil.org/blog/ice-immigrationos-palantir-ai-track-immigrants/.

[24] Id.

[25] See generally AI Act; see also European Commission, AI Act, (Oct. 23, 2025), https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai.

[26] Oskar J. Gstrein, Noman Haleem & Andrej Zwitter, General-purpose AI Regulation and the European Union AI Act, 13 Internet Pol’y Rev. 3 (Aug. 1, 2024), https://doi.org/10.14763/2024.3.1790.

[27] Laura Caroli, Will the EU AI Act work? Lessons Learned from Past Legislative Initiatives, Future Challenges,  IAPP (Apr. 17, 2024), https://iapp.org/news/a/will-the-eu-ai-act-work-lessons-learned-from-past-legislative-initiatives-future-challenges.

[28] Id.

[29] AI Act, ch. II, art. 5 (1-6).

[30] Id.

[31] Pieter Gryffroy, Understanding the Impact of the AI Act for Law Enforcement in 7 Points and 7 Key Takeaways, Timelex (Dec. 20, 2024), https://www.timelex.eu/en/blog/understanding-impact-ai-act-law-enforcement-7-points-and-7-key-takeaways#:~:text=The%20most%20common%20example%20of,of%20at%20least%20four%20years.

[32] AI Act, ch. II art. 5(2)(b).

[33] Id.; European Commission, AI Act.

[34] ISACA, Understanding the EU AI Act: Requirements and Next Steps, at 12 (Oct. 18, 2024), https://www.isaca.org/-/media/files/isacadp/project/isaca/resources/white-papers/understanding-the-eu-ai-act_1024.pdf; AI Act, ch. VI, art. 60(9).

[35] ISACA, supra note 33.

[36] Rosamund Powell, The EU AI Act: National Security Implications, Ctr. for Emerging Tech. and Sec. (Aug. 2024), https://cetas.turing.ac.uk/sites/default/files/2024-07/cetas_explainer_-_the_eu_ai_act_-_national_security_implications.pdf.

[37] Id.

[38] AI Act, ch. I, art. 2(3).

[39] Id.

[40] Gryffroy, supra note 30; ISACA, supra note 33; Powell, supra note 35.