Can military applications of AI be controlled? I would say yes, there is a possibility of restricting and limiting how AI technologies are developed and used in the military domain. However, the focus of those efforts should not be on the technological side, for example, in terms of banning certain types of weapon systems or restricting the use of particular AI models. By the time any potential treaty or any sort of other regulation might be adopted, we might already be talking about different types of technologies. It is common for legislation or regulation to be several years behind technological developments.
It might be worth slightly shifting the debate towards what should be the roles of humans interacting with AI technologies across the AI lifecycle. More specifically, the debate should focus on how to safeguard an appropriate level of human judgement in decisions that are made using AI. That is, not when AI systems replace humans as in a ‘killer-robot’ sci-fi type of scenario, but when humans remain formally in control of the process while the quality of human judgement risks being affected by the use of AI technologies.
Why This Question is Timely
First, we already see studies emerging on the impact of using generative AI and large-language models on human cognitive skills. This raises concerns about human-machine interaction and the role of the human in the dynamics of those interactions. A key question becomes: is there sufficient opportunity to exercise human agency (the capacity to act) when making military decisions on targeting?
Second, empirical trends from around the world in terms of development and use of AI in the military domain demonstrate an interest in using AI systems to support humans, rather than necessarily replacing humans in fully autonomous ‘killer robot’ types of application. Prominent cases include the Israeli Defense Forces’ uses of AI-based decision-support systems (AI DSS) in Gaza, as well as the US military’s use of the Maven Smart System in its war on Iran. There have also been reports of the US military using Claude to plan the abduction of Nicolas Maduro, with President Donald Trump reportedly using xAI’s Grok model to ask questions about how the Venezuelan people would react to such an abduction.
A recent CNN report further highlighted the relevance of this question. It mentioned that the US military almost boarded a Chinese vessel by trusting intelligence that claimed that the Chinese ship had nuclear weapons on board. This intelligence report turned out to be generated (or ‘hallucinated’) by an AI system. A military confrontation between two major powers almost started not only due to AI itself and its technical malfunctions, but due to how military personnel and decision-makers interacted with AI outputs. This case illustrates how maintaining an appropriate level of human judgement in such crucial decisions is really what is at stake here. As prominent scientists Timnit Gebru and Emily Bender argue in a recent op-ed, it is not superintelligence we should be concerned about, but existing uses of AI and what they mean for humans.
Ensuring the Role of Humans: Challenges Ahead
Safeguarding the role of humans is something that global governance debates recognize. In one way or another, the role of humans is present in all Responsible AI initiatives in the military domain: whether it is referred to as meaningful human control, human judgement, or by another term. For example, the most recent Responsible AI in the Military Domain (REAIM) Summit’s outcome document, the REAIM Pathways to Action, notes that “AI-enabled decision support systems should support, not replace, the exercise of human judgement”.
But while governance debates note the importance of this human role, the challenge is to agree on what it means in practice and implement it. Ironically enough, the REAIM Pathways to Action did not really provide concrete pathways, and the discussion remains around general principles. There are many challenges to operationalizing these principles.
First, there is a lack of support from major military AI developers. Many of them have been consistently against new regulation in this area, such as Russia, Israel, India and the US. Such states continue to interfere in diplomatic processes. The Washington Post reported that during recent negotiations on autonomous weapons in Geneva, when the Group of Governmental Experts on LAWS was drafting its final report, the US and Russia deployed their resourceful delegations to change the report’s text. For instance, they removed “a provision requiring that humans review military targets developed by AI before a strike”. With a vacuum from so-called great powers, a key question is: can middle powers rise to the challenge and lead on military AI governance?
A second challenge is that different strategic environments are interpreted differently. Many states and especially middle powers such as Canada, the UK, or Australia, are pursuing AI adoption and innovation because they consider it a priority to deal with the strategic situation in their respective regions. There is concern that imposing too many restrictions on ensuring a human role risks preventing AI development. This concern is guided by the idea that regulation automatically prevents innovation. However, middle powers also have a unique opportunity to develop defence technological capabilities while securing a high quality of agency for humans, not falling into the false innovation vs. regulation narrative. In fact, guidelines on the role of humans may strengthen innovation and secure adoption of AI.
Finally, there are different financial, human, and technical resources available to actors who seek to implement Responsible AI principles. At the Center for War Studies, we conducted the AutoPractices project, which brought together stakeholders from all over the world to co-create a practical toolkit with best practices to strengthen the exercise of human agency across the AI lifecycle. Some examples of these practices include conducting extensive risk assessment frameworks; paying attention to defence procurement and not being too reliant on vendors; and conducting continuous training and education of humans (not only military personnel, but all humans involved across the lifecycle). One of the issues that constantly came up in the project’s discussions is that many of these practices require resources that most states do not have. To address this challenge, there is much potential for middle powers to play a key role in capacity building and sharing of best practices.
Dr. Anna Nadibaidze is a Balsillie Scholar at the Balsillie School of International Affairs and a Postdoctoral Researcher at the Center for War Studies, University of Southern Denmark (SDU). She investigates artificial intelligence (AI) technologies in international security and the global governance of military applications of AI. Anna holds a PhD in political science from SDU. Her research is part of the Independent Research Fund Denmark-funded HuMach project, as well as the European Research Council-funded AutoNorms and AutoPractices projects.
She recently spoke at the CPG/CIPS Conference held on October 2nd, titled “Canada and the Middle Power Challenge“.
