AI Governance

Overview

Artificial intelligence is gaining remarkable capabilities, from solving complex mathematical problems and defeating expert human competitors in games to creating sophisticated images. These technologies have the potential to benefit humanity in numerous ways, such as detecting human trafficking, forecasting earthquakes, aiding in medical diagnoses, and accelerating scientific breakthroughs.

The capabilities mentioned pertain to "narrow AI," which excel in particular fields but lack the versatility of human skills. However, these systems also carry potential dangers. They could be engineered for destruction, exemplified by lethal autonomous weapons, misused intentionally, or produce destructive adverse outcomes inadvertently, such as through algorithmic bias.

The concept of "transformative AI" refers to future AI developments that could bring about changes as significant as the agricultural or industrial revolutions. This could be achieved through the creation of "artificial general intelligence" (AGI), which would possess human-level intelligence across all areas. AGI has the potential to significantly benefit the world by addressing critical global challenges. Conversely, it poses risks that could threaten human survival if it fails to align with human interests. For instance, a 2023 U.S. executive order zeroes in on models demanding extreme computational resources or focusing on biological data, referring to them as "dual-use foundation models" due to their broad applicability and potential security risks.

Given these stakes, the urgency of ensuring transformative AI benefits humanity cannot be overstated. Predictions indicate that transformative AI could emerge within the next few decades. Surveys from 2022 reveal a consensus among experts on the tangible risk of AI leading to catastrophic outcomes, with a substantial portion highlighting the underinvestment in mitigating these risks compared to the efforts to advance AI technology.

One way to steer the development and application of AI towards positive impacts is through AI governance research. This involves exploring and establishing international norms, policies, laws, and institutions that will influence how AI technologies affect society. This research spans broad inquiries, such as the timeline for AGI development, its geopolitical implications, and the ideal framework for AI governance. It also includes investigating AI's potential effects on cybersecurity, and formulating specific strategies, such as laboratory policies that encourage ethical research practices.

Protecting against Dangerous Capabilities and Extreme Risks

Risks are broadly categorized into Misuse and Misalignment. Misuse involves malevolent applications by individuals or entities, such as cyberattacks or bioweapons. Misalignment refers to scenarios where AI systems act contrary to their creators' intentions, which can arise from a myriad of issues, including improper encoding of human values or the system developing autonomous objectives.

Additional risk categories include Structural risks, relating to AI's interaction with societal dynamics; incompetence risks, where AI fails at its assigned tasks; and risks stemming from the AI development race or organizational failures, like neglecting safety research or ignoring AI risks.

As general-purpose models evolve, they might develop unforeseen capabilities, some potentially hazardous, amplifying risks like misuse or misalignment. Identifying, measuring, and regulating these capabilities is crucial to prevent the deployment of potentially dangerous models.

Interventions and policies

Policies can target various stages of AI development and deployment, from the ecosystem level—like forecasting or resource regulation—to specific stages like model licensing, capability audits, or deployment regulation. Policy implementation can range from lab self-regulation to national regulations, with an emphasis on international cooperation as a means to mitigate certain risks, like state-led AI arms races.

Some specific policy recommendation ideas include:

  • Software Export Controls: Implement export controls on "frontier AI models" with high general capabilities or those trained with significant compute resources, and restrict API access to limit their proliferation.

  • Require Hardware Security Features: Mandate security features on cutting-edge chips for governance, such as compliance verification, activity monitoring, usage limitation, and emergency interventions.

  • Track Chips and License Clusters: Monitor cutting-edge chips and require licenses for assembling large clusters, enhancing visibility into and control over potentially dangerous compute aggregations.

  • Licensing for Developing Frontier AI Models: Increase government oversight on frontier AI development through licensing to control their spread and mitigate risks.

  • Testing and Evaluation Requirements: Subject frontier AI models to rigorous safety tests and independent evaluations to ensure their safety.

  • Fund Alignment and Model Evaluation R&D: Allocate funds specifically to research and development in AI alignment, interpretability, and model evaluation to narrow the gap between AI capabilities and safety methodologies.

  • AI Incident Reporting Requirements: Mandate reporting of AI incidents to regulatory bodies to track potential harms and refine mitigation strategies, maintaining confidentiality as needed.

  • Clarify AI Developers' Liability: Establish clear liability rules for AI developers, particularly for harms caused by frontier models, to motivate investment in safety and security.

Regulating Frontier AI

For effective regulation of frontier AI, it is important to create foundational regulatory elements, as self-regulation alone may not adequately mitigate the risks associated with these advanced AI models. These elements include:

  • Developing Standards: Establishing and periodically updating standards for responsible development and deployment of frontier AI through inclusive, multi-stakeholder processes. These standards should not only target frontier AI specifically but also encompass broader foundational model guidelines, allowing for swift updates in line with technological advancements.

  • Enhancing Regulatory Visibility: Implementing mechanisms such as disclosure requirements, monitoring initiatives, and whistleblower protections to give regulators detailed insights into frontier AI progress. This information would cover aspects of the development process, the models themselves, and their applications, enabling regulators to identify areas requiring governance and devise effective regulatory strategies.

  • Ensuring Compliance: While voluntary self-regulation efforts like certification might contribute to safety standard adherence among frontier AI developers, government action is deemed necessary for ensuring comprehensive compliance. This could involve a governmental body converting standards into enforceable regulations, detecting and penalizing rule violations, or establishing a licensing system for the development and deployment of frontier AI models. Balancing the regulation of frontier AI to avoid both stifling innovation and lagging behind technological progress poses a significant challenge for regulators.

What might safety standards look like in practice?

  • Risk Assessments: Carrying out detailed risk evaluations based on potential dangers and manageability to prevent unforeseen hazardous capabilities or erratic behaviors in deployed models.

  • External Expert Reviews: Enlisting independent experts to scrutinize models' safety and risk profiles, enhancing the rigor of assessments and ensuring accountability.

  • Deployment Guidelines: Utilizing risk assessment outcomes to guide decisions on model deployment and the implementation of necessary precautions, ranging from unrestricted use to complete non-deployment until risks are minimized.

  • Ongoing Monitoring and Adaptation: Continuously tracking and adapting to new insights about model capabilities post-deployment, with provisions for reassessing risks and adjusting safeguards as needed.

Compute Governance

Computational power, or "compute," plays a crucial role in driving advancements in AI. In the last thirteen years, the compute used in leading AI systems has increased by a factor of 350 million, causing significant breakthroughs that have caught worldwide attention.

Noticing these developments, governments are increasingly focusing on compute governance as a strategy to achieve AI policy objectives, such as reducing misuse risks, bolstering domestic sectors, and competing on the geopolitical stage.

Governments can employ compute governance in AI through at least three methods:

  • Tracking or monitoring compute to understand AI development and usage better.

  • Adjusting access to compute to influence resource distribution among AI initiatives.

  • Integrating monitoring and restrictions directly into hardware to ensure compliance.

Compute governance emerges as a viable method for AI regulation due to several factors. Advanced AI system training requires vast numbers of sophisticated AI chips, making compute both detectable and controllable. The physical nature of AI chips allows for their distribution to be managed, and their characteristics and utilization are measurable. The efficiency of compute governance is also supported by the AI supply chain's concentration, with only a handful of companies possessing the capabilities to produce the necessary advanced chips, manufacturing equipment, or host data centers.

Why work on AI Governance?

We see the following reasons why it might be wise to prioritise working on AI governance to mitigate risks posed by AI:

  • Leverage: A few key stakeholders hold significant power when it comes to making decisions that will shape centuries to come. In periods of high “plasticity”, improving policy and governance on pressing problems can be highly impactful. 

  • Neglectedness: 80,000 Hours estimates that there are “around 400 people working directly on reducing the chances of an AI-related existential catastrophe”, and few of these work on AI Strategy or Governance. 

  • Talent bottleneck: There is a small overlap between people who have a good technical understanding and those who have the policy skills to effectively communicate knowledge to decision makers. 

  • Transferable career capital: Policy research is needed in many fields, and provides researchers with a wide variety of tools and skills that are valuable in a multitude of governmental entities, nonprofit organisations and private industry. 

Learn More

AI Governance Curriculum — AGI Safety Fundamentals

AI Governance: A Research Agenda | GovAI

Centre for the Governance of AI

AI Governance Research Group @ FHI

Center for Security and Emerging Technology (Georgetown)