On Scenario Politics

Is it legitimate to make assumptions in public?

Introduction

The AI Futures Project wrote a “Plan A”, a positive vision for “what should happen” until 2040. It outlines goals (the solution of five problems, from the „Loss of control to superintelligent AIs“ to „AI taking everyone’s jobs“) and recommends a path to reaching them.

In this post, I touch on three questions:

1. How closely should a scenario’s normative assumptions resemble its recommendations?
2. How much room should scenarios or plans like AI 2040 leave for (potentially decisive) future developments that aren’t about AI?
3. What makes scenarios like AI 2040 useful in the first place?

1 On Normative Assumptions and Exogenous Factors

To start, other people have written reactions and critiques about AI 2040, which I recommend reading: Richard Ngo, for example, does a great job explaining how the concept of a zoomed out “Plan A” rests on quite a lot of assumptions about the decisive importance of “race dynamics”, and that a scenario about the broad question on “what will be feasible and necessary” can make it difficult to separate normative assumptions from descriptive forecasts.

I had two related questions when reading the document, on the authors’ assumptions about normative questions and political trade-offs, and on their view about the (ir)relevance of ‘exogenous’ factors.

Normative assumptions and policy recommendations

If the authors’ assumptions are too similar to the recommendations at the end of a scenario, I can’t help but wonder whether the scenario is a long-form articulation of a world where these assumptions happen to be true and important. For example: if security/transparency tradeoffs are assumed to “favour paying significant security costs for improved transparency’” or if a “slowdown deal is assumed enforceable for at least a year or two even in the most pessimistic case”.

Assumptions about the irrelevance of exogenous factors

I’m unsure whether this scenario is compatible with exogenous non-“AI-related” forecasts about the future – and how much the authors have engaged with them. This question comes up in two parts in particular:  Firstly, in conditional forecasts. The first listed assumption is that AI R&D will be fully automated within ~15 years, absent “major regulation, war, or societal collapse”. I’d love to know what the authors think major regulation, war, or societal collapse would have to look like to stop AI R&D from being fully automated. For example: The US is (arguably) at war with Iran, and Russia is definitely at war with Ukraine. I don’t think the authors consider this a relevant factor influencing the AI-R&D process. Secondly, in the noticeable absence of factors we often assume to be quite influential in the next 20 years. The word “climate change” shows up once, in appendix N, which describes what it might have felt like to live through the last five centuries in five years. At the very end of the five years, “You hear about climate change, gene editing, cryptocurrency.”

Compare this vision to the National Intelligence Council’s 2021 estimate of how climate change will impact US security interests in the next 20 years.: 

“Geopolitical tensions are likely to grow as countries increasingly argue about how to accelerate the reductions in net greenhouse gas emissions that will be needed to meet the Paris Agreement goals. Debate will center on who bears more responsibility to act and to pay—and how quickly—and countries will compete to control resources and dominate new technologies needed for the clean energy transition.  Most countries will face difficult economic choices and probably will count on technological breakthroughs to rapidly reduce their net emissions later. China and India will play critical roles in determining the trajectory of temperature rise.” 

One might argue that, in practice, scoping decisions just have to be made – and imagining a “Plan A” that takes into account the whole world in intricate detail is simply not feasible. But this only makes the limitations of a plan A understandable. It doesn’t make these limitations go away. It’s helpful to acknowledge our uncertainties about the future, but it’s also helpful to engage with the chaotic, seemingly unrelated, but potentially decisive events and developments unfolding in other domains. Some of these are described in existing scenarios about these domains. A “Plan A” that assumes all relevant events will be a strict consequence of AI development, deployment and user decisions will almost certainly be less useful than one that doesn’t. Scenario scrutiny is a great practice – but it is, in fact, difficult to “achieve” in any meaningfully absolute sense.

To reflect on these two limitations – assumptions about normative claims, and assumptions about exogenous factors – and on how much they matter, it might be worth reconsidering what scenarios are used for in a policy setting.

2 What are Scenarios for?

I think there are basically three types of use-cases on a spectrum from “relying on few assumptions” to “relying on many – or only – assumptions”

On one end of the spectrum, a clearer description of the future, focusing mostly on descriptive information and empirical evidence.

On the other end, a normative case, a utopia or dystopia written to make a point about what a “happy end” should look like.

And then, there is a large space in between: A narrative exploration of potential and contingent futures, an argument for how a policy decision will affect the future, in more or less productive ways.

I’ll go into each of these a bit more in detail, starting from the purely descriptive end of the spectrum.

1. To describe the (capital F) Future

Scenarios can paint a clearer picture of what will happen in the future — so that policy makers can discuss and ideally answer all the nitty-gritty normative and political questions about what should be done “given the situation”. Scenario writers assume one very concrete event as a premise, and describe the probable effects of such an event occurring on some other part of the world. E.g. what happens to the North Pole if the global average surface temperature rises above 2 degrees compared to pre-industrial levels?

Policy-makers are then left to grapple with two questions:

1. Is this forecasted event going to happen? Do I believe the scenario?

2. If it is, can something be done to prepare for it? What should be done?

In the case of a pandemic, for example, forecasts like this are vital. Predictions about how many people will need access to hospital beds, for example, help policy-makers plan ahead. The discussion of exogenous factors is relevant in this kind of scenario writing insofar as it helps policy makers decide whether they should question the premise of the scenario. If the premise of the scenario is built on a large body of empirical evidence, the discussion of exogenous factors becomes less important.

2. To describe a potential and contingent future

People who write scenarios often don’t want to make a forecast about what they think will happen, but illustrate what they think could happen in the future. Possible “risks” and “benefits” can be described, and modulated into different stories: here are the risks if policy makers do nothing, here if they pursue policy a, policy b, policy c. Usually, the policy with the “lowest” risks and the “largest” benefits is recommended. Risks and benefits are normative terms, and how they are weighed, and which ones are considered, are normative assumptions.

This kind of scenario is more understandable (and more rhetorically effective) if future possibilities are described in vivid detail, with good prose and a lot of concrete examples and footnotes about “what this would actually look like” – the more concrete the scenario, the more imaginable the potentiality of a future event. It’s also cleaner, and more readable, if uncertainties, exogenous factors and normative assumptions remain bracketed in an appendix. 

These kinds of scenarios are often marketed as a way to “improve policy decisions”. It’s true that they can be thought provoking, provide the space to ask questions “readily open to red-teaming through ‘what if?’ questions” and make a helpfully intuitive case for why people shouldn’t immediately assume that there’s “nothing to be done”.

There are also a few reasons to doubt the objectivity of the improvement that scenarios claim to provide:

• Firstly, possibilities are much easier to justify than probabilities. Possibilities just need to be internally consistent, and seem plausible, i.e. imaginable, to a desired readership. Scenario-writers don’t need to take into account exogenous factors to describe the possibility of a catastrophic event in large enough detail to establish legitimacy for “having to do something” – and a “something” policy to “reduce risk”. This is very common in securitised discussions: De Goede (2008) describes the post-9/11 security imagination as one of “premediation”: the imaginative enactment of possible futures – through scenarios, exercises, and visualisation – to render potential catastrophe actionable in the present. Amoore (2013) calls this the “politics of possibility”, in which authorities act on what could occur rather than what will, deriving present interventions from virtual visions of low-probability, high-consequence stories about the future.
• Secondly, they can lead readers into different kinds of overconfidence: Usually, the use of “this could be a risk” scenarios is criticised because it generates artificial legitimacy for overly precautious actions, and powerful actors establishing levers of control over a large portion of the future to “minimise risks”. I also think they can produce overconfidence in the other way: Vivid stories about how risks can be “minimised” (even if they’re just called hypothetical or illustrative) can make the future seem understandable enough to become plausible, and then plausible enough to become familiar, and then familiar enough to either seem manageable (i.e. not that urgent) or unavoidable.
• Thirdly, they can bias towards concrete, visible and easily describable risks. If scenario writers want to make a potential development (e.g. growing frustration among younger generations about the lack of entry-level opportunities) seem understandable, they benefit from picking very concrete examples. But the more concrete an example, the more it can anchor the reader on that particular instance rather than the broader phenomenon (e.g. “400,000 graduates protest in London”). The audience may treat that exact example as the risk scenario writers try to describe, rather than the cause of that risk.

3. To make a normative point

Scenario writers can make normative points. By vividly describing a world where people get hurt, and others do nothing about it – or a world where a seemingly acceptable state of affairs has unacceptable consequences if you “zoom in”.  They can also describe a positive vision of a future to aim for or look forward to. In each case, they make normative points and inspire readers to reflect on their values in different ways: 

• …by choosing to focus on a particular utopian or dystopian aspect of a world, or by describing how a seemingly innocuous ideology inspires actions we might find completely unacceptable on reflection;

• ..by focusing on a particular population as being more agentic (and therefore, more responsible) for the course of history, while others are described as bystanders, passive on-lookers or victims;

• …by painting one kind of suffering as tragic, but necessary or unavoidable, and other kinds as unacceptable and contingent;

• ..by describing a future as a “happy end” to aim for, justifying all the means that lead to it.

I think people often underestimate the value of utopias and dystopias for policy debates, because they don’t seem as “fact-based” as the other two use-cases of scenarios. But the value of these scenario’s doesn’t depend on their actuality, in the same way that the value of a piece of fiction doesn’t depend on how realistic it is. Whether or not they are “correct”, they make ideologies explicit enough to be discussed. They expand the political imagination of their readers, either because they go “too far” or “not far enough”. They can encourage people to question what is considered “necessary by default” or “unrealistic”, whether they are based on accurate forecasts or not.

3 Conclusion  

I’m not sure we know that scenarios like AI 2040 actually improve policy decisions in an epistemic sense much more than, say, a thoughtful opening statement on a debate stage or a draft piece of legislation. These, too, can be thought-provoking, open up important questions, refer to important information, and help us make sense of what other people think. I think AI 2040 belongs less to the first, and more to the second, and possibly third category of scenarios.

Scenarios like this are not objective statements, infinitely transparent or verifiably rational, they don’t account for an infinite amounts of strategic surprises or exogenous factors – but, importantly, scenarios can be legitimate (and good!), even if they are not derived from a “view from nowhere”.

They can be legitimate, for example, because they discuss a moral principle, articulate a goal. Policy proposals can, similarly, be legitimate because people vote for them. Or because they provide a foundation on which to build something better. Or because they deliberately allocate responsibilities according to a process people generally endorse. Or simply because a person thinks they are worth proposing. Documents like these, with all their contestable assumptions and potential tautologies, are a contribution to a policy (and often political) debate, an intentional step towards a collective decision – which, I’d argue, we shouldn’t take for granted, even (and perhaps especially) if we disagree with them.