Hawkish nationalism vs international AI power and benefit sharing
A call against allowing nationalism to steer the AI revolution and for spearheading the frontier of AI globally
TLDR: In response to Leopold Aschenbrenner’s ‘Situational Awareness’ and its accelerationist national ambitions, we argue against the claim that artificial superintelligence will inevitably be weaponised and turn its country of origin into an untouchable hegemony. Not only do we see this narrative as extremely dangerous, but also expect that the grandest AI challenges call for global coordination between rivalling nations and companies. We lay out ways in which sharing the benefits of – and even some power over – the most capable AI systems can help to build positive-sum partnerships with the best chance of maximally good outcomes of the AI revolution. Finally, we present the multitude of challenges associated with such collaboration and discuss possible solutions and reasons for optimism.
Table of Contents
1 Introduction
“Situational Awareness” is an opinion piece written by Leopold Aschenbrenner, a former researcher at OpenAI. It advocates for a concentrated effort by the US government (USG) to race towards Artificial Superintelligence (ASI) as quickly as possible. The central narrative can be summarised as follows (not a direct quote):
The free world must achieve ASI first. It can become the ultimate weapon, powerful enough to negate even the mutually assured destruction by nuclear warheads. Military proliferation of ASI is inevitable and it is the responsibility of the USG to protect democracy and liberty by becoming an invincible hegemony.
While calls for a US-based Manhattan Project for ASI development have appeared in public discourse before, this point of view has recently grown in influence (source 1, source 2) with the appearance of Aschenbrenner’s piece. The purpose of our article is to challenge this opinion in a constructive way that adds to the discussion. More specifically, we are mainly concerned with the question of how to stabilise and improve geopolitical dynamics in the face of highly powerful AI systems with economic and military relevance. We compare the above viewpoint with a different, less antagonistic perspective:
A reckless arms race towards ASI goes to the detriment of the safety of humanity as a whole. Dealing with an AI-fueled nation to nation conflict is better handled through soft power: Sharing benefits and power provided by the most capable AI systems in a controlled, strategic manner. Positive-sum trade cannot only improve international relations, but also economic development and human wellbeing much more effectively than nationalistic strategies.
In the scope of this article, we will consider Transformative Artificial Intelligence (TAI) according to a common (albeit vague) definition where TAI is ‘an AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution’. We will not consider the impact of ASI which achieves performance at a level far above humans. The enormous power of ASI means that social interactions, economic mechanisms and geopolitical dynamics are likely to be transformed in ways so significant that current forecasting is marred by uncertainty. In particular, if one actor obtains a ‘decisive strategic advantage’, the decisions made by that actor plausibly make it very hard to predict the outcome. We believe that the relevance of this article increases with longer TAI timelines and longer transition periods between TAI and ASI. This is because slower development speeds increase the probability of multipolar outcomes and give more time for governance and collaboration to come into effect. While in a fast takeoff scenario the systemic and misuse risks will certainly be severe, the potential misalignment risks stemming from a hastily developed ASI might dwarf them.
2 Advantages of Benefit and Power Sharing
Preliminary
There is a distinction to be made between benefit sharing and power sharing. Here, we borrow the terminology and ideas from Michel Justen’s ‘Sharing the AI Windfall: A Strategic Approach to International Benefit-Sharing’, which defines these terms as follows:
Benefit-sharing: sharing AI-derived benefits that don't significantly alter power dynamics between the recipient and the provider.
Power-sharing: sharing AI-derived benefits that significantly empower the recipient actor, thereby changing the relative power between the provider and recipient.
2.1 Reducing arms race incentives
AGI risks are transnational and exceed the borders of individual countries. Given the increasing global integration and the rising use of open models, it is increasingly naive to believe that a failure could be contained within a single country (source 1, source 2). In an AI arms race scenario, global coordination will be severely limited due to a lack of trust and concerns about information sharing and rule compliance.
Competitive pressure in the AI sector is already undermining safety research, as seen in the commercial space. This is likely one of the reasons why OpenAI disbanded its Superalignment team, as its resources could be otherwise used for developing more capable products. In a military context, such a race would generate even stronger pressure (source 1, source 2). Since the risk of falling behind is often weighted higher than potential damages from misalignment, systemic risks or misuse, safety concerns are at a danger of being deprioritized. Some of those AI risks are already apparent in the real world (source 1, source 2, source 3). Moreover, early warning signs of misalignment and deception have appeared in laboratory conditions (source 1, source 2, source 3).
To mitigate these incentives, global coordination could bundle resources and expertise and incorporate diverse perspectives into AI safety research. In Section 3.3, we propose actionable approaches to address these challenges.
2.2 Improved economic development through positive-sum trade
Failing to cooperate and refusing to share the AI windfall can cause tremendous damage to both international relations as well as human wellbeing. For this, consider a thought experiment where using AI to automate a vast range of previously irreplaceable human labour becomes possible.
The implications of this within one economic zone are being debated extensively, but we want to focus on what happens between one economy that has diffused this technology successfully and another one that is refused entry. The loss of competitiveness in the latter is detrimental to its export sector and, without protectionist trade restriction, also to its native market. In the absence of benefit sharing, economic downturn leads to lowering living standards, which leads to social unrest, which leads to political tension. This goes to show that even without targeted, malicious action, just a refusal to share benefits can cause major harm.
There are opportunity costs to consider: the Marshall Plan played a key part in supporting powerful allies for the US not only from a military perspective, but also helped with creating strong markets for mutually beneficial trade. From today’s perspective, this can be seen as a success that, together with the formation of international institutions (e.g. the EU, NATO, Eurozone), helped nations to unite and cultures to overcome centuries of bitter animosity. Of course, one can argue that this is more achievable between democracies that commonly value human rights. Now, consider its counterproposal: the Morgenthau Plan. Briefly summarised, it proposed a deindustrialisation of Germany to prevent it from being powerful enough to start a war ever again. One of the most important trading partners and strategic allies of the US would not have arisen if it was for this plan.
The other side of the argument might counter that a nation with control over an unrivalled AGI has nothing to gain from trading with outsiders, as those cannot produce anything competitive to an AGI-fueled economy. This simplistic position ignores a few factors. On one hand, AI-enhanced management and organisation can make human labour much more efficient, which has the potential of unlocking unprecedented opportunities for productivity and wealth creation, especially in highly populated parts of the world like China and India. Robotics needs time and resources to scale up, while human capital is readily available. On the other hand, natural resources and land cannot be substituted even with AGI (when leaving longer timescales and asteroid mining out of the debate). In fact, one can expect that scaling up AI usage and integrating it further into the economy will drastically increase resource demand. The third point is that underdeveloped economies can grow faster than already established ones. The ‘economic wonder’ phenomenon, where countries develop into economic powerhouses in mere decades, follows a pattern: younger cases show more rapid development likely due to technology transfer allowing them to ‘skip some steps’ that earlier industrialisations went through (source 1, source 2), as well as due to improved international communication and transport. There are also other arguments, such as the fact that economies generally benefit from scale, synergy of different comparative advantages and more. But we think we have demonstrated sufficiently that the position of ‘We have AGI, you have nothing we can’t make more cheaply’ is unfounded.
2.3 Incentivising participation in international AI collaborations
This is where the exercise of soft power comes in. Benefit- and power-sharing can come with conditions attached. These conditions can be set by those with the highest bargaining power, which as of right now when it comes to AI, is the USG. It is important to note that this might be subject to change and therefore there is a motivation for the USG not to risk missing out on this opportunity. Had the USG initialised negotiations on the nuclear Non-Proliferation Treaty (NPT) back when it was the only nuclear state, it might have secured itself a more favourable negotiating position.
On a broader note, such conditions can be stated as commitments to the rules of international treaties. Those treaties can serve a multitude of purposes. Two of those have already been discussed in the Sections 2.1 and 2.2. Further commitments can come in the form of:
AI safety best practices
Non-proliferation commitments
Standards creation and building scientific consensus on AI risks
Commitments to contribute to research
Allowing external inspectors to access one’s data centres, publishing detailed data about the GPU numbers, power consumption, etc.
If membership and compliance with treaties comes with rewards such as access to safe, beneficial technology and a ‘seat at the table’ where important global decisions are made, this could be a more effective motivation to comply with rules than the threat of sanctions could ever be.
We argue that rather than gambling on the success of establishing a global singleton, the USG is better advised to use its strong negotiating position to spearhead peaceful, soft power solutions while it still has control over the only cutting-edge AI hardware supply chain in the world. If it allows for international relations to become too adversarial or for other powers to build their own TAI technology before negotiations start, this opportunity will be irreversibly lost.
3 Challenges and possible solutions
We see an international collaboration on AI of the type described in International Institutions for Advanced AI as one of the most promising ways to realise a positive outcome of TAI. However, despite the reasons for optimism listed above, many challenges remain.
3.1 Which nation states should be included in power-sharing?
Any international project on AI will naturally have to involve the dominant players of this industry. Most notably, these include the US, China, Canada and the UK, where most AI research and entrepreneurship takes place today (source 1, source 2, source 3). Other countries with leading roles might also be Germany, France, Japan, South Korea, Israel and Australia, assuming they can keep up with the pace of AI innovation in the ‘Big Four’ states. In the next few years or decades, states in the Middle East may also play a key role in frontier AI development, in particular if they decide to offer attractive financial incentives to established AI corporations – for example to fund their increasingly expensive training runs. Finally, it is certain that the so-called Global South will not want to miss out on the TAI revolution either.
It is unclear how many countries should be given ‘seats at the table’. A format analogous to that of G7 or G20 might be most natural, although a large number of players might render the decision-making process unmanageably slow, especially if all countries have the right to veto new proposals. An obstacle might be encountered even earlier if the ‘Big Four’ decide to appropriate more legislative power to themselves and leave the others out. This trend can be exacerbated to the extreme if the leading country (the US, for now), decides that it should have sole authority over benefit-sharing and refuses to budge.
Interestingly, Aschenbrenner also deals with the question of who to share power and benefits with in ‘Situational Awareness’. His ideal scenario would be a coalition of democracies (NATO, Japan and South Korea) developing ASI together, with a strong emphasis on American leadership. He notably excludes China from such a coalition. Moreover, sharing of the technology benefits with non-coalition states should, he argues, be tied to the condition that they commit to not developing ASI themselves. We can see the virtue in his worldview: states that do not respect human rights and freedoms are not to be trusted with power over TAI/ASI and need to be kept in check by those who do. However, he makes three assumptions we see as problematic:
That a rushed development of ASI does not lead to disaster
That overwhelming military superiority (even negating nuclear deterrence) is achieved before drastic escalation by those standing to lose everything
That when faced with an undefeatable enemy, China (and others) will accept submission and external control
Starting with the first one, ASI is plausibly the most dangerous technology humanity will ever have to deal with. The stakes are nothing short of existential. Building a mind that can outwit all of humanity combined is not something that should ever be rushed. It is not even clear that any amount of care and preparation can avoid putting humanity at an irreversible disadvantage. This is our main argument why a ‘race to the bottom’ has to be avoided.
Ambitions of using powerful technology to prevent adversaries from ever gaining access to it will not go unheard and will pressure them into even more careless development, for even a slight chance of catching up. Moreover, in order to make use of ASI as a weapon, one would have to authorise it to injure or kill humans, as well as to control or develop weaponry. Our – possibly controversial – view is that the creation of such systems is not unavoidable. The substantial head start of the US in AI technology may be seen by the likes of Aschenbrenner as an argument for not having to negotiate with anyone. In contrast, we see it as an argument against worrying too much about losing the lead and for seeing the technology itself as the more pressing risk.
We phrased Aschenbrenner’s second assumption in a way that already highlights the problem. Putting autocrats with access to nuclear weaponry into desperate situations (their adversary is about to become invincible, unless you strike their cloud servers first) is a risky bet on them choosing not to escalate.
The third of our arguments has much in common with the previous one. Can one assume that, say, China will choose not to fight an unwinnable battle or not to try to secretly find a way out of being defenceless, no matter how hopeless? Aschenbrenner suggests the coalition could commit to not using ASI against complying states, but we highly doubt that autocrats would trust this commitment, especially coming from states that see their autocratic power as problematic.
We should also not assume perfectly rational actors, but also consider emotions such as pride. Today’s China sees itself as a proud, 4000 year old civilisation that is rising up to reclaim its status as a global superpower – a nation and culture that has recovered from centuries of being defeated, exploited and humiliated by foreign powers. How much of this is historically accurate is up to debate, but this is the narrative that the Chinese Communist Party has diffused into the population over more than half a century. We doubt they would resign themselves into submission.
We conclude that TAI is an issue that forces powers to cooperate with one another, even when the ideological differences run deep.
3.2 Ideological differences
Indeed, we expect that in the process of formulating the power structure of a future AGI cooperation, both (or rather, all) sides might want to exclude other parties based on ideological grounds. For example, the US might be hesitant to include its biggest rival, China, because of concerns about human rights abuse and surveillance overreach. Whether these concerns are expressed genuinely or as a means of anti-competitiveness, it is true that TAI could enable non-democratic governments to strengthen their grip over their citizens, resulting in a dystopian scenario being realised. On the other hand, China itself might use accusations of war-mongering and colonial history when trying to discredit its adversaries in the West.
However, the global catastrophic risks posed by unregulated AI development may necessitate collaboration of countries with radically different worldviews. Like nuclear arms control, where the US and the Soviet Union cooperated despite profound ideological differences, preventing an AI-driven catastrophe could require a pragmatic approach that includes both allies and rivals. We see the racing dynamics that would arise from the lack of a multilateral deal as an outcome worse than maintaining the current multipolar balance in a post-TAI world. For all parties concerned, this might be a bitter pill to swallow.
3.3 Mutual trust and transparency
One of the key challenges in power- and benefit-sharing will undoubtedly be the issue of securing trust between all members of the treaty. Trust lies at the core of what makes agreements effective and successful in the long term. However, without appropriate paradigms and verification tools, it might be hard to achieve its required levels in the context of a technology as revolutionary as TAI. In this section, we review several existing and proposed ‘Confidence Building Mechanisms’ (CBMs) that serve this purpose.
Track record
A good track record of nation states honouring their commitments could contribute to an increased sense of confidence and trust in each other. However, we remain sceptical as to whether track record can be a significant factor. It is a hard task to find a state or agency that has never broken international norms or resigned from an agreement, either formally or de facto. Unfortunately, we expect that in a world where TAI is about to be developed, suspicions about other parties will be heightened as never before and no side will be willing to take bets based solely on the historical reputation of its opponents.
Verification mechanisms
More concrete verification tools will be needed to build the trust required for both sides to honour their commitments. In analogy to the inspections of the US and Soviet nuclear arsenals in the 60s and 70s, it might be possible to obtain transparency by allowing external inspectors to visit data centres in order to investigate their GPU content and power usage. Methods such as on-chip devices and renewable hardware operating licences can offer additional control tools. Nonetheless, one has to bear in mind that, due to algorithmic improvements, the number of GPUs or FLOPs performed in a given centre might not be an informative metric.
Compared to the hardware focused approaches mentioned above, software inspection is even more challenging. Due to its complexity, as well as progress in architectures, metrics such as the number of parameters in the neural network will not be robust either. Models small enough to fit on consumer devices are far behind state-of-the-art today, but may exceed human performance in the near future. Giving an auditor API access to a model will also not be enough due to concerns of behind-the-scenes manipulation. A full review of compute governance, transparency building mechanisms and verification tools is beyond the scope of this piece, but we list some ideas below:
Compute governance: This can be realised in different, not mutually exclusive ways. One idea is to keep track of the AI hardware supply chain and what data centres are being built where. FLOP counters built into the hardware could be introduced as a legal requirement and even kill switches have been proposed.
Third-party monitoring: In literature, the International Atomic Energy Agency is often used as a comparison. Among other tasks, it manages inspections of nuclear facilities with respect to compliance to the NPT. There have been cases where violations of the treaty were successfully uncovered. In the case of AI, monitoring would probably have to focus on hardware deployment and usage, though other ideas exist:
Zero-Knowledge Proofs (ZKP) in Software Verification: Use ZKP to verify model capabilities without revealing the underlying code. A common counter argument here is that deceiving inspectors is too easy of a task. According to the argument, one could simply show a complying model to them, instead of a secret, more dangerous version. That in turn can be countered by demanding that every training run above a certain compute threshold must be mapped to a resulting end product. That way a treaty organisation can keep track of what was trained where and when.
Secure Multi-Party Computation: Techniques to ensure compliance checks can be performed without revealing proprietary information. Requiring the usage of certain blockchain technologies and trusted execution environments to verify the origins of AI models is an interesting idea to explore, which does not require laying open IP protected company secrets.
Tracking Key Researchers: Countries could monitor the movement of top AI researchers, similar to how nuclear physicists have been tracked historically (the story of A.Q. Khan comes to mind). Any clustering of expertise in unexpected areas could trigger scrutiny. We should note, however, that such a solution is highly controversial and can enable grave violations of human rights. In extreme cases, it could even lead to assassinations of key figures.
Encouraging whistleblowing and establishing secure, anonymous reporting channels for researchers and engineers can provide a crucial safety net.
‘Seats at the table’ for decision-making related to AI development: This can be thought of as a power-sharing measure. Inclusion in decision making can come in different, non mutually exclusive or interdependent forms:
Influence how frontier AI is developed, evaluated or deployed
Cooperation on technical AI safety research
Invitations to join applied ethics working groups or commissions, or high stakes conferences, summits or conventions.
Other, more drastic power- and benefit-sharing like transferring leading edge AGI research towards global research organisation(s).
The global treaties, institutions and platforms for these measures do not yet exist for AI and from the perspective we present, they can be seen as long overdue. Some proposals for this exist, but they are still in their infancy and tend to be broad and vague.
We will go into more detail on verification and confidence building mechanisms in a future article, offering a more comprehensive overview on the promise and feasibility of different methods. We will review available technologies and key areas of research.
Phased implementation with reversible steps
Sharing the AI Windfall identifies a staggered rollout of commitments as a promising CBM. We think this approach is promising as it reduces the initial cost to entry and gives all sides the confidence that they could reverse back to the default state should one party defect. We also believe that after such early commitments have been implemented and respected for some time, it is important to move to the next stage of the treaty and make these early steps irreversible.
The Intermediate-Range Nuclear Forces (INF) Treaty between the US and the Soviet Union serves as a good example. Its goal was to eliminate both countries' land-based intermediate and short-range ballistic missiles. Before the final agreement, the two parties engaged in a series of smaller commitments, such as exchanging detailed information on missile types, ranges, and deployment locations. These were reversible in nature, as either side could have halted information sharing if they suspected non-compliance. Once the INF Treaty was signed, these initially reversible commitments were solidified and ‘locked in’ through a combination of legal obligations and detailed verification protocols. This approach ensured that each side’s actions were transparent and verified, preventing backsliding and cementing the commitments over time.
3.4 Implementation of punishment
Any treaty on AGI cooperation will have value only as long as all parties fear the punishment that will result from breaking it. We believe it is crucial to have predefined consequences for violating the norms of the treaty. These should be unambiguous and written down in law, rather than discussed and agreed upon only once a violation has occurred.
Historically, sanctions have been used as one of the main means of punishment. We remain sceptical as to their effectiveness in this context. First, member states might be reluctant to actually implement them. In the weeks following the Russian aggression on Ukraine in February 2022, members of the EU struggled to reach a consensus on what sanctions should be enacted out of fear of the knock-on effects they would have on their own economies. This effect is only exacerbated in the case of US-China relations, where the two countries are much more closely interlinked and depend on each other for trade benefits. In fact, China is currently the second largest holder of the US debt, only after Japan. Historically, while the US has been eager to target the Chinese military industry, it has been more reluctant to restrict the commercial sector, at least until recently.
Secondly, the adversaries of the US have gone to great lengths to evade sanctions and export controls, for example by creating networks of shell companies. A more promising way of implementing punishment seems to be the removal of the ‘seat at the table’. This effectively demotes a member state from power-sharing to benefit-sharing, permanently consigning it to lagging behind the leaders. Such a punishment is unambiguous and would be feared more than sanctions. A separate, but highly sensitive issue, is how to decide whether a condition has been violated. This calls for the greatest possible clarity of rules. For example, if our goal is to prevent a nation state from embarking on its own domestic AGI research program, we could use third-party evaluations to benchmark their models’ capabilities. More generally, third parties can be helpful when it comes to evaluating treaty compliance in an impartial manner.
3.5 Robustness of case studies
While historical case studies of international cooperation and benefit-sharing can serve as inspiration and provide useful anchors, one should also be careful not to extrapolate their conclusions too far. In particular, the analogy of ‘CERN for AI’ (source 1, source 2) suffers from one fundamental flaw – in the case of particle physics, there are no immediate monetary or power incentives to being at the forefront (although there are incentives of prestige and reputation). In contrast, it is clear that in the case of TAI, establishing a lead independently brings enormous financial and strategic gains. This might incentivise one or more players to take the gamble and defect.
3.6 Stability of policy across electoral cycles
In the case of democratic countries whose leaders and governments change frequently, it is unclear whether the stability of foreign policy on AI can be secured across administrations. The key player in the AI industry, namely the US, operates on a 4-year presidential cycle and a 2-year congressional cycle. Frequent changes of posture will almost certainly be detrimental to mutual trust and might incentivise race dynamics. Rather ironically, the foreign policy of a more authoritarian state might be easier to predict and prepare for, even if this goes against the values of democratic societies.
3.7 Benefit absorption
There is an entirely different class of problems with benefit-sharing, separate from the implementation problems described above. Consider a scenario where everything goes well. One or more players have the ‘seats at the table’ and agree on a power-sharing scheme between themselves, as well as a benefit-sharing one for players that they mutually agreed to leave out. TAI arrives, turns out to be aligned with humanity, and the whole world is ready to reap its benefits. Even in this scenario, the existing problems and inequalities are far from being solved. Worse still, we argue that due to the lack of necessary infrastructure in underprivileged countries, the global inequalities might actually grow.
Consider first the cost of API access to a popular LLM chatbot like Claude:
“Claude Pro is available for a monthly price of $20 (US), £18 (UK), €18 (EU), or local currency equivalent, plus applicable tax for your region.”
As an example, the differences in the median income between the US and Mexico mean that the relative cost of a monthly Claude Pro subscription differs by a factor of ~8 in these countries. This is particularly striking when we consider that Mexico is in the top 40 countries with highest median incomes. Such prohibitively large costs hurt not only private individuals, but also business, educational institutions and developers. Thus, relative pricing might be necessary to ensure equal access to the fruits of AI.
A fact that is often overlooked by those in developed countries is that a huge portion of the world’s population still does not have access to the internet. As of 2024, the global internet connectivity is estimated at around 67%. Naturally, this access is not evenly distributed across all continents – most notably, in Africa this figure is only 37%. Developing nations are of course not going to be direct competitors to states like the US and China any time soon. We should strive to increase their access to the internet as early as we can, such that they can also benefit from TAI once it arrives.
Even after securing internet access, the problem of skills and education remains. Underdeveloped countries may struggle to adapt to the rapid pace of changes, both at an individual and institutional level. We imagine that privileged countries should sponsor upskilling programs for local authorities and educators that can then disseminate this knowledge in their communities. Special care will need to be taken to ensure that this newfound knowledge is not usurped by these people to further entrench existing power structures and enable corruption or abuse.
Finally, more prosaic factors might be neglected by the creators of TAI – does their product have local language support and does it come with an accessible documentation? Is it geo-restricted due to local regulations (such as the GDPR policy in the EU)?
3.8 Intellectual property and government intervention
As of October 2024, the status quo is that the most capable multi-domain AI systems originate from commercial research, namely by Anthropic, OpenAI, Google DeepMind and Meta AI. The multi-billion dollar investments into these companies are tied to the expectation of the returns generated by their intellectual property – their AI products. That is the important point to make here: the technology is not owned by the USG, but by private business. IP rights in the US are strong and provide private owners of proprietary technology with the freedom to choose whether or not (and at what price) to share or sell their products and services. At what point TAI transcends the status of a commercial product and what that new status would even be, is up to definition and is likely to be shrouded by ambiguity.
OpenAI serves as an example of a case where market pressures and incentives can outcompete the philanthropic ideals of benefit sharing (which were a core value for the company when it was originally founded as a non-profit). Very recently (September 2024), OpenAI announced its transition into a for-profit structure. At the time of writing this article, the company website’s explanation of its structure contains the clauses:
Third, the board remains majority independent. Independent directors do not hold equity in OpenAI. Even OpenAI’s CEO, Sam Altman, does not hold equity directly. His only interest is indirectly through a Y Combinator investment fund that made a small investment in OpenAI before he was full-time.
Fourth, profit allocated to investors and employees, including Microsoft, is capped. All residual value created above and beyond the cap will be returned to the Nonprofit for the benefit of humanity.
These, among others, are now subject to change. The CEO, Sam Altman, has announced he will receive equity in the company. The profit by investors will be uncapped.
Fifth, the board determines when we've attained AGI. Again, by AGI we mean a highly autonomous system that outperforms humans at most economically valuable work. Such a system is excluded from IP licences and other commercial terms with Microsoft, which only apply to pre-AGI technology.
One key question for OpenAI’s future is whether this fifth clause still applies. The potential for conflicts of interest are apparent: its equity in the company incentivises the board to push the definition of AGI as far away as possible.
This conflict of interest is of course not an issue just for this particular company, but for society as a whole. Clear-cut definitions for what kinds of AI systems can count as proprietary products and which ones are too powerful to be treated (and sold) as such do not yet exist. Disagreements between different jurisdictions can cause trouble in international relations as well. It could be difficult for many nations to find a mutually satisfactory answer to this question and to agree on how to categorise different stages of TAI/AGI. Nonetheless, it is easy to argue for the usefulness of finding agreement on these questions before the answers are needed in practice.
It can be expected that this is where corporate lobbying will try to have as much of an influence on political decisions as it can, to further its interests as much as possible. Indeed, lobbying from the leading AI companies seems to have been one of the key factors behind the recent veto of the SB1047 bill in California.
Discussions are emerging as to when the USG would step in and take charge of TAI development or operation. The decision to consider certain AI technologies not only as economic goods, but also as a national security issue, is likely not for boards of companies to make. For the reasons laid out in this section, this would also make a lot more sense.
Regulatory frameworks for government intervention do exist: the Defense Production Act enables the federal government to prioritise or control production and to force companies to sell material and technology to it. The Eminent domain can similarly, if technology is classified as critical to national security, allow for disowning people or companies, under the condition that appropriate compensation is provided in return. The International Emergency Economic Powers Act enables the president to limit exports, monitor trade and – in extreme cases – freeze assets. Still, the power of the largest businesses to influence politics in their own favour should not be underestimated.
Here, we focused on the USA because of its substantial technological head start. We expect national government intervention to happen before decisions over TAI are globalised.
On an optimistic note, the owners and directors of the major AI labs are human beings as well. Many, if not most of them, want the technology to be beneficial for the whole world and want power over it to be in the right hands, even if they currently consider those ‘right hands’ to be their own. We see potential in benevolence, which we detail more in the following section.
3.9 Appeals to conscience
When considering the best ways of convincing key actors to join an AGI collaboration, one should not underestimate the importance of appeals to conscience. While such attempts might sound naive or even childish, there are historical precedents to suggest they might not be hopeless. A famous example is the story of how Ronald Reagan, who had a hardline stance on nuclear deterrence, gradually expressed a stronger interest in pursuing arms control throughout his presidency. This change of stance is often partially attributed to him seeing the movie The Day After, which presents a scenario of a nuclear exchange between the US and countries of the Warsaw Pact. In his diary, Raegan noted:
“Columbus day. In the morning at Camp D. I ran the tape of the movie ABC is running on the air Nov. 20. It’s called “The Day After.” It has Lawrence Kansas wiped out in a nuclear war with Russia. It is powerfully done—all $7 mil. worth. It’s very effective & left me greatly depressed. So far they haven’t sold any of the 25 spot ads scheduled & I can see why. Whether it will be of help to the “anti nukes” or not, I cant say. My own reaction was one of our having to do all we can to have a deterrent & to see there is never a nuclear war. Back to W.H.”
Eventually, Raegan signed the Intermediate-Range Nuclear Forces (INF) Treaty in 1987 with Soviet leader Mikhail Gorbachev, which was a major milestone in reducing the global nuclear threat. Naturally, other factors such as evolving global politics and his dialogues with Gorbachev also played significant roles in shaping his nuclear policy. However, we believe we should at least try to use such visceral representations of catastrophic outcomes of race dynamics to convince world leaders of the need for collaboration. More broadly, we can use them to raise public awareness, which might in turn create pressure from the general public on politicians to take these issues seriously.
An additional lever can come in the form of Track 2 dialogues involving top AI scientists from rival nations. If these scholars can reach consensus on prioritising AI safety over capabilities, we are hopeful that they can use their privileged position to influence policymakers in their governments.
4 Final thoughts
In this article, we have explored the issue of how to ensure that the incoming TAI revolution does not lead to a catastrophic outcome and can benefit the whole of humanity. While an international collaboration on TAI development has many advantages and incentives, it is not without problems of its own. These challenges are strikingly multidisciplinary, requiring the expertise and input of scientists, economists, policymakers, sociologists and activists. Inspired by historical case studies such as the International Space Station, the nuclear Non-Proliferation Treaty or CERN, we remain hopeful that a multilateral deal with power- and benefit-sharing provisions emerges as one of the most promising ways to reduce race dynamics and systemic risks from TAI.
The current situation, where the most powerful AI technology is owned by private business, does not have to stay that way. Large-scale state or internationally funded research efforts can still outsize what even the likes of Google can build. If and when the potential for catastrophic risks of TAI make it into the political consensus, compute thresholds and/or other limits can have a chance of making it into global treaties. Then, pioneering into uncharted and potentially dangerous territory can become the task of a global, non-profit research organisation with more transparency in its structure and more humanity-aligned incentives. Building such an organisation is unlikely to happen from scratch and will have to include public-private partnerships together with at least the hardware and cloud infrastructure providers. Regulatory capture, corruption, lobbying and other profit incentive-driven problems are major challenges to work on, as with any large partnership. In the case of the most powerful AIs of the world, these failure modes could reach new extremes.
At the heart of the Prisoner’s dilemma is the key problem that the prisoners cannot communicate and coordinate. Transparency and cooperation is how we can try to avoid an AI cold war. Power sharing may sound absurd when one is convinced of their own superiority and invulnerability. But what historical experience has shown is that overwhelming military superiority alone does not guarantee success or even peace (Vietnam War, US retreating from Afghanistan, Iraq War, etc.). Benefit sharing could allow TAI to improve living standards like no previous technological development in history. It is not in the interests of the owners and investors of AI to live in a world where everyone else is left behind and struggling, when they can benefit much more from positive-sum trade. They could also live with a much clearer conscience than in the former case.
We conclude by inviting the reader to think about the same question that motivated us to write this article:
Will the singularity be kicked off by the best qualities of humanity, or the worst?
We want to thank Jan Kirchner (Anthropic)*, Lovkush Argawal*, Jordan Taylor* and others who prefer to stay anonymous for their insightful feedback on an early draft of this article. It helped us with identifying and plugging gaps in our knowledge and adding some important considerations. *Feedback was given from personal capacity and not in representation of any company or organisation.
 | A guest post by
|
This article was written as a mini project at the end of the ML4Good (Germany) AI Safety bootcamp in September 2024. It is a distillation of existing literature on proposals on collaborative TAI development and benefit sharing. The bulk of research and writing was done across two days. Minor corrections were made afterwards. Both authors contributed equally.