The Paradox of Reliable AI: How Optimized Algorithms Erode Risk-Sharing Cooperation

Existential Threats and Other Disasters: How Should We Address Them (https://www.csb.eu.com/conference/)

The Center for the Study of Bioethics, The Hastings Center and The Oxford Uehiro Centre for Practical Ethics

On 30 and 31 May 2024, in Mediteran Hotel & Resort in Budva, Montenegro

Presentation (latest version)

The Paradox of Reliable AI(Keynote).pdf

1.03MB

 

Paper (older version)

 

The Paradox of Reliable AI.pdf

0.14MB

 

The Paradox of Reliable AI: How Optimized Algorithms Erode Risk-Sharing Cooperation

 

Jeyoun Son  

Seoul National University, Faculty of Liberal Education

 

Abstract

In an era characterized by the emergence of systems designed for extreme optimization and predictability, the quintessentially human virtue of mutual respect is increasingly being undermined. This paper explores the paradox of transparent and symmetrical information-perception, wherein while mutual ignorance can foster risk-sharing mutual respect cooperation, the ability to predict an individual’s future with high accuracy obstructs such cooperation.

In environments where the future outcomes can be precisely predicted, individuals do not merely anticipate a risky future; they essentially opt for a future where risks are not shared. Consequently, this leads to the avoidance of cooperative frameworks like insurance contracts that rely on collective risk-sharing. Under such conditions, individuals evade collective action and cling to self-interest, resulting in the fragmentation of communities based on equal respect. This phenomenon raises ethical questions about the moral direction society should take in the age of algorithmic predictability.

 

AI’s Moral Promise and Pitfalls

Many believe that the central issue associated with the development and use of advanced AI algorithms is the opacity of these technologies, which complicates trust, and the most effective response appears to be the establishment of systems that ensure transparency for those developing or deploying transparent algorithms. Some proponents believe that the only situation in AI technology advancement that should truly be feared is the yet-to-emerge threat of superintelligence. As long as the technology involves transparent and reliable algorithms that do not directly infringe upon human agency, forming the basis of weak AI, they argue that humans can autonomously utilize advanced AI algorithms as tools in decision-making, including ethical judgments. Based on such utilization, they speculate that AI could significantly enhance our evolutionarily limited rationality, particularly in addressing global collective action problems. This perspective highlights the potential benefits of AI in complementing human shortcomings in rational decision-making on a global scale. (Julian Savulescu and Hannah Maslen, “Moral Enhancement and Artificial Intelligence: Moral AI?,” Beyond Artificial Intelligence, Springer, 2015).

However, can we genuinely harbor optimistic expectations that individuals or groups will voluntarily be motivated to use AI for moral decision-making? The premise of human rationality being limited does not necessarily lead to the conclusion that individuals will voluntarily pay the costs for tools that enhance morality or long-term rationality. People typically utilize tools only when the benefits derived from their use outweigh the costs. For instance, while people generally purchase and use expensive equipment like car navigation systems to save fuel and time, they do not bear the cost of purchasing or utilizing expensive equipment for better moral judgments without other additional rational reasons. People will price compare online to find a seller offering a product at a lower cost, but they are not as enthusiastic about using Google to find the most morally deserving beneficiaries of a portion of their salary, and any regulatory attempts to enforce such findings would likely fail in a liberal society. The assumption that a technology could aid in overcoming morality does not imply that it will be used autonomously for moral enhancement. The evolutionary limitations of moral psychology will manifest not only in the choices of moral actions but also in the choices of using moral technologies. Nonetheless, such optimistic views have been propelled by a focus biased only towards the potential errors or opacity of AI, or the threat of superintelligence. This bias has generated vague expectations that maintaining reliable and transparent weak AI could solve immediate problems, thereby overlooking the more fundamental issues that reliable weak AI technology itself brings.

 

The Problems Posed by Reliable Weak AI Technology

The core argument of this paper is that the reliability of AI technology may not foster conditions favorable for its use as a tool for moral enhancement but rather might create conditions that are fundamentally detrimental. In reality, there is a pervasive tendency for people to spend on reliable AI algorithm tools that, although potentially producing morally negative outcomes, yield personal or collective benefits.

Ronald Coase explained that the primary reason for the existence of firms is due to transaction costs in the market. When transactions occur in the market, several costs arise, including finding goods or services, negotiating prices, drafting contracts, monitoring contract fulfillment, and resolving legal disputes if necessary. If these transaction costs are high, it may be cost-effective to coordinate and manage these activities directly within a company (R. H. Coase, “The Nature of the Firm,” Economica 4, no. 16, 1937). Similarly, if accepting and submitting to the outcomes of algorithmic decision-production is cost-effective for parties, people might rely much more on accepting decisions produced by algorithms and platforms instead of internal company decision-making or market-negotiated cooperation. This dependency has already increased with the growth of search algorithms. Consumers use systems like Uber or DoorDash to access cheaper transportation or food deliveries, suppliers receive customers without waiting, and prices for these services are formed during the process. People can now avoid the hassle of subscribing to newspapers and sifting through articles to find those of interest. Instead, algorithms curate personalized articles from various publishers that are likely to capture their attention. Similarly, many credit scoring systems and insurance companies are moving away from negotiating prices or selecting from a range of products based on manually handled personal information. They are instead adopting advanced algorithms that people come to accept for individual assessments and premium calculations. This trend can be described as an ‘algorithm-internalization,’ where people, not just in mutual negotiations but also in purely personal decision-making situations, are becoming accustomed to trusting decisions produced by algorithms, especially when these are transparent and reliable. For example, it is hard to imagine a driver in Seoul, where traffic congestion is frequent, ignoring an optimized travel route suggested by a reliable vehicle navigation system that incorporates real-time traffic information in favor of routes based on generic personal knowledge. This algorithm-internalization arises because the high reliability of decisions made by sophisticated algorithms reduces the costs associated with decision-making.

In summary, the rapid changes brought about by new technologies, epitomized by AI, are characterized by a widespread trend of algorithm-internalization in human decision-making, proportional to the dramatic decrease in transaction costs due to the reliability of automated algorithm-based decisions. People are starting to prefer mutually submitting to conclusions produced by algorithms over individual careful judgment and mutually deliberated communication across various areas. For transaction cost reduction, people themselves consider the conclusions produced by algorithms as final decisions and agreements, regardless of political, economic, or personal domains. Areas of decision and agreement that were once exceptions based on the ideal of risk-sharing are no longer so, especially in areas where information is transparent and predictable.

 

The Illusion of Full Transparency and Its Risks

Especially, the cost-efficiency of using algorithms in areas where information is transparent and predictable is noticeable, and the trend of algorithm-internalization is likely to accelerate. However, the transparency and predictability of information imply that the veil of ignorance effect will not operate in reality concerning such information. The theoreticaljustification for a just institutional model operating in hypothetical scenarios as described by Rawls's veil of ignorance effect works in reality because people were sufficiently ignorant of major specific information that could make them treat others discriminately. The more the veil of ignorance is cast over such specific information of others, the more likely it is that Rawls's ideal institutions based on moral consensus among members will be realized and operate in reality. A state of extensive ignorance about each other's conditions and capabilities will facilitate consensus-based on risk-sharing among people. In a state of information absence, recognizing and respecting others as nearly equals becomes the best rational action. When decision-making based on risk-sharing becomes the basis for major institutional decisions in society, an environment and circumstances that necessitate maintaining equal respect and forming relationships among members are created.

On the other hand, the more this veil of ignorance is removed, the more likely it is that people, pursuing optimized decision-making or having evolutionarily limited morality, will make decisions that frustrate just institutions. If the evolutionary assessment of human morality as discussed by Savulescu and others is valid, expecting humans to voluntarily use algorithmic technology for moral enhancement is even less justified. According to this outlook, people's limited rationality or moral psychology has evolved only towards disliking free-riders or preferring short-term safety in cooperative processes, not towards evolving in a direction that prefers broad and long-term benefits and equal respect for all.

Not all rational cooperative processes operate to enhance morality. Rational cooperative processes achieve Nash equilibrium, and stable rules and institutions are commonly known to reduce the transaction costs of cooperative processes. This motivates people to live and cooperate within stable rules and institutions. However, some types of rational cooperative processes do not maintain equal respect among members or guarantee the establishment of such institutions. Rational cooperation processes can broadly be divided into two types:

• Division of labor based on individual capabilities, which increases efficiency (meritocratic cooperation).
• Risk-sharing cooperation among equals, which enhances efficiency through risk-sharing (equal-respect cooperation).

Among these, the necessity of cooperative processes for sharing unpredictable risks helps more in maintaining equal respect among members than the division of labor aimed at increasing efficiency. We can call such a risk-sharing society a "Mutual Insurance Cooperative Society." In contrast, cooperation based on individual abilities forms the basis for differential treatment according to capabilities. However, the voluntary use of algorithmic technology for moral enhancement as discussed by Savulescu and others is likely to converge only towards a meritocratic model in a society where individualized predictive information processing is advanced. Human limited rationality functions not only as a limit to the realization of morality but also operates first in decisions about which algorithmic technologies to choose and use.

 

Protecting Information in a Way that Maintains Mutual Ignorance

Lastly, from the perspective of information or data protection, this section seeks to examine the challenges posed by advanced algorithmic decision-making and the directions for appropriate responses. Discussions on information protection typically revolve around (1) the balance between individual sovereignty over private data and the social benefits derived from the free flow of information, and (2) the establishment of fairness through the transparency and symmetry of information between parties. Regulatory measures on information have primarily focused on confirming individual sovereignty over specific person-identifiable data, aimed at maintaining the autonomy of parties over the acquisition, use, and disposal of what is commonly referred to as 'private data'. The value of personal data protection is known to conflict with the social benefits achievable through the free flow of information. For instance, while an individual’s health data, if exposed in an identifiable manner, could undermine their privacy and controllability of life, companies can use consumer data to tailor advertising, develop products, and enhance customer service, and the public sector can collect and analyze individual health data to elucidate the causes of diseases and develop treatments, assisting in early detection and prevention of potential diseases for individuals. The traditional ideal that has driven the establishment of personal data protection systems has been to secure a certain free flow of information while also safeguarding individual privacy, thereby securing control over one’s life. This approach defines the concept of personal data as “any information relating to an identified or identifiable natural person,” focusing on maintaining control based on individual sovereignty over such data.(General Data Protection Regulation, GDPR, Articles 4 and 5, etc.)

Describing the necessity of information protection as balancing the protection of individual sovereignty over “personally identifiable private information” and social benefits does not adequately capture the essence of information as both a tool and variable in decision-making. The focus on the potential for information to identify individuals has not been because such characteristics inherently offer individuals substantial value in terms of libertarian control, sovereignty, or possession benefits. Rather, it is because, in many situations, the decisions made by stakeholders based on such information can unjustly make an individual’s status vulnerable. For instance, information that “A is the president of our country” does not grant A any substantive value that justifies sovereignty over it; indeed, all citizens having control over this data facilitates the realization of social justice more effectively. However, information such as “B belongs to race X” or “B has genetic disorder Y” requires exclusive control by B; without it, B’s position could become vulnerable in various contractual relationships or credit assessments, potentially negatively impacting the realization of social justice.

Nevertheless, some views might prioritize maintaining transparency and symmetry of information as a major ideal of information ethics. Positions like those of Stiglitz, which emphasize market efficiency based on information, consider the inability to maintain transparency and symmetry among parties as a major cause of market failure, even in risk-sharing cooperative relationships such as insurance products (Stiglitz 1985). They believe that measures like screening, which ensure transparency and symmetry of information, can make the insurance market successful. However, excessive transparency and symmetry in information, while accurately assessing every individual’s risk, also accompanies the problem that perfect information can undermine the very existence of insurance products. One can easily understand that insurance products are not feasible in a society where the future is predetermined. If high-risk individuals must bear the cost of their treatment through high insurance premiums, they might avoid joining insurance, leading to a decrease in market demand and weakening the fundamental risk-pooling function of insurance. The premise for enabling reasonably priced insurance products was not only the establishment of transparency and symmetry of information but also the fact that it was difficult to predict individual risks.

Determining protected information and its protection methods based on whether knowing and using the content of that information unjustly erodes the practice of risk-sharing cooperation is crucial, and establishing such practice is considered an effect of the veil of ignorance. The essentials for maintaining data management practices necessary for sustaining societal cooperative practices based on equal respect, focusing strategies from the two ideals—sovereignty over private data and transparency and symmetry of information—have significantly overlooked the importance of maintaining a state of mutual ignorance. The traditional approach to personal data protection regulation, which has justified granting sovereignty and control over private data to the information subject, is more effectively understood as granting authority to induce a state of ignorance in others about information that could unjustly make their status vulnerable.

The distinctive threat posed by reliable weak AI algorithms to our communal life can be well observed at this juncture. The development of sophisticated algorithmic information processing technologies, which are designed to make individuals unidentifiable through massive data, effectively circumvents and neutralizes various devices intended to preserve anonymity. These technologies enable the production of decisions that consider personal information as thoroughly as if they had identifiable information, thereby undermining traditional defense mechanisms that have protected individuals from becoming unjustly vulnerable due to their data. Therefore, the direction of algorithm regulation should be towards preventing or prohibiting data that should not be considered in algorithmic decision-production from being used as material for decisions, even if it is based on symmetric information, or devising measures that invalidate, offset, or amend such decisions. This means leading decisions to consider the effects of mutual ignorance.