RESEARCH HANDBOOK
ON ARTIFICIAL INTELLIGENCE AND THE
FUTURE OF HUMAN PREFERENCES
EDWARD ELGAR
Technology continues to evolve in ways that transform many human activities. Search engines make it easier to find information, and virtual platforms allow us to connect with people. We can learn about almost any topic and, for all practical purposes, access information whenever we need it. Computation has become increasingly powerful, and our ability to program machines has advanced to the point that they can now learn under both supervised and unsupervised conditions.
These advances have brought enormous benefits. Many tasks have become easier. Repetitive work is being automated. Workplaces have become safer. Scientific discovery is moving at a much faster pace. These developments have contributed to improvements in our well-being. At the same time, we live in a market economy, where organizations must create value and generate income. As a result, companies will continue to seek opportunities to improve their processes and make products and services that attract consumers.
As technology has evolved, many processes and decisions have become easier. This is a potential weakness in our human condition. We often take the path of least resistance. We consult our phones throughout the day. Increasingly, new devices make those consultations even more seamless. Wearable technologies now collect information about us and help guide our decisions. As we interact more frequently with machines, the convenience of those interactions means that many of our decisions are gradually being outsourced to intelligent systems. As we write this, for example, we could easily turn to one of the many large language models, provide a few prompts, and receive the text we want. Instead, we are choosing to guide our own thinking through the process of deciding what this book should be about, what ideas it should convey, which topics readers may find compelling, and how the chapters should be organized.
The purpose of this book is to examine how the evolution of computing, the nature of human decision making, and market incentives are converging to produce technologies that influence our choices at nearly every moment. The news we read, the videos we watch, and the purchases we make are collected, analyzed, and incorporated into systems that can generate recommendations and content tailored to each of us. We are creating a world in which everyone increasingly lives within a unique and personalized experience
The problem we have not fully recognized is that, in this process, machines are providing more of our choices. We may still appear to be making decisions, but our role changes. We are no longer always the protagonists of our decisions. Often, we are simply choosing among options that have already been selected, ranked, or framed for us. We are often reacting rather than carefully considering options. At times, we may not even be thinking in a meaningful way. Instead, we behave as though we have made decisions when, in reality, we may simply be continuing along a path toward which we have been nudged.
We must now ask whether we are losing some measure of human autonomy and some of our ability to make decisions without the assistance of increasingly intelligent machines. As large language models and other artificial intelligence tools gather more information about our preferences, we need to ask who is truly making the decision. Even when these systems do not decide for us directly, they may shape our choices in subtle and powerful ways.
This book explores the areas in which artificial intelligence may be reducing human autonomy. It is organized into six parts. The first provides background on how humans make decisions, how we form or act on our preferences, how they change over time or adapt as a result of our conditions, emotions, or social context. The second examines how artificial intelligence is being developed from a technical perspective. The chapters explain how reinforcement learning, human feedback, and refinements to LLMs, for example, align better to our human preferences. The third explores technology and decision-making in work settings. This section is separated from other chapters around human activity to illustrate the pressures companies face to use AI to optimize decisions and increase revenues. The fourth explores how different aspects of human experience are being shaped by autonomous systems. This section examines how AI-driven personalization is reshaping multiple aspects of our lives, from virtual interactions with agents that can influence our beliefs, to the ways we choose entertainment, make purchases, and even form political preferences. It also considers how AI systems increasingly select products for us through personalized recommendations and pricing, raising concerns about whether these systems may exploit individual vulnerabilities that can harm people financially.
The sixth considers the risks of these systems and the tools, policies, and practices that can help restore greater human control.
The central social question is how we design, govern, and use these systems so they support human agency rather than quietly replacing it.
TABLE OF CONTENTS
Potential authors and chapters (not confirmed):
1.Introduction
Martha Garcia-Murillo and Ian MacInnes, University of Nebraska Omaha
PART I: PREFERENCE MAKING
2.Emotion and Influence
Rosalind Picard, Professor, MIT Media Lab; founder and director, Affective Computing Research Group
3.Preferences Are Not Fixed: The Behavioral Science of Choice Formation
Gerd Gigerenzer, Director, Harding Center for Risk Literacy, University of Potsdam; long time director at the Max Planck Institute for Human Development.
PART II: THE MECHANICS OF AI DESIGN
4.The Architecture of Digital Choice
Dietmar Jannach, Professor of Computer Science, Department of Artificial Intelligence and Cybersecurity, University of Klagenfurt, Austria; also affiliated with University of Bergen, Norway.
5.Persuasive Technology
Sandra Matz, Lulu Chow Wang Professor of Business, Columbia Business School, Columbia University
PART III: AI AND PREFERENCES AT WORK
6.Dataveillance as Business Model
Alex Rosenblat, Director of Sociotechnical Research, The Markup. Formerly Head of Marketplace Policy, Fairness and Research at Uber; formerly Data & Society.
7.Personalized AI Assistants and the New Intermediaries of Choice
Christoph Busch, Professor of Law and Director of the European Legal Studies Institute, University of Osnabrück. Research focus includes consumer law, platform governance, digital markets, and algorithmic regulation.
PART IV: PERSONAL PREFERENCES AND AI
8.AI, Marketing, and the Automation of Persuasion
Catherine Tucker, Sloan Distinguished Professor of Management and Professor of Marketing, MIT Sloan School of Management
9.The Echo Chamber of Taste
Ryan Milner,Professor and Department Chair, Department of Communication, College of Charleston.
10.The Algorithmic Culture
F. Maxwell Harper, Researcher and data scientist in recommender systems, human computer interaction, machine learning, and data science; formerly Amazon Alexa Shopping and GroupLens, University of Minnesota.
11.Factory of Desire
Kate Devlin, Professor of Artificial Intelligence and Society, Department of Digital Humanities, King's College London; Chair Director, Digital Futures Institute.
12.Conversational AI as a Preference Elicitation System
Justine Cassell, SCS Dean's Professor, School of Computer Science, Carnegie Mellon University; Senior Researcher, Inria Paris; International Chair in AI, PRAIRIE
13.The Illusion of Democratic Choice
Taina Bucher, Professor, Department of Media and Communication, University of Oslo
14.The Personalization Economy
Gedas Adomavicius, Professor of Information and Decision Sciences; Larson Endowed Chair for Excellence in Business Education, Carlson School of Management, University of Minnesota
15. Artificial Intelligence and the Hypernudging of Consumer Preferences: Rethinking Undue Influence under the UCPD
Emanuele Scattarreggia, PhD Candidate in Law, Sydney Law School
16.The Extraction Economy: Algorithmic Pricing and the Loss of Consumer Surplus
Ian MacInnes, Associate Professor, Economics, University of Nebraska Omaha
17.The Algorithmic Con
Marc Schmitt, Research Associate, Department of Computer Science, University of Oxford; Managing Director, DEIM Research Institute.
PART V: THE PITFALLS AND SOLUTIONS TO AI DRIVEN PREFERENCES
18.When AI Gets Preferences Wrong
Solon Barocas, Senior Principal Researcher, Microsoft Research; Adjunct Associate Professor, Information Science, Cornell University.
19.Designing for Agency: User Control, Explanation, and Contestability
Iyad Rahwan, Director and Scientific Member, Max Planck Institute for Human Development, Berlin; Founder and Director, Center for Humans and Machines; Honorary Professor of Electrical Engineering and Computer Science, Technical University of Berlin.
20.Machines as diagnosticians
Munmun De Choudhury, J. Z. Liang Professor, School of Interactive Computing, Georgia Institute of Technology; SocWeB Lab leader.
Editors
Martha Garcia-Murillo, University of Nebraska – Omaha (US)
Ian MacInnes, University of Nebraska – Omaha (US)
These advances have brought enormous benefits. Many tasks have become easier. Repetitive work is being automated. Workplaces have become safer. Scientific discovery is moving at a much faster pace. These developments have contributed to improvements in our well-being. At the same time, we live in a market economy, where organizations must create value and generate income. As a result, companies will continue to seek opportunities to improve their processes and make products and services that attract consumers.
As technology has evolved, many processes and decisions have become easier. This is a potential weakness in our human condition. We often take the path of least resistance. We consult our phones throughout the day. Increasingly, new devices make those consultations even more seamless. Wearable technologies now collect information about us and help guide our decisions. As we interact more frequently with machines, the convenience of those interactions means that many of our decisions are gradually being outsourced to intelligent systems. As we write this, for example, we could easily turn to one of the many large language models, provide a few prompts, and receive the text we want. Instead, we are choosing to guide our own thinking through the process of deciding what this book should be about, what ideas it should convey, which topics readers may find compelling, and how the chapters should be organized.
The purpose of this book is to examine how the evolution of computing, the nature of human decision making, and market incentives are converging to produce technologies that influence our choices at nearly every moment. The news we read, the videos we watch, and the purchases we make are collected, analyzed, and incorporated into systems that can generate recommendations and content tailored to each of us. We are creating a world in which everyone increasingly lives within a unique and personalized experience
The problem we have not fully recognized is that, in this process, machines are providing more of our choices. We may still appear to be making decisions, but our role changes. We are no longer always the protagonists of our decisions. Often, we are simply choosing among options that have already been selected, ranked, or framed for us. We are often reacting rather than carefully considering options. At times, we may not even be thinking in a meaningful way. Instead, we behave as though we have made decisions when, in reality, we may simply be continuing along a path toward which we have been nudged.
We must now ask whether we are losing some measure of human autonomy and some of our ability to make decisions without the assistance of increasingly intelligent machines. As large language models and other artificial intelligence tools gather more information about our preferences, we need to ask who is truly making the decision. Even when these systems do not decide for us directly, they may shape our choices in subtle and powerful ways.
This book explores the areas in which artificial intelligence may be reducing human autonomy. It is organized into six parts. The first provides background on how humans make decisions, how we form or act on our preferences, how they change over time or adapt as a result of our conditions, emotions, or social context. The second examines how artificial intelligence is being developed from a technical perspective. The chapters explain how reinforcement learning, human feedback, and refinements to LLMs, for example, align better to our human preferences. The third explores technology and decision-making in work settings. This section is separated from other chapters around human activity to illustrate the pressures companies face to use AI to optimize decisions and increase revenues. The fourth explores how different aspects of human experience are being shaped by autonomous systems. This section examines how AI-driven personalization is reshaping multiple aspects of our lives, from virtual interactions with agents that can influence our beliefs, to the ways we choose entertainment, make purchases, and even form political preferences. It also considers how AI systems increasingly select products for us through personalized recommendations and pricing, raising concerns about whether these systems may exploit individual vulnerabilities that can harm people financially.
The sixth considers the risks of these systems and the tools, policies, and practices that can help restore greater human control.
The central social question is how we design, govern, and use these systems so they support human agency rather than quietly replacing it.
TABLE OF CONTENTS
Potential authors and chapters (not confirmed):
1.Introduction
Martha Garcia-Murillo and Ian MacInnes, University of Nebraska Omaha
PART I: PREFERENCE MAKING
2.Emotion and Influence
Rosalind Picard, Professor, MIT Media Lab; founder and director, Affective Computing Research Group
3.Preferences Are Not Fixed: The Behavioral Science of Choice Formation
Gerd Gigerenzer, Director, Harding Center for Risk Literacy, University of Potsdam; long time director at the Max Planck Institute for Human Development.
PART II: THE MECHANICS OF AI DESIGN
4.The Architecture of Digital Choice
Dietmar Jannach, Professor of Computer Science, Department of Artificial Intelligence and Cybersecurity, University of Klagenfurt, Austria; also affiliated with University of Bergen, Norway.
5.Persuasive Technology
Sandra Matz, Lulu Chow Wang Professor of Business, Columbia Business School, Columbia University
PART III: AI AND PREFERENCES AT WORK
6.Dataveillance as Business Model
Alex Rosenblat, Director of Sociotechnical Research, The Markup. Formerly Head of Marketplace Policy, Fairness and Research at Uber; formerly Data & Society.
7.Personalized AI Assistants and the New Intermediaries of Choice
Christoph Busch, Professor of Law and Director of the European Legal Studies Institute, University of Osnabrück. Research focus includes consumer law, platform governance, digital markets, and algorithmic regulation.
PART IV: PERSONAL PREFERENCES AND AI
8.AI, Marketing, and the Automation of Persuasion
Catherine Tucker, Sloan Distinguished Professor of Management and Professor of Marketing, MIT Sloan School of Management
9.The Echo Chamber of Taste
Ryan Milner,Professor and Department Chair, Department of Communication, College of Charleston.
10.The Algorithmic Culture
F. Maxwell Harper, Researcher and data scientist in recommender systems, human computer interaction, machine learning, and data science; formerly Amazon Alexa Shopping and GroupLens, University of Minnesota.
11.Factory of Desire
Kate Devlin, Professor of Artificial Intelligence and Society, Department of Digital Humanities, King's College London; Chair Director, Digital Futures Institute.
12.Conversational AI as a Preference Elicitation System
Justine Cassell, SCS Dean's Professor, School of Computer Science, Carnegie Mellon University; Senior Researcher, Inria Paris; International Chair in AI, PRAIRIE
13.The Illusion of Democratic Choice
Taina Bucher, Professor, Department of Media and Communication, University of Oslo
14.The Personalization Economy
Gedas Adomavicius, Professor of Information and Decision Sciences; Larson Endowed Chair for Excellence in Business Education, Carlson School of Management, University of Minnesota
15. Artificial Intelligence and the Hypernudging of Consumer Preferences: Rethinking Undue Influence under the UCPD
Emanuele Scattarreggia, PhD Candidate in Law, Sydney Law School
16.The Extraction Economy: Algorithmic Pricing and the Loss of Consumer Surplus
Ian MacInnes, Associate Professor, Economics, University of Nebraska Omaha
17.The Algorithmic Con
Marc Schmitt, Research Associate, Department of Computer Science, University of Oxford; Managing Director, DEIM Research Institute.
PART V: THE PITFALLS AND SOLUTIONS TO AI DRIVEN PREFERENCES
18.When AI Gets Preferences Wrong
Solon Barocas, Senior Principal Researcher, Microsoft Research; Adjunct Associate Professor, Information Science, Cornell University.
19.Designing for Agency: User Control, Explanation, and Contestability
Iyad Rahwan, Director and Scientific Member, Max Planck Institute for Human Development, Berlin; Founder and Director, Center for Humans and Machines; Honorary Professor of Electrical Engineering and Computer Science, Technical University of Berlin.
20.Machines as diagnosticians
Munmun De Choudhury, J. Z. Liang Professor, School of Interactive Computing, Georgia Institute of Technology; SocWeB Lab leader.
Editors
Martha Garcia-Murillo, University of Nebraska – Omaha (US)
Ian MacInnes, University of Nebraska – Omaha (US)
