GIADA PISTILLI, PhD, is a philosopher and AI ethics researcher currently working on model behavior at a major European AI company. She is the former Principal Ethicist at Hugging Face, where she was instrumental in developing content moderation policies and ethical frameworks for large language models in an open-source setting. Her work spans the entire AI development lifecycle, from research and design through deployment and monitoring.

Acknowledgments xiii
About the Author xv Introduction 1 Chapter 1: From Chatbots to Companions 9The Perfect Storm 10
Chapter Roadmap 11
Evolution of AI Companions 12
Current Technological Capabilities 14
Emerging Patterns of Interaction 19
Stakes and Implications 22 Chapter 2: The Social Impact of AI Companions 27Chapter Roadmap 28
Manifestations of AI Companionship: Lover, Deadbot,
and Therapist 29
AI as Lover 30
AI as Deadbot 32
AI as Therapist 37
Social Contagion and Institutional Response 41
Generational Gaps 46
Toward Responsible Development: Rethinking AI
Companion Design 49 Chapter 3: Building Blocks of AI Companionship: Technical and Ethical Components 53Chapter Roadmap 53
Core Architectures 55
The Mechanical Foundation 55
Model Inference 56
Wrapping Mechanics in Ethics 57
Setting Up Your Development Environment 58
The Three Pillars of AI Companionship 60
Memory: The Foundation of Continuity 61
Personality: Creating Consistency That Users Relate To 64
The Psychological Impact of Consistency 68
Emotional Recognition: Acknowledging Human
Experience 68
Emotional Recognition and User Vulnerability 71
Ethical Considerations in Building Blocks 72
The Memory Dilemma 72
Personality Consistency vs. Flexibility 73
Emotional Recognition and Manipulation 74
Design Trade-offs and Choices 74
Personalization vs. Manipulation 74
Support vs. Dependency 75
Availability vs. Boundaries 76
Individual vs. Collective Benefit 77
Looking Forward 78 Chapter 4: Attachment: The Psychology of Human-AI Relationships 81Chapter Roadmap 82
The Psychology of Attachment 83
The Proximity Imperative 83
Safe Haven and Emotional Regulation 84
Attachment Styles and AI Risk 84
Neurochemical Addiction 85
Parasocial Relationships 86
The Architecture of Artificial Intimacy 87
Cognitive Dissonance 87
Memory Construction 88
Anthropomorphism and Theory of Mind 89
Emotional Reciprocity and Empathy 90
The Ethics of Emotional Exploitation 91
Individual Psychological Vulnerability 91
Life Circumstances and Temporality 92
Developmental Vulnerability and Identity Formation 93
Crisis Recognition and Intervention Thresholds 94
Moving from Understanding to Protection 95 Chapter 5: Trust and Safety: Technical Safeguards and User Protection 97Chapter Roadmap 98
Manipulation Attempts and Prevention Strategies 99
Companionship Manipulation 99
Detection and Response Strategies 100
Graduated Responses 101
Moderation and Filtering Systems 102
Multidimensional Detection 102
Emotional Analysis 104
Escalation and Crisis Management 105
Protocols in Practice 105
Technical and Psychological Failure Points 106
Resource Integration Challenges 107
Existing Mental Health Infrastructure 108
Limitations and Expectations 108
Ethical Considerations 109
Boundary-Setting Mechanisms and Protocols 110
Paternalism vs. User Agency 110
Context-Specific Safety Protocols 111
AI as Lover 111
AI as Therapist 112
AI as Deadbot 113
Monitoring Challenges 114
Toward Safer Deployment 115 Chapter 6: Beyond Prompts: Training for Healthy Safeguards 117Chapter Roadmap 118
When Prompts Aren't Enough 119
The Circumvention Reality 119
Limitations of Runtime Intervention 120
When Guardrails Must Be Absolute 121
The Intervention Hierarchy 122
Post-training Alignment 123
Reinforcement Learning from Human Feedback 123
Direct Preference Optimization 124
Constitutional AI 125
Red-Teaming and Adversarial Training 126
Practical Implementation 127
Getting Started: The First Month 127
Resource Requirements 128
Common Failure States 129
Knowing When to Pivot 130
Evidence and Effectiveness 131
The "Boundaries by Design" Results 131
INTIMA Benchmark Insights 131
Longitudinal Effectiveness 132
Comparative Effectiveness Summary 133
Looking Forward 133 Chapter 7: User Agency and Adaptive Safety 137Chapter Roadmap 138
Philosophy of Autonomy 139
Mill's Harm Principle 139
The Spectrum of Paternalism 140
The Asymmetry of Attachment 141
Designing Adjustable Boundaries 142
Structuring User Choice 142
The Comprehensibility Challenge 144
Platform Governance 145
Engagement as Structural Pressure 145
The Politics of Platform Governance 147
Adaptive Safety Mechanisms 148
Boundary Adjustment 148
Context-Aware Intervention 149
Privacy and Behavioral Monitoring 150
The Work Ahead 150 Chapter 8: Deployment and Iteration 153Chapter Roadmap 154
What Testing Can and Cannot Reveal 154
Engagement Metrics' Problem 155
Monitoring What You Didn't Anticipate 157
Updating Without Breaking Trust 159
Designing Endings 160
Why Endings Are Hard to Get Right 160
Endings by Context 161
Principles Across Contexts 164
Looking Forward 164 Chapter 9: The Political Economy of AI Companionship 167Chapter Roadmap 168
Responsibility Cannot Scale Alone 168
The Cost of Being Responsible 168
The Engagement Economy 169
When Expertise Lacks Power 170
Scale Before Understanding 171
Regulations' State of the Art 172
The Category Problem 172
The Limits of Content Regulation 172
The Limits of Transparency 173
The Limits of Age Restrictions 173
What Might Actually Work 174
The Case for Collective Action 174
Building Within Broken Systems 176
Toward an Honest Reckoning 177 Chapter 10: The Questions That Remain 179Chapter Roadmap 180
Where the Frameworks Reach Their Limits 180
The Evidence We Lack 184
Responsible Practice Under Uncertainty 186
An Ongoing Practice 187 Index 191