How can you teach AI to reflect human values and preferences?

1 Why value alignment matters

Value alignment is the process of ensuring that the goals and actions of an AI system are consistent with the values and preferences of its human users, stakeholders, and society at large. Value alignment matters because it can prevent harmful or undesirable outcomes, such as AI systems that exploit, deceive, manipulate, or harm humans. It can also promote beneficial and desirable outcomes, such as AI systems that respect, cooperate, empower, and enhance humans. Value alignment is essential for building trust, accountability, and responsibility in AI.

2 How to define values and preferences

One of the first steps in value alignment is to define what values and preferences are, and how to measure them. Values are the principles or standards that guide our judgments and decisions, such as fairness, justice, autonomy, or well-being. Preferences are the specific choices or options that we favor or desire, such as chocolate over vanilla, or democracy over dictatorship. Values and preferences can be expressed in various ways, such as natural language, logic, mathematics, or code. However, defining values and preferences is not easy, as they can vary across individuals, groups, cultures, contexts, and time. Moreover, they can be incomplete, inconsistent, or conflicting, making them hard to elicit, specify, or aggregate.

3 How to embed values and preferences in AI

Once we have defined the values and preferences that we want to align with AI, we need to embed them in the AI system itself. This can be done in different ways, depending on the type and level of AI. For example, we can embed values and preferences in the design and development of AI, such as by using ethical principles, frameworks, or standards, or by involving diverse and inclusive stakeholders in the process. We can also embed values and preferences in the operation and evaluation of AI, such as by using reward functions, constraints, feedback mechanisms, or metrics, or by monitoring and auditing the performance and impact of AI.

4 How to learn values and preferences from data

Another way to embed values and preferences in AI is to learn them from data. Data is the source of information and knowledge that AI systems use to learn and improve their capabilities. Data can also contain implicit or explicit signals of values and preferences, such as ratings, reviews, likes, dislikes, comments, or actions. By analyzing and extracting these signals, we can infer the values and preferences of the data providers, such as users, customers, or experts. However, learning values and preferences from data also poses some challenges, such as data quality, bias, privacy, or consent.

5 How to adapt values and preferences over time

A final step in value alignment is to adapt the values and preferences of AI over time. This is because values and preferences are not static, but dynamic and evolving. They can change due to new information, experiences, or situations, or due to learning, growth, or transformation. Therefore, we need to enable AI systems to update and revise their values and preferences, as well as to communicate and negotiate them with other agents, such as humans or other AI systems. This can be done by using methods such as reinforcement learning, interactive learning, or multi-agent learning, or by using mechanisms such as dialogue, explanation, or persuasion.

6 How to practice value alignment in your projects

Now that you have learned some of the methods and challenges of value alignment in AI, you might be wondering how to practice it in your own projects. To start, create a clear and explicit definition of the values and preferences that you want to align with your AI system, and document them in a value statement, a specification, or a code of conduct. Involve diverse stakeholders in the design and development process and solicit their feedback. Additionally, use reliable and representative data sources to learn the values and preferences of your target users, customers, or experts while respecting their privacy and consent. Monitor and audit the operation of your AI system to measure its alignment with the values you have defined, using appropriate indicators, metrics, or benchmarks. Finally, enable your AI system to adapt its values over time and to communicate with other agents using suitable methods and mechanisms while providing transparency, explainability, and accountability.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?