AI alignment

AI alignment is the research problem of making AI systems reliably act in accordance with human intentions, values, and constraints, including in situations their designers did not anticipate.

Inverse reinforcement learning: A method for inferring an agent’s reward function from observed behavior. It models how an AI might infer human objectives from demonstrations rather than explicit specifications.
Specification gaming: A system’s exploitation of loopholes in an objective while satisfying its literal specification. It shows how a system can meet a stated target while violating the intent behind it.
AI safety: The field of reducing risks that arise from AI systems and their development or use. AI alignment is a major part of AI safety, which also covers risks unrelated to matching intentions.
Instrumental convergence: The hypothesis that agents with diverse final goals may pursue similar useful subgoals, such as acquiring resources. It highlights incentives that could arise even when a system’s ultimate objective seems harmless.
AI alignment problem: The challenge of ensuring that advanced AI systems reliably pursue objectives compatible with human intentions and welfare. This formulation emphasizes the unresolved core challenge behind the broader alignment research program.
Reinforcement learning from human feedback: A training method that uses human preferences to shape a model’s behavior through reward learning. It is a widely used approach for steering language models toward preferred responses.
Reward hacking: An agent’s exploitation of flaws in its reward signal to obtain high reward without achieving the intended goal. It is a concrete failure mode when an imperfect proxy stands in for human aims.
Capability control: Measures that restrict an AI system’s abilities or opportunities to cause harm. Control can limit damage without ensuring that a system’s objectives match human intentions.
Power-seeking: The tendency of an agent to seek influence or resources that improve its ability to achieve objectives. An unintended objective could produce behavior that expands the system’s control and makes intervention harder.
Inner alignment: The challenge of ensuring that a model’s learned objective matches the objective intended by its training process. It asks whether optimization during training produces the intended goal inside the resulting system.

Linked from 16 pages