Dual Control

Origin. Alexander Feldbaum (1960s); the theory of dual control for systems with unknown parameters.

Mechanism. When the system's parameters are unknown, the controller faces a dilemma: exploit current knowledge to optimize, or explore to improve knowledge for future optimization. These objectives conflict — exploration sacrifices immediate performance, exploitation sacrifices learning. Dual control recognizes both objectives and balances them. The optimal control is neither pure exploitation nor pure exploration but a blend that depends on the horizon and uncertainty.

Procedure. Model the system with unknown parameters as a distribution over parameter values. At each step, compute the value of information: how much would reducing parameter uncertainty improve future control? Compare to the cost of exploration: performance lost by deviating from the myopically optimal action. Choose actions that balance immediate reward against information gain. As the horizon shortens or uncertainty resolves, shift toward exploitation.

Applies to. Adaptive control, reinforcement learning, A/B testing, any system where learning and performance must be traded off.

Limitations. The full dual control problem is computationally intractable except for special cases. Practical implementations use approximations (certainty equivalence, forced exploration, Thompson sampling) that may not optimally balance the trade-off. Also: the value of information depends on the horizon; if the system will change before learned information can be exploited, exploration is waste.