Dual Control

Origin. Alexander Feldbaum, Soviet control theory, 1960s. The theory of dual control for systems with unknown parameters. Foundational to adaptive control and later to reinforcement learning.

Mechanism. When a system's parameters are unknown, the controller faces a fundamental dilemma: exploit current knowledge to optimize immediate performance, or explore to improve knowledge for future optimization. These objectives conflict — exploration sacrifices immediate performance, exploitation sacrifices learning. Dual control recognizes both objectives simultaneously and seeks actions that balance them. The optimal control is neither pure exploitation nor pure exploration but a blend that depends on the time horizon, current uncertainty, and the cost structure.

Procedure. Design control that balances learning and performance: (1) Model uncertainty explicitly — represent the system with unknown parameters as a distribution over parameter values, not a point estimate. The distribution encodes what you know and don't know. (2) Compute the value of information — how much would reducing parameter uncertainty improve future control performance? This requires modeling the future trajectory of decisions. (3) Compute the cost of exploration — what performance is lost by deviating from the action that is optimal given current knowledge? (4) Balance the trade-off — choose actions that account for both immediate reward and information gain. Actions that reveal information about uncertain parameters may be preferred even if they sacrifice immediate performance. (5) Update beliefs — after observing outcomes, update the parameter distribution. Uncertainty should decrease as evidence accumulates. (6) Shift with horizon — as the decision horizon shortens or uncertainty resolves, shift toward exploitation. Late in the process, learning has less time to pay off.

Applies to. Adaptive control. Reinforcement learning. A/B testing and experimentation. Clinical trials. Any system where learning and performance must be traded off over time.

Limitations. The full dual control problem is computationally intractable except for special cases. Practical implementations use approximations (certainty equivalence, forced exploration, Thompson sampling) that may not optimally balance the trade-off. The value of information depends on the horizon; if the system will change before learned information can be exploited, exploration is waste. Non-stationary systems may require perpetual exploration.