Maximum Entropy

Origin. Jaynes (1957). Least biased distribution given constraints. Information-theoretic foundation. Exponential families emerge naturally. Foundation for statistical mechanics and machine learning.

Models. Maximize entropy subject to moment constraints. Entropy: H(P) = -∑P(x)log P(x). Constraints: E[fᵢ] = cᵢ. Solution: exponential family.

Formalism.

Entropy: H(P) = -∑ₓ P(x) log P(x) (discrete) H(P) = -∫ p(x) log p(x) dx (continuous)

Maximum entropy problem: maximize H(P) subject to: E_P[fᵢ(X)] = cᵢ for i = 1, ..., k ∑ₓ P(x) = 1

Lagrangian: L = H(P) - ∑ᵢ λᵢ(E[fᵢ] - cᵢ) - μ(∑P - 1)

Solution: P*(x) = (1/Z) exp(∑ᵢ λᵢfᵢ(x)) Z = ∑ₓ exp(∑ᵢ λᵢfᵢ(x)) (partition function) Exponential family.

Dual problem: minimize log Z(λ) - ∑ᵢ λᵢcᵢ Convex optimization.

Maximum entropy logic: Background: first-order constraints. Find P maximizing entropy satisfying constraints. Markov logic: weighted FOL rules.

Properties: Unique solution (convex). Sufficient statistics: fᵢ. Connection to maximum likelihood.

Symbols.

Symbol	Unicode	Name	Meaning
H	—	Entropy	Information measure
E	—	Expectation	Expected value
λ	U+03BB	Greek small lambda	Lagrange multiplier
λᵢ	—	Indexed multiplier	The multiplier attached to constraint i
μ	U+03BC	Greek small mu	Multiplier enforcing normalization
Z	—	Partition	Normalization
exp	—	Exponential	e^x
P	—	Distribution	The variable of the optimization
P*	—	Optimum	The maximizing distribution
fᵢ	—	Feature	The constrained statistic
cᵢ	—	Target moment	The value fᵢ is required to average to
L	—	Lagrangian	The objective with constraints adjoined
∑	U+2211	N-ary summation	Sum over outcomes and over constraints
∫	U+222B	Integral	The continuous form of the entropy
log	—	Logarithm	Base fixes the unit of entropy
-	U+002D	Hyphen-minus	Sign of the log-sum, and the dual objective
=	U+003D	Equals sign	The moment constraint
X	—	Random variable	The argument of the features

Metatheory. The problem is a convex program with a unique answer: entropy is strictly concave on the simplex and the moment constraints are linear, so where the feasible set is nonempty the maximizer is unique, and it has exponential-family form P*(x) ∝ exp(∑ᵢ λᵢfᵢ(x)) with the λᵢ the multipliers of the constraints. Convex duality turns the constrained maximization into the unconstrained minimization of log Z(λ) − ∑ᵢ λᵢcᵢ, and that dual is precisely the negative log-likelihood of the exponential family evaluated on data whose empirical moments are the cᵢ — so maximum entropy under moment constraints and maximum likelihood in the induced exponential family are one optimization, a duality made explicit by Della Pietra, Della Pietra and Lafferty (1997). Csiszár (1975) gave the geometric form: the solution is the I-projection of the reference measure onto the linear family, characterized by a Pythagorean identity for relative entropy, which yields the convergence of iterative proportional fitting; Darroch and Ratcliff (1972) proved generalized iterative scaling converges monotonically for nonnegative features with constant sum. Shore and Johnson (1980) derived the principle from consistency axioms — uniqueness, invariance, system independence, subset independence — rather than postulating it.

Applies to. Statistical mechanics. Natural language processing. Image reconstruction. Prior specification. Markov logic networks.

Limitations. The dual is convex but not evaluable: log Z is a sum over the whole outcome space, and for a structured model such as a Markov logic network computing Z is #P-hard, as is the gradient, which is the model expectation of each feature. Attainment can fail — if the constraints are satisfiable only on the boundary of the simplex the maximizer lies in the closure of the exponential family, no finite λ realizes it, and the scaling algorithms drive weights to infinity. In the continuous case the principle is not coordinate-free: differential entropy shifts by the log of the Jacobian under a change of variable, so "the" maximum entropy density depends on the parameterization, and the repair — maximizing relative entropy against a stated reference measure — relocates the arbitrariness rather than removing it. Updating by maximizing entropy under a new constraint conflicts with conditionalization: van Fraassen's (1981) Judy Benjamin case imposes a conditional probability and the entropy maximizer moves the marginal of the conditioning event, which Bayesian updating leaves fixed, so the two rules are incompatible rather than merely different.