Maximum Entropy

Origin. Jaynes (1957). Least biased distribution given constraints. Information-theoretic foundation. Exponential families emerge naturally. Foundation for statistical mechanics and machine learning.

Models. Maximize entropy subject to moment constraints. Entropy: H(P) = -∑P(x)log P(x). Constraints: E[fᵢ] = cᵢ. Solution: exponential family.

Formalism.

Entropy: H(P) = -∑ₓ P(x) log P(x) (discrete) H(P) = -∫ p(x) log p(x) dx (continuous)

Maximum entropy problem: maximize H(P) subject to: E_P[fᵢ(X)] = cᵢ for i = 1, ..., k ∑ₓ P(x) = 1

Lagrangian: L = H(P) - ∑ᵢ λᵢ(E[fᵢ] - cᵢ) - μ(∑P - 1)

Solution: P*(x) = (1/Z) exp(∑ᵢ λᵢfᵢ(x)) Z = ∑ₓ exp(∑ᵢ λᵢfᵢ(x)) (partition function) Exponential family.

Dual problem: minimize log Z(λ) - ∑ᵢ λᵢcᵢ Convex optimization.

Maximum entropy logic: Background: first-order constraints. Find P maximizing entropy satisfying constraints. Markov logic: weighted FOL rules.

Properties: Unique solution (convex). Sufficient statistics: fᵢ. Connection to maximum likelihood.

Symbols.

Symbol	Unicode	Name	Meaning
H	—	Entropy	Information measure
E	—	Expectation	Expected value
λᵢ	U+03BB	Lambda	Lagrange multiplier
Z	—	Partition	Normalization
exp	—	Exponential	e^x

Metatheory. Uniqueness. Duality. Iterative scaling algorithms. GIS, IIS convergence. Connection to exponential families.

Applies to. Statistical mechanics. Natural language processing. Image reconstruction. Prior specification. Markov logic networks.

Limitations. Feature engineering. Partition function computation. Continuous constraints harder. Moment constraints may be insufficient.