Entropy and Structure

Origin. Shannon entropy (1948); Kolmogorov-Sinai entropy for dynamical systems; Soviet applications to physics, biology, and linguistics.

Mechanism. Entropy measures uncertainty or disorder — the average information content per symbol. Low entropy means predictable (structure exists); high entropy means unpredictable (no structure, or structure not yet found). But entropy is relative to a model; changing the model changes the entropy. Finding structure means finding a model that reduces entropy.

Procedure. For a system or dataset, estimate entropy under different models. Raw entropy (uniform model) is maximal. Model-based entropy (given the model's predictions) is lower if the model captures structure. Conditional entropy (entropy remaining after observing some variables) reveals dependencies. Mutual information (reduction in entropy of one variable from knowing another) quantifies relationships. Use these to detect structure, evaluate models, and find dependencies.

Applies to. Data analysis, model selection, feature selection, anomaly detection, understanding system behavior.

Limitations. Entropy estimation from finite samples is biased (typically underestimated). High-dimensional data requires massive samples for reliable estimation. Entropy measures average case; it misses rare-but-important structure. Also: entropy is syntactic, not semantic — a perfectly structured but unknown message has high entropy until the structure is discovered.