Vector Space Semantics

Origin. Zellig Harris, "Distributional structure" (Word 10, 1954), and Firth, "A synopsis of linguistic theory, 1930–1955" (1957), for the hypothesis; Deerwester, Dumais, Furnas, Landauer, and Harshman (1990) for latent semantic analysis; Church and Hanks (1990) for pointwise mutual information; Turney and Pantel's survey (JAIR 37, 2010) for the vector-space model as a research programme; Mikolov and colleagues (2013) for word2vec and Pennington, Socher, and Manning (2014) for GloVe. The compositional branch is Coecke, Sadrzadeh, and Clark (2010), with Baroni and Zamparelli (2010) for the matrix treatment of adjectives.

Models. A word is a point in a real vector space whose coordinates are derived from the contexts it occurs in, so semantic relatedness becomes a geometric quantity — usually the cosine of the angle between two vectors. The representation is learned from a corpus rather than stipulated, which is what makes the approach empirical and scalable, and what makes it a theory of similarity rather than of reference. The compositional question is then how to combine vectors when syntax combines words: the type-driven answer assigns a tensor of rank matching a word's grammatical type, so a noun is a vector, an adjective a matrix, and a transitive verb a rank-three tensor, and composition is contraction.

Formalism.

Distributional hypothesis: "You shall know a word by the company it keeps" (Firth). Similar contexts → similar meaning. Co-occurrence statistics. Empirical semantics.

Word vector: v_w ∈ ℝⁿ. Vector representing word w. Dimensions: contexts or latent features. Geometric meaning.

Similarity: sim(w₁, w₂) = cos(v_{w₁}, v_{w₂}). Cosine similarity. Angle between vectors. Semantic relatedness.

Word2Vec: Skip-gram: predict context from word. CBOW: predict word from context. Neural training. Dense vectors.

Analogies: v_king - v_man + v_woman ≈ v_queen. Vector arithmetic. Relational similarity. Geometric structure.

Compositional distributional: How to combine word vectors? Matrix-vector multiplication. Tensor composition. Syntax-guided combination.

Type-driven composition: Noun: vector. Adjective: matrix. A N = M_A · v_N. Compositional types.

DisCoCat (Coecke, Sadrzadeh, Clark 2010): Grammar is a pregroup: types reduce by n · nʳ → 1 and nˡ · n → 1. Meaning space is FdVect, finite-dimensional real vector spaces. A strong monoidal functor sends grammatical reduction to tensor contraction. Both categories are compact closed, which is why the functor exists.

Weighting: Raw counts are dominated by frequency. PMI(w,c) = log[ P(w,c) / (P(w)P(c)) ] (Church and Hanks 1990). LSA truncates the SVD of the weighted matrix; by Eckart–Young the truncation is the optimal low-rank approximation.

What the neural models compute: Levy and Goldberg (2014): skip-gram with negative sampling implicitly factorizes a shifted PMI matrix, so the neural model and the count model are the same object. The dense representation is a change of algorithm, not of theory.

Symbols.

Symbol	Unicode	Name	Meaning
ℝ	U+211D	Double-struck R	The reals; ℝⁿ is the ambient vector space
⊗	U+2297	Circled times	Tensor product, the type constructor for composition
∈	U+2208	Element of	Membership of a vector in the space
≈	U+2248	Almost equal to	Approximate identity in the analogy equation
·	U+00B7	Middle dot	Matrix–vector product, and pregroup concatenation
→	U+2192	Arrow	Pregroup reduction
v_w	—	Word vector	The representation of word w
M_A	—	Adjective matrix	A linear map on the noun space
cos	—	Cosine	The similarity measure
PMI	—	Pointwise mutual information	The standard association weighting
SVD	—	Singular value decomposition	Dimensionality reduction in LSA
n, nˡ, nʳ	—	Pregroup types	A basic type with its left and right adjoints

Metatheory. Coecke, Sadrzadeh, and Clark (2010) gave the framework its one structural theorem: a pregroup grammar and the category of finite-dimensional real vector spaces are both compact closed, so there is a strong monoidal functor from the first to the second, and grammatical reduction is carried to tensor contraction — the composition rule is derived from the shared categorical structure rather than chosen. On the lexical side the apparent divide between count-based and neural models is not one: Levy and Goldberg (2014) showed that skip-gram with negative sampling implicitly factorizes a shifted pointwise-mutual-information matrix, so word2vec and a weighted co-occurrence model compute the same object by different means. Latent semantic analysis rests on Eckart–Young, which makes the truncated singular value decomposition the optimal low-rank approximation in the Frobenius norm. What the geometry does not supply is a truth condition: the space encodes similarity, and no entailment relation is definable on it without added structure.

Applies to. Computational semantics. NLP. Word similarity. Analogy.

Limitations. Vector addition is commutative, so any additive composition rule identifies "dog bites man" with "man bites dog" — the simplest and most used method cannot represent argument structure at all. The distributional signal does not distinguish opposition from synonymy, since antonyms occur in near-identical contexts, and it gives no account of negation, quantification, or scope. The type-driven repair is not free: a transitive verb needs a rank-three tensor, quadratic or worse in the dimension, so the parameters cannot be estimated from corpora for most of the lexicon. Static vectors conflate the senses of a polysemous word into one point. The analogy results are weaker than reported — Linzen (2016) showed much of the effect survives removing the offset vector — and the spaces encode social bias measurably (Caliskan, Bryson, and Narayanan 2017). Nothing in the geometry gives reference or truth.