import numpy as onp
def angle(v, w):
return onp.arccos(v.dot(w) / (onp.linalg.norm(v) * onp.linalg.norm(w)))
angle(onp.array([0, 1, 2]), onp.array([2, 3, 4]))Why a geometric view
Motivation
A vector is a list of numbers, but reading it as a picture is where the intuition lives.
- A vector is a point, or equally an arrow (a direction).
- The dot product measures alignment, giving angles and similarity.
- A hyperplane is the decision boundary of every linear classifier.
- A matrix is a map that skews, rotates, and scales space.
These pictures are the foundation for the eigendecomposition and the SVD in the sections that follow.
Two readings of a vector
Vectors
The same array names a point (its coordinates) or a direction (an arrow that may start anywhere). Deep learning mostly uses the direction view.
Reading a vector as a direction makes addition visual: follow one arrow, then the next, tip to tail.
Dot products and the angle
Vectors
Why is the dot product tied to the angle? The two vectors span a plane, so measure \|\mathbf{v}-\mathbf{w}\|^2 two ways and equate.
Algebra gives \|\mathbf{v}\|^2 - 2\,\mathbf{v}\cdot\mathbf{w} + \|\mathbf{w}\|^2; the law of cosines gives \|\mathbf{v}\|^2 + \|\mathbf{w}\|^2 - 2\,\|\mathbf{v}\|\,\|\mathbf{w}\|\cos\theta.
Cancel the common terms: \;\mathbf{v}\cdot\mathbf{w} = \|\mathbf{v}\|\,\|\mathbf{w}\|\cos\theta, so \theta=\arccos\!\big(\mathbf{v}\cdot\mathbf{w}/\|\mathbf{v}\|\,\|\mathbf{w}\|\big).
0.4189900840328574
Projection and orthogonality
Vectors
How much of \mathbf{v} points along \mathbf{w}? Drop \mathbf{v} onto the line through \mathbf{w}:
\operatorname{proj}_{\mathbf{w}}\mathbf{v} = \frac{\mathbf{v}\cdot\mathbf{w}}{\mathbf{w}\cdot\mathbf{w}}\,\mathbf{w}.
The residual \mathbf{r} leaves at a right angle, so Cauchy–Schwarz is just “a leg is no longer than the hypotenuse.” Two vectors are orthogonal when \mathbf{v}\cdot\mathbf{w} = 0.
This is least squares in one dimension (the best fit leaves an orthogonal residual); the SVD scales the same idea to any matrix.
Span, basis, dimension
Vectors
- Span: everything reachable by scaling and adding a set of vectors.
- Basis: an independent spanning set, so every vector gets unique coordinates.
- Dimension: the size of any basis.
Two subspaces ride on every matrix: the column space is what it can produce, the null space is what it sends to zero.
Random vectors are nearly orthogonal
High dimensions
Drop two unrelated unit vectors into \mathbb{R}^d: what cosine do we expect?
For a random unit vector, \;\mathbb{E}[\cos\theta] = 0 and \operatorname{Var}(\cos\theta) = \tfrac{1}{d}.
Why: rotate so \mathbf{u}=\mathbf{e}_1, so \cos\theta=v_1. Symmetry \mathbf{v}\mapsto-\mathbf{v} kills the mean; \sum_i v_i^2=1 with all coordinates alike gives \mathbb{E}[v_1^2]=\tfrac{1}{d}.
So the cosine concentrates at 0 with width 1/\sqrt{d}: a cosine well above 0 is unlikely to be an accident, which is why cosine similarity works.
A hyperplane as a decision boundary
Hyperplanes
The set \{\mathbf{x} : \mathbf{w}\cdot\mathbf{x} = b\} is a plane with normal \mathbf{w}, sitting at distance b/\|\mathbf{w}\| from the origin. Sliding b translates it without rotating.
For any point, the signed distance (\mathbf{w}\cdot\mathbf{x} - b)/\|\mathbf{w}\| is positive on one side and negative on the other. That number is exactly a linear classifier’s margin.
A classifier with nothing trained
Hyperplanes
Take Fashion-MNIST t-shirts and trousers. Average each class: the means are blurry but recognizable. The line between them, \mathbf{w} = \overline{\mathbf{x}}_1 - \overline{\mathbf{x}}_0, is our normal.
The whole picture in one projection
Hyperplanes
Reduce every image to one number, \mathbf{w}\cdot\mathbf{x}, its position along the normal. The two classes form two humps; the dashed threshold cuts between them.
A learned classifier just tilts this boundary to trim the overlap; a deep net learns features that pull the humps apart.
A matrix moves the whole grid
Linear maps
A matrix is fixed by where it sends the basis vectors, namely its columns. Every other vector follows as a weighted sum, so the entire grid is carried along: lines stay lines, the origin stays put, cells stay evenly spaced.
Matrices cannot bend space, only skew, rotate, and scale it. Multiplying two matrices composes their maps.
What a matrix destroys
Linear maps
\mathbf{B} = \bigl[\begin{smallmatrix}2&-1\\4&-2\end{smallmatrix}\bigr] has dependent columns, \mathbf{b}_1 + 2\,\mathbf{b}_2 = \mathbf{0}. The whole plane lands on one line (the column space), and the direction (1,2)^\top is crushed to the origin (the null space). Once inputs collide, no inverse can tell them apart.
Rank–nullity. \operatorname{rank}\mathbf{A} + \dim\ker\mathbf{A} = n: of the n directions coming in, \dim\ker\mathbf{A} are destroyed and \operatorname{rank}\mathbf{A} survive. For \mathbf{B}: 1 + 1 = 2.
The determinant is signed area
Linear maps
The unit square maps to a parallelogram; its signed area is \det\mathbf{A}. The sign records orientation, and a zero determinant means space was crushed to a lower dimension.
For our grid matrix, \det = 1\cdot3 - 2\cdot(-1) = 5:
import numpy as onp
onp.linalg.det(onp.array([[1, 2], [-1, 3]]))4.999999999999999Composed maps multiply their area scalings, \det(\mathbf{A}\mathbf{B})=\det\mathbf{A}\,\det\mathbf{B}. Two payoffs: \det\mathbf{Q}=\pm1 for orthogonal \mathbf{Q}, and (next section) \det\mathbf{A}=\prod_i\lambda_i, the per-axis stretches multiplied.
