Orthogonal Projections and the Gram-Schmidt Process

8 min read

Before Projection uses the dot product, a basis of the subspace, and independence. Builds on: Inner Products, Length, and Orthogonality · Null Space, Column Space, and Basis · Linear Independence

Next The closest point is the idea behind least squares, the method that finds a line of best fit.

Reference: Lay, Linear Algebra and Its Applications, Section 6.3: Orthogonal Projections; Section 6.4: The Gram-Schmidt Process

Video: Dr Lynch’s MATH 231 lectures (YouTube playlists)

Before You Start: Prerequisite Check

Can you do these? (Click to reveal self-test)

Test yourself on these prerequisite skills:

Dot product: Compute $\begin{pmatrix}3\\1\\-2\end{pmatrix} \cdot \begin{pmatrix}1\\2\\3\end{pmatrix}$.

Check

$3(1)+1(2)+(-2)(3) = 3+2-6 = -1$.
Orthogonality: Is $\begin{pmatrix}1\\2\end{pmatrix}$ orthogonal to $\begin{pmatrix}2\\-1\end{pmatrix}$?

Check

$1(2)+2(-1) = 0$. Yes, orthogonal.
Basis: What is the definition of a basis for a subspace?

Check

A linearly independent set that spans the subspace. Every vector in the subspace is a unique linear combination of the basis vectors.

If you struggled: Review Inner Products, Length, and Orthogonality.

Try This First

Find the closest point on the line $\text{Span}\left\{\begin{pmatrix}1\\2\end{pmatrix}\right\}$ to the point $\mathbf{y} = \begin{pmatrix}3\\1\end{pmatrix}$.

Before computing: draw a sketch. The closest point is the foot of the perpendicular from $\mathbf{y}$ to the line.

Show Solution

The closest point on $\text{Span}\left\{\mathbf{u}\right\}$ to $\mathbf{y}$ is the projection of $\mathbf{y}$ onto $\mathbf{u}$:

\[ \hat{\mathbf{y}} = \frac{\mathbf{y} \cdot \mathbf{u}}{\mathbf{u} \cdot \mathbf{u}}\mathbf{u} = \frac{3(1)+1(2)}{1+4}\begin{pmatrix}1\\2\end{pmatrix} = \frac{5}{5}\begin{pmatrix}1\\2\end{pmatrix} = \begin{pmatrix}1\\2\end{pmatrix} \]

The closest point is $\begin{pmatrix}1\\2\end{pmatrix}$.

Verify: the error vector is $\mathbf{y} - \hat{\mathbf{y}} = \begin{pmatrix}2\\-1\end{pmatrix}$. Check orthogonality with $\mathbf{u} = \begin{pmatrix}1\\2\end{pmatrix}$: $(2)(1)+(-1)(2) = 0$. The error is perpendicular to the line.

The Projection Idea

The projection of $\mathbf{y}$ onto a subspace $W$ gives the point in $W$ that is closest to $\mathbf{y}$. The “leftover,” $\mathbf{y} - \hat{\mathbf{y}}$, is always orthogonal to $W$.

This decomposition $\mathbf{y} = \hat{\mathbf{y}} + (\mathbf{y} - \hat{\mathbf{y}})$ separates $\mathbf{y}$ into a part inside $W$ and a part perpendicular to $W$, and it generalizes the familiar idea of dropping a perpendicular from a point to a line.

Prerequisite Map

Quick Reference

Property	Value
Course	MATH 231
Difficulty	Advanced
Time	~30 minutes
Source	Lay, Linear Algebra and Its Applications, Sections 6.3 and 6.4

Projection Formulas

Situation	Formula
Project $\mathbf{y}$ onto a vector $\mathbf{u}$	$\hat{\mathbf{y}} = \dfrac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u}$
Project $\mathbf{y}$ onto a subspace with orthogonal basis $\{\mathbf{u}_1,\ldots,\mathbf{u}_p\}$	$\hat{\mathbf{y}} = \dfrac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1}\mathbf{u}_1 + \cdots + \dfrac{\mathbf{y}\cdot\mathbf{u}_p}{\mathbf{u}_p\cdot\mathbf{u}_p}\mathbf{u}_p$
Error vector	$\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ (always orthogonal to $W$)

Key Concepts

Projection onto a Vector

The orthogonal projection of $\mathbf{y} \in \mathbb{R}^n$ onto a nonzero vector $\mathbf{u}$ is: \[ \text{proj}_\mathbf{u}\mathbf{y} = \frac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u} \]

The scalar $\frac{\mathbf{y}\cdot\mathbf{u}}{\|\mathbf{u}\|^2}$ is the coefficient; it can be positive (projection in the direction of $\mathbf{u}$), negative (opposite direction), or zero (projection is $\mathbf{0}$).

If $\mathbf{u}$ is a unit vector, the formula simplifies: $\text{proj}_\mathbf{u}\mathbf{y} = (\mathbf{y}\cdot\mathbf{u})\mathbf{u}$.

Source: Lay, Linear Algebra and Its Applications, Section 6.3: Orthogonal Projections.

Projection onto a Subspace

If $W$ has an orthogonal basis $\{\mathbf{u}_1, \ldots, \mathbf{u}_p\}$, the orthogonal projection of $\mathbf{y}$ onto $W$ is:

\[ \hat{\mathbf{y}} = \frac{\mathbf{y}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1}\mathbf{u}_1 + \frac{\mathbf{y}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2}\mathbf{u}_2 + \cdots + \frac{\mathbf{y}\cdot\mathbf{u}_p}{\mathbf{u}_p\cdot\mathbf{u}_p}\mathbf{u}_p \]

The error $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to every vector in $W$.

Orthogonal Decomposition Theorem: every $\mathbf{y}$ can be written uniquely as $\mathbf{y} = \hat{\mathbf{y}} + \mathbf{z}$ where $\hat{\mathbf{y}} \in W$ and $\mathbf{z} \in W^\perp$.

Best Approximation Theorem: $\hat{\mathbf{y}}$ is the closest point in $W$ to $\mathbf{y}$: for any other $\mathbf{v} \in W$, $\|\mathbf{y} - \hat{\mathbf{y}}\| \leq \|\mathbf{y} - \mathbf{v}\|$.

The Gram-Schmidt Process

Input: any basis $\{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_p\}$ for a subspace $W$.

Output: an orthogonal basis $\{\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_p\}$ for the same subspace $W$.

Algorithm:

\[ \mathbf{v}_1 = \mathbf{x}_1 \] \[ \mathbf{v}_2 = \mathbf{x}_2 - \frac{\mathbf{x}_2\cdot\mathbf{v}_1}{\mathbf{v}_1\cdot\mathbf{v}_1}\mathbf{v}_1 \] \[ \mathbf{v}_3 = \mathbf{x}_3 - \frac{\mathbf{x}_3\cdot\mathbf{v}_1}{\mathbf{v}_1\cdot\mathbf{v}_1}\mathbf{v}_1 - \frac{\mathbf{x}_3\cdot\mathbf{v}_2}{\mathbf{v}_2\cdot\mathbf{v}_2}\mathbf{v}_2 \]

and so on: at step $k$, subtract from $\mathbf{x}_k$ the projection of $\mathbf{x}_k$ onto each previously computed $\mathbf{v}_j$.

Gram-Schmidt preserves the span: at each step, $\text{Span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\} = \text{Span}\{\mathbf{x}_1,\ldots,\mathbf{x}_k\}$.

To get an orthonormal basis, normalize each $\mathbf{v}_j$: $\mathbf{u}_j = \mathbf{v}_j / \|\mathbf{v}_j\|$.

Source: Lay, Linear Algebra and Its Applications, Section 6.4: The Gram-Schmidt Process.

Worked Example: Gram-Schmidt in $\mathbb{R}^3$

Apply Gram-Schmidt to the basis $\mathbf{x}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}$, $\mathbf{x}_2 = \begin{pmatrix}1\\0\\1\end{pmatrix}$.

Step 1: $\mathbf{v}_1 = \mathbf{x}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}$

Step 2: Subtract the projection of $\mathbf{x}_2$ onto $\mathbf{v}_1$:

\[ \frac{\mathbf{x}_2\cdot\mathbf{v}_1}{\mathbf{v}_1\cdot\mathbf{v}_1} = \frac{1+0+0}{1+1+0} = \frac{1}{2} \]

\[ \mathbf{v}_2 = \mathbf{x}_2 - \frac{1}{2}\mathbf{v}_1 = \begin{pmatrix}1\\0\\1\end{pmatrix} - \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} = \begin{pmatrix}1/2\\-1/2\\1\end{pmatrix} \]

Verify orthogonality: $\mathbf{v}_1 \cdot \mathbf{v}_2 = \frac{1}{2} - \frac{1}{2} + 0 = 0$ (correct).

Normalize to get an orthonormal basis:

\[ \|\mathbf{v}_1\| = \sqrt{2}, \quad \mathbf{u}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix} \]

\[ \|\mathbf{v}_2\| = \sqrt{\frac{1}{4}+\frac{1}{4}+1} = \sqrt{\frac{3}{2}} = \frac{\sqrt{6}}{2}, \quad \mathbf{u}_2 = \frac{1}{\sqrt{6}/2}\begin{pmatrix}1/2\\-1/2\\1\end{pmatrix} = \frac{1}{\sqrt{6}}\begin{pmatrix}1\\-1\\2\end{pmatrix} \]

Verify: $\mathbf{u}_1 \cdot \mathbf{u}_2 = \frac{1}{\sqrt{2}} \cdot \frac{1}{\sqrt{6}}(1-1+0) = 0$ (orthogonal), $\|\mathbf{u}_1\| = 1$ (unit), $\|\mathbf{u}_2\|^2 = \frac{1}{6}(1+1+4) = 1$ (unit).

Common Pitfalls

Mistake	Why It Is Wrong	Correct Approach
Thinking the projection $\hat{\mathbf{y}}$ lies on the vector $\mathbf{u}$ itself	The projection lies in $\text{Span}\{\mathbf{u}\}$, the line through the origin in the direction of $\mathbf{u}$; it is a scalar multiple of $\mathbf{u}$	Projection = $\frac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u}$: a specific point on the line through $\mathbf{u}$
Believing Gram-Schmidt changes the subspace	Gram-Schmidt builds a different basis for the SAME subspace: the span is invariant	Verify by checking $\text{Span}\{\mathbf{v}_1,\mathbf{v}_2\} = \text{Span}\{\mathbf{x}_1,\mathbf{x}_2\}$
Applying the subspace projection formula without an orthogonal basis	The formula $\hat{\mathbf{y}} = \sum \frac{\mathbf{y}\cdot\mathbf{u}_i}{\mathbf{u}_i\cdot\mathbf{u}_i}\mathbf{u}_i$ requires the $\mathbf{u}_i$ to be ORTHOGONAL; it fails for a non-orthogonal basis	First run Gram-Schmidt to make the basis orthogonal, then project

Practice Problems

Level 1 Project onto a Vector

Find the orthogonal projection of $\mathbf{y} = \begin{pmatrix}4\\2\\-1\end{pmatrix}$ onto $\mathbf{u} = \begin{pmatrix}1\\-1\\2\end{pmatrix}$.

Then find the error vector and verify it is orthogonal to $\mathbf{u}$.

Thought Process

Compute $\mathbf{y}\cdot\mathbf{u}$ and $\mathbf{u}\cdot\mathbf{u}$. Form the scalar $\frac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}$ and multiply $\mathbf{u}$.

Show Answer

$\mathbf{y}\cdot\mathbf{u} = 4-2-2 = 0$.

$\hat{\mathbf{y}} = \frac{0}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u} = \mathbf{0}$.

The projection is the zero vector. The error is $\mathbf{y} - \mathbf{0} = \mathbf{y}$.

Orthogonality check: $\mathbf{y}\cdot\mathbf{u} = 0$ (already computed). So $\mathbf{y}$ itself is orthogonal to $\mathbf{u}$, and its projection onto $\text{Span}\{\mathbf{u}\}$ is zero.

Geometric meaning: $\mathbf{y}$ is entirely in the direction perpendicular to $\mathbf{u}$. The closest point in $\text{Span}\{\mathbf{u}\}$ to $\mathbf{y}$ is the origin.

Level 2 Project onto a 1D Subspace

Let $W = \text{Span}\left\{\begin{pmatrix}2\\1\\-1\end{pmatrix}\right\}$ and $\mathbf{y} = \begin{pmatrix}3\\-1\\2\end{pmatrix}$.

Find $\hat{\mathbf{y}} = \text{proj}_W\mathbf{y}$ and $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$.

Verify $\mathbf{z} \in W^\perp$.

Thought Process

Let $\mathbf{u} = \begin{pmatrix}2\\1\\-1\end{pmatrix}$. Compute $\frac{\mathbf{y}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u}$.

Then $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$; verify $\mathbf{z}\cdot\mathbf{u} = 0$.

Show Answer

$\mathbf{y}\cdot\mathbf{u} = 6-1-2 = 3$. $\mathbf{u}\cdot\mathbf{u} = 4+1+1 = 6$.

\[ \hat{\mathbf{y}} = \frac{3}{6}\begin{pmatrix}2\\1\\-1\end{pmatrix} = \begin{pmatrix}1\\1/2\\-1/2\end{pmatrix} \]

\[ \mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = \begin{pmatrix}2\\-3/2\\5/2\end{pmatrix} \]

Verify: $\mathbf{z}\cdot\mathbf{u} = 2(2)+(-3/2)(1)+(5/2)(-1) = 4-3/2-5/2 = 4-4 = 0$. In $W^\perp$.

Level 3 Project onto a 2D Subspace

Let $W = \text{Span}\left\{\mathbf{u}_1, \mathbf{u}_2\right\}$ where $\mathbf{u}_1 = \begin{pmatrix}1\\0\\-1\end{pmatrix}$ and $\mathbf{u}_2 = \begin{pmatrix}1\\2\\1\end{pmatrix}$ (you verified these are orthogonal in [Inner Products, Level 3]).

Find $\hat{\mathbf{y}} = \text{proj}_W\mathbf{y}$ for $\mathbf{y} = \begin{pmatrix}2\\3\\-1\end{pmatrix}$.

Thought Process

Since $\{\mathbf{u}_1, \mathbf{u}_2\}$ is an orthogonal basis for $W$, apply the projection formula directly: sum the two individual projections.

Show Answer

$\mathbf{y}\cdot\mathbf{u}_1 = 2+0+1 = 3$. $\mathbf{u}_1\cdot\mathbf{u}_1 = 1+0+1 = 2$.

$\mathbf{y}\cdot\mathbf{u}_2 = 2+6-1 = 7$. $\mathbf{u}_2\cdot\mathbf{u}_2 = 1+4+1 = 6$.

\[ \hat{\mathbf{y}} = \frac{3}{2}\begin{pmatrix}1\\0\\-1\end{pmatrix} + \frac{7}{6}\begin{pmatrix}1\\2\\1\end{pmatrix} = \begin{pmatrix}3/2\\0\\-3/2\end{pmatrix} + \begin{pmatrix}7/6\\7/3\\7/6\end{pmatrix} = \begin{pmatrix}9/6+7/6\\7/3\\-9/6+7/6\end{pmatrix} = \begin{pmatrix}8/3\\7/3\\-1/3\end{pmatrix} \]

Verify: $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = \begin{pmatrix}2-8/3\\3-7/3\\-1+1/3\end{pmatrix} = \begin{pmatrix}-2/3\\2/3\\-2/3\end{pmatrix}$.

$\mathbf{z}\cdot\mathbf{u}_1 = -2/3+0+2/3 = 0$ (correct). $\mathbf{z}\cdot\mathbf{u}_2 = -2/3+4/3-2/3 = 0$ (correct).

Level 4 Apply Gram-Schmidt

Apply the Gram-Schmidt process to the basis $\left\{\begin{pmatrix}1\\1\\1\end{pmatrix}, \begin{pmatrix}1\\2\\0\end{pmatrix}, \begin{pmatrix}1\\0\\0\end{pmatrix}\right\}$ for $\mathbb{R}^3$.

Produce an orthogonal basis $\{\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3\}$. Verify pairwise orthogonality.

Thought Process

Step 1: $\mathbf{v}_1 = \mathbf{x}_1$.

Step 2: subtract from $\mathbf{x}_2$ its projection onto $\mathbf{v}_1$.

Step 3: subtract from $\mathbf{x}_3$ its projections onto both $\mathbf{v}_1$ and $\mathbf{v}_2$.

Show Answer

$\mathbf{v}_1 = \begin{pmatrix}1\\1\\1\end{pmatrix}$. $\mathbf{v}_1\cdot\mathbf{v}_1 = 3$.

Step 2: $\mathbf{x}_2\cdot\mathbf{v}_1 = 1+2+0 = 3$.

$\mathbf{v}_2 = \begin{pmatrix}1\\2\\0\end{pmatrix} - \frac{3}{3}\begin{pmatrix}1\\1\\1\end{pmatrix} = \begin{pmatrix}0\\1\\-1\end{pmatrix}$. $\mathbf{v}_2\cdot\mathbf{v}_2 = 0+1+1 = 2$.

Step 3: $\mathbf{x}_3\cdot\mathbf{v}_1 = 1+0+0 = 1$. $\mathbf{x}_3\cdot\mathbf{v}_2 = 0+0+0 = 0$, so $\mathbf{x}_3$ is already orthogonal to $\mathbf{v}_2$.

$\mathbf{v}_3 = \begin{pmatrix}1\\0\\0\end{pmatrix} - \frac{1}{3}\begin{pmatrix}1\\1\\1\end{pmatrix} - 0\cdot\mathbf{v}_2 = \begin{pmatrix}2/3\\-1/3\\-1/3\end{pmatrix}.$

Direction is all that matters for an orthogonal basis, so scaling by $3$ gives $\mathbf{v}_3$ parallel to $\begin{pmatrix}2\\-1\\-1\end{pmatrix}$.

Orthogonal basis: $\left\{\begin{pmatrix}1\\1\\1\end{pmatrix}, \begin{pmatrix}0\\1\\-1\end{pmatrix}, \begin{pmatrix}2\\-1\\-1\end{pmatrix}\right\}$.

Verify pairwise orthogonality: $\mathbf{v}_1\cdot\mathbf{v}_2 = 0+1-1 = 0$; $\mathbf{v}_1\cdot\mathbf{v}_3 = 2-1-1 = 0$; $\mathbf{v}_2\cdot\mathbf{v}_3 = 0-1+1 = 0$. Every pair is orthogonal, so the three vectors form an orthogonal basis for $\mathbb{R}^3$.

Level 5 Gram-Schmidt and the Best Approximation

Let $W = \text{Col}(A)$ where $A = \begin{pmatrix}1&1\\1&0\\0&1\end{pmatrix}$.

(a) Apply Gram-Schmidt to the columns of $A$ to produce an orthogonal basis $\{\mathbf{v}_1, \mathbf{v}_2\}$ for $W$.

(b) Find the orthogonal projection of $\mathbf{y} = \begin{pmatrix}2\\1\\3\end{pmatrix}$ onto $W$.

(c) Verify that $\mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to both columns of $A$.

Thought Process

For (a): $\mathbf{v}_1 = \mathbf{a}_1$, then subtract the projection of $\mathbf{a}_2$ onto $\mathbf{v}_1$.

For (b): use the orthogonal basis from (a) with the subspace projection formula.

For (c): compute $\mathbf{y}-\hat{\mathbf{y}}$ and check the dot product with each column of $A$.

Show Answer

(a) Let $\mathbf{x}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}$, $\mathbf{x}_2 = \begin{pmatrix}1\\0\\1\end{pmatrix}$.

$\mathbf{v}_1 = \mathbf{x}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}$. $\mathbf{v}_1\cdot\mathbf{v}_1 = 2$.

$\mathbf{x}_2\cdot\mathbf{v}_1 = 1+0+0 = 1$.

$\mathbf{v}_2 = \begin{pmatrix}1\\0\\1\end{pmatrix} - \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} = \begin{pmatrix}1/2\\-1/2\\1\end{pmatrix}$.

$\mathbf{v}_2\cdot\mathbf{v}_2 = 1/4+1/4+1 = 3/2$.

Check: $\mathbf{v}_1\cdot\mathbf{v}_2 = 1/2-1/2+0 = 0$ (orthogonal).

(b) $\mathbf{y}\cdot\mathbf{v}_1 = 2+1+0 = 3$. $\mathbf{y}\cdot\mathbf{v}_2 = 1-1/2+3 = 7/2$.

\[ \hat{\mathbf{y}} = \frac{3}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} + \frac{7/2}{3/2}\begin{pmatrix}1/2\\-1/2\\1\end{pmatrix} = \begin{pmatrix}3/2\\3/2\\0\end{pmatrix} + \frac{7}{3}\begin{pmatrix}1/2\\-1/2\\1\end{pmatrix} = \begin{pmatrix}3/2+7/6\\3/2-7/6\\7/3\end{pmatrix} = \begin{pmatrix}16/6\\2/6\\7/3\end{pmatrix} = \begin{pmatrix}8/3\\1/3\\7/3\end{pmatrix} \]

(c) $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}} = \begin{pmatrix}2-8/3\\1-1/3\\3-7/3\end{pmatrix} = \begin{pmatrix}-2/3\\2/3\\2/3\end{pmatrix}$.

$\mathbf{z}\cdot\mathbf{x}_1 = -2/3+2/3+0 = 0$ (orthogonal to column 1).

$\mathbf{z}\cdot\mathbf{x}_2 = -2/3+0+2/3 = 0$ (orthogonal to column 2).

$\hat{\mathbf{y}}$ is the closest point in $W = \text{Col}(A)$ to $\mathbf{y}$.

Common Misconceptions

Common misconception

the projection $\hat{\mathbf{y}}$ is the foot of the perpendicular onto the vector $\mathbf{u}$ itself, not the line $\text{Span}\{\mathbf{u}\}$. The projection lands in $\text{Span}\{\mathbf{u}\}$, which is the entire line through the origin in the direction of $\mathbf{u}$. If $\mathbf{u} = \begin{pmatrix}1\\2\end{pmatrix}$, the projection of $\mathbf{y}$ onto $\mathbf{u}$ is a scalar multiple of $\mathbf{u}$, potentially far from the endpoint $\mathbf{u}$ itself.

Common misconception

Gram-Schmidt changes the subspace. At every step of the process, $\text{Span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\} = \text{Span}\{\mathbf{x}_1,\ldots,\mathbf{x}_k\}$. The algorithm builds a new DESCRIPTION (orthogonal basis) of the SAME subspace. The subspace does not change.

Common misconception

the subspace projection formula works for any basis, not just orthogonal ones. The formula $\hat{\mathbf{y}} = \sum \frac{\mathbf{y}\cdot\mathbf{u}_i}{\mathbf{u}_i\cdot\mathbf{u}_i}\mathbf{u}_i$ is valid only when $\{\mathbf{u}_i\}$ is an orthogonal set. For a non-orthogonal basis, the individual projections interact (are not independent) and the formula overcounts. First apply Gram-Schmidt, then project.

Mastery Checklist

Novice (Level 1-2):

I can compute the projection of a vector onto a single vector using the formula.
I can find the error vector and verify it is orthogonal to the subspace.
I can state the Best Approximation Theorem.

Competent (Level 3-4):

I can project onto a multi-dimensional subspace using an orthogonal basis.
I can apply Gram-Schmidt to a basis and produce an orthogonal set.
I can recognize when Gram-Schmidt reveals linear dependence (zero vector at some step).

Proficient (Level 5):

I can combine Gram-Schmidt and projection to find the closest point in Col$(A)$ to a given vector.
I can verify projection results by checking orthogonality of the error.
I can explain why the projection formula fails for non-orthogonal bases.

Mental Model

The Shadow

The orthogonal projection of $\mathbf{y}$ onto a subspace $W$ is the “shadow” of $\mathbf{y}$ on $W$ when the light shines perpendicular to $W$. The shadow is the closest point in $W$ to $\mathbf{y}$, and the gap between the shadow and $\mathbf{y}$ (the error vector) is exactly perpendicular to $W$.

Gram-Schmidt is the construction of a “coordinate frame” for $W$: starting from an arbitrary basis, it builds one direction at a time by removing any component that overlaps with the already-established directions. The result is a set of arrows that are all at right angles to each other, spanning the same space as before but with no angular overlap.

Connections

Within MATH 231:

Null Space, Column Space, and Basis: the projection onto Col$(A)$ is the least-squares solution to $A\mathbf{x} = \mathbf{b}$; Gram-Schmidt on the columns of $A$ produces the QR factorization
Eigenvalues and Eigenvectors: the Spectral Theorem says every real symmetric matrix has an orthonormal eigenvector basis; Gram-Schmidt is one way to construct it

Real-world connections:

Data fitting (least squares): the projection of $\mathbf{b}$ onto Col$(A)$ gives the best-fit coefficients when no exact solution exists.
Signal processing: projection onto orthogonal frequency components is the foundation of the Fourier transform.
Machine learning: dimensionality reduction (PCA) uses an orthogonal basis found via spectral methods.

Source: Lay, Linear Algebra and Its Applications, Section 6.3: Orthogonal Projections; Section 6.4: The Gram-Schmidt Process.

Previous	Up	Next
Inner Products, Length, and Orthogonality	MATH231 Skills Index

Last updated: 2026-06-16

Watch and Read

Lynch, MATH 231 lecture playlists (YouTube)Recorded MATH 231 lectures at EWU. Use them to see each topic worked through in a full class session.