aInner product space is vector space equipped with definition of inner product. In many literature, inner product and dot product are used interchangeably while mathematicians argue the difference appears when applying them on complex and real vector spaces. This note serves the purpose of introducing inner product spaces themselves, and the next note will discuss in more details the operators on inner product spaces.1. Inner Products and Norms 1.1. Three important theorems 2. Orthonormal Bases 2.1. Riesz Representation Theorem 3. Orthogonal Complements 3.1. Minimization problem 1. Inner Products and NormsTo make our definition of inner product more rigorous, let us first define the concept of dot product. Note here the dot product is defined in real vector spaces Rn${\displaystyle {\mathbb{R}}^n}$.

Definition1For x,y∈Rn${\displaystyle x,y\in R^n}$, the dot product of the two is difined asx⋅y=n∑i=1x1yi$$x\cdot y=\sum _{i=1}^n{x}_1{y}_i$$where x=(x1,...,xn)${\displaystyle x=(x_1,...,x_n)}$ and y=(y1,...,yn)${\displaystyle y=(y_1,...,y_n)}$.

With Definition 1, if we let x=y${\displaystyle x=y}$, then x⋅x=||x||2=∑x2i${\displaystyle x\cdot x=||x|{|}^2=\sum x_i^2}$. ||x||${\displaystyle ||x||}$ is called norm of x${\displaystyle x}$, and it represents the Euclidean length of vector x${\displaystyle x}$ that has its initial point at the origin. An inner product is a generalization of the dot product to complex vector spaces. For z∈Cn${\displaystyle z\in {\mathbb{C}}^n}$, its norm is defined as(1)||z|| =n∑i=1zi⏨zi$$||z||=\sqrt{\sum _{i=1}^n{z}_i{\overline{z}}_i}$$where ⏨zi${\displaystyle {\overline{z}}_i}$ is the conjugate of complex scalar zi${\displaystyle z_i}$. The norm definition above suggest that the inner product of w∈Cn${\displaystyle w\in {\mathbb{C}}^n}$ with v∈Cn${\displaystyle v\in {\mathbb{C}}^n}$ should be equal tow1⏨v1+⋯+wn⏨vn.$$w_1{\overline{v}}_1+\cdots +w_n{\overline{v}}_n.$$On the other hand, the inner product of v${\displaystyle v}$ with w${\displaystyle w}$ should then bev1⏨w1+⋯vn⏨wn,$$v_1{\overline{w}}_1+\cdots v_n{\overline{w}}_n,$$the complex conjugate to the inner product of w${\displaystyle w}$ with v${\displaystyle v}$. To satisfy these constraints we give the following definition for the inner product on both real and complex vector spaces.

Definition2An inner product on V${\displaystyle V}$ takes each ordered pair (u,v)${\displaystyle (u,v)}$ of elements of V${\displaystyle V}$ to a number ⟨u,v⟩∈F${\displaystyle ⟨u,v⟩\in \mathcal{F}}$.

It's worth noting here that the use of ⟨u,v⟩${\displaystyle ⟨u,v⟩}$ is pervasive in the studies of quantum mechanics. Some literature prefer (u,v)${\displaystyle (u,v)}$ for denoting the inner product, but we will stick to the former as it looks a lot like the Dirac's notation. For 𝜆∈F${\displaystyle 𝜆\in \mathcal{F}}$, and u,v∈V${\displaystyle u,v\in V}$, we have the following properties of the inner product hold:

a) definiteness: ⟨v,v⟩=0 if and only if v=0.${\displaystyle ⟨v,v⟩=0\text{ if and only if }v=0.}$b) positivity: ⟨v,v⟩≥0 for all v∈V.${\displaystyle ⟨v,v⟩\ge 0\text{ for all }v\in V.}$c) additivity in first slot: ⟨u+v,w⟩=⟨u,w⟩+⟨v,w⟩ for all u,v,w∈V.${\displaystyle ⟨u+v,w⟩=⟨u,w⟩+⟨v,w⟩\text{ for all }u,v,w\in V.}$d) homogeneity in first slot: ⟨𝜆u,v⟩=𝜆⟨u,v⟩$⟨𝜆u,v⟩=𝜆⟨u,v⟩$ for all 𝜆∈F$𝜆\in \mathbf{F}$ and all u,v∈V.$u,v\in V.$e) conjugate symmetry: ⟨u,v⟩=⏨⏨⏨⟨v,u⟩$⟨u,v⟩=\overline{⟨v,u⟩}$ for all u,v∈V$u,v\in V$.(b) ⟨0,u⟩=0$⟨0,u⟩=0$ for every u∈V$u\in V$.f) ⟨u,0⟩=0$⟨u,0⟩=0$ for every u∈V$u\in V$.g) ⟨u,v+w⟩=⟨u,v⟩+⟨u,w⟩$⟨u,v+w⟩=⟨u,v⟩+⟨u,w⟩$ for all u,v,w∈V$u,v,w\in V$.h) ⟨u,𝜆v⟩=⏨𝜆⟨u,v⟩$⟨u,𝜆v⟩=\overline{𝜆}⟨u,v⟩$ for all 𝜆∈F$𝜆\in \mathbf{F}$ and u,v∈V$u,v\in V$.

The definition of inner product results in two other definitions: norms and orthogonality. Eqn.(1) already gives a definition of norm by writting out a sum of products of complex vector and its conjugate. The same definition can be re-written in a more concise form as

Definition3The norm of v∈V${\displaystyle v\in V}$ is defined as ||v||=⟨v,v⟩${\displaystyle ||v||=\sqrt{⟨v,v⟩}}$.

For v,u∈V${\displaystyle v,u\in V}$, we can define the orthogonality between the two as the following

Definition4Two vectors v,u∈V${\displaystyle v,u\in V}$ are called orthogonal if ⟨v,u⟩=0${\displaystyle ⟨v,u⟩=0}$.

The concise definitions above should make the importance of inner product self-evident. In the next subsection we can show how important theorems can be proved by inner products of vectors easily.1.1. Three important theoremsWe will introduct in this section three important theorems: triangle inequality, Cauchy–Schwarz inequality, and parallelogram equality. We will see the power of using inner products in the proofs. As a starter, let us first state the triangle inequality in terms of norms of vectors

Theorem1For v,u∈V${\displaystyle v,u\in V}$, we have ||v+u||≤||u||+||v||${\displaystyle ||v+u||\le ||u||+||v||}$

Proof: ||v+u||2=⟨v+u,v+u⟩=||v||2+||u||2+⟨v,u⟩+⟨u,v⟩${\displaystyle ||v+u|{|}^2=⟨v+u,v+u⟩=||v|{|}^2+||u|{|}^2+⟨v,u⟩+⟨u,v⟩}$. Using the conjugate symmetry of inner product gives ||v+u||2=||v||2+||u||2+2Re(⟨v,u⟩)≤||v||2+||u||2+2||u||||v||=(||u||+||v||)2${\displaystyle ||v+u|{|}^2=||v|{|}^2+||u|{|}^2+2Re(⟨v,u⟩)\le ||v|{|}^2+||u|{|}^2+2||u||||v||=(||u||+||v||{)}^2}$. Taking square root at both sides gives desired result.We next prove the parallelogram equality, which usually makes its appearance in basic geometry textbooks.

Theorem2Suppose u,v∈V$u,v\in V$. Then||u+v‖2+||u-v‖2=2(||u||2+||v||2).$$||u+v{\Vert }^2+||u-v{\Vert }^2=2(||u|{|}^2+||v|{|}^2).$$

Proof:a

||u+v||2+||u-v||2=	⟨u+v,u+v⟩+⟨u-v,u-v⟩
=	||u||2+||v||2+⟨u,v⟩+⟨v,u⟩
	+||u||2+||v||2-⟨u,v⟩-⟨v,u⟩
=	2(||u||2+||v||2)

$$\begin{array}{rl}||u+v|{|}^2+||u-v|{|}^2=& ⟨u+v,u+v⟩+⟨u-v,u-v⟩\\ =& ||u|{|}^2+||v|{|}^2+⟨u,v⟩+⟨v,u⟩\\ & +||u|{|}^2+||v|{|}^2-⟨u,v⟩-⟨v,u⟩\\ =& 2(||u|{|}^2+||v|{|}^2)\end{array}$$We end this section by proving arguably the most important theorem for inner product vector spaces: Cauchy-Schwatz inequality. To do so, we need to first introduce an orthogonal decomposition of vector in inner product spaces:

Theorem3Suppose u,v∈V$u,v\in V$, with v≠0$v\ne 0$. Set c=⟨u,v⟩

||v||2$c=\frac{⟨u,v⟩}{||v|{|}^2}$ and w=u-⟨u,v⟩

||v||2v$w=u-\frac{⟨u,v⟩}{||v|{|}^2}v$. Then ⟨w,v⟩=0 $⟨w,v⟩=0$ and u=cv+w$u=cv+w$.

Theorem 3 should make sense to the reader as ⟨u,v⟩

||v||$\frac{⟨u,v⟩}{||v||}$ is the projection of u${\displaystyle u}$ at the direction of v${\displaystyle v}$. Now we are good to prove the Cauchy-Schwartz inequality

Theorem4Suppose u,v∈V$u,v\in V$. Then|⟨u,v⟩|≤||u||⋅||v||.$$|⟨u,v⟩|\le ||u||\cdot ||v||.$$This inequality is an equality if and only if one of u,v$u,v$ is a scalar multiple of the other.

Proof: We can use Theorem 3 to decompose u=⟨u,v⟩

||v||2v+w${\displaystyle u=\frac{⟨u,v⟩}{||v|{|}^2}v+w}$ with ⟨w,v⟩=0${\displaystyle ⟨w,v⟩=0}$. Taking the square of both sides gives ||u||2=${\displaystyle ||u|{|}^2=}$|⟨u,v⟩|2

||v||2+||w||2${\displaystyle \frac{|⟨u,v⟩{|}^2}{||v|{|}^2}+||w|{|}^2}$ and ||u||2||v||2=|⟨u,v⟩|2+||v||2||w||2${\displaystyle ||u|{|}^2||v|{|}^2=|⟨u,v⟩{|}^2+||v|{|}^2||w|{|}^2}$. Because ||v||2||w||2≥0${\displaystyle ||v|{|}^2||w|{|}^2\ge 0}$, we have |⟨u,v⟩|≤||u||⋅||v||$$|⟨u,v⟩|\le ||u||\cdot ||v||$$Also, ${\displaystyle }$|⟨u,v⟩|=||u||⋅||v||${\displaystyle |⟨u,v⟩|=||u||\cdot ||v||}$ means ||u||2=${\displaystyle ||u|{|}^2=}$|⟨u,v⟩|2

||v||2${\displaystyle \frac{|⟨u,v⟩{|}^2}{||v|{|}^2}}$, i.e., ||w||=0${\displaystyle ||w||=0}$. Therefore the equality is satisfied if and only if one of u,v${\displaystyle u,v}$ is scalar multiple of the other.2. Orthonormal BasesWe have discussed in the note of bases and dimensions that a basis of vector space is a list of linearly independent vectors. Now that we have the definition of orthogonality in our pocket, we can take a step further to demand basis vectors to be mutually orthogonal with their norms being 1. A basis satisfies these constraints is called orthonormal bases, a most used basis in quantum-mechanical calculations. The reason for choosing orthonormal bases lies in the following proposition

Proposition1Every finite-dimensional inner product space has an orthonormal basis.

In this section, we will first give the definition of orthonormal bases and discuss how we can obtain them from a basis of linearly independent vectors through Gram–Schmidt procedure. But first, let us make the definition of orthonormal basis formal as the following:

Definition5A basis of V${\displaystyle V}$, e1,e2,...,en${\displaystyle e_1,e_2,...,e_n}$ is called orthonormal if <ej,ek>=a

1		if j=k
0		if j≠k

$$⟨e_j,e_k⟩=\{\begin{array}{ll}1& \text{ if }j=k\\ 0& \text{ if }j\ne k\end{array}$$

It turns out that arbitrary vector v∈V${\displaystyle v\in V}$ can be expressed in terms of orthonormal basis as the following(2)v=n∑i=1<v,ei>ei$$v=\sum _{i=1}^n⟨v,e_i⟩e_i$$where n=dimV${\displaystyle n=dimV}$. To see this, suppose v=a1e1+⋯+anen${\displaystyle v=a_1{e}_1+\cdots +a_n{e}_n}$, and we have ai=⟨v,ei⟩${\displaystyle a_i=⟨v,e_i⟩}$ due to Definition 5. As one might expect, the norm of v${\displaystyle v}$ is then||v||=n∑i=1<v,ei>2.$$||v||=\sqrt{\sum _{i=1}^n⟨v,e_i{⟩}^2}.$$To prove this, let ai=<v,ei>${\displaystyle a_i=⟨v,e_i⟩}$ and v=a1e1+a2e2+⋯+anen${\displaystyle v=a_1{e}_1+a_2{e}_2+\cdots +a_n{e}_n}$. So ||v||2=⟨v,v⟩=a1⏨a1+a2⏨a2+⋯+an⏨an${\displaystyle ||v|{|}^2=⟨v,v⟩=a_1{\overline{a}}_1+a_2{\overline{a}}_2+\cdots +a_n{\overline{a}}_n}$ due to Definition 5 and property (h) of inner product.It is now obvious that orthonormal basis, empowered by inner product, is a natural choice for inner product spaces. Furthermore, if we a list of linear independent vectors, we can convert them into a list of orthonormal vectors using the Gram-Schmidt procedure:

Theorem5Suppose v1,…,vm$v_1,\dots ,v_m$ is a linearly independent list of vectors in V$V$. Let e1=v1/‖‖v1‖‖$e_1=v_1/\Vert v_1\Vert$. For j=2,…,m$j=2,\dots ,m$, define ej$e_j$ inductively byej=vj-<vj,e1>e1-⋯-<vj,ej-1>ej-1

‖‖vj-<vj,e1>e1-⋯-<vj,ej-1>ej-1‖‖.$$e_j=\frac{v_j-⟨v_j,e_1⟩e_1-\cdots -⟨v_j,e_{j-1}⟩e_{j-1}}{\Vert v_j-⟨v_j,e_1⟩e_1-\cdots -⟨v_j,e_{j-1}⟩e_{j-1}\Vert }.$$Then e1,…,em$e_1,\dots ,e_m$ is an orthonormal list of vectors in V$V$ such thatspan(v1,…,vj)=span(e1,…,ej)$$\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}(v_1,\dots ,v_j)=\mathrm{s}\mathrm{p}\mathrm{a}\mathrm{n}(e_1,\dots ,e_j)$$for j=1,…,m$j=1,\dots ,m$.

One of common wishful thinking is that matrix with respect to orthornomal basis is diagonal. This is usually not the case. But the next theorem shows that if an operator has an upper-triangular matrix with respect to some basis, it also has an upper-triangular matrix with respect to orthonormal basis. Note that the following theorem holds for both real and complex vector spaces.

Theorem6Suppose T∈L(V)$T\in \mathcal{L}(V)$. If T$T$ has an upper-triangular matrix with respect to some basis of V$V$, then T$T$ has an upper-triangular matrix with respect to some orthonormal basis of V$V$.

Proof: Let v1,...vn${\displaystyle v_1,...v_n}$ be a basis of V${\displaystyle V}$, from which we can get an orthonormal basis e1,...,en${\displaystyle e_1,...,e_n}$ by using Gram-Schmit procedure in Theorem 5. Because span(v1,...vn)=span(e1,..,en)${\displaystyle span(v_1,...v_n)=span(e_1,..,e_n)}$, we can conclude that span(e1,...,en)${\displaystyle span(e_1,...,e_n)}$ is invariant under T${\displaystyle T}$. From the condition c) for having an upper-triangular matrix, T${\displaystyle T}$ must have a upper-triangular matrix with respect to the orthonormal basis.Theorem 6 has many applications, one of which is helping proving the Shur's theorem:

Theorem7Suppose V$V$ is a finite-dimensional complex vector space and T∈L(V)$T\in \mathcal{L}(V)$. Then T$T$ has an upper-triangular matrix with respect to some orthonormal basis of V$V$.

Proof: recall the theorem on the existence of upper-triangular matrix in the previous notes, then we can apply Theorem 6 here to prove Theorem 7.2.1. Riesz Representation TheoremTheorem 6 and Theorem 7 give what operator matrices might look like with respect to orthonormal basis, and we will discuss more about operators on inner product space in our next note. For now, let us confine our scope to just discuss one special kind of operators, i.e., linear functional. We have briefly discussed it in our note on the duality of vector space, and its formal definition is the following

Definition6A linear functional on V$V$ is a linear map from V$V$ to F$\mathbf{F}$. In other words, a linear functional is an element of the dual space of V${\displaystyle V}$, L(V,F)$\mathcal{L}(V,\mathbf{F})$.

For v,u∈V${\displaystyle v,u\in V}$, a map from v${\displaystyle v}$ to ⟨v,u⟩${\displaystyle ⟨v,u⟩}$ is a linear functional. The following theorem (Ries representation theorem) shows that every linear functional on inner product space is essentially a inner product.

Theorem8Suppose V$V$ is finite-dimensional and 𝜑$𝜑$ is a linear functional on V$V$. Then there is a unique vector u∈V$u\in V$ such that𝜑(v)=⟨v,u⟩$$𝜑(v)=⟨v,u⟩$$for every v∈V$v\in V$.

Proof: We include the proof here because it shows some tricks that will be used in our quantum computing /information theory notes repeatedly. Suppose e1,e2,...,en${\displaystyle e_1,e_2,...,e_n}$ is an orthonormal basis of V${\displaystyle V}$, we havev=⟨v,e1⟩e1+⋯+⟨v,en⟩en$$v=⟨v,e_1⟩e_1+\cdots +⟨v,e_n⟩e_n$$anda

𝜑(v)	=⟨v,e1⟩𝜑(e1)+⋯+⟨v,en⟩𝜑(en)
	=⟨v,⏨⏨⏨𝜑(e1)e1⟩+⋯+⟨v,⏨⏨⏨𝜑(en)en⟩
	=⟨v,⏨⏨⏨𝜑(e1)e1+⋯+⏨⏨⏨𝜑(en)en⟩

$$\begin{array}{rl}𝜑(v)& =⟨v,e_1⟩𝜑(e_1)+\cdots +⟨v,e_n⟩𝜑(e_n)\\ & =⟨v,\overline{𝜑(e_1)}{e}_1⟩+\cdots +⟨v,\overline{𝜑(e_n)}{e}_n⟩\\ & =⟨v,\overline{𝜑(e_1)}{e}_1+\cdots +\overline{𝜑(e_n)}{e}_n⟩\end{array}$$where the first equivalence is from the property of linear functional, and the second one is due to the property h) of the inner product. Because ⏨⏨⏨𝜑(ei)${\displaystyle \overline{𝜑(e_i)}}$ are scalars, we recover the form of 𝜑(v)=⟨v,u⟩${\displaystyle 𝜑(v)=⟨v,u⟩}$ if we let (3)u=⏨⏨⏨𝜑(e1)e1+⋯+⏨⏨⏨𝜑(en)en$$u=\overline{𝜑(e_1)}{e}_1+\cdots +\overline{𝜑(e_n)}{e}_n$$To prove the uniqueness of u${\displaystyle u}$, suppose we have two vectors, u1${\displaystyle u_1}$ and u2${\displaystyle u_2}$, that map every v∈V${\displaystyle v\in V}$ to scalar through inner product. Then, we have𝜑(v)=⟨v,u1⟩=⟨v,u2⟩$$𝜑(v)=⟨v,u_1⟩=⟨v,u_2⟩$$So 0=⟨v,u1⟩-⟨v,u2⟩=⟨v,u1-u2⟩${\displaystyle 0=⟨v,u_1⟩-⟨v,u_2⟩=⟨v,u_1-u_2⟩}$ for every v∈V${\displaystyle v\in V}$, which indicates that u1=u2${\displaystyle u_1=u_2}$. □The u${\displaystyle u}$ in Theorem 8 is also called as Riesz representation of the linear fucntional 𝜑${\displaystyle 𝜑}$. More importantly, Eqn.(3) gives a way to calculate the Riesz representation. While Eqn.(3) seems to be dependent on the choice of orthonormal basis, we emphasize here that the Riesz representation is only determined by its associated linear functional, and the right hand side of the equation remains the same regardless of the choice of {ei}${\displaystyle \{e_i\}}$.3. Orthogonal ComplementsAs an extention to Definition 4, we can define the concept of orthogonal complement for subspaces of an inner product space.

Definition7If U$U$ is a subset of V$V$, then the orthogonal complement of U$U$, denoted U⟂$U^{⟂}$, is the set of all vectors in V$V$ that are orthogonal to every vector in U$U$ :U⟂={v∈V:⟨v,u⟩=0 for every u∈U}$$U^{⟂}=\{v\in V:⟨v,u⟩=0\text{ for every }u\in U\}$$

With Definition 7, the following propositions and theorems hold as expected:

Theorem9Suppose V$V$ is finite-dimensional and U$U$ is a subspace of V$V$. ThendimU⟂=dimV-dimU.V=U⊕U⟂U=(U⟂)⟂$$\begin{array}{c}\mathrm{d}\mathrm{i}\mathrm{m}{U}^{⟂}=\mathrm{d}\mathrm{i}\mathrm{m}V-\mathrm{d}\mathrm{i}\mathrm{m}U.\\ V=U\oplus U^{⟂}\\ U=(U^{⟂}{)}^{⟂}\end{array}$$

The definition of orthogonal complement gives the definition of othogonal projection operator, which projects vector to a specific subspace.

Definition8Suppose U$U$ is a finite-dimensional subspace of V$V$. The orthogonal projection of V$V$ onto U$U$ is the operator PU∈L(V)$P_U\in \mathcal{L}(V)$ defined as follows: For v∈V$v\in V$, write v=u+w$v=u+w$, where u∈U$u\in U$ and w∈U⟂$w\in U^{⟂}$. Then PUv=u$P_{U}v=u$.

As an example, suppose v∈V${\displaystyle v\in V}$ and U=span(v)${\displaystyle U=span(v)}$, then for u∈V${\displaystyle u\in V}$, PUu=⟨u,v⟩

||v||2v${\displaystyle P_{U}u=\frac{⟨u,v⟩}{||v|{|}^2}v}$. Note here that ⟨u,v⟩

||v||2v${\displaystyle \frac{⟨u,v⟩}{||v|{|}^2}v}$ and u-⟨u,v⟩

||v||2v${\displaystyle u-\frac{⟨u,v⟩}{||v|{|}^2}v}$ are the orthogonal decomposition of v${\displaystyle v}$, and ⟨u,v⟩

||v||2v${\displaystyle \frac{⟨u,v⟩}{||v|{|}^2}v}$ is a scalar multiple of v${\displaystyle v}$. Thus, u-⟨u,v⟩

||v||2v∈U⟂${\displaystyle u-\frac{⟨u,v⟩}{||v|{|}^2}v\in U^{⟂}}$.Before we discuss minimization problem using orthogonal projection, let us simply list its properties:

Suppose U$U$ is a finite-dimensional subspace of V$V$ and v∈V$v\in V$. Then(a) PU∈L(V)$P_U\in \mathcal{L}(V)$;(b) PUu=u$P_{U}u=u$ for every u∈U$u\in U$;(c) PUw=0$P_{U}w=0$ for every w∈U⟂$w\in U^{⟂}$;(d) range PU=U$P_U=U$;(e) null PU=U⟂$P_U=U^{⟂}$;(f) v-PUv∈U⟂$v-P_{U}v\in U^{⟂}$;(g) P2U=PU$P_U^2=P_U$;(h) ‖‖PUv‖‖≤||v||$\Vert P_{U}v\Vert \le ||v||$;(i) for every orthonormal basis e1,…,em$e_1,\dots ,e_m$ of U$U$,PUv=<v,e1>e1+⋯+<v,em>em.$$P_{U}v=⟨v,e_1⟩e_1+\cdots +⟨v,e_m⟩e_m.$$

3.1. Minimization problemGiven a subspace U⊂V${\displaystyle U\subset V}$, and a vector v∈V${\displaystyle v\in V}$, a natural question would be: how can one find u∈U${\displaystyle u\in U}$ such that the norm ||v-u||${\displaystyle ||v-u||}$ is minimized. The following proposition gives the answer to the question:

Proposition2Suppose U⊂V${\displaystyle U\subset V}$ is a finite-dimensional subspace of V${\displaystyle V}$. For v∈V${\displaystyle v\in V}$ and u∈U${\displaystyle u\in U}$, we have||v-u||≥||PUv-u||$$||v-u||\ge ||P_{U}v-u||$$where the equality is satisfied if and only if u=PUv${\displaystyle u=P_{U}v}$.

As an example of using Proposition 2(took from Axler's book), let us find a real function u(x)${\displaystyle u(x)}$ such that the value of (4)𝜋∫-𝜋|sinx-u(x)|2dx$$\underset{-𝜋}{\overset{𝜋}{\int }}|\sin x-u(x){|}^{2}dx$$is minimized. On the vector space of real fucntions defined on [-𝜋,𝜋]${\displaystyle [-𝜋,𝜋]}$, we can define its associated inner product as(5)⟨f(x),g(x)⟩=𝜋∫-𝜋f(x)g(x)dx$$⟨f(x),g(x)⟩=\underset{-𝜋}{\overset{𝜋}{\int }}f(x)g(x)dx$$If we let v(x)=sinx${\displaystyle v(x)=\sin x}$, then Eqn.(4) is just ⟨v(x)-u(x),v(x)-u(x)⟩=||v-u||2${\displaystyle ⟨v(x)-u(x),v(x)-u(x)⟩=||v-u|{|}^2}$. With this observation, the original problem is then finding a u(x)${\displaystyle u(x)}$ to minimize ||v-u||${\displaystyle ||v-u||}$. Accoding to Proposition 2, such a function must be PUv${\displaystyle P_{U}v}$. Since there is no limitation of how to choose our U${\displaystyle U}$, we can let U${\displaystyle U}$ be the space of polynomials with real coefficient and degree at most 5(i.e. P5(R)${\displaystyle {\mathcal{P}}_5(\mathbb{R})}$). A basis of P5(R)${\displaystyle {\mathcal{P}}_5(\mathbb{R})}$ is 1,x,x2,x3,x4,x5${\displaystyle 1,x,x^2,x^3,x^4,x^5}$. We make orthonormal through the Gram-Schwartz procedure. Next, we can use Eqn.(5) and the property (i) of the orthogonal projection to calculate u(x)${\displaystyle u(x)}$ asu(x)=0.987862 x-0.155271 x3+0.00564312 x5$$u(x)=0.987862x-0.155271x^3+0.00564312x^5$$The following plot shows how close u(x)${\displaystyle u(x)}$(green) and sinx${\displaystyle \sin x}$(blue) are within [-𝜋,𝜋]${\displaystyle [-𝜋,𝜋]}$. It is worth noting that such closeness is not achieved through Tyler expansion!

And finally, ladies and gentleman, we have a colored plot in our bland-and-white notes on Linear Algebra!