Engineering Math - Matrix
Inner Product/Dot Product
Inner Product is a mathematical operation for two data set (basically two vector or data set) that performs following
i) multiply two data set element-by-element
ii) sum all the numbers obtained at step i)
This may be one of the most frequently used operation in mathematics (especially in engineering math). Sometimes it is used because the result indicates a specific mathemaatical or physical meaning and sometimes it is used just because it simplifies the mathematical representation. Anyway, it is used a LOT. Basically Innerproduct is fundamental operation for what I explains in Sum of Times (Sum of Multiplication) page.
The geomatrc meaning of Inner Product is as follows. Inner Product is a kind of operation which gives you the idea of angle between the two vectors. Actually the most important application of inner product are
For more detailed descriptions of the application, see the last section (Applied meaning of Vector Inner Product) and read the whole pages if you are interested in how the vector inner product can indicate this kind of information.
There are several different ways of representing/calculating the inner product. Equation (1) gives you the geometric meaning of inner product. Equation (2) would not shows you any idea of visualization, but it gives you a way of calculating the inner product with very simple multiplication and sums(Equation (2) would be the most common ways to calculate the inner product in most of the application). Equation (3) is the form representing the inner product as a multiplication of two vectors.
What I explained above is a kind of geometric meaning (graphical meaning) of Inner product. This kind of application can be used in 2D (two element vector) and 3D (three element vector) which can be easily visualized, but it would be difficult to interpret the inner product in this way if the number of the element grow larger. In most of the applications other than computer graphics, the inner product of vectors is done in algebric procedure as shown below.
As a simple example, let's think of a case of taking the inner product of two vectors which is made up of 3 elements as shown below.
Before I ask you to take the inner product of two vectors, I would like to talk first about how to make a proper question of inner product.
First if I ask you to perform the following operation (inner product of two vectors in the following format), does it make sense to you ? Is this a valid question ?
The answer is NO. The question itself is not valid. In any matrix inner product, there is an important rule as shown below (Vector is also a kind of matrix, so vector inner product should follow this rule as well. 'm x n', 'a x b', 'm x b' represents the dimension of a vector or matrix. If you are not familiar with the concept of dimension, refer to vector/matrix dimension page)
Now let's check if the problem I gave you meets this requirement. As shown below, the problem that I gave you does not meet the requirement, so the problem itself is invalid.
What if I change the problem as below ? (In this problem, I transposed the v2). Does this make sense ?
Yes, it make sense. It is valid problem since it meets the matrix inner product rule as shown below. One think you have to notice is that the result of this form of inner product is 1 x 1 vector. 1 x 1 vector (matrix) is same as scalar.
Now let's look how this inner product is calculated. The calculation is as simple as follows. You may have a very long calculation if the size of the vector is large. But it is just long, but it is very simple calculation. Just multiply each elements of two vectors and sum them all.
Now let's think of another example of vector porduct. This may look almost same as the previous example, but it is not exactly same. The difference is that v1 is transposed as opposed to v2 in previous example.
Before you start calculation, let's check if this problem is meaning (valid) question or not. It is valid question because it meets the inner produt rule as shown below. One important thing you need to notice in this case is that the result of the inner product in this case is 3 x 3 matrix, it is not a scalar value.
The result of calculation is as follows. I recommend you to take some time to take a close look at this.
Now let's try to increase the size of a vector and see if you can figure out the result from the simple example as explained above.
Let's suppose you are give two vectors as shown below.
And you are asked to take the inner product of the two vectors as shown below.
First, check if this problem itself is valid or not. (I will leave this to you. Try on your own if this problem is valid or not). Just to give you the conclusion. the answer is 'Yes, the problem is valid'.
What is the result of the inner product in this case ? It is going to be as follows.
If the size of vector gets very large, it would be too tedious to write the result as above. If you take a little bit closer examination of the expression, you would notice that you can simplify the expressing using 'Sum (Sigma)' notation as shown below.
When you see the case of vector inner product in real application, it is very important of the practical meaning of the vector inner product.
I see two major application of the inner product. One is to figure out the angle between the two vectors as illustrated above. (First, you calculate the inner product using Equation (2) and with the result and equation (1), you can figure out the angle).
This angle in turn has additional implication. If this angle is 90 degree(0.5 pi), it means
If this angle is 0 degree, it means
If this angle is between 0 and 90, the value is between min(0) and max value and they are partially dependent to each other.
Second application is nothing to do with geometric meaning, but it is very handy form of mathematical representation of "Sum of Multiplication/Sume of Times" as shown in "Sum of Times" page.
One important thing you have to remember is that the result of inner product of two vectors is a scalar.
Example 1 >
This examples shows how the angle between two vectors can be calculated by Inner Product. Click on the figure or click here for step by step process.
Example 2 >
This examples shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process.
Example 3 >
This examples shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process. This type of operation is fundamental process in Discrete Fourier Transform.
Example 4 >
This is another that shows how the inner product for the two functions can be done. Click on the figure or click here for step by step process.