[K-MOOC]  Introductory Mathematics for Artificial Intelligence

                Translated by

  Sang-Gu LEE with Youngju NIELSEN, Yoonmee HAM

         from the original Korean text written by

      Sang-Gu LEE with Jae Hwa LEE, Yoonmee HAM, Kyung-Eun PARK

    Part Ⅳ. AI and Statistics

11. Expectation, variance, covariance, correlation coefficient, covariance matrix

  11.1 Expectation, variance, standard deviation

  11.2 Joint probability distribution

  *11.3 Covariance, correlation coefficient

  11.4 Covariance matrix

11.1 Expectation, variance, standard deviation

The expected value, or expectation, of a random variable is an average value for probabilistic events, which is the sum of products of a value

obtained by each event and each event's probability. The variance of a random variable is a measure of the dispersion of numbers (data),

which indicates how far a set of numbers (data) are spread out from their average value. The standard deviation is defined as a square root of variance.

You can find the expectation, variance, and standard deviation of a discrete random variable with these formulas.

  (1) Expectation:

  (2) Variance:

  (3) Standard deviation:

  Find the expected value, variance, and standard deviation of a random variable .

     [Source]  http://matrix.skku.ac.kr/2018-album/R-Sage-Stat-Lab-2.html

Solution.  This problem can be solved easily using the Sage code.

We can find it with R code.

■ The expectation, the variance, and the standard deviation of a continuous random variable X can be found as follows:

  (1) Expectation: 

  (2) Variance: 

  (3) Standard deviation: 

■ Here are the properties of the expected value and variance.

 ① ,        

 ② ,        

 ③ When you define ,  Z is called 'a standardized random variable'.

The expected value of Z is 0 and the variance of Z is 1 according to the properties of expected value and variance 1 and 2.

Let the probability density function of variable is . Find the variance of and

Solution.

                 ■

◩ Open Problem 2

Try to find an example of a continuous random variable from other textbooks for your exercise.

Find the expected value, the variance, and the standard deviation of .

11.2 Joint probability distribution

When there are two or more random variables, we should also look at the probability distribution for those random variables together (e.g., the event two events happen simultaneously)

and  the probability distributions for each variable. We call the joint probability the probability two random variables happen at the same time.

When we know the probability distribution of two random variables together and the probability of one random variable, we can figure out the rest of

one random variable's probability distribution. We need to know the concept of the joint probability distribution(joint distribution)  to understand it.

Let’s look at the case with two discrete random variables.

(1) If and are discrete random variables, the joint probability function for and is defined as below:            

(2) We can show for every possible value of and as in the below table. We call it the joint probability distribution.

						Sum
							the marginal probability distribution for
Sum
	the marginal probability distribution for

(3) When the joint distribution for and is given, the marginal probability distribution is defined as below.

   The marginal probability distribution is the probability distribution for one variable of two variables described in the given joint probability distribution.

Here are the properties of the joint probability distribution.

 ① For all , ,

 ② For all , the sum of is 1.

 ③ For all ,

   There are three blue balls, two red balls, and three green balls of the same size in a pocket and we are going to take out two balls randomly.

Let’s say that among the balls we took out, the number of blue balls is , and the number of red balls is . Please answer the following questions

(1) Find the joint probability function of and ,

(2) Make a table of the joint probability distribution of and .

(3) Find ,

(4) Find the marginal distribution of , and

(5) Find the marginal distribution of .

Solution. We can get the table below using the definitions of joint distribution and marginal distribution.

① and ②

③

④

⑤

◆ As for a continuous random variable, we use the joint density function. The joint density function for continuous random variable and is defined as below.

[We use double integrals here. Refer to this YouTube lecture https://youtu.be/T1z_GYt85rI.]

 ①  , for all real

 ②  , for all real

 ③  , for all real

 ④ The probability of belongs to the area is as follows.

 ⑤ We can define the Marginal probability density function of and   as follows.

              , 

*   The joint density function of the two random variables and is given below.

                ,  , 

 Find the marginal density function of , in the given area of (0,1).

Solution. Using the definition, we can solve this problem as follows.

      , 

      , 

 [Reference]   A random variable, Expectation https://youtu.be/SUsZHarQqqg

*11.3 Covariance, correlation coefficient

The first thing we use to understand the distribution of a random variable is the mean. Using the mean, information about the distribution

can be expressed as a single number (the middle part of the distribution). The second concept we use is variance.

Using the variance indicates how far the distribution is from the mean.

Then, what method is needed to understand the probability distribution with two random variables and ? F

irst, we can think of the mean of and the mean of . Next, we can use the variance to determine how each variable is spread out.

However, the concept of covariance is necessary to know the correlation between two random variables.

The covariance of random variables and is defined as follows.

The  covariance is the average of the product of the deviation of and the deviation of . However, there is a problem that the covariance

is affected by the size of the units of and . To compensate for this, we use a correlation coefficient.

The correlation (coefficient) can be thought of as standardization by dividing the covariance by the standard deviations of two random variables.

Therefore, it is not affected by the absolute size of the random variable.

The correlation coefficient between the random variables and is defined as follows.

11.4 Covariance matrix

Using matrices, it is easy to express how several random variables relate to each other. The covariance matrix is created

by using the variance and covariance of each random variable.

The covariance matrix for {, , } of random variables is defined using a matrix with the covariance between the th and th random variable

when , and the variance of the th random variable when , It is denoted by

In simple terms, The covariance matrix is a square matrix with the variance in the main diagonal and covariance of two variables in off-diagonal.

The covariance matrix can be seen as representing the distribution of data, as shown in the figure below.

         [Source]  https://www.ritchieng.com/machine-learning-anomaly-detection/ 

Given the sample data as follows, find the covariance matrix of

it.

We can easily find the covariance matrix using code.

◩ Open Problem 3

Find the covariance matrix of the sample data found in other textbooks.

☞ Note  The covariance matrix plays an important role in dimension reduction, effectively reducing the dimension

while maintaining the distribution of high-dimensional data as much as possible. A typical technique is principal component analysis (PCA).

When calculating the principal component, the singular value decomposition (SVD) is mainly used.

◩ Open Problem 1

Discuss the Monty Hall problem of conditional probability, one example  in which Bayes' theorem is applied.

https://destrudo.tistory.com/5 

                 https://en.wikipedia.org/wiki/Monty_Hall

◩ Open Problem 2

Find the expected value, the variance, and the standard deviation of the continuous random variable found in other textbooks.

◩ Open Problem 3

Find the covariance matrix of the data found in other textbooks.

Copyright @ 2021 SKKU Matrix Lab. All rights reserved.
Made by Manager: Prof. Sang-Gu Lee and Dr. Jae Hwa Lee