2021, Fall semester PBL Report (Final)
Introductory Mathematics for Artificial Intelligence
http://matrix.skku.ac.kr/intro-math4ai/
Prof : Sang-Gu LEE
Due day: Dec. 5th, 2021 (11 AM) in HW box in I-campus
Name: Kim Daniil (김다니일), Pravas Giri, Park Junho (박준호), 김다솔, …
Major: Software, Global Economics,
Software, …
Student ID: 2021, 2014, 2021,…
e-mail/HP number: kim.daniil**, junho**, giri**/ … 010-***
Ch 1. Participation (10pt)
We
have learned ‘Basic Mathematics(행렬, 도함수, 통계)’
to understand and talk about the following concepts in 14 weeks.
1.
SVD(Singular Value Decomposition)
2.
GDM(Gradient Descent Method)
3.
Data and Covariance Matrix
4.
PCA(Principal Components Analysis)
5.
Rank Reduction and the role of SVD in PCA
6.
BP(Back-Propagation) algorithm in ML(Machine Learning) and ANN(Artificial
Neural Network) as we can see in http://matrix.skku.ac.kr/2021-Final-PBL-E/
We could practice the our codes in http://matrix.skku.ac.kr/KOFAC/
○ Math Lab Review (실습실)
9학년(중3) 수학 http://matrix.skku.ac.kr/9th-Grade/
10학년(고1) 수학 http://matrix.skku.ac.kr/10th-Grade/
11학년(고2) 수학 1
http://matrix.skku.ac.kr/11th-Grade-1/
11학년(고2) 수학 2
http://matrix.skku.ac.kr/11th-Grade-2/
12학년(고3) 미적분
http://matrix.skku.ac.kr/12th-Grade-1/
12학년(고3) 확률통계 http://matrix.skku.ac.kr/12th-Grade-2/
(1) State more than 10
Math Definitions and concepts what you learned in the first 14 weeks.
1.
Data can be represented as an Ordered
Pair (n-tuple). Example:
1. Extreme point: A point at
which a function has a local
maximum or minimum.
2. GDM(Gradient Descent Method):
The method finding the
optimal point(a extreme point),
as setting initial iterate, initial
learning rate and tolerance, and
then compute the next point
equal to previous point – learning rate * slope..
3. Joint probability: The
probability of which two or more random
variables which signify events
occuring at the same time.
4. Conditional Probability: The
probability that an event occurs
under the condition that the
other event already occured.
5. Bayes’ Theorem: A conditional probability explained when we
know the probabilities of the
events having reversed priority.
6. Covariance: the measurement
of distances of two other
variables from the mean.
7. PCA(Principal Components
Analysis): Express information as
covariance matrix to reduce
dimension of data as negligible
terms.
8. ANN(Artificial Neural
Network): System having hidden layers as
operating weight matrix from
input to output.
9. Sigmoid Activation Function: , which replaces
‘Heaviside function(step function)’ for some analytic advantages.
10. BP(Back-Propagation):
Updating method for weights matrix
and activation function as
errors between predicted and observed value adjusting
|
Name |
Height (cm) |
Weight (kg) |
Age |
Data Representation (4-tuple) |
|
Kim |
160 |
80 |
19 |
(160, 80, 19) |
|
Lee |
170 |
70 |
27 |
(170, 70, 27) |
|
Park |
180 |
56 |
30 |
(180, 56, 30) |
1.
Each number is called a component
of the data.
2.
Vector
is a line (arrow), that has a starting point – the origin, and the endpoint.
3.
The components of the vector are 
numbers, which are called scalars.
4.
Data can also be represented as a
rectangular (2D) array (grid), that is called Matrix.
5. Tensors are just “data containers”. A 0-dim Tensor is a single number, a 1-dim Tensor is a Vector, a 2-dim Tensor is a Matrix and so on.
6.
Diagonal Matrix. A Diagonal Matrix is a matrix, in which the entries
outside the main diagonal are all 0.
7.
Identity Matrix. An Identity Matrix 
is
a square matrix, that has 1s along the main diagonal and 0s for all other
entries.
8.
Triangular Matrix. A Triangular Matrix is a square matrix, where all the
entries above (lower triangular) or below (upper triangular) the main diagonal
are 0s.
– lower triangular
– upper triangular
9.
Symmetric Matrix. A Symmetric Matrix is a square matrix that is equal to
its transpose.
10.
Inverse Matrix. An Inverse Matrix of an 
matrix A is an matrix 
, such that 
. denoted as 
.
11.
Singular Matrix is a matrix that cannot be inverted.
12.
Transposed Matrix is a flipped version of the initial matrix. Basically, a matrix
with rows and columns swapped.
13. The
determinant is a special number that can be calculated from a
matrix that helps us find the inverse of a matrix and the solutions of the
system of equations.
14.
Coefficient matrix is a matrix, made up of coefficients from the equations in a system
of equations.
15.
Variable matrix is a 
matrix that is made up of
unknown variables in the system of equations.
16.
Constant matrix is a matrix made up of constants in the system of equations.
17.
Augmented matrix is a result matrix, that we get by merging multiple other matrices.
18.
ERO (Elementary Row
Operations) are operations that can be
performed in matrix: Row swap, Scalar multiplication, Row sum.
19.
REF (Row Echelon Form) is a matrix form, where
o The row vectors, all entries of which are zeroes should be at the very bottom of the matrix.
o 
Each pivot should be in a
column strictly to the right of the pivots, occurring in the rows above it.
20. Matrix
Pivot is the first
non-zero entry of a particular row.
21. RREF
(Reduced Row Echelon Form) is a matrix
in REF form, where
o
All pivots are 1s.
o
And the entries above pivots
are all 0s.
22. Free
Variables are located in the columns that does not
have a pivot in REF form, and they are basically variables that can take any
number.
23. Gauss-Jordan
Elimination - method of matrix system of equations
solution by performing ERO, to get the identity matrix.
24. Cramer’s
Rule – method of matrix system of equations
solution by finding the determinants ratio.
25. Classification is a process of
identifying to which category a new data belongs, based on the data
characteristics.
26. The distance between two points is defined by the Euclidean
Distance.
27. Norm is the size/length of a vector.
28. Inner product (dot product) is the
sum of the multiplication of the same indexed numbers in two ordered tuples.
Example: 
29. Cosine theorem:
30. Normalized vector is a vector, whose
length/norm equals to 1, basically making a unit vector out of
it.
31. Unit Vector is a vector with length
= 1.
32. Eigenvector is a vector which
direction remains unchanged when a linear transformation is applied to it.
33. Eigenvalue is a value that can be
found from the following formula: 
, where 
is a square matrix, 
is an Eigen Vector and 
is an Eigen Value.
34. The Least-Squares method is used to find a straight line (the least-squares line, the best fit line) that has the minimal distance to each data point.
35. A projection is the transformation of points and lines in one plane onto another plane. The example of projection is shadow.
36. Curve fitting is a process of
finding the best fit curve (a quadratic approximation) to describe an array of
data.
37. LU Decomposition is a matrix
decomposition that results a product of the Lower triangular matrix
and the Upper triangular matrix. 
38. Permutation Matrix is a matrix 
, that shows all the changes in the row positions of the initial
matrix .
39. QR Decomposition is a matrix
decomposition that makes finding the least-squares solution easier. For the QR
Decomposition the following equation is also true: 
40. SVD Decomposition helps us to reduce the size of the original matrix, hence reducing the amount of computational power required. We also need to keep in mind that SVD always exists for any rectangular or square matrix.
41. Kernel. A Kernel of a
matrix is a matrix 
, such that 
.
42. Tangent (касательная). A Tangent is a straight line that only “touches” a function at only 1 point.
43. Limit. The limit of a
function 
is the value that the function approaches as its argument approaches 
.
44. Slope. A Slope is a number that describes both the direction and the steepness of the function.
45. Continuity. A function is Continuous
at a point , if it is defined at this point.
46. Derivative. The derivative of
a function 
at a number , denoted by 
, is the instantaneous rate of change of 
with respect to , when 
.
47. Differentiation. Differentiation is the process of finding the derivative of a function.
48. Fermat’s Theorem: If has a local maximum
or minimum at and if 
exists, then 
.
49. Gradient Descent Method/Algorithm is an optimization algorithm to find the minimum of a complex function.
50. Factorial of 
is a recursive product of all
positive integers
. 
. Note: 
.
51. Permutations is a number of ways, we
can choose 
elements out of elements, with the
consideration of their order. There are two formulas:
1. Without Repetitions: (all chosen elements are distinct)
2. With Repetitions: (all chosen elements are not necessarily distinct)
52. Combinations is a number of ways, we
can choose elements out of elements, but without
the consideration of their order. There are two formulas:
1. Without Repetitions: (all chosen elements are distinct)
2. With Repetitions: (all chosen elements are not necessarily distinct)
53. Sample Space or Probability Space is a set of possible outcomes.
54. Probability is the branch of mathematics concerning numerical descriptions of how likely an event is to occur.
55. Mathematical probability is:
56. Conditional probability is defined
as the probability of an event or outcome occurring, based on the occurrence of
a previous event or outcome. It is calculated by multiplying the probability of
the prior event by the updated probability of the succeeding, or conditional
event:
57. Bayes’ Theorem describes the probability of the occurrence of a particular event, considering the conditions, related to this event.
58. Discrete Probability Distribution describes the probability of occurrence of each value of a discrete (or in other words, explicitly defined) random variable.
59. Probability Mass Function is a function that gives the probability that a discrete random variable is exactly equal to some value.
60. Continuous probability distribution
describes the probability of occurrence of each value of a continuous (or in
other words, not explicitly defined or conditionally defined, e.g., 
) random variable.
61. Probability Density Function is a function that is providing a relative likelihood that the value of the continuous random variable would be close to a particular sample.
62. Expectation of a random variable is
an average value, that we can get from random variable. It is also called a mean
value.
63. Variance of a random variable is the
spread of a set of data, in relation to their average value.
64. Standard Deviation is the square
root of its variance.
65. Standardized Random Variables are
similar to “Normalization” (like vectors). The expected value of a Standardized
Random Variable is always 0, and its variance is 1. Standardizing makes it
easier to compare variables of different types and units:
66. Joint Probability Distribution shows a probability distribution of multiple variables and their relationship.
67. Marginal Density Function – a
density function of a one particular variable.
68. Covariance is a measure of the joint
variability of two random variables.
69. Correlation Coefficient is the
measure of relationship strength between the two variables.
70. Covariance Matrix is a square matrix
with the variance in the main diagonal and all covariances in non-diagonal
entries. Any covariance matrix is symmetric and positive semi-definite. It
visualizes the data distribution.
71. Poisson Distribution – discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event.
72. Bernoulli Trial – random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is conducted.
73. Central Limit Theorem – when independent random variables are summed up, their properly normalized sum tends toward a normal distribution even if the original variables themselves are not normally distributed.
74. Hypergeometric Distribution – probability of k successes (random draws for which the object drawn has a specified feature) in n draws, without replacement, from a finite population of size N that contains exactly K objects with that feature, wherein each draw is either a success or a failure.
75. Gamma Distribution – two-parameter family of continuous probability distributions.
76. Dimensionality Reduction Techniques – algorithms that transform the data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains meaningful properties of the original data.
77. Principal Component Analysis (PCA) is one of the most widely used dimensionality reduction techniques. The PCA tries to combine existing variables to define the new variables, called Principal Components, while minimizing the information loss.
78. Principal Components represent the directions of the data that explain a maximal amount of variance.
79. Machine Learning (ML) is a subfield of AI that studies Computer Algorithms, that are capable of self-improvement through experiencing sample data, without direct programmer’s interference.
80. Training Data – a sample data, used to teach Machine Learning.
81. Artificial Neural Networks (ANN) are
a commonly used, specific class of ML algorithms. ANNs
are modeled on the human brain, in which thousands or millions of processing
nodes, called “neurons”, are interconnected and organized into
layers.
ANNs always has an Input Layer and an Output
Layer.
82. Sometimes, Neural Networks can have a hidden layer in between I and O Layers, where the weights are adjusted. Such Neural Networks are called “Deep Learning”.
83. Backpropagation – the process of weight adjustment. The flow of the Backpropagation algorithm is as follows:
1. Data Division (~80% for learning and the rest ~20% for the test cases)
2. Weights Randomization (initial setup of random weights)
3. Matrices and Activation Functions setup
4. Standard Calculation (using mentioned formulas)
5. Calculate Error (compare with the expected result)
6. Use Gradient Descent Method, to adjust the weights, so that the error is minimal.
7. Repeat 4-6.
8. Stop when result is satisfactory (the error is almost negligible or non-existent).
84. Weight or 
decides how influential the input
will be on the output.
85. Bias or is a constant that
helps us fit our model for the given data.
86. Activation Function of a node defines the output of that node given an input or set of inputs.
87. Linear Activation Function –
simplest Activation Function. If we use a linear activation function in a
neural network, then this model can only learn linearly separable problems:
or
88. Non-Linear Activation Functions – the most popular type activation functions in modern ANNs. They allow ANNs to easily learn a non-linearly separable problem. One such function is a Sigmoid Function.
89.
Sigmoid Function – a special form of the logistic function: 
. With the addition of just one hidden layer and this activation
function in it, neural network can learn complex decision functions.
90. MNIST database (Modified
National Institute of Standards and Technology database) is a large
collection of handwritten digits. It has a training set of 60,000 examples, and
a test set of 10,000 examples.
All the examples are monochrome, centered and fully normalized (no distortion
or skew) 28x28 images.
91. t-SNE (t-Distributed Stochastic Neighbor Embedding)– a technique for dimensionality reduction (like PCA) that is particularly well suited for the visualization of high-dimensional datasets.
1.
Functions and its Graph
The relationship in which the value
of the two variables x and y is uniquely determined according to the value is
called a function 
and is
expressed as 
.
2.
Vector
A vector is an abstract data type
used to represent properties that have both direction and magnitude. Vectors
are commonly used to represent movement. For example, vector can be used to
represent the distance between the two points.
For this above figure, whose starting
point is A, ending point is B the vector will be 
.
Vector operations:
a.
Scalar multiplication: For scalar 
and vector 
.
b.
Vector addition: For two
vectors 
and 
.
3.
Matrix Operations
Matrix is a rectangular array of
numbers or polynomials arranged in rows and columns.
a.
Scalar multiplication
The term scalar multiplication refers
to the product of a real number and a matrix. In a scalar multiplication, each
entry in the matrix is multiplied by the given scalar. For scalar ,
b.
Matrix addition and subtraction
A matrix can only be added or
subtracted from another matrix if they have same dimensions.
c.
Matrix multiplication
For matrix multiplication, the number
of columns in the first matrix must be equal to the number of rows in the
second matrix.
d.
Transpose of matrix
The transpose of a matrix is an
operator which flips a matrix over its diagonal which means it switches the row
and column indices of the matrix A by producing another matrix which is denoted
by 
.
e.
Diagonal matrix
A matrix in which the entries outside
the main diagonal are all zero.
f.
Identity matrix
A square matrix that has 1’s along
the diagonal and all the remaining are zero.
g.
Triangular matrix
It is a special type of square
matrix. A square matrix having all the entries above main diagonal zero is
called lower triangular matrix. Similarly, a square matrix having all the
entries below main diagonal zero is called upper triangular matrix.
h.
Symmetric matrix
A symmetric matrix is a square matrix
which is equal to its transpose. 
i.
Inverse matrix
A square matrix with an associated matrix 
such that multiplied by and multiplied by both equal the identity matrix.
Properties of inverse matrix:
1. If A is
nonsingular, then so is A-1 and
(A-1) -1
= A
2. If A and B are
nonsingular matrices, then AB is nonsingular and
(AB) -1
= B-1A-1
-1
3. If A is
nonsingular then
(AT) -1
= (A -1)T
4. If A and B are
matrices with
AB = In
then A and B are inverses of each other.
4.
Tensor
Tensor is a container, which is a
storage that can put data together since most of them deal with numeric data.
We can think of a 1-dim Tensor as a Vector, a 2-dim tensor as a Matrix, and a
matrix’s generalized form as a tensor. However, tensor and matrix are
different. A matrix is just a container for entries and it doesn’t change if
any change occurs in the system, whereas tensor is an entity in the system that
interacts with other entities in a system an changes its values when other
values change.
5.
Data Similarity
Data similarity is calculating how
close or similar the data is in each category.
a.
Distance
The distance
between the two points 
and 
is defined by
This is also
called ‘Euclidean Distance’.
b.
Norm
For a vector 
the size of 
is called a norm.
In case of two vectors 
and 
the norm is the distance between two points and .
6.
Cosine Similarity
The cosine similarity can be
calculated by the angle between two vectors which can be defined with an inner
product and measuring the similarity with the cosine value 
of by using an inner product.
a.
Inner Product
The inner product between two vectors
and 
is
b.
Angle between two vectors
The angle between two vectors is given by
7.
Gauss-Jordan Elimination
This is an algorithm for solving
system of linear equations. While solving the final form of the augmented
matrix on the left side of the equation will become identity matrix.
We can solve a equation following
Elementary Row Operations (ERO).
a.
Exchange two equations. 
b.
Multiply a row by a nonzero real
number. 
c.
Add a nonzero multiple of a row to
another row. 
While finding RREF of an augmented
matrix 
.
If the right side of the solution of
given linear system of equations is not 0 then it has no solutions.
If it is 0, we should check if the
solution is identity matrix. If yes, it has unique solutions.
However, if there is free variable
then it will have infinitely many solutions.
8.
Singular Value Decomposition
Singular value decomposition of a
matrix is a factorization of that matrix into three matrices 
where U is an orthogonal matrix, V is an
orthogonal matrix, and 
is a rectangular diagonal matrix with
non-negative real numbers on the main diagonal.
The key point of SVD is it exists for
any sort of rectangular or square matrices.
9.
Limit of functions
The limit of a function is a
fundamental concept in calculus and analysis concerning the behavior of that
function near a particular input.
This only works when 
. Because
when 
the function cannot be defined.
10.
Derivatives and Differentiation
a.
Derivatives
When 
is
differentiable at every point x in an interval, 
is called differentiable in that interval. In
this case, the derivative at that point is called the derivative of 
at 
. This is
denoted by: 
etc.
b.
Differentiation.
The derivative of function is called differentiation of which can be found by 
=
.
c.
Tangent line
The tangent line can be found by the
following formula:
Where,
m=slope of the tangent line
)= The point
of slope
11.
Local maximum and Local Minimum
When a function 
has 
at c in the domain with 
.
- if
, then 
is a local maximum. - if
, then is local minimum.
12. Absolute Maximum
It is point where a function obtains its greatest
possible value.
13. Absolute Minimum
It is a point where a function obtains its smallest
possible value.
14. Gradient Descent Method (GDM)
This is a method which is used when we need to find
the critical value of a complicated function. The basic idea is to find the slope of the
function (derivative), move it toward downhill (learning rate), and repeat it
until it reaches the extreme value.
15. Permutations
Rules of counting where order
matter.
16. Combinations
Rules of counting where order doesn’t matter.
17. Conditional Probability
Probability that a event B occurs given that an
event A has already occurred.
18. Bayes Theorem
It describes the probability of an event based on
prior knowledge of conditions related to the event.
19. Prior Probability
Probability computed before the collection of
new data.
20. Posterior Probability
Revised or updated probability after taking new
information into considerations.
21. Expectation
The expectation of a random variable is an average
value for probabilistic events, which is the sum of products of a value
obtained by each event and each event's probability.
For discrete random
variable X:
For continuous
random variable X:
22. Variance
It is a measure of the dispersion of numbers
(data), which indicates how far a set of numbers (data) are spread out from
their average value.
For discrete random variable X:
For continuous random variable X:
23. Standard deviation
It defined as a square root of variance.
For discrete random variable X:
For continuous random variable X:
24. Covariance matrix
It is created by using the variance and covariance
of each random variable.
The covariance matrix is a square matrix with the
variance in the main diagonal and covariance of two variables in off-diagonal.
It plays an important role in dimension
reduction, effectively reducing the dimension while maintaining the
distribution of high-dimensional data as much as possible.
25. Principle Component Analysis (PCA)
It is difficult to analyze multi-dimensional data
due to its large number. In this case it would be much easy to extract and only
use some data that seems explain the data. However, data selection in this
manner might not purely reflect the attributes of the data. Therefore, PCA is
needed. PCA is process of analysis used to determine which feature of the
element should be mainly used.
26. Artificial Neural Network (ANN)
ANN refers to Artificial Neural Network. A
neural network is like a neuron of our nervous system. It receives multiple
input signals and gives one output signal.
Artificial neurons receive input data from other
artificial neurons or from outside. The input artificial neurons combines with
given weights and creates an output by linearly combining a single value with
the weight and adding the bias. However, incase of two outputs it can be
expressed using matrix product. We use 
as this
is the format in statistics or artificial intelligence. Then we substitute
the value we obtained through linear combination in activation function and
output data is obtained.
27. Backpropagation
The process of creating and updating weights
between hidden layers in a deep learning is called a backpropagation.
An input is entered and the weight of artificial
neural network is set randomly. When the hidden layer receives input and
computes with the weight then we obtain output through the activation function.
There will be an error between the predicted output and obtained output. We
then use gradient descent method to minimize the error. Now we modify the
weights. After correcting the weights the process is repeated again until the
overall error is minimized. After we find the optimal solution, we have the
neural network model.
This whole process is operated within the hidden
layers.
(2) State more than 10 things that you
know/can/find ... after you studied the
first 14 weeks.
1.
I can
solve different polynomial functions and plot them using code.
2.
I can
generate a matrix according to the dimension I wish. Apply all the matrix
operations on the matrix and find its transpose and inverse. I learnt that
3.
I can
find the data similarity by calculating the distance between the data’s using
norm or by calculating the angle between the data’s using angle between two
vectors. I am able to calculate the data similarity by deciding whether to
calculate the distance or angle according to the situation. In case of
direction, I must find the angle to determine the data similarity as the length
of the data’s maybe different and AI may conclude that the two data’s are not
well related.
4.
I know
how to find the solutions from the system of linear equations using
Gauss-Jordan elimination. I can also determine whether the equation has no
solution, unique solution of infinitely many solutions by finding the Reduced
Row Echelon Form (RREF) of the augmented matrix.
5.
While
finding the solution of equation if we use the code “solve_right” instead of
“RREF” it will show an error for the equation if the equation has no solution.
So, it will be better to use “RREF” while finding the solution of the equation.
6.
When the
coefficient matrix can be expressed as a triangular matrix, the computation
cost for solving a linear system of equations can be reduced dramatically.
7.
I can
compute the limit of a function and find the derivate by simply using the code.
8.
I can
find the local maximum, local minimum, absolute maximum, absolute minimum of a
function using the code provided.
9.
I can
use the code of GDM provided to calculate the critical point of a complicated
function.
10.
I can
calculate expected value, the variance and the standard deviation of a
continuous random variable.
11.
I can
find a covariance matrix from a sample data.
12.
I
understand artificial neural network and backpropagation.
13.
Have
basic idea regarding how MNIST is used to utilize ANN.
1.
Vector Translation
(addition or subtraction)
2.
Vector Scalar
multiplication
3.
Vector Dot product.
4.
Matrix Translation
(addition or subtraction)
5.
Matrix Scalar
multiplication
6.
Matrix Dot product.
7.
Matrix inversion.
8.
Solve
equations using Cramer’s Rule.
9.
Solve
equations using Gauss-Jordan Elimination.
10.
Find
REF.
11.
Find
RREF.
12.
Find
Euclidean Distance. 
13.
Vector Norm: 
14.
Solve
the Least-Squares problem.
15.
Find
Vector Projection.
16.
Solve
the Curve-Fitting problem.
17.
Make
LU Decomposition.
18.
Make
QR Decomposition.
19.
Make
SVD Decomposition.
20.
Find
Limits.
21.
Find
Derivative.
22.
Find
local maximum and minimum.
23.
Find
absolute maximum and minimum.
24.
Know
how to use Fermat’s Theorem.
25.
Understand
Gradient Descent Method and its limitations.
26.
Find
Factorial.
27.
Find
number of Permutations.
28.
Find
number of Combinations.
29.
Know
what is Probability Theory.
30.
Find
Mathematical Probability and know its formula.
31.
Find
Statistical Probability.
32.
Know
what is Conditional Probability.
33.
Understand
Discrete Probability Distribution and Probability Mass Function.
34.
Understand
Continuous Probability Distribution and Probability Density Function.
35.
Understand
Conditional Probability.
36.
Use
Bayes’ Theorem.
37.
Find
Expectation.
38.
Find
Variance.
39.
Find
Standard Deviation.
40.
Standardize
Random Variables.
41.
Understand
Joint Probability Distribution.
42.
Understand
Marginal Density Function.
43.
Understand
Covariance.
44.
Understand
Correlation Coefficient.
45.
Understand
Covariance Matrix.
46.
Understand
Dimensionality Reduction Techniques.
47.
Know
Principal Component Analysis.
48.
Understand
Principal Components.
49.
Know
what Machine Learning is.
50.
Understand
how ANNs work.
51.
Understand
how Deep Learning algorithms work.
52.
Understand
Backpropagation. Understand Activation Functions.
53.
Know
Linear Activation Function.
54.
Know
Non-Linear Activation Function.
55.
Know
about MNIST database.
56.
Understand
how digit recognition ANN works.
57.
Know
about t-SNE.
58.
Know
how math is used in ANNs.
59.
Understand
how Object Detection ANNs work.
60.
Can
convert image to Grayscale.
61.
Can
use Canny Edge detection algorithms.
62.
Know
some deskewing techniques.
Ch 2.
Participation and Project Proposal (20pt)
1. (10pts) Briefly describe your participation/contributions for
yourself and your classmates in Q&A of our "Introductory Math4AI"
class.
(1) # of your weekly QnA participation (Sat->Fri):
(1) Week 1 : 5 , Week 2: 6 , Week 3: 7 , Week 4: 6
(2) Week 5 : 5 , Week 6: 5 , Week 7: 4 , Week 8: 7 Exam Week
(3) Week 9 : 4 , Week 10: 6 , Week 11: 7 , Week 12: 13
(4) Week 13: 2 , Week 14: 6 , Week 15: Exam Week
Total number of sessions (Q: 15 , A: 47 , Others: 21 )
Number of I-campus online class attendances: ( 35 / 35 )
Off-line and WebEx OH attendance: ( 7 / 7 )
(2) Now I do understand the concepts below, and can
explain the meaning, and can compute (by hand or by the Code - Sage/R/python).
1.
Code to
solve the equation and plot its graph.
2.
Code for
Vector operations.
3.
Code for
matrix operations
4.
Code for
data similarity
5.
Finding
solution of equation and least square solution
6.
Finding
SVD of a matrix
7.
Finding
limit and derivative
8.
Finding
local maximum, local minimum, absolute maximum, absolute minimum.
9.
Generating
covariance matrix from a sample data.
10. Monty Hall Problem
11. Use of SVD in PCA.
12. Artificial Neural Network and Backpropagation
13. Handwritings number detection
(3) State your meaningful Comment/Answer/Discussions
in QnA.
I have summarized all our
WebEx meetings in QnA so that the classmates who missed the meeting could know
about the activities we did in the WebEx meetings. I was able to help my classmate
Park Jun Ho to understand regarding how to find out whether a system of linear
equation has no solution, unique solution, or infinitely many solutions.
Lastly, I have thanked my classmates for uploading their work on QnA as I was
able to review them and those answers helped me understand the topic even more
clearly. I was able to make a correction in week 11 open problem 2 as Mr. Kim
Daniel found out an error.
김다니일(2021####39)November
12, 4:00 PM
I
would like to point out, that I actually found that np.array(X.row(row)) is a
redundant conversion. You can just replace it with X[row], and everything would
work out fine with a higher performance. Although, it is not really critical,
since on small matrices <20x20, I did not really notice the big difference,
I only noticed it with a very large matrix, so I do not think that it is
something to worry about, just wanted to mention.
기리프라바스(2021####24)November
15, 11:56 AM
Thank
you for the suggestion. I will remember it as our program should be accurate as
well as fast.
[Final OK by SGLee] Finalized by Daniil Kim, 기리프라바스, 박준호, 김범수 Cramer's
Rule Explanation (박준호)
Author : 김다니일(2021####39) Date :
September 26, 11:50 AM [Final OK by SGLee] Finilized by Daniil Kim, 기리프라바스, 박준호, 김범수 Cramer's Rule Explanation (박준호).
As an answer to: 박준호 question.
W12 (Daniil Kim). Summary. Chapter 4 Open Problem 2
Author : 김다니일(2021####39) Date :
November 18, 8:05 PM
Very useful PCA decomposition experience, and also got more
acknowledged with Google Jupyter Notebook, NumPy, Matplotlib and Scipy.
W14 (Daniil Kim). Summary. Chapter 4 Open Problems: 6, 7.
Author : 김다니일(2021####39) Date :
December 2, 6:03 PM
Since, I did do a simple sudoku solver CV program before, it was
very interesting to dig deeper into the algorithms that I used and how they
work. And I feel like the information could be helpful.
W11 (Daniil Kim). Chapter 3 Open Problems: 2 Fixed. Thanks to 박준호.
Author : 김다니일(2021####39) Date :
November 29, 9:16 AM
I feel like it was a very good experience, trying to figure out
what is wrong with the algorithm, and finally discovering the error. Also
enjoyed cooperating with 박준호.
2. (10 points) [Your possible Team] Project proposal :
Team Project (idea) [Tentative Title] Team Project (Tentative, just idea now)
Me
to You
This
is the name in a sense that the application sees the world and describes it to
the user. Here, me is the application and you is the user.
[Motivation]
I got motivation for this
project for my aim to develop something that would be for people that do not
have the ability to see. Just like how a guide dog guides the person throughout
their journey I want to be able to provide them a medium through which they
will be able to hear what they are seeing.
[Abstract]
Just like how google
assistant or siri responds to us with just a single command that is “Hello
google” and “Hello siri”, I want my application to be able to respond with a
single command and open the application. Then when the user points the camera
of the phone then the application should be able to describe it to the user in
the human speech. Along with this, I want the application to operate with the
voice command. In the long term, I wish that the application can be further
developed in a way that user would feel as if the things that are described by
the application are being seen by the user itself.
[Method of Research]
For this project, python
would be the most suitable language as it is a powerful data analysis tool and
has been popular in this field. It also has tensorflow, which is an open-source
library created specifically for machine learning and can be used for training
and inference of deep neural networks.
This will be a
qualitative research as it will take time to understand concepts, thoughts, and
experience it. This will help us to gather in-depth insights on topics that we
are not well understood.
I will be selecting
people and ask for their personal opinion regarding how they would want the
application to function. This will give me an idea about what are the things I
need to focus while developing this application. In order to make sure that the
application will be able to explain some basics thing in its early stage, the
application should be taught to distinguish between the images it sees and
describe it. For this as I have learned, for letter and alphabet recognition we
can use MNIST along with ANN. From MNIST we can provide numeric and
alphabetical images which can be used to make distinction and recognize the
input.
For the application to
respond to the user I need to select a person for a voice over. The main
participants for the applications are blind people. Having them as a
participant will help us point our weakness and areas that should be improved
in my application.
[Reference]
https://www.editage.com/insights/ko/node/14300
https://www.scribbr.com/methodology/qualitative-quantitative-research/
https://www.bairesdev.com/blog/top-6-languages-for-artificial-intelligence/
http://matrix.skku.ac.kr/2021-Final-PBL-E/
Team Project (idea) [Tentative Title] Crime Prevention and Analysis
System
[Motivation]
This year in USA alone
happened more than 20 incidents involving School Shootings, and judging by the
reports, the majority of them could have been prevented, since before these events,
criminals posted threating comments on their social media, as well as had a
suspicious behavior. And these incidents do not limit just by the School
Shootings, in 2019, in Japan there was a huge arson “KyoAni
incident”. A huge studio was set on fire, resulting a
lot of deaths and injured innocent people, and according to the research, prior
to that incident, suspect posted threats and sent threatening emails to the
involved studio. So my proposal is an AI that will analyze social media for
violent keywords (kill, murder, and so on). And in case of multiple occurrences
or suspicious internet traffic of a suspect (visiting gun selling sites, using
suspicious VPN, such as TOR), notify a corresponding agency for the further
investigation. Later on, this system can be even used to prevent the people’s
suicide intentions.
[Abstract]
In order for it to analyze the
social media, we can use OpenAI GPT-3 language processor, so the AI can get at
least a slight understanding of the context, where particular keywords are
being used, and then by the similarity comparison in different values, it can
calculate a rough approximation of a chance that a person can commit crime. The
second one can be camera footage analysis. For this one we do not necessarily
need a very advanced analysis, we just need a rough behavior evaluation: a
person really often visiting a particular place, stalking another person and so
on.
And as mentioned before, with
all the retrieved data, AI will try to calculate a possibility of a person
committing crime, and then notify a corresponding agency for the further
investigation.
[Method of Research]
As a machine learning material,
we can feed AI past incidents and the data backups from the culprit’s social
media accounts or camera recordings. Furthermore, it can be launched as a test
and be improved on the actual crime cases.
[Reference]
https://openai.com/blog/openai-api/
https://en.wikipedia.org/wiki/GPT-3
https://openai.com/blog/openai-codex/
https://www.mitsubishielectric.com/en/about/rd/research/highlights/automation/ai.html
https://www.edweek.org/leadership/school-shootings-this-year-how-many-and-where/2021/03
http://matrix.skku.ac.kr/2021-Final-PBL-E/
[Progress]
Only two weeks ago, Google completely
released GPT-3 to public, and its still beta. Because of that, I did not have
enough time to fully understand API and all its capabilities, however, as a
proof-of-concept, I found an example on: https://beta.openai.com/playground/p/default-mood-color. And there we can see:
When I input bad or
words of anger, like “violence”, 
it outputs red colors: #FF0000 in our case, which is a hex representation of
RGB(255, 0, 0) -> a completely red color.
But, when I input
good or neutral words, like “good”,
it outputs neutral colors: #9e9e9e in our case, which is a hex representation
of RGB(158, 158, 158) -> a grey neutral color.
With that, it
proves, that GPT-3 can classify words and understand the meaning behind them
(something bad, something good or something neutral). So, it means that it can
even classify the context of whole sentences, and with that we can measure the
social media post’s “aggressiveness” levels.
Also, in order to
measure “aggressiveness”, we can use Object Recognition AIs on attached with
post images. It could be YOLO for speed or some easy-to-use ANNs like python
ImageAI:
By performing Object Recognition, we can find potential harmful things on
attached photos, like “weapons”, “guns”, etc.
And finally, by plugging the outputs
of both ANNs in another “connector” ANN, we can measure the overall “aggressiveness” level of a post and potential “crime” level of it.
P.S. The biggest problem here lies in the datasets. From my research, the amount of crimes happening after some social media posts, is still low, since it started relatively recently. So, it would be hard to give enough “training data” to the “connector” ANN.
Ch 3. Self
Evaluation 1. (10pt)
|
Subject |
Introductory Math4AI |
Major |
Software |
||||
|
Name/ID |
Kim
Daniil (2021) |
Year |
Freshman |
||||
|
Self-Checking |
|||||||
|
Activity |
Excellent |
Good |
Fair |
||||
|
1. |
I have contributed to generate ideas and facts needed to resolve the issue. |
● |
|
|
|||
|
2. |
I proposed learning issues associated with learning. |
|
● |
|
|||
|
3. |
When I study alone, I used a variety of learning materials. |
● |
|
|
|||
|
4. |
I provide new information and knowledge in this class. |
● |
|
|
|||
|
5. |
I was actively involved in the discussions. And I provided a lot of questions in order to understand these discussions. |
|
● |
|
|||
|
6. |
I have made a contribution to the learning activities for our class. |
● |
|
|
|||
※ Please record the following items by considering your learning process.
1. Do you understand the most of contents of this
learning process?
Answer : Yes.
2. What kind of learning materials have you used
to study?
Answer:
http://matrix.skku.ac.kr/intro-math4ai/W1/
http://matrix.skku.ac.kr/intro-math4ai/W2/
http://matrix.skku.ac.kr/intro-math4ai/W3/
http://matrix.skku.ac.kr/intro-math4ai/W4/
http://matrix.skku.ac.kr/intro-math4ai/W5/
http://matrix.skku.ac.kr/intro-math4ai/W6/
http://matrix.skku.ac.kr/intro-math4ai/W7/
http://matrix.skku.ac.kr/intro-math4ai/W8/
http://matrix.skku.ac.kr/intro-math4ai/W9/
http://matrix.skku.ac.kr/intro-math4ai/W10/
http://matrix.skku.ac.kr/intro-math4ai/W11/
http://matrix.skku.ac.kr/intro-math4ai/W12/
http://matrix.skku.ac.kr/intro-math4ai/W13/
http://matrix.skku.ac.kr/intro-math4ai/W14/
http://matrix.skku.ac.kr/LinearAlgebra.htm
https://tutorial.math.lamar.edu/classes/alg/augmentedmatrix.aspx
https://www.mathsisfun.com/algebra/systems-linear-equations-matrices.html
https://en.wikipedia.org/wiki/Augmented_matrix
https://virtualnerd.com/algebra-2/matrices/determinants-cramers-rule/cramers-rule/coefficient
https://www.math.ucdavis.edu/~linear/old/notes3.pdf
https://stattrek.com/matrix-algebra/matrix-rank.aspx
https://www.pyimagesearch.com/2020/08/10/opencv-sudoku-solver-and-ocr/
https://en.wikipedia.org/wiki/Canny_edge_detector
https://en.wikipedia.org/wiki/Hough_transform
https://en.wikipedia.org/wiki/Activation_function
https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6
https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/
https://www.statisticshowto.com/probability-and-statistics/monty-hall-problem/
https://towardsdatascience.com/monty-hall-problem-solution-using-bayes-theorem-cb1d6fbc0c9e
And a number of Russian Linear Algebra mathematics books.
3. What did you learn through the learning
activities of this course?
Solving equations and plotting graphs through codes. Generating matrix and vectors. Using different formulas to calculate distance and angle between two data’s. Finding the transpose, inverse and checking whether the matrix is invertible or not. Learning about LQ, QR, and SVD decomposition. Finally, finding the limit, derivative and plotting the graph of the function using codes.
Finding local maximum, local minimum, absolute maximum, absolute minimum, using GDM to find critical value of a complex function, calculating expected value, the variance and the standard deviation of a continuous random variable, finding covariance matrix, basic knowledge of PCA, ANN and backpropagation.
Basic mathematics, that will help to build a foundation for the future AI, including the best ways to solve a system of equations and finding an optimal solution.
Got to know how ANNs actually “learn” (backpropagation), and found out a lot about the back-end of ANNs: activation functions, hidden layers, etc.
Also, the math that I learned on this course, helped me in my other courses (Calculus and Discrete Math).
4. What have you learned from the other
colleagues?
From my colleagues I was able to get a clear understanding on how to determine whether A^T*A is invertible or not. From seeing their answers on the QnA, I got to see different kinds of codes and their different approach for the solution. I was able to rely on my colleagues answer in case of any confusion.
박준호 and Kim Daniel have helped by pointing out mistakes which has helped me to make corrections.
기리프라바스constantly uploads a very interesting information.
김다솔 uploaded very useful information regarding PCA which helped me to get better understanding.
Due to their participation there are various materials for a same topic which provides me an opportunity to learn better studying with different materials.
박준호 made a pretty interesting proposition about the Sage -> Excel Import/Export and I had fun exploring it and implementing it in Python and Java.
박준호actually helped me a lot a couple of times, by pointing out on mistakes in my code or understanding.
기리프라바스constantly uploads a very interesting information.
김다솔helped a lot with PCA, and thanks to his code, I managed to perform it myself successfully.
5. Self-Evaluation for Q/A Activities
My expected score: ( * ) / 10
6.
Evaluation for other students
Kim Daniel, 김다솔 and 박준호 have
been very hardworking classmates. They have provided very useful information
in their assignment. They have participated in QnA diligently.
Both 박준호 and Pravas Giri are very active and
interesting conversationalists. Besides, I think they are also very active
and have a good sense of responsibility. I genuinely respect them and count
as very diligent students.
Although late, but 김다솔also became very active,
and I also respect his commitment, since he already managed to catch up with
other students.
I also like how observant is 박준호. He constantly points out
on mistakes and helps to polish our skills.
1. Do you
understand the most of contents of this learning process?
기초적인 인공지능 수학에 대해 대부분 이해하였습니다.
2. What kind of
learning materials have you used to study?
온라인 교안, 강의과 더불어 전공
교재도 일부 참조하였습니다.
3. What did you
learn through the learning activities of this course?
다른 학생들과 소통하면서 미처 생각하지 못했던 소재에 대해 생각해볼수 있었습니다.
4. What have
you learned from the other colleagues?
김다니일 학우 덕분에 다양한 코드들을 접할 수 있었고, Pravas Giri 학우가 14주차에 올린 QnA를 통해 목소리 인식 기능에
사용되는 ANN에 대해서 알 수 있었습니다.
5. Self-Evaluation for Q/A
Activities
My expected score: ( 6.5 )
/ 10
6. Evaluation for other
students
김다니일: 10점
Pravas Giri: 10점
김다솔: 10점
Self
Evaluation 2. (10pt)
|
Subject |
Introductory Math4AI |
Major |
Software |
|||||
|
Name/ID |
Kim Daniil (2021314939) |
|
|
|||||
|
Evaluation Items |
Strongly disagree |
Disagree |
Mostly disagree |
Mostly agree |
Agree |
Strongly agree |
||
|
|
1. I participated actively in both, online and offline classes. |
|
|
|
|
|
✓ |
|
|
|
2. I participated actively on a Q&A activity. |
|
|
|
|
✓ |
|
|
|
|
3. My question and replies made on Q&A are relevant. |
|
|
|
|
✓ |
|
|
|
|
4. Information provided by my activity was useful for other students in the class. |
|
|
|
|
|
✓ |
|
|
|
5. I enthusiastically took into the consideration other students’ opinions or point of view. |
|
|
|
|
✓ |
|
|
|
|
6. I contributed to class by participating on Q&A discussions. |
|
|
|
|
|
✓ |
|
|
|
7. I am enthusiastic about taking other class with the same students (I am taking Discrete Mathematics.) |
|
|
|
|
|
✓ |
|
|
[Opinion] ► Satisfaction according to the Self-Evaluation I am glad I could make others understand
about “GAUSS JORDAN ELIMINATION”. I am happy that I was able to learn about
ANN, Backpropagation. I found these really interesting. I am glad that my “Cramer’s Rule” explanation was somehow helpful to
other students. I am also happy, that
according to comments, my summaries help other students to understand
material a bit better. ► Sorrow according to the Self-Evaluation I hope to
participate more in Q&A. Even though I
understand how particular things are used and have a decent grasp on how they
work, I do not have a very thorough understanding of some very hard concepts. I am sad that there were some problems
right before the submission. However, due to help of professor, I was able to
gather myself and complete the assignment. |
||||||||
► Satisfaction according
to the Self-Evaluation
다른 학우가 놓치고 지나갔던 부분들을 점검함으로써 더 정밀한 논의를 할 수 있게 했다는 점에서 만족하였습니다.
► Sorrow according to
the Self-Evaluation
중간고사와 대비해볼
때, 전반적인
참여정도가 저조하였던
것이 아쉬웠습니다.
특히 제때에 참여하였다면
더 활발하게 다른
학우와 의견을 나눌
수 있었을텐데 뒤늦게
참여하면서 feedback을
받기 어려웠던 것 같습니다.
Ch 4. PBL
Participation/Activity Part (30pt) [Midterm]
Your QnA HW solutions!! If you don’t have
enough, Do solve the followings.
W1
(Daniil Kim) Self-Introduction: Daniil Kim, Summary: Open Problem 4:
Self-Introduction:
My name is Daniil Kim (17 y.o.), I
am an English Track Freshman from the Uzbekistan. Since the very childhood
I was first curious about software development and later on got into the
Computer Science, today I know a couple of backend and frontend languages and
technologies, familiar with the popular frameworks.
My motivation for taking this
course was an interest in the AI as a whole and the one particular situation.
Once I found an interesting challenge on the internet about the creation of a
simple text recognition AI. So, I jumped right into it, and stumbled upon the
TensorFlow and the OpenCV. It was really hard at first, but after familiarizing
myself with these frameworks, I finally managed to get decent recognition
results. However, I was not satisfied with it. And the biggest reason was the
fact that I did not write it from scratch, and used the TensorFlow and the
OpenCV for all the hard AI and recognition math (image deskew, matrix
transformations, neural networks and so on). That is why I decided to take this
course, in order to get at least the basic understanding of the math behind the
AI, and perhaps, with further studies I hope to be able to contribute to the
source code of the TensorFlow on github.
Summary:
The first week, we got introduced
to the AI, got a thorough explanation of what exactly AI is, and even were
shown an examples of modern AI (AlphaGo, Car Auto-Parking systems, etc.).
Furthermore, Prof. Lee Sang-gu
told us about the SageMath and its possibilities. Basically, Sage is an
open-source program, that makes Python math easier. For example, in order to
draw a graph in Python, you first need to install it, then download all the
necessary libraries and their dependencies, including NumPy (for math) and
MatPlotLIB (for graph visualization), and then only after including them into
your program, you can start writing the actual code to draw the aforementioned
graph. However, you do not need to do any of these with SageMath, you can just
go to the website (SageMath Cell) and start writing the code using very
convenient functions such as "solve" or "plot". Something
that you can write on SageMath in mere seconds, would have taken you at least
15 minutes of your time on Python.
Furthermore, Professor explained
ways to solve basic polynomial equations, and even showed us how to approximate
roots of complex equations using SageMath.
Pravas Giri HW W1
Author : 기리프라바스(2021####24)Date : September
2, 1:08 AM
Count : 22
Motivation:
Hello, my name is Pravas Giri a freshman. I am from Nepal. Currently, I am
pursuing my Bachelor's Degree in Computer Science and Engineering.
Being interested in Artificial Intelligence, I have always thought of
pursuing my career in this field. AI has always fascinated me with its
capability to make human life more simpler. AI has been handling all the
boring, repetitive tasks and completes it with no error (if programmed
properly). Currently, the career in AI looks most promising in comparison to
other fields and to fulfill this wish has been my motivation behind choosing
this course.
Question:
In this course, are we going to solve all the equations through program or
do we have to solve it numerically also?
HW W1:
e7+2x−3=0
#Pravas Giri
#Solving the equation
eqn=solve(exp(7+2*x)-3==0,x)
show(eqn)
#Plotting the graph
graph=plot(exp(7+2*x)-3,(x,-4,4))
show(graph)
#The value we want must be numeric. According to the graph there is an
intersection point.
#However, the intersection point is not clear. In this case we can get
appropriate solution by enlarging the graph.
#Plotting the graph by with interval[-4,-2.8]
graph1=plot(exp(7+2*x)-3,(x,-4,-2.8))
show(graph1)
print("The solution is: -2.9 " )
#We can get the approximate value using the command find_root
final=find_root(exp(7+2*x)-3,-4,-2.8)
print("The final solution using find_root command is: ", final)
We
can practice the above code in http://matrix.skku.ac.kr/KOFAC/
Prev
Next
2comments
이상구(LEE SANGGU)September 2, 11:41 AM
Good job^^
기리프라바스(2021####24)October 5, 10:56 PM
Thank you sir.
[Final OK by SGLee] Finalized by Pravas
Giri, Park Junho, WebEx 9/14 Week 3 Meeting All can have 30/30.
[Final OK by SGLee]
Finalized by Pravas Giri, Park Junho, WebEx 9/14 Week 3 Meeting https://youtu.be/ENNxDikPG5M ,Due of the 1st Quiz is extended to this Sat. 11AM. All
can have 30/30. Now make it up if you are not 30/30.
[Final OK by SGLee]
WebEx Summary (Pravas
Giri), [Due of the 1st Quiz is extended to this Sat. 11AM.]
All can have 30/30. Now make it up if you are not 30/30.
<In Intro Math 4 AI, OH,
2021/09/14 Week 3 https://youtu.be/ENNxDikPG5M >
During our 2nd live session, we had a short review of week 1
and week 2 contents
and discussion regarding
week 3 and week 4 contents.
We were
notified that if there will be many academic participation/discussion from all
the students in Q&A Board
and if professor is convinced that every students have
clear idea regarding the courses we have studied
then there
might be PBL Report including a simple Project Proposal and the Presentation of
mid-term PBL will be enough for Your Midterm Exam. <-
It was
notified that if there will be many academic participation/discussion from all
the students in Q&A Board
and if professor is
convinced that every students have clear idea regarding the courses we have
studied
then your
Midterm PBL Report (including a simple Project Proposal) and your Presentation
of mid-term PBL will be enough for Your Midterm Exam.
This means you are preparing
your Exam in QnA by asking Questions~~ You will be free in the Exam week.
However, the main point here is that all the
students must must have knowledge on the topics we have studied.
For any
confusions we are free to ask questions on Q&A and see other students
Q&A as they may contain answers to our questions.
Webex
Meeting Summary (09.14./10:00~11:30) by Park Junho
[Google AI translation] 모두 오늘 결론을 정확히 이해하기 위한 ... 미참여자를 위한 요약
Hope All can see today's discussions
without any confusion ... Summary for who was not there.
1. 수강생들 간에 수학적 배경지식의 차이가 존재 -> QnA 게시판을 활용하여, 질문/답변/문제풀이/수정 or 모른다->질문/안다 -> 답변. 서로 배우자.
1. There is a difference in mathematical background among
students
-> So, All do use QnA
bulletin board to ask/answer/problem solving/correction or Just
say I don't know What? --> Make question or Just sat Now I
got it -> Make your answer.
Let's help to learn each other.
2. 월별 PBL Report는 단순히 위의 활동들을 포함하고, Final Comment 달아서 제출하는 형식이다.
2. The monthly PBL Report simply includes the above our
academic activities with your own final comment.
3. QnA를 통해서 충분한 지식의 교유와 이해가 되었다고 여겨지면, 시험대신 보고서, 아니면 Written Exam (중간, 기말) 을 보게 된다. (problem solving)
<-- 그러나 본 강좌는 실제 무슨 수단으로건 자신의 문제를 풀고 고치면서 원하는 답을 얻는 과정이 필요하므로 전자를 추천한다.
3. If it is considered that you have gained sufficient
knowledge and understanding through QnA, we will have the report or the Written
Exam (mid-term, final) instead of the written exam. <-- However,
I recommend the former, because this course requires a process to get the
answers you want while actually solving and fixing your own problems by any
means.
4. 한글 강의는 반드시 들을 필요는 없는 참고 자료이니 참고만 하고, 궁금한 부분은 질문하면 된다. <--- 수업과 관련된 내용이니 선택하여 들어보자.
4. The Korean lecture is a reference material for some
students who is not good in English, so please just refer to it and ask any
questions you have.
<--- Please select and listen (if you
need)
5. 또한, 영어 강의, 한글 강의에 더불어 일상에서 만나는 예시들도 수업내용과 관련 있다면 자유롭게 QnA 게시판에서 토론하자.
5. In addition to English lectures and Korean lectures,
if you encounter any mathematics related to the class content, feel free to
discuss them in the QnA bulletin board.
<W2 Open
problem 4> Finalized by 박준호, Pravas Giri , Daniil Kim <-- Dear 박준호, I
expect you do it [ finalize that I left for students] like What I did for you
[when you see them]
<W2 Open problem 4> Finalized by Pravas Giri , Daniil Kim <--
Anyone can add comment and your name in the title and Use it in your PBL
report. <-- Dear 박준호, I expect you do it [ finalize that I left for students]
like What I did for you [when you see them] ^^ Do the rest and Get the Extra
credit. It will improve your contribution and Grade.
HW :
e^(7+2)x−3=0
#Pravas
Giri
#Solving the equation
eqn=solve(exp(7+2*x)-3==0,x)
show(eqn)
#Plotting the graph
graph=plot(exp(7+2*x)-3,(x,-4,4))
show(graph)
#The value we want must be numeric. According to
the graph there is an intersection point.
#However, the intersection point is not clear.
In this case we can get appropriate solution by enlarging the graph.
#Plotting the graph by with interval[-4,-2.8]
graph1=plot(exp(7+2*x)-3,(x,-4,-2.8))
show(graph1)
print("The solution is: -2.9 " )
#We can get the approximate value using the
command find_root
final=find_root(exp(7+2*x)-3,-4,-2.8)
print("The final solution using find_root
command is: ", final)
1) Open Problem 4:
The equation that I
chose:
The
source code with the thorough explanations:
Source Code with Explanations.png
#Daniil Kim [GEDN047_I2(이상구)] W1 Assignment
# Declaring an equation
f = exp(x^3)-x^4 == x^2+x
# Solving the equation
precise_solution = solve(f, x)
# Printing the equation
show (f)
# Visualizing the graph
show (plot(f, (x, -1.3, 1.3)))
# Printing the solution
show (precise_solution)
# That is the right solution, however, we need a numeric value, so now we move
on to the approximation:
# As we can see, we have three places where our function intersects with the X
axis: (approx. -0.9, 0.7 and 1.1)
# To make our approximations more precise, we will zoom in the approximated
points by reducing the limit values
# Approximating -0.9:
show (plot(f, (x, -0.9, -0.85)))
show ("The solution is approx. -0.8821")
# Approximating 0.7:
show (plot(f, (x, 0.68, 0.71)))
show ("The solution is approx. 0.6879")
# Approximating 1.1:
show (plot(f, (x, 1.09, 1.11)))
show ("The solution is approx. 1.0957")
# Showing the result
show ("The equation ", f, " has 3 roots:")
show ("Precise roots: ", precise_solution)
show ("Approximated roots: x1 = ", -0.8821, " x2 = ",
0.6879, " x3 = ", 1.0957 )
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
The end
result:
A link
to my Solution in SageMath Cell:
Open Problem 4:
The equation that I chose:
#Daniil Kim [GEDN047_I2(이상구)] W1 Assignment
# Declaring an equation
f = exp(x^3)-x^4 == x^2+x
# Solving the equation
precise_solution = solve(f, x)
# Printing the equation
show (f)
# Visualizing the graph
show (plot(f, (x, -1.3, 1.3)))
# Printing the solution
show (precise_solution)
# That is the right solution,
however, we need a numeric value, so now we move on to the approximation:
# As we can see, we have three
places where our function intersects with the X axis: (approx. -0.9, 0.7 and
1.1)
# To make our approximations more
precise, we will zoom in the approximated points by reducing the limit values
# Approximating -0.9:
show (plot(f, (x, -0.9, -0.85)))
show ("The solution is
approx. -0.8821")
# Approximating 0.7:
show (plot(f, (x, 0.68, 0.71)))
show ("The solution is
approx. 0.6879")
# Approximating 1.1:
show (plot(f, (x, 1.09, 1.11)))
show ("The solution is
approx. 1.0957")
# Showing the result
show ("The equation ",
f, " has 3 roots:")
show ("Precise roots: ",
precise_solution)
show ("Approximated roots: x1
= ", -0.8821, " x2 = ", 0.6879, " x3 = ", 1.0957 )
The
source code with the thorough explanations:
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
(Daniil Kim, 박준호, 기리프라바스)
<W2 Summary>, Open Problems 5 and 6.
Summary:
This week, we learned a lot about
types of data representation:
- Ordered
Pairs (n-tuple). Ordered Pair is a sequence of "components" with
a particular order (order matters). The most popular example of an ordered
pair is a Vector: (x, y, z);
- Matrix.
Matrix is a way of representing data in a 2D array, or 2D grid. Matrices
have rows and columns, and each cell has a corresponding value. The most
popular example of a matrix is a grayscale image;
- Tensors.
Are a general way of saying "data container". In programming it
can be considered is a multi-dimensional array. A 0-dim Tensor is a single
number, a 1-dim Tensor is a Vector, a 2-dim Tensor is a Matrix and so on.
A 3-dim Tensor can represent an RGBA image, basically a 2D Matrix, but
instead of 1 value (grayscale), each cell has 3 values (R, G, B, and
sometimes Alpha/Opacity), hence we can consider a 3-dim Tensor as a 3D
grid.
Also, we revised the standard
matrix properties: addition, multiplication, transposition and inversion, as
well as different types of matrices: diagonal, identity, triangular and
symmetric. Learned how to do a matrix inversion. Prof. explained concepts of
Matrix rank and so on.
Equation solution using matrices:
Suppose we have a system of equations:
Now, let us try to convert it into a matrix.
First, we need to find a coefficient matrix:
In our system, 
are coefficients of the variable 
, and 
are coefficients of the variable 
. So, if we put these
coefficients in the matrix, we will get a coefficient matrix:
Hence, a coefficient matrix is a matrix, made up of coefficients from the equations in a system of equations. Keep in mind, that the correct coefficient matrix is a matrix, where columns represent the coefficients of an individual variable, and rows represent coefficients of a particular equation:
Second, we declare a variable matrix:
It is vertical, because the
dot product of two matrices is the multiplication of each entry of row vector
by each entry of column vector, so if we multiply matrix 
by 
, we will get the exact left side of our initial system of
equations:
Lastly, we find the equation’s constant matrix:
In the end, we get the matrix representation of the initial equation:
After that, we need to find
the augmented matrix, which is a matrix, that we get by merging and 
matrices:
And the only that is left now, is that we need to obtain the reduced row echelon form on the left side (by performing “ERO” or “Elementary Row Operations”), and if we do that, the right side will represent the solutions to the equation:
That method is called “Gauss-Jordan Elimination”.
However, we need to know, Elementary Row Operations, and what exactly is a Row Echelon Form (REF), and a Reduced Row Echelon Form (RREF).
Elementary Row Operations. There are three “ERO”:
1.
Row Swap. Swap any two rows.
2. Scalar Multiplication. Multiply any row
by a constant
3.
Row Sum. Add a multiple of one row to another row.
Row Echelon Form (REF). Example:
In REF:
· The row vectors, all entries of which are zeroes should be at the very bottom of the matrix (marked in red).
· Each pivot should be in a column strictly to the right of the pivots, occurring in the rows above it. A pivot is the first non-zero entry of a particular row (marked in blue).
As we can see in Row Echelon Form pivots form a stair-like structure (drawn in green) (a name “Echelon” is derived from French “échelle” – “staircase” or “ladder” https://dictionnaire.lerobert.com/definition/echelle ).
Reduced Row Echelon Form (RREF). Example:
In RREF:
· All pivots are 1s (marked in blue).
· And the entries above pivots are all 0s.
We can get an RREF from REF with some additional Elementary Row Operations. And as we can see we have a column without pivot (marked in orange). These columns are called “free columns” and they basically represent variables that cannot be found and should be “gambled”. A free variable can appear if you for example have a system of three equations with four unknown variables.
So, in normal system of equations, where the number of equations equals to the number of variables, the Reduced Row Echelon Form, simply is an Identity matrix.
Open Problems:
5:
"Perform
matrix operations with other matrices."
Code:
Result:
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
6:
"With
matrix bigger than 5x5, check whether transpose and inverse matrices
exist."
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
[Final
OK by SGLee] Finilized by 박준호,
Daniil Kim <W3 Summary> Open Problems 7-11.
Author : 김다니일(2021####39)
Date : September 16, 1:59 PM
Summary:
This week, we revised vectors and
various vector operations, including Dot/Inner Product, Vector Norm, Normalized
Vectors, Euclidean Distance and Cosine theorem. And with these, we learnt two
basic ways of data classification - process of identifying to which category
input data belongs:
- Using
Euclidean Distance.
Basically, the larger the Euclidean Distance is, larger will be the distinction between data. - Using
Cosine Theorem / Cosine Similarity.
The larger the Cosine value is, lesser will be the distinction between data.
Open Problems:
#7:
"What kinds of data can you
apply the similarity measures we just discussed (Distance similarity)?"
Using the distance similarity, we
can determine the color similarity. Defining the 3-dim vectors as an rgb pair,
and later calculating Euclidean distance between them.
#8:
"Can you think of the data
you cannot apply to the similarity measure we just discussed (Distance
similarity)?"
We cannot apply Euclidean Distance
for the words similarity, since for instance calculating the distance between
words "greet" and "greetings" will result a huge value,
since the length of the second words noticeably bigger, hence the rise in
distance. In that case it is better to apply a cosine similarity. (More
thorough explanation in Open Problem 10)
#9:
"Randomly generate two 7-dim
vectors and then calculate the distance between two vectors."
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
#10:
"What kinds of data can you
apply a cosine similarity measure to?"
Using the cosine similarity, we
can determine the words distinction. For example, we can assign each letter a
unique integer, represent words as a multi-dim vector of these unique integers,
and then just calculate the angle between them. Here, the cosine similarity is
better, because if we for instance take two words "greet" and
"greetings", the Euclidean distance between them will be large, because
of the length difference, hence the difference in the vector size, however, the
angle would be small enough to call them similar.
#11:
"Make two 5-dim vectors and
find the inner product and the angle of two vectors."
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
And I made an additional
visualization of the similarities, showing the 3 Factories Productivity. In
fact, in this graph, I showed relation between the amount of time spent and the
amount of produced products in hundreds. In that particular example, we can
evaluate the similarity using both Euclidean Distance and Cosine Similarity. In
my case, I decided to show the cosine similarity:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Finalized
by 김다니일
W4 (Daniil Kim). Summary, Open Problems 12 and 13. Question.
Author : 김다니일(2021####39)
Date : September 22, 7:37 PM
Summary:
This
week, we covered the ways to solve systems of linear equations, using two
different methods:
- Matrix
inversion. Basically, in order to find the unique solution x, we need to
multiply the constant matrix by the inverse
of the coefficient matrix. However, it is impossible to use this
method most of the times, because not all coefficient matrices are
invertible, so it is better to use another method. In SageMath, we can solve
it using "COEFFICIENT_MATRIX.solve_right(CONSTANT_MATRIX);"
- Gauss-Jordan
Elimination. A method, that uses RREF (Reduced Row Echelon Form)
and standard ERO (Elementary Row Operations).
It is a pretty easy and convenient method that even helps to determine
whether the system has an infinite amount of solutions, only one, or
multiple. Generally speaking, that is the method that we are taught in
primary schools, with the exception that we now represent the system as
matrices. In SageMath, we can solve it using
"COEFFICIENT_MATRIX.augment(CONSTANT_MATRIX).rref()".
Open Problems:
#12:
"Find a system of linear
equations from other textbooks. Find a solution to that by using the code that
you learned. (inverse matrix and solve_right)"
My solution:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
# W4 (KIM
DANIIL) Open Problem 12 (Practice in
http://matrix.skku.ac.kr/KOFAC/ or https://sagecell.sagemath.org/ )
# W4 (KIM DANIIL) Open Problem 12
(Practice in
http://matrix.skku.ac.kr/KOFAC/ or https://sagecell.sagemath.org/ )
# A convenient function I wrote,
that converts equations to the corresponding marices. However, it is still not
perfect, since it needs the normalized system of equations.
def ConvertToMatrices(eqArray,
varArray):
A=matrix(
[
[
eq.lhs().coefficient(v) for v
in varArray # Getting the LeftHandSide of the equation and retrieving the
coefficients
] for eq in eqArray
]
)
b=matrix(
[
[
eq.rhs() # Getting the values
on the RightHandSide of the equation
] for eq in eqArray
]
)
return (A,b)
# Declaring variables
x, y, z =
var("x","y","z");
eq_1 = (3*x+2*y+2*z == 13)
eq_2 = (2*x+3*y+2*z == 14)
eq_3 = (2*x+2*y+3*z == 15)
# Printing out the system
show("Our system of
equations:")
# I used the MathJax syntax here
show(sage.misc.html.math_parse(r"$$
\left\{ \begin{aligned} 3x+2y+2z&=13 \\ 2x+3y+2z&=14 \\
2x+2y+3z&=15 \end{aligned} \right. $$"))
# Converting the system to the
corresponding matrices
m =
ConvertToMatrices([eq_1,eq_2,eq_3], [x,y,z])
A = m[0]
b = m[1]
show("Solution using the
inverse matrix:")
s = A.inverse()*b
# MathJax syntax
show(sage.misc.html.math_parse(r'$$
\left\{ \begin{aligned} x&='+str(s[0])+r' \\ y&='+str(s[1])+r' \\
z&='+str(s[2])+r' \end{aligned} \right. $$'))
show("Solution using the
solve_right function:")
s = A.solve_right(b)
# MathJax syntax
show(sage.misc.html.math_parse(r'$$
\left\{ \begin{aligned} x&='+str(s[0])+r' \\ y&='+str(s[1])+r' \\
z&='+str(s[2])+r' \end{aligned} \right. $$'))
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Note: For this task, I wrote
a simple function, that utilizes standard SageMath functions and accepts the
array of equations and array of variables involved and returns a tuple of
(COEFFICIENT_MATRIX, CONSTANT_MATRIX).
#13:
"Explain
how and what we can determine for a given linear system of equations with a
unique solution or infinitely many solutions, or no solutions. Use an RREF.
Explain to others what you understand."
My
solution:
# W4 (KIM
DANIIL) Open Problem 13 (Practice in
http://matrix.skku.ac.kr/KOFAC/ or https://sagecell.sagemath.org/ )
# A
convenient function I wrote, that converts equations to the corresponding
marices. However, it is still not perfect, since it needs the normalized system
of equations.
def
ConvertToMatrices(eqArray, varArray):
A=matrix(
[
[
eq.lhs().coefficient(v) for v
in varArray # Getting the LeftHandSide of the equation and retrieving the
coefficients
] for eq in eqArray
]
)
b=matrix(
[
[
eq.rhs() # Getting the values on the
RightHandSide of the equation
] for eq in eqArray
]
)
return (A,b)
#
Declaring variables
x, y, z =
var("x","y","z");
eq_1 =
(x+2*y+3*z == 3)
eq_2 =
(3*x+y+2*z == 7)
eq_3 =
(2*x+3*y+z == 2)
# Printing
out the system
show("Our
system of equations:")
# I used
the MathJax syntax here
show(sage.misc.html.math_parse(r"$$
\left\{ \begin{aligned} x+2y+3z&=3 \\ 3x+y+2z&=7 \\ 2x+3y+z&=2
\end{aligned} \right. $$"))
#
Converting the system to the corresponding matrices
m =
ConvertToMatrices([eq_1,eq_2,eq_3], [x,y,z])
A = m[0]
b = m[1]
show("Solution
using the Gauss-Jordan Elimination:")
show("Find
variable matrix:")
varM =
matrix([[x], [y], [z]])
show(varM)
show("Find
the RREF:")
rref =
A.augment(b, subdivide=True).rref()
show(rref)
show("Final
Solution:")
show(varM,
"=", rref.matrix_from_columns([3]))
We
can practice the above code in http://matrix.skku.ac.kr/KOFAC/
So, during
the Gauss-Jordan Elimination, we can get 3 outcomes:
- We
cannot achieve the RREF, or the RREF contradicts the math (e.g. 0=1). It
means that the equation have no solutions.
- We
managed to get the RREF. Now we have 2 outcomes:
- The
RREF is an identity matrix. It means that our system has a particular
solution.
- The
RREF has "free columns" (look at my summary). It means, that
our system has an infinite amount of solutions. Basically, we can assign
random values to the "free variables", and the solutions will
be different.
In my case, everything went
perfectly, and I got the ideal RREF without free variables.
Finalized by Daniil Kim W5 (Daniil Kim). Summary, Open Problems
14. and <--- Due of the 2nd Quiz. was extended to till Mon. 11AM
Summary:
This week, we found out how to use
the Least-Squares Method of finding an approximate optimal solution for an
array of data/system of equations. Revised the definition of projection, learnt
Curve Fitting method.
Open Problems:
#14:
Answer1: The reason, this property
works is because according to the Vandermonde Determinant theorem, a
determinant of a matrix is always equals to 0, when we have a pair of xi=xj,
where i≠j. And we know, since the formula of matrix inversion is:
we can see, that if determinant is
equal to 0, the matrix is not invertible, since 1/0 is undefined. Hence,
according to the Transitivity Law, we can say, that if there exists a pair of
xi=xj, where i≠j, our matrix is not invertible. And the opposite is also true,
if we do not have a pair xi=xj, where i≠j, it means that our matrix is always
invertible. QED.
Answer2:
Code:
# W5 (KIM
DANIIL) Open Problem 14
# Defining 6 points.
points = [
vector([0,1]),
vector([1,3]),
vector([3,6]),
vector([4,6]),
vector([4.5,7]),
vector([6, 5])
]
# Storing visualized points in a
variable
v_points = list_plot(points)
# Matrix for Linear equation
A = matrix(
[
[1, points[0][0] ],
[1, points[1][0] ],
[1, points[2][0] ],
[1, points[3][0] ],
[1, points[4][0] ],
[1, points[5][0] ]
]
)
# Matrix for Cubic equation
c_A = matrix(
[
[1, points[0][0], points[0][0]^2,
points[0][0]^3 ],
[1, points[1][0], points[1][0]^2,
points[1][0]^3 ],
[1, points[2][0], points[2][0]^2,
points[2][0]^3 ],
[1, points[3][0], points[3][0]^2,
points[3][0]^3 ],
[1, points[4][0], points[4][0]^2,
points[4][0]^3 ],
[1, points[5][0], points[5][0]^2,
points[5][0]^3 ]
]
)
# Constants Matrix
consts = matrix(
[
[points[0][1]],
[points[1][1]],
[points[2][1]],
[points[3][1]],
[points[4][1]],
[points[5][1]]
]
)
x = var("x")
show("Data:",points)
show("Linear:", A,
"=", consts)
lsol = (A.transpose()*A).inverse()*A.transpose()*consts
show("The Least-Squares
solution: ", lsol)
v_lsoleq = plot((lsol[0][0] +
lsol[1][0]*x), (x, 0, 7), color="green")
show("Cubic:", c_A,
"=", consts)
ccfit =
(c_A.transpose()*c_A).inverse()*c_A.transpose()*consts
show("Cubic Curve fitting:
", ccfit)
v_ccfiteq = plot((ccfit[0][0] +
ccfit[1][0]*x + ccfit[2][0]*(x^2) + ccfit[3][0]*(x^3)), (x, 0, 7),
color="red" )
show(v_points+v_ccfiteq+v_lsoleq)
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
*****
In my code, you can easily change
the first array (array of vectors) and it will recalculate everything
accordingly.
<W6Summary-Added> W6 by Daniil Kim, Pravas Giri , Question <- include Week 7
in your Midterm Exam Report!!
W6
by Daniil Kim, Pravas Giri , <W6 Summary> Open Problems 15-16. Question
<- include Week 7 in your Midterm Exam Report!!
W6
by Daniil Kim, Pravas Giri Summary. Open Problems 15-16. Question
Daniil
Kim added on the <W6 Summary> by Pravas Giri <--- Others can add
yours on this.
Summary: Week 6 - Pravas Giri <W6 Summary>
Matrix
Decomposition: Just like the name suggests matrix decomposition is to
simplify and express the matrix to make calculation easier and to solve
difficult problems easily.
1. LU
Decomposition: Decomposing a square matrix into a lower triangular matrix
and upper triangular matrix. Lower triangular matrix where all elements above
diagonal are zero and upper triangular matrix where all elements below diagonal
are zero.
2. QR
Decomposition: Decomposition of a matrix A into a product A=QR of an
orthogonal matrix Q and an upper triangular matrix R.
3. SVD: Is
a product of two orthogonal matrix and rectangular
diagonal matrix with a non-negative real numbers on the main
diagonal. Its key feature is that it exists for any square or rectangular
matrix.
Summary:
This
week, we learnt three important decomposition methods (LU, QR, SVD), each of
which is used in Machine Learning for different purposes.
I
uploaded summary as a word screenshot, to keep the formatting:
+ Open problem 15: by Pravas Giri , Week 6
Open
problem 15:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Open problem 16:
이전
1. Open Problems: #15
by Daniil Kim :
Code:
# W6 (KIM DANIIL) Open Problem 15
# A convenient function I wrote, that converts equations to the
corresponding marices. However, it is still not perfect, since it needs the
normalized system of equations.
def ConvertToMatrices(eqArray, varArray):
A=matrix(CDF,
[
[
eq.lhs().coefficient(v) for v in
varArray # Getting the LeftHandSide of the equation and retrieving the
coefficients
] for eq in
eqArray
]
)
b=matrix(CDF,
[
[
eq.rhs() #
Getting the values on the RightHandSide of the equation
] for eq in
eqArray
]
)
return (A,b)
# Declaring variables
x, y, z = var("x","y","z");
eq_1 = (8*x+2*y+2*z == 13)
eq_2 = (4*x+5*y+2*z == 14)
eq_3 = (2*x+5*y+3*z == 15)
# Printing out the system
show("Our system of equations:")
# I used the MathJax syntax here
show(sage.misc.html.math_parse(r"$$ \left\{ \begin{aligned}
8x+2y+2z&=13 \\ 4x+5y+2z&=14 \\ 2x+5y+3z&=15 \end{aligned} \right.
$$"))
m = ConvertToMatrices([eq_1,eq_2,eq_3], [x,y,z])
A = m[0]
b = m[1]
# In recent versions, SageMath added QR decomposition function:
Q, R = A.QR()
show("x =", R.solve_right(Q.transpose()*b)) #
Least-Squares solution
Open Problem 15:
(# Make a new function ,gs_orth(), using gram_schmidt() function
def gs_orth(A): #Define the function
m, n = A.nrows(), A.ncols() #the size of the matrix
r = A.rank() #the rank of the matrix
if m < n: #Checking the size of the matrix
raise ValueError("The number of rows must be larger than
the number of columns.")
elif r < n: #Checking the full rank
raise ValueError("The matrix is not full column
rank.")
[G, mu] =
A.transpose().gram_schmidt() #Using gram_schmidt function
# Q의 transpose
#Transpose Q
Q1 = matrix([G.row(i)
/ G.row(i).norm() for i in range(0, n)])
R1 = Q1*A
Q = simplify(Q1.transpose()) # A by the method
'orthonormalization'
R = simplify(R1) #An upper triangular matrix
return Q, R
A = matrix([[1, 0, 0], [1, 1, 0], [1, 1, 1]])
#Defind the matrix A
Q, R = gs_orth(A) #QR decomposition
print("Q =")
print(Q.numerical_approx().n(digits = 3))
print()
print("R =")
print(R.numerical_approx().n(digits = 3))
print()
print("Q*R =")
print(Q*R)
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
***
We will have
Q =
[ 0.577 -0.816 0.000]
[ 0.577 0.408 -0.707]
[ 0.577 0.408 0.707]
R =
[ 1.73 1.15 0.577]
[0.000 0.816 0.408]
[0.000 0.000 0.707]
Q*R =
[1 0 0]
[1 1 0]
[1 1 1]
)
For
this part I can fully understand this answer and the labeling is very
clear.Thank you.
By 왕자기
Prev
Next
3comments
김다솔(2018####24)October 8,
11:00 PM
I got it. I tried to make python code to
solve gram-schmidt method. However, I found it exists already.
https://numpy.org/doc/stable/reference/generated/numpy.linalg.qr.html QR
factorization code of numpy library. It involves gram-schmdit method :)
이상구(LEE SANGGU)October 9, 5:04
AM
Good^^
기리프라바스(2021####24)October 11,
2:24 PM
Thank you.
<W6 Open Problems 16> Finalized by김다솔, J.Wang,박준호(added), Daniil Kim and Pravas Giri <-- Anyone can add
comment and your name in the title and Use it in your PBL report.
Open problem 16:by
Pravas Giri ,
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
#16: by Daniil
Kim
Code:
# W6 (KIM DANIIL) Open Problem 16 # W6 (KIM DANIIL) Open Problem 16
# Generating random 3x3
Matrix
M = matrix(
RDF,
[
[
random () for i in range(3)
] for j in range(3)
]
)
U, S, V = M.SVD() # A = U*S*V' # A's SVD
show("U = ", U)
show("S = ", S)
show("V = ", V)
show("M = ", M)
show("USV^T = ",
U*S*V.transpose())
show("=> M =
USV^T")
Try the following Code in
our Lab. http://matrix.skku.ac.kr/KOFAC/
# Generating random 3x3
Matrix
M = matrix(
RDF,
[
[
random () for i in range(3)
] for j in range(3)
]
)
U, S, V = M.SVD() # A = U*S*V' # A's SVD
show("U = ", U.n(digits = 3))
show("S = ", S.n(digits = 3))
show("V = ", V.n(digits = 3))
show("M = ", M.n(digits = 3))
show("USV^T = ", U*S*V.transpose().n(digits = 3))
show("=> M = USV^T")
<< Using This I try
to make the result more visible.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
by PARK Junho
Another one I did.
#J.Wang
Open Problem 16:
Code:
# Generating random 3x3 Matrix
M = matrix(
RDF, [[22,35,43,40], [43,27,15,35],[22,52,36,48], [14,17,11,15]])
U, S, V = M.SVD() # A = U*S*V' # A's SVD()
show("U = ",U)
show("S = ",S)
show("V = ",V)
show("M = ",M)
show("USV^T = ", U*S*V.transpose()
show("=> M = USV^T")
2comments
이상구(LEE SANGGU)October 10,
6:09 PM
Looks better^^ Add your comment.
기리프라바스(2021####24)October 11,
2:36 PM
Thank you.
Open Problems: #15
by Daniil Kim :
Code:
# W6 (KIM DANIIL)
Open Problem 15
# A convenient
function I wrote, that converts equations to the corresponding marices.
However, it is still not perfect, since it needs the normalized system of
equations.
def
ConvertToMatrices(eqArray, varArray):
A=matrix(CDF,
[
[
eq.lhs().coefficient(v) for v
in varArray # Getting the LeftHandSide of the equation and retrieving the
coefficients
] for eq in eqArray
]
)
b=matrix(CDF,
[
[
eq.rhs() # Getting the values on the
RightHandSide of the equation
] for eq in eqArray
]
)
return (A,b)
# Declaring
variables
x, y, z =
var("x","y","z");
eq_1 = (8*x+2*y+2*z
== 13)
eq_2 = (4*x+5*y+2*z
== 14)
eq_3 = (2*x+5*y+3*z
== 15)
# Printing out the
system
show("Our
system of equations:")
# I used the
MathJax syntax here
show(sage.misc.html.math_parse(r"$$
\left\{ \begin{aligned} 8x+2y+2z&=13 \\ 4x+5y+2z&=14 \\
2x+5y+3z&=15 \end{aligned} \right. $$"))
m = ConvertToMatrices([eq_1,eq_2,eq_3],
[x,y,z])
A = m[0]
b = m[1]
# In recent
versions, SageMath added QR decomposition function:
Q, R = A.QR()
show("x
=", R.solve_right(Q.transpose()*b))
# Least-Squares solution
We
can practice the above code in http://matrix.skku.ac.kr/KOFAC/
#16:
Code:
# W6 (KIM DANIIL)
Open Problem 16
# Generating random
3x3 Matrix
M = matrix(
RDF,
[
[
random () for i in range(3)
] for j in range(3)
]
)
U, S, V =
M.SVD() # A = U*S*V' # A's SVD
show("U =
", U)
show("S =
", S)
show("V =
", V)
show("M =
", M)
show("USV^T =
", U*S*V.transpose())
show("=> M
= USV^T")
We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
박준호(2014####20)10월 5일 오후 8:35
Looks good!
By the way, the result of
"print(" ... command is: ", final)" became
"commandis", I can't understand the computer did mistake...
and professor, I did
solve problem 4 (W1) but you didn't finalize it I think... Is there any wrong
in my solution?
이상구(LEE SANGGU)10월 5일 오후 8:49
Dear 박준호, I expect you do it [ finalize that I left for students]
like What I did for you [when you see them] ^^
Do
the rest and Get the Extra credit. It will improve your contribution and Grade.
이상구(LEE SANGGU)10월 5일 오후 8:51
Dear 박준호, Try
# W6 (KIM DANIIL) Open
Problem 16
# Generating random 3x3
Matrix
M = matrix(
RDF,
[
[
random () for i in range(3)
] for j in range(3)
]
)
U, S, V = M.SVD() # A =
U*S*V' # A's SVD
show("U = ",
U.n(digits = 3))
show("S = ",
S.n(digits = 3))
show("V = ",
V.n(digits = 3))
show("M = ",
M.n(digits = 3))
show("USV^T = ",
U*S*V.transpose().n(digits = 3))
show("=> M =
USV^T")
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Prev
Next
5comments
박준호(2014####20)October 5, 8:35 PM
Looks good! By
the way, the result of "print(" ... command is: ", final)"
became "commandis", I can't understand the computer did mistake...
and professor, I did solve problem 4 (W1) but you didn't finalize it I think...
Is there any wrong in my solution?
이상구(LEE SANGGU)October 5, 8:49 PM
Dear 박준호, I
expect you do it [ finalize that I feft for students] like What I did for you
[when you see them] ^^ Do the rest and Get the Extra credit. It will improve
your contribution and Grade.
이상구(LEE SANGGU)October 5, 8:51 PM
Dear 박준호, Try #
W6 (KIM DANIIL) Open Problem 16 # Generating random 3x3 Matrix M = matrix( RDF,
[ [ random () for i in range(3) ] for j in range(3) ] ) U, S, V = M.SVD() # A =
U*S*V' # A's SVD show("U = ", U.n(digits = 3)) show("S = ",
S.n(digits = 3)) show("V = ", V.n(digits = 3)) show("M = ",
M.n(digits = 3)) show("USV^T = ", U*S*V.transpose().n(digits = 3))
show("=> M = USV^T") in http://matrix.skku.ac.kr/KOFAC/
김다니일(2021####39)October 7, 1:05 PM
Thank you for
finalizing our answers.
기리프라바스(2021####24)October 8, 6:08 PM
Thank you.
<W3
Open problem 7> Finalized by 김다솔, 박준호, Daniil Kim, Pravas Giri <-- Anyone
can add comment and your name in the title and Use it in your PBL report.
Open Problems:
1) #7:
"What kinds of data
can you apply the similarity measures we just discussed (Distance
similarity)?"
Using the distance
similarity, we can determine the color similarity. Defining the 3-dim vectors
as an rgb pair, and later calculating Euclidean distance between them.
Prev
Next
4comments
김다솔(2018####24)October 7, 2:56
AM
I understood the problem and the answer.
이상구(LEE SANGGU)October 7, 7:15
AM
OK^^ 김다솔, Good start. Get going^^
김다니일(2021####39)October 7, 1:04
PM
Thank you for finalizing our answers.
기리프라바스(2021####24)October 8, 7:27
PM
Got it.
<W3 Open
problem 8> Finalized by 김다솔, 박준호, Daniil Kim, Pravas Giri <-- Anyone can
add comment and your name in the title and Use it in your PBL report.
<W3 Open
problem 8> Finilized by 박준호, Daniil Kim, <-- Anyone can add comment and
your name in the title and Use it in your PBL report.
<Open problem
8> "Can you think of the data you cannot apply to the
similarity measure we just discussed (Distance similarity)?"
Answer 1.
2) #8: "Can you think of the data you cannot apply to the
similarity measure we just discussed (Distance similarity)?"
Answer 2. We cannot
apply Euclidean Distance for the words similarity, since for instance
calculating the distance between words "greet" and
"greetings" will result a huge value, since the length of the second
words noticeably bigger, hence the rise in distance. In that case it is better
to apply a cosine similarity. (More thorough explanation in Open Problem 10)
..
# W3 (KIM DANIIL) Open
Problem 11 in http://matrix.skku.ac.kr/KOFAC/
# Define vectors
v0 = vector( [random() for i in range (5)] ) # 5 since 5-dim vector
v1 = vector( [random() for i in range (5)] ) # 5 since 5-dim vector
# Defining the similarity threshold
threshold = 0.4 # Approx. 23 degree
# Calculate the distance between them
cosine = v0.inner_product(v1)/(v0.norm() * v1.norm())
angle = arccos(cosine)
show (cosine)
show (angle)
show ("Are these vectors similar? ", angle <= threshold)
And I made an additional
visualization of the similarities, showing the 3 Factories Productivity. In
fact, in this graph, I showed relation between the amount of time spent and the
amount of produced products in hundreds. In that particular example, we can
evaluate the similarity using both Euclidean Distance and Cosine Similarity. In
my case, I decided to show the cosine similarity:
# W3 (KIM DANIIL)
Effectiveness Example. in http://matrix.skku.ac.kr/KOFAC/
# Color constants
_black = (0,0,0)
_red = (1,0,0)
_green = (0,1,0)
_blue = (0,0,1)
_yellow = (1,1,0)
_orange = (1, 0.5, 0)
_purple = (0.7, 0, 0.65)
# Setting up the graph base
base = circle( (0,0), 1, rgbcolor=_black, alpha=0.2, linestyle='--',
axes_labels=["Time ($t$ in hrs)", "Produced ($N^{100}$)"],
axes_labels_size=1, xmin=-3, ymin=-3, xmax=3, ymax=3 ) # Angle circle
scene = base
scene.set_legend_options(back_color=(0.5, 0.5, 0.5), shadow=False) # Changed
colors for the better visibility
# Defining all the necessary vectors
v_origin = vector ([ 0,0 ])
v_0 = vector ( [0.5, 3] ) - v_origin
v_1 = vector ( [1, 2.5] ) - v_origin
v_2 = vector ( [3, 2.5] ) - v_origin
# Adding the defined vectors to our scene
f_m_effective = arrow2d( (0,0), (0.5, 3), width=1, color=_green, arrowsize=2,
legend_label="Most Effective Factory", legend_color=_green )
scene += f_m_effective
f_1 = arrow2d ( (0,0), (1, 2.5), width=1, color=_yellow, arrowsize=2,
legend_label="Factory #1", legend_color=_yellow )
scene += f_1
f_2 = arrow2d ( (0,0), (3, 2.5), width=1, color=_red, arrowsize=2,
legend_label="Factory #2", legend_color=_red )
scene += f_2
# Adding guidelines
l_25 = line2d ( [(0, 2.5), (3, 2.5)], linestyle="-.", rgbcolor=_blue,
alpha=0.5 )
scene += l_25
l_3 = line2d ( [(0, 3), (3, 3)], linestyle="-.",
rgbcolor=_blue, alpha=0.5 )
scene += l_3
# Calculating angles
cos_01 = v_0.inner_product(v_1)/(v_0.norm() * v_1.norm())
a_01 = arccos(cos_01)
cos_02 = v_0.inner_product(v_2)/(v_0.norm() * v_2.norm())
a_02 = arccos(cos_02)
# Adding angle guidelines
d_01 = arc((0,0), 1, sector=(arccos(2*sqrt(29)/29),
arccos(2*sqrt(29)/29)+a_01), color=_orange)
d_02 = arc((0,0), 2, sector=(arccos(6*sqrt(61)/61),
arccos(6*sqrt(61)/61)+a_02), color=_purple)
scene += d_01
scene += d_02
show(scene)
show ("The cosine similarity between the Most Effective Factory and
Factory #1 = ", cos_01)
show ("The cosine similarity between the Most Effective Factory and
Factory #2 = ", cos_02)
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Image Hosting: https://imgur.com/a/IKCtt0d
기리프라바스(2021####24)October 5, 11:08 PM
Thank you.
Prev
Next
4comments
김다솔(2018####24)October 7, 3:00
AM
I understood the idea which is not able to
be calculated by usual distance measure.
이상구(LEE SANGGU)October 7,
12:09 PM
김다솔씨, 하나씩 이해하고 유사한 문제들을 실제 다룰 수 있다는 것을
보여주시면 됩니다^^ You will be fine if you catch up before the Midterm Exam. Good
luck.
김다니일(2021####39)October 7, 1:03
PM
Thank you for finalizing our answers.
기리프라바스(2021####24)October 8, 7:28
PM
Got it.
<W3 Open problem 9> Finalized by 김다솔,
박준호, Daniil Kim, Pravas Giri<-- Anyone can add comment and your name in the
title and Use it in your PBL report.
Open
problem #9: "Randomly generate two 7-dim
vectors and then calculate the distance between two vectors."
Answer 1
3) Answer 2
in http://matrix.skku.ac.kr/KOFAC/
# W3 (KIM DANIIL) Open
Problem 9
# Define vectors
v0 = vector( [random() for i in range (7)] ) # 7 since 7-dim vector
v1 = vector( [random() for i in range (7)] ) # 7 since 7-dim vector
# Defining the similarity threshold
threshold = 0.5
# Calculate the distance between them
d = (v0-v1).norm()
show (d)
show ("Are these vectors similar? ", d <= threshold)
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
Image Hosting: https://imgur.com/a/h6IMuAb
기리프라바스(2021####24)October 5, 11:11 PM
Thanks
Prev
Next
4comments
김다솔(2018####24)October 7, 3:02 AM
The random seventh dimensional vector. I understood and I can generate it.
이상구(LEE SANGGU)October 7, 7:17 AM
I am grad all of you understand this problem.
김다니일(2021####39)October 7, 1:03 PM
Thank you for finalizing our answers.
기리프라바스(2021####24)October 8, 7:29 PM
Thank you. I am able to generate a random vector and calculate the distance
between them
<W3 Open problem 10> Finalized by
김다솔, 박준호, Daniil Kim, Pravas Giri <-- Anyone can add comment and your name
in the title and Use it in your PBL report.
4) Open problem #10: "What kinds of data
can you apply a cosine similarity measure to?"
Answer 1: Using the cosine similarity, we can determine
the words distinction. For example, we can assign each letter a unique integer,
represent words as a multi-dim vector of these unique integers, and then just
calculate the angle between them. Here, the cosine similarity is better,
because if we for instance take two words "greet" and
"greetings", the Euclidean distance between them will be large,
because of the length difference, hence the difference in the vector size,
however, the angle would be small enough to call them similar.
5) Answer 2 :
이상구(LEE SANGGU)9월 14일 오후 2:47
Q1. 강의를 통해 data
similarity의 두 가지 측정방법을 알 수 있었습니다. 그런데 그렇게 측정된 data similarity를 통해 data classification을
어떻게 하는지에 대해서는 알기 어려웠습니다. 관련 내용은 추후의 강의를 통해 알 수 있는 걸까요? 아니면 제가 놓친 부분이 있을까요?
Anawer, Good^^ A Data set 에서 데이터 사이의 data similarity 를 구한 다음에는 ... 기준을 정해서 ... 닮은 것들을 어디를 중심으로 몇 개의 카테고리로 나눌지만 결정하면, 코드
한 줄만 더 하면 ... data 들을 몇개의 카테고리로
classification 하는 것은 쉬운 일이니 ... 좀 더 배운 후 ... 중간고사 후 ... 프로젝트로 ... 실습 해 보시면 됩니다. ^^
이상구(LEE SANGGU)9월 23일 오후 12:41
Add your short comments to the most poosible other's QnA issues including
mine.
Just 'Thanks!' note may be enough^^
이상구(LEE SANGGU)9월 16일 오후 2:03
Yes, You are right.^^
No Attendance Check to watch Korean Lectures that was only for Korean
advanced students for reference.
Take care only English lecture. It will not affect your grade.
PS: If you find any of my Lectures in Korean that require manditory
Attendance, ley me know. I think I did not require any attendence of those
reference lectures in Korean for those who is nott good in English.
이상구(LEE SANGGU)9월 16일 오후 7:02
I double checked. I could not find what you mentioned one in the list. So just feel free. And ask more Questions
to get A.
4comments
김다솔(2018####24)October 7, 3:46
AM
I understood the problem in terms of data
similarity. Thank you.
이상구(LEE SANGGU)October 7, 7:21
AM
OK. I am grad to see you now understand
'Data similarity'^^ Thank you^^
김다니일(2021####39)October 7, 1:03
PM
Thank you for finalizing our answers.
기리프라바스(2021####24)October 8, 7:30
PM
I understood in what type of data we apply
cosine similarity and in which we apply data similarity. Thank you.
[Final OK by SGLee] Finilized by Pravas
Giri, Daniil Kim, 박준호, <Week 4 Summary> , Open problem 12, Open Problem
13 And questions and Answer - 박준호
[Final OK by
SGLee] Finilized by Pravas
Giri, Daniil Kim, 박준호, Week 4 Summary, Open problem 12, Open Problem 13
And
questions and Answer - 박준호
Due to Chuseok, It
was busy time to study lectures.
Summary
I. There are three type
of the solution set (no matter what the system of linear equations we solve
is.)
1. a unique solution. -
RREF: original matrix except augmented vector part becomes "identity
matrix".
2. infinite
solutions. - RREF: there are "free variables", and these and
"1" decide other (not free) variables.
3. no solutions. - RREF:
there is a row vector like (0, 0 ... 0 : 1) which means 0x1 + 0x2 ... 0xn
= 1(F).
I think, this is little
bit confusing case. First one is case 1 (solution is (x=0,y=0,z=0)), but
second one is case 3 (last row vector means (0 = 1)).
II. To solve the system
of linear equations
1. make a coefficient
matrix (of system of linear equations)
2. get a RREF(reduced row
echeleon for) by Gauss-Jordan Elimination
3. just read last column
vector (if it has a unique solution).
Gauss-Jordan Elimination
is using "Elementary Row Operaion"
(1) Exchange two rows.
(2) Multiply a row by nonzero real number (how about complex
number? because of different operation between real number and complex number?)
(3) Add a nonzero multiple of a row of another row
(reference: http://matrix.skku.ac.kr/intro-math4ai/w4/ )
2. Question:
I know that there is an
alternative method of finding the roots of a system of equations, using the
matrix determinants (Cramer's Rule), and it is easier to program than the
Gauss-Jordan Elimination. So, why don't we use the Cramer's Rule then?
Answer. http://matrix.skku.ac.kr/LA/Ch-4/
Cramer's rule can be
applied to systems of linear equations with the same number of unknowns
and the equations.
Solve the following system of
linear equations by Cramer's rule.
A =
matrix(3,3,[-2,3,-1,1,2,-1,-2,-1,1]);
A1 = matrix(3,3,[1, 3, -1,4, 2,-1,-3,-1,1]);
A2 = matrix(3,3,[-2,1, -1, 1,4,-1,-2,-3,1]);
A3 = matrix(3,3,[-2,3,1, 1, 2, 4, -2,-1,-3]);
print ( A.det())
print (A1.det())
print (A2.det())
print (A3.det())
print ( "x
=", A1.det()/A.det())
print ("y =", A2.det()/A.det())
print ( "z =", A3.det()/A.det())
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
6개의 댓글
이상구(LEE SANGGU)9월 23일 오후 12:39
Good job^^
Answer. There are many different ways of finding the solution set of a linear
system of equations,.
Each has advantages and disadvantages, (Cramer's Rule) is good for a small
size, sqare, nonsingular matrix.
the Gauss-Jordan Elimination is better for a large size matrix and a
rectangular coefficent matrix.
There are other wasys too.
이상구(LEE SANGGU)9월 23일 오후 12:41
Add your short comments to the most poosible other's QnA issues including mine.
Just 'Thanks!' note may be enough^^
김다니일(2021####39)9월 24일 오후 3:28
Oh, i see. That makes sense. Actually, I made some research, and apparently the
complexity formula (amount of actions) for the Cramer's Rule is O(n!), whereas
for Gauss-Jordan it is O(n^2). So yeah, I understand what are you talking
about. Basically. for short systems (2 and 3), Cramer's Rule is faster since
2!=2 and 3!=6, whereas Gauss-Jordan will be 2^2=4 and 3^2=9. But systems with
the number of equations larger than 3 will be faster using Gauss-Jordan. Thank
you for the answer! :D
박준호(2014####20)9월 25일 오후 9:34
What is Cramer's Rule? Is there any lecture contents uploaded?
if not, can you explain about that?(of course, if you don't mind...)
and your summary is amazing... easy to read.
김다니일(2021####39)9월 26일 오전 11:51
박준호. Yeah, sure. The explanation turned out to be quite long, so
I posted it on the Q&A board. (and sent you notification)
이상구(LEE SANGGU)9월 27일 오후 3:54
Answer.
the Cramer's Rule only work for det(A) is not zero. and In real computation it
is only good for a small size matrix because it is one of the NP Problem.
http://matrix.skku.ac.kr/LA/Ch-4/
Cramer's rule can be applied to systems of linear equations with the same
number of unknowns and the equations.
Solve the following system of linear equations by Cramer's rule.
A =
matrix(3,3,[-2,3,-1,1,2,-1,-2,-1,1]);
A1 = matrix(3,3,[1, 3, -1,4, 2,-1,-3,-1,1]);
A2 = matrix(3,3,[-2,1, -1, 1,4,-1,-2,-3,1]);
A3 = matrix(3,3,[-2,3,1, 1, 2, 4, -2,-1,-3]);
print ( A.det())
print (A1.det())
print (A2.det())
print (A3.det())
print ( "x =", A1.det()/A.det())
print ("y =", A2.det()/A.det())
print ( "z =", A3.det()/A.det())
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Finilized by Pravas Giri,
박준호, Week 4 (Pravas Giri) Summary, Open problem 12, Open
Problem 13
Summary:
In this chapter we learnt
about augmented matrices, gauss-jordan elimination, and solution set for system
of linear equations.
For augmented matrices,
we must first find the coefficient of matrix and find the solution using the
code A.augment(B), where A is the coefficient of matrix.
In gauss-jordan
elimination, we learnt about the elementary row operations with which we can
find the solution. We can use the code A.augment(B).rref().
Lastly, we learnt about
the way of determining whether a system of linear equation has no solutions,
unique solutions or infinitely many solutions.
If the right side of the
solution of given linear system of equations is not 0 then it has no solutions.
If it is 0, we should
check if the solution is identity matrix. If yes, it has unique solution.
However, if there is free
variable then it will have infinitely many solutions.
6개의 댓글
박준호(2014####20)9월 25일 오후 9:46
In 3rd from the last
sentence, "If the bottom right of the solution of given linear system of
equations is not 0 then it has no solutions.",
what is "bottom
right"??
In your right above RREF,
bottom right seems to mean (0 0 1 0<<this zero), but there is a solution...
so I'm confusing what is bottom right.
Because I'm not familiar
with English expression of mathematics... I'm sorry to ask an awkward
question...
기리프라바스(2021####24)9월 26일 오전 8:34
It is not an awkward
question. Please do feel free to ask as we both are learning and can grow
together.
I would like to give you
another example:
For three variables x, y,
z.
[1 0 0 : 0]
[0 1 0 : 0]
[0 0 1 : 0]
[0 0 0 : 1]
From this we can conclude
that x=0,y=0,z=0 for first 3 rows.
However, in the last row
z.0=0 and 0 != 1.
Hence, this will have no
solution.
In my case, for three
variables x=-1, y=3 and z=0 and as it has no free variables it has unique
solution.
I hope this helps you.
박준호(2014####20)9월 26일 오후 4:34
Thank you for answer!
You means, (0 0 ... 0 :
not 0<<bottom right) -> no solution. Is my understanding correct?
기리프라바스(2021####24)9월 26일 오후 5:16
Sorry for the confusion.
Please don't focus on the term bottom right.
If there is this matrix
then:
[1 0 0 : 0]
[0 1 0 : 0]
[0 0 1 : 0]
[0 0 0 : 1]
[0 0 0 : 0]
In this the bottom right
will the R5. However, we can see that R5 makes sense but R4 doesn't. Hence,
because of R4 it has no solution.
Rather than focusing on
the term bottom right it will be easy if you focus whether the matrix makes
sense of not when assigning the variables to it.
I will make the change
and make it right side.
이상구(LEE SANGGU)9월 27일 오후 3:49
Good^^
Dear Mr. 박준호, Yes, If RREF {A:b] has a row like [0 0 0 : 1] means there
is no solution of Ax=b
박준호(2014####20)9월 27일 오후 5:45
I understand!
Question
저번 주(W3) 강의 관련 질문인데 다시 한 번 올립니다...
Q1.
<---Answer to Q1.
답. 방향이 다르면 사잇각은 아주 크게 됩니다. 그리고 방향이 다른다는 것은 취향이 아주 다르다는 것과도 일치합니다.
Q2.
각도는 서로 만나는 두 반직선으로 이루어지고, 각 직선은 점 2개를 필요로 하며, 꼭짓점을 공유하므로 총 3개의 점이 필요한 것 같습니다.
그런 맥락에서 cosine similarity는 (data1, data2, origin) 3개의 점으로 각도를 만들어서 측정하는 원리로 이해했습니다.
혹시 두 자료의 cosine similarity를 구할 때, 원점이 아닌 다른 점을 기준으로 각도를 만들어서 사용할 수도 있나요?
이상구(LEE SANGGU)9월 27일 오후 3:47
<---
Answer to Q2. 답. 수학에서는 모든 것이 가능합니다. 그러나
지금의 stting 에서는 원점을 기준으로 하는 각각의 벡터들의 닮음을 계량하여 준거를 만든 것입니다. 다른 점을 기준으로 각도를 정의하려면, 유클리드 기하학 대신 비유클리드
기하학을 만들어 가듯이, 가정 공리 증명을 해 가면서 새로운 수학적 구조를 만드는 식으로 ... 할 수 있을 것입니다. 사실은 벡터 공간에서 축만 조금 바꾸어
주어도 새로운 준거를 만들 수 있을 것입니다.
Prev
Next
2comments
이상구(LEE SANGGU)September 27, 3:47 PM
Q1.
답. 방향이 다르면 사잇각은 아주 크게 됩니다.
그리고 방향이 다른다는 것은 취향이 아주 다르다는 것과도 일치합니다. Q2. 답. 수학에서는 모든 것이 가능합니다. 그러나 지금의 stting 에서는 원점을 기준으로 하는 각각의 벡터들의 닮음을 계량하여 준거를 만든 것입니다. 다른 점을 기준으로 각도를 정의하려면, 유클리드 기하학 대신 비유클리드
기하학을 만들어 가듯이, 가정 공리 증명을 해 가면서 새로운 수학적 구조를 만드는 식으로 ... 할 수 있을 것입니다. 사실은 벡터 공간에서 축만 조금 바꾸어
주어도 새로운 준거를 만들 수 있을 것입니다.
기리프라바스(2021####24)October 3, 10:42 PM
Thank
you sir.
박준호(2014####20)9월 25일 오후 9:46
In 3rd from the last sentence, "If the bottom right of the solution of
given linear system of equations is not 0 then it has no solutions.",
what is "bottom right"??
In your right above RREF, bottom right seems to mean (0 0 1 0<<this
zero), but there is a solution... so I'm confusing what is bottom right.
Because I'm not familiar with English expression of mathematics... I'm sorry
to ask an awkward question...
기리프라바스(2021####24)9월 26일 오전 8:34
It is not an awkward question. Please do feel free to ask as we both are
learning and can grow together.
I would like to give you another example:
For three variables x, y, z.
[1 0 0 : 0]
[0 1 0 : 0]
[0 0 1 : 0]
[0 0 0 : 1]
From this we can conclude that x=0,y=0,z=0 for first 3 rows.
However, in the last row z.0=0 and 0 != 1.
Hence, this will have no solution.
In my case, for three variables x=-1, y=3 and z=0 and as it has no free
variables it has unique solution.
I hope this helps you.
박준호(2014####20)9월 26일 오후 4:34
Thank you for answer!
You means, (0 0 ... 0 : not 0<<bottom right) -> no solution. Is my
understanding correct?
기리프라바스(2021####24)9월 26일 오후 5:16
Sorry for the confusion. Please don't focus on the term bottom right.
If there is this matrix then:
[1 0 0 : 0]
[0 1 0 : 0]
[0 0 1 : 0]
[0 0 0 : 1]
[0 0 0 : 0]
In this the bottom right will the R5. However, we can see that R5 makes
sense but R4 doesn't. Hence, because of R4 it has no solution.
Rather than focusing on the term bottom right it will be easy if you focus
whether the matrix makes sense of not when assigning the variables to it.
I will make the change and make it right side.
이상구(LEE SANGGU)9월 27일 오후 3:49
Good^^
Dear Mr. 박준호, Yes, If RREF {A:b] has a row like [0 0 0 : 1] means there is
no solution of Ax=b
박준호(2014####20)9월 27일 오후 5:45
I understand!
Prev
Next
7comments
박준호(2014####20)September 25, 9:46
PM
In 3rd from the
last sentence, "If the bottom right of the solution of given linear system
of equations is not 0 then it has no solutions.", what is "bottom
right"?? In your right above RREF, bottom right seems to mean (0 0 1
0<<this zero), but there is a solution... so I'm confusing what is bottom
right. Because I'm not familiar with English expression of mathematics... I'm
sorry to ask an awkward question...
기리프라바스(2021####24)September 26, 8:34
AM
It is not an
awkward question. Please do feel free to ask as we both are learning and can
grow together. I would like to give you another example: For three variables x,
y, z. [1 0 0 : 0] [0 1 0 : 0] [0 0 1 : 0] [0 0 0 : 1] From this we can conclude
that x=0,y=0,z=0 for first 3 rows. However, in the last row z.0=0 and 0 != 1.
Hence, this will have no solution. In my case, for three variables x=-1, y=3
and z=0 and as it has no free variables it has unique solution. I hope this
helps you.
박준호(2014####20)September 26, 4:34
PM
Thank you for
answer! You means, (0 0 ... 0 : not 0<<bottom right) -> no solution.
Is my understanding correct?
기리프라바스(2021####24)September 26, 5:16
PM
Sorry for the
confusion. Please don't focus on the term bottom right. If there is this matrix
then: [1 0 0 : 0] [0 1 0 : 0] [0 0 1 : 0] [0 0 0 : 1] [0 0 0 : 0] In this the
bottom right will the R5. However, we can see that R5 makes sense but R4
doesn't. Hence, because of R4 it has no solution. Rather than focusing on the term
bottom right it will be easy if you focus whether the matrix makes sense of not
when assigning the variables to it. I will make the change and make it right
side.
이상구(LEE SANGGU)September 27, 3:49
PM
Good^^ Dear Mr. 박준호,
Yes, If RREF {A:b] has a row like [0 0 0 : 1] means there is no solution of
Ax=b
박준호(2014####20)September 27, 5:45
PM
I understand!
이상구(LEE SANGGU)October 3, 4:54 PM
Wonderful^^
<W5
Open Problems 14> by 박준호, Daniil Kim
and Pravas Giri <-- Anyone can add comment and
your name in the title and Use it in your PBL report.
3. <W5 Open Problems 14> by
Daniil Kim and Pravas Giri 기리-Giri and 김다니일
4. <--
Anyone can add comment and your name in the title and Use it in your PBL
report.
Question answer:
For a matrix to be invertible its determinant should not be
equal to zero. Hence, A matrix is only invertible if the determinant of the
matrix is not 0.
#14:
Answer1: The reason, this property works is because according
to the Vandermonde Determinant theorem, a determinant of a matrix is always
equals to 0, when we have a pair of xi=xj, where i≠j. And we know, since the
formula of matrix inversion is:
we can see, that if determinant is equal to 0, the matrix is
not invertible, since 1/0 is undefined. Hence, according to the Transitivity
Law, we can say, that if there exists a pair of xi=xj, where i≠j, our matrix is
not invertible. And the opposite is also true, if we do not have a pair xi=xj,
where i≠j, it means that our matrix is always invertible. QED.
3. open problem 14에서 A transpose * A가
가역일 조건을 구하라고 하셨는데, 저는 A의 column vector들이 선형독립인 것이라고 추측했습니다만 정확한 증명을 할 수가 없어 정확한 조건 또한
구하기 어려웠던 것 같습니다...
<--- A transpose * A가 가역일 조건 은 A 가 full
column tank 를 갖는 것입니다. 즉 컬럼들이 모두 일차 독립이면 A transpose * A 는 Full rank 를 갖는 정사각행렬이
되고, 가역행렬입니다.
박준호(2014####20)10월 3일
오후 2:32
넵 감사합니다! 확인해보겠습니다!
Prev
Next
3comments
이상구(LEE SANGGU)October 5, 7:28 PM
Dear
All , I recommend you to add your comment on the solution on Prob's by Others.
(This one and Prob. 1-11) You may take look on this problem and solution, and
Add your comment or Revise or Finalize or Re-Finalize. And add your name in the
title. And use it to fill out your PBL report. <--- I told that in today's
OH. Then all will have the same Full credit. That is my Tip for you and others.
Good luck.
김다니일(2021####39)October 7, 1:06 PM
Thank
you for finalizing our answers.
기리프라바스(2021####24)October 8, 7:35 PM
Thank
you for the explanation on whether A^TA is invertible or not. <--- A
transpose * A가 가역일 조건 은 A 가 full
column tank 를 갖는 것입니다. 즉 컬럼들이 모두 일차 독립이면 A transpose * A 는 Full rank 를 갖는 정사각행렬이
되고, 가역행렬입니다. I understood it.
W7 (Daniil Kim) Summary. Chapter 2 Open Problems: 1
Summary:
This week we
revised the previous Chapter and the basics of Calculus:
- Derivatives
(shorthand for limit x->0),
- Limits
(a value that the function approaches as x approaches some number a),
- Tangent
Lines (a line that "touches" the graph at one point),
- Slopes
(a number that describes both direction and the steepness of a function
(tan(alpha))
Chapter 2 Open Problems:
#1:
Code:
# W7 (KimDaniil) Chapter 2 Open Problem 1
f(x) = (sqrt(x-1)+(x-1)^(1/5))/((x-1)^(1/3))
show("f(x)=", f(x))
show("Derivatives:")
show("f'(x)=", diff(f(x), x))
show("f''(x)=", diff(f(x), x, 2))
show("f'''(x)=", diff(f(x), x, 3))
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
2comments
이상구(LEE SANGGU)October 9, 5:06
AM
Good problem!
기리프라바스(2021####24)October 11,
2:40 PM
Thank you sir.
Second part:
5. Summary:
My word summary:
W8
(Daniil Kim). Summary, Chapter 2 Open Problems: 2.
Author : 김다니일(2021####39)Date : October
15, 2:48 PM
Count : 9
Summary:
My
word summary:
Derivatives help us to determine function properties:
· First derivative gives us information about the critical points and the changes in the slope sign. Whenever, the derivative value is 0, it means that the function has a critical point, and at that critical point, the slope sign changes (increasing -> decreasing or decreasing -> increasing):
o If 
at all points in the
interval, then increases on that interval
o If 
at all points in the
interval, then decreases on that interval
· Second derivative gives us information about local and absolute maximums and minimums of the function (extreme values):
o Concave Upward. If the graph of lies above all of its tangents
on an interval 
, then it is called concave upward on . In other words, if 
, then is convex
downward/concave upward at 
o Concave Downward. If the graph of lies below all of its tangents
on an interval , then it is called concave downward
on . In other words, if 
, then is convex
upward/concave downward at
o
Inflection Point. An Inflection Point is a point on a curve 
, where curve changes its concavity type.
Fermat’s Theorem: If 
has a local maximum
or minimum at 
and if 
exists, then 
.
Local extreme points:
·
If and , then 
is a local maximum.
·
If and , then is a local minimum.
Absolute extreme points:
· Find all critical points.
· Compare function values at these critical points and ends of the interval.
· The biggest value will be the absolute maximum and the smallest will be the absolute minimum.
Optimal solution 
should satisfy Fermat’s
Theorem:
So, all we need to do is solve
and determine which critical points satisfy the aforementioned
equation. However, sometimes it can be very hard, so there are several methods
to solve this problem, and one of them is The Gradient Descent Method
(GDM).
Chapter 2 Open Problems:
#2:
My
Solution:
Code:
#
W8 (Kim Daniil) Chapter 2 Open Problem 2
#
Making it interactive
@interact
def
_( temp1 =
input_box(x^4-6*x^2+1,label="f(x)=", width=30),
temp2 = input_grid(1, 2,
default=["-1","3"], label="Interval:", width=6),
zoomX = slider(1, 100, step_size=0.5,
default=5, label="Zoom X:", display_value=True),
zoomY = slider(1, 100, step_size=0.5,
default=30, label="Zoom Y:", display_value=True),
scale = slider(4, 40, step_size=1,
default=15, label="Graph Scale:", display_value=True)):
# Converting a temp1 input to a function.
f(x) = temp1
# Converting a temp2 grid to the interval
temp2[0] means first row in grid temp[2], which returns a 1-dim array e.g
[-1,3]
interval = temp2[0]
show("Equation: f(x)=", f(x))
show("Interval: ", interval)
df(x) = diff(f(x),x)
show("First Derivative: f'(x)=",
df(x))
d2f(x) = diff(df(x),x)
show("Second Derivative:
f''(x)=", d2f(x))
plot_f = plot(f(x), x,
color="green", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY,
legend_label="Function", figsize=(scale,scale))
plot_df = plot(df(x), x,
color="red", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY,
legend_label="1st Derivative", figsize=(scale,scale))
plot_d2f = plot(d2f(x), x,
color="blue", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY,
legend_label="2nd Derivative", figsize=(scale,scale))
plot_interval = line2d([(interval[0],
-zoomY), (interval[0], zoomY)], linestyle="--",
color="black", alpha=0.5) + line2d([(interval[1], -zoomY),
(interval[1], zoomY)], linestyle="--", color="black",
alpha=0.5)
# Finding critical points.
t = solve(diff(f(x)) == 0, x)
# Transforming SageForm ( [x == 1, x == 2
...] ) to normal array of values ( [1, 2 ...] )
cPoints = [i.rhs() for i in t]
# Add the interval ends, as critical
points:
cPoints.append(interval[0])
cPoints.append(interval[1])
# Creating a dictionary of (x,f(x)) values:
cPointsVals = {}
# List of local extremes
locM = []
# Finding extreme points
show("All Local Extreme Points:")
for j in cPoints:
k = f(j)
v = d2f(j)
# Finding local maxima and minima:
if df(j) == 0:
l = vector([j,k])
locM.append(l)
show("Local", "
minimum" if v > 0 else " maximum",":", l)
# Calculating values of the function at
the extreme points to later find absolute extreme points
if j >= interval[0] and j <=
interval[1]:
cPointsVals[j] = k
# Finding absolute maxima and minima
absMaxX = max(cPointsVals,
key=cPointsVals.get)
absMinX = min(cPointsVals,
key=cPointsVals.get)
# List of abs extremes
absM = [
vector([absMaxX,cPointsVals[absMaxX]]),
vector([absMinX,cPointsVals[absMinX]])
]
# Printing
show("Interval Absolute Extreme
Points:")
show("Absolute Maximum:",
absM[0])
show("Absolute Minimum:",
absM[1])
plot_loc = list_plot(locM,
color="purple", legend_label="Local Extremes")
plot_abs = list_plot(absM,
color="orange", legend_label="Absolute Extremes")
show(plot_f +plot_df +plot_d2f +
plot_interval + plot_loc + plot_abs)
I
made it completely interactive, so you can easily enter your own equation and
interval, and the code will solve it.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
W9 (Daniil Kim). Summary. Chapter 2 Open Problems 3 and 4.
Summary:
Gradient Descent Method/Algorithm is an
optimization algorithm to find the minimum of a complex function.
The general idea of the Gradient Descent Method (GDM) is to:
- Find the slope/gradient of
the function. Basically, find the derivative.
- Pick a random point
and move in the direction of the negative gradient (a direction,
in which function decreases the most), by a certain coefficient , that is called learning rate (usually,
it is a number in range (10)).
- Repeat the process
until we reach the extreme value.
The GDM can
also be used to solve the Least
Squares Problem the idea is to find the solution with the
minimized Error, in other words, we apply the GDM to the error
function to find the most optimal solution.
Chapter 2 Open Problems:
I wrote another convenient
interactive sagemath module, so everyone can easily play with the values and
adapt for their particular equations.
Note: I modified the algorithm,
that was in the video. The reason is, that in this case, Python loops perform
slower (for i in range(x)), in comparison with C loops (while x).
# W9 (Kim Daniil) Chapter 2 Open
Problem 3 and 4
# Making it interactive
@interact
def _( temp1 =
input_box(9*x^2-7*x+6,label="f(x)=", width=30),
threshold = slider(0.000001, 2,
step_size=0.000001, default=0.000001, label="Threshold/Tolerance:",
display_value=True),
l_rate = slider(0.00001, 1,
step_size=0.00001, default=0.1, label="Learning Rate:",
display_value=True),
base_point = input_box(0.0,
label="Base Point:", width=20),
terminator = slider(100, 10000,
step_size=1, default=300, label="Terminate after:",
display_value=True),
zoomX = slider(1, 100, step_size=0.5,
default=5, label="Zoom X:", display_value=True),
zoomY = slider(1, 100, step_size=0.5, default=30,
label="Zoom Y:", display_value=True),
scale = slider(4, 40, step_size=1,
default=15, label="Graph Scale:", display_value=True)):
# Converting a temp1 input to a function.
f(x) = temp1
show("Equation: f(x)=", f(x))
df(x) = diff(f(x),x)
show("Slope: f'(x)=", df(x))
# GDM implementation
i_points = []
iterator = 0
my_x = base_point #Starting point
g = threshold+1
while ( iterator < terminator and g > threshold ):
g = df(my_x)
i_points.append(vector([my_x, g]))
my_x = my_x - l_rate*g
g = abs(g)
iterator +=1
final_point = vector([my_x, f(my_x)])
show("x* = ", final_point[0])
show("|g*| = ", g)
show("f(x*) = ", final_point[1])
show("Achieved in '", iterator, "' attempts.")
plot_f = plot(f(x), x, color="green", xmin=-zoomX,
ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="Function",
figsize=(scale,scale))
plot_df = plot(df(x), x, color="blue", xmin=-zoomX, ymin=-zoomY,
xmax=zoomX, ymax=zoomY, legend_label="1st Derivative",
figsize=(scale,scale))
plot_i_points = list_plot(i_points, color="red",
legend_label="Iteration Points")
plot_final_point = list_plot([final_point], color="purple",
legend_label="Result")
show( plot_f + plot_df + plot_i_points + plot_final_point )
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
#3:
#4:
이상구(LEE SANGGU)October
15, 2:59 PM
Very good^^
박준호(2014####20)October
17, 6:58 PM
I tried your code and it is
very nice!
기리프라바스(2021####24)October
20, 6:33 PM
Thank you for the clear
explanation.
PRAVAS
GIRI
SUMMARY:
In this week we learnt about
1.
Application of Derivatives.
- If the
f'(x)>0 then f increases on interval [a,b].
- If the f'(x)<0 then f decreases on interval [a,b].
- If the function is increasing then the slope of the tangent line is
+ve and the curve of the function is above tangent line.
- If the function is decreasing then the slope of the
tangent line is -ve and the curve of the function is below tangent line.
2.
Application of Second Derivatives.
- The curve y=f(x)
is concave up (convex down) at (c,f(c) if (x,f(x)) lies above the tangent
at (c,f(c)) for all near c.
- The curve y=f(x) is concave down (convex up) at (c,f(c) if (x,f(x))
lies below the tangent at (c,f(c)) for all near c.
3.
Local Maximum, Local Minimum.
When
a function f has f', f" at c in the domain with f'(c)=0.
- a. if
f"(c)>0, then f(c) is a local maximum.
- b. if
f"(c)<0, then f(c) is local minimum.
4.
Absolute Maximum, Absolute Minimum.
- Find all
critical numbers of f within the interval [a, b].
That is, set f '(x) = 0, solve for x, and
only consider those solutions x that satisfy a ≤ x ≤ b.
- Plug in each
critical number from step 1 into the function f(x).
- Plug in the
endpoints, a and b, into the
function f(x).
- The largest
value is the absolute maximum, and the smallest value is the absolute
minimum.
Prev
Next
3comments
이상구(LEE SANGGU)October
20, 7:57 PM
Good^^
기리프라바스(2021####24)October
22, 7:43 AM
Thank you sir.
김다니일(2021####39)November
29, 2:11 PM
Thank you for finalizing my
answer and adding a more thorough explanation of the abs. max/min evaluation
method.
<Week
8-Open problem 2>Pravas Giri + Graph Added
Author : 기리프라바스(2021####24)Date : October
22, 8:04 AM
Count : 6
Chapter-2
Open
Problem-2
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
2comments
이상구(LEE
SANGGU)October 20, 6:53 PM
Good.
OK. It will be better, if you plot the graph as well. Sample: var('x,y')
p1=plot(5*x^3 + 3*x^2 -4*x - 1, x, -2.5,2, color='blue'); p2 =
text("$y=5x^3 +3x^2 -4x-1 $", (-1.4,15), fontsize=15, color='blue')
show(p1+p2, ymax=20, ymin=-10)
기리프라바스(2021####24)October
22, 8:02 AM
Okay
sir. Thank you.
Prev
Next
2comments
이상구(LEE
SANGGU)October 27, 1:28 PM
Take
a look on a simple Sage interact ========================== @interact def _(a =
slider(-10, 10, step_size = 1, default = 0, label = "a",
display_value = True)): f(x) = 2^(-x^2 + 4*x + a) print("f(x) =",
f(x)) print("a =", a) p1 = plot(f(x), (x, 0, 3)) p2 = plot(4, (x, 0,
3), color = 'red') show(p1 + p2) You may practice in
http://matrix.skku.ac.kr/KOFAC/ and improve your coding to be iteracted^^
김다니일(2021####39)December
1, 00:27 AM
Yeah,
I think with the addition of the graph, it looks much better :D.
Finalized by Pravas Giri,
김다니일, 박준호 <Week 9- SUMMARY>
PRAVAS GIRI
Summary:
Gradient Descent Method
(GDM):
When a function is complicated
then the critical point can be found through GDM.
The basic idea is to find
the slope of the function (derivative), move it toward downhill (learning
rate), and repeat it until it reaches the extreme value.
6. 김다니일
7. Summary:
Gradient
Descent Method/Algorithm is an
optimization algorithm to find the minimum of a complex function.
The general idea of the Gradient Descent Method
(GDM) is to:
- Find the slope/gradient of the function.
Basically, find the derivative.
- Pick a random point and move in the direction of the negative
gradient (a direction, in which function decreases the most), by
a certain coefficient
,
that is called learning rate (usually, it is a number
in range (10)). - Repeat the process until we reach the extreme value.
The GDM can also be used to solve
the Least Squares Problem the idea is to find the solution
with the minimized Error, in other words, we apply the GDM to
the error function to find the most optimal solution.
박준호
Summary:
Gradient Descent Method:
(source : http://matrix.skku.ac.kr/intro-math4ai/w9/)
Using GDM to get Least
Square Solution:
set an Error Function E(u)
and we minimize the error.
(source : http://matrix.skku.ac.kr/intro-math4ai/w9/)
Prev
Next
1comments
김다니일(2021####39)December
2, 4:12 PM
Thank you for finalizing!
Finalized by Pravas Giri, 김다니일, 박준호, 김다솔 <Week 9- OPEN
PROBLEM 3>
PRAVAS GIRI
Open Problem 3:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
8. 김다니일
9. Chapter 2 Open
Problems:
I wrote another convenient
interactive sagemath module, so everyone can easily play with the values and
adapt for their particular equations.
Note: I modified the algorithm,
that was in the video. The reason is, that in this case, Python loops perform
slower (for i in range(x)), in comparison with C loops (while x).
# W9 (Kim Daniil) Chapter 2
Open Problem 3 and 4
# Making it interactive
@interact
def _( temp1 = input_box(9*x^2-7*x+6,label="f(x)=",
width=30),
threshold = slider(0.000001, 2,
step_size=0.000001, default=0.000001, label="Threshold/Tolerance:",
display_value=True),
l_rate = slider(0.00001, 1,
step_size=0.00001, default=0.1, label="Learning Rate:",
display_value=True),
base_point = input_box(0.0,
label="Base Point:", width=20),
terminator = slider(100, 10000,
step_size=1, default=300, label="Terminate after:",
display_value=True),
zoomX = slider(1, 100, step_size=0.5,
default=5, label="Zoom X:", display_value=True),
zoomY = slider(1, 100, step_size=0.5,
default=30, label="Zoom Y:", display_value=True),
scale = slider(4, 40, step_size=1,
default=15, label="Graph Scale:", display_value=True)):
# Converting a temp1 input to a function.
f(x) = temp1
show("Equation: f(x)=", f(x))
df(x) = diff(f(x),x)
show("Slope: f'(x)=", df(x))
# GDM implementation
i_points = []
iterator = 0
my_x = base_point #Starting point
g = threshold+1
while ( iterator < terminator and g >
threshold ):
g = df(my_x)
i_points.append(vector([my_x, g]))
my_x = my_x - l_rate*g
g = abs(g)
iterator +=1
final_point = vector([my_x, f(my_x)])
show("x* = ", final_point[0])
show("|g*| = ", g)
show("f(x*) = ", final_point[1])
show("Achieved in '", iterator,
"' attempts.")
plot_f = plot(f(x), x,
color="green", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY,
legend_label="Function", figsize=(scale,scale))
plot_df = plot(df(x), x,
color="blue", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY,
legend_label="1st Derivative", figsize=(scale,scale))
plot_i_points = list_plot(i_points,
color="red", legend_label="Iteration Points")
plot_final_point = list_plot([final_point],
color="purple", legend_label="Result")
show( plot_f + plot_df + plot_i_points +
plot_final_point )
Code:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
#3:
박준호
open problem 3
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
using Daniil Kim's
code (I changed some 'sliders' to
'input_boxes')
we can see the converge
speed is too slow and the destination(x*; f'(x*) = 0)
is not local minimum!
Therefore the initial iterator and learning rate should be properly set.
김다솔
122p 열린문제 3
함수
f(x) = 9x^2-7x+6의 최솟값을 구하시오.
단 x_1
= 0, eta=0.1, epsilon=10^-6으로 한다.
풀이
initial state
f(x_1) = 6
update formula
x <- x - eta*dx
x_1에서의 derivative를 구하면
f'(x)=18x-7
dx = f'(x_1) = -7
update formula에 맞추어서
x_2 <- x_1 - eta*dx 형식으로 표현하면
x_2 <- 0 - 0.1 * (-7) 가 된다.
x_2 = 0.7
second state
f(x_2) = 9*(0.7)^2 -7*(0.7)
+ 6 = 4.41 - 4.9 + 6 = 5.51
6 -> 5.51 로 state가 업데이트 된 것을 볼 수 있다.
third state
x_3 = x_2 - 0.1*f'(x_2) =
0.7 - 0.1*(5.6) = 0.14
f(x_3) = 5.1964
이러한 방식으로 dx가
epsilon보다 작아질 때까지 update를 반복한다.
---python code---
def function(x):
return ((x**2)*9)-(x*7)+6
def derivative(x):
return (18*x)-7
def update(x_1):
x_2 = x_1 - 0.1*derivative(x_1)
return x_2
x = 0
states=[]
while abs(derivative(x)) > 10e-6:
states.append(round(function(x),6))
x = update(x)
print(len(states), states)
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
------------------------------------
결과
61번 반복 후 4.638889로 수렴했다. (6 digits round)
[6, 5.51, 5.1964, 4.995696,
4.867245, 4.785037, 4.732424, 4.698751, 4.677201, 4.663408, 4.654581, 4.648932,
4.645317, 4.643003, 4.641522, 4.640574, 4.639967, 4.639579, 4.639331, 4.639172,
4.63907, 4.639005, 4.638963, 4.638936, 4.638919, 4.638908, 4.638901, 4.638897,
4.638894, 4.638892, 4.638891, 4.63889, 4.63889, 4.638889, 4.638889, 4.638889,
4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889,
4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889,
4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889, 4.638889,
4.638889]
Prev
Next
3comments
김다솔(2018####24)October
30, 2:39 PM
Sorry for using Korean
language
이상구(LEE SANGGU)October
30, 2:40 PM
Looks good. Take a look on
https://github.com/pod3275/-AI-Simple-linear-regression-using-GDM And add a
comment~~
기리프라바스(2021####24)December
1, 11:21 AM
Thank you.
Prev
Next
1comments
김다니일(2021####39)December
2, 4:12 PM
Thank you for finalizing!
Finalized by Pravas Giri,
김다니일, 박준호 <Week 9- OPEN PROBLEM 4>
PRAVAS GIRI
Open Problem 4:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Prev
Next
3comments
이상구(LEE SANGGU)October
27, 10:58 AM
Nice try. What did you see in
here?
기리프라바스(2021####24)October
28, 5:31 PM
From the graph below, I
identified that the local minimum is between the interval (1.35, -2.5).
김다니일(2021####39)December
1, 00:31 AM
Good code! Although if it
fails, it will probably print "garbage" (in this case default)
values, that can sometimes confuse the user, so perhaps add some flag on
"Algorithm Succeeded!" and then check it later on, when print the
result.
10. 김다니일
#4:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
3comments
이상구(LEE SANGGU)October
22, 7:30 PM
Good job^^☆☆
기리프라바스(2021####24)October
27, 9:36 AM
Thank you for the detailed
explanation. :)
박준호(2014####20)October
27, 1:11 PM
the module is useful!
박준호
open problem 4
f'(0) = 0 and f''(0) > 0,
so minimum point (x*, f(x*)) = (0, 1)
The result of GDM is very
similar to this.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
6comments
이상구(LEE SANGGU)October
27, 2:49 PM
Nice update~~ Good. Add your
Comment later in your Final PBL.
김다니일(2021####39)October
28, 8:36 PM
Yeah, I also stumbled upon
this problem and after a bit of research. It turned out that the GDM always
works ONLY if we have the convex function, and it will always return the global
minimum. In other cases, we need to be lucky to get the initial guess right, so
that we do not go into the infinity, and even more lucky to get the initial
guess near the GLOBAL minimum, to get the global minimum as a result of GDM
(instead of the usual local one).
김다니일(2021####39)October
28, 8:38 PM
https://www.researchgate.net/post/Does_anyone_know_how_to_select_the_initial_guess_for_the_steepest_descent_method_loop_to_arrive_the_minimal_point
Two last answers to be precise: Quote 1: "This method does not guarantee
you the global-optimal value. As far as I know, there are no real
recommendations for certain initial values, as before you executed the
algorithm you have no idea about the function you explore. Therefore, you can
randomly sample initial position. If you are looking for a global-optimum then
you would need to search for the algorithms, that can avoid local optima, e.g.,
VNS or GA." and the 2nd Quote: "If you have an initial state of the
problem, the best initial point is the variable values of the current state.
otherwise it is better to find it by trail and error. That is, by selecting
several initial solution and seeing which one performs better. Otherwise there
may not be a general rule for finding the initial guess."
이상구(LEE SANGGU)October
28, 8:54 PM
Yes you are right. Gradient
descent then starts at that point (somewhere around the the local min.), and it
takes one step after another in the steepest downside direction (i.e., from the
top to the bottom of the illustration) until it reaches the point where the
cost function is as small as possible.
박준호(2014####20)October
29, 10:51 PM
I guess, this method seems to
be really powerful only when we know the function and we want to get a solution
approximately.
기리프라바스(2021####24)November
2, 8:46 AM
Thank you. It helped me
understand more about GDM.
Finalized by Pravas Giri, 박준호, 김다니일 <Week-10 OPEN PROBLEM 1>
PRAVAS GIRI
Open Problem 1 (Discussion):
Monty Hall Problem:
The Monty Hall
problem is a brain teaser, in the form of a probability puzzle,
loosely based on the American television game show Let's Make a Deal and named after its original host, Monty Hall.
The problem was originally posed (and solved) in a letter by Steve Selvin to
the American Statistician in 1975.[1][2] It
became famous as a question from reader Craig F. Whitaker's letter quoted in Marilyn vos Savant's "Ask Marilyn" column in Parade magazine
in 1990:[3]
Suppose you're on a game show, and you're given the choice of
three doors: Behind one door is a car; behind the others, goats. You pick a
door, say No. 1, and the host, who knows what's behind the doors, opens another
door, say No. 3, which has a goat. He then says to you, "Do you want to
pick door No. 2?" Is it to your advantage to switch your choice?
In this problem we consider
that we are in a game show where there are 3 doors (A,B,C) in front and two of
them have a goat behind it and one has a car. I want to choose a door which has
a car behind. The door A and B has goat behind it and the door C has car. The
game show host knows what are behind all the 3 doors.
Suppose that I choose door
A. Then the game show host opens the door B and there is goat. Now the game
show host gives me an option to swap the door. Initially, I thought of not
swapping the door and staying at with my first choice because in the beginning
I had 33.33% probability of choosing the correct door. Now, after one door was
opened I have 50% chance of choosing the correct door. Even though this
explanation is sensible it is wrong.
For this we have to use
Bayes theorem.
The prior probability of the
car being behind door no. C was 1/3.
However, after the game host
opens the door no. B and shows the goat, the probability that the car is behind
door C given that the door B was opened increases to 2/3 (using Bayes formula).
No matter which door I
choose in the beginning and the game host opens, by using Bayes theorem always
swapping the door gives you 66.66% probability of getting the car.
4comments
이상구(LEE SANGGU)November 3, 10:53 AM
Others may add your comment on Mr. 기리`s work.
김다솔(2018####24)November 4, 2:28 PM
It is simple and understandable. Thank you Giri.
기리프라바스(2021####24)November 4, 5:19 PM
I am glad that you understood.
김다니일(2021####39)December 1, 00:35 AM
Was always fascinated, even though Monty Hall Problem is
probably the most famous probability problem, there are so much ways you can
explain it. Thank you for your variant. Simple and neat!
W10
(Daniil Kim). Summary, Chapter 3 Open Problems: 1
Summary:
This week, we revised the basic
combinatorics and probability theory, as well as covered a bit of Discrete
Mathematics.
Factorial of 
is a recursive product of all
positive integers
. 
. Note: 
.
Permutations is a number of ways, we can choose 
elements out of elements, with the
consideration of their order. There are two formulas:
3. Without Repetitions: (all chosen elements are distinct)
4. With Repetitions: (all chosen elements are not necessarily distinct)
Combinations is a number of ways, we can choose elements out of elements, but without
the consideration of their order. There are two formulas:
3. Without Repetitions: (all chosen elements are distinct)
4. With Repetitions: (all chosen elements are not necessarily distinct)
In probability theory, a sample space or probability space is a set of possible outcomes.
A general formula for the mathematical probability is:
But in reality, in order to determine the statistical probability, we repeat the same experiment a number of times, and then the calculate the probability using the usual formula.
It is important to note, that
the statistical probability is not equal to the mathematical probability, but
it can approach, and theoretically become equal, as the number of repetitions
increases and becomes sufficiently large. This property is called the ‘Law of
Large Number’: 
Bayes Theorem describes the probability of the occurrence of a particular event, considering the conditions, related to this event.
A discrete probability distribution describes the probability of occurrence of each value of a discrete (or in other words, explicitly defined) random variable. We define the Probability Mass Function that states:
·
Probability of occurrence of
any outcome should be in range of 
· Sum of all probabilities should equal to 1
A continuous probability
distribution describes the probability of occurrence of each value of a
continuous (or in other words, not explicitly defined or conditionally defined,
e.g., 
) random variable. In this case, we define the Probability
Density Function:
·
For all real 
·
Chapter 3 Open Problems:
#1:
My initial take on the infamous
Monty Hall problem, after reading about the Bayes Theorem was that when we
initially choose any of the doors, each door has 1/3 probability of a prize
being behind it.
By choosing the door, we sort of
"lock" the probability in place, and when Monty Hall opens an empty
door (or door with goat), it looks like nothing should change, but according to
the PMF, the sum of the probabilities should = 1, whereas if we go with our
assumption of the same probabilities, we get 1/3 + 1/3 = 2/3, which contradicts
our statement. Hence, it means that one of the doors now has an increased
probability of 2/3. But which one?
The answer is the second one,
because as I said, the door that we chose has a "locked" probability,
so the only "free" door left is the 2nd one, hence we get these
probabilities:
|
Door you chose |
1/3 |
|
The other door |
2/3 |
That makes the choice obvious: we
always need to change the door, to get the increased probability of 2/3 ~
66.67%.
I also did some research and
stumbled upon an amazing article, that explains everything more
thoroughly: https://towardsdatascience.com/monty-hall-problem-solution-using-bayes-theorem-cb1d6fbc0c9e
W11
(Daniil Kim). Summary, Chapter 3 Open Problems: 2,3
Summary:
Expectation of a random variable is an average value, that we can get from random variable. It is also called a mean value.
Example: We can calculate the expected value of a dice:
Here, we put 
, because the probability of each number on the standard dice is the
same and equals to .
This means, that if you roll a
dice, large enough number of times, and average the results, you will approach 
.
Variance of a random variable is the spread of a set of data, in relation to their average value.
Standard Deviation is the square root of its variance.
However, keep in mind, that the formulas mentioned above are only suitable for the discrete random variables.
For continuous we have a different set of formulas:
There are multiple properties of Variance and Expectation:
These are very easy to remember, because you basically extract the constant
expressions out of the brackets.
Week
11-Summary (Pravas Giri)
Author : 기리프라바스(2021####24)Date : November
11, 9:31 AM
Count : 12
Summary:
Expectation:
The
expectation of a random variable is an average value for probabilistic events,
which is the sum of products of a value obtained by each event and each event's
probability.
For
discrete random variable X:
For
continuous random variable X:
Variance:
It
is a measure of the dispersion of numbers (data), which indicates how far a set
of numbers (data) are spread out from their average value.
For
discrete random variable X:
For
continuous random variable X:
Standard deviation:
It
defined as a square root of variance.
For
discrete random variable X:
For
continuous random variable X:
Covariance matrix:
It
is created by using the variance and covariance of each random variable.
The
covariance matrix is a square matrix with the variance in the main diagonal and
covariance of two variables in off-diagonal.
It
plays an important role in dimension reduction, effectively reducing the
dimension while maintaining the distribution of high-dimensional data as much
as possible.
4comments
이상구(LEE
SANGGU)November 11, 9:35 AM
Good.
If you can make a Covariance matrix from your data. You will be on your way in
PCA.
기리프라바스(2021####24)November
11, 11:22 AM
Thank
you sir. I have made covariance matrix and uploaded it in QnA as open problem 3
with the code you have provided.
이상구(LEE
SANGGU)November 12, 1:10 PM
Add
comment on other's solutions.
김다니일(2021####39)December
1, 00:43 AM
Short,
simple and understandable. Thank you!
Standardized Random Variable:
We can treat Standardizing Random Variables as “Normalization” (similar to what we did to vectors). The expected value of a Standardized Random Variable is always 0, and its variance is 1. Standardizing makes it easier to compare variables of different types and units.
Joint Probability Distribution shows a probability distribution of multiple variables and their relationship.
In the Joint Probability Distribution, we have a Marginal Density Function, which basically a density function of a one particular variable.
Covariance is a measure of the joint variability of two random variables.
Correlation Coefficient is the measure of relationship strength between the two variables.
Covariance Matrix is a square matrix with the variance in the main diagonal and all covariances in non-diagonal entries. Any covariance matrix is symmetric and positive semi-definite. It visualizes the data distribution.
박준호
Open Problem 1.
A : A가 당첨인 사건(처음 고르는 문)
B : B가 당첨인 사건
C : C가 당첨인 사건
b : B가 공개되는 사건
c : c가 공개되는 사건
b가 공개되고 바꾸지 않았을 때 당첨확률
P(A|b) =
P(A)*P(b|A) / (P(A)*P(b|A) + P(B)*P(b|B) + P(C)*P(b|C)) = (1/3 * 1/2) / (1/3 *
1/2 + 1/3 * 0 + 1/3 * 1) = 1/3
P(A) = P(B) = P(C)
= 1/3
P(b|A) = 1/2 (A 선택 - A 당첨 -> B 또는 C 공개)
P(b|B) = 0 (A선택 - B 당첨 -> C 공개)
P(b|C) = 1 (A선택 - C 당첨 -> 반드시 B 공개)
b가 공개되고 바꾸었을 때(이 경우 반드시 C 선택/ ( P(B|b) =
P(B)*P(b|B) = 1/3 * 0 = 0 )) 당첨확률
P(C|b) =
P(C)*P(b|C) / (P(A)*P(b|A) + P(B)*P(b|B) + P(C)*P(b|C)) = 2/3
c가 공개되고 바꾸지 않았을 때 당첨확률
P(A|c) =
P(A)*P(c|A) / (P(A)*P(c|A) + P(B)*P(c|B) + P(C)*P(c|C)) = 1/3
c가 공개되고 바꾸었을 때 당첨확률
P(B|c)
= P(B)*P(c|B) / (P(A)*P(b|A) + P(B)*P(b|B) + P(C)*P(b|C)) = 2/3
4comments
이상구(LEE SANGGU)November 8, 4:13 AM
Good. You made it. You are the ... one who did solve some
problems of Week 10. ^^ Add your comment to others works in the below.
이상구(LEE SANGGU)November 11, 8:39 PM
[Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문 10-1, 순열, 조합, 확률, 조건부확률, https://youtu.be/eOVVKFB3eFc [Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문 7-1, 극한과 도함수, https://youtu.be/nm7sKIfnWro [Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문 4-2, 선형연립방정식의
해집합 구하는 방법, https://youtu.be/xT8tqbjq2Ig [Matrixtopia] 고교생과
일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문, 인공신경망과
오차역전파법, https://youtu.be/T6OSpzz00HU [Matrixtopia] 고교생과
일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문, 주성분분석과
차원축소, https://youtu.be/Skyi06WVRgA [Matrixtopia] 고교생과 일반인을
위한 [K-MOOC] 인공지능을 위한 기초수학 입문, 공분산행렬, https://youtu.be/MSu5_ehP_Ug [Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문, 경사하강법과 최적해, https://youtu.be/IADbbl_iAwI [Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학 입문, 특잇값분해(SVD)의 이해와 계산, https://youtu.be/0aCnKbrVyhQ
[Matrixtopia] 고교생과 일반인을 위한 [K-MOOC] 인공지능을 위한 기초수학
입문, QR 분해를 이용하여 최소제곱해를 구하는 방법,
https://youtu.be/iAnKfEiph7Y [Matrixtopia] 고교생과 일반인을 위한
[K-MOOC] 인공지능을 위한 기초수학 입문, 최소제곱해를 구하는 방법, https://youtu.be/xGZPBT4Q4t4 [Matrixtopia] SKKU Math4AI, 인공지능을
위한 기초수학, 강의계획서 설명,
https://youtu.be/p8eGmEbm6qk [Matrixtopia] 인공지능이란 무엇인가?
(Dummy를 위한 AI) https://youtu.be/F1HNFGAMhro
기리프라바스(2021####24)November 15, 11:57 AM
The calculation part made it more clearer for me to understand.
Thank you.
김다니일(2021####39)December 1, 00:42 AM
Short, simple and understandable. Thank you.
<Week 11- Open Problem 2
Correction Using Kim Daniels Code- Pravas Giri> THANK YOU KIM DANIEL
Author : 기리프라바스(2021####24)Date : December 4, 11:17
AM
Count : 1
#OPEN PROBLEM
2
# CORRECTION
USING KIM DANIELS CODE
#
Probability Density Function
f(x) = 4*x^3
xmin = 0
xmax = 1
show("Initial Probability Density Function: f(x)=", f(x))
# Expectation
Ex = integral(x*f(x), x, xmin, xmax)
tempEx = integral(x^2*f(x), x, xmin, xmax)
show("Expectation: E(X)=", Ex)
Var(X) = tempEx - Ex^2
show("Variance: Var(X)=", Var(X))
SD(X) = sqrt(Var(X))
show("Standard Deviation: SD(X)=", SD(X))
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Output
2comments
이상구(LEE
SANGGU)December 4, 11:18 AM
Good.
You are back. PBL due was extended 24 hours. Good luck.
기리프라바스(2021####24)December
4, 12:06 PM
Thank
you sir.
Open Problems:
#2:
# W11 (Kim Daniil) Chapter 3 Open
Problem 2
# Probability Density Function
f(x) = x^5+e^(3*x)
xmin = -0.5
xmax = 1
show("Initial Probability
Density Function: f(x)=", f(x))
# Expectation
Ex = integral(x*f(x), x, xmin,
xmax)
tempEx = integral(x^2*f(x), x,
xmin, xmax)
show("Expectation:
E(X)=", Ex)
Var(X) = tempEx - Ex^2
show("Variance:
Var(X)=", Var(X))
SD(X) = sqrt(Var(X))
show("Standard Deviation:
SD(X)=", SD(X))
Code:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
#3:
# W11 (Kim Daniil) Chapter 3 Open
Problem 3
import numpy as np
# Randomly generate Matrix
size = [5, 8]
M = matrix(
[
[ np.random.randint(0, 11) for i in
range(size[0]) ] for x in range(size[1])
]
)
show("Initial Matrix: M
=", (M))
tempM = zero_matrix(RDF, size[1],
size[0])
for y in range(size[1]):
m = mean(M[y])
for x in range(size[0]):
tempM[y,x]=M[y,x] - m
show("Covariance Matrix: CM
=", (1/(size[0]-1)*tempM*tempM.transpose().n(digits=5)))
Code: We can practice the above code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
김다니일(2021####39)November 16, 11:13 AM
Regarding
the Computing Power, if data analysis takes too much time, you can easily get a
free 3 month trial on Google Computing Engine and rent a very powerful Linux
server for 300 free credits (a quad-core 16GB RAM Ubuntu Pro server costs ~100
credits a month). Although, you would need some prior Unix knowledge, because
there is no GUI, and you would have to use Terminal and SSH to communicate with
server. But you can easily setup a Jupyter Notebook server (I think that is
what you are using here) there. https://cloud.google.com/compute
W12
(Daniil Kim). Summary. Chapter 4 Open Problem 1
Summary:
The Principal Component Analysis (PCA) is
one of the most widely used dimensionality reduction techniques – the
transformation of data from a high-dimensional space into a low-dimensional space
so that the low-dimensional representation retains some meaningful properties
of the original data. The PCA tries to combine existing variables to define the
new variables, called Principal
Components, while minimizing the information loss.
After PCA, it is much easier to
compare data and requires less computing power.
There are two main ways to
perform a PCA:
- via
eigen-decomposition and covariance matrices. I recommend reading that
article, for the thorough understanding of that method: https://medium.com/analytics-vidhya/understanding-principle-component-analysis-pca-step-by-step-e7a4bb4031d9
- using
SVD on the initial data matrix. This method is sometimes preferrable over
the eigen-decomposition because eigenvalue decomposition can only be used
for the diagonalizable matrices, whereas SVD can be used for any matrix.
More on that topic: https://jonathan-hui.medium.com/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491
Chapter
4 Open Problem 1:
SVD not only more robust and reliable, but also makes it easier
to determine Principal Components. By doing SVD, we can manually choose the
proportion that preserves the initial data distribution and select that
dimension as the basis for our PCA.
Week-11
Open problem 3 (Pravas Giri)
Author : 기리프라바스(2021####24)Date : November
11, 9:36 AM
Count : 12
Open Problem 3
code:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Open Problem 3
output:
4comments
이상구(LEE
SANGGU)November 11, 8:41 PM
Nice
try ^^
이상구(LEE
SANGGU)November 12, 1:10 PM
Add
comment on other's solutions.
김다니일(2021####39)November
12, 4:00 PM
I
would like to point out, that I actually found that np.array(X.row(row)) is a
redundant conversion. You can just replace it with X[row], and everything would
work out fine with a higher performance. Although, it is not really critical,
since on small matrices <20x20, I did not really notice the big difference,
I only noticed it with a very large matrix, so I do not think that it is
something to worry about, just wanted to mention.
기리프라바스(2021####24)November
15, 11:56 AM
Thank
you for the suggestion. I will remember it as our program should be accurate as
well as fast.
Finalized by Kim Daniil,
Pravas Giri, 김다솔 Week 12 < Open Problem 1>
PRAVAS GIRI
Principle Component Analysis (PCA):
It is difficult to analyze multi-dimensional data due to its
large number. In this case it would be much easy to extract and only use some
data that seems explain the data. However, data selection in this manner might
not purely reflect the attributes of the data. Therefore, PCA is needed. PCA is
process of analysis used to determine which feature of the element should be
mainly used.
Open Problem 1:
After deciding a data, we construct a matrix X using that data.
Then the centered matrix is found by differencing X and the matrix having mean
of the each column in matrix X.
Now, we calculate the SVD of mean-centered matrix X. Here, we
have U, V which are orthogonal matrices and S with is a diagonal matrix with
decreasing singular values. Then, the column of vector V becomes the principle
axes and the product of U and S denoted by Z becomes principle component. In
this way by computing the proportion that preserves the distribution of the
original data for each principal component we can decide the reduced
dimension.
1comments
이상구(LEE SANGGU)November
24, 12:50 PM
Other classmate shoud add
your comment on this problem/solution and Finilize it. In order to pass this
course, you will start to make your Final PBL Report.
김다솔
Open Problem 1, Explain how
does SVD work in PCA.
The most important property
of SVD is capability of partial re-composition. It means you can operate
recovery with partial SIGMA matrix from decomposition A = U SIGMA V.
This property is caused by
that the matrix SIGMA is "ordered". You can keep information of
original matrix A as much as possible, choosing upper some values in
SIGMA.
For simple example, you can
discard all 0 values in the lower of SIGMA matrix, keeping whole original
matrix.
The partial re-composition
property of SVD has a main role in PCA. PCA transforms a data matrix following
principal axes.
The number of principal axes
can be chosen manually. It means we can choose principal axes and discard minor
axes.
This property is from SVD
used in the middle of PCA. In real, choosing principal axes is choosing upper
SVD elements because the principal axes are in V matrix from the decomposition
A = U SIGMA V.
When SIGMA becomes smaller,
then the decomposition (or re-composition) represents like this: U SIGMA' V =
A' where SIGMA' is SIGMA matrix changing under elements as 0.
Prev
Next
2comments
이상구(LEE SANGGU)November
15, 2:28 PM
Reasonable explanation ^^
기리프라바스(2021####24)November
24, 12:45 PM
Thank you for the detailed
explanation.
김다니일
11. Summary:
The Principal
Component Analysis (PCA) is one of the most widely used
dimensionality reduction techniques – the transformation of data from a
high-dimensional space into a low-dimensional space so that the low-dimensional
representation retains some meaningful properties of the original data. The PCA
tries to combine existing variables to define the new variables, called Principal
Components, while minimizing the information loss.
After PCA, it is much easier
to compare data and requires less computing power.
There are two main ways to
perform a PCA:
- via eigen-decomposition and covariance
matrices. I recommend reading that article, for the thorough understanding
of that method: https://medium.com/analytics-vidhya/understanding-principle-component-analysis-pca-step-by-step-e7a4bb4031d9
- using SVD on the initial data matrix.
This method is sometimes preferrable over the eigen-decomposition because
eigenvalue decomposition can only be used for the diagonalizable matrices,
whereas SVD can be used for any matrix. More on that topic: https://jonathan-hui.medium.com/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491
12. Chapter 4 Open
Problem 1:
SVD not only more robust and reliable, but also makes it easier
to determine Principal Components. By doing SVD, we can manually choose the
proportion that preserves the initial data distribution and select that
dimension as the basis for our PCA.
2comments
이상구(LEE SANGGU)November
18, 6:43 PM
Good. I recommend you to add
your comment or ask question on OTHER'S summary or Solutions to pass this class
as I mentined many times.
기리프라바스(2021####24)November
24, 12:48 PM
Thank you for the
explanation.
2comments
이상구(LEE SANGGU)November
24, 12:50 PM
Other classmate shoud add
your comment on this problem/solution and Finilize it. In order to pass this
course, you will start to make your Final PBL Report.
김다니일(2021####39)December
1, 00:47 AM
Thank you for finalizing!
[Final OK by SGLee] Finalized
by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 2>
목록으로 이동
[Final OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 2>
박준호
다른 학우들의 과제를 잘 보았습니다.
그러나 전반적으로 관심사나 전공이 달라서 그런지 PCA를 통해 데이터를 압축할 수 있다는 사실과 별개로
데이터 자체에 대한 설명이 있었다면 더 좋았을 것 같았습니다.
그래서 저는 다른 것보다도 데이터 자체에 대한 저의 생각을 좀더 첨가해보았습니다.
사용한 데이터는 금융과목에서 사용한 바 있는 샘플 주식 가격(지수)/수익률 데이터입니다.
"Refinitve(https://www.refinitiv.com/ko)"이라는 유료 데이터 제공업체의 data stream 자료입니다.
샘플되지 않은 자료는 지금 저의 수준으로 다루기 어려울 것 같아 부득이 샘플된 자료를 사용하였습니다.
1열은 날짜이고, 15~27은 각 회사의 주가 혹은 주요 시장지표이고, 2~14는 해당 대상들의 수익률입니다.
2~14는 (y_t - y_t-1)/y_t-1 로 정의되기 때문에, 당연히 첫 행의 데이터가 존재하지 않습니다.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
R을 이용해 centered matrix를 구하고
SVD와 sigma square을 통해 분산들을 구하면 다음과 같습니다.
하지만 원본 데이터와의 유사성을 위해 꽤 많은 주성분들이 필요한 것을 알 수 있습니다.
12개의 주성분이 있어야 비로소 90% 이상의 분산을 보존할 수 있습니다.
김다솔 학우님이 사용한 data set은 단 하나의 주성분만으로 80% 이상의 분산을 보존할 수 있던 것과는 대조적입니다.
이때, 2~14열의 자료들은 사실 15~27열의 자료들을 통해 계산될 수 있다는 점을 고려해서
15~27열의 자료만을 대상으로 다시 PCA를 사용해보았고 위의 결과를 얻을 수 있었습니다.
이 경우 13개 중 4개의 주성분만으로 90% 이상의 분산을 보존할 수 있음을 확인할 수 있습니다.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
흥미로웠던 점은 2~14열의 수익률 자료만을 가지고 PCA를 하자, 전체 데이터 이상으로 무시할만한 성분들이 적어지는 것을 확인할 수 있었습니다.
사실 초기값(initial value)이 주어진다는 전제 하에서 수익률 데이터와 가격 데이터는 사실상 동일함에도 불구하고
PCA에서는 매우 상반되는 과정, 심지어는 결과물을 낼 수 있다는 사실이 몹시 흥미로웠습니다.
아마도 수익률은 주가에 비해서 훨씬 무작위적이기 때문에 그렇다고 추측해볼 따름이지만, 그 이유에 대해 다른 학우들의 의견도 들어보고 싶습니다!
PRAVAS GIRI
Open Problem 2:
Source: https://data.world/sdhilip/pizza-datasets/workspace/file?filename=Pizza.csv
Code:
# Enter the data matrix.
# Here, each column represents data corresponding to the variables Price,
Software, Aesthetics, and Brand.
# data matrix
X = matrix(RDF, [[27.82,21.43,44.87,5.11,1.77,0.77,4.93],
[28.49,21.26,43.89,5.34,1.79,1.02,4.84],
[28.35,19.99,45.78,5.08,1.63,0.8,4.95],
[30.55,20.15,43.13,4.79,1.61,1.38,4.74],
[30.49,21.28,41.65,4.82,1.64,1.76,4.67],
[31.14,20.23,42.31,4.92,1.65,1.4,4.67],
[31.21,20.97,41.34,4.71,1.58,1.77,4.63],
[28.76,21.41,41.6,5.28,1.75,2.95,4.72],
[28.22,20.48,45.1,5.02,1.71,1.18,4.93],
[27.72,21.19,45.29,5.16,1.66,0.64,4.95],
[27.35,21.2,45.59,4.94,1.65,0.92,4.98],
[26.98,21.2,45.03,5.15,1.67,1.64,4.97],])
print("X =")
print(X)
print() # One line indent
# centering
# scaling
n = X.nrows() # (number of rows)
p = X.ncols() # (number of columns)
X_ctr = zero_matrix(RDF, n, p) # Prepare memory for centered matrix
import numpy as np
for col in range(p):
v = np.array(X.column(col))
m = mean(v)
st = std(v)
for row in range(n):
X_ctr[row,col] = (X[row,col] - m)/st
# (X_ctr is a centered matrix)
print("X_ctr =")
print(X_ctr.n(digits = 4))
print() # (One line indent)
# Singular Value Decomposition(SVD). X_ctr = U*S*V.transpose()
U, S, V = X_ctr.SVD()
Var_PC = [(S[i, i]^2/n).n(digits = 4) for i in range(p)] # PC variance
Prop_Var = [100*Var_PC[i]/sum(Var_PC) for i in range(p)] # Ratio of variance
print("PC variance ", Var_PC)
print("Ratio of variance ", Prop_Var)
print() # (One line indent)
# Scree Plot. Visually show variance. Decide to what dimension to
reduce.
show(list_plot(Var_PC, plotjoined = True, color = 'red', axes_labels = ['',
'Variances']))
print() # (One line indent)
# Compute PC score. Z = U*S
# Calculate the PC score. Z = U*S
PC_score = U*S
PC12 = PC_score[:,0:2]# (Use only the first and second components for
visualization)
# (Visualize the data on a plane.)
point([PC12.row(i) for i in range(n)], color = 'red')
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Output:
X =[27.82 21.43 44.87 5.11 1.77 0.77 4.93][28.49 21.26 43.89 5.34 1.79 1.02 4.84][28.35 19.99 45.78 5.08 1.63 0.8 4.95][30.55 20.15 43.13 4.79 1.61 1.38 4.74][30.49 21.28 41.65 4.82 1.64 1.76 4.67][31.14 20.23 42.31 4.92 1.65 1.4 4.67][31.21 20.97 41.34 4.71 1.58 1.77 4.63][28.76 21.41 41.6 5.28 1.75 2.95 4.72][28.22 20.48 45.1 5.02 1.71 1.18 4.93][27.72 21.19 45.29 5.16 1.66 0.64 4.95][27.35 21.2 45.59 4.94 1.65 0.92 4.98][26.98 21.2 45.03 5.15 1.67 1.64 4.97] X_ctr =[ -0.7290 1.000 0.6295 0.4249 1.439 -0.9150 0.7250][ -0.2863 0.6797 0.05384 1.598 1.745 -0.5223 0.06144][ -0.3788 -1.713 1.164 0.2719 -0.7005 -0.8679 0.8724][ 1.075 -1.411 -0.3926 -1.207 -1.006 0.04320 -0.6758][ 1.035 0.7174 -1.262 -1.054 -0.5477 0.6401 -1.192][ 1.465 -1.261 -0.8742 -0.5439 -0.3948 0.07462 -1.192][ 1.511 0.1334 -1.444 -1.615 -1.465 0.6558 -1.487][ -0.1079 0.9623 -1.291 1.292 1.134 2.509 -0.8233][ -0.4647 -0.7896 0.7646 -0.03399 0.5222 -0.2710 0.7250][ -0.7951 0.5479 0.8762 0.6799 -0.2420 -1.119 0.8724][ -1.040 0.5667 1.052 -0.4419 -0.3948 -0.6794 1.094][ -1.284 0.5667 0.7234 0.6289 -0.08915 0.4516 1.020] PC variance [3.483, 1.841, 0.5706, 0.4104, 0.1111, 0.0003037, 5.267e-31]Ratio of variance [54.28, 28.69, 8.892, 6.396, 1.731, 0.004733, 8.209e-30]
Prev
Next
1comments
이상구(LEE SANGGU)November
24, 12:50 PM
Other classmate shoud add
your comment on this problem/solution and Finilize it. In order to pass this
course, you will start to make your Final PBL Report.
W12
(Daniil Kim). Summary. Chapter 4 Open Problem 2
Chapter 4 Open Problem 2:
Sources:
Data Source: https://data.world/exercises
Jupyter Source: https://colab.research.google.com/drive/1rMEHNEHV_21dhlX6tmt7ObfzZ9u91NJ_?usp=sharing
Prev
Next
3comments
이상구(LEE SANGGU)November
18, 8:10 PM
Add the address of your
source data^^
이상구(LEE SANGGU)November
19, 4:24 AM
Other classmate shoud add
your comment on this problem/solution and Finilize it. In order to pass this
course, you will start to make your Final PBL Report. With these academic
activities, you can add it in your Final Exam report. Then you will pass. Add your
comment and Finalize other's solutions and answers. You will learn from it a
lot.
기리프라바스(2021####24)November
24, 12:48 PM
Thank you.
Finalized
by Kim Daniil, Pravas Giri, 김다솔
Week 12 < Open Problem 1>
PRAVAS GIRI
Principle Component Analysis (PCA):
It is difficult to analyze multi-dimensional data due to its
large number. In this case it would be much easy to extract and only use some
data that seems explain the data. However, data selection in this manner might
not purely reflect the attributes of the data. Therefore, PCA is needed. PCA is
process of analysis used to determine which feature of the element should be
mainly used.
Open Problem 1:
After deciding a data, we construct a matrix X using that data.
Then the centered matrix is found by differencing X and the matrix having mean
of the each column in matrix X.
Now, we calculate the SVD of mean-centered matrix X. Here, we
have U, V which are orthogonal matrices and S with is a diagonal matrix with
decreasing singular values. Then, the column of vector V becomes the principle
axes and the product of U and S denoted by Z becomes principle component. In
this way by computing the proportion that preserves the distribution of the
original data for each principal component we can decide the reduced
dimension.
1comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
김다솔
Open Problem 1, Explain how does
SVD work in PCA.
The most important property of SVD
is capability of partial re-composition. It means you can operate recovery with
partial SIGMA matrix from decomposition A = U SIGMA V.
This property is caused by that
the matrix SIGMA is "ordered". You can keep information of original
matrix A as much as possible, choosing upper some values in SIGMA.
For simple example, you can
discard all 0 values in the lower of SIGMA matrix, keeping whole original
matrix.
The partial re-composition
property of SVD has a main role in PCA. PCA transforms a data matrix following
principal axes.
The number of principal axes can
be chosen manually. It means we can choose principal axes and discard minor
axes.
This property is from SVD used in
the middle of PCA. In real, choosing principal axes is choosing upper SVD
elements because the principal axes are in V matrix from the decomposition A =
U SIGMA V.
When SIGMA becomes smaller, then
the decomposition (or re-composition) represents like this: U SIGMA' V = A'
where SIGMA' is SIGMA matrix changing under elements as 0.
Prev
Next
2comments
이상구(LEE SANGGU)November 15, 2:28 PM
Reasonable
explanation ^^
기리프라바스(2021####24)November 24, 12:45 PM
Thank
you for the detailed explanation.
김다니일
Summary:
The Principal Component Analysis (PCA) is
one of the most widely used dimensionality reduction techniques – the
transformation of data from a high-dimensional space into a low-dimensional
space so that the low-dimensional representation retains some meaningful
properties of the original data. The PCA tries to combine existing variables to
define the new variables, called Principal
Components, while minimizing the information loss.
After PCA, it is much easier to
compare data and requires less computing power.
There are two main ways to
perform a PCA:
- via
eigen-decomposition and covariance matrices. I recommend reading that
article, for the thorough understanding of that method: https://medium.com/analytics-vidhya/understanding-principle-component-analysis-pca-step-by-step-e7a4bb4031d9
- using
SVD on the initial data matrix. This method is sometimes preferrable over
the eigen-decomposition because eigenvalue decomposition can only be used
for the diagonalizable matrices, whereas SVD can be used for any matrix.
More on that topic: https://jonathan-hui.medium.com/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491
Chapter 4 Open Problem 1:
SVD not only more robust and reliable, but also makes it easier
to determine Principal Components. By doing SVD, we can manually choose the
proportion that preserves the initial data distribution and select that
dimension as the basis for our PCA.
Prev
Next
2comments
이상구(LEE SANGGU)November 18, 6:43 PM
Good.
I recommend you to add your comment or ask question on OTHER'S summary or
Solutions to pass this class as I mentined many times.
기리프라바스(2021####24)November 24, 12:48 PM
Thank
you for the explanation.
Finalized
by Kim Daniil, Pravas Giri, 김다솔
Week 12 < Open Problem 2>
PRAVAS GIRI
Open Problem 2:
Source: https://data.world/sdhilip/pizza-datasets/workspace/file?filename=Pizza.csv
Code:
# Enter the data matrix.
# Here, each column represents data corresponding to the variables Price,
Software, Aesthetics, and Brand.
# data matrix
X = matrix(RDF, [[27.82,21.43,44.87,5.11,1.77,0.77,4.93],
[28.49,21.26,43.89,5.34,1.79,1.02,4.84],
[28.35,19.99,45.78,5.08,1.63,0.8,4.95],
[30.55,20.15,43.13,4.79,1.61,1.38,4.74],
[30.49,21.28,41.65,4.82,1.64,1.76,4.67],
[31.14,20.23,42.31,4.92,1.65,1.4,4.67],
[31.21,20.97,41.34,4.71,1.58,1.77,4.63],
[28.76,21.41,41.6,5.28,1.75,2.95,4.72],
[28.22,20.48,45.1,5.02,1.71,1.18,4.93],
[27.72,21.19,45.29,5.16,1.66,0.64,4.95],
[27.35,21.2,45.59,4.94,1.65,0.92,4.98],
[26.98,21.2,45.03,5.15,1.67,1.64,4.97],])
print("X =")
print(X)
print() # One line indent
# centering
# scaling
n = X.nrows() # (number of rows)
p = X.ncols() # (number of columns)
X_ctr = zero_matrix(RDF, n, p) # Prepare memory for centered matrix
import numpy as np
for col in range(p):
v = np.array(X.column(col))
m = mean(v)
st = std(v)
for row in range(n):
X_ctr[row,col] = (X[row,col] - m)/st
# (X_ctr is a centered matrix)
print("X_ctr =")
print(X_ctr.n(digits = 4))
print() # (One line indent)
# Singular Value Decomposition(SVD). X_ctr = U*S*V.transpose()
U, S, V = X_ctr.SVD()
Var_PC = [(S[i, i]^2/n).n(digits = 4) for i in range(p)] # PC variance
Prop_Var = [100*Var_PC[i]/sum(Var_PC) for i in range(p)] # Ratio of variance
print("PC variance ", Var_PC)
print("Ratio of variance ", Prop_Var)
print() # (One line indent)
# Scree Plot. Visually show variance. Decide to what dimension to
reduce.
show(list_plot(Var_PC, plotjoined = True, color = 'red', axes_labels = ['',
'Variances']))
print() # (One line indent)
# Compute PC score. Z = U*S
# Calculate the PC score. Z = U*S
PC_score = U*S
PC12 = PC_score[:,0:2]# (Use only the first and second components for
visualization)
# (Visualize the data on a plane.)
point([PC12.row(i) for i in range(n)], color = 'red')
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Output:
X =[27.82 21.43 44.87 5.11 1.77 0.77 4.93][28.49 21.26 43.89 5.34 1.79 1.02 4.84][28.35 19.99 45.78 5.08 1.63 0.8 4.95][30.55 20.15 43.13 4.79 1.61 1.38 4.74][30.49 21.28 41.65 4.82 1.64 1.76 4.67][31.14 20.23 42.31 4.92 1.65 1.4 4.67][31.21 20.97 41.34 4.71 1.58 1.77 4.63][28.76 21.41 41.6 5.28 1.75 2.95 4.72][28.22 20.48 45.1 5.02 1.71 1.18 4.93][27.72 21.19 45.29 5.16 1.66 0.64 4.95][27.35 21.2 45.59 4.94 1.65 0.92 4.98][26.98 21.2 45.03 5.15 1.67 1.64 4.97] X_ctr =[ -0.7290 1.000 0.6295 0.4249 1.439 -0.9150 0.7250][ -0.2863 0.6797 0.05384 1.598 1.745 -0.5223 0.06144][ -0.3788 -1.713 1.164 0.2719 -0.7005 -0.8679 0.8724][ 1.075 -1.411 -0.3926 -1.207 -1.006 0.04320 -0.6758][ 1.035 0.7174 -1.262 -1.054 -0.5477 0.6401 -1.192][ 1.465 -1.261 -0.8742 -0.5439 -0.3948 0.07462 -1.192][ 1.511 0.1334 -1.444 -1.615 -1.465 0.6558 -1.487][ -0.1079 0.9623 -1.291 1.292 1.134 2.509 -0.8233][ -0.4647 -0.7896 0.7646 -0.03399 0.5222 -0.2710 0.7250][ -0.7951 0.5479 0.8762 0.6799 -0.2420 -1.119 0.8724][ -1.040 0.5667 1.052 -0.4419 -0.3948 -0.6794 1.094][ -1.284 0.5667 0.7234 0.6289 -0.08915 0.4516 1.020] PC variance [3.483, 1.841, 0.5706, 0.4104, 0.1111, 0.0003037, 5.267e-31]Ratio of variance [54.28, 28.69, 8.892, 6.396, 1.731, 0.004733, 8.209e-30]
Prev
Next
1comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
김다니일
Chapter
4 Open Problem 2:
I found a practice data sample of
different food and amount of different kinds of vitamins, nutritions, etc in
them.
Sources:
Data Source: https://data.world/exercises
Jupyter Source: https://colab.research.google.com/drive/1rMEHNEHV_21dhlX6tmt7ObfzZ9u91NJ_?usp=sharing
Prev
Next
3comments
이상구(LEE SANGGU)November 18, 8:10 PM
Add
the address of your source data^^
이상구(LEE SANGGU)November 19, 4:24 AM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report. With
these academic activities, you can add it in your Final Exam report. Then you
will pass. Add your comment and Finalize other's solutions and answers. You
will learn from it a lot.
기리프라바스(2021####24)November 24, 12:48 PM
Thank
you.
김다솔
Open Problem 2 - Find a data
matrix from your major field. Apply PCA algorithm to the data table and debate
about it.
1) Download RNA expression of
cancer patients data from GDC data portal.
2) Preprocessing
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
3) Data matrix has 60k expression
probe (TPKM) and 174 samples
4) Centering
5) Components are limited by
sample number because the data sample size is smaller than the feature size.
and I choose 34 upper components.
It is impossible to show 34
dimensions. Question is first component has 82% PC score. It is too high, I
think.
Prev
Next
2comments
김다니일(2021####39)November 16, 11:13 AM
Regarding
the Computing Power, if data analysis takes too much time, you can easily get a
free 3 month trial on Google Computing Engine and rent a very powerful Linux
server for 300 free credits (a quad-core 16GB RAM Ubuntu Pro server costs ~100
credits a month). Although, you would need some prior Unix knowledge, because
there is no GUI, and you would have to use Terminal and SSH to communicate with
server. But you can easily setup a Jupyter Notebook server (I think that is
what you are using here) there. https://cloud.google.com/compute
기리프라바스(2021####24)November 24, 12:44 PM
Thank
you. Nice work.
Final OK by SGLee] Finalized
by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 3> <PCA
for Dimensionality Reduction> and <PCA and covariance matrix.>
[Final OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 3> <PCA for Dimensionality
Reduction> and <PCA and covariance matrix.>
PRAVAS GIRI
Open Problem 3:
(ENG) Debate the process of dimension reduction, using PCA with
covariance matrix.
(KOR) 공분산 행렬에 대한 주성분분석(PCA)을 하여, 차원을 축소하는 과정에 대하여 이해한 바를 토론하시오.
Dimension reduction is a technique of reducing high-dimensional
data into low-dimensional data. By using the process of dimension reduction by
PC's on the covariance matrix, we find the smallest number of new variables
that can preserve the information from the covariance matrix as much as
possible.
KIM DASOL Open Problem 3
공분산 행렬은 데이터의 분포를 설명해준다. 만약 공분산 행렬을 크기 순으로 정렬해 분산의 크기가 큰 (즉, 정보가 많은) 변수를 유지하고, 그렇지 않은 변수들은 버린다면, 정보를 최대한 유지한 채로 변수들의 숫자를 줄일 수 있게 된다. 즉, 원정보를 유지하며 차원을 축소 시킬 수 있게 된다. 그리고 이를 달성하기 위해 공분산 행렬에 특잇값 분해를 시행하는 것이 다름아닌 PCA가 된다.
(ENG) Debate the process of dimension reduction, using PCA with
covariance matrix.
Covariance matrix explains the distribution of data. If we
remain bigger covariances and discard smaller, then we can reduce the number of
variables remaining the information of distribution as possible. It
means we can reduce the dimension of data. For achieving this, we should
operate singular value decomposition with covariance matrix, a.k.a PCA.
PCA and covariance matrix. PCA is simply described as
“diagonalizing the covariance matrix”. ... It simply means that we need to find
a non-trivial linear combination of our original variables such that the
covariance matrix is diagonal. When this is done, the resulting variables are
uncorrelated, i.e. independent. What is the significance of the covariance
matrix and eigen vectors in PCA?
The eigenvectors and eigenvalues of a
covariance (or correlation) matrix represent the “core” of a PCA:
The eigenvectors (principal components)
determine the directions of the new feature space, and the eigenvalues
determine their magnitude.
So, PCA is a method that: Measures how each variable is
associated with one another using a Covariance matrix. Understands the
directions of the spread of our data using Eigenvectors. Brings out the
relative importance of these directions using Eigenvalues.Sep 9, 2019 How is
covariance matrix used in PCA? The eigenvectors and eigenvalues of a covariance
(or correlation) matrix represent the “core” of a PCA: The eigenvectors
(principal components) determine the directions of the new feature space, and the
eigenvalues determine their magnitude.Jan 27, 2015
박준호
High dimension is a huge matter for computing.
from difference with average.
Covariances = addtional
information from difference with new axis (fitting line).
Therefore, small variances
are negligible comparing to computing efficiency by reducing dimension.
Kim Daniil
Open Problem 3:
As everyone above
mentioned. Dimension reduction by PCs (Principal Components) on the
covariance matrix is basically what we call PCA, and what we do with the
PCs and the covariance matrix is that we simply "diagonalize the
covariance matrix". In other words, what we do is we try to find a linear
combination of PCs, such that our covariance matrix will be diagonal.
Note: i just noticed that
the Prof. Lee Sang-Gu also commented the same thing. However, in the actual
article there is a lot more information, that can be helpful. So, I decided to
leave the source: http://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf
Finalized by Kim Daniil, Pravas Giri, 김다솔 Week
12 < Open Problem 3> <PCA for Dimensionality Reduction> and <PCA
and covariance matrix.>
KIM DASOL
Open Problem 3
(KOR) 공분산 행렬에 대한 주성분분석(PCA)을 하여, 차원을 축소하는 과정에 대하여 이해한 바를 토론하시오.
공분산 행렬은 데이터의 분포를 설명해준다. 만약 공분산 행렬을 크기 순으로 정렬해 분산의 크기가 큰 (즉, 정보가 많은) 변수를 유지하고, 그렇지 않은 변수들은 버린다면, 정보를 최대한 유지한 채로 변수들의 숫자를 줄일 수 있게 된다. 즉, 원정보를 유지하며 차원을 축소 시킬 수 있게 된다. 그리고 이를 달성하기 위해 공분산 행렬에 특잇값 분해를 시행하는 것이 다름아닌 PCA가 된다.
(ENG) Debate the process of
dimension reduction, using PCA with covariance matrix.
Covariance matrix explains
the distribution of data. If we remain bigger covariances and discard smaller,
then we can reduce the number of variables remaining
the information of distribution as possible. It means we can reduce
the dimension of data. For achieving this, we should operate singular value
decomposition with covariance matrix, a.k.a PCA.
1comments
기리프라바스(2021####24)November
24, 12:49 PM
It was well explained. Thank
you.
>> by 박준호
'분산의 크기가 큰 = 정보가 많은' 문구의 의미에 대하여,
(A)는 새로운 축과 원본 데이터 간의 정보 차이가 적지만, (B)는 많을 것이라는 생각이 들었습니다.
-------------------------------------------------------------------------------------------------------------------------
PRAVAS GIRI
Open Problem 3:
Dimension reduction is a
technique of reducing high-dimensional data into low-dimensional data. By using
the process of dimension reduction by PC's on the covariance matrix, we find
the smallest number of new variables that can preserve the information from the
covariance matrix as much as possible.
이상구(LEE SANGGU)11월
25일 오후 1:46
Principal Component Analysis for Dimensionality Reduction https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
Learn how to perform PCA by learning the mathematics
behind the algorithm and executing it step-by-step with Python! Lorraine Li May
25, 2019·10 min read In the modern age of technology, increasing amounts of
data are produced and collected. In machine learning, however, too much data
can be a bad thing. At a certain point, more features or dimensions can
decrease a model’s accuracy since there is more data that needs to be
generalized — this is known as the curse of dimensionality. Dimensionality
reduction is way to reduce the complexity of a model and avoid overfitting.
There are two main categories of dimensionality reduction: feature selection
and feature extraction. Via feature selection, we select a subset of the
original features, whereas in feature extraction, we derive information from
the feature set to construct a new feature subspace. In this tutorial we will
explore feature extraction. In practice, feature extraction is not only used to
improve storage space or the computational efficiency of the learning
algorithm, but can also improve the predictive performance by reducing the
curse of dimensionality — especially if we are working with non-regularized
models. Specifically, we will discuss the Principal Component Analysis (PCA)
algorithm used to compress a dataset onto a lower-dimensional feature subspace
with the goal of maintaining most of the relevant information. We will explore:
The concepts and mathematics behind PCA How to execute PCA step-by-step from
scratch using Python How to execute PCA using the Python library scikit-learn
Let’s get started! This tutorial is adapted from Part 2 of Next Tech’s Python
Machine Learning series, which takes you through machine learning and deep
learning algorithms with Python from 0 to 100. It includes an in-browser
sandboxed environment with all the necessary software and libraries
pre-installed, and projects using public datasets. You can get started for free
here!
이상구(LEE SANGGU)11월
25일 오후 1:48
PCA and covariance matrix. PCA is simply described as “diagonalizing the covariance
matrix”. ... It simply means that we need to find a non-trivial linear
combination of our original variables such that the covariance matrix is
diagonal. When this is done, the resulting variables are uncorrelated, i.e.
independent. What is the significance of
the covariance matrix and eigen vectors in PCA?
The eigenvectors and eigenvalues of a covariance (or correlation) matrix
represent the “core” of a PCA:
The eigenvectors (principal components) determine the directions of the
new feature space, and the eigenvalues determine their magnitude.
So, PCA is a method that: Measures how each variable is
associated with one another using a Covariance matrix. Understands the
directions of the spread of our data using Eigenvectors. Brings out the
relative importance of these directions using Eigenvalues.Sep 9, 2019 How is
covariance matrix used in PCA? The eigenvectors and eigenvalues of a covariance
(or correlation) matrix represent the “core” of a PCA: The eigenvectors
(principal components) determine the directions of the new feature space, and
the eigenvalues determine their magnitude.Jan 27, 2015
Kim Daniil
Open Problem 3:
As everyone above
mentioned. Dimension reduction by PCs (Principal Components) on the
covariance matrix is basically what we call PCA, and what we do with the
PCs and the covariance matrix is that we simply "diagonalize the
covariance matrix". In other words, what we do is we try to find a linear
combination of PCs, such that our covariance matrix will be diagonal.
Note: i just noticed that
the Prof. Lee Sang-Gu also commented the same thing. However, in the actual
article there is a lot more information, that can be helpful. So, I decided to
leave the source: http://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf
>> understanded by 박준호
linear combination of PCs : substitute
original orthgonal vectors to new axes.
PCA means "diagonlizing covariance
matrix" : ordering new axes.
이상구(LEE SANGGU)November
25, 1:46 PM
Principal Component Analysis
for Dimensionality Reduction
https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
...
이상구(LEE SANGGU)November
25, 1:48 PM
PCA and covariance matrix. PCA is simply described as
“diagonalizing the covariance matrix”. ... It simply means that we need to find
a non-trivial linear combination of our original variables such that the
covariance matrix is diagonal. When this is done, the resulting variables are
uncorrelated, i.e. independent. What is the significance of the covariance
matrix and eigen vectors in PCA? So, PCA is a method that: Measures how each
variable is associated with one another using a Covariance matrix. Understands
the directions of the spread of our data using Eigenvectors. Brings out the
relative importance of these directions using Eigenvalues.Sep 9, 2019 How is
covariance matrix used in PCA? The eigenvectors and eigenvalues of a covariance
(or correlation) matrix represent the “core” of a PCA: The eigenvectors
(principal components) determine the directions of the new feature space, and
the eigenvalues determine their magnitude.Jan 27, 2015
Prev
Next
2comments
김다니일(2021####39)December
1, 00:55 AM
Thank you for finalizing!
기리프라바스(2021####24)December
1, 10:50 AM
Thank you.
Finalized by Kim Daniil,
Pravas Giri, 김다솔, Week 12 <Open Problem 4> Is there a difference between
linear regression and least squares?
Finalized by Kim Daniil,
Pravas Giri, 김다솔, Week 12 <Open Problem 4> Is there a
difference between linear regression and least squares?
KIM DASOL
Open Problem 4
(KOR) 최소제곱직선과 회귀분석에서 사용하는 선형회귀 사이의 같은 점과 다른 점에 대하여 이해한 바를 토론하시오.
최소 제곱 해는 변수들 사이에 순서가 없다. 그러나 선형회귀에는 종속변수와 독립변수가 존재한다.
'직선'에 포커스를 맞추면 변수가 세 개 이상이 되면 최소제곱직선이라고 부를 수 없게 된다. (그러나 여전히 최소제곱해는 존재한다.)
선형회귀는 두 변수 사이의 회귀직선 뿐만 아니라 세 개 이상의 변수도 사용해 회귀분석을 진행하곤 한다. 이 때에도 여전히 종속변수와 독립변수들이 나뉘게 된다.
다변수에 대한 비교는 최소제곱해와 선형회귀의 차이를 뚜렷하게 나타내준다. 만약 100개의 변수가 있다면 최소제곱해는 100 차원의 벡터들이 최소 제곱 오차를 갖게 되는 99차원의 초평면을 해로 갖게 된다. 선형회귀는 1개의 종속변수에 99개의 독립변수가 가장 잘 맞도록 하는 회귀직선을 갖게 된다.
제 이해를 바탕으로 한 제 생각이니 더 많은 의견과 사실들을 댓글로 남겨주시길 바랍니다.
(ENG) Debate the same and
different things between least-squares line and linear regression in regression
analysis.
There is no order in the
variables of least square solutions. But, in linear regression, there are one
dependent variable and independent variables. Multivariate operating clears the
difference between LSE and linear regression. With 100 variables, LSE has
99-dimensional hyperplane as a solution. Linear regression has one regression
line fitting 99 independent variables to one dependent variable.
This is just my thought from
my understanding. Please let me know your thought and the facts about the
question with under comments! Thank you :)
2comments
이상구(LEE SANGGU)November
24, 12:50 PM
Other classmate shoud add
your comment on this problem/solution and Finilize it. In order to pass this
course, you will start to make your Final PBL Report.
기리프라바스(2021####24)November
24, 12:50 PM
Thank you.
PRAVAS GIRI
Open Problem 4:
Linear regression attempts to model the relationship between two
variables by fitting a linear equation to observed data.
Least square line is the line that makes the vertical distance from the data
points to the regression line as small as possible .
What
is the difference between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares
In common parlance, they seem to be used interchangeably. But
more precisely, least squares is a method for performing linear
regression.
For now*, we can think of linear regression as the task of
fitting a straight line (or, in the case of multiple linear regression, a
"hyperplane") through a set of points. But there are many possible
strategies to fit a line through a set of points:
·
you could take the leftmost point and the rightmost point and
draw a line between them
·
you could compute the slopes of the lines connecting each pair
of points and calculate the average slope, drawing a line with this slope that
passes through the point at the average of the "x values" and the
average of the "y values"
·
you could find the line for which there are an equal number of
points above the line and below the line
·
you could draw a line, and then for each of the data points,
measure the vertical distance between the point and the line, and add these up;
the fitted line would be the one where this sum of distances is as small as
possible
·
you could draw a line, and then for each of the data points,
measure the vertical distance between the point and the line, square it, and
add these up; the fitted line would be the one where this sum of distances is
as small as possible
The last strategy is called "ordinary least squares" (because you are
trying to minimize the sum of squared prediction errors) and it is the most
commonly used (wondering why least squares is most commonly used? see * below)
. But as far as fitting a line through a set of points goes, any of the other
strategies are equally valid. The first three strategies I made up as examples
and probably do not perform well, but the fourth one is a real strategy called
"least absolute deviations" that some people prefer over least
squares.
* I want to clear up an important distinction that I glossed
over in the above:
First, precisely speaking there is a difference between linear
regression and curve fitting. Everything I said above is
actually talking about curve fitting, not linear regression: you have a set of
points, and you want to draw a curve (line) through them that fits as well as
possible. This is a purely geometric problem.
The x and y axes have no interpretation, and the "data" are just
points in Cartesian space.
On the other hand, linear regression is a statistical
inference problem. The "y values" take on the interpretation
of data you wish to model, and the "x values" take on the
interpretation of extra information you have about each data point that might
be helpful in predicting their "y values". You are trying to build a
probabilistic model that describes "y" while taking into account
"x", and a linear model is one of many ways to do this. A linear
model assumes that "y" has a different mean for each possible value
of "x", and that these means happen to follow a straight line with a
certain intercept and a certain slope. As with any statistical inference
problem, you estimate the unknown parameters using maximum likelihood
estimation. But since in this case the unknown parameters are an intercept and
a slope, the end result of maximum likelihood estimation is basically that you
are choosing a straight line that fits the observed data best, so this
essentially becomes the curve fitting problem discussed above.
Now we arrive at the question of why least squares, of all
possible curve fitting methods, is so commonly used. The reason is that when
solving the statistical linear regression problem, a very
common modeling assumption is that for every possible value of "x",
the quantity "y" is normally distributed with a mean
that is linear in "x". Therefore, the likelihood function is
essentially a product of PDFs of the normal distribution. As stated above, you
estimate the unknown parameters (and therefore find the best fitting line) by
maximizing the likelihood function. If you look at what the product of normal PDFs
looks like, you will notice that maximizing this expression happens to be
equivalent to... you guessed it... minimizing the sum of squared errors!
That is, the line you get
performing curve fitting via least squares is equivalent to the line you get performing
linear regression using a normal model.
Prev
Next
1comments
이상구(LEE SANGGU)November
25, 1:44 PM
What is the difference
between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares
Profile photo for Raj Bhuptani Raj Bhuptani , Harvard '13 (Statistics), Two
Sigma Investments Answered Feb 16, 2014 · Upvoted by Vladimir Novakovski ,
started …
Kim Daniil
Open Problem 4:
Although two terms have
similar names, they are in fact different:
- Least-Squares solution gives us optimal solutions to the equation,
and we find it via different Matrix Mutations.
- Least-Squares Regression Line is a
linear function, that visualizes the line, that has a
minimal variance (sum of squares of the errors or in other
words, vertical distances). And we usually find it via Calculus, or
Slope/Tangent and Intercept evaluation to be precise.
However, despite that, we can find the Least-Squares Regression Line by finding the
Least-Squares Solution. Prof. Lee Sang-Gu discussed it in the previous lessons
and defined it as (the least-squares line, the
best fit line).
Week
13- Pravas Giri <Open Problem 5>
Author : 기리프라바스(2021####24)Date : November
17, 10:42 PM
Count : 11
Open
Problem 5:
Artificial
Neural Network:
ANN refers to
Artificial Neural Network. A neural network is like a neuron of our nervous
system. It receives multiple input signals and gives one output signal.
Artificial neurons receive input data from other
artificial neurons or from outside. The input artificial neurons combines with
given weights and creates an output by linearly combining a single value with
the weight and adding the bias. However, incase of two outputs it can be
expressed using matrix product. We use x^T*A*T=y^T as this is the format
in statistics or artificial intelligence. Then we substitute the value we
obtained through linear combination in activation function and output data is
obtained.
Backpropagation:
The process of creating and updating weights
between hidden layers in a deep learning is called a backpropagation.
An input is entered and the weight of artificial
neural network is set randomly. When the hidden layer receives input and
computes with the weight then we obtain output through the activation function.
There will be an error between the predicted output and obtained output. We
then use gradient descent method to minimize the error. Now we modify the
weights. After correcting the weights the process is repeated again until the
overall error is minimized. After we find the optimal solution, we have the
neural network model.
This whole process is operated within the hidden
layers.
Prev
Next
5comments
이상구(LEE
SANGGU)November 17, 10:54 PM
The
First^^ Good~~
이상구(LEE
SANGGU)November 17, 10:54 PM
Others
should add your comment on this to pass this class.
이상구(LEE
SANGGU)November 18, 6:44 PM
Good.
I recommend you Giri to add your comment or ask question on OTHER'S summary or
Solutions to pass this class as I mentined many times.
김다솔(2018####24)November
22, 6:21 PM
(KOR)
인공신경망과 오차 역전파법에 대하여 아는 대로 서술하시오 Thank you,
and I will add more explanations later.
김다니일(2021####39)December
1, 00:46 AM
Short,
simple and understandable. Thank you!
W13
(Daniil Kim). Summary. Chapter 4 Open Problem 5.
13. Summary and Open Problem 5.
Open Problem 5 asks us to give a
short summary for the ANN and Backpropagation, so I thought about merging
answer with the Summary.
Machine Learning (ML) is a subfield of AI that studies Computer Algorithms, that are capable of self-improvement through experiencing sample data, known as “training data”, without direct programmer’s interference.
Artificial Neural Networks (ANN) are a commonly used, specific class of ML algorithms. ANNs are modeled on the human brain, in which thousands or millions of processing nodes, called “neurons”, are interconnected and organized into layers.
ANNs always has an Input Layer and an Output Layer. Sometimes, Neural Networks can have a hidden layer in between I and O Layers, where the weights are adjusted. Such Neural Networks are called “Deep Learning” and the process of adjustment – “Backpropagation”.
In order to calculate an output of the node (neuron), we use the following equation:
or
Here, Weight or
decides how influential the input
will be on the output, and Bias or 
is a constant that
helps us fit our model for the given data (More here: https://stackoverflow.com/questions/2480650/what-is-the-role-of-the-bias-in-neural-networks).
However, this formula works only when we have one output node. The
general formula for the outputs and 
inputs is:
These formulas are called Activation
Functions and usually denoted as 
.
Mentioned above Activation Function is called Linear Activation Function – simplest Activation Function. If we use a linear activation function in a neural network, then this model can only learn linearly separable problems.
Because of that, in modern ANNs,
developers usually use non-linear functions, that allows ANNs to
easily learn a non-linearly separable problem. One such function is a Sigmoid
Function – a special form of the logistic function: 
. With the addition of just one hidden layer and this activation
function in it, neural network can learn complex decision functions.
Initially, in a clean neural network, weights are defined randomly, but with time, they are readjusted using the Backpropagation, to reduce the error.
The flow of the Backpropagation algorithm is as follows:
1. Data Division (~80% for learning and the rest ~20% for the test cases)
2. Weights Randomization (initial setup of random weights)
3. Matrices and Activation Functions setup
4. Standard Calculation (using mentioned formulas)
5. Calculate Error (compare with the expected result)
6. Use Gradient Descent Method, to adjust the weights, so that the error is minimal.
7. Repeat 4-6.
8. Stop when result is satisfactory (the error is almost negligible or non-existent).
***************
Back Propagation Neural Network:
What is Backpropagation Algorithm in Machine Learning?
https://www.guru99.com/backpropogation-neural-network.html
What is Backpropagation?
Backpropagation is the essence of
neural network training. It is the method of fine-tuning the weights of a
neural network based on the error rate obtained in the previous epoch (i.e.,
iteration). Proper tuning of the weights allows you to reduce error rates and
make the model reliable by increasing its generalization.
Backpropagation in neural network
is a short form for “backward propagation of errors.” It is a standard method
of training artificial neural networks. This method helps calculate the
gradient of a loss function with respect to all the weights in the network.
Learn:
What is Artificial Neural
Networks?
What is Backpropagation?
How Backpropagation Algorithm
Works
Why We Need Backpropagation?
What is a Feed Forward Network?
Types of Backpropagation Networks
History of Backpropagation
Backpropagation Key Points
Best practice Backpropagation
Disadvantages of using
Backpropagation
How Backpropagation Algorithm
Works
The Back propagation algorithm in
neural network computes the gradient of the loss function for a single weight
by the chain rule. It efficiently computes one layer at a time, unlike a native
direct computation. It computes the gradient, but it does not define how the
gradient is used. It generalizes the computation in the delta rule.
Consider the following Back
propagation neural network example diagram to understand:
How Backpropagation Algorithm
Works
Inputs X, arrive through the
preconnected path
Input is modeled using real
weights W. The weights are usually randomly selected.
Calculate the output for every
neuron from the input layer, to the hidden layers, to the output layer.
Calculate the error in the outputs
ErrorB= Actual Output – Desired
Output
Travel back from the output layer
to the hidden layer to adjust the weights such that the error is decreased.
Keep repeating the process until
the desired output is achieved
Why We Need Backpropagation?
Most prominent advantages of
Backpropagation are:
Backpropagation is fast, simple
and easy to program
It has no parameters to tune apart
from the numbers of input
It is a flexible method as it does
not require prior knowledge about the network
It is a standard method that
generally works well
It does not need any special
mention of the features of the function to be learned.
What is a Feed Forward Network?
A feedforward neural network is an
artificial neural network where the nodes never form a cycle. This kind of
neural network has an input layer, hidden layers, and an output layer. It is
the first and simplest type of artificial neural network.
Types of Backpropagation Networks
Two Types of Backpropagation
Networks are:
Static Back-propagation
Recurrent Backpropagation
Static back-propagation:
It is one kind of backpropagation
network which produces a mapping of a static input for static output. It is
useful to solve static classification issues like optical character
recognition.
Recurrent Backpropagation:
Recurrent Back propagation in data
mining is fed forward until a fixed value is achieved. After that, the error is
computed and propagated backward.
The main difference between both
of these methods is: that the mapping is rapid in static back-propagation while
it is nonstatic in recurrent backpropagation.
History of Backpropagation
In 1961, the basics concept of
continuous backpropagation were derived in the context of control theory by J.
Kelly, Henry Arthur, and E. Bryson.
In 1969, Bryson and Ho gave a
multi-stage dynamic system optimization method.
In 1974, Werbos stated the
possibility of applying this principle in an artificial neural network.
In 1982, Hopfield brought his idea
of a neural network.
In 1986, by the effort of David E.
Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, backpropagation gained
recognition.
In 1993, Wan was the first person
to win an international pattern recognition contest with the help of the
backpropagation method.
Backpropagation Key Points
Simplifies the network structure
by elements weighted links that have the least effect on the trained network
You need to study a group of input
and activation values to develop the relationship between the input and hidden
unit layers.
It helps to assess the impact that
a given input variable has on a network output. The knowledge gained from this
analysis should be represented in rules.
Backpropagation is especially
useful for deep neural networks working on error-prone projects, such as image
or speech recognition.
Backpropagation takes advantage of
the chain and power rules allows backpropagation to function with any number of
outputs.
Best practice Backpropagation
Backpropagation in neural network
can be explained with the help of “Shoe Lace” analogy
Too little tension =
Not enough constraining and very
loose
Too much tension =
Too much constraint (overtraining)
Taking too much time (relatively
slow process)
Higher likelihood of breaking
Pulling one lace more than other =
Discomfort (bias)
Disadvantages of using Backpropagation
The actual performance of
backpropagation on a specific problem is dependent on the input data.
Back propagation algorithm in data
mining can be quite sensitive to noisy data
You need to use the matrix-based
approach for backpropagation instead of mini-batch.
Summary
A neural network is a group of
connected it I/O units where each connection has a weight associated with its
computer programs.
Backpropagation is a short form
for “backward propagation of errors.” It is a standard method of training artificial
neural networks
Back propagation algorithm in
machine learning is fast, simple and easy to program
A feedforward BPN network is an
artificial neural network.
Two Types of Backpropagation
Networks are 1)Static Back-propagation 2) Recurrent Backpropagation
In 1961, the basics concept of
continuous backpropagation were derived in the context of control theory by J.
Kelly, Henry Arthur, and E. Bryson.
Back propagation in data mining
simplifies the network structure by removing weighted links that have a minimal
effect on the trained network.
It is especially useful for deep
neural networks working on error-prone projects, such as image or speech
recognition.
The biggest drawback of the
Backpropagation is that it can be sensitive for noisy data.
Prev
Next
2comments
이상구(LEE SANGGU)November 25, 3:58 PM
Back
Propagation Neural Network: What is Backpropagation Algorithm in Machine
Learning? https://www.guru99.com/backpropogation-neural-network.html What is
Backpropagation? Backpropagation is the essence of neural network training. It
is the method of fine-tuning the weights of a neural network based on the error
rate obtained in the previous epoch (i.e., iteration). Proper tuning of the
weights allows you to reduce error rates and make the model reliable by
increasing its generalization. Backpropagation in neural network is a short
form for “backward propagation of errors.” It is a standard …
Finalized
by Kim Daniil, Pravas Giri, 김다솔
Week 12 < Open Problem 3> <PCA for Dimensionality Reduction> and
<PCA and covariance matrix.>
14. Finalized by Kim Daniil,
Pravas Giri, 김다솔 Week 12
< Open Problem 3> <PCA for Dimensionality Reduction> and <PCA
and covariance matrix.>
KIM DASOL
Open Problem 3
(KOR) 공분산 행렬에 대한 주성분분석(PCA)을 하여,
차원을 축소하는 과정에 대하여 이해한 바를 토론하시오.
공분산 행렬은 데이터의 분포를 설명해준다.
만약 공분산 행렬을 크기 순으로 정렬해 분산의 크기가 큰
(즉,
정보가 많은) 변수를 유지하고,
그렇지 않은 변수들은 버린다면, 정보를 최대한 유지한 채로 변수들의 숫자를 줄일 수 있게 된다.
즉,
원정보를 유지하며 차원을 축소 시킬 수 있게 된다. 그리고 이를 달성하기 위해 공분산 행렬에 특잇값 분해를 시행하는 것이 다름아닌 PCA가 된다.
(ENG) Debate the process of
dimension reduction, using PCA with covariance matrix.
Covariance matrix explains the
distribution of data. If we remain bigger covariances and discard smaller, then
we can reduce the number of variables remaining the information of
distribution as possible. It means we can reduce the dimension of data. For
achieving this, we should operate singular value decomposition with covariance
matrix, a.k.a PCA.
1comments
기리프라바스(2021####24)November 24, 12:49 PM
It
was well explained. Thank you.
PRAVAS GIRI
Open Problem 3:
Dimension reduction is a technique
of reducing high-dimensional data into low-dimensional data. By using the
process of dimension reduction by PC's on the covariance matrix, we find the
smallest number of new variables that can preserve the information from the
covariance matrix as much as possible.
이상구(LEE SANGGU)11월 25일 오후 1:46
Principal
Component Analysis for Dimensionality Reduction
https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
Learn
how to perform PCA by learning the mathematics behind the algorithm and
executing it step-by-step with Python! Lorraine Li May 25, 2019·10 min read In
the modern age of technology, increasing amounts of data are produced and
collected. In machine learning, however, too much data can be a bad thing. At a
certain point, more features or dimensions can decrease a model’s accuracy
since there is more data that needs to be generalized — this is known as the
curse of dimensionality. Dimensionality reduction is way to reduce the
complexity of a model and avoid overfitting. There are two main categories of
dimensionality reduction: feature selection and feature extraction. Via feature
selection, we select a subset of the original features, whereas in feature
extraction, we derive information from the feature set to construct a new
feature subspace. In this tutorial we will explore feature extraction. In
practice, feature extraction is not only used to improve storage space or the
computational efficiency of the learning algorithm, but can also improve the
predictive performance by reducing the curse of dimensionality — especially if
we are working with non-regularized models. Specifically, we will discuss the
Principal Component Analysis (PCA) algorithm used to compress a dataset onto a
lower-dimensional feature subspace with the goal of maintaining most of the
relevant information. We will explore: The concepts and mathematics behind PCA
How to execute PCA step-by-step from scratch using Python How to execute PCA
using the Python library scikit-learn Let’s get started! This tutorial is
adapted from Part 2 of Next Tech’s Python Machine Learning series, which takes
you through machine learning and deep learning algorithms with Python from 0 to
100. It includes an in-browser sandboxed environment with all the necessary
software and libraries pre-installed, and projects using public datasets. You
can get started for free here!
이상구(LEE SANGGU)11월 25일 오후 1:48
PCA
and covariance matrix. PCA is simply described as
“diagonalizing the covariance matrix”. ... It simply means that we need to find
a non-trivial linear combination of our original variables such that the
covariance matrix is diagonal. When this is done, the resulting variables are
uncorrelated, i.e. independent. What is the
significance of the covariance matrix and eigen vectors in PCA?
The
eigenvectors and eigenvalues of a covariance (or correlation) matrix
represent the “core” of a PCA:
The
eigenvectors (principal components) determine the directions of the new feature
space, and the eigenvalues determine their magnitude.
So,
PCA is a method that: Measures how each variable is associated with one another
using a Covariance matrix. Understands the directions of the spread of our data
using Eigenvectors. Brings out the relative importance of these directions
using Eigenvalues.Sep 9, 2019 How is covariance matrix used in PCA? The
eigenvectors and eigenvalues of a covariance (or correlation) matrix represent
the “core” of a PCA: The eigenvectors (principal components) determine the
directions of the new feature space, and the eigenvalues determine their
magnitude.Jan 27, 2015
Kim Daniil
Open Problem 3:
As everyone above mentioned. Dimension reduction by PCs (Principal Components) on
the covariance matrix is basically what we call PCA, and what we do
with the PCs and the covariance matrix is that we simply "diagonalize the
covariance matrix". In other words, what we do is we try to find a linear
combination of PCs, such that our covariance matrix will be diagonal.
Note: i just noticed that the
Prof. Lee Sang-Gu also commented the same thing. However, in the actual article
there is a lot more information, that can be helpful. So, I decided to leave
the source: http://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf
이상구(LEE SANGGU)November 25, 1:46 PM
Principal
Component Analysis for Dimensionality Reduction
https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
...
이상구(LEE SANGGU)November 25, 1:48 PM
PCA
and covariance matrix. PCA is simply described as “diagonalizing the covariance
matrix”. ... It simply means that we need to find a non-trivial linear
combination of our original variables such that the covariance matrix is
diagonal. When this is done, the resulting variables are uncorrelated, i.e.
independent. What is the significance of the covariance matrix and eigen
vectors in PCA? So, PCA is a method that: Measures how each variable is
associated with one another using a Covariance matrix. Understands the
directions of the spread of our data using Eigenvectors. Brings out the
relative importance of these directions using Eigenvalues.Sep 9, 2019 How is
covariance matrix used in PCA? The eigenvectors and eigenvalues of a covariance
(or correlation) matrix represent the “core” of a PCA: The eigenvectors
(principal components) determine the directions of the new feature space, and
the eigenvalues determine their magnitude.Jan 27, 2015
Finalized
by Kim Daniil, Pravas Giri, 김다솔,
Week 12 <Open Problem 4> Is there a difference between linear regression
and least squares?
Finalized by Kim Daniil, Pravas
Giri, 김다솔,
Week 12 <Open Problem 4> Is there a difference between
linear regression and least squares?
KIM DASOL
Open Problem 4
(KOR) 최소제곱직선과 회귀분석에서 사용하는 선형회귀 사이의 같은 점과 다른 점에 대하여 이해한 바를 토론하시오.
최소 제곱 해는 변수들 사이에 순서가 없다.
그러나 선형회귀에는 종속변수와 독립변수가 존재한다.
'직선'에 포커스를 맞추면 변수가 세 개 이상이 되면 최소제곱직선이라고 부를 수 없게 된다. (그러나 여전히 최소제곱해는 존재한다.)
선형회귀는 두 변수 사이의 회귀직선 뿐만 아니라 세 개 이상의 변수도 사용해 회귀분석을 진행하곤 한다.
이 때에도 여전히 종속변수와 독립변수들이 나뉘게 된다.
다변수에 대한 비교는 최소제곱해와 선형회귀의 차이를 뚜렷하게 나타내준다. 만약 100개의 변수가 있다면 최소제곱해는
100 차원의 벡터들이 최소 제곱 오차를 갖게 되는
99차원의 초평면을 해로 갖게 된다.
선형회귀는
1개의 종속변수에 99개의 독립변수가 가장 잘 맞도록 하는 회귀직선을 갖게 된다.
제 이해를 바탕으로 한 제 생각이니 더 많은 의견과 사실들을 댓글로 남겨주시길 바랍니다.
(ENG) Debate the same and
different things between least-squares line and linear regression in regression
analysis.
There is no order in the variables
of least square solutions. But, in linear regression, there are one dependent
variable and independent variables. Multivariate operating clears the
difference between LSE and linear regression. With 100 variables, LSE has
99-dimensional hyperplane as a solution. Linear regression has one regression
line fitting 99 independent variables to one dependent variable.
This is just my thought from my
understanding. Please let me know your thought and the facts about the question
with under comments! Thank you :)
2comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
기리프라바스(2021####24)November 24, 12:50 PM
Thank
you.
PRAVAS GIRI
Open Problem 4:
Linear regression attempts to
model the relationship between two variables by fitting a linear equation to
observed data.
Least square line is the line that makes the vertical distance from the data
points to the regression line as small as possible .
What
is the difference between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares
In common parlance, they seem
to be used interchangeably. But more precisely, least squares is a
method for performing linear regression.
For now*, we can think of
linear regression as the task of fitting a straight line (or, in the case of
multiple linear regression, a "hyperplane") through a set of points.
But there are many possible strategies to fit a line through a set of points:
·
you could take the leftmost
point and the rightmost point and draw a line between them
·
you could compute the slopes
of the lines connecting each pair of points and calculate the average slope,
drawing a line with this slope that passes through the point at the average of
the "x values" and the average of the "y values"
·
you could find the line for
which there are an equal number of points above the line and below the line
·
you could draw a line, and
then for each of the data points, measure the vertical distance between the
point and the line, and add these up; the fitted line would be the one where
this sum of distances is as small as possible
·
you could draw a line, and
then for each of the data points, measure the vertical distance between the
point and the line, square it, and add these up; the fitted line would be the
one where this sum of distances is as small as possible
The last strategy is called "ordinary least squares" (because you are
trying to minimize the sum of squared prediction errors) and it is the most
commonly used (wondering why least squares is most commonly used? see * below)
. But as far as fitting a line through a set of points goes, any of the other
strategies are equally valid. The first three strategies I made up as examples
and probably do not perform well, but the fourth one is a real strategy called
"least absolute deviations" that some people prefer over least
squares.
* I want to clear up an
important distinction that I glossed over in the above:
First, precisely speaking
there is a difference between linear regression and curve
fitting. Everything I said above is actually talking about curve fitting,
not linear regression: you have a set of points, and you want to draw a curve
(line) through them that fits as well as possible. This is a purely geometric problem.
The x and y axes have no interpretation, and the "data" are just
points in Cartesian space.
On the other hand, linear
regression is a statistical inference problem. The "y
values" take on the interpretation of data you wish to model, and the
"x values" take on the interpretation of extra information you have
about each data point that might be helpful in predicting their "y
values". You are trying to build a probabilistic model that describes
"y" while taking into account "x", and a linear model is
one of many ways to do this. A linear model assumes that "y" has a
different mean for each possible value of "x", and that these means
happen to follow a straight line with a certain intercept and a certain slope.
As with any statistical inference problem, you estimate the unknown parameters
using maximum likelihood estimation. But since in this case the unknown
parameters are an intercept and a slope, the end result of maximum likelihood
estimation is basically that you are choosing a straight line that fits the
observed data best, so this essentially becomes the curve fitting problem
discussed above.
Now we arrive at the question
of why least squares, of all possible curve fitting methods, is so commonly
used. The reason is that when solving the statistical linear
regression problem, a very common modeling assumption is that for every
possible value of "x", the quantity "y" is normally
distributed with a mean that is linear in "x". Therefore,
the likelihood function is essentially a product of PDFs of the normal
distribution. As stated above, you estimate the unknown parameters (and
therefore find the best fitting line) by maximizing the likelihood function. If
you look at what the product of normal PDFs looks like, you will notice that
maximizing this expression happens to be equivalent to... you guessed it...
minimizing the sum of squared errors!
That is, the line you get performing curve fitting via least
squares is equivalent to the line you get performing linear regression using a
normal model.
Prev
Next
1comments
이상구(LEE SANGGU)November 25, 1:44 PM
What
is the difference between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares
...
Kim Daniil
Open Problem 4:
Although two terms have similar
names, they are in fact different:
- Least-Squares
solution gives us optimal solutions to the equation, and we find it via
different Matrix Mutations.
- Least-Squares
Regression Line is a linear function, that visualizes the line, that has
a minimal variance (sum of squares of the errors or in other words,
vertical distances). And we usually find it via Calculus, or Slope/Tangent
and Intercept evaluation to be precise.
However, despite that, we can
find the Least-Squares Regression Line by finding the Least-Squares Solution.
Prof. Lee Sang-Gu discussed it in the previous lessons and defined it as (the
least-squares line, the best fit line).
김다니일(2021####39)November 29, 9:06 AM
Oh,
I see. Found the error. I looked through everyone's code and, apparently, all of
us did the same mistake :D. The problem with the code is: Ex2-Ex^2 (Variance
function). The priority is incorrect, It should be a squared difference of the
mean values, but what we have is a "mean_value1 - square of
mean_value2". So, we should add parentheses (Ex2-Ex)^2. And now everything
falls in place.
김다니일(2021####39)November 29, 9:06 AM
Thank
you for pointing out! I posted solution with a fix.
Finalized
by Kim Daniil, Pravas Giri, 김다솔,
박준호
Week 12 < Open Problem 1>
Finalized by Kim Daniil, Pravas Giri, 김다솔 Week 12 < Open Problem 1>
PRAVAS GIRI
Principle Component Analysis (PCA):
It is difficult to analyze multi-dimensional data due to its
large number. In this case it would be much easy to extract and only use some
data that seems explain the data. However, data selection in this manner might
not purely reflect the attributes of the data. Therefore, PCA is needed. PCA is
process of analysis used to determine which feature of the element should be
mainly used.
이해한 바 >> To PROPERLY select(extract)
partial data, use PCA.
Open Problem 1:
After deciding a data, we construct a matrix X using that data.
Then the centered matrix is found by differencing X and the matrix having mean
of the each column in matrix X.
Now, we calculate the SVD of mean-centered matrix X. Here, we
have U, V which are orthogonal matrices and S with is a diagonal matrix with
decreasing singular values. Then, the column of vector V becomes the principle
axes and the product of U and S denoted by Z becomes principle component. In
this way by computing the proportion that preserves the distribution of the
original data for each principal component we can decide the reduced
dimension.
1comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
김다솔
Open Problem 1, Explain how does
SVD work in PCA.
The most important property of SVD
is capability of partial re-composition. It means you can operate recovery with
partial SIGMA matrix from decomposition A = U SIGMA V.
This property is caused by that
the matrix SIGMA is "ordered". You can keep information of original
matrix A as much as possible, choosing upper some values in SIGMA.
For simple example, you can
discard all 0 values in the lower of SIGMA matrix, keeping whole original
matrix.
The partial re-composition
property of SVD has a main role in PCA. PCA transforms a data matrix following
principal axes.
The number of principal axes can
be chosen manually. It means we can choose principal axes and discard minor
axes.
This property is from SVD used in
the middle of PCA. In real, choosing principal axes is choosing upper SVD
elements because the principal axes are in V matrix from the decomposition A =
U SIGMA V.
When SIGMA becomes smaller, then
the decomposition (or re-composition) represents like this: U SIGMA' V = A'
where SIGMA' is SIGMA matrix changing under elements as 0.
이해한 바 >> because of "ORDERED SIGMAs" of SVD,
we can know what are
Principal Components (new axes) for PCA
added my opinion
>> very small sigmas(>= 0) and highly dependent vectors (axes) on
other variables seems to be similar concept. The errors (variances from fitting
curve) are negligible derived from linearly dependent two vectors, so plane
spanned by these two vectors became one fitting curve.
Prev
Next
2comments
이상구(LEE SANGGU)November 15, 2:28 PM
Reasonable
explanation ^^
기리프라바스(2021####24)November 24, 12:45 PM
Thank
you for the detailed explanation.
김다니일
15. Summary:
The Principal Component Analysis (PCA) is
one of the most widely used dimensionality reduction techniques – the
transformation of data from a high-dimensional space into a low-dimensional
space so that the low-dimensional representation retains some meaningful
properties of the original data. The PCA tries to combine existing variables to
define the new variables, called Principal
Components, while minimizing the information loss.
After PCA, it is much easier to
compare data and requires less computing power.
There are two main ways to
perform a PCA:
- via
eigen-decomposition and covariance matrices. I recommend reading that
article, for the thorough understanding of that method: https://medium.com/analytics-vidhya/understanding-principle-component-analysis-pca-step-by-step-e7a4bb4031d9
- using
SVD on the initial data matrix. This method is sometimes preferrable over
the eigen-decomposition because eigenvalue decomposition can only be used
for the diagonalizable matrices, whereas SVD can be used for any matrix.
More on that topic: https://jonathan-hui.medium.com/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491
<Week
14- Open Problem 6 PRAVAS GIRI>
Author : 기리프라바스(2021####24)Date : December
2, 10:08 PM
Count : 11
Open
Problem 6:
Voice
Recognition using Neural Networks
In this system ANN is used for pattern matching. We know that ANN is composed
of neurons which are connected to each other in the forward path by some
multipliers called the connection weights. Neural networks have self learning
capacity and are used in system identification, pattern recognition,
classification, speak recognition etc.
Like
shown in the figure, a three layer feedforward neural network with a sigmoidal
hidden layer followed by a linear layer is employed in this application for
pattern matching. Here, the neural network is trained using the backpropagation
algorithm. In this application, an adaptive backpropagation learning method is
employed that is the learning gain is adjusted during the training to enhance
faster and global convergence.
The ANNs were trained with seven voice samples recorded at different instants
of time of four different speakers uttering the same phrase at all times. An
initial learning rate, an allowable error and the maximum number of training
cycles are the parameters that specified during the training phase. The neural
network is constructed in the MATLAB environment [3]. In diagram below the
first diagram shows a plot of the sum squared error versus the number of epochs
during the training phase. The sum squared error goal was reached in just 174
epochs. The second diagram shows the different learning rates used during the
training.
There
was a 100% rate in success when ANN was tested with trained samples. However,
only 66% success rate was experienced when untrained sample was used. This was
due to PSDs of input sample not being consistent. When tested with unknown
samples ANN successfully classified it as unidentified voices.
The
use of ANN in this research for voice recognition has so far proved to be
successful.
reference link:
https://scholarsmine.mst.edu/cgi/viewcontent.cgi?article=1691&context=ele_comeng_facwork
Prev
Next
3comments
김다니일(2021####39)December
2, 10:53 PM
That
was pretty informative. Thank you! Btw, it is so fascinating. I can't really
imagine how much data samples is needed to get decent enough results for
completely different voices (child's voice (high-pitched), old person's voice
(low-pitched and hardly distinguishable sounds))
이상구(LEE
SANGGU)December 3, 00:21 AM
Good
^^☆☆
박준호(2014####20)December
3, 10:35 AM
Thanks!
so interesting! It is curious about does it work well if the data have
different phrases and different voices. I will search for that.
<Week
14- OPEN PROBLEM 7 PRAVAS GIRI>
Author : 기리프라바스(2021####24)Date : December
2, 10:32 PM
Count : 7
Open
Problem 7:
Probability
theory is incorporated into machine learning, particularly the subset of
artificial intelligence concerned with predicting outcomes and making
decisions. In computer science, functions are used to limit the functions
outcome to a value between 0 and 1. These functions, also known as squashing
functions, are useful in an algorithms process of assigning outcomes a
probability value. The values assigned by these functions assist the neural
network in making better decisions, and is often the final step in a neural
network function.
Probabilistic
reasoning is a way of knowledge representation where we apply the concept of
probability to indicate the uncertainty in knowledge. In probabilistic
reasoning, we combine probability theory with logic to handle the uncertainty.
We use probability in probabilistic reasoning because it provides a way to
handle the uncertainty.
In
the above diagram listed out Bayesian Networks based Probabilistic programs for
making reasoning, reasoning over time and for decisions. The models listed out
with respect to its area like Learning-based for Machine Learning, State-based
for Problem Solving, Logic-based for Bayesian Networks, Logical-based for First
Order Logic, and Communication for NLP.
Prev
Next
2comments
김다니일(2021####39)December
2, 10:54 PM
Oh,
that's an interesting scheme: simple and concrete. Thank you for sharing!
이상구(LEE
SANGGU)December 3, 00:21 AM
Good
이해한 바 >> PCA reduce dimension of
data set as leaving Prinipal Components.
16. Chapter 4 Open Problem 1:
SVD not only more robust and reliable, but also makes it easier
to determine Principal Components. By doing SVD, we can manually choose the
proportion that preserves the initial data distribution and select that
dimension as the basis for our PCA.
Prev
Next
2comments
이상구(LEE SANGGU)November 18, 6:43 PM
Good.
I recommend you to add your comment or ask question on OTHER'S summary or
Solutions to pass this class as I mentined many times.
기리프라바스(2021####24)November 24, 12:48 PM
Thank
you for the explanation.
W11
(Daniil Kim). Chapter 3 Open Problems: 2 Fixed. Thanks to 박준호.
Thank you, 박준호, for pointing out, that the code
was incorrect. I suggest everyone to look through their code, because from what
I have found, everyone had the same problem: There are missing parentheses in
the variance function, where we subtract mean values and find the square of the
result.
17. Open Problems:
#2:
<<<
Added by Park Junho
Because of
improper PDF(P(x) < 0, and total sum of P(x) != 1), the result is incorrect.
For example,
negative variance.
It is
impossible because it is sum of the squared nunbers((x-m)^2))
The correct code is as follows:
# W11 (Kim Daniil) Chapter 3
Open Problem 2 (Fixed. Thanks 박준호)
# Probability Density
Function
f(x) = x^5+e^(3*x)
xmin = -0.5
xmax = 1
show("Initial
Probability Density Function: f(x)=", f(x))
# Expectation
Ex = integral(x*f(x), x,
xmin, xmax)
tempEx = integral(x^2*f(x),
x, xmin, xmax)
show("Expectation:
E(X)=", Ex)
# Previously, the variance
function was incorrect:
# Var(X) = tempEx - Ex^2
# Now, the correct one, with
parenthesis:
Var(X) = (tempEx - Ex)^2
show("Variance:
Var(X)=", Var(X))
SD(X) = sqrt(Var(X))
show("Standard
Deviation: SD(X)=", SD(X))
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
[Final
OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 2>
목록으로 이동
[Final
OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 2>
박준호
다른 학우들의 과제를 잘 보았습니다.
그러나 전반적으로 관심사나 전공이 달라서 그런지
PCA를 통해 데이터를 압축할 수 있다는 사실과 별개로
데이터 자체에 대한 설명이 있었다면 더 좋았을 것 같았습니다.
그래서 저는 다른 것보다도 데이터 자체에 대한 저의 생각을 좀더 첨가해보았습니다....
흥미로웠던 점은 2~14열의 수익률 자료만을 가지고 PCA를 하자, 전체 데이터 이상으로 무시할만한 성분들이 적어지는 것을 확인할 수 있었습니다.
사실 초기값(initial value)이 주어진다는 전제 하에서 수익률 데이터와 가격 데이터는 사실상 동일함에도 불구하고
PCA에서는 매우 상반되는 과정, 심지어는 결과물을 낼 수 있다는 사실이 몹시 흥미로웠습니다.
아마도 수익률은 주가에 비해서 훨씬 무작위적이기 때문에 그렇다고 추측해볼 따름이지만, 그 이유에 대해 다른 학우들의 의견도 들어보고 싶습니다!
---------------------------------
PRAVAS GIRI
Open Problem 2:
Source: https://data.world/sdhilip/pizza-datasets/workspace/file?filename=Pizza.csv
Code:
# Enter the data matrix.
# Here, each column represents
data corresponding to the variables Price, Software, Aesthetics, and Brand.
# data matrix
X = matrix(RDF,
[[27.82,21.43,44.87,5.11,1.77,0.77,4.93],
[28.49,21.26,43.89,5.34,1.79,1.02,4.84],
[28.35,19.99,45.78,5.08,1.63,0.8,4.95],
[30.55,20.15,43.13,4.79,1.61,1.38,4.74],
[30.49,21.28,41.65,4.82,1.64,1.76,4.67],
[31.14,20.23,42.31,4.92,1.65,1.4,4.67],
[31.21,20.97,41.34,4.71,1.58,1.77,4.63],
[28.76,21.41,41.6,5.28,1.75,2.95,4.72],
[28.22,20.48,45.1,5.02,1.71,1.18,4.93],
[27.72,21.19,45.29,5.16,1.66,0.64,4.95],
[27.35,21.2,45.59,4.94,1.65,0.92,4.98],
[26.98,21.2,45.03,5.15,1.67,1.64,4.97],])
print("X =")
print(X)
print() # One line indent
# centering
# scaling
n = X.nrows() # (number of
rows)
p = X.ncols() # (number of
columns)
X_ctr = zero_matrix(RDF, n,
p) # Prepare memory for centered matrix
import numpy as np
for col in range(p):
v =
np.array(X.column(col))
m = mean(v)
st = std(v)
for row in range(n):
X_ctr[row,col] = (X[row,col] - m)/st
# (X_ctr is a centered matrix)
print("X_ctr =")
print(X_ctr.n(digits = 4))
print() # (One line indent)
# Singular Value
Decomposition(SVD). X_ctr = U*S*V.transpose()
U, S, V = X_ctr.SVD()
Var_PC = [(S[i, i]^2/n).n(digits =
4) for i in range(p)] # PC variance
Prop_Var =
[100*Var_PC[i]/sum(Var_PC) for i in range(p)] # Ratio of variance
print("PC variance
", Var_PC)
print("Ratio of
variance ", Prop_Var)
print() # (One line indent)
# Scree Plot. Visually show
variance. Decide to what dimension to reduce.
show(list_plot(Var_PC, plotjoined
= True, color = 'red', axes_labels = ['', 'Variances']))
print() # (One line indent)
# Compute PC score. Z = U*S
# Calculate the PC score. Z
= U*S
PC_score = U*S
PC12 = PC_score[:,0:2]# (Use only
the first and second components for visualization)
# (Visualize the data on a plane.)
point([PC12.row(i) for i in
range(n)], color = 'red')
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
Output:
X =[27.82 21.43 44.87 5.11 1.77 0.77 4.93][28.49 21.26 43.89 5.34 1.79 1.02 4.84][28.35 19.99 45.78 5.08 1.63 0.8 4.95][30.55 20.15 43.13 4.79 1.61 1.38 4.74][30.49 21.28 41.65 4.82 1.64 1.76 4.67][31.14 20.23 42.31 4.92 1.65 1.4 4.67][31.21 20.97 41.34 4.71 1.58 1.77 4.63][28.76 21.41 41.6 5.28 1.75 2.95 4.72][28.22 20.48 45.1 5.02 1.71 1.18 4.93][27.72 21.19 45.29 5.16 1.66 0.64 4.95][27.35 21.2 45.59 4.94 1.65 0.92 4.98][26.98 21.2 45.03 5.15 1.67 1.64 4.97] X_ctr =[ -0.7290 1.000 0.6295 0.4249 1.439 -0.9150 0.7250][ -0.2863 0.6797 0.05384 1.598 1.745 -0.5223 0.06144][ -0.3788 -1.713 1.164 0.2719 -0.7005 -0.8679 0.8724][ 1.075 -1.411 -0.3926 -1.207 -1.006 0.04320 -0.6758][ 1.035 0.7174 -1.262 -1.054 -0.5477 0.6401 -1.192][ 1.465 -1.261 -0.8742 -0.5439 -0.3948 0.07462 -1.192][ 1.511 0.1334 -1.444 -1.615 -1.465 0.6558 -1.487][ -0.1079 0.9623 -1.291 1.292 1.134 2.509 -0.8233][ -0.4647 -0.7896 0.7646 -0.03399 0.5222 -0.2710 0.7250][ -0.7951 0.5479 0.8762 0.6799 -0.2420 -1.119 0.8724][ -1.040 0.5667 1.052 -0.4419 -0.3948 -0.6794 1.094][ -1.284 0.5667 0.7234 0.6289 -0.08915 0.4516 1.020] PC variance [3.483, 1.841, 0.5706, 0.4104, 0.1111, 0.0003037, 5.267e-31]Ratio of variance [54.28, 28.69, 8.892, 6.396, 1.731, 0.004733, 8.209e-30]
Prev
Next
1comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
김다니일
Chapter 4 Open Problem 2:
I found a practice data sample of
different food and amount of different kinds of vitamins, nutritions, etc in
them….
Sources:
Data Source: https://data.world/exercises
Jupyter Source: https://colab.research.google.com/drive/1rMEHNEHV_21dhlX6tmt7ObfzZ9u91NJ_?usp=sharing
Prev
Next
3comments
이상구(LEE SANGGU)November 18, 8:10 PM
Add
the address of your source data^^
이상구(LEE SANGGU)November 19, 4:24 AM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report. With
these academic activities, you can add it in your Final Exam report. Then you
will pass. Add your comment and Finalize other's solutions and answers. You
will learn from it a lot.
기리프라바스(2021####24)November 24, 12:48 PM
Thank
you.
김다솔
Open Problem 2 - Find a data
matrix from your major field. Apply PCA algorithm to the data table and debate
about it.
1) Download RNA expression of
cancer patients data from GDC data portal.
2) Preprocessing
3) Data matrix has 60k expression
probe (TPKM) and 174 samples
4) Centering
5) Components are limited by
sample number because the data sample size is smaller than the feature size.
and I choose 34 upper components.
It is impossible to show 34
dimensions. Question is first component has 82% PC score. It is too high, I
think.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
2comments
김다니일(2021####39)November 16, 11:13 AM
Regarding
the Computing Power, if data analysis takes too much time, you can easily get a
free 3 month trial on Google Computing Engine and rent a very powerful Linux
server for 300 free credits (a quad-core 16GB RAM Ubuntu Pro server costs ~100
credits a month). Although, you would need some prior Unix knowledge, because
there is no GUI, and you would have to use Terminal and SSH to communicate with
server. But you can easily setup a Jupyter Notebook server (I think that is
what you are using here) there. https://cloud.google.com/compute
기리프라바스(2021####24)November 24, 12:44 PM
Thank
you. Nice work
[Final
OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 3>
<PCA for Dimensionality Reduction> and <PCA and covariance matrix.>
[Final
OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 < Open Problem 3>
<PCA for Dimensionality Reduction> and <PCA and covariance matrix.>
PRAVAS GIRI
Open Problem
3:
(ENG) Debate
the process of dimension reduction, using PCA with covariance matrix.
(KOR)
공분산 행렬에 대한 주성분분석(PCA)을 하여,
차원을 축소하는 과정에 대하여 이해한 바를 토론하시오.
Dimension
reduction is a technique of reducing high-dimensional data into low-dimensional
data. By using the process of dimension reduction by PC's on the covariance
matrix, we find the smallest number of new variables that can preserve the
information from the covariance matrix as much as possible.
KIM
DASOL Open Problem 3
공분산 행렬은 데이터의 분포를 설명해준다. 만약 공분산 행렬을 크기 순으로 정렬해 분산의 크기가 큰 (즉, 정보가 많은) 변수를 유지하고, 그렇지 않은 변수들은 버린다면, 정보를 최대한 유지한 채로 변수들의 숫자를 줄일 수 있게 된다. 즉, 원정보를 유지하며 차원을 축소 시킬 수 있게 된다. 그리고 이를 달성하기 위해 공분산 행렬에 특잇값 분해를 시행하는 것이 다름아닌 PCA가 된다.
(ENG) Debate
the process of dimension reduction, using PCA with covariance matrix.
Covariance
matrix explains the distribution of data. If we remain bigger covariances and
discard smaller, then we can reduce the number of variables remaining
the information of distribution as possible. It means we can reduce
the dimension of data. For achieving this, we should operate singular value
decomposition with covariance matrix, a.k.a PCA.
PCA and
covariance matrix. PCA is simply described as “diagonalizing the covariance
matrix”. ... It simply means that we need to find a non-trivial linear
combination of our original variables such that the covariance matrix is
diagonal. When this is done, the resulting variables are uncorrelated, i.e.
independent. What is the significance of the covariance matrix and eigen
vectors in PCA?
The
eigenvectors and eigenvalues of a covariance (or correlation) matrix
represent the “core” of a PCA:
The
eigenvectors (principal components) determine the directions of the new feature
space, and the eigenvalues determine their magnitude.
So, PCA is a
method that: Measures how each variable is associated with one another using a
Covariance matrix. Understands the directions of the spread of our data using
Eigenvectors. Brings out the relative importance of these directions using Eigenvalues.Sep
9, 2019 How is covariance matrix used in PCA? The eigenvectors and eigenvalues
of a covariance (or correlation) matrix represent the “core” of a PCA: The
eigenvectors (principal components) determine the directions of the new feature
space, and the eigenvalues determine their magnitude.Jan 27, 2015
박준호
High dimension
is a huge matter for computing.
from
difference with average.
Covariances = addtional information from difference with
new axis (fitting line).
Therefore, small variances are negligible comparing to
computing efficiency by reducing dimension.
Kim Daniil
Open Problem 3:
As everyone above mentioned. Dimension reduction by
PCs (Principal Components) on the covariance matrix is basically what we
call PCA, and what we do with the PCs and the covariance matrix is that we
simply "diagonalize the covariance matrix". In other words, what we
do is we try to find a linear combination of PCs, such that our covariance
matrix will be diagonal.
Note: i just noticed that the Prof. Lee Sang-Gu also
commented the same thing. However, in the actual article there is a lot more
information, that can be helpful. So, I decided to leave the source: http://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf
Finalized by Kim Daniil, Pravas Giri, 김다솔 Week 12 < Open Problem 3> <PCA for
Dimensionality Reduction> and <PCA and covariance matrix.>
KIM DASOL
Open Problem 3
(KOR) 공분산 행렬에 대한 주성분분석(PCA)을 하여,
차원을 축소하는 과정에 대하여 이해한 바를 토론하시오.
공분산 행렬은 데이터의 분포를 설명해준다.
만약 공분산 행렬을 크기 순으로 정렬해 분산의 크기가 큰
(즉,
정보가 많은) 변수를 유지하고,
그렇지 않은 변수들은 버린다면, 정보를 최대한 유지한 채로 변수들의 숫자를 줄일 수 있게 된다.
즉,
원정보를 유지하며 차원을 축소 시킬 수 있게 된다. 그리고 이를 달성하기 위해 공분산 행렬에 특잇값 분해를 시행하는 것이 다름아닌 PCA가 된다.
(ENG) Debate the process of
dimension reduction, using PCA with covariance matrix.
Covariance matrix explains the
distribution of data. If we remain bigger covariances and discard smaller, then
we can reduce the number of variables remaining the information of
distribution as possible. It means we can reduce the dimension of data. For
achieving this, we should operate singular value decomposition with covariance
matrix, a.k.a PCA.
1comments
기리프라바스(2021####24)November 24, 12:49 PM
It
was well explained. Thank you.
>> by 박준호
'분산의 크기가 큰 = 정보가 많은' 문구의 의미에 대하여,
(A)는 새로운 축과 원본 데이터 간의 정보 차이가 적지만, (B)는 많을 것이라는 생각이 들었습니다.
-------------------------------------------------------------------------------------------------------------------------
PRAVAS GIRI
Open Problem 3:
Dimension reduction is a technique
of reducing high-dimensional data into low-dimensional data. By using the
process of dimension reduction by PC's on the covariance matrix, we find the
smallest number of new variables that can preserve the information from the
covariance matrix as much as possible.
이상구(LEE SANGGU)11월 25일 오후 1:46
Principal
Component Analysis for Dimensionality Reduction
https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
Learn
how to perform PCA by learning the mathematics behind the algorithm and
executing it step-by-step with Python! Lorraine Li May 25, 2019·10 min read In
the modern age of technology, increasing amounts of data are produced and
collected. In machine learning, however, too much data can be a bad thing. At a
certain point, more features or dimensions can decrease a model’s accuracy
since there is more data that needs to be generalized — this is known as the
curse of dimensionality. Dimensionality reduction is way to reduce the
complexity of a model and avoid overfitting. There are two main categories of
dimensionality reduction: feature selection and feature extraction. Via feature
selection, we select a subset of the original features, whereas in feature
extraction, we derive information from the feature set to construct a new
feature subspace. In this tutorial we will explore feature extraction. In
practice, feature extraction is not only used to improve storage space or the
computational efficiency of the learning algorithm, but can also improve the
predictive performance by reducing the curse of dimensionality — especially if
we are working with non-regularized models. Specifically, we will discuss the
Principal Component Analysis (PCA) algorithm used to compress a dataset onto a
lower-dimensional feature subspace with the goal of maintaining most of the
relevant information. We will explore: The concepts and mathematics behind PCA
How to execute PCA step-by-step from scratch using Python How to execute PCA
using the Python library scikit-learn Let’s get started! This tutorial is
adapted from Part 2 of Next Tech’s Python Machine Learning series, which takes
you through machine learning and deep learning algorithms with Python from 0 to
100. It includes an in-browser sandboxed environment with all the necessary
software and libraries pre-installed, and projects using public datasets. You
can get started for free here!
이상구(LEE SANGGU)11월 25일 오후 1:48
PCA
and covariance matrix. PCA is simply described as
“diagonalizing the covariance matrix”. ... It simply means that we need to find
a non-trivial linear combination of our original variables such that the
covariance matrix is diagonal. When this is done, the resulting variables are
uncorrelated, i.e. independent. What is the
significance of the covariance matrix and eigen vectors in PCA?
The
eigenvectors and eigenvalues of a covariance (or correlation) matrix
represent the “core” of a PCA:
The
eigenvectors (principal components) determine the directions of the new feature
space, and the eigenvalues determine their magnitude.
So,
PCA is a method that: Measures how each variable is associated with one another
using a Covariance matrix. Understands the directions of the spread of our data
using Eigenvectors. Brings out the relative importance of these directions
using Eigenvalues.Sep 9, 2019 How is covariance matrix used in PCA? The
eigenvectors and eigenvalues of a covariance (or correlation) matrix represent
the “core” of a PCA: The eigenvectors (principal components) determine the
directions of the new feature space, and the eigenvalues determine their
magnitude.Jan 27, 2015
Kim Daniil
Open Problem 3:
As everyone above
mentioned. Dimension reduction by PCs (Principal Components) on the
covariance matrix is basically what we call PCA, and what we do with the
PCs and the covariance matrix is that we simply "diagonalize the
covariance matrix". In other words, what we do is we try to find a linear
combination of PCs, such that our covariance matrix will be diagonal.
Note: i just noticed that the
Prof. Lee Sang-Gu also commented the same thing. However, in the actual article
there is a lot more information, that can be helpful. So, I decided to leave
the source: http://pmaweb.caltech.edu/~physlab/lab_21_current/Ph21_5_Covariance_PCA.pdf
>> understanded by 박준호
linear combination of PCs : substitute original orthgonal vectors
to new axes.
PCA means "diagonlizing covariance
matrix" : ordering new axes.
이상구(LEE SANGGU)November 25, 1:46 PM
Principal
Component Analysis for Dimensionality Reduction
https://towardsdatascience.com/principal-component-analysis-for-dimensionality-reduction-115a3d157bad
...
이상구(LEE SANGGU)November 25, 1:48 PM
PCA and covariance matrix. PCA is
simply described as “diagonalizing the covariance matrix”. ... It simply means
that we need to find a non-trivial linear combination of our original variables
such that the covariance matrix is diagonal. When this is done, the resulting
variables are uncorrelated, i.e. independent. What is the significance of the
covariance matrix and eigen vectors in PCA? So, PCA is a method that: Measures
how each variable is associated with one another using a Covariance matrix.
Understands the directions of the spread of our data using Eigenvectors. Brings
out the relative importance of these directions using Eigenvalues.Sep 9, 2019
How is covariance matrix used in PCA? The eigenvectors and eigenvalues of a
covariance (or correlation) matrix represent the “core” of a PCA: The
eigenvectors (principal components) determine the directions of the new feature
space, and the eigenvalues determine their magnitude.Jan 27, 2015
[Final
OK by SGLee] Finalized by Kim Daniil, Pravas Giri, 김다솔, 박준호 Week 12 <Open Problem 4>
[Final OK by SGLee] Finalized
by Kim Daniil, Pravas Giri, 김다솔,
박준호
Week 12 <Open Problem 4>
박준호
1.
선형회귀와 최소제곱직선
공통점:
거리를 최소화하는 과정을 통해
fitting curve 생성
차이점:
선형회귀는 y축 기준 거리인 반면 최소제곱직선은
fitting curve 기준 거리
( 이 둘
linear regression and least squares 은
they seem to be used interchangeably. But more precisely, least squares is a
method for performing linear regression 입니다.
부분집합의 관계 정도로 이해 됩니다.
LSM(least square method) is only one way of a linear
regression. )
Kim Daniil
Open Problem 4:
Although two terms have similar
names, they are in fact different:
- Least-Squares solution gives us optimal solutions to the
equation, and we find it via different Matrix Mutations.
- Least-Squares Regression Line is a linear function, that
visualizes the line, that has a minimal variance (sum of squares of the
errors or in other words, vertical distances). And we usually find it via
Calculus, or Slope/Tangent and Intercept evaluation to be precise.
However, despite that, we can
find the Least-Squares Regression Line by finding the Least-Squares Solution.
Prof. Lee Sang-Gu discussed it in the previous lessons and defined it as (the
least-squares line, the best fit line).
PRAVAS GIRI
Open Problem 4:
Linear regression attempts to
model the relationship between two variables by fitting a linear equation to
observed data.
Least square line is the line
that makes the vertical distance from the data points to the regression line as
small as possible .
-------------------------------------------------------------------------------------------------------------
Finalized by Kim Daniil, Pravas
Giri, 김다솔,
Week 12 <Open Problem 4> Is there a difference between
linear regression and least squares?
KIM DASOL
Open Problem 4
(KOR) 최소제곱직선과 회귀분석에서 사용하는 선형회귀 사이의 같은 점과 다른 점에 대하여 이해한 바를 토론하시오.
최소 제곱 해는 변수들 사이에 순서가 없다.
그러나 선형회귀에는 종속변수와 독립변수가 존재한다.
'직선'에 포커스를 맞추면 변수가 세 개 이상이 되면 최소제곱직선이라고 부를 수 없게 된다. (그러나 여전히 최소제곱해는 존재한다.)
선형회귀는 두 변수 사이의 회귀직선 뿐만 아니라 세 개 이상의 변수도 사용해 회귀분석을 진행하곤 한다.
이 때에도 여전히 종속변수와 독립변수들이 나뉘게 된다.
다변수에 대한 비교는 최소제곱해와 선형회귀의 차이를 뚜렷하게 나타내준다. 만약 100개의 변수가 있다면 최소제곱해는
100 차원의 벡터들이 최소 제곱 오차를 갖게 되는
99차원의 초평면을 해로 갖게 된다.
선형회귀는
1개의 종속변수에 99개의 독립변수가 가장 잘 맞도록 하는 회귀직선을 갖게 된다.
제 이해를 바탕으로 한 제 생각이니 더 많은 의견과 사실들을 댓글로 남겨주시길 바랍니다.
(ENG) Debate the same and
different things between least-squares line and linear regression in regression
analysis.
There is no order in the variables
of least square solutions. But, in linear regression, there are one dependent
variable and independent variables. Multivariate operating clears the
difference between LSE and linear regression. With 100 variables, LSE has
99-dimensional hyperplane as a solution. Linear regression has one regression
line fitting 99 independent variables to one dependent variable.
This is just my thought from my
understanding. Please let me know your thought and the facts about the question
with under comments! Thank you :)
>> 이해한 바 by
박준호
종속변수인 y를 추측하기 위해 선형회귀를 구하는 만큼, y가 curve에 영향을 주지 않도록 하는 과정이라고 이해하였습니다.
2comments
이상구(LEE SANGGU)November 24, 12:50 PM
Other
classmate shoud add your comment on this problem/solution and Finilize it. In
order to pass this course, you will start to make your Final PBL Report.
기리프라바스(2021####24)November 24, 12:50 PM
Thank
you.
PRAVAS GIRI
Open Problem 4:
Linear regression attempts to
model the relationship between two variables by fitting a linear equation to
observed data.
Least square line is the line that makes the vertical distance from the data
points to the regression line as small as possible .
What
is the difference between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares
…
First, precisely speaking
there is a difference between linear regression and curve
fitting. Everything I said above is actually talking about curve fitting,
not linear regression: you have a set of points, and you want to draw a curve
(line) through them that fits as well as possible. This is a purely geometric problem.
The x and y axes have no interpretation, and the "data" are just
points in Cartesian space.
On the other hand, linear
regression is a statistical inference problem. The "y
values" take on the interpretation of data you wish to model, and the
"x values" take on the interpretation of extra information you have
about each data point that might be helpful in predicting their "y
values". You are trying to build a probabilistic model that describes
"y" while taking into account "x", and a linear model is
one of many ways to do this. A linear model assumes that "y" has a
different mean for each possible value of "x", and that these means
happen to follow a straight line with a certain intercept and a certain slope.
As with any statistical inference problem, you estimate the unknown parameters
using maximum likelihood estimation. But since in this case the unknown
parameters are an intercept and a slope, the end result of maximum likelihood
estimation is basically that you are choosing a straight line that fits the
observed data best, so this essentially becomes the curve fitting problem discussed
above.
Now we arrive at the question
of why least squares, of all possible curve fitting methods, is so commonly
used. The reason is that when solving the statistical linear
regression problem, a very common modeling assumption is that for every possible
value of "x", the quantity "y" is normally
distributed with a mean that is linear in "x". Therefore,
the likelihood function is essentially a product of PDFs of the normal
distribution. As stated above, you estimate the unknown parameters (and
therefore find the best fitting line) by maximizing the likelihood function. If
you look at what the product of normal PDFs looks like, you will notice that
maximizing this expression happens to be equivalent to... you guessed it...
minimizing the sum of squared errors!
That is, the line you get performing curve fitting via least
squares is equivalent to the line you get performing linear regression using a
normal model.
>> 이해한 바 by 박준호
'least squares is one method for performing linear regression.' 라는 문장을 고려하면
최소제곱법은 선형회귀에도 사용되는 하나의 방법 이라 이해하였습니다. 선형회귀는 훨씬 큰 개념입니다.
그러나 통계에서 최소제곱직선이 아닌 선형회귀를 사용하는 이유가 있을 것이고, 그것이 종속변수의 존재때문이라고 생각하게 되었습니다.
Prev
Next
1comments
이상구(LEE SANGGU)November 25, 1:44 PM
What
is the difference between linear regression and least squares?
https://www.quora.com/What-is-the-difference-between-linear-regression-and-least-squares...
minimizing the sum of squared errors! That is, the line you get performing
curve fitting via least squares is equivalent to the line you get performing
linear regression using a normal model.
Kim Daniil
Open Problem 4:
Although two terms have similar
names, they are in fact different:
- Least-Squares
solution gives us optimal solutions to the equation, and we find it via
different Matrix Mutations.
- Least-Squares
Regression Line is a linear function, that visualizes the line, that has a
minimal variance (sum of squares of the errors or in other words, vertical
distances). And we usually find it via Calculus, or Slope/Tangent and
Intercept evaluation to be precise.
However, despite that, we can find
the Least-Squares Regression Line by finding the Least-Squares Solution. Prof.
Lee Sang-Gu discussed it in the previous lessons and defined it as (the
least-squares line, the best fit line).
>> 이해한 바 by
박준호
이 둘 linear regression and least
squares 은
they seem to be used interchangeably. But more precisely, least squares is a
method for performing linear regression 입니다. 부분집합의 관계 정도로 이해 됩니다. LSM(least square
method) is on;ly one way of a linear regression.
'Least-Squares
Regression ... has a minimal variance' 의 variance는 y값의 오차이고, 최소제곱해를 구함으로서 도출됩니다,
따라서 최소제곱직선과의 차이점이라고 한다면 최소화하고자 하는 variance에 있을 것이라 생각할 수 있었습니다.
Finalized
by Pravas Giri, 김다니일
<Week-8 Summary>
Kim Daniil:
Summary:
My word summary:
Prev
Next
3comments
이상구(LEE SANGGU)October 15, 2:59 PM
Very
good^^
박준호(2014####20)October 17, 6:58 PM
I
tried your code and it is very nice!
기리프라바스(2021####24)October 20, 6:33 PM
Thank
you for the clear explanation.
PRAVAS GIRI
SUMMARY:
In this week we learnt
about
1. Application of
Derivatives.
- If the f'(x)>0 then f
increases on interval [a,b].
- If the f'(x)<0 then f
decreases on interval [a,b].
- If the function is
increasing then the slope of the tangent line is +ve and the curve of the
function is above tangent line.
- If the function is
decreasing then the slope of the tangent line is -ve and the
curve of the function is below tangent line.
2. Application of Second
Derivatives.
- The curve y=f(x) is
concave up (convex down) at (c,f(c) if (x,f(x)) lies above the tangent at
(c,f(c)) for all near c.
- The curve y=f(x) is
concave down (convex up) at (c,f(c) if (x,f(x)) lies below the tangent at
(c,f(c)) for all near c.
3. Local Maximum, Local
Minimum.
When a function f has f',
f" at c in the domain with f'(c)=0.
- a. if f"(c)>0,
then f(c) is a local maximum.
- b. if f"(c)<0,
then f(c) is local minimum.
4. Absolute Maximum,
Absolute Minimum.
- Find all critical numbers
of f within
the interval [a, b]. That is, set f '(x) = 0, solve
for x,
and only consider those solutions x that
satisfy a ≤ x ≤ b.
- Plug in each critical
number from step 1 into the function f(x).
- Plug in the
endpoints, a and b, into the
function f(x).
- The largest value is the
absolute maximum, and the smallest value is the absolute minimum.
Prev
Next
3comments
이상구(LEE SANGGU)October 20, 7:57 PM
Good^^
기리프라바스(2021####24)October 22, 7:43 AM
Thank
you sir.
김다니일(2021####39)November 29, 2:11 PM
Thank
you for finalizing my answer and adding a more thorough explanation of the abs.
max/min evaluation method.
Finalized
by Pravas Giri, 김다니일,
박준호
<Week 8-Chapter 2 OPEN PROBLEM 2>
18. Kim Danil
19. Chapter 2 Open Problems:
#2:
My Solution:
Code:
I made it completely interactive,
so you can easily enter your own equation and interval, and the code will solve
it.
# W8 (Kim Daniil) Chapter 2 Open
Problem 2
# Making it interactive
@interact
def _( temp1 =
input_box(x^4-6*x^2+1,label="f(x)=", width=30),
temp2 = input_grid(1, 2,
default=["-1","3"], label="Interval:", width=6),
zoomX = slider(1, 100, step_size=0.5,
default=5, label="Zoom X:", display_value=True),
zoomY = slider(1, 100, step_size=0.5,
default=30, label="Zoom Y:", display_value=True),
scale = slider(4, 40, step_size=1,
default=15, label="Graph Scale:", display_value=True)):
# Converting a temp1 input to a function.
f(x) = temp1
# Converting a temp2 grid to the interval temp2[0] means first row in
grid temp[2], which returns a 1-dim array e.g [-1,3]
interval = temp2[0]
show("Equation: f(x)=", f(x))
show("Interval: ", interval)
df(x) = diff(f(x),x)
show("First Derivative: f'(x)=", df(x))
d2f(x) = diff(df(x),x)
show("Second Derivative: f''(x)=", d2f(x))
plot_f = plot(f(x), x, color="green", xmin=-zoomX,
ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="Function",
figsize=(scale,scale))
plot_df = plot(df(x), x, color="red", xmin=-zoomX,
ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="1st Derivative",
figsize=(scale,scale))
plot_d2f = plot(d2f(x), x, color="blue", xmin=-zoomX,
ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="2nd Derivative",
figsize=(scale,scale))
plot_interval = line2d([(interval[0], -zoomY), (interval[0], zoomY)],
linestyle="--", color="black", alpha=0.5) +
line2d([(interval[1], -zoomY), (interval[1], zoomY)], linestyle="--",
color="black", alpha=0.5)
# Finding critical points.
t = solve(diff(f(x)) == 0, x)
# Transforming SageForm ( [x == 1, x == 2 ...] ) to normal array of values
( [1, 2 ...] )
cPoints = [i.rhs() for i in t]
# Add the interval ends, as critical points:
cPoints.append(interval[0])
cPoints.append(interval[1])
# Creating a dictionary of (x,f(x)) values:
cPointsVals = {}
# List of local extremes
locM = []
# Finding extreme points
show("All Local Extreme Points:")
for j in cPoints:
k = f(j)
v = d2f(j)
# Finding local maxima and minima:
if df(j) == 0:
l = vector([j,k])
locM.append(l)
show("Local", "
minimum" if v > 0 else " maximum",":", l)
# Calculating values of the function at
the extreme points to later find absolute extreme points
if j >= interval[0] and j <=
interval[1]:
cPointsVals[j] = k
# Finding absolute maxima and minima
absMaxX = max(cPointsVals, key=cPointsVals.get)
absMinX = min(cPointsVals, key=cPointsVals.get)
# List of abs extremes
absM = [
vector([absMaxX,cPointsVals[absMaxX]]),
vector([absMinX,cPointsVals[absMinX]])
]
# Printing
show("Interval Absolute Extreme Points:")
show("Absolute Maximum:", absM[0])
show("Absolute Minimum:", absM[1])
plot_loc = list_plot(locM, color="purple", legend_label="Local
Extremes")
plot_abs = list_plot(absM, color="orange",
legend_label="Absolute Extremes")
show(plot_f +plot_df +plot_d2f + plot_interval + plot_loc + plot_abs)
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
3comments
이상구(LEE SANGGU)October 15, 2:59 PM
Very
good^^
박준호(2014####20)October 17, 6:58 PM
I
tried your code and it is very nice!
기리프라바스(2021####24)October 20, 6:33 PM
Thank
you for the clear explanation.
박준호
Daniil's code is very useful to
me, but I think it can be improved!
(source: http://matrix.skku.ac.kr/intro-math4ai/w8/)
There are critical points at which
f(x) is non-differentiable.
t
= solve(diff(f(x)) == 0, x)
#
Transforming SageForm ( [x == 1, x == 2 ...] ) to normal array of values ( [1,
2 ...] )
cPoints = [i.rhs() for i in t]
so I
suggest that 'cpoints'(or 't') contain points at which not only f'(c) = 0 but
also f'(c) is undefined.
In
fact, some errors occur as differentiate the functions with non-differentiable
points, so I can't correct the problem in my hand...
Prev
Next
6comments
이상구(LEE SANGGU)October 17, 10:23 PM
Good
point~%
김다니일(2021####39)October 17, 11:14 PM
Yeah,
I see. But, the problem is, after digging into the SageMath documentation, I am
yet to find a way to implement Fermat's Theorem. I found one implementation,
but it seems over-engineered (https://wiki.sagemath.org/Differential_Calculus_of_many_variables),
and honestly not a fan of it. Will try to figure something out. Perhaps, you
can suggest something? Anyways, thank you for pointing out!
김다니일(2021####39)October 18, 2:15 PM
So,
my best bet is, to find the inflection points (since inflection points do not
have the condition for f''=0 be undefined), then get these intervals (from 1
inflection point to another), and plug them into the standard
find_local_maximum(INTERVAL). https://doc.sagemath.org/html/en/reference/numerical/sage/numerical/optimize.html#sage.numerical.optimize.find_local_maximum
박준호(2014####20)October 18, 5:40 PM
Using
"find_local_maximum" seems to give just an interval, not an exact
point. I don't know what the algorithm of that code, so it may not be a good
way... In my opinion, we should use directly differentiability, but there is no
reference about finding non-diff points of f(x) or even undefined points of
f'(x) in calculus section. I find "differentiable" as an keyword in
this section(https://doc.sagemath.org/html/en/reference/manifolds/sage/manifolds/differentiable/manifold.html)
though it may not be useful to us.
이상구(LEE SANGGU)October 18, 8:49 PM
Your
Discussions were very valuable. Yes, it can not be done by a simple line of
code. Now move on to the next topics in the text book.
기리프라바스(2021####24)October 20, 6:35 PM
Thank
you.
20. Pravas Giri
Chapter-2
Open Problem-2
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
2comments
이상구(LEE SANGGU)October 20, 6:53 PM
Good.
OK. It will be better, if you plot the graph as well. Sample: var('x,y')
p1=plot(5*x^3 + 3*x^2 -4*x - 1, x, -2.5,2, color='blue'); p2 = text("$y=5x^3
+3x^2 -4x-1 $", (-1.4,15), fontsize=15, color='blue') show(p1+p2, ymax=20,
ymin=-10)
기리프라바스(2021####24)October 22, 8:02 AM
Okay
sir. Thank you.
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
2comments
이상구(LEE SANGGU)October 27, 1:28 PM
Take
a look on a simple Sage interact ========================== @interact def _(a =
slider(-10, 10, step_size = 1, default = 0, label = "a",
display_value = True)): f(x) = 2^(-x^2 + 4*x + a) print("f(x) =",
f(x)) print("a =", a) p1 = plot(f(x), (x, 0, 3)) p2 = plot(4, (x, 0,
3), color = 'red') show(p1 + p2) You may practice in http://matrix.skku.ac.kr/KOFAC/
and improve your coding to be iteracted^^
김다니일(2021####39)December 1, 00:27 AM
Yeah,
I think with the addition of the graph, it looks much better :D.
Finalized
by Pravas Giri, 김다니일,
박준호
<Week 9- SUMMARY>
PRAVAS GIRI
Summary:
Gradient Descent Method (GDM):
When a function is complicated
then the critical point can be found through GDM.
The basic idea is to find the
slope of the function (derivative), move it toward downhill (learning rate), and
repeat it until it reaches the extreme value.
21. 김다니일
22. Summary:
Gradient Descent Method/Algorithm is an
optimization algorithm to find the minimum of a complex function.
The general idea of the Gradient Descent Method (GDM) is to:
- Find the slope/gradient of
the function. Basically, find the derivative.
- Pick a random point
and move in the direction of the negative
gradient (a direction, in which function decreases the
most), by a certain coefficient , that is called learning rate (usually,
it is a number in range (10)).
- Repeat the process
until we reach the extreme value.
The GDM can
also be used to solve the Least
Squares Problem the idea is to find the solution with the
minimized Error, in other words, we apply the GDM to the error
function to find the most optimal solution.
박준호
Summary:
Gradient Descent Method:
(source : http://matrix.skku.ac.kr/intro-math4ai/w9/)
Using GDM to get Least Square
Solution:
set an Error Function E(u) and we
minimize the error.
(source : http://matrix.skku.ac.kr/intro-math4ai/w9/)
Finalized
by Pravas Giri, 김다니일,
박준호,
김다솔
<Week 9- OPEN PROBLEM 3>
PRAVAS GIRI
Open Problem 3:
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
23. 김다니일
24. Chapter 2 Open Problems:
I wrote another convenient interactive
sagemath module, so everyone can easily play with the values and adapt for
their particular equations.
Note: I modified the algorithm,
that was in the video. The reason is, that in this case, Python loops perform
slower (for i in range(x)), in comparison with C loops (while x).
Code:
# W9 (Kim
Daniil) Chapter 2 Open Problem 3 and 4
# Making it
interactive
@interact
def _( temp1 = input_box(9*x^2-7*x+6,label="f(x)=", width=30),
threshold = slider(0.000001, 2, step_size=0.000001, default=0.000001, label="Threshold/Tolerance:", display_value=True),
l_rate = slider(0.00001, 1, step_size=0.00001, default=0.1, label="Learning
Rate:", display_value=True),
base_point = input_box(0.0, label="Base Point:", width=20),
terminator = slider(100, 10000, step_size=1, default=300, label="Terminate
after:", display_value=True),
zoomX = slider(1, 100, step_size=0.5, default=5, label="Zoom X:", display_value=True),
zoomY = slider(1, 100, step_size=0.5, default=30, label="Zoom Y:", display_value=True),
scale = slider(4, 40, step_size=1, default=15, label="Graph
Scale:", display_value=True)):
#
Converting a temp1 input to a function.
f(x) = temp1
show("Equation:
f(x)=", f(x))
df(x) = diff(f(x),x)
show("Slope:
f'(x)=", df(x))
# GDM
implementation
i_points = []
iterator = 0
my_x = base_point
#Starting point
g = threshold+1
while ( iterator < terminator
and g > threshold ):
g = df(my_x)
i_points.append(vector([my_x, g]))
my_x = my_x - l_rate*g
g = abs(g)
iterator +=1
final_point = vector([my_x, f(my_x)])
show("x* = ", final_point[0])
show("|g*| = ", g)
show("f(x*) = ", final_point[1])
show("Achieved in
'", iterator, "' attempts.")
plot_f = plot(f(x), x, color="green", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="Function", figsize=(scale,scale))
plot_df = plot(df(x), x, color="blue", xmin=-zoomX, ymin=-zoomY, xmax=zoomX, ymax=zoomY, legend_label="1st
Derivative", figsize=(scale,scale))
plot_i_points = list_plot(i_points, color="red", legend_label="Iteration
Points")
plot_final_point = list_plot([final_point], color="purple", legend_label="Result")
show( plot_f + plot_df + plot_i_points + plot_final_point )
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
#3:
박준호
open problem 3
We can practice the above
code in http://matrix.skku.ac.kr/KOFAC/
and in https://sagecell.sagemath.org/
using Daniil Kim's code (I
changed some 'sliders' to 'input_boxes')
we can see the converge speed
is too slow and the
destination(x*; f'(x*) = 0) is not local minimum!
Therefore the initial iterator and learning rate should
be properly set.
김다솔
122p 열린문제 3
함수
f(x) = 9x^2-7x+6의 최솟값을 구하시오.
단
x_1 = 0, eta=0.1, epsilon=10^-6으로 한다.
풀이
initial state
f(x_1) = 6
update formula
x <- x - eta*dx
x_1에서의 derivative를 구하면
f'(x)=18x-7
dx = f'(x_1) = -7
update formula에