[K-MOOC]  Introductory Mathematics for Artificial Intelligence

                Translated by

  Sang-Gu LEE with Youngju NIELSEN, Yoonmee HAM

         from the original Korean text written by

      Sang-Gu LEE with Jae Hwa LEE, Yoonmee HAM, Kyung-Eun PARK

    Part Ⅴ. PCA and ANN

14.  Hand Writing Numbers detection using ANN on MNIST Dataset

  14.1 Hand Writing Numbers detection using Artificial Neural Network

For a long time, post offices have spent a lot of effort in recognizing handwritten zip codes, sorting them, and then the physical process of delivering mail.

This section shows how the artificial neural network recognizes the handwritten numbers in the MNIST data set and finds the closest number.

It is called <Handwriting Number Recognition Using Artificial Neural Network>. We will learn about the process and principle of how artificial intelligence,

instead of humans, does the process of automatically classifying envelopes so that the postal delivery department in charge of the area

can carry out their work with minimum delay. We practice the process that AI figure out what the zip code we wrote.

We explain <Examples of Handwritten Number Recognition Using Artificial Intelligence> and introduce related codes.

The Backpropagation and the Gradient descent method are the main algorithms to update Artificial Neural Network.

14.1 Hand Writing Numbers detection using ANN           

In the past, many people were involved in [Recognition of Handwritten (postal) ZIP Codes in a Postal Sorting system].

In this section, we learn about how Computer/AI do recognize Handwritten ZIP Codes by artificial neural networks.

Now, artificial intelligence (Robot) is replacing the workforce at the post office by recognizing and sorting ZIP code and sorting them out by themselves.

If we look at the images here, the letters 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 can be written in various ways by each person.

In order to utilize the artificial neural network, as shown in the picture above, hand-written numeral images (number) are needed first.

MNIST(Modified National Institute of Standards and Technology) database contains approximately 70,000 numeric images in black and white,

28 pixels wide by 28 pixels tall. Of these, we divide it into 60,000 learning data and 10,000 test data. Each data contains a hand-written image (with a label)

of numbers from 0 to 9. The most widely used MNIST data for benchmarking the performance of machine learning can be found on the Web site below.

  [Source]  THE MNIST DATABASE of handwritten digits

           http://yann.lecun.com/exdb/mnist/

Now let's check out the number images of the MNIST data.

Here, we have modified the Python code to Sage, which was released on a GitHub page in https://github.com/freebz/Make-Your-Own-Neural-Network .

As the title 'Make your own neural network' show, it works with the code that fits us. We added detailed explanations to make it easier to be understood.

Let's read the code and try to understand the algorithm. Then we can practice it in the Lab http://matrix.skku.ac.kr/math4ai-intro/W14/.  

The first part of the code calls <numpy and matplotlib> libraries and some part of the code needs libraries.

Let's check the numerical image of the MNIST data. The following was implemented in Sage with some modifications of the Python code from

https://github.com/freebz/Make-Your-Own-Neural-Network 

When we copy and execute the command in our lab, the number 7 of training_data_list[0] appears.

And the image is created in a png file foo.png as follows.

At this time, the number stored in training_data_list[0] was 7. And when we click on this, the number 7 is shown as an image as we see now.

If we change this number [0] to training_data_list[1] in the same way, the stored number is 2. So the image 2 will be seen.

In other words, if we choose another (number) image, ANN will tell us what that number was.

             all_values = training_data_list[1]

An image of a number is saved as a matrix.

① Let's take a look at how the images of a number were converted into a matrix. Divide the image into sizes of 28 pixels wide by 28 pixels tall.

② Now, we have an image/matrix of 784= pixels. And we classify the brightness of each pixel with the natural numbers.

The brightness of 0 is set to black and the number 255 to white, and then we have a matrix whose entries indicating its brightness.

③ For example, “the horizontal 12th and vertical 6th pixel and brightness is 82” would be expressed as (12, 6, 82).

Thus, a number image can be represented in a matrix as shown below.

Here, all black image is a zero matrix. A colored area is marked with the number 82.  Now the letter 8 is shown below.

It can be understood that this matrix data means a hand-written letter.

Now, if we look at the process of recognizing the image of a number through an ANN, the image of a number in our learning data is a matrix of 28×28,

which converts it into a 784×1 vector.

Each component of this vector is an input signal and will be sent to the node of the input layer. Then the output layer through the ANN

will give us an answer.

■

If there is an error from comparing this with the correct answer, the neural network's weight is adjusted

using the gradient descent method and the Backpropagation algorithm. After all these learning data were used (when the learning process is completed),

the remaining test data will be used to determine our neural network's performance.

The following is a neural network of the input, hidden layers, and output layers. This code shows how ANN worked

when we use 100 learning data and 10 test data in it.

[Source]

https://github.com/freebz/Make-Your-Own-Neural-Network 

https://media.githubusercontent.com/media/freebz/Make-Your-Own-Neural-Network/master/mnist_dataset/mnist_train_100.csv 

https://media.githubusercontent.com/media/freebz/Make-Your-Own-Neural-Network/master/mnist_dataset/mnist_test_10.csv 

The neural network's input layer for recognizing numbers consists of 784 nodes, and the output layer consists of 10 nodes representing numbers from 0 to 9.

There are no specific regulations concerning the number of nodes in the hidden layer, but the number of nodes in this section was set at 100.

Let's take a closer look at the lines in the code.

This code explains that it learns from the data with a three-layer neural network. It starts to import a 'numpy' and load 'scipy.special' to use the 'sigmoid function'.

It imports a library for visualizing matrices. It defines 'Class of neural network' as 'neuralNetwork' and initializes a 'Neural network'.

Then it defines 'input, hidden layer, output, learning rate, and weight matrix, and the weights in the array are denoted by

which means connecting from node to node , in the next layer. It defines the learning rate and the activation function as a Sigmoid function.

After we run the above algorithm (the result of our practice), we learned the following. When we started this ANN with 100 training data and 10 test data,

this neural network's accuracy was 70%, which is not bad. It took only a very short time. However, as the number of data increases to 60,000 training data

and 10,000 test data from 100 training data and 10 test data, we see that the same algorithm's accuracy is roughly 95% accurate

(more accurate than humans). More details in https://linux-blog.anracom.com/2019/09/29/a-simple-program-for-an-ann-to-cover-the-mnist-dataset-i/.

The case of recognizing hand-written numbers in the MNIST data set was possible because enough data was available to train the neural network.

Previously, there was not much data available, and the computer took a lots of time to perfome the task. But nowadays, a lot of data

is being produced every day, and AI systems have been improving better and better with vast amounts of training data.

As with the example of [Recognition of Hand-written number with MNIST dataset], it is possible to recognize speech, signals, or text similarly.

Furthermore, research that processes our speech (our natural language is more challenging to recognize than hand-writing, numbers,

voices, signals, and letters) is ongoing.

Note: PCA is used to analyze MNIST data. The original data has 6304 rows and 785 columns. The shape of handwriting data is described

using 24 by 24 (784) pixels. The Principal Components were constructed using 784 pixel data from the columns. The figure shows 0 to 9

based on the first two Principal Components.

[Source]  https://ryanwingate.com/intro-to-machine-learning/unsupervised/pca-on-mnist/

We showed the result of using PCA in MNIST data. The following figure is the result of using t-Stochastic Neighbor Embedding(t-SNE).

t-SNE does a similar job as PCA. It represents high dimension data on a lower dimension as PCA does. However, it uses a non-linear methodology,

vs. PCA is a linear methodology. Satisfactory results require computationally extensive methods such as t-SNE.

[Source]  https://bigsnarf.files.wordpress.com/2016/11/tsne_mnist_all.png

With the help of Natural Language Process, ANN will analyze the real meaning of what we said and will try to give us the best possible answers.

This should enable us to react faster and make better decision.

For example, when I say "I want to pizza!" to a secretary in my smartphone, the AI system goes further to understand the meaning.

'Oh, this person does not want to make pizza but wants to buy or eat pizza from recognizing there is no term in the text such as 'recipe'.

'After AI understand the true meaning of what I said', AI understand that I said, "I want to eat pizza!". Next, AI uses GPS information

to find restaurants that serve pizza near me. Then AI order the restaurants by distance and cost-effectiveness. After finding a restaurant in this way

and figuring out the restaurant's name and address. AI is ready to make recommendations. Lastly, a message is generated in natural language sentences.

For example, AI may returned the message "There is a Gino's Pizza house in the right of 1km ahead." And if we ask AI in a driveless car

to take me to COEX convention center, AI will recognize the meaning and find the best route through the navigation and drive me safely to COEX.

While AI driving the car, we can read, prepare for a talk, and be ready for the work to be done there. It will be a lifestyle that we will enjoy sooner or later.

◩ Open Problem 6

  Find out some other examples on number recognition, face recognition, voice recognition, and natural language

generation technologies currently used with ANN.

◩ Open Problem 7

Find and discuss topics in mathematics that have been used in artificial intelligence.  http://matrix.skku.ac.kr/math4ai/part4/

[Week 13]

◩ Open Problem 1

Explain how a singular value decomposition (SVD) is used in the  principal component analysis.

◩ Open Problem 2

Take a data matrix from sources related to your major, then use the code above to apply a Principal Component Analysis (PCA)

algorithm to this matrix. Then, post the results on the QnA Board for discussion. It will help you to understand the topic.

◩ Open Problem 3

Discuss what you understand on the process of dimension reduction by PC’s on the covariance matrix.

◩ Open Problem 4

Discuss what you  understand on the similarity and difference between the least-squares line and the linear regression.

◩ Open Problem 5

Describe ANN and Backpropagation as you understand.

[Week 14]

◩ Open Problem 6

  Find out some other examples on number recognition, face recognition, voice recognition, and natural language

generation technologies currently used with ANN .

◩ Open Problem 7

Find and discuss topics in mathematics that have been used in artificial intelligence.

            [Source] http://matrix.skku.ac.kr/KOFAC2/

Copyright @ 2021 SKKU Matrix Lab. All rights reserved.
Made by Manager: Prof. Sang-Gu Lee and Dr. Jae Hwa Lee