Feedforward Neural Networks

Mathematically, a perceptron works by taking many inputs as a vectorIn school, you typically get taught that a vector is a thing with both a magnitude and a direction. However, in the context of neural networks and computer science in general, it's more helpful to just think of a vector as a long (or short) list of numbers. \(( x_1, x_2, …, x_{n-1}, x_n)\) and multiplying each value by a corresponding weightA weight is a number that determines how much influence a particular input has on the perceptron’s output. \(w\) from \((w_1, w_2, \cdots, w_{n-1}, w_n)\). These results are added with a bias termA bias is an extra number added to the weighted sum of inputs that allows the perceptron to shift its decision boundary. \(b\), which produces a weighted sum: \(z = w_1x_1 + w_2 x_2 + \cdots + w_n x_n + b\). This value \(z\) then goes through an activation function An activation function is a rule that decides the perceptron’s output based on the weighted sum of its inputs.. In Rosenblatt's design, the perceptron used a simple step function
(H.Y. using Desmos)
Step functions are piecewise functions made up of sections of horizontal lines. A common variation of the step function equals \(0\) up until \(x=0\), after which the function equals \(1\): if \(z > 0\), the perceptron gives \(1\) as its output. If not, it returns \(0\), therefore dividing the input spaceThe input space is the set of all possible values that the perceptron’s inputs can take, often visualised as a coordinate system where each axis represents one input. into 2 distinct regions separated by a straight line (or plane).

Perceptrons are valuable because of how they learn these weights and the bias. When a perceptron makes an incorrect prediction, the weights are changed slightly using the rule:

\[w_i \leftarrow w_i + \eta(y-\hat{y}) x_i\]

In this formula, \(y\) is the actual output, \(\hat{y}\) is what the perceptron guessed, and the Greek letter \(\eta\) (eta) is the learning rate. This rate is a constant that controls how large the change should be. This rule slowly moves the decision boundary until the perceptron properly divides the training data, assuming the problem can be separated by a straight line. This concept is quite simple, but it is the basis for how neural networks adjust when training, even now.

If this doesn't make sense to you right now, don't worry. In the next tutorial we will look more intuitively and deeply into how this concept of self-learning works. In the meantime, just know that for 1 perceptron, its pretty easy to make it "learn", and that there exists a formula that dictates this.

If you already know the basics of how to manipulate matrices and vectors, and you understand summation notation, continue reading, but if you don't, here's a crash course on it:

I suppose now is a good time to introduce you to the standard notation used in neural networks. This notation helps us describe how data flows through the layers of a network:

• We typically use the letter \(x\) to represent our inputs as a vectorIn school, you typically get taught that a vector is a thing with both a magnitude and a direction. However, in the context of neural networks and computer science in general, it's more helpful to just think of a vector as a long (or short) list of numbers.. For example, we can have the input vector \(x=(1.98, 12.04, 16.33, 0.70, \cdots)\). Each individual number inside this vector is represented as \(x_i\) with subscript \(i\), where \(i\) is the "index" or place where the number is in the vector. For example, \(x_2=12.04\) in the above example. (Quick side note: most coding languages start counting from \(i=0\), but because mathematicians like to start counting from \(1\), neural network notation also starts at \(x_1\)).
• Now, we write the vector output of the \(l^{th}\) layer of a network as \(a^{(l)}\) (superscripted \(l\)). This means that the \(j^{th}\) neuron in the \(l^{th}\) hidden layer of a neural network is \(a_j^{(l)}\). For example, the second node in the first hidden layer is represented as \(a_2^{(1)}\).
• Since the vector output is a weighted sum with the activation function applied, we can also write \(a^{(l)}=f(z^{(l)})\), where \(z^{(l)}\) is the weighted sum and \(f\) is the activation function. For example, if we had the third layer equal the weighted sum of the neurons in the second layer with a sigmoid
  (H.Y. using Desmos)
  Equation: \(\sigma(x)=\frac{1}{1+e^{-x}}\) activation, we would write that \(a^{(3)}=\sigma(z^{(3)})\). **. Similar to the previous bits of notation, we can also reference individual nodes using subscript indexing like \(\sigma(z_i^{(l)})\).
• Now the weighted sum, otherwise known as the "linear combination", \(z^{(l)}\) is itself composed of the previous nodes multiplied by some weights with an added bias term. All of the weights between layers \((l-1)\) and \(l\) are represented as \(W^{(l)}\), while the individual weight can be found by using 2 indices (\(i\) for the input neuron and \(j\) for the output neuron) as such: \(w_{ji}^{(l)}\) (Why not \(ij\)? This will be explained very shortly, but its basically to make our lives easier when doing some of the maths.). It is important to remember that while \(x\) and \(a^{(l)}\) are vectors (1D arrays of numbers), \(W^{(l)}\) is a matrix \[W^{(l)} = \begin{bmatrix} w_{11}^{(l)} & w_{12}^{(l)} & w_{13}^{(l)} & \cdots & w_{1n}^{(l)}\\ w_{21}^{(l)} & w_{22}^{(l)} & w_{23}^{(l)} & \cdots & w_{2n}^{(l)}\\ \vdots & \vdots & \vdots & \ddots & \vdots \\ w_{m1}^{(l)} & w_{m2}^{(l)} & w_{m3}^{(l)} & \cdots & w_{mn}^{(l)} \end{bmatrix}\]
  A matrix is a rectangular grid of numbers or variables. In the context of neural networks, we usually talk about 2D matrices that represent all the weights (and sometimes biases) between two layers. These matrices normally have \(m \times n\) shape where \(n\) is the input index and the number of columns, and \(m\) is the output index and the number of rows. (a 2D array of numbers).

  Weight Matrix Diagram (H.Y. using Canva and Overleaf)

  Therefore, we write \(z^{(l)}=W^{(l)} a^{(l-1)} + b^{(l)}\), or \(z_{j}^{(l)}=w_{j1}^{(l)}a_1^{(l-1)}+w_{j2}^{(l)}a_2^{(l-1)}+\cdots+w_{jn}^{(l)}a_n^{(l-1)}+b_j^{(l)}\).
This equivalence is made possible by the very specific way that matrix / vector multiplication works.

Say for example we had the first hidden layer with 4 nodes, and the second with 3. Then we would write: \[a^{(2)}=f(Wa^{(1)}+b^{(2)})\] \[=f\left(\begin{bmatrix} w_{11}^{(2)} & w_{12}^{(2)} & w_{13}^{(2)} & w_{14}^{(2)}\\ w_{21}^{(2)} & w_{22}^{(2)} & w_{23}^{(2)} & w_{24}^{(2)}\\ w_{31}^{(2)} & w_{32}^{(2)} & w_{33}^{(2)} & w_{34}^{(2)} \end{bmatrix}\begin{pmatrix}a_{1}^{(1)} \\ a_{2}^{(1)} \\ a_{3}^{(1)} \\ a_{4}^{(1)}\end{pmatrix}+ \begin{pmatrix}b_{1}^{(2)} \\ b_{2}^{(2)} \\ b_{3}^{(2)}\end{pmatrix}\right)\] \[a^{(2)}=f\left(\begin{bmatrix} w_{11}^{(2)}a_{1}^{(1)} + w_{12}^{(2)}a_{2}^{(1)} + w_{13}^{(2)}a_{3}^{(1)} + w_{14}^{(2)}a_{4}^{(1)}\\ w_{21}^{(2)}a_{1}^{(1)} + w_{22}^{(2)}a_{2}^{(1)} + w_{23}^{(2)}a_{3}^{(1)} + w_{24}^{(2)}a_{4}^{(1)}\\ w_{31}^{(2)}a_{1}^{(1)} + w_{32}^{(2)}a_{2}^{(1)} + w_{33}^{(2)}a_{3}^{(1)} + w_{34}^{(2)}a_{4}^{(1)} \end{bmatrix} + \begin{pmatrix}b_{1}^{(2)} \\ b_{2}^{(2)} \\ b_{3}^{(2)}\end{pmatrix}\right)\]
Notice here how this only works if we index our nodes with the output index first and input second.

▬▬▬▬▬▬

Now that we have the mathematical tools to tackle MLPs, we can begin by writing out the formula for the output of a single neuron in our huge network:

\[a_j^{(l)}=f(w_{j1}^{(l)}a_1^{(l-1)}+w_{j2}^{(l)}a_2^{(l-1)}+\cdots+w_{jn}^{(l)}a_n^{(l-1)}+b_j^{(l)})\]

Or, using summation notation,

\[a_j^{(l)}=f\left(b_j^{(l)}+\sum_{i=1}^n w_{ji}^{(l)}a_i^{(l-1)}\right)\]

where \(n\) is the number of neurons in the previous layer.

Then, we can expand this to an entire layer in our neural network:

\[a^{(l)}=f(W^{(l)}a^{(l-1)}+b^{(l)})\]

Now, for our multilayer perceptron, we simply combine many of these single layer equations together:

\[a^{(1)}=f(W^{(1)}x+b^{(1)})\]

\[a^{(2)}=f(W^{(2)}a^{(1)}+b^{(2)})\]

\[\vdots\]

\[a^{(L)}=f(W^{(L-1)}a^{(L-2)}+b^{(L-1)})\]

\[\hat{y}=f(W^{(L)}a^{(L-1)}+b^{(L)})\]

where \(\hat{y}\) is the model's output or prediction, and \(L\) is the number of layers in the neural network (excluding the input neurons, since they don't do anything except hold data).