Flipping \(e_{simple}\)

So far we've been defining \(e_{simple} = (g - a)\), but in some places you'll see error defined as \(e_{simple} = (a - g)\). How does defining \(e_{simple}\) in this opposite way affect the adjustments to our weight? Let's recalculate the derivatives for MSE using this new \((a - g)\) measure for error, which we'll call \(e_{flipped}\).

\[ \begin{eqnarray*} \alignThreeEqs{ e_{flipped} \vphantom{\frac{\partial e_{mse}}{\partial e_{flipped}}} }{ a - g }{ -e_{simple} } \end{eqnarray*} \]

This flips the sign of our error, does it also flip the sign of our adjustment? Why or why not?

\[ \begin{eqnarray*} \alignThreeEqs{ \frac{\partial e_{mse}}{\partial e_{flipped}} }{ \frac{\partial}{\partial e_{flipped}}(e_{flipped})^2 }{ 2 \times e_{flipped} }\\ \alignThreeEqs{ \frac{\partial e_{flipped}}{\partial a} }{ \frac{\partial}{\partial a} (a - g) }{ 1 \tag{different} }\\ \alignThreeEqs{ \frac{\partial a}{\partial w_1} }{ \frac{\partial}{\partial w_1}(w_1 \times input + w_2) }{ input }\\ \alignThreeEqs{ \frac{\partial a}{\partial w_2} }{ \space\space \frac{\partial}{\partial w_2}(w_1 \times input + w_2) }{ 1 } \end{eqnarray*} \]

And now we see that the adjustments to our weights are flipped as well, but remember that the sign of \(e_{simple}\) has flipped as well, so our weights are changed by the same amount in the same direction.

\[ \begin{eqnarray*} e_{w1}'(input, e_{flipped}) &=& \frac{\partial e}{\partial w_1} &=& (2 \times e_{flipped})(1)(input)\\ &\vphantom{\frac{\partial e}{\partial w_1}}& &=& 2 \times e_{flipped} \times input\\ &\vphantom{\frac{\partial e}{\partial w_1}}& &=& -2 \times e_{simple} \times input \tag{same!}\\ e_{w2}'(1, e_{flipped}) &=& \frac{\partial e}{\partial w_2} &=& (2 \times e_{flipped})(1)(1)\\ &\vphantom{\frac{\partial e}{\partial w_1}}& &=& 2 \times e_{flipped}\\ &\vphantom{\frac{\partial e}{\partial w_1}}& &=& -2 \times e_{simple} \tag{same!}\\ \end{eqnarray*} \]

So if you ever see error definitions using \(a - g\) instead of \(g - a\), you can rest easy that all of the math works out exactly the same. Our neuron's weights will be updated the same direction either way.

Activation vs Error

As I was first learning about neural networks, it was at this point that things really started to make sense from the math perspective, but I was still unsure from the code perspective.

Most tutorials either used a library like Keras that had already separated error from activation, or by building a bare bones python network that had the error formula baked in and unchangeable, as we've done so far.

Specifically, in our case, the error calculations in errorDerivativeForWeight are tied much too tightly to the neuron itself.

    function errorDerivativeForWeight0(e_simple){
        e_deriv = 2 * e_simple // updated
        e_simple_deriv = -1
        a_deriv = input
        return e_deriv * e_simple_deriv * a_deriv
    }

It's convenient for the neuron to calculate its own error from a goal, and then to calculate its own e_deriv, but it makes it much harder to hot-swap out different error methods to see how they behave.

Ideally, the error calculations would be factored out from the neuron's code, and it wasn't immediately clear which pieces of that function should stay in the neuron, if any, and which would be pulled out into an Error class.

Looking at the math again, we can separate out the forward propagation into two sections: everything up to the neuron's activation (the weights, input, and neuron formula), and everything after the neuron's activation (the simple error and final error).

\[ \underbrace{ w, i \Rightarrow a }_{\text{neuron}} \Rightarrow \underbrace{ e_{simple} \Rightarrow e_{final} }_{\text{error}} \]

For backpropagation, we align these same responsibilities to their derivatives. Since we are splitting \(\frac{\partial e}{\partial w}\) into parts with the chain rule, the error function is responsible for the first part \(\frac{\partial e}{\partial a}\), and the neuron is responsible for calculating the last part \(\frac{\partial a}{\partial w}\).

\[ \begin{equation} \frac{\partial e}{\partial w} = \underbrace{ \frac{\partial e_{final}}{\partial e_{simple}} \frac{\partial e_{simple}}{\partial a} }_{\text{error}} \, \underbrace{ \vphantom{\frac{\partial e_{final}}{\partial e_{simple}}} \,\, \times \frac{\partial a}{\partial w} \,\, }_{\text{neuron}} \end{equation} \]

For us to be able to swap out error implementations, the derivative \(\frac{\partial e}{\partial a}\) that we calculate in e_deriv should be handled outside of the neuron, and \(\frac{\partial a}{\partial w}\) from a_deriv onward should stay inside the neuron. Our neuron shouldn't care at all about how the error's derivative was calculated, it only needs to know the final derivative.

In that separation above, the neuron needs to know about \(\frac{\partial e_{final}}{\partial a}\), and then it can calculate its own \(\frac{\partial a}{\partial w}\) and apply that error to each weight.

In code, this separation of responsibilities looks like:

    // Somewhere else, outside the Neuron class
    e_final_deriv = 2 * e_simple
    e_simple_deriv = -1
    e_deriv = e_final_deriv * e_simple_deriv

    // a much simpler function for the neuron
    function errorDerivativeForWeight0(e_deriv){
        a_deriv = input
        return e_deriv * a_deriv
    }

So our future error calculator will calculate e_deriv, and our neuron will still calculate a_deriv. That's the correct separation of powers, as the neuron controls its activation, so it should control the \(\frac{\partial a}{\partial w}\), and the error calculator handles calculating error from activation, so it should handle \(\frac{\partial e}{\partial a}\).

Refactoring Error Functions

Currently, our neuron's code contains 100% of the formulas that make the neural network work. Just as we've done in the math formulas above, I find it very helpful to separate out the error function from the neuron in our code too. This way, I can think of our \(a = w_1i + w_2\) neuron separately from our \((e_{simple})^2\) or \(\big|e_{simple}\big|\) error functions, whereas right now all of that code is jumbled into the same class.

Let's start by pull the errors out into their own classes. Each error option should be able to calculate its absolute error and its derivative with respect to activation, so we'll only need those two functions for each error.

abstract class SimpleError{
    function error(goal, activation){
        return goal - activation
    }
    function derivative(goal, activation){
        return -1
    }
}

class ABSError : SimpleError{
    function error(goal, activation){
        return abs(super.error(goal, activation))
    }

    function derivative(goal, activation){
        derivSimpleError = super.derivative(goal, activation)
        if (super.error(goal, activation) > 0) {
            derivABS = 1
        } else {
            derivABS = -1
        }
        return derivABS * derivSimpleError
    }
}

class MSEError : SimpleError{
    function error(goal, activation){
        simpleErr = super.error(goal, activation)
        return simpleErr * simpleErr
    }

    function derivative(goal, activation){
        derivSimpleError = super.derivative(goal, activation)
        derivMSE = 2 * super.error(goal, activation)
        return derivMSE * derivSimpleError
    }
}

Looking at these error classes, it's easier to see the requirements for building even more error functions: we need to calculate the error given a goal and activation, and also be able to compute its derivative with the same inputs. As long as we can compute the total error and derivative with just a goal and activation as inputs, then we have enough to build a new error function.

Now that we have our error functions separated out into their own classes. Let's update our neuron class to use these error classes instead of calculating its own error.

class Neuron{
    property input
    property activation
    property weights = [0.1, 0.2]
    property learningRate = 0.01

    function feedForward(){
        activation = weights[0] * input + weights[1]
        return activation
    }

    function backpropagate(e_deriv){
        delta_w0 = errorDerivativeForWeight0(e_deriv)
        delta_w1 = errorDerivativeForWeight1(e_deriv)
        weights[0] -= learningRate * delta_w0
        weights[1] -= learningRate * delta_w1
    }

    function errorDerivativeForWeight0(e_deriv){
        a_deriv = input
        return e_deriv * a_deriv
    }

    function errorDerivativeForWeight1(e_deriv){
        a_deriv = 1
        return e_deriv * e_simple_deriv * a_deriv
    }
}

These new error classes and updated Neuron class let us easily swap out the error method used during training.

n = new Neuron()
n = new Neuron()
errCalc = MSEError.singleton
// or errCalc = ABSError.singleton

for(1 ... 3600){
    celsius = random() % 20 - 10
    fahrenheit = 1.8 * celsius + 32

    n.input = celsius
    n.feedForward()

    e_deriv = errCalc.derivative(fahrenheit, n.activation)
    n.backpropagate(e_deriv)

    log("Asked for %℃ => %℉. Predicted %℉. Error %", 
        celsius, fahrenheit, n. activation, error)
}

That's all we need to do to change how we calculate a neuron's error. Next, we'll start looking at different kinds of neurons. So far we've only seen linear neurons \(w_1i + w_2\), but in Chapter 3 we'll look into non-linear neurons.

I Want Your Feedback!

Thanks for reading! I'd love to know what you thought of Chapter 1 and 2, and what you'd like to see in upcoming chapters. Please send me your feedback at adam@milestonemade.com.

And if you haven't already, sign up at https://www.milestonemade.com/building-with-neurons/ to be notified when new chapters are available!