Neural Network Training

🎯 Lab Objectives

• Implement a small neural network in MATLAB.
• Calculate forward propagation step by step.
• Calculate the loss function.
• Implement backpropagation using analytical gradients.
• Update weights using gradient descent.
• Stop training when the error becomes smaller than ε.

1. Network Structure

We use a simple 1–2–1 neural network:

2. Activation Function

We use the Sigmoid activation function:

Its derivative is:

3. MATLAB Sigmoid Function

sigma = @(z) 1 ./ (1 + exp(-z));

This anonymous function calculates the Sigmoid value for any input z.

4. Initial Parameters

Start with initial weights and biases:

W.w1 = 0.4;
W.w2 = -0.3;

W.v1 = 0.2;
W.v2 = 0.5;

W.b1 = 0.1;
W.b2 = -0.1;
W.b3 = 0.0;

5. Training Sample

x = 1;
t = 0.8;

Here:

6. Forward Propagation

Hidden neuron 1:

Hidden neuron 2:

Output neuron:

7. MATLAB Forward Propagation

z1 = W.w1*x + W.b1;
h1 = sigma(z1);

z2 = W.w2*x + W.b2;
h2 = sigma(z2);

z3 = W.v1*h1 + W.v2*h2 + W.b3;
y  = sigma(z3);

8. Loss Function

We use:

MATLAB:

E = 0.5*(y - t)^2;

9. Backpropagation: Output Error

From Lecture 05:

MATLAB:

dO = (y - t) * y * (1 - y);

10. Output-Layer Gradients

dV1 = dO*h1;
dV2 = dO*h2;
dB3 = dO;

11. Hidden-Layer Error

d1 = dO*W.v1*h1*(1-h1);
d2 = dO*W.v2*h2*(1-h2);

12. Input-to-Hidden Gradients

dW1 = d1*x;
dB1 = d1;

dW2 = d2*x;
dB2 = d2;

13. Gradient Descent Update

W.w1 = W.w1 - eta*dW1;
W.w2 = W.w2 - eta*dW2;

W.v1 = W.v1 - eta*dV1;
W.v2 = W.v2 - eta*dV2;

W.b1 = W.b1 - eta*dB1;
W.b2 = W.b2 - eta*dB2;
W.b3 = W.b3 - eta*dB3;

14. Learning Rate

Choose:

eta = 0.5;

15. Stopping Criterion

Define a required accuracy:

epsilon = 1e-6;

Stop when:

and also use a maximum epoch limit:

maxEpochs = 10000;

16. Complete MATLAB Program

clear; clc; close all;

sigma = @(z) 1 ./ (1 + exp(-z));

x = 1;
t = 0.8;

eta = 0.5;
epsilon = 1e-6;
maxEpochs = 10000;

W.w1 = 0.4;
W.w2 = -0.3;

W.v1 = 0.2;
W.v2 = 0.5;

W.b1 = 0.1;
W.b2 = -0.1;
W.b3 = 0.0;

lossHist = zeros(1,maxEpochs);

epoch = 0;
converged = false;

while ~converged && epoch < maxEpochs

    epoch = epoch + 1;

    % ----- Forward propagation -----

    z1 = W.w1*x + W.b1;
    h1 = sigma(z1);

    z2 = W.w2*x + W.b2;
    h2 = sigma(z2);

    z3 = W.v1*h1 + W.v2*h2 + W.b3;
    y  = sigma(z3);

    % ----- Loss -----

    E = 0.5*(y - t)^2;

    lossHist(epoch) = E;

    % ----- Backpropagation -----

    dO = (y - t) * y * (1 - y);

    dV1 = dO*h1;
    dV2 = dO*h2;
    dB3 = dO;

    d1 = dO*W.v1*h1*(1-h1);
    d2 = dO*W.v2*h2*(1-h2);

    dW1 = d1*x;
    dB1 = d1;

    dW2 = d2*x;
    dB2 = d2;

    % ----- Update parameters -----

    W.w1 = W.w1 - eta*dW1;
    W.w2 = W.w2 - eta*dW2;

    W.v1 = W.v1 - eta*dV1;
    W.v2 = W.v2 - eta*dV2;

    W.b1 = W.b1 - eta*dB1;
    W.b2 = W.b2 - eta*dB2;
    W.b3 = W.b3 - eta*dB3;

    % ----- Stopping criterion -----

    if E < epsilon
        converged = true;
    end
end

disp('Final weights and biases:')
disp(W)

fprintf('Epochs = %d\n',epoch)
fprintf('Final output y = %.6f\n',y)
fprintf('Target t = %.6f\n',t)
fprintf('Final error E = %.10f\n',E)

figure
semilogy(1:epoch,lossHist(1:epoch),'LineWidth',1.6)
grid on

xlabel('Epoch')
ylabel('Loss E')
title('Neural Network Training Convergence')

17. Understanding the Training Loop

18. Convergence Plot

The command:

semilogy(1:epoch,lossHist(1:epoch),'LineWidth',1.6)

plots the loss on a logarithmic vertical axis.

19. Experiment 1 — Change Learning Rate

Try:

eta = 0.05;
eta = 0.5;
eta = 2.0;

Compare:

• Number of epochs
• Stability
• Final loss

20. Experiment 2 — Change Initial Weights

Modify the initial values of:

W.w1
W.w2
W.v1
W.v2

21. Power-System Interpretation

The same training mechanism can be used for a power-system prediction problem.

For example:

With more inputs:

22. Lab Tasks

1. Run the original program.
2. Record the final output and number of epochs.
3. Change the learning rate and compare convergence.
4. Change the initial weights.
5. Change the target value t.
6. Explain every gradient expression mathematically.
7. Explain why the loss decreases during training.

📝 Questions

1. What does dO represent?
2. Why does dO contain y(1-y)?
3. Why is dV1 equal to dO*h1?
4. Why does the hidden error contain W.v1?
5. Why do we subtract the gradient?
6. What happens if η is too large?
7. What is the role of ε?