Rohit Dua, Samuel A. Mulder, Steve E. Watkins, and Donald C. Wunsch

A Brief Overview of Neural Networks

• By
• Rohit Dua, Samuel A. Mulder, Steve E. Watkins, and Donald C. Wunsch

Overview

• Relation to Biological Brain: Biological Neural Network
• The Artificial Neuron
• Types of Networks and Learning Techniques
• Supervised Learning & Backpropagation Training Algorithm
• Learning by Example
• Applications
• Questions

Biological Neuron

Artificial Neuron

• Σ

• f(n)

• W

• W

• W

• W

• Outputs

• Activation
• Function

• INPUTS

• W=Weight

• Neuron

Transfer Functions

• 1

• 0

• Input

• Output

Types of networks

• Multiple Inputs and Single Layer

• Multiple Inputs and layers

Types of Networks – Contd.

• Feedback

• Recurrent Networks

Learning Techniques

• Supervised Learning:

• Inputs from the environment

• Neural Network

• Actual System

• Σ

• Error

• +

• -

• Expected Output

• Actual Output

• Training

Multilayer Perceptron

• Inputs

• First Hidden layer

• Second Hidden Layer

• Output Layer

Signal Flow Backpropagation of Errors

• Function Signals

• Error Signals

Learning by Example

• Hidden layer transfer function: Sigmoid function = F(n)= 1/(1+exp(-n)), where n is the net input to the neuron.
• Derivative= F’(n) = (output of the neuron)(1-output of the neuron) : Slope of the transfer function.
• Output layer transfer function: Linear function= F(n)=n; Output=Input to the neuron
• Derivative= F’(n)= 1

Learning by Example

• Training Algorithm: backpropagation of errors using gradient descent training.
• Colors:
• Red: Current weights
• Orange: Updated weights
• Black boxes: Inputs and outputs to a neuron
• Blue: Sensitivities at each layer

First Pass

• 0.5

• 0.5

• 0.5

• 0.5

• 0.5

• 0.5

• 0.5

• 0.5

• 1

• 0.5

• 0.5

• 0.6225

• 0.6225

• 0.6225

• 0.6225

• 0.6508

• 0.6508

• 0.6508

• 0.6508

• Error=1-0.6508=0.3492

• G3=(1)(0.3492)=0.3492

• G2= (0.6508)(1-0.6508)(0.3492)(0.5)=0.0397

• G1= (0.6225)(1-0.6225)(0.0397)(0.5)(2)=0.0093

• Gradient of the neuron= G =slope of the transfer function×[Σ{(weight of the neuron to the next neuron) × (output of the neuron)}]

• Gradient of the output neuron = slope of the transfer function × error

Weight Update 1

• New Weight=Old Weight + {(learning rate)(gradient)(prior output)}

• 0.5+(0.5)(0.3492)(0.6508)

• 0.6136

• 0.5124

• 0.5124

• 0.5124

• 0.6136

• 0.5124

• 0.5047

• 0.5047

• 0.5+(0.5)(0.0397)(0.6225)

• 0.5+(0.5)(0.0093)(1)

Second Pass

• 0.5047

• 0.5124

• 0.6136

• 0.6136

• 0.5047

• 0.5124

• 0.5124

• 0.5124

• 1

• 0.5047

• 0.5047

• 0.6391

• 0.6391

• 0.6236

• 0.6236

• 0.8033

• 0.6545

• 0.6545

• 0.8033

• Error=1-0.8033=0.1967

• G3=(1)(0.1967)=0.1967

• G2= (0.6545)(1-0.6545)(0.1967)(0.6136)=0.0273

• G1= (0.6236)(1-0.6236)(0.5124)(0.0273)(2)=0.0066

Weight Update 2

• New Weight=Old Weight + {(learning rate)(gradient)(prior output)}

• 0.6136+(0.5)(0.1967)(0.6545)

• 0.6779

• 0.5209

• 0.5209

• 0.5209

• 0.6779

• 0.5209

• 0.508

• 0.508

• 0.5124+(0.5)(0.0273)(0.6236)

• 0.5047+(0.5)(0.0066)(1)

Third Pass

• 0.508

• 0.5209

• 0.6779

• 0.6779

• 0.508

• 0.5209

• 0.5209

• 0.5209

• 1

• 0.508

• 0.508

• 0.6504

• 0.6504

• 0.6243

• 0.6243

• 0.8909

• 0.6571

• 0.6571

• 0.8909

Weight Update Summary

• W1: Weights from the input to the input layer
• W2: Weights from the input layer to the hidden layer
• W3: Weights from the hidden layer to the output layer

Training Algorithm

• The process of feedforward and backpropagation continues until the required mean squared error has been reached.
• Typical mse: 1e-5
• Other complicated backpropagation training algorithms also available.

Why Gradient?

• O1

• O2

• O = Output of the neuron
• W = Weight
• N = Net input to the neuron

• W1

• W2

• N = (O1×W1)+(O2×W2)

• O3 = 1/[1+exp(-N)]

• Error = Actual Output – O3

• To reduce error: Change in weights:
• Learning rate
• Rate of change of error w.r.t rate of change of weight
• Gradient: rate of change of error w.r.t rate of change of ‘N’
• Prior output (O1 and O2)

• 0

• Input

• Output

• 1

Gradient in Detail

• Gradient : Rate of change of error w.r.t rate of change in net input to neuron
• For output neurons
• Slope of the transfer function × error
• For hidden neurons : A bit complicated ! : error fed back in terms of gradient of successive neurons
• Slope of the transfer function × [Σ (gradient of next neuron × weight connecting the neuron to the next neuron)]
• Why summation? Share the responsibility!!
• Therefore: Credit Assignment Problem

An Example

• 1

• 0.4

• 0.731

• 0.598

• 0.5

• 0.5

• 0.5

• 0.5

• 0.6645

• 0.6645

• 0.66

• 0.66

• 1

• 0

• Error = 1-0.66 = 0.34

• Error = 0-0.66 = -0.66

• G1=0.66×(1-0.66)×(-0.66)= -0.148

• G1=0.66×(1-0.66)×(0.34)= 0.0763

• Reduce more

• Increase less

Improving performance

• Changing the number of layers and number of neurons in each layer.
• Variation in Transfer functions.
• Changing the learning rate.
• Training for longer times.
• Type of pre-processing and post-processing.

Applications

• Used in complex function approximations, feature extraction & classification, and optimization & control problems
• Applicability in all areas of science and technology.