CS 294-4: Connectionist and Neural Computation
Lecture 5 - September 9, 1997
The Bishop and Rojas books are good references for a detailed
treatment of the topics covered in this lecture.
Backpropagation as gradient descent:
Choice of input representation, network architecture, data,
termination criteria play an important role.
Domain knowledge is crucial for successful design of neural
network models using backpropagation
- Vanilla backprop uses a fixed learning rate
- Learning rate should be chosen with care
- Limitations of using fixed learning rate
- Use of a "momentum" term can help
- More powerful quasi-newton methods such as BFGS may be used for
moderate size networks
- Conjugate gradient techniques are good alternatives if the
number of weights in the network is very large.
Some viewgraphs from Lecture 4
Some viewgraphs from Lecture 5