The Bishop and Rojas books are good references for a detailed treatment of the topics covered in this lecture.

Backpropagation as gradient descent:

Vanilla backprop uses a fixed learning rate
Learning rate should be chosen with care
Limitations of using fixed learning rate
Use of a "momentum" term can help
More powerful quasi-newton methods such as BFGS may be used for moderate size networks
Conjugate gradient techniques are good alternatives if the number of weights in the network is very large.

Choice of input representation, network architecture, data, termination criteria play an important role.

Domain knowledge is crucial for successful design of neural network models using backpropagation

Lokendra Shastri