Issues in ANN Learning and Related Topics

• High-dimensional ANNs are less likely to settle into local minima than low dimensional ones - Why? Discuss this. By the same token can we not also say that high-dimensional ANNs are less likely to settle into global minima? Good question -- There is a good answer as well.

• A momentum term is sometimes added to the weight update rule as follows:
  
  $$\Delta w_{ji}(n) = \eta \delta_j x_{ji} + \alpha (n-1) w_{ji}$$
  
  
  where $0 \le \alpha < 1$ is a constant called the momentum and n is the iteration number. Why do this? Mainly to keep the ball rolling through local minima and to decelerate gradually on plane regions of the error surface.

• Research the technique of ``Simulated Annealing'' Kirkpatrick, Gelatt Jr. and Vecchi, 1983, in Science 220(4598), pp.671-680 and discuss how one can apply it to reduce the chances of settling in local minima.

• How can you characterize the hypothesis space and inductive bias of ANNs?

• Read Section 4.6.5 of Mitchell on problems due to overfitting in ANNs, esp re. the various heuristics people have been using to address this issue.

• Discuss how alternative functions for thresholding and error can be used.

• Read the rest of Mitchell's Chapter 4, esp. on simple recurrent nets and on dynamically adding and deleting nodes from an ANN.