5.1. Master's thesis

My Master's thesis (called “ diploma work” in german) at Kaiserslautern dealt with the mathematical aspects of some neural networks, the exact title being “ Neural learning rules for linear association and separation”. After giving a formal mathematical definition of neural networks, I consider the linear association problem and some so-called “ learning rules” for its solution:

• Hebb's rule

• Delta rule

• Generalized inverses and the Greville algorithm

I show that the Greville algorithm has the same general form as the Delta rule but does not have its limitations regarding orthogonality (Hebb) or linear independence (Delta) of the input vectors and infinite repetition of the learning process (in fact, it converges to a solution after a finite number of steps, so that the input vectors need only presented once to the network).

Continuing from linear association to linear separation, I consider

• the Perceptron rule

• the relaxation rule

• the Ho-Kashyap rule

I give a generalized convergence proof for the Perceptron learning rule. I present corrected proofs of the relaxation and Ho-Kashyap rules and touch on a solution by linear programming. As you can read in a 1997 paper by Prof. G. Labonté, the scientific community did not seem to realize that the relaxation method can be used in the context of neural networks. In my 1992 Master's thesis, I devote a whole chapter (5.2, pp.30-37) to the treatment of the relaxation method for the linear separation problem which is fundamental to neural networks (you can transform a non-linear separation problem to a linear one, using an aproppriate non-linear transformation of the input space).

Finally all methods are investigated theoretically as well as practically regarding their asymptotic time behaviour.

If you are interested in a printed copy of my Master's thesis (in german) and the source code (FORTRAN), you can buy it online: Master's thesis on Neural Networks. The following table contains the key data: