ICSI Speech FAQ:
6.9 How does neural net randomization affect performance?

Answer by: dpwe - 2000-08-11

Our neural net trainings involve several sources of randomization:

Note that although we call these values random, they are all deterministically produced, so that two trainings run with the same parameters on the same architecture and the same data should give identical results. However, due to the very large number of calculations involved, small arithmetic differences (e.g. SPARC versus Intel, and certainly floating point versus fixed point such as the SPERT) can result in measurable differences in performance.

To investigate the possible influence of these randomization values, Eric and I conducted a number of trainings of essentially the same net, varying only the randomization in each case. The results are summarized in my status reports of 2000-04-28 and 2000-05-12.

Essentially, we found that different initial randomizations can indeed vary resulting net performance, but only by a maximum of 3-4% relative. We also experimented with posterior combination of these nets (i.e. combining two nets trained on exactly the same data), which, as you might expect, doesn't help very much - maybe 2% relative if you're lucky.

Previous: 6.8 How does neural net size affect performance? - Next: 6.10 What is posterior combination? What other kinds of combination are possible?
Back to ICSI Speech FAQ index

Generated by build-faq-index on Tue Mar 24 16:18:16 PDT 2009