ICSI Broadcast News Results Page

This page is a central location to gather and organize the results of the various forays we are making into the Hub 4 - Broadcast News task here at ICSI. As our experiments continue, it will hopefully form a kind of narrative of our approach to the problem.
 

1998aug03: Progress of the current trainings

Contents

• Single Net Test Results
• The 13 Epochs of PLP and MSG
• PLP Context Window Experiments
• Combo Net Test Results
• Per Spoke (acoustic condition) Results
• Notes
• Bias Histograms of PLP
• Multiband MSG Results

Single Net Test Results

• Approximate word-error-rates are shown for evaluations where the decoding did not complete, in all cases because the decode ran out of memory (presumably as a result of weak acoustic models generating too large a hypothesis space). Numbers reported are based on at least 50% of the test set, but are bound to be somewhat biassed since the utterance that caused failure must be one of the hardest. In the h4e97sb set, the majority of incomplete runs ran out of RAM on utterance 93 (of 173), Clinton_F2.
• The feature abbreviations are as follow: "ras" is Log-RASTA-PLP features; "plp" is plain PLP features; "nplp" is PLP features normalized on a per-utterance basis; "plpN" is supposed to be the same as "nplp", but reimplemented at ICSI (with pfile_normutts). The next number (8 or 12) is the order of the LPC smoothing used, and also the number of cepstral parameters used (in addition to log energy) in the feature vector. "-8k" indicates features derived from the 8kHz (downsampled) original data; "-16k" is based on the full-bandwidth material. "+d" indicates a feature vector that also included derivatives of each dimension.
• The notation +dN indicates that the delta features were computed over a window of N frames. If the N is omitted, the default is 9.
• Training times are shown for several different types of training hardware: "SP" is a single SPERT board, running in an Ultra-1; "ultra" is running on the CPU of an Ultra-30 TEQUILA). "Tetra" refers to the four-SPERT board setup in GUINNESS, an Ultra-1. "Duo" is the two-SPERT Sparc-2, ICSIB46.
• The testing time values indicate how many times real time the decodes took. The test sets are all about 30 minutes long, so this is simply related to the total time taken for decode. Only TEQUILA (Ultra-30, 300 MHz UltraSPARC-II) and STOUT (Ultra-1, 170 MHz UltraSPARC) have enough memory to do decodes; they are, however, machines of different speed, thus I have attempted to indicate which machine ("teq" or "sto") gave each timing result.
• The 20shows training set was an arbitrarily-chosen collection of 20 complete shows consisting of 2.7M training frames or about 12 hours of data.
• The 1/6 of 74h was the first sixth of a randomly-ordered list of the complete set of training utterances (the 74 hours of hub4_96 training data) prepared by Gary. It also has 2.7M frames.
• The 1/2 of 74h was the first half of the randomized utterance list, with 8.1M frames.
• The k960614 test set was a single, complete show of NPR's Marketplace. There was no Marketplace in the 20shows training set.
• The 100xF0s training set was a random selection of F0-condition (studio prepared speech) utterances only, and as such should be much easier than the mix of conditions in the other training sets.
• h4e97sb is a 30 minute subset of the 1997 evaluation data, used as a standard for development at Cambridge.
• 27hyp means the following Gethin-derived noway pruning parameters:

-n_hyps 27 -beam 5 -state_beam 6 -prob_min 0.00020
• 7hyp means the following BeRP-demo-derived noway pruning parameters:

-n_hyps 7 -beam 2 -state_beam 4 -prob_min 0.00020
• 31hyp means the following Gary-derived noway pruning parameters:

-n_hyps 31 -beam 3.5 -state_beam 4.5 -prob_min 0.00020
• 31*hyp implies the following noway pruining parameters:

-n_hyps 31 -beam 5 -state_beam 6 -prob_min 0.000075
• The " FPU Train " test reproduced the "first try" net but running on the main processor of the Ultra-30 TEQUILA, to confirm that the SPERT's fixed-point arithmetic wasn't affecting these results
• The " Bunch=16 " test reproduced the "first try" net but using the special version of qnstrn that permits bunch-mode training. It is almost twice as fast for training on a single SPERT, but I'm still waiting to confirm an absence of WER% influence.
• The " Rasta ftrs " training will reproduce Gary's base, but using RASTA features instead of the plain CU version of PLP, allowing a direct comparison between these feature choices.
• The " 8k compare " training will reproduce Gary's base, but based on the wav files downsampled to 8kHz, so the impact of this change can be measured.
• The " ras12+d-8k " training is intended to be a best-shot at a `different' network, to permit a preliminary test-of-combined-nets with Gary's best PLP net, and also hopefully the RNN classifier.
• Combinations are systems that combine the probability outputs from two acoustic models (by averaging in the log probability domain, at least for now) and decode the resulting probability stream.
• The " RNN1 " system is an LNA from Cambridge derived from four RNN context-independent systems, trained in both directions.
• For the combination experiments, a new aggressive decode' configuration was used, noted as " b2.5 ", chosen from Gary's grid search:

-n_hyps 31 -beam 2.5 -state_beam 3.5 -prob_min 0.00020
It also uses a smaller trigram grammar, bn96-65k-1-2.bin (94.2M) rather than the full bn97-65k-1-1.bin (154M). This helps decode in finite memory.
Later, I redid all the combos with "7hyp" pruning since Gary had done a bunch of baselines, and it seemed about the same.
• Note: by using "b2.5" pruning instead of "7hyp", the overall word error rates were exactly traded between RNN+ras12+d-half and RNN+ras12-8k-1/3!
• Compare the 37h-trained sets plp12-8k-half (ICSI unnormalized plp12 features based on 8kHz sampled wavs) and nplp12-16k-half (CUCon normalized plp12 features based on 16kHz wavs). Surprisingly, the 8kHz sampling appears to hurt us negligably overall. Looking at the per-condition results, the 8kHz downsampling hurt the F0 spoke by about 1% absolute, but improved the F2 spoke (telephone etc.) by some 8% from 54% to 46%. Comparing plp12-8k-half to ras12-8k-half indicates that using Rasta is hurting us by around 10%; however, as Eric pointed out, this might be expected since the targets we are training against are defined based on plp features, and a match between the alignments and the training features is likely to show a benefit. Doing a Rasta-based embedded training would be the fairest way to test.
• The WER breakdown by spoke table has the following attributes constant across all systems: 2000 hidden units, trained on half the data (37h), tested on h4e97sub, "7hyp" pruning.
• In the WER column, the notation (N) indicates that the training was run for N epochs. If (N) is omitted, the default is 8.
• The features for plp+msg include the 28 features from msg0+2N and the 13 features from plp12N to form an input feature vector of 41 features. All are samples at 8k and have a context window of 9.

MSG Multiband Results

The subbands are labelled "a", "b", "c", and "d", and consist of the following features:
 

Band	Top Half Features	Bottom Half Features
A	0,1,2,3,4	14,15,16,17,18
B	5,6,7,8	19,20,21,22
C	9,10,11	23,24,25
D	11,12,13	25,26,27

Notation: a-b means subband a and subband b combined at the feature level by adding the log probabilities (using mergeLna). a-b-c-d-abcdx4 means each of the subbands combined with 4x the fullband. a+b means subband a and subband b combined using a 500 HU mlp. So a+b+c+d-abcdx4 means the 4 subbands are combined with an mlp, and the results are merged with 4x the fullband using mergeLna. "No norm" indicates that the probabilities fed into the merging net were not normalized (e.g. the .norm file contained 0.0 for the means and 1.0 for the variances).
 
 

Bands	WER
abcd	38.7
abcd-rnn*	30.4
a-b-c-d	64.9
a-b-c-d-abcdx4	40.5
a-b-c-d-abcdx4-rnnx8	31.8
a+b+c+d	48.6
a+b+c+d-abcdx4	38.5
a+b+c+d-abcdx4-rnnx8	30.7
abc-abcd	38.7
abc-abd-acd-bcd	37.9
abc-abd-acd-bcd-abcdx4	37.7
abc-abd-acd-bcd-abcdx4-rnnx8	30.2
cep a-b-c-d	Out of memory on teq
klt a-b-c-d	62.7
cep a-b-c-d-abcdx4	40.2
klt a-b-c-d-abcdx4	40.2
cep a-b-c-d-abcdx4-rnnx8	31.6
klt a-b-c-d-abcdx4-rnnx8	31.4
klt a+b+c+d	47.6
klt a+b+c+d no norm	54.7
klt a+b+c+d-abcd	38.7
klt a+b+c+d-abcdx4	38.2
klt a+b+c+d-abcdx4 no norm	37.7
klt a+b+c+d-abcdx4-rnnx8	30.3
klt a+b+c+d-abcdx4-rnnx8 no norm	30.5

* - These are from Dan's decodes in /u/drspeech/data/bn/experiments/dpwe/train17-msg1+align2.

From these results, single multiband hurts overall recognition, even when combined using an MLP. The "drop one" multiband doesn't help enough to be worth the extra complexity (although I haven't tried combining the "drop one" multiband using an MLP).

See the bnspokes database for a breakdown of the focus conditions.