nbest-optimize - optimize score combination for N-best word error minimization
nbest-optimize [ -help ] option ... [ scoredir ... ]
reads a set of N-best lists, additional score files, and corresponding
reference transcripts and optimizes the score combination weights
so as to minimize the word error of a classifier that performs
word-level posterior probability maximization.
The optimized weights are meant to be used with
determines both the best relative weighting of knowledge source scores
and the optimal
parameter that controls the peakedness of the posterior distribution.
can also optimize weights for a standard, 1-best hypothesis rescoring that
selects entire (sentence) hypotheses
In this mode sentence-level error counts may be read from external files,
or computed on the fly from the reference strings.
The weights obtained are meant to be used for N-best list rescoring with
A third optimization criterion is the BLEU score used in machine translation.
This also requires the associated scores to be read from external files.
One of three optimization algorithms are available:
The default optimization method is gradient descent on a smoothed (sigmoidal)
approximation of the true 0/1 word error function (Katagiri et al. 1990).
Therefore, the result can only be expected to be a
minimum of the error surface.
(A more global search can be attempted by specifying different starting
Another approximation is that the error function is computed assuming a fixed
multiple alignment of all N-best hypotheses and the reference string,
which tends to slightly overestimate the true pairwise error between any
single hypothesis and the reference.
An alternative search strategy uses a simplex-based "Amoeba" search on
the (non-smoothed) error function (Press et al. 1988).
The search is restarted multiple times to avoid local minima.
A third algorithm uses Powell search (Press et al. 1988)
on the (non-smoothed) error function.
Each filename argument can be an ASCII file, or a
compressed file (name ending in .Z or .gz), or ``-'' to indicate
Print option summary.
Print version information.
- -debug level
Controls the amount of output (the higher the
At level 1, error statistics at each iteration are printed.
At level 2, word alignments are printed.
At level 3, full score matrix is printed.
At level 4, detailed information about word hypothesis ranking is printed
for each training iteration and sample.
- -nbest-files file-list
Specifies the set of N-best files as a list of filenames.
Three sets of standard scores are extracted from the N-best files:
the acoustic model score, the language model score, and the number of
words (for insertion penalty computation).
In BLEU optimization mode, since there is no acoustic score, the
of the first score is taken by the "ac-replacement" score, which can be
any score used by the machine translation system.
A typical example is a score measuring
word order distortion between the source and target languages.
Parse n-best list in SRInterp format, which has
features and text in the same line. n-best optimize will also generate
rover-control file in SRInterp format, where each line is in the form of:
F1=V1 F2=V2 ... Fm=Vm W1 W2 ...Wn
are feature names,
are feature values,
Also generate SRInterp control file, in the format of:
F1:S1 F2:S2 ... Fm:Sm
are scaling factors (weights) for feature
- -refs references
Specifies the reference transcripts.
Each line in
must contain the sentence ID (the last component in the N-best filename
path, minus any suffixes) followed by zero or more reference words.
- -insertion-weight W
Weight insertion errors by a factor
This may be useful to optimize for keyword spotting tasks where
insertions have a cost different from deletion and substitution errors.
- -word-weights file
Read a table of words and weights from
Each word error is weighted according to the word-specific weight.
The default weight is 1, and used if a word has no specified weight.
Also, when this option is used, substitution errors are counted
as the sum of a deletion and an insertion error, as opposed to counting
as 1 error as in traditional word error computation.
- -anti-refs file
Read a file of "anti-references" for use with the
option (see below).
- -anti-ref-weight W
Compute the hypothesis errors for 1-best optimization by adding the
edit distance with respect to the "anti-references" times the weight
to the regular error count.
is negative this will tend to generate hypotheses that are different from
the anti-references (hence the name).
Select optimization for standard sentence-level hypothesis selection.
Optimized first using
mode, then switch to full optimization.
This is an effective way to quickly bring the score weights near an
optimal point, and then fine-tune them jointly with the posterior scale
- -errors dir
In 1-best mode, optimize for error counts that are stored in separate files
Each N-best list must have a matching error counts file of the same
Each file contains 7 columns of numbers in the format
wcr wer nsub ndel nins nerr nw
Only the last two columns (number of errors and words, respectively) are used.
If this option is omitted, errors will be computed from the N-best hypotheses
and the reference transcripts.
- -bleu-counts dir
Perform BLEU optimization, reading BLEU reference counts from directory
Each N-best list must have a matching counts file of the same
containing the following information:
N M L1 ... LM
is the number of hypotheses in the N-best list,
is the number of references for the utterance,
are the reference lengths (word counts) for each reference.
Following this line, there are
lines of the form
K C1 C2 ... Cm
is the number of words in the hypothesis and
are the N-gram counts occurring in the references for each N-gram order
is limited to 4.
Use shortest reference length to compute the BLEU brevity penalty.
Use closest reference length for each translation hypothesis to compute
the BLEU brevity penalty.
Use average reference length to compute the BLEU brevity penalty.
- -error-bleu-ratio R
Specifies the weight of error rate when combined with BLEU as optimization
objective: (1-BLEU) + ERR x R.
is error rate computed by #errors/#references.
- -max-nbest n
Limits the number of hypotheses read from each N-best list to the first
- -rescore-lmw lmw
Sets the language model weight used in combining the language model log
probabilities with acoustic log probabilities.
This is used to compute initial aggregate hypotheses scores.
- -rescore-wtw wtw
Sets the word transition weight used to weight the number of words relative to
the acoustic log probabilities.
This is used to compute initial aggregate hypotheses scores.
- -posterior-scale scale
Initial value for scaling log posteriors.
The total weighted log score is divided by
when computing normalized posterior probabilities.
This controls the peakedness of the posterior distribution.
The default value is whatever was chosen for
so that language model scores are scaled to have weight 1,
and acoustic scores have weight 1/lmw.
Compute aggregate scores by linear combination, rather than log-linear
(This is appropriate if the input scores represent log-posterior probabilities.)
Constrain search to non-negative weight values.
- -vocab file
Read the N-best list vocabulary from
This option is mostly redundant since words found in the N-best input
are implicitly added to the vocabulary.
Map vocabulary to lowercase, eliminating case distinctions.
Split multiwords (words joined by '_') into their components when reading
- -multi-char C
Character used to delimit component words in multiwords
(an underscore character by default).
Do not reorder the hypotheses for alignment, and start the alignment with
the reference words.
The default is to first align hypotheses by order of decreasing scores
(according to the initial score weighting) and then the reference,
which is more compatible with how
- -noise noise-tag
as a vocabulary item that is to be ignored in aligning hypotheses with
each other (the same as the -pau- word).
This is typically used to identify a noise marker.
- -noise-vocab file
Read several noise tags from
instead of, or in addition to, the single noise tag specified by
- -hidden-vocab file
Read a subvocabulary from
and constrain word alignments to only group those words that are either all
in or outside the subvocabulary.
This may be used to keep ``hidden event'' tags from aligning with
- -dictionary file
Use word pronunciations listed in
to construct word alignments when building word meshes.
This will use an alignment cost function that reflects the number of
inserted/deleted/substituted phones, rather than words.
should contain one pronunciation per line, each naming a word in the first
field, followed by a string of phone symbols.
- -distances file
Use the word distance matrix in
as a cost function for word alignments.
Each line in
defines a row of the distance matrix.
The first field contains the word that is the row index,
followed by one or more word/number pairs, where the word represents the
column index and the number the distance value.
- -init-lambdas 'w1 w2 ...'
Initialize the score weights to the values specified
(zeros are filled in for missing values).
The default is to set the initial acoustic model weight to 1,
the language model weight from
the word transition weight from
and all remaining weights to zero initially.
Prefixing a value with an equal sign (`=')
holds the value constant during optimization.
(All values should be enclosed in quotes to form a single command-line
Hypotheses are aligned using the initial weights; thus, it makes sense
to reoptimize with initial weights from a previous optimization in order
to obtain alignments closer to the optimimum.
- -alpha a
Controls the error function smoothness;
the sigmoid slope parameter is set to
- -epsilon e
The step-size used in gradient descent (the multiple of the gradient vector).
- -min-loss x
Sets the loss function for a sample effectively to zero when its value falls
- -max-delta d
Ignores the contribution of a sample to the gradient if the derivative
This helps avoid numerical problems.
- -maxiters m
Stops optimization after
In Amoeba search, this limits the total number of points in the parameter space
that are evaluated.
- -max-bad-iters n
Stops optimization after
iterations during which the actual (non-smoothed) error has not decreased.
- -max-amoeba-restarts r
Perform only up to
repeated Amoeba searches.
The default is to search until
searches give the same results, where
is the dimensionality of the problem.
- -max-time T
Abort search if new lower-error point isn't found in
- -epsilon-stepdown s
- -min-epsilon m
is a value greater than zero, the learning rate will be multiplied by
every time the error does not decrease after a number of iterations
Training stops when the learning rate falls below
in this manner.
- -converge x
Stops optimization when the (smoothed) loss function changes relatively by less
from one iteration to the next.
Use the approximate second-order method known as "QuickProp" (Fahlman 1989).
- -init-amoeba-simplex 's1 s2 ...'
Perform Amoeba simplex search.
The argument defines the step size for the initial Amoeba simplex.
One value for each non-fixed search dimension should be specified,
plus optionally a value for the posterior scaling parameter
(which is searched as an added dimension).
- -init-powell-range 'a1,b1 a2,b2 ...'
Perform Powell search.
The argment initializes the weight ranges for Powell search.
One comma-separated pair of values for each search dimension should
be specified. For each dimension, if the upper bound equals lower bound
and initial lambda, that dimension will be fixed, even if not so specified by
- -num-powell-runs N
Sets the number of random runs for quick Powell grid search
(default value is 20).
Use time and process ID to initialize seed for pseudo random series used
in Powell search.
This will make results unrepeatable but may yield better results through
- -print-hyps file
Write the best word hypotheses to
- -print-top-n N
Write out the top
In this case
specifies a directory (not a file)
and one file per N-best list is generated.
Eliminate duplicate hypotheses when writing out N-best hypotheses.
Output the original hypothesis ranks when writing out N-best hypotheses.
Find the lowest error rate or the highest BLEU score achievable by choosing
among all N-best hypotheses.
- -print-oracle-hyps file
Print output oracle hyps to
- -write-rover-control file
Writes a control file for
reflecting the names of the input directories and the optimized parameter
The format of
is described in
The file is rewritten for each new minimal error weight combination found.
In BLEU optimization, the weight for the ac-replacement score will be written
in the place of the posterior scale,
since posterior scaling is not used in BLEU optimization.
Skip optimization altogether, such as when only the
function is to be exercised.
Signals the end of options, such that following command-line arguments are
interpreted as additional scorefiles even if they start with `-'.
- scoredir ...
Any additional arguments name directories containing further score files.
In each directory, there must exist one file named after the sentence
ID it corresponds to (the file may also end in ``.gz'' and contain compressed
The total number of score dimensions is thus 3 (for the standard scores from
the N-best list) plus the number of additional score directories specified.
nbest-lattice(1), nbest-scripts(1), nbest-format(5).
S. Katagiri, C.H. Lee, & B.-H. Juang, "A Generalized Probabilistic Descent
Proceedings of the Acoustical Society of Japan, Fall Meeting,
pp. 141-142, 1990.
S. E. Fahlman, "Faster-Learning Variations on Back-Propagation: An
Empirical Study", in D. Touretzky, G. Hinton, & T. Sejnowski (eds.),
Proceedings of the 1988 Connectionist Models Summer School, pp. 38-51,
Morgan Kaufmann, 1989.
W. H. Press, B. P. Flannery, S. A. Teukolsky, & W. T. Vetterling,
Numerical Recipes in C: The Art of Scientific Computing,
Cambridge University Press, 1988.
Gradient-based optimization is not supported (yet) in 1-best or BLEU mode
or in conjunction with the
use simplex or Powell search instead.
The N-best directory in the control file output by
is inferred from the
first N-best filename specified with
and will therefore only work if all N-best lists are placed in the same
options only affect the word error computation, not the construction
of hypothesis alignments.
Also, they only apply to sausage-based, not 1-best error optimization.
(1-best errors may be explicitly specified using the
options do not work for sausage-based or BLEU optimization.
Andreas Stolcke <email@example.com>
Dimitra Vergyri <firstname.lastname@example.org>
Jing Zheng <email@example.com>
Copyright (c) 2000-2012 SRI International, 2012 Microsoft Corp.