nbest-optimize

NAME

SYNOPSIS

nbest-optimize [ -help ] option ... [ scoredir ... ]

DESCRIPTION

Alternatively, nbest-optimize can also optimize weights for a standard, 1-best hypothesis rescoring that selects entire (sentence) hypotheses (-1best option). In this mode sentence-level error counts may be read from external files, or computed on the fly from the reference strings. The weights obtained are meant to be used for N-best list rescoring with rescore-reweight (see nbest-scripts(1)).

A third optimization criterion is the BLEU score used in machine translation. This also requires the associated scores to be read from external files.

One of three optimization algorithms are available:

1.

The default optimization method is gradient descent on a smoothed (sigmoidal) approximation of the true 0/1 word error function (Katagiri et al. 1990). Therefore, the result can only be expected to be a local minimum of the error surface. (A more global search can be attempted by specifying different starting points.) Another approximation is that the error function is computed assuming a fixed multiple alignment of all N-best hypotheses and the reference string, which tends to slightly overestimate the true pairwise error between any single hypothesis and the reference.

2.

An alternative search strategy uses a simplex-based "Amoeba" search on the (non-smoothed) error function (Press et al. 1988). The search is restarted multiple times to avoid local minima.

3.

A third algorithm uses Powell search (Press et al. 1988) on the (non-smoothed) error function.

OPTIONS

Each filename argument can be an ASCII file, or a compressed file (name ending in .Z or .gz), or ``-'' to indicate stdin/stdout.

-help

Print option summary.

-version

Print version information.

-debug level

Controls the amount of output (the higher the level, the more). At level 1, error statistics at each iteration are printed. At level 2, word alignments are printed. At level 3, full score matrix is printed. At level 4, detailed information about word hypothesis ranking is printed for each training iteration and sample.

-nbest-files file-list

Specifies the set of N-best files as a list of filenames. Three sets of standard scores are extracted from the N-best files: the acoustic model score, the language model score, and the number of words (for insertion penalty computation). See nbest-format(5) for details.
In BLEU optimization mode, since there is no acoustic score, the position of the first score is taken by the "ac-replacement" score, which can be any score used by the machine translation system. A typical example is a score measuring word order distortion between the source and target languages.

-srinterp-format

Parse n-best list in SRInterp format, which has features and text in the same line. n-best optimize will also generate rover-control file in SRInterp format, where each line is in the form of:

	F1=V1 F2=V2 ... Fm=Vm W1 W2 ...Wn

where F1 through Fm are feature names, V1 through Vm are feature values, W1 through Wn are words. Also generate SRInterp control file, in the format of:

	F1:S1 F2:S2 ... Fm:Sm

where S1 through Sm are scaling factors (weights) for feature F1 through Fm .

-refs references

Specifies the reference transcripts. Each line in references must contain the sentence ID (the last component in the N-best filename path, minus any suffixes) followed by zero or more reference words.

-insertion-weight W

Weight insertion errors by a factor W. This may be useful to optimize for keyword spotting tasks where insertions have a cost different from deletion and substitution errors.

-word-weights file

Read a table of words and weights from file. Each word error is weighted according to the word-specific weight. The default weight is 1, and used if a word has no specified weight. Also, when this option is used, substitution errors are counted as the sum of a deletion and an insertion error, as opposed to counting as 1 error as in traditional word error computation.

-anti-refs file

Read a file of "anti-references" for use with the -anti-ref-weight option (see below).

-anti-ref-weight W

Compute the hypothesis errors for 1-best optimization by adding the edit distance with respect to the "anti-references" times the weight W to the regular error count. If W is negative this will tend to generate hypotheses that are different from the anti-references (hence the name).

-1best

Select optimization for standard sentence-level hypothesis selection.

-1best-first

Optimized first using -1best mode, then switch to full optimization. This is an effective way to quickly bring the score weights near an optimal point, and then fine-tune them jointly with the posterior scale parameter.

-errors dir

In 1-best mode, optimize for error counts that are stored in separate files in directory dir. Each N-best list must have a matching error counts file of the same basename in dir. Each file contains 7 columns of numbers in the format

	wcr wer nsub ndel nins nerr nw

Only the last two columns (number of errors and words, respectively) are used.
If this option is omitted, errors will be computed from the N-best hypotheses and the reference transcripts.

-bleu-counts dir

Perform BLEU optimization, reading BLEU reference counts from directory dir. Each N-best list must have a matching counts file of the same basename in dir, containing the following information:

	N M L1 ... LM

where N is the number of hypotheses in the N-best list, M is the number of references for the utterance, and L1 through LM are the reference lengths (word counts) for each reference. Following this line, there are N lines of the form

	K C1 C2 ... Cm

where K is the number of words in the hypothesis and C1 through Cm are the N-gram counts occurring in the references for each N-gram order 1, ..., m. Currently, m is limited to 4.

-minimum-bleu-reference

Use shortest reference length to compute the BLEU brevity penalty.

-closest-bleu-reference

Use closest reference length for each translation hypothesis to compute the BLEU brevity penalty.

-average-bleu-reference

Use average reference length to compute the BLEU brevity penalty.

-error-bleu-ratio R

Specifies the weight of error rate when combined with BLEU as optimization objective: (1-BLEU) + ERR x R. ERR is error rate computed by #errors/#references.

-max-nbest n

Limits the number of hypotheses read from each N-best list to the first n.

-rescore-lmw lmw

Sets the language model weight used in combining the language model log probabilities with acoustic log probabilities. This is used to compute initial aggregate hypotheses scores.

-rescore-wtw wtw

Sets the word transition weight used to weight the number of words relative to the acoustic log probabilities. This is used to compute initial aggregate hypotheses scores.

-posterior-scale scale

Initial value for scaling log posteriors. The total weighted log score is divided by scale when computing normalized posterior probabilities. This controls the peakedness of the posterior distribution. The default value is whatever was chosen for -rescore-lmw, so that language model scores are scaled to have weight 1, and acoustic scores have weight 1/lmw.

-combine-linear

Compute aggregate scores by linear combination, rather than log-linear combination. (This is appropriate if the input scores represent log-posterior probabilities.)

-non-negative

Constrain search to non-negative weight values.

-vocab file

Read the N-best list vocabulary from file. This option is mostly redundant since words found in the N-best input are implicitly added to the vocabulary.

-tolower

Map vocabulary to lowercase, eliminating case distinctions.

-multiwords

Split multiwords (words joined by '_') into their components when reading N-best lists.

-multi-char C

Character used to delimit component words in multiwords (an underscore character by default).

-no-reorder

Do not reorder the hypotheses for alignment, and start the alignment with the reference words. The default is to first align hypotheses by order of decreasing scores (according to the initial score weighting) and then the reference, which is more compatible with how nbest-lattice(1) operates.

-noise noise-tag

Designate noise-tag as a vocabulary item that is to be ignored in aligning hypotheses with each other (the same as the -pau- word). This is typically used to identify a noise marker.

-noise-vocab file

Read several noise tags from file, instead of, or in addition to, the single noise tag specified by -noise.

-hidden-vocab file

Read a subvocabulary from file and constrain word alignments to only group those words that are either all in or outside the subvocabulary. This may be used to keep ``hidden event'' tags from aligning with regular words.

-dictionary file

Use word pronunciations listed in file to construct word alignments when building word meshes. This will use an alignment cost function that reflects the number of inserted/deleted/substituted phones, rather than words. The dictionary file should contain one pronunciation per line, each naming a word in the first field, followed by a string of phone symbols.

-distances file

Use the word distance matrix in file as a cost function for word alignments. Each line in file defines a row of the distance matrix. The first field contains the word that is the row index, followed by one or more word/number pairs, where the word represents the column index and the number the distance value.

-init-lambdas 'w1 w2 ...'

Initialize the score weights to the values specified (zeros are filled in for missing values). The default is to set the initial acoustic model weight to 1, the language model weight from -rescore-lmw, the word transition weight from -rescore-wtw, and all remaining weights to zero initially. Prefixing a value with an equal sign (`=') holds the value constant during optimization. (All values should be enclosed in quotes to form a single command-line argument.)
Hypotheses are aligned using the initial weights; thus, it makes sense to reoptimize with initial weights from a previous optimization in order to obtain alignments closer to the optimimum.

-alpha a

Controls the error function smoothness; the sigmoid slope parameter is set to a.

-epsilon e

The step-size used in gradient descent (the multiple of the gradient vector).

-min-loss x

Sets the loss function for a sample effectively to zero when its value falls below x.

-max-delta d

Ignores the contribution of a sample to the gradient if the derivative exceeds d. This helps avoid numerical problems.

-maxiters m

Stops optimization after m iterations. In Amoeba search, this limits the total number of points in the parameter space that are evaluated.

-max-bad-iters n

Stops optimization after n iterations during which the actual (non-smoothed) error has not decreased.

-max-amoeba-restarts r

Perform only up to r repeated Amoeba searches. The default is to search until D searches give the same results, where D is the dimensionality of the problem.

-max-time T

Abort search if new lower-error point isn't found in T seconds.

-epsilon-stepdown s

-min-epsilon m

If s is a value greater than zero, the learning rate will be multiplied by s every time the error does not decrease after a number of iterations specified by -max-bad-iters. Training stops when the learning rate falls below m in this manner.

-converge x

Stops optimization when the (smoothed) loss function changes relatively by less than x from one iteration to the next.

-quickprop

Use the approximate second-order method known as "QuickProp" (Fahlman 1989).

-init-amoeba-simplex 's1 s2 ...'

Perform Amoeba simplex search. The argument defines the step size for the initial Amoeba simplex. One value for each non-fixed search dimension should be specified, plus optionally a value for the posterior scaling parameter (which is searched as an added dimension).

-init-powell-range 'a1,b1 a2,b2 ...'

Perform Powell search. The argment initializes the weight ranges for Powell search. One comma-separated pair of values for each search dimension should be specified. For each dimension, if the upper bound equals lower bound and initial lambda, that dimension will be fixed, even if not so specified by -init-lambda .

-num-powell-runs N

Sets the number of random runs for quick Powell grid search (default value is 20).

-dynamic-random-series

Use time and process ID to initialize seed for pseudo random series used in Powell search. This will make results unrepeatable but may yield better results through multiple trials.

-print-hyps file

Write the best word hypotheses to file after optimization.

-print-top-n N

Write out the top N rescored hypotheses. In this case -print-hyps specifies a directory (not a file) and one file per N-best list is generated.

-print-unique-hyps

Eliminate duplicate hypotheses when writing out N-best hypotheses.

-print-old-ranks

Output the original hypothesis ranks when writing out N-best hypotheses.

-compute-oracle

Find the lowest error rate or the highest BLEU score achievable by choosing among all N-best hypotheses.

-print-oracle-hyps file

Print output oracle hyps to file.

-write-rover-control file

Writes a control file for nbest-rover to file, reflecting the names of the input directories and the optimized parameter values. The format of file is described in nbest-scripts(1). The file is rewritten for each new minimal error weight combination found.
In BLEU optimization, the weight for the ac-replacement score will be written in the place of the posterior scale, since posterior scaling is not used in BLEU optimization.

-skipopt

Skip optimization altogether, such as when only the -print-hyps function is to be exercised.

--

Signals the end of options, such that following command-line arguments are interpreted as additional scorefiles even if they start with `-'.

scoredir ...

Any additional arguments name directories containing further score files. In each directory, there must exist one file named after the sentence ID it corresponds to (the file may also end in ``.gz'' and contain compressed data). The total number of score dimensions is thus 3 (for the standard scores from the N-best list) plus the number of additional score directories specified.

SEE ALSO

BUGS

The -insertion-weight and -word-weights options only affect the word error computation, not the construction of hypothesis alignments. Also, they only apply to sausage-based, not 1-best error optimization. (1-best errors may be explicitly specified using the -errors option).

The -anti-refs and -anti-ref-weight options do not work for sausage-based or BLEU optimization.

AUTHORS