nbest-lattice
nbest-lattice
NAME
nbest-lattice - rescore N-best lists and lattices
SYNOPSIS
nbest-lattice [ -help ] option ...
DESCRIPTION
nbest-lattice
rescores N-best lists or optimizes word-level recognition scores
(as opposed to sentence-level scores).
There are two rescoring modes.
In
N-best word error minimization
mode, the program computes the posterior expected word error for each
hypothesis relative to all hypotheses in the N-best list, choosing the one
with the lowest value.
In
lattice word error minimization
mode, the program constructs a word lattice from all the N-best hypotheses
and extracts the path with the lowest expected word error.
This is similar to N-best word error minimization but allows
hypotheses not contained in the N-best list.
A variant of this mode uses a word ``mesh'' instead of a word lattice,
in which all hypotheses are aligned into a grid of word positions,
and one is allowed to chose a word from each grid position, thus allowing an
even greater number of potential hypotheses.
OPTIONS
Each filename argument can be an ASCII file, or a
compressed file (name ending in .Z or .gz), or ``-'' to indicate
stdin/stdout.
- -help
-
Print option summary.
- -version
-
Print version information.
- -debug level
-
Controls the amount of output (the higher the
level,
the more).
At level 1, the expected word error counts for the chosen hypotheses
are printed.
At level 2, the word posterior probabilities are printed in addition
(only for lattice mode, similar to
-dump-posteriors).
- -wer
-
Chooses N-best word error minimization mode.
- -lattice-wer
-
Chooses lattice word error minimization mode (the default).
- -use-mesh
-
Choose the variant of lattice mode that uses word meshes
instead of simple lattices.
- -deletion-bias D
-
Causes the probabilities of deletions to be biased by a factor
D
in doing mesh-based word error minimization.
This controls the trade-off between insertion and deletion errors.
The default is 1 (no bias).
- -rescore file
-
Reads the N-best list from
file.
The N-best list can be in any of the formats described in
nbest-format(5).
- -nbest file
-
A synonym for
-rescore.
- -write-nbest file
-
Outputs the N-best list to a file, after sorting and processing
(for validation or format conversion purposes).
- -nbest-files file-list
-
Rescores multiple N-best lists whose filenames are read from
file-list.
- -write-nbest-dir directory
-
Outputs N-best lists to
directory,
to files named after the input N-best lists,
for when multiple N-best lists are processed (see
-nbest-files).
- -write-vocab file
-
Outputs vocabulary used in N-best list.
- -decipher-nbest
-
Output N-best list in Decipher
nbest-format(5),
rather than the default native SRILM format.
(All N-best formats are accepted for input regardless of this option.)
- -no-rescore
-
Suppress rescoring of lattices;
useful if only the operations of lattice/N-best list reading/writing
are desired.
- -max-nbest n
-
Limits the number of hypotheses read from each N-best list to the first
n.
- -max-rescore m
-
In N-best mode, only choose among the top
m
hypotheses when optimizing word error.
This is convenient to limit computation for long N-best lists.
The cutoff is made after reading all hypotheses (subject to
-max-nbest)
and reordering them according to the posterior probabilities.
The worst-case time taken in N-best error minimization is proportional to
m
times
n,
where
n
is the length of the N-best list (or the value given to
-max-nbest).
However, in practice the average time per sentence is independent of
m,
so this option is usually not necessary.
In lattice mode, only align the top
m
scoring hypotheses (after reweighting and sorting) into the lattice.
- -posterior-prune threshold
-
Don't process N-best hypotheses whose cumulative posterior probability
is below
threshold.
This is another strategy to speed up the algorithm.
- -no-reorder
-
Process N-best hypotheses in the order in which they appear.
By default, hypotheses are first sorted by their aggregate scores.
- -nbest-backtrace
-
Preserve backtrace information (word-level timemarks and scores) when reading
N-best lists containing such information (see
nbest-format(5)).
The default is to ignore backtrace information and record only sentence-level
scores and the word identities.
- -output-ctm
-
Output word hypotheses in NIST CTM (conversation time mark) format.
Note that word start times will be relative to the segment start times,
the first column will contain the N-best filename, and the channel field
is always 1.
The word confidence field contains posterior probabilities.
This option also implies
-nbest-backtrace.
- -rescore-lmw lmw
-
Sets the language model weight used in combining the language model log
probabilities with acoustic log probabilities
(only relevant if separate scores are given in the N-best input).
- -rescore-wtw wtw
-
Sets the word transition weight used to weight the number of words relative to
the acoustic log probabilities
(only relevant if separate scores are given in the N-best input).
If
-no-reorder
is not specified, and either
lmw
or
wtw
are specified to be non-zero, the aggregate scores are recomputed using
those weights; otherwise aggregate scores supplied in the input N-best lists
are used to sort hypotheses.
- -posterior-scale scale
-
Divide the total weighted log score by
scale
when computing normalized posterior probabilities.
This controls the peakedness of the posterior distribution.
The default value is whatever was chosen for
-rescore-lmw,
so that language model scores are scaled to have weight 1,
and acoustic scores have weight 1/lmw.
- -posterior-amw amw
-
Sets the acoustic model weight for computing posteriors;
the default is 1.
This and the next two options allow posteriors to be computed using a
different weighting than that used in ranking and reordering the
hypotheses.
- -posterior-lmw lmw
-
Sets the language model weight for computing posteriors.
The default is to use whatever was specified for
-rescore-lmw.
- -posterior-wtw wtw
-
Sets the word transition weight for computing posteriors.
The default is to use whatever was specified for
-rescore-wtw.
If all three of
amw,
lmw,
and
wtw
are set to zero the posteriors are computed directly from the
aggregate scores stored in the N-best input.
- -vocab file
-
Read the N-best list vocabulary from
file.
This option is mostly redundant since words found in the N-best input
are implicitly added to the vocabulary.
- -vocab-aliases file
-
Reads vocabulary alias definitions from
file,
consisting of lines of the form
alias word
This causes all tokens
alias
to be mapped to
word.
- -tolower
-
Map vocabulary to lowercase, eliminating case distinctions.
- -multiwords
-
Split multiwords (words joined by '_') into their components when reading
N-best lists.
- -multi-char C
-
Character used to delimit component words in multiwords
(an underscore character by default).
- -noise noise-tag
-
Designate
noise-tag
as a vocabulary item that is to be ignored in aligning hypotheses with
each other (the same as the -pau- word).
This is typically used to identify a noise marker.
- -noise-vocab file
-
Read several noise tags from
file,
instead of, or in addition to, the single noise tag specified by
-noise.
- -keep-noise
-
Do not remove pause or noise tokens from hypotheses. The default
is to preserve noise tags but still eliminate pauses.
- -nbest-error
-
Compute the N-best error (minimum word error) of the N-best list read with
-nbest.
Pause and noise tokens (as specified with
-noise)
in the N-best list are ignored.
- -dump-posteriors
-
Output posterior probabilities of all N-best hypotheses
instead of choosing the best hypothesis.
In N-best mode, only the posterior probability for each hypothesis is output.
In lattice mode, the hyp posterior is followed by word posterior probabilities
for each (non-pause, non-noise) token in the hypothesis.
The
-max-rescore
option limits the number of hypotheses per N-best list processed.
- -dump-errors
-
Output word correctness indicators for all N-best hypotheses
instead of choosing the best hypothesis.
For each hypothesis, a line is output containing first the total number of
errors and the list of indicators of whether the corresponding word is
correct, substituted or inserted relative to the reference string.
The location of deleted words is also indicated by a corresponding marker.
The
-max-rescore
option limits the number of hypotheses per N-best list processed.
- -reference w1 w2 ...
-
Specifies a reference word string for
-dump-errors,
-nbest-error,
and
-lattice-error
options.
Additionally, in
-use-mesh
mode, the reference words are recorded in the word mesh and can be output
with
-write,
indicating which word in each alignment position is the correct one.
- -refs references
-
Read a table of reference transcripts from file
reference,
for when multiple N-best lists are processed (see
-nbest-files).
Each line in
references
must contain the sentence ID (the last component in the N-best filename
path, minus any suffixes) followed by zero or more reference words.
The following options only affect lattice mode.
- -read file
-
Reads an initial lattice from
file,
to be merged with additional paths constructed from the
N-best hypotheses.
- -lattice-files file
-
Reads the names of one or more lattices from
file
and aligns those lattices with the main lattice being built.
Each line of
file
must contain a lattice filename, optionally followed by a weight.
- -write file
-
Writes the resulting word posterior lattice or mesh to
file,
in
wlat-format(5).
- -write-dir directory
-
Write the resulting N-best lattices to
directory,
in files named after the input N-best lists,
for when multiple N-best lists are processed (see
-nbest-files).
- -prime-lattice
-
Start building the lattice with the best hypothesis obtained from
N-best error minimization. This produces slightly better alignments
and sometimes lower error rates. The default is to start with the
top-scoring hypothesis.
- -prime-with-1best
-
Similar to
-prime-lattice,
but uses the top-ranked sentence hypothesis for priming.
(Experience shows that
-no-reorder -prime-lattice
gives best results.)
- -prime-with-refs
-
Similar to
-prime-lattice,
but uses the reference words for priming.
- -no-merge
-
Build a lattice from the N-best hypotheses without merging edges
(string/lattice alignment). This creates a lattice with one disjoint path
per hypothesis, and is useful mainly for debugging purposes.
This option has no effect with
-use-mesh
since word meshes can represent only one word type per
alignment position.
- -lattice-error
-
Compute the lattice error (minimum word error) of the lattice read with
-read
or built with
-nbest.
- -dictionary file
-
Use word pronunciations listed in
file
to construct word alignments when building word meshes.
This will use an alignment cost function that reflects the number of
inserted/deleted/substituted phones, rather than words.
The dictionary
file
should contain one pronunciation per line, each naming a word in the first
field, followed by a string of phone symbols.
- -hidden-vocab file
-
Read a subvocabulary from
file
and constrain word meshes to only align those words that are either all
in or outside the subvocabulary.
This may be used to keep ``hidden event'' tags from aligning with
regular words.
- -record-hyps
-
Record the ranks of the hyps contributing to each word hypothesis in the
resulting word lattice;
the information is included in
-write
output.
SEE ALSO
ngram(1), nbest-optimize(1), nbest-scripts(1), nbest-format(5), wlat-format(5).
A. Stolcke, Y. Konig, and M. Weintraub,
``Explicit Word Error Minimization in N-best List Rescoring,''
Proc. Eurospeech, 163-166, 1997.
The ``word meshes'' used here are equivalent to the ``confusion networks''
described in:
L. Mangu, E. Brill, and A. Stolcke, ``Finding Consensus Among Words:
Lattice-based Word Error Minimization.'' Proc. Eurospeech,
vol. 1, 495-498, 1999.
BUGS
Several functions are not uniformly implemented for all rescoring modes
(e.g.,
-lattice-files,
-dictionary,
-record-hyps,
and
-nbest-backtrace
are currently effective only in mesh-lattice mode).
It is a common mistake (not a bug) to use the default LM weight with
N-best lists directly from Decipher.
Decipher N-best lists have the recognizer's LM weight already
built in, so they should be processed with
nbest-lattice -rescore-lmw 1 -posterior-scale LMW
where
LMW
is the LM weight during recognition.
This is not an issue if the N-best lists have been rescored with
rescore-decipher.
AUTHOR
Andreas Stolcke <stolcke@speech.sri.com>.
Copyright 1996-2010 SRI International