select-vocab - Select a maximum-likelihood vocabulary from a mixture of corpora.
select-vocab [ option ... ] -heldoutfile f1 f2 ...
picks a vocabulary from the union of the vocabularies of files
in order to maximize the likelihood of the heldout file. When invoked
as above, the program will print out (unsorted) the list of words in
all of the input corpora together with their weights. This list may
subsequently be sorted to put the words in decreasing order of weight
and a vocabulary may be chosen by picking a suitable threshold weight
and ignoring words with weight less than this.
A number of automatically detected formats are supported for the input
They can be count files, which are characterized by each line ending
in a number, ARPA language models in
or simply text files. If they are text-files, further, and
their names end in ".sentid", it is assumed that the first field of
each line is a sentence identifier that is then discarded.
Furthermore, all of the input files can also be compressed (if gzip is
installed and available on the system).
Prints a short help message.
Likelihood maximization is performed on the contents of
This file may also be in any of the formats supported for the input
corpora, namely: text, counts, sentid, or ARPA-lm.
Suppresses printing of progress and other informative messages during
execution. By default the script writes these out to the output error
The combined final counts are scaled by
before being written out. This makes it possible to sort the output
list numerically with sort(1). The default scale is 1e6.
This implementation corrects a minor error in the algorithm
specification in . The paper describes corpus level interpolation,
but the script actually does word-level interpolation.
The program is written in perl(1) and requires it to be installed in
order to run.