From Joris.Pelemans at esat.kuleuven.be  Wed Jul  9 07:03:27 2014
From: Joris.Pelemans at esat.kuleuven.be (Joris Pelemans)
Date: Wed, 09 Jul 2014 16:03:27 +0200
Subject: [SRILM User List] reproduce Penn Treebank KN5 results
Message-ID: <53BD4BAF.5060106@esat.kuleuven.be>

Hi all,

I'm trying to reproduce some reported N-gram perplexity results on the 
Penn Treebank with SRILM, but somehow my results are always different by 
a large degree. Since I will be interpolating with these models and 
comparing the interpolated model with others, I would really prefer to 
start on the same level :-).

The data set I'm using is the one that comes with Mikolov's RNNLM 
toolkit and applies the same processing of data as used in many LM 
papers, including "Empirical Evaluation and Combination of Advanced 
Language Modeling Techniques". In that paper, Mikolov et al report a KN5 
perplexity of 141.2. It's not entirely clear (1) whether they ignore OOV 
words or simply use the <unk> probability; and (2) whether it's a 
back-off or interpolated model, but I assume the latter as this has been 
reported as best many times. They do report using SRILM and no count 
cut-offs.

I have tried building the same model in many ways:

*regular:* ngram-count -order 5 -text data/penn/ptb.train.txt -lm 
models/ptb.train_5-gram_kn.arpa2 -kndiscount -interpolate
*open vocab:* ngram-count -order 5 -text data/penn/ptb.train.txt -lm 
models/ptb.train_5-gram_kn.arpa3 -kndiscount -interpolate -unk
*no sentence markers:* ngram-count -order 5 -text 
data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa4 -kndiscount 
-interpolate -no-sos -no-eos
*open vocab + no sentence markers:* ngram-count -order 5 -text 
data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5 -kndiscount 
-interpolate -unk -no-sos -no-eos
*back-off (just in case**):* ngram-count -order 5 -text 
data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5 -kndiscount 
-unk

None of them however, give me a perplexity of 141.2:

<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa2 -order 5
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
0 zeroprobs, logprob= -172723 ppl= 167.794 ppl1= 217.791

<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa3 -order 5 -unk
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
0 zeroprobs, logprob= -178859 ppl= 147.852 ppl1= 187.743

<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa4 -order 5
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
0 zeroprobs, logprob= -179705 ppl= 206.4 ppl1= 270.74

<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
0 zeroprobs, logprob= -186444 ppl= 182.746 ppl1= 234.414

<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
0 zeroprobs, logprob= -181381 ppl= 158.645 ppl1= 202.127

So... what am I missing here? 147.852 is close, but still not quite 141.2.

Joris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140709/33d396a0/attachment.html>

From s.gangireddy at sms.ed.ac.uk  Wed Jul  9 07:24:15 2014
From: s.gangireddy at sms.ed.ac.uk (Siva Reddy Gangireddy)
Date: Wed, 9 Jul 2014 23:24:15 +0900
Subject: [SRILM User List] reproduce Penn Treebank KN5 results
In-Reply-To: <53BD4BAF.5060106@esat.kuleuven.be>
References: <53BD4BAF.5060106@esat.kuleuven.be>
Message-ID: <CAL6CX2n5VOWzoCYKf8gxXFXsiN-ia3eSWrH6-HpM9boT+gQycw@mail.gmail.com>

Hi Joris,

Use the count cut-offs like this.

ngram-count -order 5 -text ptb.train.txt -lm templm -kndiscount
-interpolate -unk -gt3min 1 -gt4min 1
ngram -ppl ptb.test.txt -lm templm -order 5 -unk

By default SRILM uses different count cut-offs.

---
Siva



On Wed, Jul 9, 2014 at 11:03 PM, Joris Pelemans <
Joris.Pelemans at esat.kuleuven.be> wrote:

>  Hi all,
>
> I'm trying to reproduce some reported N-gram perplexity results on the
> Penn Treebank with SRILM, but somehow my results are always different by a
> large degree. Since I will be interpolating with these models and comparing
> the interpolated model with others, I would really prefer to start on the
> same level :-).
>
> The data set I'm using is the one that comes with Mikolov's RNNLM toolkit
> and applies the same processing of data as used in many LM papers,
> including "Empirical Evaluation and Combination of Advanced Language
> Modeling Techniques". In that paper, Mikolov et al report a KN5 perplexity
> of 141.2. It's not entirely clear (1) whether they ignore OOV words or
> simply use the <unk> probability; and (2) whether it's a back-off or
> interpolated model, but I assume the latter as this has been reported as
> best many times. They do report using SRILM and no count cut-offs.
>
> I have tried building the same model in many ways:
>
> *regular:* ngram-count -order 5 -text data/penn/ptb.train.txt -lm
> models/ptb.train_5-gram_kn.arpa2 -kndiscount -interpolate
> *open vocab:* ngram-count -order 5 -text data/penn/ptb.train.txt -lm
> models/ptb.train_5-gram_kn.arpa3 -kndiscount -interpolate -unk
> *no sentence markers:* ngram-count -order 5 -text data/penn/ptb.train.txt
> -lm models/ptb.train_5-gram_kn.arpa4 -kndiscount -interpolate -no-sos
> -no-eos
> *open vocab + no sentence markers:* ngram-count -order 5 -text
> data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5 -kndiscount
> -interpolate -unk -no-sos -no-eos
> *back-off (just in case**):* ngram-count -order 5 -text
> data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5 -kndiscount
> -unk
>
> None of them however, give me a perplexity of 141.2:
>
> <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm
> models/ptb.train_5-gram_kn.arpa2 -order 5
> file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
> 0 zeroprobs, logprob= -172723 ppl= 167.794 ppl1= 217.791
>
> <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm
> models/ptb.train_5-gram_kn.arpa3 -order 5 -unk
> file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
> 0 zeroprobs, logprob= -178859 ppl= 147.852 ppl1= 187.743
>
> <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm
> models/ptb.train_5-gram_kn.arpa4 -order 5
> file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
> 0 zeroprobs, logprob= -179705 ppl= 206.4 ppl1= 270.74
>
> <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm
> models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
> file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
> 0 zeroprobs, logprob= -186444 ppl= 182.746 ppl1= 234.414
>
> <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm
> models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
> file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
> 0 zeroprobs, logprob= -181381 ppl= 158.645 ppl1= 202.127
>
> So... what am I missing here? 147.852 is close, but still not quite 141.2.
>
> Joris
>
> _______________________________________________
> SRILM-User site list
> SRILM-User at speech.sri.com
> http://www.speech.sri.com/mailman/listinfo/srilm-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140709/629009b0/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140709/629009b0/attachment.ksh>

From Joris.Pelemans at esat.kuleuven.be  Thu Jul 10 01:43:57 2014
From: Joris.Pelemans at esat.kuleuven.be (Joris Pelemans)
Date: Thu, 10 Jul 2014 10:43:57 +0200
Subject: [SRILM User List] reproduce Penn Treebank KN5 results
In-Reply-To: <CAL6CX2n5VOWzoCYKf8gxXFXsiN-ia3eSWrH6-HpM9boT+gQycw@mail.gmail.com>
References: <53BD4BAF.5060106@esat.kuleuven.be>
	<CAL6CX2n5VOWzoCYKf8gxXFXsiN-ia3eSWrH6-HpM9boT+gQycw@mail.gmail.com>
Message-ID: <53BE524D.7020904@esat.kuleuven.be>

Hi Siva,

Thanks a lot, with these arguments the perplexity is very close to the 
reported 141.2 (still not entirely the same though):

<jpeleman at spchcl23:~/exp/025> ngram-count -order 5 -text 
data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa7 -kndiscount 
-interpolate -unk -gt3min 1 -gt4min 1
<jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt -lm 
models/ptb.train_5-gram_kn.arpa7 -order 5 -unk
file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
0 zeroprobs, logprob= -177278 ppl= *141.464* ppl1= 179.251

I wonder about the value of experiments that include <unk> in the 
perplexity calculation. Does it not make the problem a lot easier 
(predicting a huge class is not hard - imagine mapping all words to 
<unk>) and as such yield misleading results?

Joris


On 07/09/14 16:24, Siva Reddy Gangireddy wrote:
> Hi Joris,
>
> Use the count cut-offs like this.
>
> ngram-count -order 5 -text ptb.train.txt -lm templm -kndiscount 
> -interpolate -unk -gt3min 1 -gt4min 1
> ngram -ppl ptb.test.txt -lm templm -order 5 -unk
>
> By default SRILM uses different count cut-offs.
>
> ---
> Siva
>
>
>
> On Wed, Jul 9, 2014 at 11:03 PM, Joris Pelemans 
> <Joris.Pelemans at esat.kuleuven.be 
> <mailto:Joris.Pelemans at esat.kuleuven.be>> wrote:
>
>     Hi all,
>
>     I'm trying to reproduce some reported N-gram perplexity results on
>     the Penn Treebank with SRILM, but somehow my results are always
>     different by a large degree. Since I will be interpolating with
>     these models and comparing the interpolated model with others, I
>     would really prefer to start on the same level :-).
>
>     The data set I'm using is the one that comes with Mikolov's RNNLM
>     toolkit and applies the same processing of data as used in many LM
>     papers, including "Empirical Evaluation and Combination of
>     Advanced Language Modeling Techniques". In that paper, Mikolov et
>     al report a KN5 perplexity of 141.2. It's not entirely clear (1)
>     whether they ignore OOV words or simply use the <unk> probability;
>     and (2) whether it's a back-off or interpolated model, but I
>     assume the latter as this has been reported as best many times.
>     They do report using SRILM and no count cut-offs.
>
>     I have tried building the same model in many ways:
>
>     *regular:* ngram-count -order 5 -text data/penn/ptb.train.txt -lm
>     models/ptb.train_5-gram_kn.arpa2 -kndiscount -interpolate
>     *open vocab:* ngram-count -order 5 -text data/penn/ptb.train.txt
>     -lm models/ptb.train_5-gram_kn.arpa3 -kndiscount -interpolate -unk
>     *no sentence markers:* ngram-count -order 5 -text
>     data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa4
>     -kndiscount -interpolate -no-sos -no-eos
>     *open vocab + no sentence markers:* ngram-count -order 5 -text
>     data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5
>     -kndiscount -interpolate -unk -no-sos -no-eos
>     *back-off (just in case**):* ngram-count -order 5 -text
>     data/penn/ptb.train.txt -lm models/ptb.train_5-gram_kn.arpa5
>     -kndiscount -unk
>
>     None of them however, give me a perplexity of 141.2:
>
>     <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt
>     -lm models/ptb.train_5-gram_kn.arpa2 -order 5
>     file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
>     0 zeroprobs, logprob= -172723 ppl= 167.794 ppl1= 217.791
>
>     <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt
>     -lm models/ptb.train_5-gram_kn.arpa3 -order 5 -unk
>     file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
>     0 zeroprobs, logprob= -178859 ppl= 147.852 ppl1= 187.743
>
>     <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt
>     -lm models/ptb.train_5-gram_kn.arpa4 -order 5
>     file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 4794 OOVs
>     0 zeroprobs, logprob= -179705 ppl= 206.4 ppl1= 270.74
>
>     <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt
>     -lm models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
>     file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
>     0 zeroprobs, logprob= -186444 ppl= 182.746 ppl1= 234.414
>
>     <jpeleman at spchcl23:~/exp/025> ngram -ppl data/penn/ptb.test.txt
>     -lm models/ptb.train_5-gram_kn.arpa5 -order 5 -unk
>     file data/penn/ptb.test.txt: 3761 sentences, 78669 words, 0 OOVs
>     0 zeroprobs, logprob= -181381 ppl= 158.645 ppl1= 202.127
>
>     So... what am I missing here? 147.852 is close, but still not
>     quite 141.2.
>
>     Joris
>
>     _______________________________________________
>     SRILM-User site list
>     SRILM-User at speech.sri.com <mailto:SRILM-User at speech.sri.com>
>     http://www.speech.sri.com/mailman/listinfo/srilm-user
>
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140710/d586f093/attachment.html>

From nimrodia2 at yahoo.com  Sat Sep  6 07:10:30 2014
From: nimrodia2 at yahoo.com (nimrod d)
Date: Sat, 6 Sep 2014 07:10:30 -0700
Subject: [SRILM User List] make fails: libiconv functions undefined
Message-ID: <1410012630.98967.YahooMailNeo@web140104.mail.bf1.yahoo.com>

Hello,

When drying to build SRILM  make fails for me because it cannot find the definition for libincov functions.
How can I tell SRILM libinconv's location on my computer?

I'm running Ubuntu 14.04 

I have libinconv installed at: /nimrod/libinconv-1.14
and srilm installed at: /nimrod/srilm

I do need unicode support so I cannot compile with the NO_ICONV flag activated.

Thank you,

following is the last part of the output I get for $ Make World


g++ -march=athlon64 -m64 -Wall -Wno-unused-variable -Wno-uninitialized -DINSTANTIATE_TEMPLATES -fopenmp   -I. -I../../include   -u matherr -L../../lib/i686-m64  -g -O3  -o ../bin/i686-m64/ngram ../obj/i686-m64/ngram.o ../obj/i686-m64/liboolm.a ../../lib/i686-m64/libflm.a ../../lib/i686-m64/libdstruct.a ../../lib/i686-m64/libmisc.a ../../lib/i686-m64/libz.a  -lm  -lpthread 2>&1 | c++filt
../../lib/i686-m64/libmisc.a(File.o): In function `File::~File()':
/nimrod/srilm/misc/src/File.cc:125: undefined reference to `libiconv_close'
../../lib/i686-m64/libmisc.a(File.o): In function `File::reopen(char const*, char const*)':
/nimrod/srilm/misc/src/File.cc:165: undefined reference to `libiconv_close'
../../lib/i686-m64/libmisc.a(File.o): In function `File::reopen(char const*)':
/nimrod/srilm/misc/src/File.cc:214: undefined reference to `libiconv_close'
../../lib/i686-m64/libmisc.a(File.o): In function `File::reopen(char const*, unsigned long, int)':
/nimrod/srilm/misc/src/File.cc:246: undefined reference to `libiconv_close'
../../lib/i686-m64/libmisc.a(File.o): In function `File::reopen(std::string&, int)':
/nimrod/srilm/misc/src/File.cc:270: undefined reference to `libiconv_close'
../../lib/i686-m64/libmisc.a(File.o): In function `File::fgetsUTF8(char*, int)':
/nimrod/srilm/misc/src/File.cc:341: undefined reference to `libiconv_open'
/nimrod/srilm/misc/src/File.cc:379: undefined reference to `libiconv'
/nimrod/srilm/misc/src/File.cc:337: undefined reference to `libiconv_open'
collect2: error: ld returned 1 exit status
test -f ../bin/i686-m64/ngram
make[2]: *** [../bin/i686-m64/ngram] Error 1
make[2]: Leaving directory `/nimrod/srilm/lm/src'
make[1]: *** [release-programs] Error 1
make[1]: Leaving directory `/nimrod/srilm'
make: *** [World] Error 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140906/b0c57925/attachment.html>

From stolcke at icsi.berkeley.edu  Sat Sep  6 11:54:57 2014
From: stolcke at icsi.berkeley.edu (Andreas Stolcke)
Date: Sat, 06 Sep 2014 11:54:57 -0700
Subject: [SRILM User List] make fails: libiconv functions undefined
In-Reply-To: <1410012630.98967.YahooMailNeo@web140104.mail.bf1.yahoo.com>
References: <1410012630.98967.YahooMailNeo@web140104.mail.bf1.yahoo.com>
Message-ID: <540B5881.8080004@icsi.berkeley.edu>

On 9/6/2014 7:10 AM, nimrod d wrote:
> Hello,
>
> When drying to build SRILM  make fails for me because it cannot find 
> the definition for libincov functions.
> How can I tell SRILM libinconv's location on my computer?
>
> I'm running Ubuntu 14.04
> I have libinconv installed at: /nimrod/libinconv-1.14
> and srilm installed at: /nimrod/srilm
>
> I do need unicode support so I cannot compile with the NO_ICONV flag 
> activated.
>
> Thank you,
>
> following is the last part of the output I get for $ Make World
>
> g++ -march=athlon64 -m64 -Wall -Wno-unused-variable -Wno-uninitialized 
> -DINSTANTIATE_TEMPLATES -fopenmp   -I. -I../../include   -u matherr 
> -L../../lib/i686-m64  -g -O3  -o ../bin/i686-m64/ngram 
> ../obj/i686-m64/ngram.o ../obj/i686-m64/liboolm.a 
> ../../lib/i686-m64/libflm.a ../../lib/i686-m64/libdstruct.a 
> ../../lib/i686-m64/libmisc.a ../../lib/i686-m64/libz.a  -lm  -lpthread 
> 2>&1 | c++filt

The -liconv appears nowhere in the link command, so that's why you're 
getting the undefined references.  Try putting

     SYS_LIBRARIES = -L/nimrod/libinconv-1.14 -liconv

in common/Makefile.site.i686-m64 .   Make sure the -L option points to 
the directory that has the libiconv.so or libiconv.a file.

Andreas


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140906/2af5f466/attachment.html>

From shahramk at gmail.com  Tue Sep 23 17:41:13 2014
From: shahramk at gmail.com (Shahram)
Date: Wed, 24 Sep 2014 10:41:13 +1000
Subject: [SRILM User List] merging two language models
Message-ID: <CABjq+1-qEpo6BSaG6-VLNhgg1r8Hj_1td3OTv_PhnxuLNYZLNw@mail.gmail.com>

Hi,

Is there any way I can merge two language models?
I don't have access to the actual text of the two language models, so I
guess I cannot create the N-gram count of them to use in ngram-merge tool.
I wan wondering if I can merge the two language models together and create
a single LM out of them?

-- 
---
Regards

Shahram Kalantari
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140924/1528edd7/attachment.html>

From stolcke at icsi.berkeley.edu  Tue Sep 23 20:23:31 2014
From: stolcke at icsi.berkeley.edu (Andreas Stolcke)
Date: Tue, 23 Sep 2014 20:23:31 -0700
Subject: [SRILM User List] merging two language models
In-Reply-To: <CABjq+1-qEpo6BSaG6-VLNhgg1r8Hj_1td3OTv_PhnxuLNYZLNw@mail.gmail.com>
References: <CABjq+1-qEpo6BSaG6-VLNhgg1r8Hj_1td3OTv_PhnxuLNYZLNw@mail.gmail.com>
Message-ID: <54223933.9030405@icsi.berkeley.edu>

On 9/23/2014 5:41 PM, Shahram wrote:
> Hi,
>
> Is there any way I can merge two language models?
> I don't have access to the actual text of the two language models, so 
> I guess I cannot create the N-gram count of them to use in ngram-merge 
> tool. I wan wondering if I can merge the two language models together 
> and create a single LM out of them?
You can merge two ngram backoff LMs using

ngram -order N  -lm LM1  -mix-lm LM2 -lambda W -write-lm MIXLM

Where N is the maximum ngram order in the two LMs, LM1, LM2 are the 
input models, W is the weight to give to LM1, and MIXLM is the merged 
model file.

Using appropriate options you can merge more than 2 LMs.  Check out the 
ngram man page description of options
-mix-lm2 etc.
-mix-lambda2 etc.
-read-mix-lms
and related options.

Andreas



From maxim_korenevski at mail.ru  Thu Sep 25 23:28:02 2014
From: maxim_korenevski at mail.ru (=?UTF-8?B?0JzQsNC60YHQuNC8INCa0L7RgNC10L3QtdCy0YHQutC40Lk=?=)
Date: Fri, 26 Sep 2014 10:28:02 +0400
Subject: [SRILM User List] =?utf-8?q?Questions_on_conversion_word_lattice_?=
	=?utf-8?q?to_mesh?=
Message-ID: <1411712882.492526412@f356.i.mail.ru>

 Hi, all,

I use lattice-tool.exe to convert word lattices (in HTK-like SLF format) obtained from recognition pass into a word confusion networks (meshes). SLFs contains both acoustic and language model scores and lm_scale parameter (used by recognizer) in its header. Word insertion penalty was set to 0.

When I scale both acoustic and LM scores with a constant factor C, I see that the 1-best path through mesh depends strongly on it. When C is large the mesh 1-best sentence coincides to word lattice 1-best sentence (which is in turn recognizer 1-best output), but when C goes down to zero, WER of mesh 1-best sequence increases monotonically.
I believed that optimal value of this factor should be about 1/lm_scale (as proposed in several papers, for example, "Confidence measures for Large Vocabulary Speech Recognition" by F.Wessel et al., 2001), but I observe an average WER increase about 5% absolute over large number of files for such factor value.

Is it caused by incorrect use of lattice-tool for mesh generation or this situation is normal ?

Maxim.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140926/4c3bab51/attachment.html>

From stolcke at icsi.berkeley.edu  Fri Sep 26 13:30:43 2014
From: stolcke at icsi.berkeley.edu (Andreas Stolcke)
Date: Fri, 26 Sep 2014 13:30:43 -0700
Subject: [SRILM User List] Questions on conversion word lattice to mesh
In-Reply-To: <1411712882.492526412@f356.i.mail.ru>
References: <1411712882.492526412@f356.i.mail.ru>
Message-ID: <5425CCF3.702@icsi.berkeley.edu>

On 9/25/2014 11:28 PM, ?????? ??????????? wrote:
> Hi, all,
>
> I use lattice-tool.exe to convert word lattices (in HTK-like SLF 
> format) obtained from recognition pass into a word confusion networks 
> (meshes). SLFs contains both acoustic and language model scores and 
> lm_scale parameter (used by recognizer) in its header. Word insertion 
> penalty was set to 0.
>
> When I scale both acoustic and LM scores with a constant factor C, I 
> see that the 1-best path through mesh depends strongly on it. When C 
> is large the mesh 1-best sentence coincides to word lattice 1-best 
> sentence (which is in turn recognizer 1-best output), but when C goes 
> down to zero, WER of mesh 1-best sequence increases monotonically.
What you're seeing is expected.   In fact, the scaling of of scores can 
be achieved using the lattice-tool -posterior-scale option, you don't 
have to do it yourself by manipulating the scores in the lattices.

        -posterior-scale S
               Scale the transition weights by dividing by S for the 
purpose of
               posterior probability computation.  If the input weights  
repre-
               sent combined acoustic-language model scores then this 
should be
               approximately the language model weight  of  the 
recognizer  in
               order  to  avoid  overly peaked posteriors (the default 
value is
               8).


> I believed that optimal value of this factor should be about 
> 1/lm_scale (as proposed in several papers, for example, "Confidence 
> measures for Large Vocabulary Speech Recognition" by F.Wessel et al., 
> 2001), but I observe an average WER increase about 5% absolute over 
> large number of files for such factor value.
Now the default posterior-scale (see above) is equal to the LM score 
weight,  just as advocated in the paper you mention.  BTW, the rationale 
for this choice can be found in our earlier work on expected error 
minimization, e.g., in section 3.6 of this paper 
<http://www.speech.sri.com/cgi-bin/run-distill?ftp:papers/eurospeech99-consensus.ps.gz>.
So if you are scaling the scores yourself and also use the default 
-posterior-scale then you would end up with the wrong scaling.

If you are not seeing a lower WER using the default posterior scaling 
then you probably won't see a gain from confusion networks on your task. 
This could be for various reasons, e..g, the lattices are too thin, or 
the utterances too short.

Andreas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.speech.sri.com/pipermail/srilm-user/attachments/20140926/0d60c092/attachment.html>