<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
On 1/8/2010 3:57 AM, Manuel Alves wrote:
<blockquote
cite="mid:495c9ccd1001080357u61df2223gae7b1c20775716be@mail.gmail.com"
type="cite"><br>
<br>
<div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Manuel Alves</b> <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:beleira@gmail.com">beleira@gmail.com</a>></span><br>
Date: Fri, Jan 8, 2010 at 10:40 AM<br>
Subject: Re: Fwd: ngram-count<br>
To: Andreas Stolcke <<a moz-do-not-send="true"
href="mailto:stolcke@speech.sri.com">stolcke@speech.sri.com</a>><br>
<br>
<br>
1. ngram-count -text CETEMPublico1.7 -lm LM<br>
2.I test it in this way:<br>
I use the client-server architecture of
SRILM<br>
SERVER : ngram -lm ../$a -server-port 100
-order 3 <br>
CLIENT : ngram -use-server
100\@localhost -cache-served-ngrams -ppl $ficheiro -debug 2 2>&1<br>
where $ficheiro is this:<br>
</div>
</blockquote>
<br>
<blockquote
cite="mid:495c9ccd1001080357u61df2223gae7b1c20775716be@mail.gmail.com"
type="cite">
<div class="gmail_quote"> <br>
<br>
p( observássemos | que ...) = 0 [ -inf ]<br>
</div>
</blockquote>
<br>
<blockquote
cite="mid:495c9ccd1001080357u61df2223gae7b1c20775716be@mail.gmail.com"
type="cite">
<div class="gmail_quote">file final.txt: 6 sentences, 126 words, 0
OOVs<br>
6 zeroprobs, logprob= -912.981 ppl= 1.7615e+07 ppl1= 4.05673e+07<br>
</div>
</blockquote>
<br>
It looks to me like everything is working as intended. You are
getting zeroprobs, but not a large number of them.<br>
They are low-frequency words (like the one above), so it makes sense,
since they are probably not contained in the training corpus.<br>
<br>
The perplexity is quite high, but that could be because of a small, or
mismatched training corpus. You didn't include the output of the
ngram-count program, it's possible that the GT (default) discounting
method reported some problems that are not evident from your mail.<br>
<br>
One thing to note is that with network-server LMs you don't get OOVs,
because all words are implicitly added to the vocabulary. Consequently,
OOVs are counted as zeroprobs instead, but both types of tokens are
equivalent for perplexity computation.<br>
Still, you could run <br>
ngram -lm ../$a -order 3 -ppl $ficheiro -debug 2<br>
just to make sure you're getting the same result.<br>
<br>
Andreas<br>
<br>
<blockquote
cite="mid:495c9ccd1001080357u61df2223gae7b1c20775716be@mail.gmail.com"
type="cite">
<div class="gmail_quote"><u><font color="#888888">Manuel Alves. </font></u><br>
<div>
<div class="h5"><br>
<div class="gmail_quote">On Thu, Jan 7, 2010 at 8:35 PM, Andreas
Stolcke <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:stolcke@speech.sri.com" target="_blank">stolcke@speech.sri.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
<div>
<div>On 1/6/2010 10:34 AM, Manuel Alves wrote:
<blockquote type="cite"><br>
<br>
<div class="gmail_quote">---------- Forwarded message ----------<br>
From: <b class="gmail_sendername">Manuel Alves</b> <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:beleira@gmail.com" target="_blank">beleira@gmail.com</a>></span><br>
Date: Wed, Jan 6, 2010 at 6:33 PM<br>
Subject: ngram-count<br>
To: <a moz-do-not-send="true" href="mailto:srilm-user@speech.sri.com"
target="_blank">srilm-user@speech.sri.com</a><br>
<br>
<br>
Hi people.<br>
I need help whith ngram-count because i am training a model but when
after i try to use it some test example he gives me Zeroprobs in the
output.<br>
This means that the model is bad trained?<br>
Please answer me.<br>
Best regards,<br>
<font color="#888888">Manuel Alves.<br>
</font></div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>