<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12pt"><div>Dear all,</div><div><br></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;">I have some questions regarding perplexity...I am very thankful for your time/ answers.</div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;"><br></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;">Settings:</div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family:
HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;">- one language model LM_A estimated using training corpus A </div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;">- one language model LM_B estimated using training corpus B (B = corpus_A + corpus_X)</div><div style="background-color: transparent;"><br class="Apple-interchange-newline">My intention is to prove that model B is better than model A so I though I should show that the perplexity decreased (which can be seen from the ppl files).</div><div><br></div><div style="color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; background-color: transparent; font-style: normal;">Commands used to
estimate ppl:</div><div style="background-color: transparent;">$NGRAM_FILE -order 3 -lm $WORKING_DIR"lm_A/lmodel.lm" -ppl $WORKING_DIR"test.lowercased."$TARGET > $WORKING_DIR"ppl_A.ppl"<br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;">$NGRAM_FILE -order 3 -lm $WORKING_DIR"lm_B/lmodel.lm" -ppl $WORKING_DIR"test.lowercased."$TARGET > $WORKING_DIR"ppl_B.ppl"<br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><br></div><div
style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;">This contents of the two ppl files is (A then B):</div><div style="background-color: transparent;">1000 sentences, 21450 words, 0 OOVs</div><div style="background-color: transparent;">0 zeroprobs, logprob= -57849.4 ppl= 377.407 ppl1= 497.67</div><div style="background-color: transparent;">-------------------------------------------------------------------------------------------</div><div style="background-color: transparent;">1000 sentences, 21450 words, 0 OOVs</div><div style="background-color: transparent;">0 zeroprobs, logprob= -55535.3 ppl= 297.67 ppl1= 388.204</div><div style="background-color: transparent;"><br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial,
'Lucida Grande', sans-serif; font-style: normal;">Questions:</div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;">1. Why do I get 0 OOVs? I checked using the compute-oov-rate script how many OOV there are in the test data compared to the training and it gave me the result "OOV tokens: 393 / 21450 (1.83%) excluding fragments: 390 / 21442 (1.82%)".</div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;">2. I read on the srilm-faq that "<span style="font-family: 'Times
New Roman'; font-size: 16px;">Note that perplexity comparisons are only ever meaningful if the vocabularies of all LMs are the same." </span><span style="font-size: 12pt;">Since I want to compare perplexities of two LM I am wondering if I did the right thing with my settings and commands used. The two LM were estimated on different training corpora so the vocabularies are not identical, right? Please tell me what am I doing wrong.</span></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 12pt; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><span style="font-size: 12pt;"><br></span></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><span style="font-size: 12pt;">3. If those two perplexities were computed
correctly, then could you please tell me if their difference means that the LM model has been really improved and if there is a measure that says if this improvement is significantly? </span></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 12pt; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><span style="font-size: 12pt;"><br></span></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;">Thank you very much for your time.</div><div style="background-color: transparent;"><br></div><div style="background-color: transparent; color: rgb(0, 0, 0); font-size: 16px; font-family: HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-style: normal;"><br></div></div></body></html>