<br><div class="gmail_quote">Dear SRILM List Members,<br><br>I am using / augmenting SRILM for our own language modeling purposes. One decision that I make is to separate language models for different types of tokens. In my corpus, one type of token starts with a '+' character, whereas another does not. The difference between these is that although their counts are exactly the same and their respective count files, language models generated by them have similar sizes, I am observing significant differences in their respective performances in running the ngram-count command. <br>
<br>For instance, the tokens that does not start with a '+' may finish creating a language model for a training data count file by using ngram-count in 6 seconds (by using the -read option), whereas the other one would finish in 42 seconds. Thus there seems to be a 6-7 times difference in ngram-count performance using count files generated for tokens that start with a '+' and for the ones that do not.<br>
<br>I am curious if there is an internal decision that prevents model building procedure for tokens that start with a '+' character to perform as fast as tokens of other types. What might be causing this performance difference?<br>
<br clear="all">Thanks,<br>
Ergun<br><br>end<br><font color="#888888"><br></font></div>-- <br>Ergun Bicici<br>Koc University<br><br>