Search SRILM-USER Archives

Re: limitations in ngram-merge

In message <159323335F97074D9A594D676652B06754A028 at ADDRESS HIDDENabo>you wrote:
> Hi
>
> We are currently having a problem with the merging of count files using ngram
> -merge.
> It seems that there is a limitation in the size of the resulting file of 2GB.
> Can you give us some information if this is limitation is due to the program
> or if it is a limitation due to the configuration of our system. We are runni
> ng ngram-merge in a PIV 2,66GHz 1GB RAM in Suse 10.0.

It's probably an OS limitation.  SRILM uses level-2 I/O functions
(see fopen(3)).  

We have certainly handled files larger than 2 GB on our Linux machines.
But those files that are usually gzipped (ending in .gz).  SRILM
doesn't read or write those directly, since the I/O is to a pipe
that talks to the gzip program.  Maybe you can try using gzipped files
in your case too.

--Andreas

Click here to go to the SRILM home page.