<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 8/3/2012 3:18 AM, Meng Chen wrote:<br>
</div>
<blockquote
cite="mid:CA+bc0moHaOd-e0LPfoC=LqTUi7kLn_5_PrOMvXW42svVixVNXQ@mail.gmail.com"
type="cite">Hi, in <b>make-batch-counts</b>, we need to set the
batch-size in order to count faster. it says "For maximum
performance, batch-size should be as large as possible without
triggering paging". However, sometimes I found it would crash if I
set it too large (eg. 500). So I want to ask if there is any
limitation to batch-size. Suppose every text in file list is <b>a</b>
MB, the memory of server is <b>b</b> MB,the batch-size should not
be larger than <b>b/a</b>, is it right? Or some other
limitations?</blockquote>
<br>
make-batch-counts actually works sequentially, so you can devote all
of a machine's memory to computing counts, unless you have other
things running. If you want to parallelize the counting you have to
devise your own method for that.<br>
<br>
Of course in general there other things running on a machine, and
some systems start randomly killing processes when you exhaust their
memory. I suspect that's what is happening in your case. There is
no built-in limitation in make-batch-counts, other than the limits
imposed by the system. Another reason your job might have crashed
is that you are using 32bit binaries and you were hitting against
the 2 or 4 GB limit inherent in 32bit memory addresses.<br>
<br>
Andreas<br>
<br>
</body>
</html>