[Teammetrics-discuss] Never ending nntpstat problems

[Andreas Tille]

On Thu, Dec 08, 2011 at 01:23:30PM +0530, Sukhbir Singh wrote:
> 1. For a given list X, we fetch ALL the headers and the message bodies.
> 2. Then we write them once we have all of them.
> 
> I can optimize this by:
> 
> 1. For a given list X, fetch a single message body.
> 2. Write it to the file.

I would definitely always write what needs to be written - so yes, write
immediately.
 
> But don't you think this is going to cause a lot of I/O activity? It
> *might* solve the MemoryError, but is it a good approach?

It is *definitely* the best approach.  I noticed in several cases that
you tend to keep a lot in memory what in principle is data ready for
writing.  While I did not studied IT science and I do not know what your
teachers told you this is not a good strategy in general.

You do not need to care about I/O activity yourself!  That's the job of
your operating system which will cache writing anyway (if you do not
explicitely call fflush - well, that's C, but there will be a similar
Python command).  So just write any data that can be written once you
see no need for changing them any more.

Kind regards

       Andreas.

-- 
http://fam-tille.de