[Freedombox-discuss] [tahoe-dev] Recommendations for minimal RAM usage ?

Pierre Alexandre SCHEMBRI pierre.alexandre.schembri at gmail.com
Tue Mar 6 08:25:27 UTC 2012


Hi,

about RAM usage, there a very interesting way to improve performance :
gzipped swapping.
There is  a tool / or a kernel tool (don't remember which was it)
called zram i think which allow to create a gzipped ramdisk upon boot,
then it is possible to swap on it.
This optimization is used on MeeGo (I got a N9 running with this optim.).
However, below 64M, we can't expect great perf.

P.A.

2012/3/6 Eugen Leitl <eugen at leitl.org>:
> ----- Forwarded message from Johannes Nix <johannes.nix at gmx.net> -----
>
> From: Johannes Nix <johannes.nix at gmx.net>
> Date: Mon, 5 Mar 2012 21:44:31 +0100
> To: tahoe-dev at tahoe-lafs.org
> Subject: Re: [tahoe-dev] Recommendations for minimal RAM usage ?
> X-Mailer: Claws Mail 3.7.4 (GTK+ 2.20.1; i486-pc-linux-gnu)
> Reply-To: Tahoe-LAFS development <tahoe-dev at tahoe-lafs.org>
>
> Hello Zooko,
>
>> It is cool that the Tahoe-LAFS storage server works at all on that
>> tiny NAS device! Does it work okay? Is your desire to reduce the
>> memory footprint to eliminate some failures, or to improve
>> performance, or to allow concurrent use of other processes on the same
>> device, or to move it to an even smaller device, or something else?
>
> I think it's cool as well. What I like
> especially is the idea that people can put such
> an inexpensive  box as personal server to share information
> and make themselves more independent from
> data-gathering companies. It would also be
> quite easy to provide a ready-to use image or
> software package.
>
> For me, the server worked very well in the last
> eight days.  Today however, we've seen on VG2 huge latencies of
> about 6 seconds, which occured when I did a backup upload at
> the same time. I ran also an NFS daemon to
> upload data and maybe that was sometimes becoming
> too much. Neither NFS nor tahoe-backup uses
> much RAM but I suspect that this leads occasionally
> to a kind of memory gridlock in kernel space; NFS
> can sometimes stall for many seconds.
>
> So, I think the reliability and speed could be increased
> when reducing RAM further.
>
>> > The storage-server side of Tahoe doesn't really do that much, so
>> > I'd be pleased as punch if it could fit into 10 or 20MB. To do
>> > that, I think we'd need to avoid loading all of the code when we're
>> > not running a client
>
>>
>> Wouldn't the code (.py's, .pyc's, or .so's) tend to be swapped out if
>> it unused? Or maybe unused code gets interspersed with code that is
>> needed, so it ends up bloating the memory usage?
>
> Memory is managed by the OS in 4k size pages. Entirely unused
> pages are simply swapped out. A reduction in the resident set size,
> in the amount of file buffering required or a grouping of often used
> code / data could probably achieve some speed-up.
> The simplest measure could be to import modules
> in a grouped order, with server and client
> functions in different groups.
>
> How difficult would it be to rip out a server-only
> program? This could run with increased
> priority. Is there a fundamental reason that
> both processes run in the same interpreter instance? I've
> experimented some time with real-time audio processing
> in Python, and a low-hanging fruit for multiprocessor
> systems is to separate different processes which
> are not serialized by Python's GIL (Global Interpreter Lock).
>
> Of course, my NAS has only one core but most future
> desktop and server systems will have much more than one.
>
> Regards,
>
> Johannes
>
>
>
> Am Sun, 4 Mar 2012 21:33:08 -0700
> schrieb "Zooko Wilcox-O'Hearn" <zooko at zooko.com>:
>
>> Dear Johannes Nix:
>>
>> It is cool that the Tahoe-LAFS storage server works at all on that
>> tiny NAS device! Does it work okay? Is your desire to reduce the
>> memory footprint to eliminate some failures, or to improve
>> performance, or to allow concurrent use of other processes on the same
>> device, or to move it to an even smaller device, or something else?
>>
>> The numbers you give sound about right. I measured resident RAM usage
>> on 32-bit x86 servers a couple of times, and found that it started at
>> around 32 MB and the high water mark of resident RAM usage was about
>> 56 MB.
>>
>> I don't know of any technique that could further reduce it, aside from
>> profiling the memory usage and reporting back to this list. If your
>> profiling turned up some "low hanging fruit" of wasted memory, then
>> perhaps some Tahoe-LAFS developer would be motivated to try reducing
>> that.
>>
>> Regards,
>>
>> Zooko
>> _______________________________________________
>> tahoe-dev mailing list
>> tahoe-dev at tahoe-lafs.org
>> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>
> _______________________________________________
> tahoe-dev mailing list
> tahoe-dev at tahoe-lafs.org
> http://tahoe-lafs.org/cgi-bin/mailman/listinfo/tahoe-dev
>
> ----- End forwarded message -----
> --
> Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
> ______________________________________________________________
> ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
> 8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
>
> _______________________________________________
> Freedombox-discuss mailing list
> Freedombox-discuss at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/freedombox-discuss



More information about the Freedombox-discuss mailing list