[KLUG Members] Memory problems

[Adam Williams]

>>              total       used       free     shared    buffers     cached
>> Mem:        517260     512028       5232     149724      52232     362952
>> -/+ buffers/cache:      96844     420416
>>Perfectly normal.  420Mb is being used to cache I/O,  which is
>>all good.
>Mind if I go off on a tangent here?  Apparently "cached" is *mostly*
>or *usually* just disk cache.  I gather disk cache is like disk
>buffer except one is a cache of what's been read, and the other's a
>cache of what's been written (I'm not sure which is which).

My understanding is that cache represents something from somewhere else
(disk) in order to speed up operations.  Buffers contains data either in
or out bound in order to handle devices that operate at various speeds. 
TCP for example can consume quite a bit in the way if buffer,  but I'd
wager most of it is dirty disk pages.

>But "cached" can also refer to other ways that RAM is being used
>besides mirroring disk sectors.  I know this because a machine with
>4 GB of RAM that's running MySQL under heavy load is getting to the
>point where the kernel needs to kill the "mysqld" process because it
>runs out of RAM... even though there are about 3 GB of RAM listed in
>the "cached" category in "free"'s output.
>This leaves little footprints in /var/log/messages like:
>Dec  1 12:34:56 myhost kernel: Out of Memory: Killed process 23798 (mysqld-max).

Are you sure you aren't just hitting a ulimit?  

Is there a call to ulimit in the mysqld startup script?  By default a
given uid/process can only open so many files, consume so much RAM, have
so many processess, etc...  This keeps one "user" from crunching the
machine.  I know I have had to adjust ulimits on web servers (for httpd)
and samba servers.

http://www.linuxdoc.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/x4733.html

/etc/security $ ulimit -a
core file size (blocks)     1000000
data seg size (kbytes)      unlimited
file size (blocks)          unlimited
max locked memory (kbytes)  unlimited
max memory size (kbytes)    unlimited
open files                  1024
pipe size (512 bytes)       8
stack size (kbytes)         8192
cpu time (seconds)          unlimited
max user processes          2048
virtual memory (kbytes)     unlimited

Limits are pretty generous these days, but the max user processes one is
still pretty easy to hit on a modern machine.

Also you can tune how aggresively the kernel will take back memeory
pages from cache using sysctl.

If the mysqld use shared memory for manageing locks, etc... and you have
lots of them you may need to raise the shared memory limits (depending
on your kernel).  I don't know anything about mysql.
 
>So... does anyone know what gets "cached" besides disk?

Nothing that I know of.