[KLUG Members] NFS / server tuning for mail storage?

[Adam Tauno Williams]

> Having solved my courier-imap/vmailmgr, then having moved to qmail-ldap
> with courier-imap/pop3, I'm getting farther along in my testing and
> noticing that when mailboxes get larger than a few thousand messages,
> the mailbox is effectively useless.

Yep.

> Netapp, with gigabit connection to the switch.
> Server: 800 MHz serverblade, 512 MB RAM, 100Mbit connection to switch.
> My plans are to use LVS and balance say, 3 to 4 servers from the 800 to
> 1200 MHz range handling mail services, with them all connecting to the
> netapp back end via NFS.

You are crazy.

(a) NFS latency and locking issues will *****DESTROY****** your network.
(b) With regards to performance one multi-processor highly-redundant
server will blow the above setup away, even setting NFS issues aside.

> But my fear is that people are going to have huge mailboxes, imapd will
> sit for minutes on end, the users will get frustrated, and then complain.

Oh yea!  That is exactly what will happen. 

> Are there more effective NFS options for these servers to use?  Right
> now, I'm using soft, noatime, tcp as my options. I don't know if using
> TCP is any *faster*, or if setting the rsize or wsize will help any either.

(a) tcp is slower
(b) using "soft" is NOT safe for this application
(c) yes, having a big rsize/wsize will help - but not enough

SCSI-over-IP or ATA-over-IP are MUCH faster than any CIFS/NFS mount.
BUT they will *NOT* give you the kind of redundancy or fail-over you are
looking for (unless you spend a heap load of money).

I'm afraid you have only one solution if you want something resembling
the above configuration: Cyrus IMAPd & Murder (Murder is a Cyrus IMAPd
subservice for distributing the mail spool across multiple boxes).

Some data is easier to distribute than others, IMAP service is actually
pretty hard to do for a variety of gnarly protocol reasons.  I'd suggest
one bullet-proof box and a hot standby spare if necessary.  

> Furthermore, should I be splitting up my services? The 1200 MHz systems
> will have 1 G of ram instead of 512, should I use those for imap/pop
> processing and the slower, 800 MHz systems for SMTP?

Yes, splitting SMTP from IMAP is good.  But usually (given a sane
configuration) even an 800MHz box can handle a massive amount of SMTP
traffic.  Make darn sure you have a local DNS cache and a local LDAP
replicant and that you use LDAPI (domain socket) communication if at all
possible [ domain socket communication is many many many many many times
faster that IP communication, has almost zero latency, and can be
dispatched with less CPU effort ].

> Any ideas would be most appreciated. I've worked pretty hard on this,
> and I'm wondering if there's a fundamnetal speed problem that I won't be
> able to overcome with more serverblades, or if there's other ways I
> could squeak more performance out of them.

More servers in this case will make the problem worse.  Portmap/RPC
locking is darn slow, NFS writes are not asyncronous (at least unless
you set them so - and if you do you are certifiably mad), and NFS just
isn't efficient.