[KLUG Members] Need more space for /var issue resolved

[Bob Kanaley]

Peter,

>44% (of what?) should be enough drive space. Are you using sendmail?
44% of a 251MB partition. I am running Postfix in a chroot jail with a
Vexira anti-virus filter.

>What do the sendmail and POP3 server logs say?
Nothing really of note in maillog, messages, or syslog.

The first insufficient system storage error message came from my off-site
backup mail server. I got a message addressed to Postmaster on Saturday
5/10/03. A similar but longer message came in Sunday.

Transcript of session follows.

 Out: 220 dmz02.agdia.com ESMTP Postfix
 In:  EHLO localhost.localdomain
 Out: 250-dmz02.agdia.com
 Out: 250-PIPELINING
 Out: 250-SIZE 20480000
 Out: 250-ETRN
 Out: 250 8BITMIME
 In:  MAIL FROM:<briannayoung_25j@mail.com>
 Out: 250 Ok
 In:  RCPT TO:<diane@agdia.com>
 Out: 452 Insufficient system storage
 In:  QUIT
 Out: 221 Bye

No message was collected successfully.

When I checked my mail Monday morning, I received the error messages along
with a host of other messages.

I ran df -h and found the following:

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda8             251M   39M  198M  17% /
/dev/hda1              23M  2.5M   19M  12% /boot
/dev/hda6             8.4G   56M  7.8G   1% /home
/dev/hda5             8.4G  382M  7.5G   5% /usr
/dev/hda7             251M  226M   12M  95% /var

As you can see, there was plenty of empty space on the 18 GB drive, but only
251 MB in the /var partition.

I suspect that the /var partition filled up over the weekend when most users
were not making regular POP3 email runs. A few computers left on over the
weekend continued to do POP3 mail runs that cleared out enough space in
/var/spool/mail for some mail to continue to come in. Monday morning as
machines came online, /var continued to clear out.

I tried to repartition with parted
http://www.gnu.org/manual/parted-1.6.1/html_mono/parted.html#SEC7 but got no
place fast.

To avoid the build and library dependencies on the stripped down and
hardened RedHat 7.0 distro with the default kernel running in my DMZ, I
downloaded a bootable parted floppy disk image off the GNU Parted website.
There was no parted on the disk image! Go figure.

I gave up on parted, ordered a new hard disk drive and began to worry about
how I was going to copy all of /var to the new hard disk on a running
system. To keep the system running I looked at disk use in /var with ls
|xargs "du -s -h". I found most of the disk use was in /var/log. I dumped my
rotated maillogs and posted my original message for help on how to do this.

Adam was kind enough to point out that I really didn't need to replace all
of /var just mount /dev/hdb1, cp /var/spool/mail to /dev/hdb1 and mount
/dev/hdb1 as /var/spool/mail in initab.

While this was unfolding, I had numerous users report rather bizarre email
behavior:

The company founder got a call from one of the list members on a small list
we host. The list member called to complain about problems posting to the
list. The boss sent a message to the list and CC'ed himself. The boss did
not get a bounced message error, but neither he nor the list got the
message. THERE WERE NO ERRORS IN LOGFILES.

Similarly another end user sent a meeting reminder to three in-house company
employees. In talking to the intended recipients he discovered no one had
gotten the message. He did not get a bounced message error and a copy of the
message was in his sent folder. From his desk I resent the message with an
extra CC to myself. It took over an hour for the message to be received at
my desktop.

When other users called to say they were having email problems I did not
document or investigate, I just told them I was aware that there was a
problem and I working on it.

Now that I know that space is not an issue I can begin to investigate the
backlog of email problem reports to determine what is happening. If I can't
figure it out, I know you and the KLUG experts will get me pointed in the
right direction.

Thanks for the help.

Regards,

Bob

Robert V. Kanaley
Manager Information Systems
Agdia, Inc.
rvk@agdia.com
http://www.agdia.com


-----Original Message-----
From: "Peter Buxton" <somercet@core.com>
> Moving all the rotated log files onto a different partition cut my du from
> 92% to 56%. However around the time I did this, I got additional user
> reports that mail is still not being sent or received as expected. Since I
> know that space on /var is an issue, I will address that first, then start
> looking into end user reports of email problems more closely.

44% (of what?) should be enough drive space. Are you using sendmail?
What do the sendmail and POP3 server logs say?

> My new hard disk drive just arrived so, I shall plod through making that
> /var on the existing system.

It's easy. Really. Just boot the kernel in "single" mode when you move
anything.