www.lewiz.org

The last couple of days have involved some seriously weird /var behaviour over at CompSoc.

I’d narrowed the filling up of /var down to Apache’s error log, and had been removing it more or less twice a day. Today I disabled the error.log file and restarted Apache… a measure designed to last us until I had chance to properly fix the issue.

This evening I come along and have a prod:

# df -h /var
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1d    496M    488M    -32M   107%    /var

Yikes! That’s not good, but… what’s this?

# du -sh /var
143M    /var

wtf? This got me pretty confused, but some searching around and I came upon a handy article at www.cyberciti.biz/tips/freebsd-why-command-df-and-du-reports-different-output.html that led me down the path of lsof. Here’s what I did:

# lsof|grep var | sort -r -k 7
lsof: WARNING: compiled for FreeBSD release 6.0-RELEASE-p6; this is 6.2-RELEASE.
httpd     91413          root   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd     10028           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd     10022           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd     10021           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9973           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9881           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9865           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9809           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9807           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      9654           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      7245           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      6896           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
httpd      5909           www   11w    VREG       0,92  361496571    32972 /var (/dev/ad0s1d)
syslogd    9972          root   19w    VREG       0,85  110849268 58880006 /data/var/log/all.log
httpd     91413          root   14w    VREG       0,92    4767735    32988 /var/log/httpd/access.log
httpd     91413          root   13w    VREG       0,92    4767735    32988 /var/log/httpd/access.log
httpd     10028           www   14w    VREG       0,92    4767735    32988 /var/log/httpd/access.log
httpd     10028           www   13w    VREG       0,92    4767735    32988 /var/log/httpd/access.log

[snip]

The first few are just fine, but what the hell is Apache doing with /var/log/httpd/access.log? It’s clearly not that big…

Anyway, a quick restart of Apache and /var returned to a far more reasonable size:

# df -h /var
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1d    496M    144M    312M    31%    /var

Panic over… for now.

Tags: bsd, compsoc

This entry was posted on Wednesday, February 7th, 2007 at 8:17 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.