how do i rotate my apache logs

[anyweb]

hi all,

 

i'm looking for some guidance as to the best way to rotate my apache logs, they are now large....

 

i want to retain the data as much as possible,

 

any advice ?

 

Code:

-rw-r--r--   1 root   root           0 Jan 10 21:34 access_bad.log
-rw-r--r--   1 root   root   434839497 Jan 15 12:00 access_log
-rw-r--r--   1 root   root   130760500 Jan 15 11:59 agent_log
-rw-r--r--   1 root   root    11575178 Jan 15 11:59 error_log
-rw-r--r--   1 root   root           6 Jan 13 21:37 httpd.pid
-rw-r--r--   1 root   root   149330258 Jan 15 11:59 referer_log

 

cheers

anyweb

[znx]

First off, there are replacements to the basic syslog (or syslog-ng) that have builtin logrotating skills but anyway... its easy to do.

 

First install "logrotate", this is the utility for rotating logs.

 

Example /etc/logrotate.conf

Code:

# rotate log files weekly
weekly

# keep 4 rotations (4 weeks in this case)
rotate 4

# create new log files
create

# compress the files after rotation
compress

# packages can have their own personal files (i like this)
include /etc/logrotate.d

# dont rotate if empty, dont mail, dont move rotated into another dir
notifempty
nomail
noolddir

 

Ok so thats the default for packages, now lets create a file for apache /etc/logrotate.d/apache

Code:

/var/log/apache/*log {
missingok
notifempty
sharedscripts
postrotate
/etc/init.d/apache reload > /dev/null 2>&1 || true
endscript
}

 

Then make a cron entry /etc/cron.daily/logrotate.cron (in daily in case you switch to daily rotation and forget the cron like i do ;))

Code:

#! /bin/sh

/usr/sbin/logrotate /etc/logrotate.conf

 

You will need to touch up on the paths for your system but should be it. :) weee

[anyweb]

ok thanks znx

 

done,

 

now what i'm seeing is this

 

[anyweb@www ~]$ ls -al /usr/local/apache/logs/total 762196

drwxr-xr-x 2 anyweb anyweb 4096 Jan 22 04:26 .

drwxr-xr-x 15 anyweb anyweb 4096 Apr 3 2005 ..

-rw-r--r-- 1 root root 0 Jan 10 21:34 access_bad.log

-rw-r--r-- 1 root root 0 Jan 22 04:26 access_log

-rw-r--r-- 1 root root 487006197 Jan 25 15:05 access_log.1

-rw-r--r-- 1 root root 0 Jan 22 04:26 agent_log

-rw-r--r-- 1 root root 130760500 Jan 15 11:59 agent_log.1

-rw-r--r-- 1 root root 0 Jan 22 04:26 error_log

-rw-r--r-- 1 root root 12588341 Jan 25 15:04 error_log.1

-rw-r--r-- 1 root root 6 Jan 13 21:37 httpd.pid

-rw-r--r-- 1 root root 0 Jan 22 04:26 referer_log

-rw-r--r-- 1 root root 149330258 Jan 15 11:59 referer_log.1

 

does that look correct to you ?

 

cheers

 

anyweb

[znx]

Er yes .. but only if it was instantly after the rotation, if not it looks like your apache isn't logging again, this is an issue with

 

1) your httpd.conf either moving the logs elsewhere or not logging at all

2) you failed to reload apache on the log rotation

 

Also I note that you chose not to compress (which is possibly a good idea, as those logs will take a year to compress ;))

 

If 1) mod your apache conf and restart it.

If 2) restart apache manually to get it logging again and look at the line just after "postrotate".

 

Hope this helps

[anyweb]

ok

 

heres the log part of httpd.conf

 

look ok ?

 

cheers

 

anyweb

 

# Possible values include: debug, info, notice, warn, error, crit,# alert, emerg.

#

LogLevel warn

 

#

# The following directives define some format nicknames for use with

# a CustomLog directive (see below).

#

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

LogFormat "%h %l %u %t \"%r\" %>s %b" common

LogFormat "%{Referer}i -> %U" referer

LogFormat "%{User-agent}i" agent

 

#

# The location and format of the access logfile (Common Logfile Format).

# If you do not define any access logfiles within a <VirtualHost>

# container, they will be logged here. Contrariwise, if you *do*

# define per-<VirtualHost> access logfiles, transactions will be

# logged therein and *not* in this file.

#

#CustomLog /usr/local/apache/logs/access_log common

 

#

# If you would like to have agent and referer logfiles, uncomment the

# following directives.

#

#CustomLog /usr/local/apache/logs/referer_log referer

#CustomLog /usr/local/apache/logs/agent_log agent

 

#

# If you prefer a single logfile with access, agent, and referer information# (Combined Logfile Format) you can use the following directive.

#

 

#CustomLog /usr/local/apache/logs/access_log combined

CustomLog /usr/local/apache/logs/access_log combined env=!BAD

CustomLog /usr/local/apache/logs/access_bad.log combined env=BAD

after restarting apache i now get 

-rw-r--r-- 1 root root 17368 Jan 25 15:31 access_log-rw-r--r-- 1 root root 487170458 Jan 25 15:31 access_log.1

-rw-r--r-- 1 root root 0 Jan 22 04:26 agent_log

-rw-r--r-- 1 root root 130760500 Jan 15 11:59 agent_log.1

-rw-r--r-- 1 root root 161 Jan 25 15:31 error_log

-rw-r--r-- 1 root root 12589458 Jan 25 15:31 error_log.1

-rw-r--r-- 1 root root 6 Jan 25 15:31 httpd.pid

-rw-r--r-- 1 root root 0 Jan 22 04:26 referer_log

 

i think that means its working now,

 

but why didnt it work before ?

[znx]

i think that means its working now, 

but why didnt it work before ?

 

That means that the postrotate failed, what did you use for the postrotate?

 

For instance, if you never start apache with an initscript, what I have above wouldn't work, if you use apachectl then thats the line you want.

 

Basically you need to reload (or restart if you wish but its not needed) apache once you "steal" its log files with logrotate.

 

Ensure that the postrotate line actually works :)

[anyweb]

ok znx i'll check it

 

heres the code

 

/usr/local/apache/logs/*log { missingok

notifempty

sharedscripts

postrotate

/usr/local/apache/bin/apachectl reload > /dev/null 2>&1 || true

endscript

}

 

testing...

 

Code:

[root@www longhorn]#  /usr/local/apache/bin/apachectl reload > /dev/null 2>&1 || true

 

cheers

anyweb

[znx]

Via some live IRC stuff :P

 

snip'd btw

Code:

<anyweb> [root@www longhorn]#  /usr/local/apache/bin/apachectl reload > /dev/null 2>&1 || true
<znx> /usr/local/apache/bin/apachectl reload     try just that
<anyweb> [root@www longhorn]# /usr/local/apache/bin/apachectl reload
<anyweb> usage: /usr/local/apache/bin/apachectl (start|stop|restart|fullstatus|status|graceful|configtest|help)
<anyweb> lol
<znx> so its not working :) hehe
<anyweb> okies
<anyweb> /usr/local/apache/bin/apachectl restart > /dev/null 2>&1 || true
<anyweb> got it
<znx> yup

 

ain't irc great :)

 

Thus:

Code:

/usr/local/apache/logs/*log {
missingok
notifempty
sharedscripts
postrotate
/usr/local/apache/bin/apachectl restart > /dev/null 2>&1 || true
endscript
}

[anyweb]

sorted thanks again znx :)

[anyweb]

yet another problem

 

after upgrading to fcr5 on the webserver, and adding logrotate again

 

i tried to manually run webalizer today and get 'no valid records found'

 

Code:

[root@www ~]# /usr/bin/webalizer
No valid records found!

 

 

[root@www ~]# cd /usr/local/apache/logs/

here's the contents of the logs dir

 

[root@www logs]# ls -altotal 164380

drwxr-xr-x 2 root root 4096 Oct 29 04:05 .

drwxr-xr-x 14 root root 4096 Jan 1 1970 ..

-rw-r--r-- 1 root root 0 Oct 29 04:05 access_log

-rw-r--r-- 1 root root 98791114 Nov 1 07:30 access_log.1

-rw-r--r-- 1 root root 0 Oct 29 04:05 agent_log

-rw-r--r-- 1 root root 33517582 Nov 1 07:30 agent_log.1

-rw-r--r-- 1 root root 0 Oct 29 04:05 error_log

-rw-r--r-- 1 root root 1396758 Nov 1 07:30 error_log.1

-rw-r--r-- 1 root root 5 Oct 27 09:44 httpd.pid

-rw-r--r-- 1 root root 0 Oct 29 04:05 referer_log

-rw-r--r-- 1 root root 34414268 Nov 1 07:30 referer_log.1

 

as i can see it's rotating the logs alright but it's not looking right at all, hence the webalizer failure

 

the webalizer.conf file seems to be looking at one file only so how do i get it to work with rotated log files

 

# LogFile defines the web server log file to use. If not specified# here or on on the command line, input will default to STDIN. If

# the log filename ends in '.gz' (ie: a gzip compressed file), it will

# be decompressed on the fly as it is being read.

 

LogFile /usr/local/apache/logs/access_log

any ideas ?

 

cheers

anyweb