Rotating Linux Log Files

Rotating Linux Log Files

Logrotate is a great utility to manage and administer log files in a system. It provides you with a lot of options, namely, automatic rotation, compression and mailing of log files. Each log file can be rotated hourly, daily, weekly or monthly. In this article, I will show you how to manage your log files using logrotate.

Any type of logs can be managed using logrotate. It runs as a daily cron job. These days most of the operating systems come with logrotate pre-installed so you probably won't have to deal with the installation. Lets start with looking at the files and directories used by logrotate.

Files and Directories used by Logrotate

The 2 important files in case of logrotate are

/etc/logrotate.conf

/var/lib/logrotate.status    or    /var/lib/logrotate/status

The first file is where the whole configuration and default options of logrotate is stored. The second file contains the information about the last rotation times of different log files.

The directory thats usually used by logrotate is

/etc/logrotate.d

The options specified in /etc/logrotate.conf are used as default for rotating and managing log files. But if we want to specify separate settings for a set of log files then we can add their config files in this directory /etc/logrotate.d. Although we can even use another directory to save config files by changing an option in /etc/logrotate.conf.

Now, enough of the theory let’s see how these config files really look and what all the different options in it mean.

Logrotate Options

Let me first show you some commonly used options of logrotate and then we will see how to configure various options in the config file.

rotate <num>	Log files are rotated "num" times before getting deleted or mailed.
daily	When used this means log files should be rotated daily
weekly	Log files should be rotated weekly
monthly	Log files should be rotated monthly
notifempty	Don't rotate the log if its empty
compress	Compress the files after rotating them
delaycompress	This options means to delay the compression of a log file to the next rotation cycle. This is used in combination with compress.
missingok	If the log file is missing, go on to the next one without issuing an error message.
create <mode> <owner> <group>	After rotation create a new empty file with the following properties. e.g create 640 root adm. Just "create" will ensure to inherit the properties of previous log files.

Logrotate config files

As stated above, /etc/logrotate.conf is where the default options to handle logs are specified. A typical log file will look something like this.

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# Packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

The options mentioned in the config file above means that the logs are going to be rotated "weekly" and only 4 copies of backlogs will be maintained. On every rotation "create" a new empty file with properties inherited from the previous logs. Compression of backlogs is disabled.

The "include" directive is used to mention the directory in which the package specific log config files will be saved.

Managing Package specific log files in /etc/logrotate.d

Now, if there is an application which generates logs how would it be able to use logrotate to control and manage its logs. This is all done using its /etc/logrotate.d directory. This is where applications drop the information required by logrotate to manage their logs. e.g consider the example of apache logs. My apache logrotate config file looks something like this (its /etc/logrotate.d/apache2).

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 5
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
    endscript
}

Now, cross checking  the options from the table mentioned above the following properties of the logs can be inferred.

·         It’s going to be a weekly rotation

·         5 backlogs will be maintained in compressed format. But since we are using 'delaycompress' as well, the compression of a backlog will take place in the next rotation.

·         No rotation will take place if file is empty.

·         No error in case no log is found.

·         On every rotation a new file will be created with owner root, group adm and permission 640

We haven't discussed anything about 'sharedscripts' or 'postrotate' yet. Let’s begin with prerotate/endscript and postrotate/endscript.

Prerotate and Postrotate

Both of these options control the scripts that will need to be run either before or after rotating a log file. Let’s take the above example.

postrotate
        if [ -f "`. /etc/apache2/envvars ; echo ${APACHE_PID_FILE:-/var/run/apache2.pid}`" ]; then
            /etc/init.d/apache2 reload > /dev/null
        fi
endscript

'endscript' is mandatory to be mentioned in both prerotate and postrotate to specify the ending of the script.

You would have also noticed the sharedscripts option in the apache2 logrotate config file. Now, if sharedscripts options is not specified, prerotate and postrotate will run the scripts on each log file which is rotated. But if you look carefully at the file logs mentioned in the apache2 logrotate conf file, its a wild card /var/log/apache2/*.log i.e. not just a single file but a bunch of files matching a pattern. These files in our case will be /var/log/apache2/access.log and error.log. So, for every rotation we will have 2 log files.

But look at the script that we want to run after rotation. It reloads the apache server and there is probably no need to run this script twice. So, by using sharedscripts we are ensuring that the script doesn't run on every rotated log file but only once.

I hope this howto will be helpful for a lot of you. If you have any doubts or want something to be corrected here, feel free to comment.