How to use logrotate to manage log files in Linux

Log files contain useful information about what is going on within the system. They are often inspected during troubleshooting processes or as part of server performance analysis. For a busy server, log files may grow quickly into very large sizes. This becomes a problem as the server will soon run out of space. Besides, opening and inspecting a single large log file can often be tricky.

logrotate is a very useful tool that can automate the process of breaking up (or rotating), compressing, and deleting old log files. For example, you can set up logrotate such that the log file /var/log/foo is rotated every 30 days, and logs older than 6 months are deleted. Once configured, the process is fully automated using logrotate without any further need for human interaction. Optionally, old logs can be emailed as well, but that option is beyond the scope of this tutorial.

The logrotate package is typically installed by default on major Linux distros. If, for some reason, logrotate is not present, you can install it using apt-get or yum command.

On Debian or Ubuntu:

# apt-get install logrotate cron

On Fedora, CentOS or RHEL:

# yum install logrotate crontabs

The configuration file for logrotate is /etc/logrotate.conf. Generally no modification is needed here. The log files to be rotated are defined in separate configuration file(s) placed under /etc/logrotate.d/ directory.

Example One

In the first example, we will create a 10 MB log file /var/log/log-file. We will see how we can use logrotate to manage this log file.

We start by creating a log file, and populating it with a 10 MB worth of random bit stream.

# touch /var/log/log-file
# head -c 10M < /dev/urandom > /var/log/log-file

Now that the log file is ready, we will configure logrotate to rotate this log file. Let's create a configuration file for this.

# vim /etc/logrotate.d/log-file
/var/log/log-file {
    monthly
    rotate 5
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

Where:

  • monthly: The log file will now be rotated monthly. Other possible values are 'daily', 'weekly' or 'yearly'.
  • rotate 5: A total of 5 archived logs will be stored at a time. For the 6th archive, the oldest stored archive will be deleted.
  • compress: The rotated archive will be compressed using gzip, after the rotation task is complete.
  • delaycompress: Always used together with compress option, the delaycompress parameter instructs logrotate to not run compression on the most recent archive. Compression will be performed during the next rotation cycle. This is useful if you or any software still needs to access the fresh archive.
  • missingok: During log rotation, any errors will be ignored, e.g., "file not found".
  • notifempty: Rotation will not be performed if the log file is empty.
  • create 644 root root: A fresh log file will be created with specified permissions as logrotate may rename the original log file.
  • postrotate/endscript: The command(s) specified between postrotate and endscript will be carried out after all other instructions are completed. In this case, the process rsyslogd will re-read its configuration on the fly and continue running.

The above template is generic, and the configuration parameters may vary based on your requirements. Not all the parameters may be necessary.

Example Two

In this example, we want to rotate a log file only when the size of the log file grows over 50 MB.

# vim /etc/logrotate.d/log-file
/var/log/log-file {
    size=50M
    rotate 5
    create 644 root root
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

Example Three

We want old log files to be named with the date of creation. This can be achieved by adding dateext parameter.

# vim /etc/logrotate.d/log-file
/var/log/log-file {
    monthly
    rotate 5
    dateext
    create 644 root root
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

This will cause the archived files to contain the date in their name.

Troubleshooting

Here are a few troubleshooting tips for logrotate setup.

1. Running logrotate manually

logrotate can be invoked manually from the command line at any time.

To invoke logrotate on all logs as configured in /etc/logrotate.d/*:

# logrotate /etc/logrotate.conf

To invoke logrotate for a particular configuration:

# logrotate /etc/logrotate.d/log-file

2. Dry run

The best option during troubleshooting is to run logrotate as a dry run using '-d' option. For verification, a dry run simulates log rotation and displays its output without actually rotating any log files.

# logrotate -d /etc/logrotate.d/log-file

As we can see from the above output, logrotate decided that rotation is not necessary. This can happen if the age of the file is less than one day.

3. Force run

We can force logrotate to rotate log files even when rotation conditions are not met, by using '-f' option. The '-v' parameter provides verbose output.

# logrotate -vf /etc/logrotate.d/log-file
reading config file /etc/logrotate.d/log-file
reading config info for /var/log/log-file

Handling 1 logs

rotating pattern: /var/log/log-file  forced from command line (5 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/log-file
  log needs rotating
rotating log /var/log/log-file, log->rotateCount is 5
dateext suffix '-20140916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/log/log-file.5.gz to /var/log/log-file.6.gz (rotatecount 5, logstart 1, i 5),
old log /var/log/log-file.5.gz does not exist
renaming /var/log/log-file.4.gz to /var/log/log-file.5.gz (rotatecount 5, logstart 1, i 4),
old log /var/log/log-file.4.gz does not exist
. . .
renaming /var/log/log-file.0.gz to /var/log/log-file.1.gz (rotatecount 5, logstart 1, i 0),
old log /var/log/log-file.0.gz does not exist
log /var/log/log-file.6.gz doesn't exist -- won't try to dispose of it
renaming /var/log/log-file to /var/log/log-file.1
creating new /var/log/log-file mode = 0644 uid = 0 gid = 0
running postrotate script
compressing log with: /bin/gzip

4. Logrotate logging

Logs for logrotate itself are usually stored in the directory /var/lib/logrotate/status. If we want logrotate to log to any specific file for troubleshooting purposes, we can specify that from the command line as follows.

# logrotate -vf –s /var/log/logrotate-status /etc/logrotate.d/log-file

5. Logrotate cron job

The cron jobs needed for logrotate should automatically be created during installation. I am posting the contents of the cron file for reference.

# cat /etc/cron.daily/logrotate
#!/bin/sh

# Clean non existent log file entries from status file
cd /var/lib/logrotate
test -e status || touch status
head -1 status > status.clean
sed 's/"//g' status | while read logfile date
do
    [ -e "$logfile" ] && echo "\"$logfile\" $date"
done >> status.clean
mv status.clean status

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

To sum up, logrotate is a very useful tool for preventing gigantic log files from using up storage space. Once configured, the process is fully automated, and can run without human intervention for a long time. This tutorial focused on several basic examples of how to use logrotate. You can customize it even further to match your requirements.

Hope this helps.

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.


Support Xmodulo

Did you find this tutorial helpful? Then please be generous and support Xmodulo!

The following two tabs change content below.
Sarmed Rahman is an IT professional based in Australia. He writes tutorial articles on technology every now and then from a belief that knowledge grows through sharing. During his free time, he loves gaming and spending time with his friends.

10 thoughts on “How to use logrotate to manage log files in Linux

  1. Hello, how do I rotate log files with dynamic name including date in it, for example some apache logs with non static file name. I know that logrotate does not support wildcard.
    Thanks

    • Hi,

      I have checked one of my Ubuntu server, and looks like logrotate already has something that meet your requirements. You could use it as a template and test on your required logs.

      # cat /etc/logrotate.d/apache2
      /var/log/apache2/*.log {
      weekly
      missingok
      rotate 52
      compress
      delaycompress
      notifempty
      create 640 root adm
      sharedscripts
      postrotate
      /etc/init.d/apache2 reload > /dev/null
      endscript
      prerotate
      if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
      run-parts /etc/logrotate.d/httpd-prerotate; \
      fi; \
      endscript
      }

  2. How about non-system logs produced by applications that are not stored in /var/log and that are owned by some application login? It would be very useful to make use of logrotate for these too. I briefly looked into this but could not see a way of using logrotate.
    $man logrotate
    logrotate ‐ rotates, compresses, and mails system logs

    System logs! is it really only system logs? What a shame that useful logrotate function is locked away.

  3. Actually there seems to be a relevant option:
    su user group
    Rotate log files set under this user and group instead of using
    default user/group (usually root). user specifies the user name
    used for rotation and group specifies the group used for rota‐
    tion.
    So I suppose if we touch a file in the /var/log directory and change ownership to some non system login, then we can use this option and get logrotate working for non-system logs. Not tested yet.

  4. I did some tests running under a login called "user":
    Here is the log file:
    -rw-rw-r-- 1 user root 10485760 Sep 20 18:18 /var/log/log-file
    Here is the log file rotate config:
    /var/log/log-file {
    size=10M
    compress
    create 644 user root
    su user root
    }
    Here is a force run:
    ##############################################################
    ## Run force: logrotate -vf /etc/logrotate.d/log-file ... ##
    ##############################################################
    reading config file /etc/logrotate.d/log-file

    Handling 1 logs

    rotating pattern: /var/log/log-file forced from command line (no old logs will be kept)
    empty log files are not rotated, old logs are removed
    switching euid to 1000 and egid to 0
    considering log /var/log/log-file
    log needs rotating
    rotating log /var/log/log-file, log->rotateCount is 0
    dateext suffix '-20140920'
    glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
    glob finding logs to compress failed
    glob finding old rotated logs failed
    renaming /var/log/log-file to /var/log/log-file-20140920
    error: failed to rename /var/log/log-file to /var/log/log-file-20140920: Permission denied
    disposeName will be /var/log/log-file-20140920.gz
    switching euid to 0 and egid to 0

    See the 'Permission denied' ?

    • I have tested using an Ubuntu server, and the following works file. What distro are you using?
      ###############################################################
      ls -l /opt/
      drwxr-xr-x 2 user root 4096 Sep 22 17:52 test
      ###############################################################
      cat /etc/logrotate.d/log-test
      /opt/test/log-test {
      size=10M
      compress
      rotate 5
      create 644 user root
      su user root
      postrotate
      /usr/bin/killall -HUP rsyslogd
      endscript
      }
      ###########################################################################
      logrotate -vf /etc/logrotate.d/log-test
      reading config file /etc/logrotate.d/log-test
      reading config info for /opt/test/log-test
      error: /etc/logrotate.d/log-test:6 unknown option 'su' -- ignoring line
      error: /etc/logrotate.d/log-test:6 unexpected text

      Handling 1 logs

      rotating pattern: /opt/test/log-test forced from command line (5 rotations)
      empty log files are rotated, old logs are removed
      considering log /opt/test/log-test
      log needs rotating
      rotating log /opt/test/log-test, log->rotateCount is 5
      dateext suffix '-20140922'
      glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
      renaming /opt/test/log-test.5.gz to /opt/test/log-test.6.gz (rotatecount 5, logstart 1, i 5),
      old log /opt/test/log-test.5.gz does not exist
      renaming /opt/test/log-test.4.gz to /opt/test/log-test.5.gz (rotatecount 5, logstart 1, i 4),
      old log /opt/test/log-test.4.gz does not exist
      renaming /opt/test/log-test.3.gz to /opt/test/log-test.4.gz (rotatecount 5, logstart 1, i 3),
      old log /opt/test/log-test.3.gz does not exist
      renaming /opt/test/log-test.2.gz to /opt/test/log-test.3.gz (rotatecount 5, logstart 1, i 2),
      renaming /opt/test/log-test.1.gz to /opt/test/log-test.2.gz (rotatecount 5, logstart 1, i 1),
      renaming /opt/test/log-test.0.gz to /opt/test/log-test.1.gz (rotatecount 5, logstart 1, i 0),
      old log /opt/test/log-test.0.gz does not exist
      log /opt/test/log-test.6.gz doesn't exist -- won't try to dispose of it
      renaming /opt/test/log-test to /opt/test/log-test.1
      creating new /opt/test/log-test mode = 0644 uid = 1002 gid = 0
      running postrotate script
      compressing log with: /bin/gzip

      Hope this helps.

Leave a comment

Your email address will not be published. Required fields are marked *