Log Rotation with RotatingFileHandler
Overview
Log rotation is the practice of automatically creating a new log file when the current one gets too large or too old, renaming the old files (e.g., `app.log.1`, `app.log.2`), and deleting the oldest ones. In Python, this is done using `RotatingFileHandler` (by size) or `TimedRotatingFileHandler` (by time). Think of it like a security camera system that only keeps the last 7 days of footage, overwriting the oldest tapes so the hard drive never fills up.
If a long-running application writes to a single file forever, that file will eventually consume all available disk space and crash the server.
Where used: Linux Servers, Docker Volumes, Background Workers
Why learn this
- Prevent servers from crashing due to `Disk Out Of Space` errors
- Keep log files small enough to quickly open in a text editor
- Comply with data retention policies by automatically deleting old logs
Code walkthrough
import logging
from logging.handlers import RotatingFileHandler
import os
logger = logging.getLogger('demo')
logger.setLevel(logging.INFO)
handler = RotatingFileHandler('app.log', maxBytes=50, backupCount=2)
logger.addHandler(handler)
for i in range(10):
logger.info(f'Message {i}')
print([f for f in sorted(os.listdir('.')) if 'app.log' in f])Focus: handler = RotatingFileHandler('app.log', maxBytes=50, backupCount=2)
Common mistakes
- Using standard FileHandler in production: `logging.FileHandler` never deletes old data. Always use a rotating handler for services that run continuously.
- Multiple processes rotating the same file: If running multiple worker processes (like Gunicorn), standard rotating handlers will clash and corrupt the file. You must use a specialized concurrent handler or let the OS (like logrotate or Docker) handle it.
Glossary
- data retention policies
- Rules set by an organization that dictate how long specific types of data, like logs, must be kept before being deleted, e.g. keeping logs for 30 days then auto-deleting them.
- worker processes
- Separate instances of a program running simultaneously to handle multiple tasks or requests at the same time, e.g. running `gunicorn -w 4 app:app` starts 4 worker processes.
Recall questions
- What happens if a long-running script uses a standard `FileHandler` instead of a rotating handler?
- What is the primary difference between `RotatingFileHandler` and `TimedRotatingFileHandler`?
Understanding checks
What does this print, and why?
True
The maxBytes is 20. The first message writes ~15 bytes. The second message makes the file exceed 20 bytes, triggering a rollover, which moves the current `test.log` to `test.log.1`.
If you configure `backupCount=3`, how many log files will eventually exist on disk?
4 log files: the currently active `app.log` plus 3 numbered backups (`app.log.1`, `app.log.2`, `app.log.3`).
It keeps the current active file (`app.log`) plus exactly 3 backups (`app.log.1`, `app.log.2`, `app.log.3`).
Practice tasks
Configure time-based rotation
Given a script that logs to a standard file, modify it to use `TimedRotatingFileHandler` from `logging.handlers`. Configure it to rotate every 1 'midnight' (`when='midnight'`) and keep only the last 7 days (`backupCount=7`).
Continue learning
Previous: Loggers, Handlers, and Formatters