Structured Logging with JSON
Overview
Structured logging means emitting logs as machine-readable data structures (like `JSON`) rather than plain text strings. Instead of logging a raw string: logger.error("User 123 failed login from 192.168.1.1") You emit a dictionary of queryable data: logger.error("login_failed", extra={"user_id": 123, "ip": "192.168.1.1"}) Think of it like filling out a spreadsheet with specific columns instead of writing a free-form diary entry; machines can easily filter and `aggregate` the spreadsheet. When building structured logs, you must avoid key collisions with Python's built-in `LogRecord` attributes. Passing a reserved key in your `extra` dictionary will raise an exception at runtime. # Raises KeyError: "Attempt to overwrite 'name' in LogRecord" logger.info("Server started", extra={"name": "auth-service"})
Plain text logs require brittle regular expressions (`regex`) to parse in log aggregation tools (like `Datadog` or `Splunk`). `JSON` logs are instantly parsed into queryable fields, making production debugging exponentially faster.
Where used: FastAPI, CloudWatch, Microservices
Why learn this
- Prepare your app for professional log aggregators (`ELK`, `Datadog`)
- Stop writing brittle `regex` parsing rules for plain text logs
- Attach request IDs or user IDs to every log line cleanly
Code walkthrough
import logging
import json
import sys
class CustomJSONFormatter(logging.Formatter):
def format(self, record):
return json.dumps({
'lvl': record.levelname,
'msg': record.getMessage()
})
logger = logging.getLogger('app')
logger.setLevel(logging.WARNING)
h = logging.StreamHandler(sys.stdout)
h.setFormatter(CustomJSONFormatter())
logger.addHandler(h)
logger.warning('Disk low')Focus: return json.dumps({
Common mistakes
- String interpolation in the logger: If you do `logger.info(f"User {user_id} logged in")`, the structured logger receives a single string. To pass structured data, you must pass it as an `extra` dictionary: logger.info("User logged in", extra={"user_id": user_id})
- Non-serializable objects: Passing Python objects like `datetime` or custom class instances in the `extra` dict without converting them will crash the `JSON` encoder.
Glossary
- machine-readable
- Data formatted in a specific way so that computers and software can easily process and understand it. Example: `{"event": "login", "user_id": 42}`.
- aggregate
- To collect and combine many separate pieces of data into a single, summarized view or total. Example: `aggregate(logs)`.
Recall questions
- What is the primary advantage of structured logging over plain text logging?
- How do you pass structured data fields when calling a standard Python logger?
- What happens if you pass a key like `name` or `msg` in the `extra` dictionary when logging?
Understanding checks
What is printed to the console?
{"level": "INFO", "msg": "Login", "user": 42}
The `JSONFormatter` extracts `levelname` and `msg` from the `LogRecord`, as well as any custom attributes passed via `extra` (which Python attaches to the record), and returns a `JSON` string.
Why does this log `{"msg": "Hello %s"}` instead of `{"msg": "Hello World"}`?
It accesses `record.msg` (the raw string with '%s') instead of calling `record.getMessage()`, which performs the string interpolation.
The `LogRecord` stores the raw message template and arguments separately. You must call `getMessage()` to safely merge them before `JSON` encoding.
Practice tasks
Add request ID to JSON logs
Given a working `JSON` formatter, modify the `format` method so that it also includes a `request_id` field in the `JSON` output, but ONLY if `req_id` was passed in the logger's `extra` dictionary. (Hint: use `hasattr(record, 'req_id')`).
Continue learning
Previous: Loggers, Handlers, and Formatters
Previous: Dictionaries: operations and use cases