Structured Logging: JSON Format, Context Fields and Log Aggregation

When I started handling production logs fifteen years ago, tailing a single syslog file on one server was enough to find most problems. That approach collapses the moment you run a dozen microservices across multiple hosts and need to answer a question like "show me all failed payments for user 123 in the last hour." Structured logging is the practice that makes that question answerable without regex acrobatics. Instead of writing human-friendly strings, you emit machine-readable JSON objects where every piece of information is a discrete field. In this article, I'll walk through how to design those JSON logs, how to add context fields correctly, and how to get them into a central aggregator where they become genuinely useful. If you are just getting started with centralized logging, this will give you a practical foundation you can apply this week without overhauling your entire stack.

Why Plain Text Logs Break Down When You Run More Than One Service

In my experience, the pain of unstructured logs doesn't show up in development. A line like 2024-03-12 10:15:33 INFO User login succeeded for joe@example.com from 192.168.1.10 is perfectly readable in your terminal. The problem starts when you try to aggregate thousands of those lines per second from different services and query them. You end up writing brittle regular expressions to extract the username, the IP address, and the timestamp, and every team formats their messages slightly differently. One service writes user=joe, another writes username: joe, a third writes User joe logged in. Your log aggregator sees three incompatible formats and you can't build a single dashboard or alert on "failed logins."

Plain text logs also force you to choose between readability and parsability. If you make the message more detailed for humans, you make it harder for machines to parse. If you keep it rigid for machines, developers stop reading it locally. Structured logging resolves that trade-off. A JSON log line is still readable with a formatter like jq, but it is primarily designed for machines. Every attribute is a key-value pair, so filtering becomes level="error" AND service="checkout" instead of grep "ERROR" | grep "checkout" | sed .... When you ship those logs to a system like Elasticsearch or Loki, those fields become indexed columns you can filter, group, and alert on instantly.

What Unstructured Logs Cost You During an Incident

During an outage, speed matters. I once spent 40 minutes during a checkout failure manually correlating Nginx access logs with application logs because the application only logged Payment failed: timeout without an order ID or user ID. We had the data in two different places, but no common field to join them. If that application had logged a JSON object containing order_id, user_id, payment_gateway, and duration_ms, a single query would have shown that all failures came from one gateway region. Structured logs give you that correlation for free, provided you include the right context from the start. The extra effort to emit JSON pays for itself the first time you don't have to SSH into five boxes to grep production logs.

Designing JSON Log Schemas That Survive Real Workloads

A good JSON schema is boring and consistent. I've seen teams treat JSON logging as just "stringify whatever object we have," which creates logs where field names change from line to line and types are unpredictable. That defeats the purpose. You want a small, stable set of top-level fields that every log line includes, plus a flexible way to add service-specific context. Think of your log schema like a database table: a few required columns and a JSON column for extras is better than a different table for every service.

Here is the baseline I recommend for every service, regardless of language. It balances usefulness with storage cost and stays compatible with most aggregators.

{
  "timestamp": "2024-03-12T10:15:33.421Z",
  "level": "info",
  "message": "user login succeeded",
  "service": "auth-service",
  "environment": "production",
  "version": "1.4.2",
  "request_id": "req_abc123xyz",
  "duration_ms": 42,
  "user_id": "user_789",
  "error_code": null
}

Notice a few decisions here. The timestamp is in ISO 8601 UTC with milliseconds, which is what Elasticsearch and Grafana Loki expect by default. The level field is lowercase and constrained to debug, info, warn, error, fatal - this lets you filter effectively without guessing. The message field remains a human-readable string, but it is short and never contains variable data that belongs in its own field. Don't put the user ID inside the message if you already have a user_id field.

The Core Fields Every JSON Log Should Contain

At a minimum, enforce these fields across all services: timestamp, level, message, service, environment, and request_id. The service and environment fields are critical for multi-tenant aggregators; without them, you can't tell if an error spike is in staging or production. The request_id or correlation ID is the single most valuable field for debugging distributed systems. It should be generated at the edge (load balancer or API gateway) and propagated through every downstream call. When every log line for a single user request shares the same ID, tracing a request across services becomes a simple query.

Naming Conventions That Save You Later

Pick one convention and enforce it with a linter. I strongly prefer snake_case for log fields because it works cleanly in Elasticsearch, BigQuery, and most query languages without needing quotes. Avoid mixing userId, user_id, and user-id across services. Also, keep field types stable. If duration_ms is a number in one service, it should never be a string like "42ms" in another. Inconsistent types cause mapping conflicts in Elasticsearch that are painful to fix. As described in the Kubernetes Documentation on cluster logging architecture, Kubernetes expects container logs on stdout to be JSON when you configure structured logging, and type mismatches there will break field extraction at the node level.

Injecting Context Fields Without Polluting Your Application Code

The biggest mistake I see beginners make is manually adding context to every log call: logger.info(f"user {user_id} did something on order {order_id}"). That is tedious and easy to forget. The better approach is to bind context once per request and let the logger include it automatically. Most modern logging libraries support this concept of a child logger or contextual logger.

For example, in Node.js with Pino, you create a child logger per request in your middleware. In Python, you can use structlog with contextvars. The goal is the same: request-scoped fields like request_id, user_id, and trace_id are attached without passing them through every function call.

// Node.js + Pino: bind context in Express middleware
const pino = require('pino');
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level(label) { return { level: label }; }
  },
  timestamp: pino.stdTimeFunctions.isoTime
});

function requestLogger(req, res, next) {
  // request_id comes from load balancer or is generated here
  const requestId = req.headers['x-request-id'] || crypto.randomUUID();
  // Create a child logger that always includes these fields
  req.log = logger.child({ 
    request_id: requestId, 
    service: 'checkout-api',
    environment: process.env.NODE_ENV,
    trace_id: req.headers['traceparent'] // from OpenTelemetry
  });
  res.setHeader('x-request-id', requestId);
  next();
}

// Later in a route handler - no need to pass context manually
app.post('/orders', requestLogger, async (req, res) => {
  req.log.info({ order_id: 'ord_123', amount: 49.99 }, 'order creation started');
  try {
    await createOrder(req.body);
    req.log.info({ order_id: 'ord_123', duration_ms: 87 }, 'order creation succeeded');
  } catch (err) {
    req.log.error({ err, order_id: 'ord_123' }, 'order creation failed');
  }
});

This pattern keeps your business logic clean. The handler doesn't need to know about request_id or trace_id; it just uses req.log and the fields appear automatically. I've found that teams adopt structured logging much faster when they don't have to remember to add five extra parameters to every log statement.

Propagating Request IDs and Trace IDs Across Services

Context is only useful if it follows the request. When your API calls another internal service, forward the x-request-id and traceparent headers. Most HTTP clients make this straightforward. If you are using OpenTelemetry, the W3C Trace Context header does this propagation for you. This connection between logs and traces is powerful. A log line with a trace_id lets you jump directly to the corresponding trace in a tool like Jaeger. For more on that workflow, see Distributed Tracing with Jaeger and Grafana Tempo: Setup and Best Practices, which covers how trace IDs unify logs and traces in one view.

Be careful about what you put in context. Log user IDs, request IDs, and business identifiers, but never log passwords, tokens, or full PII like credit card numbers. I recommend adding a filter in your logger to redact fields like password, authorization, and ssn before they are serialized. It is much easier to prevent sensitive data from being emitted than to scrub it from your aggregator later.

Shipping JSON Logs to Aggregators: Filebeat, Fluent Bit and the OpenTelemetry Collector

Writing JSON to stdout is only half the job. Those lines need to be collected from the host or container and shipped to a central system where you can query them. How you do that depends on your environment. On virtual machines, you typically write to a file and run an agent like Filebeat or Fluent Bit that tails it. In containers, the best practice is to log to stdout/stderr and let the container runtime handle collection, as recommended in the Docker Documentation for logging drivers.

For Docker, configure the json-file driver to avoid double-encoding and to handle rotation on the host. For Kubernetes, you don't configure anything inside the app; you write JSON to stdout and a DaemonSet collector like Fluent Bit or the OpenTelemetry Collector does the rest.

# Fluent Bit configuration for Kubernetes: parse Docker JSON and keep original JSON fields
[INPUT]
    Name              tail
    Path              /var/log/containers/*.log
    Parser            docker
    Tag               kube.*
    Refresh_Interval  5
    Mem_Buf_Limit     50MB

[FILTER]
    Name              kubernetes
    Match             kube.*
    Kube_URL          https://kubernetes.default.svc:443
    Merge_Log         On
    Keep_Log          Off
    K8S-Logging.Parser On

[FILTER]
    Name              parser
    Match             kube.*
    Key_Name          log
    Parser            json
    Reserve_Data      On

[OUTPUT]
    Name              es
    Match             kube.*
    Host              elasticsearch.logging.svc
    Port              9200
    Logstash_Format   On
    Logstash_Prefix   app-json-logs
    Replace_Dots      On

# Parser definition
[PARSER]
    Name        json
    Format      json
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

This Fluent Bit pipeline does three important things. First, it tails container logs from /var/log/containers/*.log. Second, it uses the Kubernetes filter to enrich logs with pod, namespace, and label metadata, which are invaluable as context fields. Third, it parses the log field as JSON and merges it with that metadata. The result in Elasticsearch is a document that contains both your application fields like request_id and infrastructure fields like k8s.pod.name. In my experience, Fluent Bit is lighter and simpler to operate than Logstash for this job, though Logstash offers more powerful transformation if you need it. A detailed comparison of those options is covered in ELK Stack Architecture: Elasticsearch, Logstash and Kibana at Scale.

An alternative that is quickly becoming my default for new clusters is the OpenTelemetry Collector. It can receive logs via the OTLP protocol, parse JSON, and export to multiple backends (Loki, Elasticsearch, S3) with one pipeline. If you are already using OpenTelemetry for traces and metrics, using its Collector for logs avoids running a separate agent. Just be sure to set memory limits correctly; an OOMKilled collector will cause log loss on that node.

Parsing, Storing and Querying JSON Logs Without Losing Your Mind

Once your JSON logs reach the aggregator, the work is not finished. How they are parsed and stored determines whether querying is fast or frustrating. The advantage of JSON is that no custom grok pattern is needed, but you still need to control field mappings and retention.

In Elasticsearch, JSON fields become indexed fields automatically if you use dynamic mapping, but that can backfire. I've seen clusters where a developer logged response_time: "123ms" as a string for a week, then changed it to a number, causing mapping conflicts and rejected documents. Define an index template that explicitly sets types for your core fields: timestamp as date, level as keyword, duration_ms as long, and message as text. For Loki, fields are handled differently; you extract JSON fields at query time with | json and only index a small set of labels like service and environment to keep cardinality low.

Querying JSON logs is where the investment pays off. Instead of regex, you write structured queries. In Kibana's KQL: service: "payment-service" and level: "error" and duration_ms > 1000 instantly shows slow errors. In Grafana Loki's LogQL: {service="payment-service"} | json | level="error" | duration_ms > 1000. You can aggregate, graph error rates, or create alerts on these fields without additional parsing.

Capability Plain Text Logs Structured JSON Logs
Filtering by field Requires regex / grok patterns, fragile Direct field filters: user_id="123"
Aggregation & dashboards Manual parsing before grouping Native group-by, stats on any field
Alerting precision High false positives from keyword matches Accurate alerts on numeric / keyword fields
Onboarding new service New parser needed per service format Zero parser change if schema is shared
Storage & indexing cost Lower per line, higher query cost Slightly larger lines, far lower query cost
Correlation across services Manual, time-based guessing Join on request_id / trace_id

The table shows why teams migrate even though JSON logs are slightly larger on disk. The text logs look smaller at first, but the operational cost of parsing and the time lost during incidents outweigh the bytes saved. You should also plan for that size. JSON can increase log volume by 20-40% compared to minimal text lines, which affects retention and cost. Tier your storage, set sensible retention, and sample debug logs in production. I cover storage trade-offs and compliance-driven retention in more detail in Log Rotation and Retention: Storage Optimization and Compliance Requirements.

Avoiding Mapping Explosions and Costly Mistakes

Two practical rules will save you trouble. First, limit high-cardinality fields. Don't put a unique message string or a full SQL query into an indexed field; that will blow up your index mapping. Put variable, high-cardinality content in the message field as text and keep indexed fields to low-cardinality keywords and numbers. Second, never log large objects or entire HTTP bodies as JSON fields. Log a hash, an ID, or a truncated snippet. I once debugged a cluster where a service logged the full 2MB JSON payload on every error, filling the Elasticsearch disk in hours. A simple size check in the logger would have prevented it.

Frequently Asked Questions

Should I log JSON in development or only in production?

Log JSON everywhere, but make it readable locally. In development, configure your logger to pretty-print JSON with colors using a tool like pino-pretty, jq, or structlog's console renderer. That way you get human-friendly output in your terminal while your code always emits the same JSON structure it will in production. Switching formats between environments hides bugs where a field is missing or malformed only in the production format.

How do I handle multi-line logs like stack traces in JSON?

Never emit a stack trace as multiple log lines. Most collectors treat each line as a separate event, so your trace will be split across many documents and become unsearchable. Instead, serialize the entire stack trace into a single string field, typically error.stack or exception, with newline characters escaped inside the JSON string. Most libraries do this automatically if you pass the error object as a field (e.g., logger.error({err: error})) rather than calling logger.error(error.stack).

Does structured JSON logging hurt application performance?

When done correctly, the impact is negligible. JSON serialization is fast, and asynchronous logging prevents your request handler from blocking on I/O. The performance problems I have seen come from logging too much, logging synchronously to disk or network, or serializing huge objects. Use async appenders, set the log level to info or warn in production, and avoid logging inside tight loops. Benchmark your logger choice; in Node.js, Pino is significantly faster than Winston or Bunyan for JSON output, and in Python, structlog with orjson is a solid fast option.

Can I convert existing plain text logs to JSON without changing application code?

Partly, but it is a workaround. You can use Fluent Bit or Logstash to parse plain text with regex and re-emit JSON, but that parsing is brittle and fails whenever the message format changes. It is useful as a migration step for legacy services you cannot modify. For any service you actively develop, instrument it to emit native JSON directly. The long-term maintainability and reliability of native JSON far outweigh the one-time code change, and you gain proper type handling for numbers and booleans that regex parsing cannot provide.

Related Articles

References & Standards: Kubernetes Documentation · Docker Documentation · Terraform Documentation