Why Log Volume Grows Faster Than Traffic
Log volume rarely grows linearly with user traffic. Debug-level logging left on in production, verbose third-party libraries, retry loops that log every attempt, and duplicate logging across a request's middleware chain all compound quietly until a routine deploy suddenly doubles your daily ingestion. Left unmanaged, this either inflates your storage bill (see our cost comparison post for how that plays out under usage-based pricing) or forces a reactive scramble to filter things out under pressure.
Techniques That Actually Preserve Visibility
Sample at the trace level, not the log level
Instead of randomly dropping individual log lines, sample entire traces. This way, if you keep a request, you keep every log line associated with it — no missing context.
Always keep errors and warnings, sample info/debug
Error-level logs are rare and high-value; sampling them defeats the purpose. Apply sampling selectively to high-volume, low-severity levels instead.
Deduplicate repeated messages
A retry loop that logs "connection failed" 500 times in a minute is one event, not 500. Collapse repeats into a single entry with a count, at the collector level.
Structure logs instead of writing prose
A structured log with fields (service, status, latency_ms) compresses far better than a free-text sentence carrying the same information, and is cheaper to query later.
Where in the Pipeline to Apply These
Filtering at the application layer (only log what you need) is the cheapest option but requires discipline across every service. Filtering at the OpenTelemetry Collector — using processors to drop, sample, or deduplicate before export — centralizes the policy in one place and doesn't require touching application code. See our Collector configuration guide for where sampling processors fit into a pipeline.
What Not to Cut
Resist the temptation to drop logs purely because they're verbose — verbosity and value aren't the same thing. A payment service's transaction logs might be high-volume and still non-negotiable for compliance or dispute resolution. The right question isn't "how do we log less," it's "how do we make sure every byte we keep is worth its storage cost" — which usually means cutting duplication and noise rather than cutting categories of information wholesale.
Conclusion
Reducing log volume and preserving visibility aren't in tension if you target the right things: duplication, unstructured text, and low-value debug noise. Cut those, and volume drops significantly without losing the information that actually gets used during an incident.
Store Logs Efficiently by Default
XplurData's Apache Doris backend achieves ~8:1 average compression on OTel logs out of the box.
Deploy XplurData