Why Tiering Beats a Single Flat Retention Window
The simplest retention policy is "keep everything for N days on the same storage, then delete it." This is easy to reason about but wastes money: data from an hour ago and data from three weeks ago are queried at wildly different frequencies, yet a flat policy stores both identically. Tiering matches storage cost and query performance to actual access patterns — recent data gets fast, more expensive storage; older data moves to cheaper storage as the odds of anyone querying it drop.
Hot, Warm, and Cold, Defined
Hot tier — hours to a few days
The window where active incidents are being investigated. Needs the fastest possible query performance, typically on local SSD storage with data fully indexed for both full-text and structured search.
Warm tier — days to a few weeks
Data still queried reasonably often — post-incident reviews, week-over-week comparisons — but rarely under real-time pressure. Can tolerate somewhat slower storage in exchange for lower cost per byte.
Cold tier — weeks to months or longer
Mostly held for compliance, audit, or rare historical investigation. Object storage (cheap, slower to query) is usually the right fit, sometimes with data compacted or aggregated rather than kept at full granularity.
How This Maps Onto a Doris-Backed Stack
Apache Doris supports partitioning data by time range and assigning different storage mediums (local SSD vs remote object storage) per partition, plus configurable Time-To-Live (TTL) per partition to automatically age data out. In practice, this means the hot tier lives on fast local storage, and a scheduled job (or Doris's own cooldown policies) moves older partitions to cheaper storage without any manual data migration — the query interface stays the same regardless of which tier the data physically sits in.
Setting Tier Boundaries for Your Team
There's no universal cutoff for where "hot" ends and "warm" begins — it depends on your incident review cadence and compliance requirements. A reasonable starting point is sizing the hot tier to your longest typical incident investigation window (often 3-7 days), warm tier to your sprint or monthly review cycle, and cold tier to whatever your compliance or audit retention mandate requires. Revisit these boundaries once you have real query pattern data rather than guessing indefinitely — most teams overestimate how often they query month-old logs at full detail.
Conclusion
Retention doesn't have to be a binary "keep it fast forever" or "delete it" decision. Tiering by age lets you keep the data you need for compliance or rare investigation without paying hot-tier storage and compute costs for all of it. Combined with the volume-reduction techniques in How to Reduce Log Volume Without Losing Visibility, tiering is one of the more effective levers for keeping observability costs proportional to actual value.
Tier Your Retention on Apache Doris
XplurData's Doris backend supports partition-level storage tiering and TTL policies out of the box.
Deploy XplurData