Setting Up Native OpenTelemetry Log and Trace Storage with Zero Cost

Deploy Enterprise Observability Without Enterprise Licensing

Most observability platforms start simple but quickly become expensive as your telemetry volume grows. Licensing models based on hosts, ingestion rates, or indexed data often make long-term retention and organization-wide adoption difficult.

XplurData takes a different approach by combining OpenTelemetry with Apache Doris, giving engineering teams complete ownership of their observability platform while avoiding vendor lock-in.

Instead of maintaining multiple databases for logs, traces, and analytics, all telemetry is stored inside Apache Doris where it can be searched, filtered, aggregated, and correlated using standard SQL.

Platform Architecture

Unlike traditional observability stacks that require separate databases for logs, metrics and traces, XplurData stores operational data inside Apache Doris while OpenTelemetry Collector acts as the unified ingestion layer.

Why This Architecture Works

Every component has a clearly defined responsibility.

• Applications generate telemetry using OpenTelemetry SDKs.
• OpenTelemetry Collector standardizes ingestion and buffering.
• XplurData API validates and enriches incoming events.
• Apache Doris stores telemetry using a massively parallel columnar engine.
• The XplurData frontend provides search, dashboards, tracing and operational analytics.

Prerequisites

Before deploying XplurData, make sure your environment meets the following minimum requirements. While the platform can run on a laptop for evaluation, a production deployment should use dedicated infrastructure.

Step 1 — Clone the Repository

Download the latest XplurData deployment package from GitHub.

git clone https://github.com/xplurdata/xplurdata.git

cd xplurdata

The repository contains everything required to deploy the platform, including Docker Compose files, OpenTelemetry Collector configuration, Apache Doris initialization scripts and the XplurData frontend.

Repository Structure

xplurdata/

├── api/

├── collector/

├── dashboard/

├── doris/

├── docker/

├── frontend/

├── scripts/

├── docker-compose.yml

└── README.md

Step 2 — Start the Platform

Once the repository has been cloned, start every service using Docker Compose.

docker compose up -d

The first startup may take several minutes while Docker downloads the required container images.

Containers Started

Step 3 — Verify the Deployment

Check that every container is running correctly.

docker ps

You should see all platform services reporting a healthy status.

✔ Apache Doris FE

✔ Apache Doris BE

✔ XD API

✔ OTel Collector

✔ XplurData Frontend

Step 4 — Open the Platform

After deployment completes, open the following endpoints in your browser.

Step 5 — Configure the OpenTelemetry Collector

The OpenTelemetry Collector is responsible for receiving telemetry from your applications, processing it, batching requests and forwarding them to the XplurData API. Instead of every application talking directly to the backend database, applications send telemetry using the standard OTLP protocol while the collector handles routing, retries and buffering.

Collector Configuration (otelcol.yaml)

The following configuration enables OTLP receivers, uses the batch processor for improved throughput, and exports telemetry to the XplurData API.

receivers:

  otlp:

    protocols:

      grpc:

      http:



processors:

  batch:

    timeout: 5s

    send_batch_size: 1000



exporters:

  otlphttp:

    endpoint: http://xd-api:8080/otlp



service:

  pipelines:

    logs:

      receivers: [otlp]

      processors: [batch]

      exporters: [otlphttp]



    traces:

      receivers: [otlp]

      processors: [batch]

      exporters: [otlphttp]

Understanding Receivers

Receivers define how telemetry enters the collector. In most deployments, applications communicate using the OpenTelemetry Protocol (OTLP), making the OTLP receiver the primary entry point.

Understanding Processors

Processors modify telemetry before it reaches storage. The batch processor groups events together, reducing network overhead and significantly improving throughput.

Understanding Exporters

Exporters define where processed telemetry is delivered. In this deployment, the OTLP HTTP exporter sends logs and traces directly to the XplurData API, which validates incoming telemetry before storing it inside Apache Doris.

Telemetry Flow

Production Best Practices

• Always enable batching to improve throughput.
• Configure memory limits to avoid collector crashes.
• Use HTTPS for external OTLP endpoints.
• Deploy multiple collectors behind a load balancer for high availability.
• Monitor collector CPU and queue depth.
• Enable compression for WAN deployments.

Step 6 — Create the Apache Doris Schema

Apache Doris is the analytics engine powering XplurData. Instead of storing telemetry in multiple databases, logs and traces are stored in columnar OLAP tables that support extremely fast filtering, aggregation and full-text search.

This architecture enables billions of log records to be queried with interactive response times while significantly reducing storage costs through compression and columnar encoding.

Creating the Log Table

The following example creates a simplified log table used by XplurData.

CREATE TABLE otel_logs (

timestamp DATETIME,

trace_id STRING,

span_id STRING,

service_name STRING,

severity STRING,

body STRING,

attributes JSON

)

ENGINE=OLAP

DUPLICATE KEY(timestamp, service_name)

DISTRIBUTED BY HASH(service_name)

BUCKETS 16

PROPERTIES (

"replication_num"="1",

"compression"="zstd"

);

Creating the Trace Table

Distributed tracing data is stored separately, allowing traces to be searched independently while still supporting correlation with logs through the Trace ID.

CREATE TABLE otel_traces (

trace_id STRING,

span_id STRING,

parent_span_id STRING,

service_name STRING,

operation_name STRING,

start_time DATETIME,

duration BIGINT,

status STRING,

attributes JSON

)

ENGINE=OLAP

DUPLICATE KEY(trace_id, span_id)

DISTRIBUTED BY HASH(trace_id)

BUCKETS 16

PROPERTIES (

"replication_num"="1",

"compression"="zstd"

);

Storage Strategy

Partitioning Strategy

Large observability deployments should partition data by time to improve query performance and simplify retention management.

Indexing Recommendations

Although Apache Doris scans columns efficiently, choosing appropriate keys and distribution columns greatly improves performance.

Example Query

Once telemetry is stored, engineers can analyze operational events using standard SQL.

SELECT

service_name,

COUNT(*) AS total_errors

FROM otel_logs

WHERE severity='ERROR'

AND timestamp >= NOW() - INTERVAL 1 HOUR

GROUP BY service_name

ORDER BY total_errors DESC;

Storage Best Practices

• Use ZSTD compression for production deployments.
• Distribute tables using service_name or trace_id.
• Partition large tables by time.
• Separate logs and traces into dedicated tables.
• Archive historical data using lifecycle policies.
• Monitor compaction jobs regularly.

Step 7 — Instrument Your Application

Now that the platform is running and Apache Doris has been configured, it's time to send your first telemetry. One of OpenTelemetry's biggest advantages is that every supported language follows the same concepts. Whether you're using Java, Python, Go, .NET or Node.js, your applications communicate with the collector using OTLP.

Python Example

Install the required OpenTelemetry packages.

pip install \
opentelemetry-sdk \
opentelemetry-exporter-otlp

Configure the OTLP endpoint.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

OTEL_SERVICE_NAME=payment-service

Java Example

Java applications can use the OpenTelemetry Java Agent without modifying application code.

java \
-javaagent:opentelemetry-javaagent.jar \
-Dotel.service.name=payment-service \
-Dotel.exporter.otlp.endpoint=http://localhost:4317 \
-jar app.jar

Node.js Example

npm install

@opentelemetry/sdk-node

@opentelemetry/exporter-trace-otlp-http

@opentelemetry/auto-instrumentations-node

Step 8 — Verify Incoming Logs

Once applications begin sending telemetry, verify that the OpenTelemetry Collector is receiving requests.

docker logs otel-collector

--follow

You should see successful export messages similar to:

Exporting 542 log records...

Exporting 38 traces...

Batch completed successfully.

Verify Data Inside Apache Doris

Connect to Apache Doris and run a simple SQL query.

SELECT

COUNT(*)

FROM otel_logs;

The row count should begin increasing as applications continue sending telemetry.

Step 9 — Explore Telemetry

Open the XplurData dashboard. You should now be able to search, filter, aggregate and correlate telemetry across services.

Troubleshooting

Performance Tuning

A default installation is suitable for evaluation and small deployments, but production environments should be tuned based on expected telemetry volume, retention requirements and query workload.

Production Recommendations

• Deploy three Apache Doris Frontend nodes for high availability.
• Use three or more Backend nodes to distribute storage and queries.
• Store data on NVMe SSDs for optimal query performance.
• Run multiple OpenTelemetry Collectors behind a load balancer.
• Enable HTTPS and authentication for all production endpoints.
• Monitor collector queue depth and exporter latency.
• Configure automatic backups and retention policies.

Deploying on Kubernetes

Although Docker Compose is perfect for local evaluation, Kubernetes provides better scalability, resilience and lifecycle management for production deployments.

Frequently Asked Questions

Conclusion

Modern observability should not require expensive licensing or proprietary agents. By combining OpenTelemetry, Apache Doris and XplurData, engineering teams can build a powerful, standards-based observability platform that scales from local development to enterprise production environments.

With Docker-based deployment, native OTLP ingestion, distributed analytics and an intuitive user interface, XplurData enables teams to focus on improving application reliability instead of managing complex observability infrastructure.