Log pipelines, SIEM & data retention
Logs are the memory of a network, and they are only useful if they arrive, are parsed and can be found months later. Syslog (RFC 5424) is still the common language of network gear and servers: a layered protocol with a structured message format that works over several transports. OpenTelemetry adds a vendor-neutral way to produce traces, metrics and logs, with a Collector that receives, processes and exports them.
What we do
- Syslog and OpenTelemetry collection design
- Parsing and normalization pipelines
- Kafka buffering for lossless ingestion
- Elasticsearch, OpenSearch and Splunk deployments
- Sentinel detection rules and playbooks
- Retention tiers tied to regulation
- Immutable and write-once log storage
- Pipeline health monitoring and cost control
Between sources and storage sits a pipeline. Agents and routers such as Fluent Bit, Fluentd or Vector collect, parse, filter and forward; Apache Kafka buffers events durably so a slow or failed back end does not lose data, and lets several consumers read the same stream. Storage and search are Elasticsearch, OpenSearch, Splunk or a cloud SIEM such as Microsoft Sentinel, which adds detection rules, hunting and automated response on top.
Retention is a design decision with a cost. Elasticsearch's hot, warm, cold and frozen tiers show the idea: recent data on fast disks, older data on cheaper storage or searchable snapshots on object storage. We map each source to a retention period driven by regulation and incident-response needs, protect integrity with time sync and write-once storage where required, and watch the pipeline itself for dropped events. Works with our SIEM, network automation and compliance practices.
Leading platforms we work with
- OpenTelemetry
- Fluent Bit
- Fluentd
- Apache Kafka
- Elasticsearch
- OpenSearch
- Splunk
- Microsoft Sentinel
- Vector
- Graylog
- Grafana Loki
- Datadog
- Cribl
- Wazuh
Platforms and practices
- Syslog (RFC 5424)
- OpenTelemetry Collector
- Log parsing and normalization
- Kafka buffering and fan-out
- Hot, warm, cold and frozen tiers
- Searchable snapshots on object storage
- SIEM detection rules and hunting
- Retention and legal hold policy
- Pipeline health and dropped-event alerts
References
- RFC 5424: The Syslog Protocol
- OpenTelemetry: what is OpenTelemetry
- Fluent Bit manual
- Fluentd
- Apache Kafka: introduction
- Elasticsearch
- Elasticsearch: data tiers
- OpenSearch: about
- Splunk Enterprise
- Microsoft Sentinel overview
Sources last checked .
Product names are trademarks of their owners and are listed to describe our experience; they do not imply endorsement or partnership.
Five million events, three escalations
What a managed SOC does to a day of log volume for a 250‑seat organisation. Collection produces millions of events; correlation rules turn them into a thousand or so alerts; triage merges and dismisses most of those into dozens of incidents; only a few need a human on the customer side. Bar length is the logarithm of the count, so each step is orders of magnitude. Illustrative figures.
How we use Log pipelines, SIEM & data retention for clients
Cybersecurity
We find out whether everything you run is safe from attackers, then we fix it.
Learn moreCyber Automation
Custom AI agents that do repetitive work around the clock, with human review where it matters.
Learn moreManaged IT Services
We run your IT day to day so it stays secure: help desk, devices, Microsoft 365, Exchange, Active Directory, backups, networks and monitoring.
Learn moreFrequently asked questions
Direct answers to the questions we hear most. Something else on your mind?
Send us a messageWhy not send logs straight to the SIEM?
A pipeline between sources and storage protects the data. Apache Kafka buffers events durably, so a slow or failed back end does not lose logs, and several consumers can read the same stream. Agents such as Fluent Bit, Fluentd or Vector parse, filter and forward before the SIEM sees anything.
How long should we keep logs?
It is a design decision with a cost. We map each source to a retention period driven by regulation and incident-response needs, then use tiers: recent data on fast disks, older data on cheaper storage or searchable snapshots on object storage. Your counsel confirms legal requirements.
What is syslog, and is it still relevant?
Syslog (RFC 5424) is a layered protocol with a structured message format that works over several transports. It is still the common language of network gear and servers. OpenTelemetry adds a vendor-neutral way to produce traces, metrics and logs, with a Collector that receives, processes and exports them.
Which SIEM should we choose?
It depends on your team, your data volume and your cloud footprint. Elasticsearch, OpenSearch, Splunk and Microsoft Sentinel all work. We compare cost per gigabyte, detection content and operating effort for your situation. We do not resell these products and have no partnership with their makers.
How do you know the pipeline is not dropping events?
We watch the pipeline itself: dropped-event and queue-depth alerts, agent heartbeats and end-to-end test events from each source. We also protect integrity with time synchronization and write-once storage where required. A silent log gap is worse than a loud failure.
Can this feed detection and response?
Yes. A cloud SIEM such as Microsoft Sentinel adds detection rules, hunting and automated response on top of the stored data. We help write and tune rules, route alerts to people who can act, and test them with exercises. This works with our SIEM and compliance practices.