Data Plane: An Alert's Journey

This walkthrough follows a single Suricata alert from the wire to the operator's dashboard. It ties together the Sensor, the Data Collector, the Event Aggregator, and the Deployment pipeline.

The data plane carries alert traffic only. Trust and commands travel the Control Plane.

---

Flow A — From wire to OpenSearch

Step by step:

1. Suricata detects a rule match and writes an event to eve.json exposed over a unix socket.
2. The sensor reads the socket, parses and de-duplicates events (internal/suricata), then streams them over gRPC (internal/transport) to the data collector.
3. The data collector (sensor-api :50051) queues the events and produces them to the Kafka topic sensor_events, using an Avro schema registered in the Schema Registry.
4. The event aggregator consumes sensor_events, enriches each event with geo/IP data, and produces the result to snort_alerts.
5. Logstash consumes snort_alerts, reads tenant_id from the Kafka message header, stamps received_opensearch_at, and indexes the document into the per-tenant OpenSearch index mataelang-sensor-events-<tenant_id>.

---

Flow B — From storage to the operator

Two read paths surface the data in the UI:

• Analytics / alert browsing — the frontend queries the backend, which runs analytics queries against OpenSearch (OpenSearch__IndexName) and returns alert data, aggregations, and search results.
• Live threat map — in parallel with indexing, Logstash also produces geo events to the Kafka topic ravenxcope_threat_geo_events. The backend's threat-map consumer reads that topic and pushes live events to the frontend through the threat-map live hub (controlled by ThreatMap__* settings).

---

On the two ingestion paths

You may notice the alert path above goes sensor → data collector → Kafka, while the backend also exposes an mTLS gRPC SensorEventStreamService.

• Operational path (today): sensor → data collector (sensor-api :50051) → Kafka pipeline → OpenSearch. This is what the deployment compose wires up.
• Target path: the backend's mTLS SensorEventStreamService. Today it authenticates the client certificate and enforces liveness (matching the cert serial against the active sensor list) rather than persisting events.

The system is mid-transition; treat the data-collector path as the source of truth for where alerts are actually stored.

---

Summary

Stage	Component	Topic / Index
Detect	Suricata	eve.json
Relay	ravenxcope-sensor	gRPC stream
Collect	Data Collector (sensor-api)	→ sensor_events
Enrich	event-stream-aggr	→ snort_alerts
Index	Logstash	→ mataelang-sensor-events-<tenant>
Live map	Logstash → backend	→ ravenxcope_threat_geo_events
Present	Backend → Frontend	OpenSearch queries + live hub