Analytics and Metrics

Introduction

The analytics module provides access to alert/event data from OpenSearch, host-level sensor telemetry from InfluxDB, and snapshot-grounded analytics chat with optional free enrichment. OpenSearch remains keyed by virtual sensor identity, while InfluxDB stores physical sensor online status and host metrics emitted by the single host agent installed on each sensor.

Controllers

Controller	Route	Description
`OpenSearchController`	`api/opensearch`	OpenSearch analytics and dashboard endpoints
`VirtualSensorsAnalyticsController`	`api/analytics/virtual-sensors`	Virtual sensor filter data for analytics
`AnalyticsChatController`	`api/analytics/chat`	Analytics chat status, saved conversations, and SSE streaming endpoints

Files:

API/Controllers/Analytics/OpenSearchController.cs → delegates to IOpenSearchAnalyticsService
API/Controllers/Analytics/VirtualSensorsAnalyticsController.cs → delegates to IVirtualSensorsAnalyticsService
API/Controllers/Analytics/AnalyticsChatController.cs → delegates to IAnalyticsChatService

Service Layer

Service Interface	Implementation	Responsibility
`IOpenSearchAnalyticsService`	`OpenSearchAnalyticsService`	OpenSearch query orchestration with caching, timeout policies, and graceful fallback
`IVirtualSensorsAnalyticsService`	`VirtualSensorsAnalyticsService`	Virtual sensor analytics
`IAnalyticsChatService`	`AnalyticsChatService`	Snapshot-grounded analytics chat orchestration, prompt building, persistence, and SSE response streaming
`IAnalyticsChatEnrichmentService`	`AnalyticsChatEnrichmentService`	Parallel orchestration of MISP, history, and playbook enrichment providers

OpenSearchAnalyticsService, VirtualSensorsAnalyticsService, and AnalyticsChatService handle analytics query composition and response shaping on top of the configured OpenSearch, database, and model dependencies.

OpenSearch Integration

Service

File: Infrastructure/Services/OpenSearchAnalyticsService.cs Lifetime: Scoped (registered as IOpenSearchAnalyticsService) Library: OpenSearch.Client v1.8.0

Configuration

{
  "OpenSearch": {
    "Url": "{{OpenSearch__Url}}",
    "Username": "{{OpenSearch__Username}}",
    "Password": "{{OpenSearch__Password}}",
    "IndexName": "{{OpenSearch__IndexName}}"
  }
}

{
  "OpenSearchAnalytics": {
    "DefaultTimeoutSeconds": 20,
    "DashboardTimeoutSeconds": 30,
    "AggregationTimeoutSeconds": 15,
    "ListQueryTimeoutSeconds": 25
  },
  "AnalyticsCacheWarming": {
    "Enabled": true,
    "IntervalMinutes": 5,
    "InitialDelaySeconds": 20
  }
}

Key	Type	Default	Description
`Url`	`string`	`https://localhost:9200`	OpenSearch endpoint URL
`Username`	`string`	`admin`	Authentication username
`Password`	`string`	(required)	Authentication password
`IndexName`	`string`	`mataelang-sensor-events-stream`	Default index to query

Startup Validation

OpenSearch availability is verified during startup health checks:

HTTP GET to the root URL with Basic Auth
SSL certificate validation is bypassed (DangerousAcceptAnyServerCertificateValidator)
Retries up to 3 times with 10-second timeout per attempt

Usage

The OpenSearchAnalyticsService provides methods for:

Aggregating alerts for dashboard cards and charts
Filtering alert data by organization, virtual sensor, time range, and IP/protocol metadata
Returning paginated alert lists and IP-flow data for the analytics UI
Computing bounded recurrence windows used by analytics chat historical enrichment

Sensor Identity Contract

OpenSearch analytics expects metadata.sensor_id to store the virtual sensor UUID
Backend analytics filters compare metadata.sensor_id.keyword against virtual_sensors.id
Heartbeat and InfluxDB health/metrics flows continue using the physical sensor UUID
This mapping fix is future-only; historical OpenSearch documents indexed under a physical sensor UUID remain unchanged and will not appear in the virtual-sensor analytics UI

Analytics Chat Enrichment Architecture

Current v1 analytics chat is analytics-first and snapshot-grounded.

The pinned OpenSearch snapshot remains the primary evidence source
MISP is queried through a direct REST API to a self-hosted instance
Historical enrichment comes from internal OpenSearch recurrence queries
Playbook guidance is loaded from a backend-managed local JSON file

Core services

Service	Responsibility
`AnalyticsChatService`	Builds snapshot evidence, extracts IOC candidates, requests enrichment, builds prompt context, and streams the answer
`AnalyticsChatEnrichmentService`	Fans out to the available enrichment providers and merges compact citations into the response
`MispThreatIntelProvider`	Performs read-only MISP REST lookups for top IOC candidates
`HistoricalAnalyticsProvider`	Computes first seen, last seen, 24h count, lookback count, and distinct virtual sensor count
`PlaybookGuidanceProvider`	Matches classifications, signatures, protocols, and CVEs against local playbook entries

Enrichment flow

Build the normal dashboard evidence from OpenSearch summary, alert samples, and IP-flow samples.
Extract IOC candidates from the latest user message plus alert samples.
Execute the available enrichment providers in parallel.
Append the enrichment block before the model call:
- === MISP THREAT INTEL ===
- === HISTORICAL CONTEXT ===
- === TRIAGE PLAYBOOK ===

Guardrails

MISP access is read-only in v1; there is no write-back path.
IOC lookups are capped by MaxIocsPerRequest and MaxMispMatchesPerIoc.
MISP timeout or parse failures are logged and fail open; chat can continue with history, playbooks, or analytics-only evidence.
Threat intel is supporting context only. Local analytics evidence remains primary when signals conflict.
Enrichment fields are normalized before prompt assembly to reduce prompt injection risk.

Status and capability flags

GET /api/analytics/chat/status exposes backend availability and enrichment capability flags:

ThreatIntel
HistoricalEnrichment
Playbooks

InfluxDB Integration

Service

File: Infrastructure/Services/InfluxDbService.cs Lifetime: Singleton Library: InfluxDB.Client v4.18.0

Configuration

{
  "InfluxDb": {
    "Url": "{{InfluxDb__Url}}",
    "Token": "{{InfluxDb__Token}}",
    "Org": "{{InfluxDb__Org}}",
    "Bucket": "{{InfluxDb__Bucket}}",
    "Precision": "ms",
    "TimeoutSeconds": 30
  }
}

Key	Type	Default	Description
`Url`	`string`	—	InfluxDB server URL
`Token`	`string`	—	Authentication token
`Org`	`string`	—	InfluxDB organization
`Bucket`	`string`	—	Default bucket for queries
`Precision`	`string`	`ms`	Write precision
`TimeoutSeconds`	`int`	`30`	Client timeout

Startup Validation

InfluxDB availability is verified during startup health checks:

HTTP GET to {url}/health endpoint
Must return a success status code
Retries up to 3 times with 10-second timeout per attempt

Usage

The InfluxDbService provides methods for:

Writing online status to the sensor_status measurement
Writing host CPU, memory, and eligible interface traffic to the sensor_metrics measurement
Querying host metrics for the sensor metrics UI
Retrieving latest online heartbeat timestamps for health monitoring

Metrics DTO

File: Application/DTOs/SensorHeartbeatMetricsDto.cs

Used for transferring heartbeat and metrics data:

public class SensorHeartbeatMetricsDto
{
    // Heartbeat and metric fields for sensor monitoring
}

File: Application/DTOs/SensorMetricsRequest.cs

Used for requesting metric data:

public class SensorMetricsRequest
{
    // Time range and filter parameters for metrics queries
}

Runtime Behavior

Startup Requirements

Check	Behavior on Failure
OpenSearch health	Startup fails after 3 retries
InfluxDB health	Startup fails after 3 retries

Both services are fail-fast at startup — the application will not start if either is unavailable. This ensures data integrity and prevents requests from silently failing.

Request-Time Behavior

OpenSearch analytics endpoints query OpenSearch synchronously through OpenSearchAnalyticsService
High-frequency analytics endpoints use Redis cache and stale-cache fallback patterns
OpenSearch analytics uses per-query-type timeout options (OpenSearchAnalyticsOptions)
Circuit breaker is applied in analytics query execution
Background cache warming runs via hosted service (AnalyticsCacheWarmingService) for key dashboard paths
AnalyticsChatService builds evidence from the pinned snapshot and then calls enrichment providers in parallel when they are available
MISP enrichment failures are non-fatal at request time; the chat flow degrades to history, playbooks, or analytics-only mode
AnalyticsChatController status responses expose capability flags so the frontend can distinguish full chat availability from individual enrichment availability
InfluxDB-backed metrics remain available through their own service layer outside the removed legacy analytics routes

Rollout Notes

Each physical sensor runs exactly one host agent installed through the backend-generated install command
Virtual sensor deployments no longer include the heartbeat sidecar; only Suricata and sensor-client are deployed per virtual sensor
MES_CLIENT_SENSOR_ID continues to use the virtual sensor UUID for OpenSearch analytics identity

Service Registration

Relevant analytics services are registered in ServiceCollectionExtensions:

services.AddSingleton<InfluxDbService>();
services.AddScoped<IOpenSearchAnalyticsService, OpenSearchAnalyticsService>();
services.AddScoped<IVirtualSensorsAnalyticsService, VirtualSensorsAnalyticsService>();
services.AddHttpClient<IMispThreatIntelProvider, MispThreatIntelProvider>();
services.AddScoped<IHistoricalAnalyticsProvider, HistoricalAnalyticsProvider>();
services.AddSingleton<IPlaybookGuidanceProvider, PlaybookGuidanceProvider>();
services.AddScoped<IAnalyticsChatEnrichmentService, AnalyticsChatEnrichmentService>();
services.AddScoped<IAnalyticsChatService, AnalyticsChatService>();
services.AddSingleton<CircuitBreakerService>();
services.AddHostedService<AnalyticsCacheWarmingService>();

They receive their configuration through IOptions<T> typed options:

InfluxDbService ← IOptions<InfluxDbOptions>
OpenSearchAnalyticsService ← IOptions<OpenSearchOptions> and IOptions<OpenSearchAnalyticsOptions>
MispThreatIntelProvider, HistoricalAnalyticsProvider, and PlaybookGuidanceProvider ← IOptions<AnalyticsChatEnrichmentOptions>

Future Improvements

Completed

Response caching for key analytics endpoints.
Per-query-type timeout configuration.
Circuit breaker execution path in analytics service.
Pagination support for alerts list (page, pageSize).

Open

Query result aggregation breadth (cache warming covers key paths, not all query shapes).
Graceful degradation metadata contract standardization across all endpoint response shapes.

Introduction​

Controllers​

Service Layer​

OpenSearch Integration​

Service​

Configuration​

Startup Validation​

Usage​

Sensor Identity Contract​

Analytics Chat Enrichment Architecture​

Core services​

Enrichment flow​

Guardrails​

Status and capability flags​

InfluxDB Integration​

Service​

Configuration​

Startup Validation​

Usage​

Metrics DTO​

Runtime Behavior​

Startup Requirements​

Request-Time Behavior​

Rollout Notes​

Service Registration​

Future Improvements​

Completed​

Open​

Introduction

Controllers

Service Layer

OpenSearch Integration

Service

Configuration

Startup Validation

Usage

Sensor Identity Contract

Analytics Chat Enrichment Architecture

Core services

Enrichment flow

Guardrails

Status and capability flags

InfluxDB Integration

Service

Configuration

Startup Validation

Usage

Metrics DTO

Runtime Behavior

Startup Requirements

Request-Time Behavior

Rollout Notes

Service Registration

Future Improvements

Completed

Open