Analytics and Metrics

Introduction

The analytics module provides access to alert/event data from OpenSearch and host-level sensor telemetry from InfluxDB. OpenSearch remains keyed by virtual sensor identity, while InfluxDB now stores physical sensor online status and host metrics emitted by the single host agent installed on each sensor.

---

Controllers

Controller	Route	Description
OpenSearchController	api/opensearch	OpenSearch analytics and dashboard endpoints
VirtualSensorsAnalyticsController	api/analytics/virtual-sensors	Virtual sensor filter data for analytics

Files:

• API/Controllers/Analytics/OpenSearchController.cs → delegates to IOpenSearchAnalyticsService
• API/Controllers/Analytics/VirtualSensorsAnalyticsController.cs → delegates to IVirtualSensorsAnalyticsService

Service Layer

Service Interface	Implementation	Responsibility
IOpenSearchAnalyticsService	OpenSearchAnalyticsService	OpenSearch query orchestration with caching, timeout policies, and graceful fallback
IVirtualSensorsAnalyticsService	VirtualSensorsAnalyticsService	Virtual sensor analytics

OpenSearchAnalyticsService and VirtualSensorsAnalyticsService handle analytics query composition and response shaping on top of the configured OpenSearch and database dependencies.

---

OpenSearch Integration

Service

File: Infrastructure/Services/OpenSearchAnalyticsService.cs Lifetime: Scoped (registered as IOpenSearchAnalyticsService) Library: OpenSearch.Client v1.8.0

Configuration

{
  "OpenSearch": {
    "Url": "{{OpenSearch__Url}}",
    "Username": "{{OpenSearch__Username}}",
    "Password": "{{OpenSearch__Password}}",
    "IndexName": "{{OpenSearch__IndexName}}"
  }
}

{
  "OpenSearchAnalytics": {
    "DefaultTimeoutSeconds": 20,
    "DashboardTimeoutSeconds": 30,
    "AggregationTimeoutSeconds": 15,
    "ListQueryTimeoutSeconds": 25
  },
  "AnalyticsCacheWarming": {
    "Enabled": true,
    "IntervalMinutes": 5,
    "InitialDelaySeconds": 20
  }
}

Key	Type	Default	Description
Url	string	https://localhost:9200	OpenSearch endpoint URL
Username	string	admin	Authentication username
Password	string	(required)	Authentication password
IndexName	string	mataelang-sensor-events-stream	Default index to query

Startup Validation

OpenSearch availability is verified during startup health checks:

• HTTP GET to the root URL with Basic Auth
• SSL certificate validation is bypassed (DangerousAcceptAnyServerCertificateValidator)
• Retries up to 3 times with 10-second timeout per attempt

Usage

The OpenSearchAnalyticsService provides methods for:

• Aggregating alerts for dashboard cards and charts
• Filtering alert data by organization, virtual sensor, time range, and IP/protocol metadata
• Returning paginated alert lists and IP-flow data for the analytics UI

Sensor Identity Contract

• OpenSearch analytics expects metadata.sensor_id to store the virtual sensor UUID
• Backend analytics filters compare metadata.sensor_id.keyword against virtual_sensors.id
• Heartbeat and InfluxDB health/metrics flows continue using the physical sensor UUID
• This mapping fix is future-only; historical OpenSearch documents indexed under a physical sensor UUID remain unchanged and will not appear in the virtual-sensor analytics UI

---

InfluxDB Integration

Service

File: Infrastructure/Services/InfluxDbService.cs Lifetime: Singleton Library: InfluxDB.Client v4.18.0

Configuration

{
  "InfluxDb": {
    "Url": "{{InfluxDb__Url}}",
    "Token": "{{InfluxDb__Token}}",
    "Org": "{{InfluxDb__Org}}",
    "Bucket": "{{InfluxDb__Bucket}}",
    "Precision": "ms",
    "TimeoutSeconds": 30
  }
}

Key	Type	Default	Description
Url	string	—	InfluxDB server URL
Token	string	—	Authentication token
Org	string	—	InfluxDB organization
Bucket	string	—	Default bucket for queries
Precision	string	ms	Write precision
TimeoutSeconds	int	30	Client timeout

Startup Validation

InfluxDB availability is verified during startup health checks:

• HTTP GET to {url}/health endpoint
• Must return a success status code
• Retries up to 3 times with 10-second timeout per attempt

Usage

The InfluxDbService provides methods for:

• Writing online status to the sensor_status measurement
• Writing host CPU, memory, and eligible interface traffic to the sensor_metrics measurement
• Querying host metrics for the sensor metrics UI
• Retrieving latest online heartbeat timestamps for health monitoring

Metrics DTO

File: Application/DTOs/SensorHeartbeatMetricsDto.cs

Used for transferring heartbeat and metrics data:

public class SensorHeartbeatMetricsDto
{
    // Heartbeat and metric fields for sensor monitoring
}

File: Application/DTOs/SensorMetricsRequest.cs

Used for requesting metric data:

public class SensorMetricsRequest
{
    // Time range and filter parameters for metrics queries
}

---

Runtime Behavior

Startup Requirements

Check	Behavior on Failure
OpenSearch health	Startup fails after 3 retries
InfluxDB health	Startup fails after 3 retries

Both services are fail-fast at startup — the application will not start if either is unavailable. This ensures data integrity and prevents requests from silently failing.

Request-Time Behavior

• OpenSearch analytics endpoints query OpenSearch synchronously through OpenSearchAnalyticsService
• High-frequency analytics endpoints use Redis cache and stale-cache fallback patterns
• OpenSearch analytics uses per-query-type timeout options (OpenSearchAnalyticsOptions)
• Circuit breaker is applied in analytics query execution
• Background cache warming runs via hosted service (AnalyticsCacheWarmingService) for key dashboard paths
• InfluxDB-backed metrics remain available through their own service layer outside the removed legacy analytics routes

Rollout Notes

• Each physical sensor runs exactly one host agent installed through the backend-generated install command
• Virtual sensor deployments no longer include the heartbeat sidecar; only Suricata and sensor-client are deployed per virtual sensor
• MES_CLIENT_SENSOR_ID continues to use the virtual sensor UUID for OpenSearch analytics identity

---

Service Registration

Relevant analytics services are registered in ServiceCollectionExtensions:

services.AddSingleton<InfluxDbService>();
services.AddScoped<IOpenSearchAnalyticsService, OpenSearchAnalyticsService>();
services.AddScoped<IVirtualSensorsAnalyticsService, VirtualSensorsAnalyticsService>();
services.AddSingleton<CircuitBreakerService>();
services.AddHostedService<AnalyticsCacheWarmingService>();

They receive their configuration through IOptions<T> typed options:

• InfluxDbService ← IOptions<InfluxDbOptions>
• OpenSearchAnalyticsService ← IOptions<OpenSearchOptions> and IOptions<OpenSearchAnalyticsOptions>

---

Future Improvements

Completed

• Response caching for key analytics endpoints.
• Per-query-type timeout configuration.
• Circuit breaker execution path in analytics service.
• Pagination support for alerts list (page, pageSize).

Open

• Query result aggregation breadth (cache warming covers key paths, not all query shapes).
• Graceful degradation metadata contract standardization across all endpoint response shapes.