Startup and Pipeline

Introduction

This document provides a detailed walkthrough of the application startup sequence, service registration, middleware pipeline configuration, and dependency health check mechanisms. Understanding this flow is critical for extending the application or debugging startup issues.

---

Program.cs Bootstrap Flow

The Program.cs file follows a strictly ordered initialization sequence. The order matters significantly: configuration validation must complete before service registration, and health checks must pass before migrations.

Initialization Sequence

1. Configure Logging: Creates the builder and initializes Serilog to capture all subsequent startup events.
2. Validate Configuration: Validates all required configuration keys. Fails fast if any are missing.
3. Security Checks: Emits warnings for insecure configurations (e.g., short JWT secrets).
4. Log Settings: Outputs the current auto-migration settings for debugging.
5. Register Services: Binds options, configures the database, connects to Redis, and registers all DI services.
6. Build Application: Compiles the service collection into the WebApplication instance.
7. Dependency Health Checks: Pings PostgreSQL, Redis, InfluxDB, and OpenSearch. Fails fast if unreachable.
8. Apply Migrations: Automatically applies pending EF Core migrations (if enabled).
9. Configure Pipeline: Sets up middleware (CORS, Rate Limiting, Auth) and starts the server.

Implementation

// 1. Configure logging
var builder = WebApplication.CreateBuilder(args);
builder.Host.UseSerilog((context, services, loggerConfiguration) =>
    loggerConfiguration
        .ReadFrom.Configuration(context.Configuration)
        .ReadFrom.Services(services)
        .Enrich.FromLogContext());

// 2. Validate configuration
var validatedConfigGroups = builder.Configuration.ValidateRequiredConfiguration();
Log.Information("Configuration validation passed for groups: {ConfigGroups}",
    string.Join(", ", validatedConfigGroups));

// 3. Security checks
var jwtSecretLength = builder.Configuration[BackendConfigurationKeys.JwtSecretKey]?.Length ?? 0;
if (jwtSecretLength < 32)
{
    Log.Warning("JwtSettings:Secret length is {SecretLength}. Use at least 32 characters.",
        jwtSecretLength);
}

// 4. Log settings
Log.Information("Auto-migration settings: AutoMigrate={AutoMigrate}, MaxRetries={MaxRetries}...",
    builder.Configuration.GetValue(BackendConfigurationKeys.AutoMigrateKey, true), ...);

// 5. Register services
await builder.Services.AddBackendServicesAsync(builder.Configuration);

// 6. Build application
var app = builder.Build();

// 7. Dependency health checks
await app.RunStartupDependencyHealthChecksAsync();

// 8. Apply migrations
await app.ApplyDatabaseMigrationsAsync();

// 9. Configure pipeline
app.UseBackendPipeline();
await app.RunAsync();

---

Service Registration — AddBackendServicesAsync

The ServiceCollectionExtensions.AddBackendServicesAsync method registers all application dependencies in a single call. It is async because it eagerly connects to Redis.

Registration Order

1. Bind typed options from configuration sections
   • PostgresqlSettingsOptions, RedisSettingsOptions, JwtSettingsOptions
   • InfluxDbOptions, OpenSearchOptions, AnsibleSettingsOptions
   • SensorApiSettingsOptions, DataCollectorOptions, DockerRegistryOptions
   • SensorSettingsOptions, DatabaseOptions
   • BackendAppOptions, SensorRuntimeOptions
2. Register Controllers with JSON serializer options
   • PropertyNameCaseInsensitive = true
3. Register EndpointsApiExplorer
4. Register Swagger with JWT Bearer security definition
5. Register gRPC services
6. Register rate limiters
   • auth-login: fixed window, 5 requests/minute
   • auth-register: fixed window, 3 requests/minute
7. Configure PostgreSQL DbContext
   • Build connection string from typed options
   • services.AddDbContext<ApplicationDbContext>(UseNpgsql)
8. Connect to Redis (eager, async)
   • Build connection string from typed options
   • ConnectionMultiplexer.ConnectAsync(...)
   • Register IConnectionMultiplexer as singleton
   • Register IRedisService (RedisService implementation) as singleton
9. Register singleton services
   • JwtService, InfluxDbService, OpenSearchService, ResilientHttpService
10. Register scoped repositories
    • EfRepository<T> (generic base)
    • PermissionRepository, RoleRepository, OrganizationRepository
    • UserRepository, UserRoleRepository, OrganizationUserRoleRepository
    • SensorRepository, SensorHeartbeatRepository, VirtualSensorRepository
    • AssetRepository
11. Register scoped services
    • PermissionService, RoleService, LocationService, AnalyticsService
    • OrganizationService, UserService, AuthService
    • VirtualSensorsAnalyticsService, SensorHeartbeatService
    • SensorInterfaceAllocationService, SensorProvisioningScriptService
    • SensorActivationService, SensorsService, AssetService
    • VirtualSensorsService, OpenSearchAnalyticsService
12. Register HttpClient factory
13. Configure JWT Bearer authentication
    • ValidateIssuer, ValidateAudience, ValidateLifetime = true
    • ClockSkew = TimeSpan.Zero
    • OnTokenValidated: check Redis token blacklist
14. Configure authorization policies
    • "manage-users", "manage-sensors", "manage-assets", "manage-roles", "edit-organization"
15. Register CORS policy

Typed Options Classes

All configuration sections are bound to strongly-typed options classes defined in BackendConfiguration.cs:

Options Class	Config Section	Key Properties
PostgresqlSettingsOptions	PostgresqlSettings	Host, Port, Username, Password, Database, MaxPoolSize
RedisSettingsOptions	RedisSettings	Host, Port, Password, DefaultDatabase
JwtSettingsOptions	JwtSettings	Secret, Issuer, Audience, ExpiryMinutes
InfluxDbOptions	InfluxDb	Url, Token, Org, Bucket
OpenSearchOptions	OpenSearch	Url, Username, Password, IndexName
AnsibleSettingsOptions	AnsibleSettings	ServiceUrl
SensorApiSettingsOptions	SensorApiSettings	ApiKey
DataCollectorOptions	DataCollector	Endpoint, Port
DockerRegistryOptions	DockerRegistry	Registry, Username, Password
SensorSettingsOptions	SensorSettings	HeartbeatTimeoutMinutes, ProvisioningSudoPassword
DatabaseOptions	Database	AutoMigrate, MigrationMaxRetries, MigrationRetryDelaySeconds
OpenSearchAnalyticsOptions	OpenSearchAnalytics	DefaultTimeoutSeconds, DashboardTimeoutSeconds, AggregationTimeoutSeconds, ListQueryTimeoutSeconds
AnalyticsCacheWarmingOptions	AnalyticsCacheWarming	Enabled, IntervalMinutes, InitialDelaySeconds
BackendAppOptions	(composite)	Url
SensorRuntimeOptions	(composite)	Aggregates multiple sections for sensor ops

---

Middleware Pipeline — UseBackendPipeline

The WebApplicationExtensions.UseBackendPipeline method configures the request processing pipeline:

public static WebApplication UseBackendPipeline(this WebApplication app)
{
    app.UseSerilogRequestLogging();         // 1. Log all HTTP requests
    app.UseGlobalExceptionHandling();       // 2. Global exception handler
    app.UseSwagger();                       // 3. Serve OpenAPI spec
    app.UseSwaggerUI(c =>                   // 4. Swagger UI at /swagger
    {
        c.SwaggerEndpoint("/swagger/v1/swagger.json", "Ravenxcope Backend API v1");
        c.DocExpansion(DocExpansion.None);
    });
    app.UseCors();                          // 5. Apply CORS policy
    app.UseHttpsRedirection();              // 6. Redirect HTTP → HTTPS
    app.UseRateLimiter();                   // 7. Apply rate limiting policies
    app.UseAuthentication();                // 8. JWT token validation
    app.UseAuthorization();                 // 9. Policy enforcement
    app.MapControllers();                   // 10. Map attribute-routed controllers
    app.MapGrpcService<SensorHealthcheckService>();  // 11. Map gRPC service
    return app;
}

Pipeline Order: The middleware order is significant. UseGlobalExceptionHandling must come early to catch all downstream exceptions. UseRateLimiter must come before UseAuthentication so rate limits apply before token validation. UseAuthentication must come before UseAuthorization, and both must come before MapControllers for auth attributes to work.

---

Configuration Validation

The ConfigurationValidationExtensions.ValidateRequiredConfiguration method validates all required keys at startup:

Validated Keys

Config Section	Required Keys
PostgresqlSettings	Host, Port, Username, Password, Database
RedisSettings	Host, Port
JwtSettings	Secret
InfluxDb	Url, Token, Org, Bucket
OpenSearch	Url, Username, Password, IndexName
AnsibleSettings	ServiceUrl
SensorApiSettings	ApiKey
SensorSettings	ProvisioningSudoPassword
BackendUrl	(root-level key)
DataCollector	Endpoint, Port

Validation Rules

1. Missing or empty values → throws InvalidOperationException
2. Placeholder values →{{...}} double-brace patterns are detected and treated as missing
3. Known placeholder strings → "your-sensor-api-key-here" is treated as missing
4. Port validation → PostgreSQL, Redis, DataCollector ports must be positive integers
5. Migration settings → MigrationMaxRetries and MigrationRetryDelaySeconds must be positive integers

---

Dependency Health Checks

The StartupDependencyHealthChecksExtensions.RunStartupDependencyHealthChecksAsync method verifies all external dependencies before serving traffic.

Check Details

Dependency	Check Method	Retry Count	Timeout
PostgreSQL	dbContext.Database.CanConnectAsync()	3	10s
Redis	redis.GetDatabase().PingAsync()	3	10s
InfluxDB	HTTP GET {url}/health	3	10s
OpenSearch	HTTP GET {url}/ with Basic Auth	3	10s

Retry Behavior

Checks use HealthCheckHelper.RunHealthCheckWithRetry<T>:

public static async Task<bool> RunHealthCheckWithRetry<T>(
    IServiceProvider serviceProvider,
    ILogger logger,
    Func<T, CancellationToken, Task> checkFunc,
    string serviceName,
    int maxRetries = 3,
    int timeoutSeconds = 10)

• Each attempt has a CancellationTokenSource with the configured timeout
• After a timeout or exception, waits 1 second before retrying
• If all retries fail, logs an error and returns false
• If any single dependency fails, the startup throws InvalidOperationException

OpenSearch Special Handling

The OpenSearch health check uses DangerousAcceptAnyServerCertificateValidator to bypass SSL certificate validation, and authenticates with Basic Auth credentials from the configuration.

---

Database Migration

The DatabaseMigrationExtensions.ApplyDatabaseMigrationsAsync method handles automatic database schema migration:

Migration Flow

1. Check Configuration: Read Database:AutoMigrate from options (default: true)
2. Evaluation: If false, skip migration entirely.
3. Execution Loop: For each attempt (up to MigrationMaxRetries):
   • Create a scoped ApplicationDbContext
   • Call dbContext.Database.MigrateAsync()
   • On success: Log successful migration and break the loop.
   • On failure: Log a warning, wait for RetryDelaySeconds, then retry.
4. Final Evaluation: If all attempts fail, log a critical error and throw an exception to halt startup.

Migration Configuration

Key	Type	Default	Description
Database:AutoMigrate	bool	true	Enable/disable auto-migration
Database:MigrationMaxRetries	int	10	Max retry attempts
Database:MigrationRetryDelaySeconds	int	5	Delay between retries