High-Concurrency Monitoring API: Local Development Setup
Security Operations Platform Project Summary (6W Pitch Framework) Who (Target Customers and Core Roles): Designed for security operations teams at large enterprises, with an AI full-stack engineer leading
Security Operations Platform
Project Summary (6W Pitch Framework)
- Who (Target Customers and Core Roles): Designed for security operations teams at large enterprises, with an AI full-stack engineer leading the core architecture design and business-domain development.
- What (Core Product and Problems Addressed): An integrated enterprise-grade security operations management platform that addresses delayed processing of high-volume security alerts and fragmented risk management. It provides unified capabilities for cybersecurity monitoring, asset management, vulnerability remediation, and AI-powered intelligent analysis.
- Why (Core Differentiators and Competitive Advantages): Implements a three-tier data delivery strategy combining scheduled tasks, Redis caching, and real-time aggregation with cache backfilling on misses, reducing core API response times to under 50 ms. It enables a fully automated closed loop from risk identification to protection response.
- Where (Use Cases and Deployment Environments): Supports multi-scenario enterprise security operations, firewall integration, and domain security monitoring. It also provides Docker-based container orchestration for isolated multi-environment deployments and dynamic switching between data sources.
- When (Timeliness and Real-Time Capabilities): Processes more than 100,000 raw alerts from network security devices in real time each day. It supports five-minute scheduled aggregation and cache refresh cycles, while completing full-table aggregation across more than 150 tables containing tens of thousands of records within three seconds.
- How (Technical Approach and Delivery Model): Uses a microservices architecture to separate business modules and reuse shared capability libraries. It establishes a real-time analysis and automated device-response pipeline powered by Kafka, InfluxDB, a rules engine.
Business
- AI: Integrates AI Agents to perform intelligent attack-path analysis, three-dimensional risk assessment across systems, assets, and vulnerabilities, and automated generation of remediation recommendations.
- Automation: Provides a public IP blacklist analysis API that enables firewall devices to retrieve data, creating a fully automated process from risk identification to protection response.
- Concurrency: Supports real-time processing across more than 150 business tables and over 100,000 alerts per day. Core monitoring API response times were reduced from over two seconds to under 50 ms, while Redis-only APIs support more than 500 queries per second.
- Availability: Automatically falls back to database queries when the cache becomes unavailable, ensuring 99.9% system availability.
- Localization: Enables seamless switching between the Dameng database and the PostgreSQL.
- Efficiency: Improves development and delivery efficiency through independent microservice iteration and reuse of shared capability libraries.
Tech Stack
- Backend: Spring Boot 3.2.6
- Frontend: Vue 3
- Database: InfluxDB
- Cache: Redis
- Middleware: Kafka
- Containerization: Docker
This guide creates a local Spring Boot 3.2.6 monitoring API that demonstrates how a database-backed endpoint taking more than two seconds can be redesigned into a Redis-first endpoint capable of responding in under 50 milliseconds under suitable local conditions. It also includes a repeatable k6 workload for testing a target of more than 500 queries per second on the Redis-only path.
The implementation uses Java 17, Maven, PostgreSQL 16, Redis 7.2, Docker Compose, Spring Data JPA, Spring Data Redis, Micrometer, and k6. PostgreSQL remains the authoritative store. A scheduled task prepares monitoring summaries. Redis serves the normal read path. A controlled cache miss performs real-time aggregation and backfills Redis. A per-key single-flight lock prevents concurrent requests from repeating the same database work inside one application process.
The latency and throughput figures in this guide are development targets, not universal guarantees. Results depend on workstation CPU, memory, Docker allocation, operating system, payload size, Java warm-up, network path, Redis state, database volume, and test configuration.
This guide does not include production acceptance criteria or production acceptance tests.
Step 1: Prepare the Local Development Environment
Install the following software:
- Java Development Kit 17
- Apache Maven 3.9 or later
- Docker Desktop or Docker Engine with Docker Compose v2
- Git
- curl
- k6
Confirm the installations:
java -version
mvn -version
docker --version
docker compose version
git --version
curl --version
k6 version
Create the project:
mkdir high-concurrency-monitoring-api
cd high-concurrency-monitoring-api
mkdir -p src/main/java/com/example/monitoring/config
mkdir -p src/main/java/com/example/monitoring/controller
mkdir -p src/main/java/com/example/monitoring/domain
mkdir -p src/main/java/com/example/monitoring/dto
mkdir -p src/main/java/com/example/monitoring/repository
mkdir -p src/main/java/com/example/monitoring/service
mkdir -p src/main/resources
mkdir -p src/test/java/com/example/monitoring/service
mkdir -p load-test
The local API will expose three paths. The baseline path always aggregates from PostgreSQL and includes an intentional two-second delay to reproduce the original performance problem. The optimized path reads Redis first and uses PostgreSQL only after a cache miss. A refresh path recomputes and publishes a new cache entry.
Step 2: Start PostgreSQL and Redis
Create compose.yaml:
services:
postgres:
image: postgres:16-alpine
container_name: monitoring-postgres
environment:
POSTGRES_DB: monitoring
POSTGRES_USER: monitoring_user
POSTGRES_PASSWORD: monitoring_password
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U monitoring_user -d monitoring"]
interval: 5s
timeout: 3s
retries: 20
redis:
image: redis:7.2-alpine
container_name: monitoring-redis
command:
- redis-server
- --appendonly
- "yes"
- --maxmemory
- 256mb
- --maxmemory-policy
- allkeys-lfu
ports:
- "6379:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 20
volumes:
postgres_data:
redis_data:
Start both services:
docker compose up -d
Check their health:
docker compose ps
Confirm Redis:
docker exec -it monitoring-redis redis-cli ping
Confirm PostgreSQL:
docker exec -it monitoring-postgres \
psql -U monitoring_user -d monitoring \
-c "SELECT version();"
Step 3: Create the Spring Boot Project
Create pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.2.6</version>
<relativePath/>
</parent>
<groupId>com.example</groupId>
<artifactId>high-concurrency-monitoring-api</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Create src/main/resources/application.yml:
server:
port: 8080
tomcat:
threads:
max: 200
min-spare: 20
accept-count: 200
max-connections: 8192
spring:
application:
name: high-concurrency-monitoring-api
datasource:
url: jdbc:postgresql://localhost:5432/monitoring
username: monitoring_user
password: monitoring_password
hikari:
maximum-pool-size: 20
minimum-idle: 5
connection-timeout: 1000
validation-timeout: 500
jpa:
open-in-view: false
hibernate:
ddl-auto: validate
properties:
hibernate:
jdbc:
time_zone: UTC
data:
redis:
host: localhost
port: 6379
connect-timeout: 500ms
timeout: 500ms
lettuce:
pool:
max-active: 32
max-idle: 16
min-idle: 4
max-wait: 200ms
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus
endpoint:
health:
show-details: always
metrics:
distribution:
percentiles-histogram:
http.server.requests: true
slo:
http.server.requests: 10ms,25ms,50ms,100ms,250ms,500ms,1s,2s
app:
monitoring:
cache-prefix: monitoring:v1
cache-ttl: 10m
refresh-delay-ms: 300000
summary-window-hours: 24
baseline-delay-ms: 2000
tenants:
- tenant-a
- tenant-b
logging:
level:
com.example.monitoring: INFO
The HikariCP pool is intentionally bounded. A larger pool does not make PostgreSQL infinitely faster. It can increase contention during a cache failure. Redis uses short connection and command timeouts because a slow cache must not consume the entire HTTP request budget.
Create src/main/java/com/example/monitoring/MonitoringApplication.java:
package com.example.monitoring;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.scheduling.annotation.EnableScheduling;
@EnableScheduling
@SpringBootApplication
public class MonitoringApplication {
public static void main(String[] args) {
SpringApplication.run(MonitoringApplication.class, args);
}
}
Step 4: Create the Database Schema and Development Data
Create src/main/resources/schema.sql:
CREATE TABLE IF NOT EXISTS security_alert (
id BIGSERIAL PRIMARY KEY,
tenant_id VARCHAR(64) NOT NULL,
severity VARCHAR(16) NOT NULL,
status VARCHAR(16) NOT NULL,
source_ip VARCHAR(45) NOT NULL,
asset_id VARCHAR(128) NOT NULL,
detected_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_alert_tenant_time
ON security_alert (tenant_id, detected_at DESC);
CREATE INDEX IF NOT EXISTS idx_alert_tenant_status_severity_time
ON security_alert (tenant_id, status, severity, detected_at DESC);
CREATE INDEX IF NOT EXISTS idx_alert_tenant_asset_time
ON security_alert (tenant_id, asset_id, detected_at DESC);
Create src/main/resources/data.sql:
INSERT INTO security_alert
(tenant_id, severity, status, source_ip, asset_id, detected_at)
SELECT
CASE WHEN n % 2 = 0 THEN 'tenant-a' ELSE 'tenant-b' END,
CASE
WHEN n % 20 = 0 THEN 'CRITICAL'
WHEN n % 5 = 0 THEN 'HIGH'
WHEN n % 2 = 0 THEN 'MEDIUM'
ELSE 'LOW'
END,
CASE WHEN n % 4 = 0 THEN 'CLOSED' ELSE 'OPEN' END,
'203.0.113.' || ((n % 250) + 1),
'asset-' || LPAD(((n % 5000) + 1)::text, 5, '0'),
CURRENT_TIMESTAMP - ((n % 1440) || ' minutes')::interval
FROM generate_series(1, 100000) AS n;
Add SQL initialization to application.yml under spring:
sql:
init:
mode: always
continue-on-error: true
The seed script creates 100,000 local alerts across two tenants. It is synthetic development data. The first startup can take longer while PostgreSQL inserts rows and builds indexes.
Start the application once so Spring initializes the schema and data:
mvn spring-boot:run
After initialization, confirm the count:
docker exec -it monitoring-postgres \
psql -U monitoring_user -d monitoring \
-c "SELECT tenant_id, COUNT(*) FROM security_alert GROUP BY tenant_id;"
To prevent duplicate seed rows on later starts, change spring.sql.init.mode from always to never after the first successful initialization:
spring:
sql:
init:
mode: never
Restart the application after making that change.
Step 5: Implement the Monitoring Data Model
Create src/main/java/com/example/monitoring/domain/SecurityAlert.java:
package com.example.monitoring.domain;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import java.time.Instant;
@Entity
@Table(name = "security_alert")
public class SecurityAlert {
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
@Column(name = "tenant_id", nullable = false)
private String tenantId;
@Column(nullable = false)
private String severity;
@Column(nullable = false)
private String status;
@Column(name = "source_ip", nullable = false)
private String sourceIp;
@Column(name = "asset_id", nullable = false)
private String assetId;
@Column(name = "detected_at", nullable = false)
private Instant detectedAt;
@Column(name = "updated_at", nullable = false)
private Instant updatedAt;
protected SecurityAlert() {
}
}
Create src/main/java/com/example/monitoring/repository/SeverityCount.java:
package com.example.monitoring.repository;
public interface SeverityCount {
String getSeverity();
Long getTotal();
}
Create src/main/java/com/example/monitoring/repository/MonitoringRepository.java:
package com.example.monitoring.repository;
import com.example.monitoring.domain.SecurityAlert;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import java.time.Instant;
import java.util.List;
public interface MonitoringRepository extends JpaRepository<SecurityAlert, Long> {
@Query(value = """
SELECT severity AS severity, COUNT(*) AS total
FROM security_alert
WHERE tenant_id = :tenantId
AND detected_at >= :fromTime
GROUP BY severity
""", nativeQuery = true)
List<SeverityCount> countBySeverity(
@Param("tenantId") String tenantId,
@Param("fromTime") Instant fromTime
);
@Query(value = """
SELECT COUNT(*)
FROM security_alert
WHERE tenant_id = :tenantId
AND status = 'OPEN'
AND detected_at >= :fromTime
""", nativeQuery = true)
long countOpen(
@Param("tenantId") String tenantId,
@Param("fromTime") Instant fromTime
);
@Query(value = """
SELECT COUNT(DISTINCT asset_id)
FROM security_alert
WHERE tenant_id = :tenantId
AND detected_at >= :fromTime
""", nativeQuery = true)
long countAffectedAssets(
@Param("tenantId") String tenantId,
@Param("fromTime") Instant fromTime
);
}
The repository lets PostgreSQL aggregate the data and returns compact results. The application does not load 50,000 rows into Java memory to count them.
Step 6: Define the Cached Summary Contract
Create src/main/java/com/example/monitoring/dto/MonitoringSummary.java:
package com.example.monitoring.dto;
import java.time.Instant;
import java.util.Map;
public record MonitoringSummary(
String tenantId,
Instant windowStart,
Instant generatedAt,
Instant dataThrough,
long totalAlerts,
long openAlerts,
long affectedAssets,
Map<String, Long> severityCounts,
String deliverySource,
long processingTimeMs,
int schemaVersion
) {
public MonitoringSummary deliveredFrom(String source, long elapsedMs) {
return new MonitoringSummary(
tenantId,
windowStart,
generatedAt,
dataThrough,
totalAlerts,
openAlerts,
affectedAssets,
severityCounts,
source,
elapsedMs,
schemaVersion
);
}
}
The response includes generatedAt and dataThrough so a fast response does not hide its age. deliverySource distinguishes the cache, real-time backfill, scheduled refresh, and baseline database path. processingTimeMs measures service-side work and is not a replacement for end-to-end client latency.
Step 7: Configure Redis Serialization
Create src/main/java/com/example/monitoring/config/RedisConfig.java:
package com.example.monitoring.config;
import com.example.monitoring.dto.MonitoringSummary;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.Jackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
@Configuration
public class RedisConfig {
@Bean
RedisTemplate<String, MonitoringSummary> monitoringRedisTemplate(
RedisConnectionFactory connectionFactory,
ObjectMapper objectMapper
) {
RedisTemplate<String, MonitoringSummary> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory);
template.setKeySerializer(new StringRedisSerializer());
template.setValueSerializer(
new Jackson2JsonRedisSerializer<>(objectMapper, MonitoringSummary.class)
);
template.afterPropertiesSet();
return template;
}
}
Explicit JSON serialization keeps local values inspectable and avoids Java native serialization. The performance test includes deserialization and HTTP JSON serialization because they are part of the real endpoint cost.
Step 8: Implement Database Aggregation
Create src/main/java/com/example/monitoring/service/MonitoringAggregationService.java:
package com.example.monitoring.service;
import com.example.monitoring.dto.MonitoringSummary;
import com.example.monitoring.repository.MonitoringRepository;
import com.example.monitoring.repository.SeverityCount;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.time.Clock;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.LinkedHashMap;
import java.util.Map;
@Service
public class MonitoringAggregationService {
private final MonitoringRepository repository;
private final long windowHours;
private final Clock clock = Clock.systemUTC();
public MonitoringAggregationService(
MonitoringRepository repository,
@Value("${app.monitoring.summary-window-hours}") long windowHours
) {
this.repository = repository;
this.windowHours = windowHours;
}
@Transactional(readOnly = true)
public MonitoringSummary aggregate(String tenantId, String source) {
long started = System.nanoTime();
Instant generatedAt = clock.instant();
Instant windowStart = generatedAt.minus(windowHours, ChronoUnit.HOURS);
Map<String, Long> counts = new LinkedHashMap<>();
counts.put("CRITICAL", 0L);
counts.put("HIGH", 0L);
counts.put("MEDIUM", 0L);
counts.put("LOW", 0L);
for (SeverityCount row : repository.countBySeverity(tenantId, windowStart)) {
counts.put(row.getSeverity(), row.getTotal());
}
long total = counts.values().stream().mapToLong(Long::longValue).sum();
long open = repository.countOpen(tenantId, windowStart);
long assets = repository.countAffectedAssets(tenantId, windowStart);
long elapsedMs = (System.nanoTime() - started) / 1_000_000;
return new MonitoringSummary(
tenantId,
windowStart,
generatedAt,
generatedAt,
total,
open,
assets,
Map.copyOf(counts),
source,
elapsedMs,
1
);
}
}
All delivery paths use this one aggregation method. This prevents the scheduled path and cache-miss path from developing different business definitions.
Step 9: Implement the Redis Cache Layer
Create src/main/java/com/example/monitoring/service/MonitoringCache.java:
package com.example.monitoring.service;
import com.example.monitoring.dto.MonitoringSummary;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;
import java.time.Duration;
import java.util.Optional;
@Service
public class MonitoringCache {
private static final Logger log = LoggerFactory.getLogger(MonitoringCache.class);
private final RedisTemplate<String, MonitoringSummary> redisTemplate;
private final String prefix;
private final Duration ttl;
private final Counter hitCounter;
private final Counter missCounter;
private final Counter errorCounter;
public MonitoringCache(
RedisTemplate<String, MonitoringSummary> redisTemplate,
MeterRegistry meterRegistry,
@Value("${app.monitoring.cache-prefix}") String prefix,
@Value("${app.monitoring.cache-ttl}") Duration ttl
) {
this.redisTemplate = redisTemplate;
this.prefix = prefix;
this.ttl = ttl;
this.hitCounter = meterRegistry.counter("monitoring.cache.hit");
this.missCounter = meterRegistry.counter("monitoring.cache.miss");
this.errorCounter = meterRegistry.counter("monitoring.cache.error");
}
public Optional<MonitoringSummary> get(String tenantId) {
try {
MonitoringSummary value = redisTemplate.opsForValue().get(key(tenantId));
if (value == null) {
missCounter.increment();
return Optional.empty();
}
hitCounter.increment();
return Optional.of(value);
} catch (DataAccessException ex) {
errorCounter.increment();
log.warn("Redis read failed for tenant {}", tenantId);
return Optional.empty();
}
}
public void put(String tenantId, MonitoringSummary summary) {
try {
redisTemplate.opsForValue().set(key(tenantId), summary, ttl);
} catch (DataAccessException ex) {
errorCounter.increment();
log.warn("Redis write failed for tenant {}", tenantId);
}
}
public void evict(String tenantId) {
redisTemplate.delete(key(tenantId));
}
public String key(String tenantId) {
return prefix + ":tenant:" + tenantId + ":summary:24h";
}
}
Redis errors become misses. The request can continue through the database fallback, but the fallback will be protected by a concurrency limiter in the next step.
Step 10: Implement Single-Flight Backfill and Fallback Protection
Create src/main/java/com/example/monitoring/service/MonitoringQueryService.java:
package com.example.monitoring.service;
import com.example.monitoring.dto.MonitoringSummary;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Service;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
@Service
public class MonitoringQueryService {
private final MonitoringCache cache;
private final MonitoringAggregationService aggregationService;
private final ConcurrentHashMap<String, ReentrantLock> keyLocks =
new ConcurrentHashMap<>();
private final Semaphore fallbackPermits = new Semaphore(10);
private final Counter fallbackCounter;
private final Counter rejectedCounter;
public MonitoringQueryService(
MonitoringCache cache,
MonitoringAggregationService aggregationService,
MeterRegistry registry
) {
this.cache = cache;
this.aggregationService = aggregationService;
this.fallbackCounter = registry.counter("monitoring.fallback.total");
this.rejectedCounter = registry.counter("monitoring.fallback.rejected");
}
public MonitoringSummary getOptimized(String tenantId) {
long started = System.nanoTime();
return cache.get(tenantId)
.map(value -> value.deliveredFrom("REDIS", elapsedMs(started)))
.orElseGet(() -> backfillOnce(tenantId, started));
}
public MonitoringSummary refresh(String tenantId, String source) {
MonitoringSummary summary = aggregationService.aggregate(tenantId, source);
cache.put(tenantId, summary);
return summary;
}
public void evict(String tenantId) {
cache.evict(tenantId);
}
private MonitoringSummary backfillOnce(String tenantId, long requestStarted) {
ReentrantLock lock = keyLocks.computeIfAbsent(tenantId, ignored -> new ReentrantLock());
lock.lock();
try {
return cache.get(tenantId)
.map(value -> value.deliveredFrom("REDIS_AFTER_WAIT", elapsedMs(requestStarted)))
.orElseGet(() -> aggregateWithPermit(tenantId, requestStarted));
} finally {
lock.unlock();
if (!lock.hasQueuedThreads()) {
keyLocks.remove(tenantId, lock);
}
}
}
private MonitoringSummary aggregateWithPermit(String tenantId, long requestStarted) {
boolean acquired = false;
try {
acquired = fallbackPermits.tryAcquire(100, TimeUnit.MILLISECONDS);
if (!acquired) {
rejectedCounter.increment();
throw new IllegalStateException("Database fallback is temporarily busy");
}
fallbackCounter.increment();
MonitoringSummary summary = refresh(tenantId, "REAL_TIME_BACKFILL");
return summary.deliveredFrom("REAL_TIME_BACKFILL", elapsedMs(requestStarted));
} catch (InterruptedException ex) {
Thread.currentThread().interrupt();
throw new IllegalStateException("Fallback interrupted", ex);
} finally {
if (acquired) {
fallbackPermits.release();
}
}
}
private long elapsedMs(long started) {
return (System.nanoTime() - started) / 1_000_000;
}
}
The per-key lock prevents a local cache stampede. The semaphore prevents unlimited database fallback across different keys. Ten fallback operations are permitted concurrently. Additional requests wait for at most 100 milliseconds and then fail fast instead of filling every request thread and database connection.
This local lock coordinates one application process. Multiple instances require distributed single-flight coordination or a stale-while-revalidate strategy.
Step 11: Implement Scheduled Cache Refresh
Create src/main/java/com/example/monitoring/service/MonitoringRefreshScheduler.java:
package com.example.monitoring.service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.util.List;
@Component
public class MonitoringRefreshScheduler {
private static final Logger log = LoggerFactory.getLogger(MonitoringRefreshScheduler.class);
private final MonitoringQueryService queryService;
private final List<String> tenants;
public MonitoringRefreshScheduler(
MonitoringQueryService queryService,
@Value("${app.monitoring.tenants}") List<String> tenants
) {
this.queryService = queryService;
this.tenants = List.copyOf(tenants);
}
@Scheduled(
initialDelayString = "10000",
fixedDelayString = "${app.monitoring.refresh-delay-ms}"
)
public void refresh() {
for (String tenantId : tenants) {
try {
queryService.refresh(tenantId, "SCHEDULED_REFRESH");
log.info("Refreshed monitoring cache for tenant {}", tenantId);
} catch (Exception ex) {
log.error("Monitoring refresh failed for tenant {}", tenantId, ex);
}
}
}
}
The scheduler waits ten seconds after startup, fills the two tenant keys, and refreshes them every five minutes. The ten-minute TTL gives the scheduler one refresh interval of buffer.
Step 12: Implement Baseline and Optimized APIs
Create src/main/java/com/example/monitoring/controller/MonitoringController.java:
package com.example.monitoring.controller;
import com.example.monitoring.dto.MonitoringSummary;
import com.example.monitoring.service.MonitoringAggregationService;
import com.example.monitoring.service.MonitoringQueryService;
import jakarta.validation.constraints.Pattern;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.ResponseEntity;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@Validated
@RestController
@RequestMapping("/api/v1/tenants/{tenantId}/monitoring-summary")
public class MonitoringController {
private final MonitoringAggregationService aggregationService;
private final MonitoringQueryService queryService;
private final long baselineDelayMs;
public MonitoringController(
MonitoringAggregationService aggregationService,
MonitoringQueryService queryService,
@Value("${app.monitoring.baseline-delay-ms}") long baselineDelayMs
) {
this.aggregationService = aggregationService;
this.queryService = queryService;
this.baselineDelayMs = baselineDelayMs;
}
@GetMapping("/baseline")
public MonitoringSummary baseline(@PathVariable @Pattern(
regexp = "[a-zA-Z0-9-]{1,64}"
) String tenantId) throws InterruptedException {
long started = System.nanoTime();
Thread.sleep(baselineDelayMs);
MonitoringSummary summary = aggregationService.aggregate(tenantId, "DATABASE_BASELINE");
long elapsed = (System.nanoTime() - started) / 1_000_000;
return summary.deliveredFrom("DATABASE_BASELINE", elapsed);
}
@GetMapping
public MonitoringSummary optimized(@PathVariable @Pattern(
regexp = "[a-zA-Z0-9-]{1,64}"
) String tenantId) {
return queryService.getOptimized(tenantId);
}
@PostMapping("/refresh")
public MonitoringSummary refresh(@PathVariable String tenantId) {
return queryService.refresh(tenantId, "MANUAL_REFRESH");
}
@DeleteMapping("/cache")
public ResponseEntity<Void> evict(@PathVariable String tenantId) {
queryService.evict(tenantId);
return ResponseEntity.noContent().build();
}
}
The two-second delay exists only to reproduce the initial condition in a controlled development environment. Remove it when adapting the sample to a real service. The optimized endpoint does not contain an artificial delay.
Step 13: Build and Run the Application
Build the service:
mvn clean test
Start it with a fixed JVM heap so local test runs are easier to compare:
MAVEN_OPTS='-Xms512m -Xmx512m' mvn spring-boot:run
Check health:
curl -s http://localhost:8080/actuator/health
Wait at least ten seconds for scheduled cache warming. Confirm the key:
docker exec -it monitoring-redis redis-cli \
GET monitoring:v1:tenant:tenant-a:summary:24h
Check its TTL:
docker exec -it monitoring-redis redis-cli \
TTL monitoring:v1:tenant:tenant-a:summary:24h
Step 14: Compare Baseline and Redis Latency
Call the baseline endpoint:
curl -s -o /dev/null -w 'baseline=%{time_total}s\n' \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/baseline
The total should be slightly above two seconds because the development baseline includes an intentional two-second delay plus database aggregation.
Call the optimized endpoint:
curl -s -o /dev/null -w 'optimized=%{time_total}s\n' \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
After warm-up, the local target is under 50 milliseconds. A first request can be slower because of JVM class loading, JIT compilation, Redis connection creation, Docker scheduling, or cold operating-system caches. Run several calls before comparing steady-state latency:
for i in $(seq 1 10); do
curl -s -o /dev/null -w '%{time_total}\n' \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
done
Inspect the response body:
curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
deliverySource should be REDIS. processingTimeMs measures work inside the service. The curl total also includes local HTTP transport.
Step 15: Verify Cache Miss and Backfill
Evict the cache key:
curl -i -X DELETE \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/cache
Call the optimized endpoint once:
curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
The response should report REAL_TIME_BACKFILL. It will normally be slower than a Redis hit because PostgreSQL is calculating the summary.
Call it again:
curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
The second response should report REDIS, demonstrating that the miss result was backfilled.
Step 16: Verify Single-Flight Behavior
Evict the key again:
curl -s -X DELETE \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/cache
Send 20 requests concurrently:
seq 1 20 | xargs -n1 -P20 -I{} \
curl -s http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
One request should perform REAL_TIME_BACKFILL. Waiting requests should return from REDIS_AFTER_WAIT after the first request populates Redis. The database should not execute the same aggregate 20 times inside one application instance.
Check fallback metrics:
curl -s http://localhost:8080/actuator/metrics/monitoring.fallback.total
The counter should increase once for the single missing tenant key.
Step 17: Create the 500 QPS k6 Test
Create load-test/redis_500_qps.js:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
redis_read_path: {
executor: 'constant-arrival-rate',
rate: 550,
timeUnit: '1s',
duration: '60s',
preAllocatedVUs: 100,
maxVUs: 400,
},
},
thresholds: {
http_req_failed: ['rate<0.01'],
http_req_duration: ['p(95)<50', 'p(99)<100'],
checks: ['rate>0.99'],
},
};
export default function () {
const response = http.get(
'http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary',
{ tags: { path: 'redis-monitoring-summary' } }
);
check(response, {
'status is 200': (r) => r.status === 200,
'response comes from redis': (r) => {
try {
return JSON.parse(r.body).deliverySource === 'REDIS';
} catch (e) {
return false;
}
},
});
}
The test requests 550 iterations per second for 60 seconds so that a result above 500 QPS remains possible after minor scheduling variation. It requires fewer than one percent failed requests, p95 under 50 milliseconds, p99 under 100 milliseconds, and more than 99 percent successful checks.
Warm the key before testing:
curl -s -X POST \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/refresh \
>/dev/null
Run the test:
k6 run load-test/redis_500_qps.js
Review http_reqs, http_req_failed, http_req_duration, dropped iterations, and check rate. A sustainable result requires the requested arrival rate to be maintained without a growing number of dropped iterations.
If k6 and the application run on the same small laptop, the load generator competes with Java, Redis, PostgreSQL, and Docker. Increase Docker CPU and memory allocation or run k6 from another machine for a cleaner measurement.
Step 18: Add a Mixed Hit-and-Miss Test
Create load-test/mixed_path.js:
import http from 'k6/http';
import { check } from 'k6';
export const options = {
scenarios: {
mixed_path: {
executor: 'constant-arrival-rate',
rate: 200,
timeUnit: '1s',
duration: '60s',
preAllocatedVUs: 50,
maxVUs: 200,
},
},
thresholds: {
http_req_failed: ['rate<0.02'],
http_req_duration: ['p(95)<100'],
},
};
export default function () {
const tenant = Math.random() < 0.99 ? 'tenant-a' : `cold-${__VU}-${__ITER}`;
const response = http.get(
`http://localhost:8080/api/v1/tenants/${tenant}/monitoring-summary`
);
check(response, { 'status is 200': (r) => r.status === 200 });
}
This test creates approximately one percent cold keys. At 200 QPS, that is around two database fallbacks per second. It illustrates that cache hit ratio must be interpreted with miss cost.
Run it:
k6 run load-test/mixed_path.js
Inspect database pool and fallback metrics while it runs:
curl -s http://localhost:8080/actuator/metrics/hikaricp.connections.pending
curl -s http://localhost:8080/actuator/metrics/monitoring.fallback.total
curl -s http://localhost:8080/actuator/metrics/monitoring.fallback.rejected
Step 19: Inspect HTTP and Cache Metrics
Inspect HTTP duration metrics:
curl -s http://localhost:8080/actuator/metrics/http.server.requests
Filter by URI if the available tags include the endpoint template:
curl -s 'http://localhost:8080/actuator/metrics/http.server.requests?tag=uri:/api/v1/tenants/{tenantId}/monitoring-summary'
Inspect cache counters:
curl -s http://localhost:8080/actuator/metrics/monitoring.cache.hit
curl -s http://localhost:8080/actuator/metrics/monitoring.cache.miss
curl -s http://localhost:8080/actuator/metrics/monitoring.cache.error
Inspect Prometheus output:
curl -s http://localhost:8080/actuator/prometheus | \
grep -E 'http_server_requests|monitoring_cache|monitoring_fallback|hikaricp'
The optimized load test should produce a very high Redis hit ratio. If misses increase unexpectedly, check TTL, scheduled refresh logs, Redis eviction, and cache key construction.
Step 20: Inspect PostgreSQL Query Plans
Enable pg_stat_statements for deeper local analysis by adding this command to the PostgreSQL service in compose.yaml:
command:
- postgres
- -c
- shared_preload_libraries=pg_stat_statements
Recreate the PostgreSQL container after changing the command:
docker compose down
docker compose up -d
Create the extension:
docker exec -it monitoring-postgres \
psql -U monitoring_user -d monitoring \
-c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;"
Inspect expensive statements:
docker exec -it monitoring-postgres \
psql -U monitoring_user -d monitoring \
-c "SELECT calls, total_exec_time, mean_exec_time, rows, query FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;"
Inspect the severity aggregation plan:
docker exec -it monitoring-postgres \
psql -U monitoring_user -d monitoring \
-c "EXPLAIN (ANALYZE, BUFFERS) SELECT severity, COUNT(*) FROM security_alert WHERE tenant_id = 'tenant-a' AND detected_at >= CURRENT_TIMESTAMP - INTERVAL '24 hours' GROUP BY severity;"
A development dataset may use a sequential scan when PostgreSQL estimates that a large percentage of the table must be read. The indexes are still useful for narrower windows and more selective predicates. Do not force index use without evidence.
Step 21: Profile the Java Application
Start Java Flight Recorder for a running process. Find the process ID:
jcmd
Start a 60-second recording while k6 runs:
jcmd <PID> JFR.start \
name=monitoring-load \
duration=60s \
filename=/tmp/monitoring-load.jfr \
settings=profile
Run the k6 test during the recording. Open /tmp/monitoring-load.jfr in JDK Mission Control.
Inspect CPU hot methods, allocation pressure, lock contention, socket activity, garbage collection, and thread states. If JSON parsing dominates the Redis path, reduce payload size or evaluate a more compact serializer. If lock contention dominates, confirm that the load test is not repeatedly evicting the same key.
Step 22: Verify Redis Failure Fallback
Stop Redis:
docker compose stop redis
Call the optimized endpoint:
curl -i http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary
The cache read should fail quickly and the request should use database fallback. Repeated high concurrency during this state is intentionally limited by the ten-permit semaphore.
Do not run the 550 QPS Redis-only test while Redis is stopped. That test is designed to measure the Redis path. Use a small controlled test to observe fallback protection:
seq 1 50 | xargs -n1 -P50 -I{} \
curl -s -o /dev/null -w '%{http_code}\n' \
http://localhost:8080/api/v1/tenants/cold-{}/monitoring-summary
Some requests may receive HTTP 500 because the sample maps the busy fallback exception through Spring's default handler. A practical application should add a @RestControllerAdvice that maps this condition to HTTP 503 with a stable error code.
Restart Redis:
docker compose start redis
Warm the cache again:
curl -s -X POST \
http://localhost:8080/api/v1/tenants/tenant-a/monitoring-summary/refresh \
>/dev/null
Step 23: Tune the Local Environment
Keep the cache TTL longer than the scheduled refresh interval. A five-minute refresh and ten-minute TTL prevent normal expiry between successful scheduler runs.
Do not increase Tomcat threads, HikariCP connections, or k6 virtual users without checking downstream capacity. More concurrency can create longer queues rather than more throughput.
Use a separate k6 host when validating sustained throughput. If everything runs on one machine, record the Docker CPU and memory allocation with the result.
Keep payloads small. The sample summary is compact. Adding large alert lists to the same response will increase Redis transfer, Jackson work, network time, and client parsing.
Use canonical cache keys. Do not include unbounded raw query strings. Normalize supported filters and hash them when necessary.
Do not cache arbitrary low-reuse requests. A large key space can consume memory while producing a poor hit ratio.
For faster local experimentation, change the scheduler delay to 30 seconds:
app:
monitoring:
refresh-delay-ms: 30000
Restore five minutes after observing the refresh behavior.
Step 24: Add Focused Development Tests
Create src/test/java/com/example/monitoring/service/MonitoringSummaryTest.java:
package com.example.monitoring.service;
import com.example.monitoring.dto.MonitoringSummary;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class MonitoringSummaryTest {
@Test
void changesDeliveryMetadataWithoutChangingBusinessValues() {
MonitoringSummary original = new MonitoringSummary(
"tenant-a",
Instant.parse("2026-11-18T10:00:00Z"),
Instant.parse("2026-11-19T10:00:00Z"),
Instant.parse("2026-11-19T10:00:00Z"),
100,
80,
25,
Map.of("CRITICAL", 5L, "HIGH", 20L),
"SCHEDULED_REFRESH",
120,
1
);
MonitoringSummary delivered = original.deliveredFrom("REDIS", 3);
assertThat(delivered.totalAlerts()).isEqualTo(100);
assertThat(delivered.openAlerts()).isEqualTo(80);
assertThat(delivered.deliverySource()).isEqualTo("REDIS");
assertThat(delivered.processingTimeMs()).isEqualTo(3);
assertThat(delivered.generatedAt()).isEqualTo(original.generatedAt());
}
}
Run tests:
mvn test
These are local development tests. They are not production acceptance tests.
Step 25: Reset the Local Environment
Stop the Spring Boot application with Ctrl+C.
Stop PostgreSQL and Redis while preserving data:
docker compose stop
Restart them later:
docker compose start
Remove containers while preserving named volumes:
docker compose down
Remove containers and all local data:
docker compose down -v
Remove Maven build output:
mvn clean
The completed local implementation demonstrates the performance transition clearly. The baseline endpoint repeats database work and deliberately reproduces a response above two seconds. The scheduled process prepares a reusable summary. Redis serves the steady-state API through a compact key and value. Cache misses use one controlled database calculation and backfill the result. Single-flight locking prevents repeated work for the same key. A semaphore limits fallback pressure. Micrometer exposes the fast and slow paths separately. k6 provides a repeatable method for testing more than 500 Redis-only queries per second and sub-50-millisecond p95 latency under the declared local workload.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.