🔎 Building IncidentCopilot: Designing a Deterministic Log Correlation Engine

Milestone 4 of IncidentCopilot — an open-source, local-first AI DevOps Incident Copilot

📌 Introduction

Logs are plentiful during outages — the challenge is understanding what they mean together.

• Nginx → HTTP 502 errors
• Applications → DB connection timeouts
• Docker → container exits
• Kubernetes → failed readiness probes

IncidentCopilot is being built to help developers collect, correlate, and explain logs before introducing AI.

Milestone 4 focused on implementing a deterministic log correlation engine that evaluates signals and produces explainable relationships.

1️⃣ From Ingestion to Correlation

Milestone 3 handled ingestion for five sources: Nginx, Kubernetes, Docker, application logs, GitHub Actions.

Correlation adds context: are events related, or just coincidental?

2️⃣ Common Event Representation

The engine uses a typed event contract:

• id
• source
• timestamp
• severity
• service
• event_type
• message
• metadata
• raw_message

This keeps correlation logic independent of the database.

3️⃣ Correlation Signals

Five weighted signals drive relevance:

• Temporal proximity (25%) → closeness in time
• Service relationships (25%) → shared service identities
• Error signatures (25%) → recognizable failure patterns (connection refused, CrashLoopBackOff, HTTP 502)
• Severity weighting (15%) → critical events matter more
• Causal ordering (10%) → earlier warnings/errors may precede failures

4️⃣ Calculating Relevance

Weights are explicit heuristics, not probabilities:

Formula:

S = 0.25St + 0.25Sv + 0.25Se + 0.15Ss + 0.10So

Scores are capped at 1.0. Timing alone does not create correlations.

5️⃣ Explainability

The engine returns structured signals:

• log_id
• related_log_id
• relevance_score
• relationship_type
• reason

Explanations include time differences, matching services, shared error signatures, and ordering.

This makes correlations transparent and inspectable.

6️⃣ Independence from the Database

The correlation engine:

• Does not query PostgreSQL
• Does not call LLMs
• Does not use vector DBs

Benefits:

• Testability with fixtures
• Predictability
• Separation of concerns
• Flexibility for future integration

7️⃣ Testing the Engine

Coverage included:

• Temporal correlation
• Error-signature matching
• Severity weighting
• Service correlation
• Negative tests (unrelated events)
• Ordering explanations
• Invalid window values
• Timezone-aware timestamps
• Score bounds

🚫 What It Does Not Do

At Milestone 4, the engine does not:

• Prove root cause
• Understand arbitrary error messages semantically
• Use LLMs for relationships
• Build full incident timelines

Its role: produce deterministic correlation signals for future workflows.

📚 Lessons Learned

1. Normalization is prerequisite for cross-source analysis.
2. Explainability must be part of the output.
3. Deterministic ≠ infallible.
4. Correlation ≠ causation.
5. Incremental development controls complexity.

🔮 What’s Next?

Milestone 5 → Incident Timeline & Lifecycle.

This will organize correlated events into timelines and introduce lifecycle behavior. Later milestones will add:

• Retrieval-augmented generation (Qdrant)
• Structured diagnosis (Ollama)
• Frontend for investigation

🏁 Conclusion

Milestone 4 delivered a deterministic correlation engine that evaluates temporal proximity, service identity, error signatures, severity, and ordering.

It produces structured scores and explanations, distinguishing possible relationships from proven causation.

IncidentCopilot is being built milestone by milestone, with local-first operation, testability, and transparency as core priorities.

🔗 GitHub: github.com/richardatodo/incidentcopilot

➡️ Next: Milestone 5 — Incident Timeline & Lifecycle