From LLM Tool Calling to Deterministic AIOps: Lessons from Kubernetes Troubleshooting
From LLM Tool Calling to Deterministic AIOps: Lessons from Kubernetes Troubleshooting How three local LLM experiments changed the way I think about AI-assisted operations, evidence validation, and reliable troubleshoot
From LLM Tool Calling to Deterministic AIOps: Lessons from Kubernetes Troubleshooting
How three local LLM experiments changed the way I think about AI-assisted operations, evidence validation, and reliable troubleshooting.
Introduction
AI agents are becoming increasingly popular in infrastructure operations.
The idea is attractive: connect a language model to monitoring tools, let it investigate incidents, and ask it to explain what went wrong.
But there is an important question:
Can an AI agent reliably distinguish between what the system actually observed and what it assumes happened?
I wanted to explore that question through a practical experiment.
As part of my SRE Reliability Lab, I built a small AIOps proof of concept focused on Kubernetes readiness troubleshooting.
The initial objective was to evaluate whether a small local language model could:
- Select and call diagnostic tools.
- Collect information about a Kubernetes incident.
- Reference structured evidence.
- Classify the incident correctly.
- Recognize when the available information was insufficient.
After three controlled experiments, the results led me in a different direction.
Instead of continuing to improve the prompts, I built a deterministic troubleshooting foundation designed to support a future, optional AI layer.
This article explains the process, the failures, and the engineering decisions behind that change.
1. Understanding the Kubernetes problem
A Kubernetes Pod can be running without being ready to receive traffic.
These are different conditions.
The Pod phase describes its lifecycle state, while readiness indicates whether it should receive traffic through the relevant Service endpoints.
For example:
Pod phase: Running
Ready: false
This does not necessarily mean the container has crashed.
The application might be running but failing its readiness probe.
A useful investigation needs to examine several sources of information:
- Pod status and readiness.
- Kubernetes events.
- Endpoint readiness.
Consider the following fictional incident:
Pod:
Phase: Running
Ready: false
Events:
Reason: Unhealthy
Message: Readiness probe failed
Endpoints:
Ready endpoints: 0
Together, these observations support a specific diagnosis: the Pod is running but not ready, and no ready endpoint was observed in that snapshot.
However, the exact root cause may still be unknown.
A readiness probe failure could result from application startup delays, configuration issues, unavailable dependencies, or other conditions.
Observing a failed readiness probe does not automatically prove why the probe failed.
That distinction became central to the experiment.
2. Building a deterministic reference
Before introducing AI, I created a small Python implementation using fictional Kubernetes snapshots stored in a versioned JSON fixture.
The lab exposes three read-only tools:
get_pod_status(...)
get_events(...)
get_endpoints(...)
Each tool returns a structured result containing:
{
"tool": "get_pod_status",
"state": "ok",
"evidence": {
"phase": "Running",
"ready": false
}
}
The tools do not connect to a real Kubernetes cluster.
They operate exclusively on local test data.
I also implemented a deterministic reference function:
diagnose(...)
It uses fixed rules to classify the collected evidence.
Three scenarios were defined:
| Scenario | Evidence | Expected classification |
|---|---|---|
| Running/NotReady | Pod NotReady, readiness failure event, no ready endpoints | running_not_ready |
| Healthy | Pod Ready, matching ready endpoint | healthy |
| Incomplete | Pod NotReady, events and endpoints unavailable | insufficient_evidence |
This provided an important baseline.
I could now compare the model's conclusions with a predictable reference.
The goal was not to prove that the reference rules could diagnose every Kubernetes problem.
The goal was to create a small, reproducible experiment where incorrect conclusions could be identified.
3. Adding a local LLM
For the AI experiment, I used:
- Ollama as the local inference runtime.
- Qwen2.5 1.5B Instruct, quantized to Q4_K_M.
- CPU-only inference.
- Python standard-library tooling.
- Three fictional incident scenarios.
I selected a small model because I wanted to understand what could run on a modest local development machine.
The model was allowed to request only the three predefined diagnostic tools.
It had no access to production systems, credentials, a real Kubernetes cluster, or infrastructure-changing commands.
The expected workflow was simple:
Incident
|
v
Local LLM
|
v
Structured tool calls
|
+-- get_pod_status
|
+-- get_events
|
+-- get_endpoints
|
v
Collected evidence
|
v
LLM classification
|
v
Independent validation
Initially, I focused on whether the model could use tools correctly.
But successful tool calling turned out to be only one part of the problem.
4. Three experiments, three important lessons
I performed three controlled evaluations, improving the output contract between them.
Experiment 1: Tool calling worked, but inconsistently
In the first experiment, the model completed six of the nine expected tool calls.
The healthy scenario was classified correctly.
However, the Running/NotReady scenario encountered a timeout during the initial model-loading process, and the incomplete scenario included unsupported conclusions.
The model also struggled to consistently produce the required structured output.
This suggested that I needed stronger execution controls and a more explicit response contract.
Experiment 2: Better structure did not eliminate hallucinations
In the second experiment, I introduced stricter validation and structured JSON responses.
The results improved:
- 9 of 9 expected tool calls succeeded.
- All 3 responses followed the JSON schema.
- 2 of 3 classifications matched the deterministic reference.
However, none of the responses passed the complete evidence validation.
The model generated incorrect paths to observations.
More importantly, in the incomplete scenario, it still made claims about readiness failures and unavailable endpoints without supporting evidence.
The problem was no longer just output formatting.
It was the relationship between evidence and conclusions.
Experiment 3: Valid evidence references, incorrect decisions
For the third experiment, I simplified the evidence contract.
Instead of asking the model to construct JSON paths, the executor generated short observation IDs.
The model only needed to select relevant IDs.
This eliminated the earlier reference-formatting problem.
The results were:
- 9 of 9 valid tool calls.
- 3 of 3 schema-compliant JSON responses.
- 8 of 8 evidence IDs valid.
- Only 1 of 3 correct classifications.
- Only 1 of 3 responses accepted without repair.
The two incorrect classifications were particularly interesting.
In the Running/NotReady scenario, the model selected healthy even though the referenced evidence described a readiness failure and no ready endpoints.
In the incomplete scenario, the model recognized that events and endpoints were missing, but still selected running_not_ready.
That classification required evidence that was not available.
Comparing the experiments
| Metric | Experiment 1 | Experiment 2 | Experiment 3 |
|---|---|---|---|
| Valid tool calls | 6/9 | 9/9 | 9/9 |
| JSON-compliant responses | 1/3 | 3/3 | 3/3 |
| Correct classifications | 1/3 | 2/3 | 1/3 |
| Responses accepted by evidence validation | Not fully established | 0/3 | 1/3 |
The experiment became more reliable at the interface level, but classification accuracy did not improve consistently.
The sample was intentionally small. These numbers should not be treated as a general benchmark of Qwen2.5 or other language models.
They describe the behavior observed in this specific workflow.
Nevertheless, the engineering lesson was clear:
Correct tool calls and valid JSON do not guarantee correct operational decisions.
5. Missing evidence is not negative evidence
This was probably the most important lesson from the project.
Imagine an endpoint query returning:
{
"tool": "get_endpoints",
"state": "ok",
"evidence": []
}
The query succeeded.
It observed an empty list.
Now compare it with:
{
"tool": "get_endpoints",
"state": "missing_evidence",
"evidence": null
}
These two results have different meanings.
The first says that no endpoints were returned in the successful query.
The second says the information is unavailable.
In the second situation, the system cannot conclude that there are no ready endpoints.
The same principle applies to events.
An unavailable event source does not mean that no events occurred.
It also does not prove that a readiness probe failed.
Unknown is a valid operational state.
A reliable diagnostic system must preserve uncertainty instead of replacing it with a plausible explanation.
6. Changing direction: deterministic AIOps
After the third experiment, I decided to stop refining autonomous classification with that model for this proof of concept.
The next step was not to add more prompts, tools, or frameworks.
It was to make the trustworthy part of the system the foundation.
I implemented Mode A, a deterministic troubleshooting assistant.
The architecture looks like this:
Fictional incident
|
v
Read-only collection
|
+-------+-------+
| | |
v v v
Pod Events Endpoints
| | |
+-------+-------+
|
v
Frozen tool results
|
+-------+-------+
| |
v v
diagnose() Evidence catalog
| |
+-------+-------+
|
v
Structured JSON report
|
v
Mode A
No language model used
The implementation uses Python's standard library.
It does not require Ollama, a framework for AI agents, an external API, or a Kubernetes cluster.
A single collection snapshot
One design decision was especially important.
Each diagnostic tool should be called only once per incident.
The collected results are preserved and reused by both the deterministic diagnosis and the evidence catalog.
This avoids a situation where different components accidentally evaluate different observations.
The original diagnose() function remains responsible for classification.
Its rules were not copied into a second implementation.
Immutable evidence catalog
The evidence catalog gives each observation a stable identifier.
Each entry is linked to the original tool result, including its source, state, and value.
For example:
demo-c:1 -> Pod status observation
demo-c:2 -> Missing events evidence
demo-c:3 -> Missing endpoints evidence
The catalog preserves the difference between observed facts and unavailable information.
It also allows reports to provide traceability without inventing explanations.
7. Running Mode A locally
The implementation is available in the repository:
github.com/roger-oliveira86/sre-reliability-lab
To execute the incomplete scenario from the repository root:
python3 -B -m poc_001 \
--mode deterministic \
--incident-id demo-c \
--scenario incomplete \
--namespace fictional-lab \
--pod demo-unknown \
--service demo-web
The -B option prevents Python from writing bytecode files.
The command produces a structured JSON report.
Its main fields include:
schema_version
incident
mode
collection_quality
diagnosis
observations
gaps
errors
ai
timings
For the incomplete scenario, the report preserves:
- The observed Pod state: Running/NotReady.
- Missing events evidence.
- Missing endpoints evidence.
- The
insufficient_evidenceclassification.
The AI section explicitly identifies that no model was involved:
{
"ai": {
"state": "not_used"
}
}
This is intentional.
A deterministic result should never be presented as an AI-generated diagnosis.
8. Collection quality and error handling
Another important part of Mode A is distinguishing incident uncertainty from collection failures.
The report includes a collection_quality field with three possible values.
| Value | Meaning |
|---|---|
complete |
All three queries completed successfully |
missing_evidence |
At least one source lacked evidence, with no operational errors |
operational_error |
At least one collection operation failed |
The distinction matters because not every incomplete diagnosis has the same cause.
A missing source in a fictional snapshot is different from a timeout or an unavailable fixture.
The CLI also uses explicit exit codes:
-
0: Collection completed without an operational error, including cases with missing evidence. -
1: A collection error occurred. -
2: Input validation failed.
This makes the tool suitable for repeatable local tests and future automation experiments.
9. Validation and integration
I added tests covering:
- The three original incident scenarios.
- Exactly one query per source.
- Reuse of collected results.
- Evidence traceability and immutability.
- Empty observations versus missing evidence.
- Operational errors and timeouts.
- CLI output and exit codes.
- Independence from network access and AI runtimes.
At the time of integration, the full repository suite passed 36 tests.
The existing CI also successfully completed Kubernetes schema validation, container build, and application smoke tests.
The POC implementation remained isolated from the existing HTTP service and container image.
The deterministic Mode A was merged into the repository, followed by an English-language documentation update.
10. What about AI-assisted troubleshooting?
I still see value in using AI for SRE and Platform Engineering.
However, this experiment changed where I would place that responsibility.
For a future hybrid architecture, I would separate the components as follows.
| Responsibility | Component |
|---|---|
| Collect diagnostic signals | Deterministic tools |
| Validate observed facts | Evidence catalog and validators |
| Classify supported incident states | Deterministic diagnosis |
| Explain findings | Optional AI assistant |
| Suggest next investigation steps | Optional AI assistant |
| Reject unsupported claims | Deterministic validation |
| Apply infrastructure changes | Outside this POC |
The AI layer could receive an already validated diagnosis and propose limited hypotheses or next checks.
It should not silently change the classification or present a suggestion as a confirmed fact.
For example, if the evidence confirms a readiness failure event, the AI could suggest reviewing the readiness probe configuration.
But if the event source is unavailable, it should not claim that the probe failed.
The distinction between facts, hypotheses, and missing information must remain visible.
Mode B, the optional AI-assisted layer, is an architectural proposal. It has not been implemented in this project.
11. Lessons for SRE and Platform Engineering
This project reinforced several principles that I believe are useful beyond Kubernetes troubleshooting.
1. Tool calling is a capability, not a reliability guarantee
An LLM can invoke the correct tools and still misunderstand the results.
Successful execution does not mean successful reasoning.
2. Structured output is necessary but insufficient
JSON schemas can control response format.
They cannot automatically guarantee that the conclusions are supported by the data.
Semantic validation is still required.
3. Missing information must remain missing
A model should not transform absent evidence into a convenient explanation.
Uncertainty is part of the diagnosis.
4. Deterministic systems provide valuable boundaries
Known rules, bounded tools, and reproducible snapshots make troubleshooting easier to validate and debug.
They also provide a foundation for evaluating future AI components.
5. AI should provide measurable value
Adding a model introduces complexity, runtime requirements, and new failure modes.
Before doing so, it is worth asking what the model can contribute that the existing system cannot already deliver reliably.
Conclusion
I started POC-001 with a question:
Can a small local language model reliably investigate and classify Kubernetes readiness incidents?
In this controlled experiment, the answer was not reliable enough for autonomous classification.
The model learned to call the tools correctly and produce valid structured responses, but it continued to make unsupported classification decisions.
Instead of treating those failures as reasons to add more prompts indefinitely, I changed the architecture.
The result is a small, deterministic AIOps foundation with traceable evidence, explicit uncertainty, predictable classification, and no dependency on a language model.
For me, this is a useful direction for AI-assisted operations.
Not every part of an operational workflow needs to be intelligent.
Some parts need to be predictable, observable, and trustworthy.
Build reliability first. Add AI where it creates measurable value.
Explore the project
GitHub repository: SRE Reliability Lab
Implementation: poc_001/
Technology: Python, Kubernetes troubleshooting concepts, structured evidence, deterministic diagnosis, and experimental local LLM tool calling.
This project uses fictional incident data and does not interact with production Kubernetes environments.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.