Many organizations run critical applications as microservices on Kubernetes. These applications are often distributed across multiple services, such as order management, payments, inventory, fulfillment, and customer-facing APIs. While Oracle Kubernetes Engine (OKE) provides a strong foundation for running and scaling these workloads, operations teams still face a common challenge: understanding whether a technical issue is actually affecting the business, as well as correlating and understanding this information.
The OKE platform can be running, but an order flow can still be degraded. A deployment can exist, but a new version might be failing because of an image pull error. An Horizontal Pod Autoscaling HPA can reach its maximum number of replicas, but that condition only represents a real scaling issue if CPU utilization is still above the configured target. Traditional infrastructure monitoring is necessary, but it is not always enough to explain business impact.
This blog explores the architecture that connects Oracle Kubernetes Engine, kagent, OCI Generative AI, and OCI DevOps to create a business-aware operations model. The goal is to show how AI-assisted diagnostics can help platform and application teams identify real incidents, understand root cause, evaluate business impact, and trigger governed remediation workflows.
Business scenario
A business application is composed of three microservices running on Oracle Kubernetes Engine:
- Orders: manages the order flow and exposes business observability signals.
- Inventory: validates product availability and supports the order process.
- Payments: processes payment activity and supports order completion.
From a platform point of view, these services run as Kubernetes workloads. From a business point of view, they represent an end-to-end transaction path. If one service is degraded, the impact can appear as pending orders, failed transactions, manual reviews, revenue at risk, or poor customer experience.
The key idea is simple: Kubernetes health must be correlated with business health.
Solution overview
Architecture design
Logical flow of the solution:
The solution uses a Python-based remediator deployed as a Kubernetes CronJob. The CronJob runs periodically and performs three independent checks:
- Business application health
- HPA autoscaling behavior
- ImagePull failures
For each check, the remediator sends a focused prompt to kagent through Model Context Protocol (MCP), kagent uses its Kubernetes and observability capabilities to inspect the environment and can use OCI Generative AI to reason over the collected signals. The expected result is a structured JavaScript Object Notation (JSON) response that includes status, root cause, evidence, business impact, and recommended action.
When a real issue is detected, the remediator generates a diagnostic report and triggers an OCI DevOps deployment pipeline designed to address the problem. The pipeline is not executed automatically without oversight. A human approval step (Human in the Middle) is included so that an operator can review the diagnostic evidence and approve or reject the remediation before any action is taken.
Operational Issues Detected
Why guardrails matter
AI can help accelerate diagnosis, but operational decisions require control. For that reason, the remediator uses deterministic guardrails before sending notifications or triggering remediation.
For Horizontal Pod Autoscaler (HPA), a problem is only classified as an autoscaling failure when both conditions are true:
desiredReplicas == maxReplicas
AND
currentCpuUtilization > targetCpuUtilization
If the deployment is at its maximum number of replicas but CPU is below the target, the condition is not treated as an HPA failure.
For ImagePull, the remediator validates active Kubernetes evidence such as:
ErrImagePull
ImagePullBackOff
InvalidImageName
CreateContainerConfigError
If all current pods are running and no active image pull condition is observed, the system does not classify the application as having an ImagePull failure.
For business application health, the remediator evaluates real business observability data such as pending review orders, manual review rate, confirmed orders, failed orders, revenue at risk, and service dependency status. This prevents the system from treating every technical symptom as a business incident.
Governed remediation with OCI DevOps
When the system detects a validated issue, it sends an email report containing:
- Executive summary
- Root cause
- Business impact
- Evidence
- Recommended action
- OCI DevOps remediation approval link
This creates a governed remediation process. kagent helps diagnose. OCI Generative AI helps explain. The remediator validates the evidence. OCI DevOps manages the execution path. A human operator remains in control of the final remediation decision.
Diagnostic Emails Sent for Human-in-the-Middle Remediation
OCI DevOps Remediation Pipelines
Operation flow
The CronJob runs every few minutes and evaluates the application and platform state. If the Orders flow is healthy, no action is required. If the application shows business degradation, the system prepares a business-focused diagnostic report. If an ImagePull error prevents a new version from becoming ready, the ImagePull remediation path takes priority. If HPA reaches its maximum replicas while CPU remains above target, the HPA remediation path is triggered.
This approach helps avoid unnecessary alerts and focuses attention on incidents that have real evidence and potential business impact.
Conclusion
Cloud-native operations are moving beyond infrastructure status. Platform teams need to understand how Kubernetes behavior affects customer experience, revenue, and service continuity.
By combining Oracle Kubernetes Engine, kagent, OCI Generative AI, and OCI DevOps, organizations can build a more intelligent and governed operations model. The result is not just faster diagnosis, but better decision-making: incidents are evaluated with context, remediations are controlled, and business-critical applications can be protected with greater confidence.