Kentroller is the in-cluster control-plane component that coordinates Cast AI's node rebalancing and automation features with Karpenter. It runs inside your cluster as a Kubernetes controller, watches Cast AI custom resources, and drives optimization workflows by interacting directly with Karpenter's NodeClaims and NodePools.

What Kentroller does

Kentroller is responsible for:

• Generating and reconciling rebalancing plans using CRDs in the autoscaling.cast.ai/v1alpha API group
• Handling claim- and schedule-driven rebalancing flows
• Running continuous in-cluster rebalancing to find cost-saving node replacements
• Replacing spot instances proactively based on interruption predictions
• Coordinating node consolidation with Karpenter NodeClaim deletion

How it works

Component interactions

Kentroller interacts with the following systems:

Kentroller maintains a persistent connection to the Cast AI backend for real-time plan coordination. This connection is automatically re-established if it drops.

CRDs managed by Kentroller

Kentroller defines and reconciles the following Custom Resource Definitions (CRDs) in the autoscaling.cast.ai/v1alpha API group:

Feature gating

All Kentroller features are enabled and disabled dynamically through the Cast AI console without restarting the controller.

Rebalancing workflows

Kentroller supports three ways to initiate rebalancing:

Schedule-driven rebalancing

When a RebalancePlanSchedule fires, Kentroller creates a RebalancePlanClaim from the schedule template and submits it to Cast AI for plan generation. For details, see Scheduled rebalancing for Karpenter clusters.

Claim-driven rebalancing

A RebalancePlanClaim can also be created manually without a schedule. This triggers a single on-demand rebalancing operation with the configuration specified in the claim's spec.

Continuous in-cluster rebalancing

The continuous rebalancing controller runs periodically inside the cluster and optimizes nodes without calling the Cast AI backend for plan generation. It works entirely from local cluster state and AWS pricing data:

1. Resource collection — Kentroller reads all Nodes, NodeClaims, NodePools, Pods, DaemonSets, PodDisruptionBudgets, and EC2NodeClasses from the cluster.
2. Node analysis — Nodes are classified as candidates for replacement based on age, workload compatibility, and NodePool constraints. Nodes with blocking workloads (such as local PVs or pods without a controller) are excluded unless aggressive mode is configured.
3. Optimization search — Kentroller evaluates candidate node sets to find the largest subset of nodes that can be replaced for a net cost saving.
4. Savings validation — A plan is only executed if it meets both a savings percentage threshold (default: 0%) and an absolute monthly savings threshold (default: $50/month).
5. Plan creation — A RebalancePlan is created directly in the cluster without a RebalancePlanClaim. Kentroller then executes the plan by provisioning replacement NodeClaims and deleting the originals.

The cycle repeats on a configurable polling period (default: 10 seconds). Only one active plan can exist at a time — if a plan is already running, the cycle skips until it completes.

Spot interruption prediction

The spot interruption prediction service polls the Cast AI ML API at a configurable interval (default: every 1 minute) to identify spot nodes at risk of interruption. When a node is predicted to be interrupted:

1. Kentroller labels the node with autoscaling.cast.ai/predicted-interruption=true
2. A RebalancePlanClaim is created with the at-risk node in its scope, triggering proactive replacement
3. A SpotInterruptionPredicted event is emitted on the node
4. After replacement, a cooldown period (default: 15 minutes) prevents the same node from being replaced again immediately

This gives workloads significantly more lead time before the actual interruption compared to AWS's standard two-minute warning.

Rebalance execution

Plan lifecycle

Once a RebalancePlanClaim is created, it progresses through the following states:

Execution phases

When a RebalancePlan runs, Kentroller proceeds in two phases:

Creation phase — New NodeClaims are provisioned through Karpenter. If a provisioning attempt fails due to insufficient capacity, Kentroller retries with the next available instance type from the NodePool's requirements. Each attempt is recorded in status.nodesCreation:

Deletion phase — Old NodeClaims are drained and deleted. If container live migration is available, Kentroller creates a RebalanceMigrationPlan to move workloads without restarts before deleting the node. Deletion events are recorded in status.nodesDeletion:

Savings validation

If executionPolicy.achievedSavingsPercentageThreshold is set on the claim, Kentroller validates actual savings after new nodes are provisioned but before old nodes are deleted. If the realized savings do not meet the threshold, the plan is aborted with FailureReasonInsufficientSavings.

Validated savings are stored in status.validatedSavings on the RebalancePlan. If a fallback instance type was used due to insufficient capacity, the originally planned price is used as a conservative estimate.

Failure recovery

When a plan fails, Kentroller records which phase caused the failure in status.failurePhase to ensure correct cleanup after a controller restart:

Savings reporting

After plan generation, Cast AI populates status.savings on the RebalancePlan:

Node consolidation

The node consolidation controller watches Node objects and deletes the corresponding Karpenter NodeClaim when consolidation conditions are met. Deleting the NodeClaim delegates final node and cloud instance termination to Karpenter.

Consolidation is triggered through three paths:

Node consolidation is feature-gated and can be enabled or disabled dynamically without restarting Kentroller.

Configuration

Required environment variables

View resources

List RebalancePlans

To list all rebalancing plans in the cluster:

kubectl get rebalanceplans
# or using the short name:
kubectl get rp

Example output:

NAME                          BEFORE     AFTER      SAVINGS%   STATE   AUTO-EXECUTED   AGE
nightly-rebalance-1735776000  1234.56    1111.11    10.0       Done    true            2h

List RebalancePlanClaims

To list all rebalancing plan claims:

kubectl get rebalanceplanclaims
# or using the short name:
kubectl get rpc

Example output:

NAME                                  STATE       PLANID   NODES   SAVINGS%   REBALANCE   AGE
nightly-spot-rebalance-1735776000     Completed   abc123   5       12.4       rp-xyz      2h

Inspect a RebalancePlan

To view the full details of a specific plan:

kubectl describe rebalanceplan <plan-name>

The status section shows:

• state — current execution state: Pending, Running, Done, DoneWithWarnings, or Failed
• savings — cost details for current and projected node configurations
• nodesCreation — per-node creation attempt history
• nodesDeletion — per-node deletion event history
• failurePhase — if failed, which phase caused the failure (Creation or Deletion)

Troubleshooting

RebalancePlanClaim stuck in Pending or Generating

1. Check the claim status:

   Shell

   kubectl describe rpc <claim-name>

   Review status.conditions and status.errorMessage.

2. Verify the Cast AI agent is connected and communicating with Cast AI.

3. Check if minSavingsPercentage is set too high — if projected savings don't meet the threshold, the claim will fail without executing.

RebalancePlan stuck in Running

1. Check for failed node creation attempts:

   Shell

   kubectl describe rp <plan-name>

   Look at status.nodesCreation for InsufficientCapacityError on all instance types. If all attempts are exhausted, the plan transitions to Failed with FailureReasonNodeCreateFailed.

2. Check status.nodesDeletion for stuck live migrations. If a RebalanceMigrationPlan is stuck, inspect it directly:

   Shell

   kubectl get rebalancemigrationplans
   kubectl describe rebalancemigrationplan <name>

Continuous rebalancing not running

1. Verify the continuous rebalancing feature is enabled in Cast AI.

2. Check that enough candidate nodes exist — CONTINUOUS_REBALANCING_MIN_NODES_TO_CONSIDER must be met.

3. Verify the savings thresholds are reachable. If the cluster is already well-optimized, plans may be generated but discarded because projected savings fall below CONTINUOUS_REBALANCING_SAVINGS_THRESHOLD_COST_MONTHLY.

4. Check Kentroller logs:

   Shell

   kubectl logs -n castai-agent -l app=castai-kentroller | grep "continuous-rebalancing"

Node consolidation not triggering

1. Verify the node consolidation feature is enabled in Cast AI.

2. For the empty node path, check that the NodePool has consolidation enabled and that Karpenter is setting the Consolidatable=True condition on empty nodes.

3. Check Kentroller logs:

   Shell

   kubectl logs -n castai-agent -l app=castai-kentroller | grep "empty-node-deleter"

Spot interruption replacements not happening

1. Verify the spot interruption prediction feature is enabled in Cast AI.

2. Check that SPOT_INTERRUPTION_PREDICTION_MODEL_NAME is set and that Kentroller has connectivity to the Cast AI ML API.

3. Look for SpotInterruptionPredicted events on spot nodes:

   Shell

   kubectl get events --field-selector reason=SpotInterruptionPredicted

Related resources