Scheduled rebalancing

How it works

Scheduled rebalancing enables full or partial rebalancing process on a user-defined schedule, scope, and trigger to automate various use cases. Most commonly scheduled rebalancing is used in the following scenarios:

  • Identify the most expensive Spot Instances and replace them with cheaper alternatives
  • Perform full rebalancing of clusters on weekends
  • Roll old nodes
  • Periodically target and replace nodes with specific labels

A rebalancing schedule is an organization-wide object that can be assigned to one or multiple clusters. Once triggered, it creates a rebalancing plan scoped by parameters provided in the node selection preferences, while the problematic workloads are excluded automatically.

To set up scheduled rebalancing, navigate to the organizational-level Optimization menu or access the Rebalancer view within the cluster.

Rebalancing schedules view with the schedule creation drawer open

Setup

There are two main components in the scheduled rebalancing functionality

  • A rebalancing job is a job that triggers a rebalancing schedule on an associated cluster.
  • Rebalancing schedule - an organization-wide rebalancing schedule that can be used on multiple rebalancing jobs.

The Rebalancing schedule consists of four parts:

  • The schedule describes when to run the scheduling (periodically, within the maintenance window, etc.)
  • Node selection preferences contain rules for picking specific nodes, how many to target, and similar rules that mimic decision-making when manually rebalancing a cluster.
  • Trigger requirements are decision rules for executing the rebalancing plan. They include rules like the “Savings threshold,” determining whether the generated plan should be executed.
  • Execution safeguard: Cast AI stops rebalancing before draining the original nodes if it finds that the user-specified minimum level of savings can't be achieved. This additional protection layer is required as planned nodes might temporarily be unavailable in the cloud provider's inventory during the rebalancing.

Settings

The following settings can be adjusted when setting up the scheduled rebalancing:

SettingDescription
Specify resource offeringSpot, On-Demand, or Any
Target using labels

Key-value pairs are provided as nodeSelector terms. In the UI, when multiple terms are provided, the values are handled using AND logic, i.e., only nodes that satisfy all listed selector terms will be targeted.

For OR logic between label values, use the Cast AI API endpoint POST /v1/rebalancing-schedules. In the API request, use the In operator and specify multiple values for the same label key in the matchExpressions section. This API-only feature allows targeting nodes matching any specified label values. Note that the UI may not display all values correctly when using this API method.

Minimum node ageAmount of time since the node creation before the node can be considered for rebalancing. 0 - means a node of any age can be considered.
Evict nodes gracefully

Defines whether nodes that fail to drain within a configurable timeout (default: 20 minutes, maximum: 180 minutes) will be kept with a rebalancing.cast.ai/status=drain-failed annotation instead of being forcefully drained.

When disabled (default): Nodes are forcefully drained after the timeout, which may override Pod Disruption Budgets (PDBs) to complete the rebalancing operation.

When enabled: Nodes are annotated, cordoned, and kept in the cluster. PDBs are strictly respected. Cast AI automatically uncordons drain-failed Nodes after 30 minutes by default; the delay is configurable via drainFailureConfig. See Handling failed Node drains.

Enable via the API by setting evictGracefully to true (Terraform: evict_gracefully = true). By default, evictGracefully is false (disabled). The field replaces the deprecated keepDrainTimeoutNodes; the value keeps its meaning, so copy it unchanged.

Drain timeout

Maximum time to wait for each node to drain during rebalancing. Default: 20 minutes. Maximum: 180 minutes (10800 seconds). Default: 20 minutes. Configure this in Advanced settings when setting up a rebalancing schedule.

After this timeout expires, the node is either forcefully drained (default) or annotated and kept in the cluster (when Evict nodes gracefully is enabled).

Maximum simultaneous drainsMaximum number of nodes to drain at the same time (API: maxSimultaneousDrains, Terraform: max_simultaneous_drains). Default: unlimited.
Batch executionThe value paces execution into batches so that at most this many extra nodes (created minus drained) are alive at any moment (API: splitNodesInBatchesOf, Terraform: split_nodes_in_batches_of). The plan drains a node only after its replacement node exists. When unset, zero, or negative, the plan executes in a single batch. Unlike Maximum batch size, which caps how many nodes the rebalancing plan may target, batching does not change node selection; it only paces execution.
Maximum batch sizeMaximum number of nodes that can be selected for a single rebalancing operation. Use the Batch size type selector in the schedule form to set the limit in one of two ways:
  • Node count (numTargetedNodes) — the maximum number of nodes to select. 0 means all matching nodes in the cluster can be selected.
  • Percentage of eligible nodes (targetedNodesPercentage, 1-100) — caps each rebalancing operation at a percentage of the eligible nodes, regardless of cluster size. This allows reusing one schedule across clusters that change in size. The percentage is rounded up to a whole number of nodes, with at least one node selected. When set, it takes precedence over the node count.
Sort selected nodesThe algorithm used to sort selected nodes:
  • Highest normalized CPU price* - sorts by the most expensive nodes based on the price of normalized CPU (node cost / CPU count).
  • Highest requested resource price (CPU + RAM)* - sorts by the most expensive nodes based on the requested resource price, considering both CPU and RAM in a 1:7 ratio. This helps optimize costs for memory-focused workloads and node templates.
  • Least utilized* - sorts by the least utilized nodes first, regardless of price.
Aggressive modeDefines whether rebalancing may also target nodes running problematic workloads. In the console, Enable Aggressive mode reveals the per-category options listed in Aggressive mode configuration. The deprecated aggressiveMode boolean is not offered in the console; it remains available via the API (Terraform: aggressive_mode) only.
Savings threshold
  • Target savings* (savings_percentage)
    The minimum projected savings percentage required before executing a rebalancing plan. Plans that don't meet this threshold will be skipped.\

  • Guaranteed minimum savings* (achieved_savings_percentage)
    An execution safeguard that prevents rebalancing when minimum savings cannot be achieved. When planned nodes become unavailable during execution, alternative nodes are created. If alternatives result in higher costs that don't meet the guaranteed threshold, new nodes are deleted, and rebalancing is aborted.

Execution timesExecution time can be adjusted by providing a timezone and a crontab expression

Node targeting

When configuring node targeting for scheduled rebalancing, you can choose between two methods: Labels or Fields. This selection determines how you'll identify nodes for the rebalancing operation.

Using labels

To configure label-based targeting:

  1. Select "Labels" under "Target using"
  2. Configure your label-matching rules:
    • Key: Enter the Kubernetes label key (e.g., scheduling.cast.ai)
    • Operator: Choose from:
      • Exists: Match nodes where the label exists
      • DoesNotExist: Match nodes where the label is absent
      • In: Match nodes where the label value matches one in a list
      • NotIn: Match nodes where the label value doesn't match any in a list
    • Value: Enter the label value to match (required for In and NotIn operators)
  3. Add additional label-matching expressions using the "+ Add" button if needed

Multiple node label matching expressions are combined using the AND logic – nodes must match all specified conditions to be selected for rebalancing.

For example, if you configure:

  • field1 In spot_tmp
  • field2 In wrestle

Only nodes with both field1=spot_tmp AND field2=wrestle will be selected.

Technical details

Under the hood, the Console UI creates a single nodeSelectorTerm with multiple matchExpressions. According to Kubernetes selector logic:

  • All expressions within a nodeSelectorTerm are ANDed together
  • Multiple nodeSelectorTerms are ORed together (API-only, see below)

Using fields

Field-based targeting provides an alternative method when label-based selection isn't sufficient. When selecting Fields, you can target nodes based on their Kubernetes field values rather than labels.

Example

An example use case for field targeting is rebalancing a specific node by name. This is useful when you've identified a particular node that needs replacement. For example, a node running an outdated configuration or one flagged for maintenance.

  1. Select Fields under Target using
  2. Configure the field selector:
    • Key: metadata.name
    • Operator: =
    • Value: the name of the node to target (e.g., ip-10-0-1-42.ec2.internal)

This configuration targets a single node by its Kubernetes node name, allowing you to schedule its replacement without affecting other nodes in the cluster.

For more information on using field selectors, refer to Kubernetes documentation.

⚠️

Known limitation: cordoned-node targeting may cause schedules to be skipped

Targeting cordoned Nodes with spec.unschedulable=true can cause the scheduled rebalancing operation to be skipped entirely rather than rebalancing the targeted Nodes. Cast AI automatically uncordons drain-failed Nodes after 30 minutes by default (configurable via drainFailureConfig), but scheduled rebalancing still skips them while cordoned.

For recovery options, see Handling failed Node drains.

Using OR conditions for node labels in scheduled rebalancing

The console UI supports AND logic only. To use OR conditions between node label values, use the Cast AI API:

  1. Use the Cast AI API endpoint for creating rebalancing schedules: POST /v1/rebalancing-schedules
  2. Specify multiple nodeSelectorTerms in your request to create OR logic between terms
  3. Within each term, use matchExpressions with the In operator to target multiple values for the same label key

Example: OR logic for multiple values of the same label

"nodeSelectorTerms": [
  {
    "matchExpressions": [
      {
        "key": "nodetemplate",
        "operator": "In",
        "values": ["customerA", "customerB", "customerC"]
      }
    ]
  }
]

This configuration targets nodes with the nodetemplate label matching any of the specified values (customerA OR customerB OR customerC).

Example: OR logic between different label conditions

To create OR logic between multiple distinct label conditions, add multiple objects to the nodeSelectorTerms array:

"nodeSelectorTerms": [
  {
    "matchExpressions": [
      {
        "key": "nodetemplate1",
        "operator": "In",
        "values": ["customerA"]
      }
    ]
  },
  {
    "matchExpressions": [
      {
        "key": "nodetemplate2",
        "operator": "In",
        "values": ["customberB"]
      }
    ]
  }
]

This selects nodes with either nodetemplate=customerA OR nodetemplate2=customerB.

🚧

Warning

When using multiple nodeSelectorTerms via the API, the UI may not display all values correctly. The UI is limited to displaying a single term with AND logic only.

For complete details, refer to the Cast AI API reference documentation for creating rebalancing schedules.

Maximum batch size

The Maximum batch size setting limits how many nodes a single rebalancing operation can select. In the schedule form, use the Batch size type selector to set the limit in one of two ways:

  • Node count — the maximum number of nodes to target in a single operation. 0 means all matching nodes in the cluster can be selected.
  • Percentage of eligible nodes — caps each rebalancing operation at a percentage of the eligible nodes, regardless of cluster size. This allows reusing one schedule across clusters that change in size. The percentage is rounded up to a whole number of nodes, with at least one node selected. When set, it takes precedence over the node count.

Sort selected nodes

When configuring scheduled rebalancing, the Sort selected nodes setting determines how nodes are prioritized for rebalancing. Each algorithm serves different optimization goals.

Highest normalized CPU price

  • Sorts nodes by dividing the node's total cost by its CPU count, which prioritizes removing expensive nodes based on their provisioned (total available) CPUs.
  • Best for: Spot Instance clusters when you want to remove the most expensive instances or a partial rebalancing where you just want to remove the most expensive nodes
📘

Note

It doesn't consider actual resource utilization, only the raw cost per CPU.

Highest requested resource price (CPU + RAM)

  • Sorts nodes by calculating the cost of actually requested (used) resources, considering both CPU and memory in a 1:7 ratio. For example, a $5/hour node using 1 CPU and 7GB RAM has a resource price of $2.50/unit, while the same node using 4 CPU and 28GB RAM has a resource price of $0.63/unit.
  • Best for: Most scenarios, especially mixed workloads with varying memory and CPU needs, as it provides the best overall cost optimization by targeting nodes with the highest wasted resource cost.

Least utilized

  • Sorts nodes purely based on CPU utilization percentage, regardless of node size or cost. For example, a 2-core node with 1 core used (50% utilized) would be prioritized over a 64-core node with 20 cores used (31% utilized), even though the larger node has more wasted resources
  • Best for: Specific use cases where you want to consolidate many small, underutilized nodes, regardless of cost impact
📘

Note

It may miss opportunities to remove larger, more expensive nodes that have higher percentage utilization but more wasted resources as a sum.

For most cost optimization scenarios, the Highest requested resource price (CPU + RAM) sorting algorithm provides the best balance for removing expensive and underutilized nodes.

Aggressive mode configuration

By default, the Rebalancer treats nodes running problematic workloads as not ready for rebalancing. Aggressive mode lifts these protections. Rebalancing can then also target nodes running pods without a controller, job pods, pods with the removal-disabled annotation, and pods annotated with rebalancing.cast.ai/prevented-drain=true.

In the console, the rebalancing schedule form groups these options under Aggressive mode.

The standalone aggressiveMode boolean is deprecated. Use aggressiveModeConfig instead: it enables aggressive rebalancing on its own and lets you choose exactly which problematic workload categories rebalancing may evict:

aggressiveModeConfig fieldWorkloads rebalancing may then evict
ignoreProblemPodsWithoutControllerPods without a controller (bare pods); no controller exists to restart them after eviction.
ignoreProblemJobPodsPods that belong to Jobs or CronJobs; eviction loses their progress.
ignoreProblemRemovalDisabledPodsPods marked with the removal-disabled annotation.
ignoreProblemPreventedDrainPodsPods annotated with rebalancing.cast.ai/prevented-drain=true. In the console: Pods that previously failed to drain. Also lets rebalancing target Nodes marked with rebalancing.cast.ai/status=drain-failed.
ignoreLocalPersistentVolumesPods using local-path Persistent Volumes; their data does not move with them.
🚧

Warning

Ignoring these protections lets the Rebalancer evict pods that lose their progress (Job pods), never restart (bare pods), or were marked protected from eviction. ignoreLocalPersistentVolumes additionally results in data loss for applications using Local Persistent Volumes.

How aggressiveMode interacts with aggressiveModeConfig

These rules apply when you set the fields through the API or Terraform. The console offers no control for the deprecated aggressiveMode boolean.

  • aggressiveModeConfig is the recommended way to configure aggressive rebalancing. Keep aggressiveMode unset or false when using it.
  • If the deprecated aggressiveMode is true, it forces all four problematic-pod flags on. Only ignoreLocalPersistentVolumes still follows the config.
  • If only aggressiveMode=true is set, all four categories are ignored and Local Persistent Volumes stay protected.
  • To reproduce aggressiveMode=true with the config, set the four ignoreProblem* flags to true. Local Persistent Volumes stay protected unless you also set ignoreLocalPersistentVolumes.

Local Persistent Volumes configuration

By default, even in aggressive mode, nodes with Local Persistent Volumes (LPVs) are marked as "Not ready" for rebalancing. This is because LPVs are tied to specific nodes through node affinity rules (typically using the kubernetes.io/hostname topology label) and cannot be migrated between nodes.

You can configure aggressive mode to ignore these local volume constraints via the API or Terraform:

When creating or updating a rebalancing schedule through the API, include the aggressiveModeConfig in your request:

{
  "launchConfiguration": {
    "rebalancingOptions": {
      "aggressiveModeConfig": {
        "ignoreLocalPersistentVolumes": true
      }
    }
  }
}

This configuration allows rebalancing to include nodes with Local Persistent Volumes. For the full schema, see the Scheduled Rebalancing API Reference.

🚧

Warning

Setting ignoreLocalPersistentVolumes (Terraform: ignore_local_persistent_volumes) to true allows rebalancing to evict pods with local storage. This will result in data loss for applications using Local Persistent Volumes. Only use this option when data persistence is not critical.

Configuration examples

Using the API

Create a rebalancing schedule using the API:

{
  "schedule": {
    "cron": "*/30 * * * *"
  },
  "launchConfiguration": {
    "selector": {
      "nodeSelectorTerms": [
        {
          "matchExpressions": [
            {
              "key": "scheduling.cast.ai/spot",
              "operator": "Exists"
            }
          ]
        }
      ]
    },
    "rebalancingOptions": {
      "executionConditions": {
        "enabled": true,
        "achievedSavingsPercentage": 10
      },
      "minNodes": 2,
      "evictGracefully": false,
      "maxSimultaneousDrains": 5,
      "splitNodesInBatchesOf": 10
    },
    "targetNodeSelectionAlgorithm": "TargetNodeSelectionAlgorithmNormalizedPrice",
    "numTargetedNodes": 3,
    "nodeTtlSeconds": 300
  },
  "triggerConditions": {
    "savingsPercentage": 20
  },
  "name": "rebalance spots at every 30th minute"
}

The example sets evictGracefully to false for illustration only; false is the default, so you can omit the field.

maxSimultaneousDrains caps how many nodes drain at the same time. splitNodesInBatchesOf paces execution so that at most 10 extra nodes (created minus drained) are alive. Neither changes which nodes the plan selects; that is numTargetedNodes (Maximum batch size).
Alternatively, set targetedNodesPercentage in launchConfiguration to cap each rebalancing operation at a percentage of eligible nodes rather than a node count. See the Create Rebalancing Schedule API reference for the full field specification.

"launchConfiguration": {
  "targetedNodesPercentage": 50
}

The percentage is rounded up to a whole number of nodes, with at least one node selected, and takes precedence over numTargetedNodes. This allows reusing a single schedule across clusters that vary in size.

Using Terraform

Define a rebalancing schedule using the Terraform provider:

resource "castai_rebalancing_schedule" "spots" {
  name = "rebalance spots at every 30th minute"
  schedule {
    cron = "*/30 * * * *"
  }
  trigger_conditions {
    savings_percentage = 20
  }
  launch_configuration {
    # only consider instances older than 5 minutes
    node_ttl_seconds         = 300
    num_targeted_nodes       = 3
    rebalancing_min_nodes = 2
    evict_gracefully      = false
    # drain at most 5 nodes at the same time
    max_simultaneous_drains = 5
    # execute in batches, keeping at most 10 additional (created minus drained) nodes alive
    split_nodes_in_batches_of = 10
    selector = jsonencode({
      nodeSelectorTerms = [{
        matchExpressions = [
          {
            key      = "scheduling.cast.ai/spot"
            operator = "Exists"
          }
        ]
      }]
    })
    execution_conditions {
      enabled                     = true
      achieved_savings_percentage = 10
    }
  }
}

Did this page help you?