Workload Autoscaler Configuration

Workload Autoscaling can be configured in different ways to suit your specific needs. This can be achieved by using the CAST AI API (or changing the fields via the UI) or controlling the autoscaling settings at the workload level using workload annotations.

Upgrading

Currently, workload autoscaler is installed as an in-cluster component via helm and can be upgraded by simply running the following:

helm upgrade -i castai-workload-autoscaler -n castai-agent castai-helm/castai-workload-autoscaler --reuse-values

Dynamically Injected containers

By default, containers that are injected during runtime (e.g.,istio-proxy) won't be managed by workload autoscaler, and recommendations won't be applied. To enable that, you must configure the in-cluster component with the following command:

helm upgrade castai-workload-autoscaler castai-helm/castai-workload-autoscaler -n castai-agent --reuse-values --set webhook.reinvocationPolicy=IfNeeded

Available Workload Settings

The following settings are currently available to configure CAST AI Workload Autoscaling:

• Automation - on/off marks whether CAST AI should apply or just generate recommendations.
• Scaling policy- allows for the selection of policy names. It must be one of the policies available for a cluster.
• Recommendation Percentile - which percentile CAST AI will recommend, looking at the last day of the usage. The recommendation will be the average target percentile across all pods spanning the recommendation period. Setting the percentile to 100% will no longer use the average of all pods but the maximum observed value over the period.
• Overhead - marks how many extra resources should be added to the recommendation. By default, it's set to 10% for memory and 0% for CPU.
• Optimization Threshold - when automation is enabled, how much of a difference should there be between the current pod requests and the new recommendation so that the recommendation can be applied immediately? Defaults to 10% for both memory and CPU.
• Workload autoscaler constraints - sets the minimum and maximum values for resources, which will dictate that workload autoscaler cannot scale CPU/Memory above the max or below the minimum limits. The limit is set for all containers.
• Ignore startup metrics - allows excluding a specified duration of startup metrics from recommendation calculations for workloads with high initial resource usage (e.g., Java applications).
• Look-back period - defines a custom timeframe (between 24 hours and 7 days) the Workload Autoscaler uses to observe CPU and memory usage when calculating scaling recommendations. It can be set separately for CPU and memory.

📘

Note

It is recommended to wait for a week before enabling Workload Autoscaling for "all workloads", so that the system has understanding how the resource consumption varies on weekdays and weekends.

Ignore startup metrics

Some workloads, notably Java and .NET applications, may have increased resource usage during startup that can negatively impact autoscaling recommendations. To address this, CAST AI allows you to ignore startup metrics for a specified duration when calculating workload autoscaling recommendations.

You can configure this setting in the CAST AI console under Advanced Settings of a vertical scaling policy:

1. Enable the feature by checking the "Ignore workload startup metrics" box.
2. Set the duration to exclude from recommendation generation after a workload starts (between 2 and 60 minutes).

This feature helps prevent inflated recommendations and unnecessary restarts caused by temporary resource spikes during application initialization.

You can also configure this setting via the API or Terraform.

Look-back period

The look-back period defines the timeframe the Workload Autoscaler uses to observe CPU and memory usage when calculating scaling recommendations. This feature allows you to customize the historical data window used for generating recommendations, which can be particularly useful for workloads with atypical resource usage patterns.

You can configure the look-back period in the CAST AI console under Advanced Settings of a vertical scaling policy:

1. Set the look-back period for CPU and memory separately.
2. Specify the duration in days (d) and hours (h). The minimum allowed period is 24 hours, and the maximum is 7 days.

This feature allows you to:

• Adjust the recommendation window based on your workload's specific resource usage patterns.
• Account for longer-term trends or cyclical resource usage in your applications.

You can configure this setting at different levels:

• Policy level: Apply the setting to all workloads assigned to a specific scaling policy.
• Individual workload level: Configure the setting for a specific workload using annotations or the UI by overriding policy-level settings.

The look-back period can also be configured via Annotations, the API, or Terraform.

Choosing the right look-back period

The optimal look-back period largely depends on your workload's resource usage patterns. Most applications benefit from a shorter look-back period of 1-2 days. This approach works particularly well for standard web applications, capturing daily usage patterns while maintaining high responsiveness to changes. Shorter periods enable more aggressive optimization and often lead to higher savings.

Some workloads, however, require longer observation periods of 3-7 days. Applications with significant differences between weekday and weekend usage patterns benefit from a 7-day period to capture these weekly variations. Batch processing jobs that run every few days need a look-back period that covers at least one full job cycle to prevent potential out-of-memory (OOM) situations.

Common use cases and recommended periods:

• Standard web applications: 1-2 days captures daily patterns while maintaining responsiveness to changes
• Batch processing jobs: Set to cover at least one full job cycle to account for periodic resource spikes
• Weekend-sensitive workloads: 7 days to capture both weekday and weekend patterns
• Variable workloads: Start with 1-2 days and adjust based on observed scaling behavior

💡

Tip

For workloads with variable or uncertain patterns, start with a shorter period and adjust based on observed behavior. The key is to match the look-back period to your application's actual resource usage patterns – whether that's daily consistency, weekly cycles, or periodic processing jobs.

Custom workload support

The workload autoscaler supports the scaling of custom workloads through label-based selection. This allows autoscaling for:

• Bare pods (pods without controllers)
• Pods created programmatically (as Spark Executors or Airflow Workers).
• Jobs without parent controllers
• Workloads with custom controllers not natively supported by Cast AI
• Groups of related workloads that should be scaled together

Label-based workload selection

To enable autoscaling for custom workloads, add the workload.cast.ai/custom-workload label to the Pod template specification. This is crucial - the label must be present in the Pod template, not just on the controller or running Pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    workloads.cast.ai/custom-workload: "my-custom-workload"
spec:
  containers:
    - name: app

Workloads with the same label value will be treated as a single workload for autoscaling purposes. The value acts as a unique identifier for the workload group.

Workloads are uniquely identified by their:

• Namespace
• Label value
• Controller kind (coming soon)

🚧

Important change coming

Currently, workloads are uniquely identified by namespace + label value. In an upcoming release, this will change to namespace + label value + controller kind to provide better isolation between different types of controllers.

Configuring autoscaling behavior

Both labels and annotations used to configure autoscaling behavior must be specified in the Pod template specification, not on the controller or running Pod.

Key points about label-based workload configuration:

• Workloads are grouped per controller kind (deployments and StatefulSets with the same label will be treated as separate workloads)
• For grouped workloads, the newest/latest matching controller's pod template configuration is used as the workload specification
• Only workloads with the workload.cast.ai/custom-workload label will be discovered for custom workload autoscaling
• The label value must be unique for each distinct workload or group of workloads you want to scale together
• All configuration labels and annotations must be specified in the Pod template specification

Examples

Scale a bare pod:

apiVersion: v1
kind: Pod
metadata:
  labels:
    workload.cast.ai/custom-workload: "standalone-pod"
spec:
  containers:
    - name: app
      # Container spec...

Group related jobs:

apiVersion: batch/v1
kind: Job
spec:
  template:
    metadata:
      labels:
        workload.cast.ai/custom-workload: "batch-processors"
    spec:
      containers:
        - name: processor
          # Container spec...

Schedule recurring workloads:

apiVersion: batch/v1
kind: CronJob
spec:
  schedule: "*/10 * * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            workload.cast.ai/custom-workload: "scheduled-processor"
        spec:
          containers:
            - name: processor
              # Container spec...

Scale workloads with custom controllers:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-managed-app
  ownerReferences: # Custom controller resource
    - apiVersion: customcontroller.example.com/v1alpha1
      kind: CustomResourceType
      name: custom-resource
      uid: abc123
      controller: true
spec:
  template:
    metadata:
      labels:
        workload.cast.ai/custom-workload: "custom-controlled-app"
    spec:
      containers:
        - name: app
          # Container spec...

The workload autoscaler will track and scale these workloads based on resource usage patterns, applying the same autoscaling policies and recommendations as standard workloads, except:

• These workloads are only scaled vertically using Vertical Pod Autoscaling (VPA)
• Only the deferred recommendation mode is supported

📘

Note

Custom workload autoscaling uses deferred mode, meaning recommendations are only applied when pods are naturally restarted. This helps ensure safe scaling behavior for workloads without native scaling support.

Configuration via API/UI

We can configure the aforementioned settings via the UI.

Configuration via Annotations

All settings are also available by adding annotations on the workload controller. When any workloads.cast.ai annotation is detected on a workload, it will be considered managed by annotations. This allows for flexible configuration, combining annotations and scaling policies.
Changes to the settings via the API/UI are no longer permitted for workloads with annotations. When a workload does not have an annotation for a specific setting, the default or scaling policy value is used.

📘

Note

Workloads can be managed through a combination of annotations and scaling policies. For example, you can set the workloads.cast.ai/scaling-policy annotation on a workload and toggle vertical autoscaling on/off in the scaling policy itself. This provides more flexibility in managing workload configurations.

The annotations generally follow a pattern of workloads.cast.ai/{resource}-{setting}. Currently, the available resources are cpu and memory. Available settings:

workloads.cast.ai/vertical-downscale-apply-type

The workloads.cast.ai/vertical-downscale-apply-type annotation is fully compatible with the workloads.cast.ai/apply-type annotation and is meant to be used in combination with it. This allows for fine-grained control over both upscaling and downscaling. Here's how they interact:

1. If both annotations are set to the same value (both immediate or both deferred), the behavior remains unchanged.
2. If apply-type is set to immediate and vertical-downscale-apply-type is set to deferred:
   • Upscaling operations will be applied immediately.
   • Downscaling operations will be deferred to natural pod restarts.
3. If apply-type is set to deferred and vertical-downscale-apply-type is set to immediate:
   • Upscaling operations will be deferred to natural pod restarts.
   • Downscaling operations will be applied immediately.

Example config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
  annotations:
    workloads.cast.ai/vertical-autoscaling: "on" # enable vertical automatic scaling
    workloads.cast.ai/scaling-policy: "my-custom" # select my-custom scaling policy
    workloads.cast.ai/apply-type: "immediate" # apply recommendations immediately for upscaling
    workloads.cast.ai/vertical-downscale-apply-type: "deferred" # defer downscaling to natural pod restarts

    workloads.cast.ai/cpu-overhead:                 "0"      # 0%
    workloads.cast.ai/cpu-apply-threshold:          "0.05"   # 5% 
    workloads.cast.ai/cpu-target:                   "p80"    # 80th percentile
    workloads.cast.ai/cpu-max:                      "400m"   # max 0.4 cpu
    workloads.cast.ai/cpu-min:                      "120m"   # min 0.12 cpu
    workloads.cast.ai/cpu-look-back-period-seconds: "259200" # 3 days

    workloads.cast.ai/memory-overhead:                 "0.1"    # 10%
    workloads.cast.ai/memory-apply-threshold:          "0.05"   # 5%
    workloads.cast.ai/memory-target:                   "max"    # max usage
    workloads.cast.ai/memory-max:                      "2Gi"    # max 2Gi
    workloads.cast.ai/memory-min:                      "1Gi"    # min 1Gi
    workloads.cast.ai/memory-look-back-period-seconds: "172800" # 2 days

Configuration Errors

If the workload manifest contains an invalid configuration, as an example workloads.cast.ai/autoscaling: "unknown-value" the configuration will not be updated (old configuration values will be used until the erroneous configuration is fixed), and you should be able to see the error in the workload details in the CAST AI Console. Since scaling policy names are not restricted character-wise -- any value can be set, but a non-existent policy will be treated as an invalid configuration.

🚧

Important: Annotation Deprecation Notice

The workloads.cast.ai/autoscaling annotation is deprecated but still supported for backward compatibility.

• Deprecated: workloads.cast.ai/autoscaling
• Recommended: workloads.cast.ai/vertical-autoscaling

While existing configurations using the deprecated annotation will continue to function, we strongly recommend updating your workloads to use the new workloads.cast.ai/vertical-autoscaling annotation for future compatibility and access to the latest features.