Cluster controller

The Cluster Controller is responsible for handling specific Kubernetes actions, including draining and deleting nodes, adding labels, and approving CSR requests.

Install Cluster Controller

The Cluster Controller is installed during Phase 2 (automation) cluster onboarding using the Helm chart available at https://github.com/castai/helm-charts/tree/main/charts/castai-cluster-controller.

Installation methods:

• Via Cast AI Operator (recommended): When onboarding new clusters with automation enabled (Phase 2), the Cast AI Operator automatically installs and manages Cluster Controller when extendedPermissions:true is set. This ensures you receive updates through the Cast AI console without running additional scripts.

• Via Helm (manual): If Cluster Controller was uninstalled or you're managing an older cluster without the Operator, you can reinstall it manually using the instructions below.

Manual installation

If Cluster Controller was uninstalled or you need to reinstall it manually:

Add the Cast AI helm charts repository:

helm repo add castai-helm https://castai.github.io/helm-charts
helm repo update

You can list all available components and versions:

helm search repo castai-helm

Expected example output:

NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
castai-helm/castai-agent                0.18.0          v0.23.0         CAST AI agent deployment chart.
castai-helm/castai-cluster-controller   0.17.0          v0.14.0         CAST AI cluster controller deployment chart.
castai-helm/castai-evictor              0.10.0          0.5.1           Cluster utilization defragmentation tool
castai-helm/castai-spot-handler         0.3.0           v0.3.0          CAST AI spot handler daemonset chart.

Now let's install it.

helm upgrade castai castai-helm/castai -n castai-agent \
  --reset-then-reuse-values \
  --set autoscaler.castai-cluster-controller.castai.apiKey=<your-api-token> \
  --set autoscaler.castai-cluster-controller.castai.clusterID=<your-cluster-id>

Upgrade Cluster Controller

Upgrade to the latest version:

helm repo update
helm upgrade castai castai-helm/castai -n castai-agent --reset-then-reuse-values

To stay on the current Cluster Controller version and only change configuration, pin the chart version:

helm upgrade castai castai-helm/castai -n castai-agent --version <your-current-chart-version> --reset-then-reuse-values

For RBAC-related upgrades that require new permissions, see Helm Charts for the recommended workflow.

Troubleshooting

Check Cluster Controller logs:

kubectl logs -l app.kubernetes.io/name=castai-cluster-controller -n castai-agent

Forbidden access error

In some scenarios, during multiple onboarding attempts, failing updates, or other issues, the cluster token used by the cluster controller can get invalidated. By becoming forbidden from accessing the Cast AI API, it fails to operate the cluster.

To renew it, you should run the following Helm commands:

helm repo update
helm upgrade castai castai-helm/castai -n castai-agent \
  --reset-then-reuse-values \
  --set autoscaler.castai-cluster-controller.castai.apiKey=$CASTAI_API_TOKEN \
  --set autoscaler.castai-cluster-controller.castai.clusterID=<your-cluster-id>

Throttling due to rate limiting

The Cluster Controller implements a client-side rate limiter to regulate requests to the Kubernetes API server, preventing excessive load on the control plane. This rate limiter uses the token bucket algorithm with default settings designed for typical cluster environments.

While the default configuration works well for most deployments, large or highly dynamic clusters may experience performance issues due to conservative rate limits. You can adjust the rate-limiting parameters if you observe throttling-related delays in a cluster with an appropriately scaled control plane.

Adjust rate limit settings

Modify the following environment variables in the Cluster Controller deployment:

# Requests per second rate limit (tokens replenished per second). This should be higher than the observed continuous load
KUBECLIENT_QPS=<number>

# Maximum allowed burst of requests. This can be adjusted to match extreme spikes or expected bursts of operations
KUBECLIENT_BURST=<number>

When to adjust settings

Consider increasing these values when:

• The cluster has a scaled-up control plane capable of handling higher throughput
• Logs show frequent throttling messages
• Operations that modify many resources (like large-scale rebalancing) are taking longer than expected

How to apply changes

When using Helm, you can set these values through the additionalEnv or envFrom parameters:

helm upgrade castai castai-helm/castai -n castai-agent \
  --reset-then-reuse-values \
  --set autoscaler.castai-cluster-controller.additionalEnv.KUBECLIENT_QPS=50 \
  --set autoscaler.castai-cluster-controller.additionalEnv.KUBECLIENT_BURST=200

For the complete default configuration, refer to the castai-cluster-controller Helm chart.