Migrate GitOps and IaC installations with castctl

Older Cast AI installations deployed one Helm chart per component, with each component running as a separate release, upgraded and configured independently. The recommended installation today is the umbrella chart (castai-helm/castai): a single Helm release that bundles every Cast AI component.

With one release, a single upgrade moves all components to mutually compatible versions. Helm tags control which components the umbrella installs, and one values file passes custom settings down to each sub-component. New clusters connect with it, and existing standalone installations should move to it.

If your standalone releases are managed by GitOps (ArgoCD, Flux) or Terraform, you can't migrate them imperatively. The next sync or terraform apply would undo any manual Helm change. Instead, castctl cluster migrate provides three subcommands that split the migration into phases. Each change is committed to your source of truth and applied by your own tooling:

• generate (read-only): inspects the cluster and writes an umbrella chart values file, a manifest for your tool, and a step-by-step MIGRATION-STEPS.md tailored to what it found.
• pre-umbrella-install: prepares the cluster for adoption. Preserves Workload Autoscaler CRDs and strips standalone Helm ownership without touching running workloads.
• post-umbrella-install: once the umbrella is healthy, it cleans up stale ReplicaSets and orphaned pods. On ArgoCD, it also disarms the Workload Autoscaler pre-delete hook, so a later Application delete can't strip finalizers from Recommendation CRs.

The phase split is deliberate: you can pause between phases, check the cluster state, and roll back before the final cleanup runs.

Before you begin

• castctl 0.15.0 or later, updated and authenticated.
• Your kubeconfig context points at the cluster you're migrating.
• Cluster-admin or equivalent RBAC on that cluster: pre-umbrella-install deletes Helm release secrets and orphan-deletes Deployments and DaemonSets in the Cast AI namespace (default castai-agent).
• Access to commit to the GitOps repository or apply the Terraform code that manages your Cast AI releases.
• Helm 3.17.0 or later, if you might reinstall a standalone chart with --take-ownership (used in the rollback steps below).

Check the castctl and Helm versions, and log in if you haven't already:

castctl version
helm version
castctl auth login

How the migration works

Phase 1: Generate

Run castctl cluster migrate generate --format <your tool>. It only reads from the cluster, so nothing changes and there is nothing to undo.

Phase 2: Prepare

Run castctl cluster migrate pre-umbrella-install. It protects Workload Autoscaler CRDs, deletes the Helm release secrets, and orphan-deletes the old Deployments and DaemonSets. Pods keep running. To roll back here, reinstall the standalone charts with --take-ownership. Use the values from the step 0 backup in MIGRATION-STEPS.md.

Phase 3: Install

Sync or apply the umbrella chart through your tool. Umbrella pods start alongside the old generation, which stays as warm standby, and only the new pods receive traffic. To roll back, uninstall the umbrella; the old generation is still running.

Phase 4: Clean up

Run castctl cluster migrate post-umbrella-install. It deletes the old-generation ReplicaSets and pods, and on ArgoCD it also disarms the standalone Workload Autoscaler's pre-delete hook. After this, the old pods are gone, so rolling back means a full restart of Cast AI pods.

Because pods are never deleted before the last phase, the migration has no downtime window: Cast AI components keep running throughout the process.

Generate the migration plan

1. Run generate with the format matching your tool. The command only reads from the cluster, so it's safe to run at any time and repeat:

   Shell

   castctl cluster migrate generate \
     --output castai-migration \
     --format <argocd|flux|terraform|generic> \
     --repo-url <https://github.com/org/gitops-repo.git>

   Only --format argocd needs --repo-url. Without it, the generated Application contains a <YOUR-GITOPS-REPO> placeholder. For Cast AI Anywhere clusters, add --profile autoscaler-anywhere.

   generate detects standalone Helm releases via their Helm records and ArgoCD-managed components by scanning ArgoCD Applications cluster-wide. For ArgoCD components, it reconstructs their values from each Application's spec.source.helm section. If the umbrella chart is already installed, the command stops and tells you there's nothing to migrate.

2. Review the generated files in the output directory:

   • values.yaml: umbrella chart values, including your preserved per-component custom values, the mode tag covering your detected components, and Workload Autoscaler CRD protection (crds.keep=true) where applicable. The API key is referenced via apiKeySecretRef, never embedded.
   • standalone-patch.yaml: a patch that protects Workload Autoscaler CRDs and scales the controller down before adoption. Only generated when a Workload Autoscaler installation is detected, and skipped for Flux (Flux setups patch the HelmRelease instead).
   • argocd-application.yaml, flux-helmrelease.yaml, or castai-helm-release.tf: the umbrella manifest for your tool, with the current umbrella chart version pinned.
   • MIGRATION-STEPS.md: a step-by-step checklist for your exact setup, including the step 0 backup, per-tool commands, and rollback instructions.

3. Follow MIGRATION-STEPS.md, starting with the step 0 backup of your current values and the castai-agent-metadata ConfigMap. That file is the authoritative checklist for your cluster. This guide covers the same phases next: Prepare the cluster for adoption, Install the umbrella chart through your tool, and Clean up after the install.

Prepare the cluster for adoption

1. Preview what the preparation phase will do:

   Shell

   castctl cluster migrate pre-umbrella-install --dry-run

   Check that the output lists only your standalone Cast AI releases: Helm release secrets to delete, Deployments and DaemonSets to orphan-delete, and Workload Autoscaler CRD preservation if you have Workload Autoscaler.

2. Run the preparation:

   Shell

   castctl cluster migrate pre-umbrella-install

   Type yes at the confirmation prompt, or pass --yes in automation.

   The command protects Workload Autoscaler Recommendation CRDs by scaling the controller to 0 and pinning crds.keep=true. It deletes the standalone releases' Helm secrets, so your GitOps tool no longer fights Helm ownership. It also orphan-deletes the Deployments and DaemonSets that the umbrella will re-create. Pods are not deleted and keep running.

If you're migrating a kubectl-applied manifest installation rather than Helm releases, the command automatically uses the server-side-apply strategy instead of deleting Helm secrets.

Install the umbrella chart through your tool

Commit the generated artifacts and let your own tooling install the umbrella chart. The exact steps are in MIGRATION-STEPS.md; in short:

Because pre-umbrella-install stripped the old Helm ownership, the umbrella release adopts the surviving resources. Its pods start as a new generation next to the old one. The umbrella's Services select only the new pods. The old pods remain running as your rollback safety net.

While both generations run, the namespace temporarily carries about twice the usual pod count. Before installing the umbrella, make sure the cluster has spare capacity for that: on a tightly packed cluster, the new pods can stay Pending until post-umbrella-install removes the old generation. If new-generation pods stay Pending, check why:

kubectl describe pod <pod-name> -n castai-agent

Scheduling pressure clears itself once post-umbrella-install removes the old generation. On a full cluster, you can also delete a few old-generation pods for that component; the new ones then schedule immediately.

Clean up after the install

1. Wait until the umbrella pods are Ready:

   Shell

   kubectl rollout status deployment,daemonset -n castai-agent

2. Run the cleanup (preview first with --dry-run if you want):

   Shell

   castctl cluster migrate post-umbrella-install

   This deletes the stale ReplicaSets and orphaned pods of the old generation. On ArgoCD, it also sets preDeleteHook.enabled=false on any Application that still runs the standalone workload-autoscaler chart. This stops a later Application delete from stripping finalizers from your Recommendation CRs and deleting the CRD.

3. Finish the tool-specific cleanup from MIGRATION-STEPS.md:

   Delete each old standalone Application with --cascade=orphan. This deletes only the resources the umbrella didn't adopt. Then re-enable auto-sync on the umbrella:

   Shell

   argocd app delete <standalone-app> --cascade=orphan
   # repeat for every standalone Application generate detected, or list them first: argocd app list
   argocd app set castai --sync-policy automated --auto-prune --self-heal

   Older argocd CLI versions use the deprecated --cascade=false instead.

Verify the migration

kubectl get pods -n castai-agent
kubectl rollout status deployment,daemonset -n castai-agent
kubectl get cm castai-agent-metadata -n castai-agent -o jsonpath='{.data.CLUSTER_ID}'

All pods should be running under the umbrella release, and the cluster ID should match the one from before the migration. The Cast AI console shows your components as managed by the unified chart.

Roll back if something goes wrong

Before post-umbrella-install runs, the old pod generation is still alive, so rollback is quick:

1. Delete the umbrella release, or the Application/HelmRelease/helm_release that installed it. For plain Helm:

   Shell

   helm uninstall castai -n castai-agent

   The Workload Autoscaler and its exporter CRDs survive because generate pins crds.keep=true in the generated values.

2. Re-create the standalone releases from your source of truth (restore the blocks you removed). Pass --take-ownership on every one; this requires Helm 3.17.0 or later. The surviving Workload Autoscaler and exporter CRDs still carry the umbrella release's ownership metadata. Their standalone charts need --take-ownership to adopt them; otherwise the install fails on a metadata conflict. Passing the flag on the other components too is harmless. MIGRATION-STEPS.md has the exact command for each release, in the form:

   Shell

   helm upgrade --install <release> castai-helm/<chart> -n castai-agent -f backup-<release>.yaml --take-ownership

If you already ran post-umbrella-install, the old pods are gone. The same steps still work, but every Cast AI pod is recreated, so expect a full restart. The tool-specific rollback checklists (including how to revert the Workload Autoscaler scale-down) are in the Rollback & Recovery section of your generated MIGRATION-STEPS.md.

Limitations and notes

• Only components owned by the umbrella chart are migrated. Standalone releases the umbrella doesn't manage are left untouched and reported with a warning during the prepare and cleanup phases. This currently covers castai-egressd, OMNI, and Database Optimizer components.
• The Workload Autoscaler and exporter CRDs (recommendations, gpurecommendations, custommetricsexporterconfigs) are pinned to survive an uninstall, on both the standalone and umbrella charts. Other component CRDs are not pinned; for example, the ones for Cast AI Live Migration and Pod Mutations. An uninstall deletes those, and the next install recreates them. Any custom resources of that kind are lost during that window.
• castai-agent-cpvpa-style helper workloads from older installs are preserved automatically.
• Clusters using the Cast AI Terraform module should wait for the module's migration path to complete rather than use this flow.
• generate never mutates the cluster, so you can re-run it freely. For example, run it again with a different --format for another tool (OpenTofu and Spacelift use --format terraform; OpenShift GitOps and ApplicationSets use --format argocd).

See also