Certificate rotation is a regular maintenance task that helps maintain cluster security. This guide explains how to handle the process in clusters managed by Cast AI.

The process involves:

1. Rotating certificates at your cloud provider level
2. Ensuring Cast AI components properly recognize and use the new certificates

Before you begin

• Ensure you have access to perform node rebalancing in the Cast AI console
• Review your cloud provider's certificate rotation documentation:
  • GCP - Rotating Certificates

🚧

Important

Incorrect certificate rotation steps may cause Cast AI nodes to lose connection with the control plane. Follow the steps in exact order to prevent connectivity issues.

Additionally, rotating Kubernetes certificates may result in your cluster being temporarily unavailable as components are restarted. For production environments, perform this action during a maintenance window.

Rotate certificates in GKE

1. Start certificate rotation

   • Access your GKE console or use the gcloud CLI to initiate the rotation process
   • Do not complete the rotation yet - this happens in step 5

2. Delete Cast AI node pools

   • Delete all GKE node pools used by Cast AI. Including cast-pool:
     

• If you have the kube-gke-pools-per-node-configuration-enabled feature flag enabled for your organization, delete all Cast AI-created node pools.

📘

Note

Review your CSP's documentation for instructions on deleting node pools:

• GCP - Rotating Certificates

3. Trigger cluster reconciliation
   • In the Cast AI console, navigate to the cluster list
   • In the row of your cluster, click "Actions" ("⋮") and select "Trigger reconcile":
     
     This step ensures Cast AI GKE Node Pools use the updated certificates.
4. Replace existing nodes
   Choose one of these methods to replace nodes with the new certificates:
   • Use Scheduled rebalancing to gradually replace nodes. Set "Minimum Node Age" to zero to ensure all nodes get rotated.
   • Manually rebalance specific node groups to control the process.

📘

Note

For large clusters, consider rebalancing in smaller batches to avoid hitting cloud provider quotas.

5. Complete certificate rotation
   Return to GKE to complete the rotation process. Follow the GCP documentation for detailed steps.

If you need assistance during the certificate rotation process, contact Cast AI support via our community Slack channel.

Steps for other cloud providers

Guides for certificate rotation in EKS and AKS clusters are coming soon.