This reference documents all labels, annotations, and Kubernetes events used by Container Live Migration. Use this page to configure workloads, troubleshoot migrations, or monitor migration status.

Labels

Labels control or show how Cast AI's live controller and Evictor handle workloads during migration operations.

Node labels

These labels are applied by Cast AI to nodes to indicate live migration capability.

Pod labels

These labels are applied to pods either automatically by the live controller or manually to control migration behavior.

Opting out of live migration

To disable live migration for specific workloads, add the opt-out label to your pod spec:

metadata:
  labels:
    autoscaling.cast.ai/live-migration-disabled: "true"

To prevent removal entirely (allowing only migration, with recovery on failure):

metadata:
  labels:
    autoscaling.cast.ai/removal-disabled: "true"

These labels can be combined. When both are set, the workload is excluded from both live migration and traditional eviction.

Annotations

Annotations control specific behaviors during the migration restore phase. These are applied to the source pod before migration.

Restore behavior annotations

By default, init containers, PostStart hooks, and startup probes are skipped on the restored pod because the container state is fully preserved. In rare cases, you may need to override this behavior.

These annotations on the source pod control which init containers, startup probes, and postStart lifecycle hooks re-run when the pod is restored from the checkpoint. By default, the restored process resumes as-is — init containers do not re-run and lifecycle hooks are skipped. You opt in to re-running specific ones with these annotations.

All three accept the same value format:

When to use each annotation

cast.ai/restore-run-init-containers

• When to use: Your pod has init containers that must re-run on restore (e.g. config bootstrap, iptables setup, filesystem preparation). Alternatively, keep it absent if your init container is not idempotent and you want to skip it on restore (the default behavior).
• When it does NOT make sense: Your init containers only do image-pulling or one-time setup already reflected in the checkpointed filesystem. Re-running them is wasteful or harmful.

cast.ai/restore-run-startup-probe

• When to use: Your startup probe performs a check that must re-validate after restore (e.g. verifying external dependencies are reachable from the new node).
• When it does NOT make sense (default): The startup probe only gates initial container readiness. The restored container is already past startup. Re-running it can cause a false failure if the probe target is transiently unavailable.

cast.ai/restore-run-post-start-exec

• When to use: Your postStart hook performs registration or initialization that must happen on the new node (e.g. re-registering with a service mesh, re-opening a sidecar channel).
• When it does NOT make sense (default): The postStart hook does one-time setup already reflected in the checkpoint. Re-running it may conflict with the already-running process state.

Example: Forcing startup probe execution

metadata:
  annotations:
    cast.ai/restore-run-startup-probe: "container1,container2"

Example: Forcing init container execution

metadata:
  annotations:
    cast.ai/restore-run-init-containers: "init-config,init-deps"

Migration options

These options control individual migration behavior — TCP preservation, memory dump tuning, pod replacement, and restore hooks. They can be set per-migration, per-pod, or as cluster-wide defaults.

For cluster-wide Helm chart values that configure components, scheduling, and deployment, see the chart's values.yaml.

How options are resolved

Most migration options can be set at three levels. The system checks them in order — the first value set wins:

1. Migration CR — options on the Migration object created for a specific migration. This is the per-migration override.
2. Pod annotation — annotations on the source pod being migrated. This is the per-pod override; it only takes effect when the Migration CR does not already specify the option.
3. Helm chart default — cluster-wide defaults in the Helm values. This is the fallback when neither the CR nor the pod annotation sets the option.

Once resolved, the effective values are written back into the Migration object and a MigrationReconfigured event is emitted so you can audit what was applied.

TCP/IP stack options

These control whether and how established TCP connections survive migration. For the conceptual overview, see TCP migration modes. For the TC and VPC CNI implementation details, see Traffic Control and AWS VPC CNI.

For detailed configuration reference for each TCP preservation path, see Traffic Control and AWS VPC CNI.

withTCP

Controls whether the migration attempts to preserve established TCP sockets.

When it makes sense:

• Your workload has long-lived TCP connections (databases, message queues, gRPC streams) that must survive migration.
• You are running on EKS with AWS VPC CNI (the chart autodiscovers this and defaults to true), or you have the TC DaemonSet enabled.

When it does NOT make sense:

• Your application tolerates reconnects and you want the simplest, fastest migration. Set withTCP: false.
• You are on a cluster where neither the VPC CNI path nor the TC DaemonSet is available. TCP state will be checkpointed but cannot be meaningfully restored because the pod IP changes and peers have no translation mechanism.

withTCPDnatRewrite

When true, Kubernetes Service destination IPs in the checkpoint are rewritten to the real backend pod IPs so connections survive the migration. Without this, connections to Service IPs may route differently after the pod moves.

When it makes sense:

• Your pod talks to other pods through Kubernetes ClusterIP Services. Enabled by default for this reason.

When it does NOT make sense:

• Your pod connects directly to pod IPs (headless Services, direct pod-to-pod). The rewrite is harmless but unnecessary. There is rarely a reason to disable it.

withTCPSnat

Controls what happens to connections created through Source NAT (SNAT) — typically outbound connections to external endpoints.

• "close" — SNATed connections are closed before restore. The application must reconnect after migration.
• Any other value (e.g. "keep") — SNATed connections are kept in the checkpoint as-is.

When "close" makes sense (default):

• Almost always. SNATed connections tie back to the source node's IP. After migration, the source node changes and the SNAT state is no longer valid. Keeping them usually results in broken connections that the application cannot detect cleanly.

When "keep" makes sense:

• Rarely. Only if you have verified that the source IP and SNAT state are stable across migration (e.g. an egress proxy with a stable IP). Keeping stale SNATed connections will cause silent connection failures.

Iterative dump tuning

Live migration uses iterative pre-dumps to reduce the freeze time: the pod keeps running while successive memory snapshots are taken, and only the final dump freezes the process. These options control how many iterations are done and when to stop.

iterations

The maximum number of pre-dump iterations before the final freeze-and-dump. The actual number may be lower if the dirty-pages threshold is reached first.

When a higher value makes sense:

• The pod has a large memory footprint with moderate write churn. More iterations give more chances to converge on a small set of dirty pages, reducing freeze time.

When a lower value makes sense:

• The pod has very high memory churn (it dirties pages faster than they can be tracked). Additional iterations will not converge and just add total migration time. A lower value or optimizeFor: DumpTime is better.
• The pod is small or low-traffic. The default 10 is usually fine, but you can lower it to reduce total migration time.

dirtyPagesThreshold

The threshold (in 4 KB pages) below which the system decides the remaining dirty pages are small enough to do the final freeze-and-dump. This acts as the early-exit condition for the iterative dump strategy.

When optimizeFor is FreezeTime (the default), the system first tries to compute a statistical threshold from historical dirty-page stats. This Helm/CR value serves as the fallback if the statistical calculation fails (e.g. no stats available yet).

When to adjust:

• Lower if you want stricter convergence (fewer dirty pages at final dump → shorter freeze, but more iterations and longer total time).
• Higher if you want to finish faster at the cost of a slightly longer freeze. The freeze time is roughly proportional to the number of dirty pages that must be written during the final dump.

When it does NOT matter:

• When optimizeFor is DumpTime, this threshold is unused — the fast-dump strategy does a single dump with no iteration.

optimizeFor

Selects the dump strategy. Two values:

When FreezeTime makes sense (default):

• The application is latency-sensitive and must minimize the freeze window. This is the common case for user-facing services.

When DumpTime makes sense:

• The application tolerates a longer freeze (e.g. batch jobs, background workers) and you want to minimize total migration time and network transfer.
• The pod has very high memory churn where iterative pre-dumps never converge.

Requested freeze time

requestedFreezeTime

The expected freeze time for the migration. This is not a hard limit on the actual freeze — it is an estimate used by the scheduler to decide whether a migration can proceed given the available network bandwidth.

The scheduler estimates the migration data transfer size, divides it by the requested freeze time to get the required bandwidth, and checks whether the source and destination nodes have enough available bandwidth. If not, the migration is held until capacity frees up.

The annotation accepts a Go duration string (e.g. "5s", "10s", "500ms").

When a lower value makes sense:

• The workload is latency-sensitive and you want the scheduler to be conservative — it will only start migrations it estimates can finish within a tight freeze window. Be aware that this may cause migrations to be deferred if bandwidth is limited.

When a higher value makes sense:

• The workload tolerates a longer freeze and you want migrations to proceed more readily (the scheduler will accept migrations with larger estimated transfer sizes). This is appropriate for batch or background workloads.

When the default (5s) is fine:

• For most general-purpose workloads. Only adjust if the scheduler is either too aggressively deferring or too aggressively starting migrations.

Pod replacement policy

podReplacementPolicy

Controls how the source pod is replaced by the restored clone. Three values:

Resolution: first non-empty wins, highest precedence first:

1. Pod annotation live.cast.ai/migration-default-pod_replacement_policy (values: SameName, EvictFirst, KeepSource).
2. CR spec.options.pod_replacement_policy (per-migration override).
3. Helm default features.migrationOptions.default.podReplacementPolicy.
4. Per-owner default: StatefulSet → SameName; all others → KeepSource.

An invalid annotation value is logged as a warning and treated as absent; resolution falls through to the next tier.

For the full migration path matrix (PVC impact, pre-creation, controller blocking), see Pod replacement policy above.

When SameName makes sense:

• StatefulSets (enforced). The clone takes the source pod's name so the StatefulSet controller recognizes it as the same ordinal.
• Any workload that depends on the pod name (e.g. DNS-based discovery via pod name).

When EvictFirst makes sense:

• You want the source gone before the clone starts to free resources immediately. This is the fastest cleanup path but is not recoverable if the clone fails.

When KeepSource makes sense (default for non-StatefulSet):

• You want safety: the source pod stays alive until the clone is confirmed ready. If the clone fails, the original is recovered. This is the safest option for production workloads.

System annotations

These annotations are applied by Cast AI during the migration process. Do not modify these manually.

If you are unfamiliar with clone pods or destination nodes, please refer to the Overview document.

Internal annotations and labels (do not set manually)

The following are set and managed internally by the controller during migration.

On the restored (clone) pod

On the Migration object

Other internal labels

Quick reference

Per-migration options (CR → annotation → Helm default)

Restore control annotations (source pod only)

Migration events

Live migration progress is tracked through Kubernetes events on the migration custom resource. Use these events to monitor migration status and troubleshoot failures.

Viewing migration events

# List all migrations
kubectl get migrations -A

# Watch migrations in real-time
kubectl get migrations -A -w

# Get detailed events for a specific migration
kubectl describe migration <migration-name> -n <namespace>

Event reference

Migration phases

Events occur during specific phases of the migration process:

1. Pre-dump phase: Iterative memory snapshots while the container continues running. PreDumpFinished events indicate successful iterations. Multiple pre-dumps reduce final checkpoint time.

2. Checkpoint phase: Final memory dump and state capture. No specific event due to the briefness of this phase.

3. Restore phase: Container state restoration on the destination node. PodCreateFinished indicates the pod was created successfully.

4. Completion: MigrationFinished indicates the entire migration succeeded and the pod is running on the new node.

Pod naming

After migration, pods receive modified names to indicate they are restored clones:

StatefulSets retain their original pod names because their identity (and associated PVCs) must remain stable.

Quick reference

This section provides commonly used commands for working with Container Live Migration. Each command helps you verify configuration, check eligibility, or monitor migration activity.

Check if a node supports live migration

To verify which nodes in your cluster are configured for Container Live Migration, query for the migration-enabled label. Only nodes provisioned from CLM-enabled node templates will have this label applied.

kubectl get nodes -l live.cast.ai/migration-enabled=true

If no nodes are returned, either CLM is not enabled in your node templates, or you need to trigger a rebalancing operation to replace existing nodes with CLM-enabled ones.

Check if a pod is eligible for live migration

The live controller automatically labels pods that meet all eligibility requirements. Use this command to see which pods in your cluster are currently eligible for live migration.

kubectl get pods -A -l live.cast.ai/migration-enabled=true

Pods that do not appear in this list either fail to meet the technical requirements or are running on nodes where the live controller has not yet completed its assessment.

Opt a workload out of live migration

If you need to exclude a specific workload from live migration while still allowing traditional eviction, add the opt-out label to your pod specification. This is useful for workloads that have known compatibility issues or require additional testing before enabling migration.

metadata:
  labels:
    autoscaling.cast.ai/live-migration-disabled: "true"

Monitor active migrations

During a rebalancing operation or when Evictor triggers migrations, you can watch migration resources in real-time to observe progress and catch any failures as they occur.

kubectl get migrations -A -w

The -w flag keeps the command running and displays updates as migration status changes. Look for MigrationFinished events to confirm successful migrations, or MigrationFailed events that may require investigation.

See also