This guide walks you through setting up OMNI for your cluster, from initial onboarding through creating your first edge location, configuring node templates, and provisioning compute capacity in the edge.

Before you begin

Ensure you meet the following requirements before setting up OMNI.

Cluster requirements

• Phase 2 cluster — A cluster with Cast AI automation enabled. Phase 2 clusters have the Cluster Controller, Evictor, and other automation components installed, allowing Cast AI to provision and manage nodes. If your cluster is in Phase 1 (read-only), you'll need to enable automation first.
• Supported cluster type — EKS, GKE, or AKS

Required tools

The following tools must be installed on your local machine:

brew install kubectl

brew install helm

brew install curl jq

brew install awscli

brew install --cask google-cloud-sdk

bash -c "$(curl -L https://raw.githubusercontent.com/oracle/oci-cli/master/scripts/install/install.sh)"

Cloud permissions

You'll need sufficient permissions in your cloud account(s) to create networking resources (VPCs, subnets, security groups) and compute instances in the regions where you want to create edge locations. See Required cloud permissions for details.

Step 1: Onboard your cluster to OMNI

Onboarding deploys the OMNI components to your cluster. You can onboard in two ways.

Option A: During Phase 2 onboarding (recommended for new clusters)

If you're onboarding a cluster from Phase 1 (Read-only) to Phase 2 (Automation), you can enable OMNI at the same time.

1. In the Cast AI console, navigate to your cluster

2. Follow the standard Phase 2 onboarding flow

3. Select Extend cluster to other regions and cloud providers under Advanced settings

The script will be updated to include INSTALL_OMNI=true.

4. Copy and run the script in your terminal

5. Wait for the script to complete (typically 1-2 minutes)

Option B: Enable OMNI on an existing Phase 2 cluster

If you already have a Phase 2 cluster with cluster optimization enabled:

1. Navigate to your cluster in the Cluster list
2. Click on the ellipsis and choose Cast AI features

3. Under Other features, check the box for Extend cluster to other regions and cloud providers

4. Copy the updated script and run it in your terminal

5. Wait for the script to complete (typically 1-2 minutes)

kubectl get pods -n castai-omni

NAME                                       READY   STATUS    RESTARTS   AGE
liqo-controller-manager-7cf59bcc64-xxxxx   1/1     Running   0          2m
liqo-crd-replicator-687bdc6f66-xxxxx       1/1     Running   0          2m
liqo-fabric-xxxxx                          1/1     Running   0          2m
liqo-ipam-8667dbccbb-xxxxx                 1/1     Running   0          2m
liqo-metric-agent-55cd8748c5-xxxxx         1/1     Running   0          2m
liqo-proxy-77c66dfb88-xxxxx                1/1     Running   0          2m
liqo-webhook-6f648484cc-xxxxx              1/1     Running   0          2m
omni-agent-595c4b97d9-xxxxx                1/1     Running   0          2m

Step 2: Create and onboard an edge location

Edge locations define the regions where edge nodes can be provisioned. Each edge location is cluster-specific and requires its own setup.

1. In the Cast AI console, navigate to Autoscaler → Edge locations

2. Click Create edge location to open up the creation and configuration drawer
3. Configure the edge location:
   • Name: A descriptive name (e.g., aws-us-west-2 or gcp-europe-west4)
   • Cloud provider: Select AWS or GCP, or OCI
   • Region: Select the target region

   📘

   GCP

   For GCP, providing the Project ID is also required.

4. Click Next
5. Copy and run the provided script in your terminal to establish the connection with the edge location

📘

Note

Before running the script, ensure your cloud CLI is authenticated and configured for the correct account and region.

AWS

Set your AWS profile to ensure the script creates resources in the correct AWS account:

Bash

export AWS_PROFILE=<your-aws-profile>
# Then run the provided onboarding script

If you're already using your default AWS credentials, you can skip setting the profile.

GCP

Set your active GCP project to ensure the script creates resources in the correct project:

Bash

gcloud config set project <your-project-id>
# Then run the provided onboarding script

This should match the Project ID you provided when creating the edge location.

The script will:

• Create a VPC/network and subnet (if needed)
• Configure firewall rules and security groups
• Create service accounts or IAM users with appropriate permissions
• Register the edge location with Cast AI

Wait for the script to complete (typically 2-3 minutes).

After successful completion, the edge location appears in the Edge locations list with an Incomplete setup status and a notification confirms creation.

Create additional edge locations (optional)

You can create multiple edge locations for the same cluster. Repeat the process above for each region where you want to provision edge nodes.

Step 3: Configure node templates for edge locations

Node templates control where the Autoscaler can provision nodes. To enable edge node provisioning, add edge locations to your node templates.

1. Navigate to Autoscaler → Node templates
2. Select an existing node template or create a new one
3. In the node template editor, find the Edge locations section and check the box to Enable provisioning in edge locations
4. Select one or more edge locations from the dropdown

5. Click Save

When edge locations are selected:

• The Instance constraints section is updated to account for inventory from all selected edge locations
• The Available instances list includes instances from the main cluster region and all selected edge locations
• Autoscaler can now provision nodes in any of these locations based on cost and availability

After saving, the edge location status changes from Incomplete setup to In use.

Your cluster is now configured for edge node provisioning. The Autoscaler will automatically provision edge nodes as needed.

Edge node provisioning

Once configured, edge nodes are provisioned automatically by the Autoscaler based on:

• Cost optimization: Autoscaler compares Spot and On-Demand prices across the main cluster region and all edge locations configured in the node template
• Instance availability: Considers instances that are available in each region, including edge ones
• Node template constraints: Respects all CPU, memory, architecture, GPU requirements, and other constraints otherwise defined in the node template, as one would expect

How edge nodes appear in your cluster

Cast AI Console

In the Cast AI Console, edge nodes are identified in the Nodes list via an additional External region label in the Node list:

Using kubectl

Edge nodes appear as virtual nodes in your cluster:

kubectl get nodes

Example output:

NAME                                              STATUS   ROLE     AGE     VERSION
ip-192-168-56-192.eu-central-1.compute.internal   Ready    <none>   6h2m    v1.30.14-eks-113cf36
cast-7f6821f2-b9fd-47e0-ab38-1f80c9c32dc0         Ready    agent    6m20s   v1.30.14-eks-b707fbb
# The 2nd node is an edge node with ROLE=agent

Edge nodes can be identified by several characteristics:

Node labels:

• liqo.io/type=virtual-node: Identifies the node as a Liqo virtual node
• kubernetes.io/role=agent: Role designation for edge nodes
• omni.cast.ai/edge-location-name: Name of the edge location
• omni.cast.ai/edge-id: Unique edge identifier
• omni.cast.ai/csp: Cloud provider of the edge (e.g., gcp, aws)
• topology.kubernetes.io/region: Region where the edge node is located

Node taints:

• virtual-node.omni.cast.ai/not-allowed=true:NoExecute: Applied to all edge nodes by default

ProviderID: Edge nodes have a special provider ID format:

castai-omni://<identifier-string>

You can inspect an edge node to see all these identifiers:

kubectl describe node <node-name>

Scheduling workloads on edge nodes

To enable workloads to run on edge nodes, label the namespace to allow offloading:

kubectl label ns <namespace-name> omni.cast.ai/enable-scheduling=true

To verify the label was applied:

kubectl get ns <namespace-name> --show-labels

When you deploy workloads to a labeled namespace, a mutating webhook automatically adds the required toleration to your pods, allowing them to be scheduled on edge nodes.

This label enables Liqo's offloading mechanism for the namespace.

Explicit edge placement with nodeSelector

If you want to explicitly place specific workloads on edge nodes (rather than letting Autoscaler decide), you can use a nodeSelector in your pod specification. This approach gives you direct control over which workloads run on edge nodes.

apiVersion: v1
kind: Pod
metadata:
  name: my-edge-workload
  namespace: <your-offloaded-namespace>
spec:
  nodeSelector:
    liqo.io/type: virtual-node
  tolerations:
    - key: virtual-node.omni.cast.ai/not-allowed
      operator: Equal
      value: "true"
      effect: NoExecute
  containers:
    - name: my-container
      image: my-image:latest

You can also target a specific edge location:

nodeSelector:
  omni.cast.ai/edge-location-name: aws-us-west-2

Or a specific cloud provider:

nodeSelector:
  omni.cast.ai/csp: gcp

Workload compatibility

Not all workloads are suitable for running on edge nodes. Consider the following when deciding which workloads to offload:

Requirements (hard constraints):

• Linux x86_64 architecture only (ARM-based workloads are not supported)
• Stateless workloads or workloads that don't depend on persistent volumes (PVs cannot be offloaded to edge nodes)
• Must not be in default, kube-system, kube-public, or castai-omni namespaces

Recommendations:

• Workloads that can tolerate some additional network latency (cross-region or cross-cloud communication adds latency)
• Workloads with minimal to no dependencies on other in-cluster services
• Workloads that don't require Kvisor security/netflow monitoring

Given the above, workloads such as batch processing, GPU-intensive workloads that benefit from GPU availability more than low latency, and workloads where cost savings from cheaper GPU or compute instances justify the operational trade-offs, are prime candidates to be tested and offloaded to edge nodes.

Running DaemonSets on edge nodes

DaemonSets require an additional toleration to schedule pods on edge virtual nodes. Add this to your DaemonSet spec:

tolerations:
  - key: virtual-node.omni.cast.ai/not-allowed
    operator: Exists
    effect: NoExecute

Example DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-daemonset
  namespace: <your-offloaded-namespace>
spec:
  selector:
    matchLabels:
      app: my-daemonset
  template:
    metadata:
      labels:
        app: my-daemonset
    spec:
      tolerations:
        - key: virtual-node.omni.cast.ai/not-allowed
          operator: Exists
          effect: NoExecute
      containers:
        - name: my-container
          image: my-image:latest

Evictor behavior with edge nodes

The Evictor works with edge nodes but respects the edge node toleration requirement:

What Evictor can do:

• Evict workloads from edge nodes back to nodes in the main cluster when capacity is available
• Pack workloads across multiple edge nodes to optimize resource utilization
• Consider edge nodes in its bin-packing decisions

What Evictor cannot do:

• Place workloads on edge nodes unless they explicitly tolerate virtual-node.omni.cast.ai/not-allowed=true:NoExecute

This means:

• Workloads without the edge toleration will never be moved to edge nodes by Evictor
• Workloads with the edge toleration can be evicted from the main cluster to edges (and vice versa)
• You maintain control over which workloads can run on edge nodes through tolerations

📘

Note

Only add the edge node toleration to workloads that are compatible with running in different regions or clouds. Consider all requirements when deciding which workloads to allow on edge nodes.

Edge node provisioning time

Edge nodes typically take the same amount of time to become ready as nodes in your main cluster region would.

For GPU instances, provisioning may take slightly longer due to driver installation.

Troubleshooting

Common issues and solutions

Verification checklist

If you're experiencing issues, work through this checklist:

1. OMNI components deployed?

   Shell

   kubectl get ns castai-omni
   kubectl get pods -n castai-omni

   All pods should be in Running status.

2. Edge location status healthy? Check the Cast AI console → Autoscaler → Edge locations. Status should be In use, not Pending, Incomplete setup, or Failed.

3. Edge location added to node template? Verify your node template has edge locations enabled and at least one edge location selected.

4. Architecture set to x86_64? If edge locations are selected, the node template architecture must be x86_64.

5. Namespace labeled?

   Shell

   kubectl get ns <namespace> --show-labels | grep omni.cast.ai/enable-scheduling

6. Edge nodes visible?

   Shell

   kubectl get nodes | grep agent