What are node templates?

Node templates are a key part of the Autoscaler component. They allow users to define virtual buckets of constraints and properties for nodes to be added to the cluster during upscaling. Users can:

• Specify various constraints to limit the inventory of instance types to be used.
• Select a preferred instance resource offering (e.g., build a spot-only cluster without the need for specific tolerations or selectors).
• Define how Cast AI should manage Spot Instance interruptions.
• Specify labels and taints to be applied on the nodes.
• Link a Node configuration to be applied when creating new nodes.

When a user enables Cast AI, the default Node template is created in the cluster. This template is essential and serves as the default Autoscaler behavior. In addition to the default template, users can set up multiple templates to suit their specific use case.

📘

Note

For the Cast AI autoscaler to function, the Unscheduled Pods policy and at least one Node template (default or not) must be enabled.

Main attributes

The table below outlines the Node template attributes and their availability across different cloud providers.

Processor Architecture Support

Cast AI supports different processor architectures across cloud providers through node templates. You can create architecture-specific node templates or multi-architecture templates that include both x86 and ARM options.

Using ARM on AKS

Cast AI supports ARM-based instances on Azure AKS. This feature currently requires:

1. Feature flag enablement for your organization (contact Cast AI support)
2. Specific configuration steps for your environment
3. Deployment in an ARM-supported Azure region
4. Quota for required ARM SKUs (such as Standard_D2ps_v5)

There may be additional costs associated with ARM image storage.

Contact Cast AI support for detailed guidance on implementing ARM support for your AKS clusters.

Create a Node template

• You have the following options to create Node templates:

  • Create Node template through the API
  • Create Node template through the UI ( Cluster --> Autoscaler --> Node templates )
  • Terraform

• Sometimes, when you create a Node template, you may want to associate it with custom Node configurations for provisioning nodes. You can achieve this by linking the template with a custom Node configuration.

Using the shouldTaint flag

While creating a Node template, you can choose whether the nodes created by the Cast AI Autoscaler should be tainted. This is controlled through shouldTaint property in the API payload.

🚧

When shouldTaint is set to false

Since no taints will be applied on the nodes created by Cast AI, any pods being deployed to the cluster, even the ones without nodeSelector, might get scheduled on these nodes. This effect might not always be desirable.

When shouldTaint is set to true

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  nodeSelector:
    scheduling.cast.ai/node-template: spark-jobs
  tolerations:
    - key: scheduling.cast.ai/node-template
      value: spark-jobs
      operator: Equal
      effect: NoSchedule
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

When shouldTaint is set to false

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  nodeSelector:
    scheduling.cast.ai/node-template: spark-jobs
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

Using nodeSelector

You can use nodeSelector to schedule pods on the nodes created using the template. By default, you construct nodeSelector using the template name. However, you can opt for a custom label if it suits your use case better.

Using a Node template name in nodeSelector

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  nodeSelector:
    scheduling.cast.ai/node-template: spark-jobs
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

Using multiple custom labels in nodeSelector

In case you have a Node template with multiple custom labels custom-label-key-1=custom-label-value-1 and custom-label-key-2=custom-label-value-2. You can schedule your pods on a node created using that Node template by providing nodeSelector with all the custom labels as described below:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  nodeSelector:
    custom-label-key-1: custom-label-value-1
    custom-label-key-2: custom-label-value-2
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

Using nodeAffinity

You can use nodeAffinity to schedule pods on the nodes created using the template. By default, you construct nodeAffinity using the template name. However, you may use a custom label to better fit your use case. The only supported nodeAffinity operator is In.

Using the Node template name in nodeAffinity

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: scheduling.cast.ai/node-template
              operator: In
              values:
                - "spark-jobs"
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

Using a Node template's custom labels in nodeAffinity

In case you have a Node template with multiple custom labels custom-label-key-1=custom-label-value-1 and custom-label-key-2=custom-label-value-2. You can schedule your pods on a node created using that Node template by providing nodeAffinity with all the custom labels as described below:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: custom-label-key-1
              operator: In
              values:
                - "custom-label-value-1"
            - key: custom-label-key-2
              operator: In
              values:
                - "custom-label-value-2"
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

Using a mix of nodeAffinity and nodeSelector

In case you have a Node template with multiple custom labels custom-label-key-1=custom-label-value-1 and custom-label-key-2=custom-label-value-2, you can schedule your pods on a node created using that Node template by providing nodeAffinity and nodeSelector with all the custom labels as described below:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-sleep
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
            - key: custom-label-key-1
              operator: In
              values:
                - "custom-label-value-1"
  nodeSelector:
    custom-label-key-2: custom-label-value-2
  containers:
    - name: busybox
      image: busybox:1.28
      args:
        - sleep
        - "1200"

More Specific Requirements/Constraints

Templates also support further instance type selection refinement via additional nodeAffinity/ nodeSelector values. This can be done if the additional constraints don't conflict with the template constraints.

Here's an example with some assumptions:

• The template has no constraints;
• The template is named my-template;
• The template has custom labels enabled, and they are as follows:

product: "my-product"
team: "my-team"

Here's an example deployment that would further specify what a pod supports/needs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "team"
                operator: In
                values: ["my-team"]
                # Pick only nodes that are compute-optimized
              - key: "scheduling.cast.ai/compute-optimized"
                operator: In
                values: ["true"]
                # Pick only nodes that are storage-optimized
              - key: "scheduling.cast.ai/storage-optimized"
                operator: In
                values: ["true"]
      nodeSelector:
        # template selector (can also be in affinities)
        product: "my-product"
        team: "my-team"
        # Storage optimized nodes will also have a taint, so we need to tolerate it.
      tolerations:
        - key: "scheduling.cast.ai/storage-optimized"
          operator: Exists
          # toleration for the template
        - key: "scheduling.cast.ai/node-template"
          value: "my-template"
          operator: "Equal"
        effect: "NoSchedule"
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 700m
            memory: 700Mi
          limits:
            memory: 700Mi

Using this example, you'd get a storage-optimized, compute-optimized node on a template that doesn't have those requirements for the whole pool.

Note: If a template has one of those constraints (on the template itself), there is currently no ability to loosen requirements for some pods based on their nodeAffinity/ nodeSelector values.

Node template matching

By default, workloads must specify node selectors or affinity that match all terms defined in the Node template's Custom labels. If at least one term is not matched, then the Node template is not considered.

A workload can match Multiple Node templates. In such cases, the templates are scored primarily by the number of terms that the Node template contains (there are additional scoring criteria). The greater the number of terms, the higher the score.

Partial Node template matching

Partial matching allows workloads to specify at least one of all terms of the Node template for it to be considered when provisioning a node for the workload. To enable partial Node template matching, go to Autoscaler settings, navigate to Unscheduled pods policy, and select Partial node template matching.

When partial matching is enabled, the Unscheduled Pods Policy Applied event will include information on how the Node template was selected. This can be used to troubleshoot autoscaler decisions.

Node template selection

When multiple Node templates are matched, they are sorted, and the first one is selected to create a node for the workload. The sorting criteria, in order of their priority, are:

• Node templates where all terms were matched are sorted higher.
• The count of matched terms, in descending order.
• The average price per CPU of instance types within the Node template, in descending order.
• The name of the template, in alphabetical order.

Node template CPU limit

You can enable and configure CPU limits on node templates to control the total number of CPU cores provisioned during cluster upscaling and rebalancing. When configured, the total number of CPU cores of all nodes provisioned using that template will not exceed the specified limit.

🚧

Caution

When using partial node template matching, it's important to note that if CPU limits are reached on the best-matching template, the pods will remain pending. This can impact your cluster's ability to schedule workloads effectively.

To set a CPU limit:

1. Enable the CPU limit toggle
2. Enter the maximum number of CPU cores allowed to provision for nodes under this template

Impact on Autoscaler

When a CPU limit is configured, the autoscaler will only provision nodes up to the specified CPU limit when responding to unscheduled pods. If pods cannot be scheduled due to the CPU limit, they will receive a pod event indicating that autoscaling failed due to the node template CPU limit.

Impact on Rebalancing

During rebalancing, the total CPU count may temporarily exceed the configured limit. However, after rebalancing, the final cluster state will respect the configured CPU limits.

Support of Dedicated (Sole tenant) nodes

📘

This is a new feature currently supported only on GKE clusters

Please note that the sole tenancy node group must be shared with the GKE cluster's project or organization.

Dedicated nodes (also known as Sole tenant nodes) are dedicated hosts in the customer's cloud account that can host VMs that are used in the Kubernetes clusters. The setup for GKE clusters can be found in the Google Cloud documentation.

Cast AI Autoscaler can be instructed to prefer dedicated hosts until capacity is available when the following parameters are set in the Node template:

Example of configuration setup in the API response:

    "dedicatedNodeAffinity": [
      {
        "instanceTypes": [
          "c2-node-60-240"
        ],
        "name": "test-nodegroup",
        "affinity": [
          {
            "operator": "IN",
            "key": "stnodegroup",
            "values": [
              "test-group"
            ]
          },
          {
            "operator": "IN",
            "key": "compute.googleapis.com/project",
            "values": [
              "test-123"
            ]
          }
        ],
        "azName": "europe-west1-b"

Please note that all constraints configured in the node template will be respected (e.g., the minimum CPU minCPU set to 16).

Once the dedicated host capacity is saturated, Cast AI will not add additional dedicated hosts to the node group. Instead, it will return to multi-tenant instances configured in the Node template. During the fallback to multi-tenant instances, the same availability zone will be selected as configured in the dedicatedNodeAffinity.

🚧

Currently, the Rebalancer does not simulate the deletion of instances running on dedicated nodes; therefore, rebalancing the whole cluster might produce some suboptimal results.

High availability

Maintaining minimum nodes per template

While Cast AI doesn't directly support setting a minimum node count per template, you can maintain a fixed number of nodes using a helper deployment. This is useful for maintaining a fixed set of nodes at all times for specific workloads.

Implementation

You can maintain a minimum number of nodes by creating a helper deployment. This deployment uses minimal resources to keep nodes active without affecting your actual workloads:

• Places one pod per node using podAntiAffinity
• Uses the desired node template via nodeSelector
• Prevents node removal using Cast AI's removal-disabled label:
  YAML

  autoscaling.cast.ai/removal-disabled: "true"
  

Use this helper deployment with the desired node template for which you always wish to have a fixed number of nodes.

Helper deployment

This is an example of a helper deployment that maintains 6 nodes for a specific template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: min-nodes-nt1 # Replace `nt1` with node template name
  namespace: kube-system
  labels:
    app.kubernetes.io/name: min-nodes-nt1 # Replace `nt1` with node template name
    autoscaling.cast.ai/removal-disabled: "true"
spec:
  replicas: 6  # This sets the minimum number of nodes; modify as needed
  selector:
    matchLabels:
      app.kubernetes.io/name: min-nodes-nt1 # Replace `nt1` with node template name
      autoscaling.cast.ai/removal-disabled: "true"
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: min-nodes-nt1 # Replace `nt1` with node template name
        autoscaling.cast.ai/removal-disabled: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - topologyKey: kubernetes.io/hostname
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: min-nodes-nt1 # Replace `nt1` with node template name
      nodeSelector:
        scheduling.cast.ai/node-template: nt1  # Specify your node template name here
      # Add required tolerations if using a spot-only node template
      tolerations:
        - key: scheduling.cast.ai/spot
          operator: Exists
      containers:
      - name: placeholder
        image: busybox
        command: ["sh", "-c", "tail -f /dev/null"]
        resources:
          requests:
            memory: 10Mi

This helper deployment works by:

1. Creating lightweight placeholder pods that consume minimal resources
2. Using podAntiAffinity to ensure one pod per node
3. Targeting your specific node template using nodeSelector
4. Preventing node removal with the autoscaling.cast.ai/removal-disabled label set to true
5. Including spot tolerations if the node template is configured for spot-only instances (optional)

Depending on your requirements, modify this helper deployment manifest to suit your desired node count.