Node configuration

What is node configuration?

The CAST AI provisioner allows you to set node configuration parameters that the platform will apply to provisioned nodes.

A cluster can have multiple Node Configurations. However, you can select only one as the default the Autoscaler will use. You can link the default and other node configurations to Node Templates.

📘

You can link node configuration to multiple node templates, but one node template can have just a single node configuration link.

Node configuration on its own does not influence workload placement. Its sole purpose is to apply user-provided configuration settings on the node during the provisioning process.

You can manage node configurations via UI, API or Terraform.

Feature availability

Shared configuration options

The following table provides a list of supported cloud-agnostic configuration parameters:

📘

Check the list of images available for EKS in the AWS documentation

EKS-specific subnet rules

📘

In EKS only subnets which match one of the rules below are allowed to be added to Node Configuration:

• association with a route table that has a 0.0.0.0/0 route to Internet Gateway, it's known as a public subnet. Subnet also must have "MapPublicIpOnLaunch: true" set.
• association with a route table that has a 0.0.0.0/0 route to Transit Gateway, it's known as a private subnet
• association with a route table that has a 0.0.0.0/0 route to NAT Gateway, it's known as a private subnet

Some configuration options are cloud provider specific. See the table below:

EKS-specific configuration options

KMS key for EBS volume

The key that you provide for the encryption of EBS volume must have the following policy:

  {
            "Sid": "Allow access through EBS for all principals in the account that are authorized to use EBS",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt",
                "kms:CreateGrant"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:CallerAccount": "<<account_ID>",
                    "kms:ViaService": "ec2.<<region>>.amazonaws.com"
                }
            }
}
    

module "kms" {
  source = "terraform-aws-modules/kms/aws"

  description = "EBS key"
  key_usage   = "ENCRYPT_DECRYPT"

  # Policy

  key_statements = [
    {
      sid =  "Allow access through EBS for all principals in the account that are authorized to use EBS",
      principals = [
        {
          type        = "AWS"
          identifiers = ["*"]
        }
      ]
      actions = [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:CreateGrant",
        "kms:DescribeKey"
      ],
      resources =  ["*"],
      conditions = [
        {

          test     = "StringEquals"
          variable = "kms:ViaService"
          values   = [
            "ec2.${var.cluster_region}.amazonaws.com",
          ]
        },
        {
          test     = "StringEquals"
          variable = "kms:CallerAccount"
          values   = [
            data.aws_caller_identity.current.account_id
          ]
        }
      ]}
  ]

# Aliases
  aliases = ["mycompany/ebs"]

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

GKE-specific configuration options

AKS-specific configuration options

How to create a node configuration

A default node configuration is created during cluster onboarding in the CAST AI-managed mode.

You can choose to modify this configuration or create a new one. If you add a new node configuration that will be applied to all newly provisioned nodes, you will have to mark it as default.

Node configurations are versioned, and when the CAST AI provisioner adds a new node, the latest version of the node configuration is applied.

A new configuration can't be applied to an existing node. If you want to upgrade node configuration on a node or a set of nodes, you need to delete an existing node and wait until Autoscaler replaces it with a new one or rebalance the cluster (fully or partially).

Kubelet configuration examples

You can find all available kubelet settings in the Kubernetes documentation – Kubelet Configuration. Please refer to the version of your cluster.

For example, if you want to add some specific custom taints during node startup, you could do it with the following snippet:

{
    "registerWithTaints": [
        {
            "effect": "NoSchedule",
            "key": "nodes-service-critical",
            "value": "true"
        }
    ]
}

The second example involves configuring kubelet image pulling and setting kube API limits like following:

{
    "eventBurst": 20,
    "eventRecordQPS": 10,
    "kubeAPIBurst": 20,
    "kubeAPIQPS": 10,
    "registryBurst": 20,
    "registryPullQPS": 10
}

Create node configuration with the CAST AI Terraform provider

Use the resource castai_node_configuration from CAST AI terraform provider.

Reference example:

resource "castai_node_configuration" "test" {
  name           = local.name
  cluster_id     = castai_eks_cluster.test.id
  disk_cpu_ratio = 5
  subnets        = aws_subnet.test[*].id
  tags           = {
    env = "development"
  }
  eks {
    instance_profile_arn = aws_iam_instance_profile.test.arn
    dns_cluster_ip       = "10.100.0.10"
    security_groups      = [aws_security_group.test.id]
  }
}