Node configuration
What is node configuration?
The CAST AI provisioner allows you to set node configuration parameters that the platform will apply to provisioned nodes.
A cluster can have multiple Node Configurations. However, you can select only one as the default the Autoscaler will use. You can link the default and other node configurations to Node Templates.
You can link node configuration to multiple node templates, but one node template can have just a single node configuration link.
Node configuration on its own does not influence workload placement. Its sole purpose is to apply user-provided configuration settings on the node during the provisioning process.
You can manage node configurations via UI, API or Terraform.
Feature availability
EKS | AKS | GKE | KOPS |
---|---|---|---|
+ | + | + | - |
Shared configuration options
The following table provides a list of supported cloud-agnostic configuration parameters:
Configuration | Description | Default value |
---|---|---|
Root volume ratio | CPU to storage (GiB) ratio | 1 CPU: 0 GiB |
Image | Image to be used when building CAST AI provisioned node | The latest available for the Kubernetes release |
SSH key | Base64-encoded public key or AWS key ID | "" |
Subnets | Subnet IDs for CAST AI provisioned nodes | All subnets pointing to NAT/Internet Gateways inside the cluster's VPC |
Instance tags | Tags/VM labels to be applied on CAST AI provisioned nodes | [] |
Check the list of images available for EKS in the AWS documentation
EKS-specific subnet rules
In EKS only subnets which match one of the rules below are allowed to be added to Node Configuration:
- association with a route table that has a 0.0.0.0/0 route to Internet Gateway, it's known as a public subnet. Subnet also must have "MapPublicIpOnLaunch: true" set.
- association with a route table that has a 0.0.0.0/0 route to Transit Gateway, it's known as a private subnet
- association with a route table that has a 0.0.0.0/0 route to NAT Gateway, it's known as a private subnet
Some configuration options are cloud provider specific. See the table below:
EKS-specific configuration options
Configuration | Description | Default value |
---|---|---|
Security groups | Security group IDs for nodes provisioned in CAST AI | Tagged and CAST AI SG |
Instance profile ARN | Instance profile ARN for CAST AI provisioned nodes | cast-<cluster-name>-eks-<cluster-id> (only the last 8 digits of the cluster ID) |
Dns-cluster-ip | Override the IP address to be used for DNS queries within the cluster | "" |
Container runtime | Container runtime engine selection: docker or containerd | Unspecified |
Init script | A script to be run when building the node | bash "" |
Docker configuration | A set of values that will be overwritten in the Docker daemon configuration | JSON {} |
Kubelet configuration | A set of values that will be added or overwritten in the kubelet configuration | JSON {} |
Volume type | EBS volume type to be used for provisioned nodes | gp3 |
Volume IOPS | EBS volume IOPS value to be used for provisioned nodes | 3000 |
KMS Key ARN | Customer-managed KMS encryption key to be used when encrypting EBS volumes | Unspecified |
Volume throughput | EBS volume throughput in MiB/s to be used for provisioned nodes | 125 |
Use IMDS v1 | IMDSv1 and v2 are enabled by default, else only IMDSv2 will be allowed | True |
KMS key for EBS volume
The key that you provide for the encryption of EBS volume must have the following policy:
{
"Sid": "Allow access through EBS for all principals in the account that are authorized to use EBS",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:Encrypt",
"kms:DescribeKey",
"kms:Decrypt",
"kms:CreateGrant"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:CallerAccount": "<<account_ID>",
"kms:ViaService": "ec2.<<region>>.amazonaws.com"
}
}
}
module "kms" {
source = "terraform-aws-modules/kms/aws"
description = "EBS key"
key_usage = "ENCRYPT_DECRYPT"
# Policy
key_statements = [
{
sid = "Allow access through EBS for all principals in the account that are authorized to use EBS",
principals = [
{
type = "AWS"
identifiers = ["*"]
}
]
actions = [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:CreateGrant",
"kms:DescribeKey"
],
resources = ["*"],
conditions = [
{
test = "StringEquals"
variable = "kms:ViaService"
values = [
"ec2.${var.cluster_region}.amazonaws.com",
]
},
{
test = "StringEquals"
variable = "kms:CallerAccount"
values = [
data.aws_caller_identity.current.account_id
]
}
]}
]
# Aliases
aliases = ["mycompany/ebs"]
tags = {
Terraform = "true"
Environment = "dev"
}
}
GKE-specific configuration options
Configuration | Description | Default value |
---|---|---|
Network tags | A string to be added to a tags field in a GCP VM resource | Empty |
Max pods per node | Maximum number of pods to be hosted on a node | 30 |
Boot disk | Boot disk storage type | balanced as per GKE documentation |
AKS-specific configuration options
Configuration | Description | Default value |
---|---|---|
Max pods per node | Maximum number of pods to be hosted on a node | 30 |
How to create a node configuration
A default node configuration is created during cluster onboarding in the CAST AI-managed mode.
You can choose to modify this configuration or create a new one. If you add a new node configuration that will be applied to all newly provisioned nodes, you will have to mark it as default.
Node configurations are versioned, and when the CAST AI provisioner adds a new node, the latest version of the node configuration is applied.
A new configuration can't be applied to an existing node. If you want to upgrade node configuration on a node or a set of nodes, you need to delete an existing node and wait until Autoscaler replaces it with a new one or rebalance the cluster (fully or partially).
Kubelet configuration examples
You can find all available kubelet settings in the Kubernetes documentation – Kubelet Configuration. Please refer to the version of your cluster.
For example, if you want to add some specific custom taints during node startup, you could do it with the following snippet:
{
"registerWithTaints": [
{
"effect": "NoSchedule",
"key": "nodes-service-critical",
"value": "true"
}
]
}
The second example involves configuring kubelet image pulling and setting kube API limits like following:
{
"eventBurst": 20,
"eventRecordQPS": 10,
"kubeAPIBurst": 20,
"kubeAPIQPS": 10,
"registryBurst": 20,
"registryPullQPS": 10
}
Create node configuration with the CAST AI Terraform provider
Use the resource castai_node_configuration
from CAST AI terraform provider.
Reference example:
resource "castai_node_configuration" "test" {
name = local.name
cluster_id = castai_eks_cluster.test.id
disk_cpu_ratio = 5
subnets = aws_subnet.test[*].id
tags = {
env = "development"
}
eks {
instance_profile_arn = aws_iam_instance_profile.test.arn
dns_cluster_ip = "10.100.0.10"
security_groups = [aws_security_group.test.id]
}
}
Updated 3 days ago