Configuring Kvisor features
Which upgrade method to useThe Helm commands on this page use the umbrella chart (
castai-helm/castai) by default. If you need to use a different method:
- castctl: To upgrade all Cast AI components at once without managing Helm flags:
This preserves your existing configuration. See the castctl documentation for installation and authentication instructions.castctl castware upgrade- Individual charts: If you installed each component as a separate Helm release (e.g., for ArgoCD or custom GitOps), replace the release name and chart reference with the component-specific ones (e.g.,
castai-workload-autoscalerandcastai-helm/castai-workload-autoscaler) and remove theautoscaler.castai-workload-autoscaler.value prefix.Not sure which method you used? Run
helm list -n castai-agent. A single release namedcastaimeans umbrella chart; separate releases likecastai-workload-autoscalermean individual charts.
This guide explains how to configure Kvisor's monitoring, telemetry, and security insight features. Kvisor provides several specialized capabilities that can be enabled and customized according to your needs.
Configuration overview
Kvisor supports multiple configuration options that can be set via Helm during installation or upgrade. The basic format for enabling or modifying features is:
Configuration-only changesThis command upgrades all components in the umbrella chart, not just Kvisor. To change configuration without upgrading component versions, pin the chart to your current version by adding
--version <your-current-chart-version>to the command. Runhelm list -n castai-agentto check your current version.
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.[configuration-option]=[value]The Kvisor chart has two kinds of settings. Chart values cover the agent and controller images, resources, and feature switches such as agent.gpu.enabled. Feature settings pass through the controller.extraArgs and agent.extraArgs values, one flag per setting. This page lists the settings for each feature.
To view the chart values for the chart version deployed in your cluster, run:
helm show values castai-helm/castai-kvisorScanning frequencies and intervals
Cast AI performs different types of security scans at various intervals. You can customize these frequencies to meet your specific requirements:
Image vulnerability scanning
Default behavior:
- Detection interval: Checks for new images every 30 seconds
- Concurrent scans: 1 image scanned at a time
- Trigger: Scans start automatically when new running images are detected
- Scan timeout: 10 minutes per image
- Scan job resources: CPU request 1m, limit 2; memory request 1Mi, limit 2Gi
Configure image scan detection interval
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-interval=60sConfigure concurrent image scans
To increase scanning performance for environments with many images:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-concurrent-scans=6Set the scan timeout
A single image scan runs for at most 10 minutes by default. To change the limit:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-timeout=15mConfigure the scan job resources
Each image scan runs as a Kubernetes Job with its own CPU and memory settings. The default CPU request is 1m with a limit of 2. The default memory request is 1Mi with a limit of 2Gi.
To change the resources for image scans:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-cpu-request=500m \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-cpu-limit=2 \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-memory-request=512Mi \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-memory-limit=4GiInclude secret detection in image scans
By default, image scans do not look for exposed secrets inside container images. This keeps scans fast. To include secret detection:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-disabled-analyzers=""Compliance checking
Compliance runs two kinds of checks. Configuration checks look for misconfigurations and RBAC issues in your cluster. Node benchmark checks validate kubelet and system components against the Cast AI security benchmark.
Default behavior:
- Configuration checks: Every 60 seconds
- Node benchmark checks: Every 5 minutes
Configure how often configuration checks run
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.kube-linter-scan-interval=120sConfigure how often node benchmark checks run
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.kube-bench-scan-interval=10mSummary of scan frequencies
| Security Feature | Default Frequency | Configuration Parameter |
|---|---|---|
| Image vulnerability scanning | Every 30 seconds | image-scan-interval |
| Configuration checks | Every 60 seconds | kube-linter-scan-interval |
| Node benchmark checks | Every 5 minutes | kube-bench-scan-interval |
Use a custom image
You can configure Kvisor to use a custom image from your private registry:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.image.repository=my-kvisor-repository \
--set autoscaler.castai-kvisor.image.tag=my-tag \
--set 'autoscaler.castai-kvisor.imagePullSecrets[0].name=my-pull-secret'Private image scanning
For detailed instructions on configuring private image scanning, refer to the dedicated Private Image Scanning documentation.
Excluding namespaces from scanning
You may want to exclude specific namespaces from image scanning, particularly for third-party monitoring tools or systems that manage their own security. To configure namespace exclusions:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-ignored-namespaces=namespace1,namespace2For example, to exclude the Dynatrace namespace from scanning:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-ignored-namespaces=dynatraceYou can specify multiple namespaces by separating them with commas.
Network traffic monitoring
Kvisor can collect Kubernetes network flows using eBPF. This data powers network cost monitoring and gives you visibility into pod-to-pod and pod-to-external traffic patterns.
To enable network traffic monitoring:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.netflow-enabled=trueKvisor batches the collected flows and exports them to Cast AI every 15 seconds by default. To change the export interval:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.extraArgs.netflow-export-interval=30s
NoteIf you have the
egressdcomponent running, it should be uninstalled before enabling network traffic monitoring in Kvisor:helm uninstall egressd -n castai-agent
Enrich network flows with cloud contextTo enrich netflow data with VPC, subnet, and region information from your cloud provider, see Cloud network context.
Resource statistics monitoring
Kvisor can collect resource usage statistics from containers and nodes. These monitoring capabilities are controlled by two independent flags, each enabling a distinct category of metrics.
Kubernetes storage metrics
Kvisor collects storage utilization data for both ephemeral storage and persistent volumes across your cluster nodes. This data surfaces per-node storage usage in the Node list of the Cast AI console.
Storage metrics collection requires both the Kvisor controller and agent to be enabled.
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.controller.enabled=true \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.storage-stats-enabled=trueBesides storage utilization directly in the node, Kvisor is also able to collect metrics on cloud volumes that are
being used. Collection happens inside the controller component and requires cloud access permissions, in order to fetch
volume related information. For now, only AWS and GCP are supported.
AWS
If IMDS (Instance Metadata Service) access is enabled for the Kvisor controller pod, no additional configuration is
required. Kvisor will automatically use the node's instance role to authenticate with AWS.
If IMDS access is not available (e.g. it has been restricted at the instance or pod level), you must set up
IRSA (IAM Roles for Service Accounts)
and annotate the Kvisor controller service account with the appropriate IAM role ARN. The IAM role must have
the following permission:
ec2:DescribeVolumes
Once authentication is configured, enable cloud volume metrics collection:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.controller.enabled=true \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.storage-stats-enabled=true \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider=aws \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider-aws-region=<your-aws-region> \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider-storage-sync-enabled=trueReplace <your-aws-region> with the AWS region your cluster is running in.
GCP
Cloud volume metrics collection on GCP requires Workload Identity
to be configured so that the Kvisor controller service account can authenticate with the GCP API. The bound GCP
service account must have the following permissions in your project:
compute.disks.listcompute.instances.list
Once Workload Identity is configured, enable cloud volume metrics collection:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.controller.enabled=true \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.storage-stats-enabled=true \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider=gcp \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider-gcp-project-id=<your-gcp-project-id> \
--set autoscaler.castai-kvisor.controller.extraArgs.cloud-provider-storage-sync-enabled=trueReplace <your-gcp-project-id> with your GCP project ID.
Pressure Stall Information (PSI) metrics
Kvisor can collect Pressure Stall Information (PSI) metrics from containers and nodes, along with CPU, memory, and I/O usage statistics. PSI metrics indicate how often workloads experience delays due to contention for CPU, memory, or I/O resources, helping you identify performance bottlenecks before they affect application behavior.
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.stats-enabled=trueKvisor collects resource statistics every 60 seconds by default. To change the collection interval:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.extraArgs.stats-scrape-interval=120sGPU metrics
Kvisor collects GPU metrics from NVIDIA GPUs in your cluster using DCGM Exporter. The GPU export pipeline and dcgm-exporter run only on nodes with GPUs attached, regardless of cloud provider. Kvisor itself continues to run on all cluster nodes.
To enable GPU metrics collection:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.agent.gpu.enabled=trueConfigure the GPU metrics interval
Kvisor collects GPU metrics every 15 seconds by default. To change the interval:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.gpu.exportInterval=30sUse an existing DCGM exporter
By default, Kvisor adds a DCGM exporter sidecar to its agent pods on GPU nodes. If your cluster already runs its own DCGM exporter, you can use it instead of the sidecar.
ImportantOnly disable the sidecar when every GPU node in your cluster runs a DCGM exporter. Kvisor reads the exporter on each node and reports no GPU metrics for nodes without one. The exporter must expose its metrics on port 9400. If your exporter uses a different port, set
agent.gpu.dcgmExporter.portto match.
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.gpu.dcgmExporter.enabled=falseShow per-pod GPU usage with GPU sharing
On clusters that use GPU sharing, such as MIG partitions or time slicing, each pod can use part of a GPU. Enable per-pod GPU metrics to see the usage of each pod:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.gpu.dcgmExporter.virtualGPUs.enabled=trueSupport DRA clusters
On clusters that use dynamic resource allocation (DRA) for GPUs, enable DRA support:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.gpu.dra.enabled=trueDRA support requires Kubernetes 1.32 or later with the DRA feature gates enabled and the NVIDIA DRA driver installed. See DRA.
Attribute GPU usage to custom workloads
If a pod label identifies your GPU workloads, you can attribute GPU metrics to the label value:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.extraArgs.gpu-workload-label-keys=workloads.cast.ai/custom-workloadThe label value overrides the workload name that Kvisor resolves for the pod.
Migrating from gpu-metrics-exporter
If you are currently using the standalone gpu-metrics-exporter component, follow these steps to migrate:
- Upgrade Kvisor and enable GPU metrics:
helm repo update castai-helm
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.agent.gpu.enabled=true- Uninstall the old component:
helm uninstall gpu-metrics-exporter -n castai-agent
ImportantKvisor detects if
gpu-metrics-exporteris present in the cluster and will not report GPU metrics while it remains installed. Uninstall the old component for GPU metrics to start flowing.
Resource configuration
For large clusters or environments with intensive monitoring requirements, you may need to increase the resources allocated to Kvisor:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.controller.resources.requests.cpu=100m \
--set autoscaler.castai-kvisor.controller.resources.requests.memory=2Gi \
--set autoscaler.castai-kvisor.controller.resources.limits.memory=2GiThe agent runs as a DaemonSet on every node, with a default memory request of 64Mi and a limit of 512Mi. To change the agent resources:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.resources.requests.memory=128Mi \
--set autoscaler.castai-kvisor.agent.resources.limits.memory=1GiAdjust these values based on your specific cluster size and monitoring needs.
Container runtime settings
Kvisor reads the container runtime socket to collect container details such as image digests. Clusters that use containerd with the default socket path need no configuration.
If your runtime socket is at a different path on your nodes, set the socket path:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.containerdSocketPath=<your-socket-path>On clusters that use CRI-O, also disable the containerd-specific features:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.agent.disableContainerD=trueKvisor then does not collect process trees and does not resolve image digests.
Combine multiple features
You can enable multiple features and configure scan intervals in a single Helm command:
helm upgrade castai castai-helm/castai -n castai-agent \
--reset-then-reuse-values \
--set autoscaler.castai-kvisor.castai.apiKey=<your-api-token> \
--set autoscaler.castai-kvisor.castai.clusterID=<your-cluster-id> \
--set autoscaler.castai-kvisor.controller.enabled=true \
--set autoscaler.castai-kvisor.agent.enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.netflow-enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.stats-enabled=true \
--set autoscaler.castai-kvisor.agent.extraArgs.storage-stats-enabled=true \
--set autoscaler.castai-kvisor.agent.gpu.enabled=true \
--set autoscaler.castai-kvisor.controller.extraArgs.image-concurrent-scans=6 \
--set autoscaler.castai-kvisor.controller.extraArgs.image-scan-interval=30s \
--set autoscaler.castai-kvisor.controller.extraArgs.kube-linter-scan-interval=60sVerify your configuration
To verify that your configuration changes have been applied correctly:
helm get values castai -n castai-agentThis command displays the current configuration values for Kvisor.
Troubleshooting
If you encounter issues after changing Kvisor's configuration:
Check controller logs
kubectl logs -l app.kubernetes.io/name=castai-kvisor-controller -n castai-agentCheck agent logs
kubectl logs -l app.kubernetes.io/name=castai-kvisor-agent -n castai-agentRestart the pods
If configuration changes don't seem to be taking effect, you can restart the Kvisor pods:
kubectl rollout restart deployment castai-kvisor-controller -n castai-agent
kubectl rollout restart daemonset castai-kvisor-agent -n castai-agentUpdated 6 days ago
