Reliability metrics

Reliability metrics gives you continuous visibility into the health of services running in your Kubernetes cluster. Once installed, it collects request rate, error rate, and latency data for all your workloads automatically. No changes to your application code, no sidecars, and no instrumentation libraries required.

The data appears in the Cast AI console under Cost Report → Reliability, alongside your cost data, so you can see which workloads are expensive and unreliable at the same time.

How reliability metrics works

A collector agent runs on each node in your cluster and observes HTTP and gRPC network traffic at the network level. The agent inspects request and response traffic without intercepting it. Your application code and behavior are not affected.

The collector ignores health-check and infrastructure traffic automatically, so routes like /healthz, /readyz, /metrics, and /actuator/* do not inflate your request rate or error rate figures.

Measurements are aggregated within the cluster and exported to Cast AI, where they appear in the Reliability views within a few minutes.

Storage in your cluster

Reliability metrics provisions a ClickHouse time-series database inside your cluster for temporary metric storage. By default, 10 GiB of persistent storage is allocated. castctl sizes both the collector agent and this database automatically based on your cluster. See how castctl sizes components for details.

Supported protocols

Understanding the metrics

The RED method

Reliability metrics follows the RED method: Rate (how many requests per second a service handles), Errors (how many are failing), and Duration (how long they take to complete). These three signals together are enough to detect most service health problems and are the standard starting point for reliability monitoring.

Metrics available in the console

Unhealthy workloads

A workload is classified as unhealthy when its error rate exceeds 5% over the selected time range. The Reliability tab shows a summary count of how many workloads in your cluster meet this threshold.

Data freshness

Metrics appear in the console within approximately 2–3 minutes of the corresponding requests occurring. The most recent 1–2 minutes of data may not yet be reflected at any given time.

Navigating the Reliability tab

Cluster Reliability view

Navigate to Cost Report → Reliability to see your cluster's overall health. This view shows:

• Summary cards: current request rate, error rate, and P95 latency across the entire cluster, each with a trend indicator comparing the current period to the previous one
• Traffic chart: time series of request rate and error rate
• Latency chart: time series of P50, P95, and P99 latency
• Daily details table: one row per day in the selected range, with availability %, request rate, error rate %, P50, P95, and P99 latency

If reliability metrics is not yet installed, this view shows an Enable reliability metrics button instead.

Workload Reliability view

Navigate to Cost Report → Workloads → [workload name] → Reliability to see the same charts and table scoped to a single workload.

Workloads list with reliability columns

The Reliability tab in the workloads list shows all workloads that have collected data. Columns include request rate, error rate, P95 latency, and cost, letting you identify workloads that are both expensive and unhealthy in one view.

Time range options

The Reliability views use shorter preset ranges than the general cost report, suited for operational monitoring:

• Last 15 minutes
• Last 30 minutes
• Last hour
• Last 6 hours
• Last 24 hours

Ask the Cast AI agent

If your organization uses the Cast AI agent, you can ask natural language questions about workload reliability directly in the console. Examples:

• "What is the error rate of the payments service in production?"
• "Is the checkout deployment experiencing high latency?"
• "Show me the request rate for the api-gateway over the last hour."

The agent queries the same data shown in the Reliability views and returns a plain-language answer.

Enable reliability metrics

The recommended way to enable reliability metrics is with castctl, which auto-sizes the collector and metrics database for your cluster. If you prefer to manage the installation yourself with Helm, see Helm charts in the reference.

Prerequisites

Before enabling reliability metrics, confirm:

• You have kubectl configured for the target cluster with cluster administrator permissions
• You have a Cast AI API token
• Your cluster has a default storage class (required for the metrics database)

You can install reliability metrics with castctl (recommended) or directly with the Cast AI Helm chart. The steps below cover the castctl path. For Helm, including GitOps, see Helm charts in the reference.

Install and authenticate castctl

Install castctl following the castctl installation guide.

Once authenticated, enable reliability metrics on your cluster.

Run the install command

Run castctl cluster connect with the --reliability-metrics flag. This command handles the full installation: it inspects your cluster's workload density, sizes the collector agent and metrics database accordingly, and waits until it confirms data is flowing before finishing.

Interactive (prompts you to select features):

castctl cluster connect

When the Feature Selection prompt appears, choose Reliability Metrics from the list.

Non-interactive (for scripts and CI):

castctl cluster connect \
  --non-interactive \
  --reliability-metrics

Combined with other Cast AI features:

castctl cluster connect \
  --non-interactive \
  --cluster-name my-cluster \
  --reliability-metrics \
  --cluster-optimization \
  --workload-autoscaler

Preview what will be installed without making changes:

castctl cluster connect --dry-run --reliability-metrics

After the install finishes, castctl confirms the full pipeline is running. If it times out, it prints a verification command you can run to check the status manually.

Available castctl flags for reliability metrics

Use these flags with castctl cluster connect to control the installation:

Enable via the console (for already-connected clusters)

If your cluster is already connected to Cast AI, you can enable reliability metrics without running castctl cluster connect again:

1. Navigate to Cost Report → Reliability for the cluster.
2. Click Enable reliability metrics.
3. A setup dialog opens with your API token and cluster ID pre-filled.
4. Copy and run the castctl command shown in the dialog.

Upgrade reliability metrics

To upgrade all Cast AI components in your cluster, including reliability metrics:

castctl castware upgrade

This checks for newer versions of all installed components and upgrades them, preserving your existing configuration. Any new data migrations run automatically when the collector restarts. Use --non-interactive in CI pipelines:

castctl castware upgrade --non-interactive

Limitations

Default ports only. The collector monitors traffic on ports 8080, 8443, 8090, and 6379 by default. When installing with Helm, you can add ports by setting autoscaler.castai-kvisor.agent.reliabilityMetrics.obi.openPorts (a comma-separated list). See Helm charts in the reference.

Server-side metrics only. The collector measures each request from the receiving service's perspective. This intentionally avoids double-counting when the client and server of a request are both running in the same cluster. Outbound call metrics from client services are not shown separately.

Database and messaging metrics not yet in the console. Database query duration and message queue operation metrics are collected by the agent but are not yet visible in the Cast AI console. They will appear in a future release.

Data freshness. The most recent approximately 2 minutes of data are not yet available in the console at any given time. This is by design to ensure accuracy before data is displayed.

Latency percentile accuracy. Latency percentiles (P50, P95, P99) use linear interpolation within measurement buckets. For workloads with unusual latency distributions (such as strongly bimodal response times), percentile values near histogram bucket boundaries may be approximate.

ClickHouse operator requirement. Installing reliability metrics provisions a ClickHouse database in your cluster using the Altinity ClickHouse Operator. If your cluster restricts Custom Resource Definition permissions, you may need to install the operator CRDs manually before running the installer. castctl detects an existing operator automatically and skips that step if one is already running.

Troubleshooting

The Reliability tab shows "Enable reliability metrics"

Reliability metrics is not installed on this cluster. Click the button to open the setup dialog and follow the installation steps.

The Reliability tab shows "No data" after installation

The collector may still be starting up. Wait 2–5 minutes after installation and refresh the page.

If the issue persists, confirm that your services are running and receiving traffic on ports 8080, 8443, 8090, or 6379. Traffic on other ports is not collected.

Data appears but stops updating

Check that the collector DaemonSet is healthy on all nodes:

kubectl get daemonset castai-kvisor-agent -n castai-agent

The READY count should match the DESIRED count. If nodes are missing, check the pod logs for errors:

kubectl logs -n castai-agent -l app=castai-kvisor-agent --tail=50

A service is not appearing in the Reliability workloads list

The service is likely listening on a port outside the default set (8080, 8443, 8090, 6379). When installing with Helm, you can add ports by setting autoscaler.castai-kvisor.agent.reliabilityMetrics.obi.openPorts (a comma-separated list). See Helm charts in the reference.