View detailed breakdown of cluster costs

This page shows you how to view a breakdown of your Google Kubernetes Engine (GKE) cluster costs in Cloud Billing BigQuery export using GKE cost allocation.

You can use this information to distribute the costs of a cluster to its users.

Potential data volume increase

Enabling GKE cost allocation can increase your storage and querying costs in BigQuery. The exact increase depends on the number of distinct label and namespace combinations you use across Pods and clusters.

Enabling GKE cost allocation does not change the total cost of your GKE usage. The sum of the cost line items in your export is the same and existing queries and reports you have built on the billing export return the same values.

GKE cost allocation and cluster usage metering

GKE cost allocation is different from cluster usage metering in the following ways:

• GKE cost allocation provides an alternative to cluster usage metering for getting cluster information.
• GKE cost allocation calculates cluster costs in your Cloud Billing account instead of aggregating the data in a Looker Studio template.
• GKE cost allocation lets you view a clusters cost data in the Cloud Billing Google Cloud console and in the Cloud Billing detailed export.

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

• Ensure that you have completed the steps to Export detailed usage cost data to BigQuery. If your organization is already exporting data, you must have access to query the tables.

Limitations and restrictions

• You can only view GKE cost allocation data in the Cloud Billing BigQuery detailed usage cost data export.
• You must have Google Cloud CLI version 392.0.0 or later
• If you enable GKE cost allocation, your billing export starts including additional line items for your GKE resources from that date. The billing export does not backfill data.
• If you disable GKE cost allocation, your billing export stops including additional line items, but does not modify or remove previous line items that were generated while the feature was enabled.
• GKE cost allocation data is based on resource requests, not resources consumed. To learn more about the differences between resource requests, resource limits, and resource consumption, see Kubernetes best practices: Resource requests and limits.
• GKE cost allocation supports the following resource SKU types:
  • Compute Engine VM Instance vCPU SKUs
  • Compute Engine VM Instance RAM SKUs
  • Compute Engine VM Instance GPU SKUs
  • Compute Engine PD Capacity SKUs
• If a Pod has more than 50 Kubernetes labels, none of those labels will be available in the Cloud Billing section of the Google Cloud console or the Cloud Billing detailed export.
• After you enable GKE cost allocation, it can take up to three days for data to appear in Cloud Billing.
• GKE cost allocation includes data from Persistent Disks with the following exceptions:
  • Only supports PersistentVolume resources provisioned dynamically through PersistentVolumeClaims or Generic Ephemeral Volumes.
  • Only supports PersistentVolume resources provisioned by the Compute Engine persistent disk CSI driver or with the deprecated kubernetes.io/gce-pd storage plugin with ReadWriteOnce or ReadWriteOncePod access mode.
  • Persistent Disks with an uptime of less than 30 minutes might not be tracked. Persistent Disks with at least 30 minutes uptime are always tracked.

View GKE cluster costs

When you enable GKE cost allocation, the cluster name and namespace of your GKE workloads appear in the labels field of the billing export to BigQuery:

For more information, see Schema of the detailed usage cost data.

Additionally, the following namespaces are used to track overhead and unallocated resources:

• kube:system-overhead: This accounts for node resources that are not available to Pods. These resources are set aside by the system for running the Kubernetes framework (including but not limited to kubelet, kube-proxy, containerd). This is the difference between the Node's capacity and allocatable resources.
• kube:unallocated: Resources are neither requested by workloads nor requested for system overhead.

If GKE cannot determine the cost allocation for a resource, the export includes one of the following values:

• goog-k8s-unknown: Cloud Billing couldn't process the SKU. This can occur when a new Compute Engine instance is being provisioned. You can expect some goog-k8s-unknown values during node startup and shutdown, for example when GKE autoscales a cluster.
• goog-k8s-unsupported-sku: GKE cost allocation does not support this SKU. Treat this the same as <blank>/NULL. There is no guarantee that a SKU is always labeled with this value.
• <blank>/NULL: GKE cost allocation does not track this resource. This can occur when GKE cost allocation is not enabled, or the resource does not belong to a GKE managed cluster.

Costs for Persistent Disks are tracked when the Persistent Disk's backing PersistentVolumeClaims are claimed by Pods. The Persistent Disk costs inherit all of the metadata, such as Pod labels and namespaces, of the claiming Pod, until their PersistentVolumeClaims are deleted. For PersistentVolumeClaims that are deleted but retain PersistentVolumes according to the reclaim policy, costs of the Persistent Disks are tracked as unallocated.

Labels applied directly to PersistentVolumeClaims do not appear in the Cloud Billing section of the Google Cloud console or the Cloud Billing detailed export.

Enable GKE cost allocation

You can enable GKE cost allocation on a new or existing cluster using the gcloud CLI or the Google Cloud console.

Create a new cluster

gcloud container clusters create CLUSTER_NAME \
    --enable-cost-allocation \
    --region=COMPUTE_REGION

Update a cluster

gcloud container clusters update CLUSTER_NAME \
    --enable-cost-allocation

Verify GKE cost allocation is enabled

gcloud container clusters describe CLUSTER_NAME

...
costManagementConfig:
  enabled: true
...

Filter GKE cost data

The following example queries show you how to filter and group your GKE costs for supported resource types by cluster name, namespace, and label.

SELECT
  labels.value as namespace,
  SUM(cost) + SUM(IFNULL((SELECT SUM(c.amount) FROM UNNEST(credits) c), 0)) AS cost_after_credits,
FROM `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
LEFT JOIN UNNEST(labels) as labels
  ON labels.key = "k8s-namespace"
GROUP BY namespace
;

SELECT
  SUM(cost) AS cost_before_credits,
  labels.value AS cluster_name
FROM  `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
LEFT JOIN UNNEST(labels) as labels
  ON labels.key = "goog-k8s-cluster-name"
GROUP BY labels.value
;

SELECT
  project.id AS project_id,
  labels.value AS cluster_name,
  sku.id AS sku_id,
  sku.description AS sku_description,
  SUM(cost) AS cost
FROM `project-ID.dataset.gcp_billing_export_resource_v1_XXXXXX-XXXXXX-XXXXXX`
JOIN UNNEST(labels) AS labels
  ON labels.key = "goog-k8s-cluster-name"
GROUP BY
  cluster_name, project_id, sku_description, sku_id
;

Replace BILLING_DATASET_TABLE with the name of the dataset that you created in BigQuery.

The table name is similar to gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this page, disable GKE cost allocation for the cluster:

gcloud container clusters update CLUSTER_NAME \
    --no-enable-cost-allocation

You can also delete the cluster that you created.

What's next

• View more Example queries for Cloud Billing data export.
• Learn how to Visualize your costs with Looker Studio.