We have just released version v0.14.0 of our managed collection for Kubernetes and v2.45.3-gmp.9-gke.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.31 or newer. Self-deployed collection users should upgrade their binaries to use v2.45.3-gmp.9-gke.0.

Changes in managed collection v0.14.0:

• [SECURITY] Use scale subresource to update replica counts.
• [SECURITY] Do not automount initcontainer service account token in Managed Alertmanager.
• [SECURITY] Remove operator RBAC permissions to update managed collector, rule-evaluator, and alertmanager deployments.
• [ENHANCEMENT] Add mdox formating and binary flag doc autogen for better READMEs for our binaries.
• [ENHANCEMENT] Prevent collector pod restarts on startup.
• [ENHANCEMENT] Default OperatorConfig's external labels for project_id, location, and cluster from values passed to or auto-discovered by the operator to be explicit about relabeling.
• [ENHANCEMENT] Default to Google Cloud links for Alerts in Managed Alertmanager.
• [ENHANCEMENT] Allow updating collector, rule-evaluator, and alertmanager options at runtime.
• [ENHANCEMENT] Add samples sent error counter gcm_export_samples_sent_errors_total when encountering errors on export.
• [ENHANCEMENT] Allow GRAFANA_SERVICE_ACCOUNT_TOKEN environment variable for datasource-syncer
• [ENHANCEMENT] Use protobuf encoding for core K8s API communication
• [BUGFIX] cadvisor and kubelet ClusterNodeMonitoring examples regex fix.

Changes in self-deployed collection v2.45.3-gmp.9-gke.0:

• [ENHANCEMENT] Google Cloud settings can now be configured via the config file. This allows operators to update Prometheus without needing DaemonSet PATCH/UPDATE RBAC permissions.

We have just released version v0.13.0 of our managed collection for Kubernetes and v2.45.3-gmp.9 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.31 or newer. Self-deployed collection users should upgrade their binaries to use v2.45.3-gmp.9.

Changes in managed collection v0.13.0:

• [FEATURE] Vertifical Pod Autoscaling (VPA) can now be enabled via OperatorConfig.
• [ENHANCEMENT] Collectors now skip WAL playback on startup, which can avoid slow startup and crashloop issues.
• [ENHANCEMENT] Webhooks are now optional. In this case, the operator now performs validation and updates the status subresources in the reconcile loop.
• [ENHANCEMENT] Add status field to Rules, ClusterRules, and GlobalRules to inform whether a successful configuration was generated or not.
• [ENHANCEMENT] ClusterNodeMonitoring now supports insecure-skip-verify, achieving full feature parity with the existing Kubelet scraping configuration.
• [ENHANCEMENT] The collector and rule-evaluator can now start up without previously flags if there is no configuration to parse.
• [SECURITY] Use readOnlyRootFilesystem where possible to mitigate attack vectors.
• [SECURITY] Disable automountServiceAccountToken for the alertmanager, which does not use the K8s API.
• [BUGFIX] Fix ARM64 builds by using upstream go bas image.
• [BUGFIX] Allow datasource-syncer manifest to run on ARM64 nodes.
• [BUGFIX] Use the shard count value instead of batch size when handling sample export.

We have just released version v0.12.1 of our managed collection for Kubernetes and v2.45.3-gmp.7 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.29 or newer. Self-deployed collection users should upgrade their binaries to use v2.45.3-gmp.7.

Changes in managed collection v0.12.1:

• [ENHANCEMENT] Use Docker image tags in manifests.
• [BUGFIX] Use proper regex for ClusterNodeMonitoring example to collect kubelet and cadvisor metrics.

We have just released version v0.13.1 of our managed collection for Kubernetes.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.30 or newer.

Changes in managed collection v0.13.1:

• [BUGFIX] Fix an issue where target status reporting errors when parsing ClusterNodeMonitoring configs for cadvisor and kubelet scraping.

We have just released version v0.12.0 of our managed collection for Kubernetes and v2.45.3-gmp.7-rc.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.28 or newer. Self-deployed collection users should upgrade their binaries to use v2.45.3-gmp.7-rc.0.

Changes in managed collection v0.12.0:

• [ENHANCEMENT] Added support for specifying a remote write URL.
• [ENHANCEMENT] Enabled Boringcrypto.
• [ENHANCEMENT] Added support for compression for Rules resources.
• [ENHANCEMENT] Added the ability to reference BasicAuth passwords used to authenticate against scrape endpoints.
• [ENHANCEMENT] Added the ability to reference Authentication credentials used to authenticate against scrape endpoints.
• [ENHANCEMENT] Added the ability to reference OAuth client secrets used to authenticate against scrape endpoints.
• [ENHANCEMENT] Added the ability to reference TLS certificates used to authenticate against scrape endpoints.
• [ENHANCEMENT] The operator now supports healthz and readyz endpoints.
• [BUGFIX] Config reloader is more resiliant against errors.

We have just released version v0.10.0 of our managed collection for Kubernetes and prometheus:v2.43.1-gmp.0-gke.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.27 or newer. Self-deployed collection users should upgrade their binaries to use gke.gcr.io/prometheus-engine/prometheus:v2.43.1-gmp.0-gke.0.

Changes in managed collection v0.10.0:

• [ENHANCEMENT] Introduced the datasource-syncer as a preferred way to authorize Grafana for querying, replacing the frontend binary.
  • [ENHANCEMENT] Added mTLS parameters for datasource-syncer.
• [ENHANCEMENT] Added mTLS settings (excluding certificates) for scrape configurations.
• [ENHANCEMENT] Removed some unused RBAC permissions.
• [ENHANCEMENT] Managed Rule-evaluator and Alertmanager now scale down when no rules or alerts are configured.
• [ENHANCEMENT] Added BasicAuth Username setting for scrape endpoints (password coming soon)
• [ENHANCEMENT] Added Authorization Header Scheme setting for scrape endpoints (credentials coming soon)
• [ENHANCEMENT] Added OAuth 2 settings (excluding client secrets) for scrape endpoints
• [ENHANCEMENT] Add support for web.external-url for managed alertmanager
• [SECURITY] Reduce RBAC permissions of components

We have just released version v0.8.2 of our managed collection for Kubernetes and prometheus:v2.41.0-gmp.9-gke.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation.

Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.25 or newer.

Self-deployed collection users should upgrade their binaries to use gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.9-gke.0.

Changes in managed collection v0.8.2:

• [SECURITY] Build images using go 1.20.12 to mitigate CVE-2023-45285 and CVE-2023-39326 vulnerabilities.

We have just released version v0.8.0 of our managed collection for Kubernetes and prometheus:v2.41.0-gmp.7-gke.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation.

Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.25 or newer.

Self-deployed collection users should upgrade their binaries to use gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.7-gke.0.

Changes in managed collection v0.8.0:

• [FEATURE] Add gzip-compression option to generated Prometheus config
• [ENHANCEMENT] Add rule_evaluator_query_requests_total and rule_evaluator_query_requests_latency_seconds metrics to rule-evaluator to measure query performance.
• [ENHANCEMENT] Added BasicAuth, Authorization header, OAuth 2 and mTLS support into the example app.

Changes in self-deployed collection v2.35.0-gmp.7-gke.0

• [ENHANCEMENT] Patched various security vulnerabilities through dependency upgrades.

We have just released version v0.7.4 of our managed collection for Kubernetes and v2.41.0-gmp.4-gke.1 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation.

Users who deploy the service using gcloud or the GKE UI are being upgraded on clusters running version 1.27 or newer. Older minor versions will be backported in the coming weeks

Self-deployed collection users should upgrade their binaries to use gke.gcr.io/prometheus-engine/prometheus:v2.41.0-gmp.4-gke.1.

Changes in managed collection v0.7.4:

• [ENHANCEMENT] Remove two high-cardinality, barely-queried metrics from the cAdvisor package (container_blkio_device_usage_total and container_memory_failures_total)
• [ENHANCEMENT] Log an error when unable to fetch Google Cloud metadata
• [ENHANCEMENT] frontend: Updated UI to Prometheus v2.45.0; fixed scripts & added docs on how to dev UI
• [ENHANCEMENT] export: Exposed internal shard + buffer sizing flags as exorted options.
• [ENHANCEMENT] Use JSON logger for all components
• [ENHANCEMENT] Patch various security vulnerabilities through go 1.20 patch upgrade and Docker buildbase upgrade to Debian 12.
• [ENHANCEMENT] Set default interval for ScrapeEndpoint
• [BUGFIX] Add syntax validation for GlobalRules
• [BUGFIX] Fix cluster label auto-populate in GCE

We have just released version v0.7.0 of our managed collection for Kubernetes and v2.35.0-gmp.5-gke.0 of self-deployed collection.

Users who deploy managed collection using kubectl should reapply the manifests in the documentation. Users who deploy the service using gcloud or the GKE UI are already upgraded on clusters running version 1.25 or newer. Self-deployed collection users should upgrade their binaries to use gke.gcr.io/prometheus-engine/prometheus:v2.35.0-gmp.5-gke.0.

Changes in managed collection v0.7.0:

• [FEATURE] PodMonitoring and ClusterPodMonitoring now can be configured to display statistics and samples of active targets in its status field, which is useful to determine whether scrape failures might be caused by malformed scrape configs that match no targets. See the documentation for how to use the target status report.
• [FEATURE] Support for ingestion and querying of Prometheus exemplars, which allow you to correlate histogram metrics (such as a distribution of latencies) with traces. See the documentation for how to use exemplars.
• [ENHANCEMENT] Support for multi-arch builds (both AMD64 and ARM64 node architectures are now supported).
• [ENHANCEMENT] Remedy GKE security posture alerts so all containers run as non-root.
• [BUGFIX] Handle exporters that write histogram series out-of-order.
• [ENHANCEMENT] Deprecated v1alpha1 CRDs.
• [BUGFIX] Fix a bug to tolerate if the managed alert manager is deployed with a pod IP address that is not part of RFC 6890 with a default route.
• [ENHANCEMENT] Re-format user agent to auto-discover operating environment.
• [BUGFIX] Fix stand-alone rule-evaluator init container bug
• [BUGFIX] Fix config-reloader bug to mitigate Prometheus collectors transient crash looping on startup
• [ENHANCEMENT] Clean-up and refactor some collector “self-metrics”.
• [BUGFIX] Allow Prometheus to default to “replace” when RelabelingRule.Action is unspecified, as per the docs.
• [ENHANCEMENT] Added safeguard checks to our webhooks to reject empty secret names
• [ENHANCEMENT] Remove CPU limits to prevent underutilizing CPU cycles

Changes in self-deployed collection v2.35.0-gmp.5-gke.0:

• [FEATURE] Support for ingestion and querying of Prometheus exemplars, which allow you to correlate histogram metrics (such as a distribution of latencies) with traces. See the documentation for how to use exemplars.
• [ENHANCEMENT] Support for multi-arch builds (AMD64 and ARM64 node architectures are now supported).