Skip to main content

Exporting to Grafana Cloud using CloudFlow

Monitoring with Grafana Cloud

Here we provide a simple example of how to scrape CloudFlow metrics into Grafana Cloud by using a deployment of the Prometheus Agent on CloudFlow itself as a project separate from your production workload project. The basic idea is that you need to run a Prometheus Agent that regularly scrapes CloudFlow’s /federate endpoint to fetch metrics for your entire account, and writes the results into Grafana Cloud using “remote write”. (At the time of this writing Grafana Cloud has no way to do the scraping itself.) We have another guide that shows how to do this with docker run if you prefer. But here we show how to make a deployment to CloudFlow that runs 24x7 in a single location. A single Prometheus agent collects metrics for all projects in your account.

Grafana

Obtain the following information from your instance of Grafana Cloud:

  • GRAFANA_METRICS_INSTANCE_ID: obtain from Grafana Cloud
  • GRAFANA_API_KEY: obtain from Grafana Cloud
  • GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT:
    • Is of the form: https://…/api/prom/push

Deployment

The following deployment will run the Prometheus agent on CloudFlow.

grafana-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: grafanaagent
name: grafanaagent
spec:
replicas: 1
selector:
matchLabels:
app: grafanaagent
template:
metadata:
labels:
app: grafanaagent
spec:
containers:
- image: prom/prometheus
imagePullPolicy: Always
name: grafanaagent
volumeMounts:
- name: grafanaagent-config
mountPath: /etc/prometheus
resources:
requests:
memory: ".5Gi"
cpu: "500m"
limits:
memory: ".5Gi"
cpu: "500m"
volumes:
- name: grafanaagent-config
configMap:
name: grafanaagent-config

Deploy it with kubectl apply -f grafana-deployment.yaml.

Configuration

The following YAML file defines a ConfigMap with configuration for the Prometheus agent. Replace the following accordingly: CLOUDFLOW_ACCOUNT_ID, CLOUDFLOW_API_TOKEN, GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT, GRAFANA_METRICS_INSTANCE_ID, GRAFANA_API_KEY. Learn how to obtain the SECTION items here.

configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: grafanaagent-config
data:
prometheus.yml: |
# my global config
global:
scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'cloudflow-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first.rules"
# - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'cloudflow-federation'
metrics_path: '/prometheus/account/CLOUDFLOW_ACCOUNT_ID/federate'
params:
'match[]':
- '{__name__=~".+"}'
scheme: 'https'
authorization:
type: Bearer
credentials: CLOUDFLOW_API_TOKEN
static_configs:
- targets: ['console.section.io']
remote_write:
- url: GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT
basic_auth:
username: GRAFANA_METRICS_INSTANCE_ID
password: GRAFANA_API_KEY

Deploy it with kubectl apply -f configmap.yaml.

Location Strategy

By default, CloudFlow will run this project in 2 locations. We only need to collect metrics from your account once, so let's provide a location optimizer strategy that runs the project in only a single location. Read more about location strategies.

location-optimizer.yaml
apiVersion: v1
kind: ConfigMap
data:
strategy: |
{
"strategy": "SolverServiceV1",
"params": {
"policy": "dynamic",
"minimumLocations": 1,
"maximumLocations": 1
}
}
metadata:
name: location-optimizer

Deploy it with kubectl apply -f location-optimizer.yaml.

View Metrics in Grafana

Login to your Grafana Cloud account in order to see your metrics. Try our sample dashboard to get you started.