Exporting Individual Pod Telemetry for Custom Metrics

Monitoring individual pods

While CloudFlow will provide general metrics on things like cpu and memory usage, you may have a use case where your application generates custom metrics that you want to export.

One solution for this would be to run have a process run in each container which remotely writes those metrics to some endpoint. In this article we will describe how to run a Prometheus Agent which can scrape metrics from all your pods and remote write them to destinations of your choosing. In this configuration you will run a prometheus agent in the same project as the pods you want to monitor, and an agent will end up running in each of the locations that your application runs. This means that if you inspect individual Prometheus Agents you will only see them succeed in scraping the pods that are local to them, but they will all ultimately write results to your remote_write configured endpoints.

Configuration

The following YAML file defines a ConfigMap with configuration for the Prometheus agent. Replace the following accordingly: CLOUDFLOW_KUBERNETES_URL, CLOUDFLOW_API_TOKEN. You will also need to fill in the remote_write portion of the config for the destinations you want, or remove it. Fine more information configuring remote_write here. Learn how to obtain the CLOUDFLOW items here.

"config

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheusagent-config
data:
  prometheus.yml: |
    # my global config
    global:
      scrape_interval:     30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).

      # Attach these labels to any time series or alerts when communicating with
      # external systems (federation, remote storage, Alertmanager).
      external_labels:
          monitor: 'cloudflow-monitor'

    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first.rules"
      # - "second.rules"

    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'mykubernetespods'
        metrics_path: '/metrics'
        params:
          'match[]':
            - '{__name__=~".+"}'
        scheme: 'http'
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_name]
          separator: ;
          regex: (.*)
          target_label: pod
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_container_image]
          separator: ;
          regex: (.*)
          target_label: container_image
          replacement: $1
          action: replace
        - source_labels: [__meta_kubernetes_pod_container_name]
          separator: ;
          regex: (.*)
          target_label: container_name
          replacement: $1
          action: replace
        kubernetes_sd_configs:
        - role: 'pod'
          api_server: CLOUDFLOW_KUBERNETES_URL
          authorization:
            credentials: CLOUDFLOW_API_TOKEN
    remote_write:
    - url: <FILL IN REMOTE WRITE URL>
      authorization:
        credentials: <FILL IN CREDENTIAL>

Deploy it with kubectl apply -f configmap.yaml.

Deployment

The following deployment will run the Prometheus agent on CloudFlow.

prometheus-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheusagent
  name: prometheusagent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheusagent
  template:
    metadata:
      labels:
        app: prometheusagent
    spec:
      containers:
      - image: prom/prometheus
        imagePullPolicy: Always
        name: prometheusagent
        ports:
        - containerPort: 9090
        volumeMounts:
          - name: prometheusagent-config
            mountPath: /etc/prometheus
        resources:
          requests:
            memory: ".5Gi"
            cpu: "500m"
          limits:
            memory: ".5Gi"
            cpu: "500m"
      volumes:
        - name: prometheusagent-config
          configMap:
            name: prometheusagent-config

Deploy it with kubectl apply -f grafana-deployment.yaml.

Mark pods for scraping

With the above setup, you can now add the label prometheus.io/scrape: "true" to the pods in your deployments and their metrics will be scraped from /metrics. You can learn more about customing the agent configuration here

View Metrics

You can now view metrics in the remote systems you have configured, or you can try and test things are working by running kubectl port-forward <prometheus agent pod name> 9090 and view data at localhost:9090. Not that as described earlier, you will only see the local data with this test method, not the data from ever location your app is deployed.