Exporting to Datadog using CloudFlow

Monitoring with Datadog

Here we provide a simple example of how to scrape CloudFlow metrics into Datadog by using a deployment of the Datadog Agent on CloudFlow itself as a project separate from your production workload project. The basic idea is that you need to run a Datadog Agent that regularly scrapes CloudFlow’s /federate endpoint to fetch metrics for your entire account, and writes the results into your Datadog cloud instance. We have another guide that shows how to do this with docker run if you prefer. But here we show how to make a deployment to CloudFlow that runs 24x7 in a single location. A single Datadog agent collects metrics for all projects in your account.

Obtain the following information from your instance of Datadog:

• DATADOG_API_KEY: this will appear during the Datadog setup wizard. Or generate one by visiting the API Keys area of Organization Settings.

Useful Links

• Get started with Datadog.
• Specifically, you are using the Datadog Agent, so you will want to learn about that.

Deployment

The following deployment will run the agent on CloudFlow. Substitute DATADOG_API_KEY accordingly.

datadog-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ddagent
  name: ddagent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ddagent
  template:
    metadata:
      labels:
        app: ddagent
    spec:
      containers:
      - image: gcr.io/datadoghq/agent:7
        imagePullPolicy: Always
        name: ddagent
        env:
          - name: DD_API_KEY
            value: DATADOG_API_KEY
          - name: DD_SITE
            value: "datadoghq.com"
          - name: DD_HOSTNAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        volumeMounts:
          - name: ddagent-config
            mountPath: /etc/datadog-agent/conf.d/openmetrics.d
        resources:
          requests:
            memory: "250Mi"
            cpu: "250m"
          limits:
            memory: "250Mi"
            cpu: "250m"
      volumes:
        - name: ddagent-config
          configMap:
            name: ddagent-config

Deploy it with kubectl apply -f datadog-deployment.yaml.

Configuration

The following YAML file defines a ConfigMap with configuration for the Datadog agent. It identifies the CloudFlow /federate endpoint for your account, names the metrics to collect, and identifies the CLOUDFLOW_API_TOKEN.

• Replace CLOUDFLOW_ACCOUNT_ID with your CloudFlow Account ID, typically of the form 1234.
• The token section should be filled with one line containing your CLOUDFLOW_API_TOKEN. It will be of the form *********************************************************************4c57.
• Learn how to obtain the SECTION items here.
• Note that the Bearer <TOKEN> is not something that you should replace, just leave it alone.

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: ddagent-config
data:
  conf.yaml: |
    init_config:
    instances:
      - namespace: cloudflow
        openmetrics_endpoint: https://console.section.io/prometheus/account/CLOUDFLOW_ACCOUNT_ID/federate
        auth_token:
          reader:
            type: file
            path: /etc/datadog-agent/conf.d/openmetrics.d/token
            pattern: ^(.+)$
          writer:
            type: header
            name: Authorization
            value: "Bearer <TOKEN>"
        metrics:
          - section.+:
              type: gauge
  token: |
    CLOUDFLOW_API_TOKEN

Deploy it with kubectl apply -f configmap.yaml.

Location Strategy

By default, CloudFlow will run this project in 2 locations. We only need to collect metrics from your account once, so let's provide a location optimizer strategy that runs the project in only a single location. Read more about location strategies.

location-optimizer.yaml

apiVersion: v1
kind: ConfigMap
data:
  strategy: |
    {
        "strategy": "SolverServiceV1",
        "params": {
            "policy": "dynamic",
            "minimumLocations": 1,
            "maximumLocations": 1
        }
    }
metadata:
  name: location-optimizer

Deploy it with kubectl apply -f location-optimizer.yaml.

View Metrics in Datadog

Login to your Datadog account in order to see your metrics. Try our sample dashboard to get you started.