Exporting Telemetry

Connecting to Section’s /federate Endpoint

Section provides a /federate endpoint from which you can gather live metrics pertaining to your account. The systems underlying this endpoint are Prometheus systems. But the metrics returned are OpenMetrics compatible, and therefore you can use any compatible tooling in order to process them.

In the examples below we will give details that help you process the metrics with Grafana Cloud and Datadog.

To setup remote collection for any of the following 3rd party systems you will need to collect the following information from Section Console:

• SECTION_ACCOUNT_ID is 4 or more digits, as in “1234”. Learn about accounts here.
• URL: console.section.io/prometheus/account/SECTION_ACCOUNT_ID/federate
• Authentication type: Bearer
• SECTION_API_TOKEN: get an API token using instructions here.

Useful Links

• Learn more about Prometheus and OpenMetrics at this primer.

Monitoring with Grafana Cloud

Here we provide a simple example of how to scrape Section metrics into Grafana Cloud. The basic idea is that you need to run a Prometheus agent that regularly scrapes Section’s /federate endpoint to fetch metrics, and writes the results into Grafana Cloud using “remote write”. (At the time of this writing Grafana Cloud has no way to do the scraping itself.)

You may run the Prometheus agent on any infrastructure of your choice. It could be in a docker container on your PC, hosted on a cloud provider, or even hosted at Section. A single agent can be used to scrape all of the metrics for your account.

Obtain the following information from your instance of Grafana Cloud:

• GRAFANA_METRICS_INSTANCE_ID: obtain from Grafana Cloud
• GRAFANA_API_KEY: obtain from Grafana Cloud
• GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT:
  • Is of the form: https://…/api/prom/push

Useful Links

• Get started with Grafana Cloud
• Learn about how to setup your Prometheus Agent

Example YAML Configuration for a Prometheus Docker Container

The following YAML file configures a docker container that will run Prometheus, scrape Section’s /federate endpoint, and write the results into Grafana Cloud. Replace the following accordingly: SECTION_ACCOUNT_ID, SECTION_API_TOKEN, GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT, GRAFANA_METRICS_INSTANCE_ID, GRAFANA_API_KEY.

prometheus.yaml

# my global config
global:
  scrape_interval:     30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'section-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'section-federation'
    metrics_path: '/prometheus/account/SECTION_ACCOUNT_ID/federate'
    params:
      'match[]':
        - '{__name__=~".+"}'
    scheme: 'https'
    authorization:
      type: Bearer
      credentials: SECTION_API_TOKEN
    static_configs:
      - targets: ['console.section.io']
remote_write:
- url: GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT
  basic_auth:
    username: GRAFANA_METRICS_INSTANCE_ID
    Password: GRAFANA_API_KEY

The command to run the container is as follows. Replace C:/myfiles accordingly.

docker run \
    -p 9090:9090 \
    -v C:/myfiles/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

Monitoring with Datadog

Here we provide a simple example of how to scrape Section metrics into Datadog. The basic idea is that you need to run a Datadog Agent that regularly scrapes Section’s /federate endpoint to fetch metrics, and writes the results into your Datadog cloud instance. You may run this agent on any infrastructure of your choice. It could be in a docker container on your PC, hosted on a cloud provider, or even hosted at Section. A single agent can be used to scrape all of the metrics for your account.

Obtain the following information from your instance of Datadog:

• DATADOG_API_KEY: this will appear during the Datadog setup wizard. Or generate one by visiting the API Keys area of Organization Settings.

Useful Links

• Get started with Datadog.
• Specifically, you are using the Datadog Agent, so you will want to learn about that.

Example YAML Configuration for a Datadog Agent Docker Container

The following YAML file defines a docker container that will run the Datadog agent, scrape Section’s /federate endpoint, and write the results into your Datadog cloud instance.

• Replace SECTION_ACCOUNT_ID with your Section Account ID, typically of the form 1234.
• Note that the Bearer TOKEN is not something that you are supposed to replace, just leave it alone.

ddagent.yaml

init_config:
instances:
  - namespace: section
    openmetrics_endpoint: https://console.section.io/prometheus/account/SECTION_ACCOUNT_ID/federate
    auth_token:
      reader:
        type: file
        path: /etc/datadog-agent/conf.d/openmetrics.d/token
        pattern: ^(.+)$
      writer:
        type: header
        name: Authorization
        value: "Bearer <TOKEN>"
    metrics:
      - .+:
          type: gauge

The token file should be filled with one line containing your SECTION_API_TOKEN. Shown below is an obfuscated Section API token.

*********************************************************************4c57

The command to run the container above follows. Note the following:

• You'll need to properly specify the location of the files datadog/openmetrics.d/conf.yaml and datadog/openmetrics.d/token. In the example below they are specified as being in C:/myfiles.
• And be sure to replace DATADOG_API_KEY.

docker run -d --name dd-agent \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -v C:/myfiles/datadog/openmetrics.d/conf.yaml:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml \
    -v C:/myfiles/datadog/openmetrics.d/token:/etc/datadog-agent/conf.d/openmetrics.d/token \
    -e DD_API_KEY=DATADOG_API_KEY \
    -e DD_SITE="datadoghq.com" \
    gcr.io/datadoghq/agent:7

If you are having trouble, try adding debugging to the list of command line arguments.

    -e DD_LOG_LEVEL=debug \