Skip to main content

Exporting Telemetry

Connecting to Section’s /federate Endpoint

Section provides a /federate endpoint from which you can gather live metrics pertaining to your account. The systems underlying this endpoint are Prometheus systems. But the metrics returned are OpenMetrics compatible, and therefore you can use any compatible tooling in order to process them.

In the examples below we will give details that help you process the metrics with Grafana Cloud and Datadog.

To setup remote collection for any of the following 3rd party systems you will need to collect the following information from Section Console:

  • SECTION_ACCOUNT_ID is 4 or more digits, as in “1234”. Learn about accounts here.
  • URL: console.section.io/prometheus/account/SECTION_ACCOUNT_ID/federate
  • Authentication type: Bearer
  • SECTION_API_TOKEN: get an API token using instructions here.
  • Learn more about Prometheus and OpenMetrics at this primer.

Monitoring with Grafana Cloud

Here we provide a simple example of how to scrape Section metrics into Grafana Cloud. The basic idea is that you need to run a Prometheus agent that regularly scrapes Section’s /federate endpoint to fetch metrics, and writes the results into Grafana Cloud using “remote write”. (At the time of this writing Grafana Cloud has no way to do the scraping itself.)

Grafana Cloud

You may run the Prometheus agent on any infrastructure of your choice. It could be in a docker container on your PC, hosted on a cloud provider, or even hosted at Section. A single agent can be used to scrape all of the metrics for your account.

Obtain the following information from your instance of Grafana Cloud:

  • GRAFANA_METRICS_INSTANCE_ID: obtain from Grafana Cloud
  • GRAFANA_API_KEY: obtain from Grafana Cloud
  • GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT:
    • Is of the form: https://…/api/prom/push

Example YAML Configuration for a Prometheus Docker Container

The following YAML file configures a docker container that will run Prometheus, scrape Section’s /federate endpoint, and write the results into Grafana Cloud. Replace the following accordingly: SECTION_ACCOUNT_ID, SECTION_API_TOKEN, GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT, GRAFANA_METRICS_INSTANCE_ID, GRAFANA_API_KEY.

prometheus.yaml
# my global config
global:
scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global default (10s).

# Attach these labels to any time series or alerts when communicating with
# external systems (federation, remote storage, Alertmanager).
external_labels:
monitor: 'section-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first.rules"
# - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'section-federation'
metrics_path: '/prometheus/account/SECTION_ACCOUNT_ID/federate'
params:
'match[]':
- '{__name__=~".+"}'
scheme: 'https'
authorization:
type: Bearer
credentials: SECTION_API_TOKEN
static_configs:
- targets: ['console.section.io']
remote_write:
- url: GRAFANA_METRICS_INSTANCE_REMOTE_WRITE_ENDPOINT
basic_auth:
username: GRAFANA_METRICS_INSTANCE_ID
Password: GRAFANA_API_KEY

The command to run the container is as follows. Replace C:/myfiles accordingly.

docker run \
-p 9090:9090 \
-v C:/myfiles/prometheus.yml:/etc/prometheus/prometheus.yml \
prom/prometheus

Monitoring with Datadog

Here we provide a simple example of how to scrape Section metrics into Datadog. The basic idea is that you need to run a Datadog Agent that regularly scrapes Section’s /federate endpoint to fetch metrics, and writes the results into your Datadog cloud instance. You may run this agent on any infrastructure of your choice. It could be in a docker container on your PC, hosted on a cloud provider, or even hosted at Section. A single agent can be used to scrape all of the metrics for your account.

Datadog

Obtain the following information from your instance of Datadog:

  • DATADOG_API_KEY: this will appear during the Datadog setup wizard. Or generate one by visiting the API Keys area of Organization Settings.
  • Get started with Datadog.
  • Specifically, you are using the Datadog Agent, so you will want to learn about that.

Example YAML Configuration for a Datadog Agent Docker Container

The following YAML file defines a docker container that will run the Datadog agent, scrape Section’s /federate endpoint, and write the results into your Datadog cloud instance.

  • Replace SECTION_ACCOUNT_ID with your Section Account ID, typically of the form 1234.
  • Note that the Bearer TOKEN is not something that you are supposed to replace, just leave it alone.
ddagent.yaml
init_config:
instances:
- namespace: section
openmetrics_endpoint: https://console.section.io/prometheus/account/SECTION_ACCOUNT_ID/federate
auth_token:
reader:
type: file
path: /etc/datadog-agent/conf.d/openmetrics.d/token
pattern: ^(.+)$
writer:
type: header
name: Authorization
value: "Bearer <TOKEN>"
metrics:
- .+:
type: gauge

The token file should be filled with one line containing your SECTION_API_TOKEN. Shown below is an obfuscated Section API token.

*********************************************************************4c57

The command to run the container above follows. Note the following:

  • You'll need to properly specify the location of the files datadog/openmetrics.d/conf.yaml and datadog/openmetrics.d/token. In the example below they are specified as being in C:/myfiles.
  • And be sure to replace DATADOG_API_KEY.
docker run -d --name dd-agent \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-v C:/myfiles/datadog/openmetrics.d/conf.yaml:/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml \
-v C:/myfiles/datadog/openmetrics.d/token:/etc/datadog-agent/conf.d/openmetrics.d/token \
-e DD_API_KEY=DATADOG_API_KEY \
-e DD_SITE="datadoghq.com" \
gcr.io/datadoghq/agent:7

If you are having trouble, try adding debugging to the list of command line arguments.

    -e DD_LOG_LEVEL=debug \