Skip to main content
Classic

Manage Log Streaming

Learn how to enable and configure Log Streaming in your Section application.

Introduction

Log streaming on Section is a standard Unified Logging Layer using a managed Cloud Native Foundation FluentD infrastructure.

Section supports, among others, the following FluentD destination endpoints:

  • AWS S3
  • Datadog
  • Elastic Cloud
  • Elastic Search
  • Google Cloud
  • Grafana Cloud Loki
  • Logtail
  • New Relic
  • Rsyslog
  • Splunk
  • Sumo Logic

Log streaming can be enabled simply by adding a single configuration file to an applications git repository.

High Level Overview

Here’s an overview of how to initiate log streaming:

  1. Review existing documentation of your Logging solution such as Datadog or Splunk and get the required API key or logging solution credentials.
  2. Clone your Section Application git repository to your local computer.
  3. Add a new file to the root of your repository named fluent-match.conf.
  4. Paste the example Fluentd match configuration from below into the new file.
  5. Replace the text INSERT_YOUR_LOGGING_PROVIDER_KEY_HERE in the file with your Provider API key.
  6. Save the file.
  7. Commit and push the changes.
  8. Logs will begin streaming within 30-60 seconds.

See below - Enable Log Streaming - for provider specific examples of the configuration file (fluent-match.conf).

note

Your Section Application logs should begin to appear in your Log Streaming destination in a few minutes.

Enable Log Streaming to supported Log Streaming destinations

Follow the steps above to add a fluent-match.conf to your Section Application git repository, but use the example contents below for your respective destination.

note

The fluent-match.conf file cannot contain the @include directive, and must be smaller than 1 Megabyte.

AWS S3

Ensure your S3 bucket already exists, and create a AWS IAM user with an API access key that is at least has the s3:PutObject action on the bucket contents. An example IAM policy is available here

This example will upload a log file per PoP every 5 minutes, each log file will be named with a timestamp followed by a unique identifier.

<match **>
@type s3

aws_key_id INSERT_YOUR_AWS_API_KEY_ID_HERE
aws_sec_key INSERT_YOUR_AWS_API_SECRET_KEY_HERE
check_bucket false
check_object false
path logs/ # adjust as required
s3_bucket INSERT_YOUR_AWS_S3_BUCKET_NAME_HERE
s3_region INSERT_YOUR_AWS_S3_BUCKET_REGION_HERE
# Default s3_object_key_format is "%{path}%{time_slice}_%{index}.%{file_extension}"
s3_object_key_format %{path}%{time_slice}_%{uuid_flush}.%{file_extension}
store_as gzip # or "json" for uncompressed logs
time_slice_format %Y%m%d%H%M

<buffer>
@type file
path /var/log/fluent/s3
timekey 300 # 5 minute partition
timekey_wait 30s
timekey_use_utc true
chunk_limit_size 256m
</buffer>
</match>

Datadog

Follow the steps below to enable Log Streaming to Datadog:

  1. Follow the Datadog documentation to create a Datadog API key to use with your Section application.
  2. Clone your Section Application git repository to your local computer.
  3. Add a new file to the root of your repository named fluent-match.conf.
  4. Paste the example Fluentd match configuration from below into the new file.
  5. Replace the text INSERT_YOUR_DATADOG_API_KEY_HERE in the file with your Datadog API key created above.
  6. Save the file.
  7. Add the fluent-match.conf file to your git repository, commit the change, and push the commits to Section.

Example Datadog configuration for Fluentd:

<match **>
# from https://docs.datadoghq.com/integrations/fluentd/?tab=host#log-collection
@type datadog
@id awesome_agent
api_key "INSERT_YOUR_DATADOG_API_KEY_HERE"

<buffer>
@type memory
flush_thread_count 4
flush_interval 3s
chunk_limit_size 5m
chunk_limit_records 500
</buffer>
</match>

Grafana Cloud Loki

The Grafana Cloud URL and credentials are available in the Loki Stack Details page. Your Grafana Cloud API Key should have the MetricsPublisher role.

Note: use either <label>...</label> or extra_labels to set at least one label. (Docs)

<match **>
@type loki
url "INSERT_YOUR_GRAFANA_CLOUD_LOKI_URL_HERE"
username "INSERT_YOUR_GRAFANA_CLOUD_LOKI_USERNAME_HERE"
password "INSERT_YOUR_GRAFANA_CLOUD_LOKI_PASSWORD_HERE"
extra_labels {"env":"dev"}
flush_interval 10s
flush_at_shutdown true
buffer_chunk_limit 1m
</match>

Google Cloud

Google Cloud requires a separate credentials file to be added to your repository. Add the fluent-match.conf as follows to send all your logs to Google Cloud.

<match **>
# From https://cloud.google.com/logging/docs/agent/logging/configuration
@type google_cloud
use_metadata_service false
project_id GOOGLE_CLOUD_PROJECT_ID
vm_id none
zone none
</match>

Then add a second file next to that called fluent-google-cloud.json. See the instructions under Creating a service account on Google's docs for instructions on how to generate this file. Download it from the console and add it to the root of your Section repository as fluent-google-cloud.json.

Logtail

Make sure you replace YOUR_LOGTAIL_SOURCE_TOKEN with your own source token from Logtail.com.

<match *>
@type logtail
@id output_logtail
source_token YOUR_LOGTAIL_SOURCE_TOKEN
flush_interval 2 # in seconds
</match>

New Relic

Replace YOUR_LICENSE_KEY below with your New Relic license key. For more details, see New Relic's Fluentd log forwarding documentation.

<match **>
@type newrelic
license_key YOUR_LICENSE_KEY
</match>

Sumo Logic

Your SUMOLOGIC_COLLECTOR_URL can be found by going to your collection under App Catalog then Collection and clicking on the Show URL link on the collection.

<match **>
# From https://github.com/SumoLogic/fluentd-output-sumologic
@type sumologic
endpoint SUMOLOGIC_COLLECTOR_URL
log_format json
source_category SUMOLOGIC_SOURCE_CATEGORY
source_name SUMOLOGIC_SOURCE_NAME
open_timeout 10
</match>

Disable Log Streaming

Delete the fluent-match.conf file from the Section Application git repository to disable Log Streaming.