Skip to main content
Git Interface

Manage Log Streaming

Learn how to enable and configure Log Streaming in your CloudFlow project.

Introduction​

Log streaming on CloudFlow is a standard Unified Logging Layer using a managed Cloud Native Foundation Fluentd infrastructure.

CloudFlow supports, among others, the following Fluentd destination endpoints:

  • AWS S3
  • Datadog
  • Elastic Cloud
  • Elastic Search
  • Google Cloud
  • Grafana Cloud Loki
  • Logtail
  • New Relic
  • Rsyslog
  • Splunk
  • Sumo Logic
  • Azure Log Analytics

Log streaming can be enabled simply by adding a single configuration file to a project's git repository.

High Level Overview​

Here’s an overview of how to initiate log streaming:

  1. Review existing documentation of your Logging solution such as Datadog or Splunk and get the required API key or logging solution credentials.
  2. Clone your CloudFlow Project git repository to your local computer.
  3. Add a new file to the root of your repository named fluent-match.conf.
  4. Paste the example Fluentd match configuration from below into the new file.
  5. Replace the text INSERT_YOUR_LOGGING_PROVIDER_KEY_HERE in the file with your Provider API key.
  6. Save the file.
  7. Commit and push the changes.
  8. Logs will begin streaming within 30-60 seconds.

See below - Enable Log Streaming - for provider specific examples of the configuration file (fluent-match.conf).

note

Your CloudFlow Project logs should begin to appear in your Log Streaming destination in a few minutes.

Enable Log Streaming to supported Log Streaming destinations​

Follow the steps above to add a fluent-match.conf to your CloudFlow Project git repository, but use the example contents below for your respective destination.

note

The fluent-match.conf file cannot contain the @include directive, and must be smaller than 1 Megabyte.

AWS S3​

Ensure your S3 bucket already exists, and create a AWS IAM user with an API access key that is at least has the s3:PutObject action on the bucket contents. An example IAM policy is available here

This example will upload a log file per PoP every 5 minutes, each log file will be named with a timestamp followed by a unique identifier.

<match **>
  @type s3

  aws_key_id INSERT_YOUR_AWS_API_KEY_ID_HERE
  aws_sec_key INSERT_YOUR_AWS_API_SECRET_KEY_HERE
  check_bucket false
  check_object false
  path logs/ # adjust as required
  s3_bucket INSERT_YOUR_AWS_S3_BUCKET_NAME_HERE
  s3_region INSERT_YOUR_AWS_S3_BUCKET_REGION_HERE
  # Default s3_object_key_format is "%{path}%{time_slice}_%{index}.%{file_extension}"
  s3_object_key_format %{path}%{time_slice}_%{uuid_flush}.%{file_extension}
  store_as gzip # or "json" for uncompressed logs
  time_slice_format %Y%m%d%H%M

  <buffer>
    @type file
    path /var/log/fluent/s3
    timekey 300 # 5 minute partition
    timekey_wait 30s
    timekey_use_utc true
    chunk_limit_size 256m
  </buffer>
</match>

Datadog​

Follow the steps below to enable Log Streaming to Datadog:

  1. Follow the Datadog documentation to create a Datadog API key to use with your CloudFlow project.
  2. Clone your CloudFlow Project git repository to your local computer.
  3. Add a new file to the root of your repository named fluent-match.conf.
  4. Paste the example Fluentd match configuration from below into the new file.
  5. Replace the text INSERT_YOUR_DATADOG_API_KEY_HERE in the file with your Datadog API key created above.
  6. Save the file.
  7. Add the fluent-match.conf file to your git repository, commit the change, and push the commits to CloudFlow.

Example Datadog configuration for Fluentd:

<match **>
  # from https://docs.datadoghq.com/integrations/fluentd/?tab=host#log-collection
  @type datadog
  @id awesome_agent
  api_key "INSERT_YOUR_DATADOG_API_KEY_HERE"

  <buffer>
    @type memory
    flush_thread_count 4
    flush_interval 3s
    chunk_limit_size 5m
    chunk_limit_records 500
  </buffer>
</match>

Grafana Cloud Loki​

The Grafana Cloud URL and credentials are available in the Loki Stack Details page. Your Grafana Cloud API Key should have the MetricsPublisher role.

Note: use either <label>...</label> or extra_labels to set at least one label. (Docs)

<match **>
  @type loki
  url "INSERT_YOUR_GRAFANA_CLOUD_LOKI_URL_HERE"
  username "INSERT_YOUR_GRAFANA_CLOUD_LOKI_USERNAME_HERE"
  password "INSERT_YOUR_GRAFANA_CLOUD_LOKI_PASSWORD_HERE"
  extra_labels {"env":"dev"}
  flush_interval 10s
  flush_at_shutdown true
  buffer_chunk_limit 1m
</match>

Google Cloud​

Google Cloud requires a separate credentials file to be added to your repository. Add the fluent-match.conf as follows to send all your logs to Google Cloud.

<match **>
  # From https://cloud.google.com/logging/docs/agent/logging/configuration
  @type google_cloud
  use_metadata_service false
  project_id GOOGLE_CLOUD_PROJECT_ID
  vm_id none
  zone none
</match>

Then add a second file next to that called fluent-google-cloud.json. See the instructions under Creating a service account on Google's docs for instructions on how to generate this file. Download it from the console and add it to the root of your CloudFlow repository as fluent-google-cloud.json.

Logtail​

Make sure you replace YOUR_LOGTAIL_SOURCE_TOKEN with your own source token from Logtail.com.

<match *>
  @type logtail
  @id output_logtail
  source_token YOUR_LOGTAIL_SOURCE_TOKEN
  flush_interval 2 # in seconds
</match>

New Relic​

Replace YOUR_LICENSE_KEY below with your New Relic license key. For more details, see New Relic's Fluentd log forwarding documentation.

Note: The New Relic plugin for Fluentd overwrites the message field with the contents of the log field before sending the data to New Relic, therefore the record_transformer filter is required for logs to be shipped appropriately.

<filter **>
  @type record_transformer
  <record>
    offset ${record["log"]["offset"]}
  </record>
  enable_ruby true
  remove_keys log
</filter>

<match **>
  @type newrelic
  license_key YOUR_LICENSE_KEY
</match>

Sumo Logic​

Your SUMOLOGIC_COLLECTOR_URL can be found by going to your collection under App Catalog then Collection and clicking on the Show URL link on the collection.

<match **>
  # From https://github.com/SumoLogic/fluentd-output-sumologic
  @type sumologic
  endpoint SUMOLOGIC_COLLECTOR_URL
  log_format json
  source_category SUMOLOGIC_SOURCE_CATEGORY
  source_name SUMOLOGIC_SOURCE_NAME
  open_timeout 10
</match>

Azure Log Analytics​

<match **>
  # From https://github.com/yokawasa/fluent-plugin-azure-loganalytics
  @type azure-loganalytics
  customer_id CUSTOMER_ID   # Customer ID aka WorkspaceID String
  shared_key KEY_STRING     # The primary or the secondary Connected Sources client authentication key
  log_type EVENT_TYPE_NAME  # The name of the event type. ex) ApacheAccessLog
  endpoint myendpoint
  add_time_field true
  time_field_name mytime
  time_format %s
  localtime true
  add_tag_field true
  tag_field_name mytag
</match>

Disable Log Streaming​

Delete the fluent-match.conf file from the CloudFlow Project git repository to disable Log Streaming.