Log Streaming to External Tools

Stream logs to external tools to get more robust support for log viewing than is possible using kubectl logs.

If your Project includes a Kubernetes Service decorated with the label section.io/logstream-destination: "true" and listening on UDP port 5160, the CloudFlow Platform will collect the container logs for all the Pods in your Project and forward them to this Service's Endpoint(s) in JSON format, similar to this (pretty-printed here):

example-log-entry.json

{
  "@timestamp": "2022-07-01T23:45:56.999Z",
  "kubernetes": {
    "container": {
      "name": "frontend"
    },
    "labels": {
      "app": "contoso",
      "pod-template-hash": "69bc8cd654"
    },
    "namespace": "default",
    "node": {
      "name": "syd-umvnp"
    },
    "pod": {
      "ip": "172.17.0.5",
      "name": "contoso-69bc8cd654-6lhrm"
    }
  },
  "log": {
    "offset": 23456
  },
  "message": "Application has completed initialization",
  "stream": "stdout"
}

Your Service Endpoint Pods can receive these log packets, process them, and forward them to an external log ingestion service for storage and querying. The JSON over UDP/5160 wire format is simple enough if you want to implement your own application to process, however the popular CNCF project Fluentd has a UDP input plugin exactly for this. Here is an example Fluentd source configuration:

example-source-configuration

<source>
  @type udp
  tag all_cloudflow_logs
  <parse>
    @type json
  </parse>
  port 5160
  message_length_limit 1MB # only 4096 bytes by default
</source>

And then you'll need a matching output configuration, which contains specifics for your log destination. The following matches all logs tagged with all_cloudflow_logs from the <source> configuration.

prototype-output-configuration

<match all_cloudflow_logs>
  YOUR_LOG_DESTINATION_CONFIG_HERE
</match>

Popular log ingestion services will have a Fluentd output plugin and configuration for shipping logs to their system. The table below lists a number of examples with detailed CloudFlow integrations where available.

Log Destination	Documentation	Plugin	CloudFlow Log Integration
Datadog	Documentation	Plugin	Log Integration
S3	Documentation
Splunk	Documentation	Plugin
Sumo Logic		Plugin
Grafana Loki	Documentation	Plugin	Log Integration
Dynatrace		Plugin
Elasticsearch		Plugin
New Relic		Plugin	Log Integration
Google Cloud		Plugin
Logtail		Plugin

Here is an example Kubernetes yaml to deploy Fluentd for log shipping where we put it all together:

log-shipping.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: fluentd
    section.io/logstream-destination: "true"
  name: fluentd
  namespace: default
spec:
  ports:
  - name: fluentdudp
    port: 5160
    protocol: UDP
    targetPort: 5160
  selector:
    app: fluentd
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
  namespace: default
  labels:
    app: fluentd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
        section.io/logstream-collect: "false"
    spec:
      containers:
      - name: fluentd
        image: YOUR_FLUENTD_IMAGE_HERE
        imagePullPolicy: Always
        resources:
          requests:
            memory: ".5Gi"
            cpu: "500m"
          limits:
            memory: ".5Gi"
            cpu: "500m"
        volumeMounts:
        - name: config
          mountPath: /fluentd/etc/fluent.conf
          readOnly: true
          subPath: fluent.conf
      volumes:
      - name: config
        configMap:
          name: fluent-conf
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-conf
  namespace: default
data:
  fluent.conf: |-
    <source>
      @type udp
      tag all_cloudflow_logs
      <parse>
        @type json
      </parse>
      port 5160
      message_length_limit 1MB
    </source>

    <match all_cloudflow_logs>
      YOUR_LOG_DESTINATION_CONFIG_HERE
    </match>

Our specific integration guides give exactly the code needed for YOUR_LOG_DESTINATION_CONFIG_HERE and YOUR_FLUENTD_IMAGE_HERE to make configuration easy.

Differences from Metric Collection

It is worth noting that our metric collection guides are presented as a single deployment that scrapes metrics from all projects in an account. Whereas our log forwarders are presented as a deployment in each project to be logged.