Skip to main content

Log Streaming to External Tools

Stream logs to external tools to get more robust support for log viewing than is possible using kubectl logs.

If your Project includes a Kubernetes Service decorated with the label section.io/logstream-destination: "true" and listening on UDP port 5160, the Section Platform will collect the container logs for all the Pods in your Project and forward them to this Service's Endpoint(s) in JSON format, similar to this (pretty-printed here):

example-log-entry.json
{
"@timestamp": "2022-07-01T23:45:56.999Z",
"kubernetes": {
"container": {
"name": "frontend"
},
"labels": {
"app": "contoso",
"pod-template-hash": "69bc8cd654"
},
"namespace": "default",
"node": {
"name": "syd-umvnp"
},
"pod": {
"ip": "172.17.0.5",
"name": "contoso-69bc8cd654-6lhrm"
}
},
"log": {
"offset": 23456
},
"message": "Application has completed initialization",
"stream": "stdout"
}

Your Service Endpoint Pods can receive these log packets, process them, and forward them to an external log ingestion service for storage and querying. The JSON over UDP/5160 wire format is simple enough if you want to implement your own application to process, however the popular CNCF project Fluentd has a UDP input plugin exactly for this. Here is an example Fluentd source configuration:

example-source-configuration
<source>
@type udp
tag all_section_logs
<parse>
@type json
</parse>
port 5160
message_length_limit 1MB # only 4096 bytes by default
</source>

And then you'll need a matching output configuration, which contains specifics for your log destination. The following matches all logs tagged with all_section_logs from the <source> configuration.

prototype-output-configuration
<match all_section_logs>
YOUR_LOG_DESTINATION_CONFIG_HERE
</match>

Popular log ingestion services will have a Fluentd output plugin and configuration for shipping logs to their system. The table below lists a number of examples with detailed Section integrations where available.

Log DestinationDocumentationPluginSection Log Integration
DatadogDocumentationPluginLog Integration
S3Documentation
SplunkDocumentationPlugin
Sumo LogicPlugin
Grafana LokiDocumentationPluginLog Integration
DynatracePlugin
ElasticsearchPlugin
New RelicPluginLog Integration
Google CloudPlugin
LogtailPlugin

Here is an example Kubernetes yaml to deploy Fluentd for log shipping where we put it all together:

log-shipping.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: fluentd
section.io/logstream-destination: "true"
name: fluentd
namespace: default
spec:
ports:
- name: fluentdudp
port: 5160
protocol: UDP
targetPort: 5160
selector:
app: fluentd
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fluentd
namespace: default
labels:
app: fluentd
spec:
replicas: 1
selector:
matchLabels:
app: fluentd
template:
metadata:
labels:
app: fluentd
section.io/logstream-collect: "false"
spec:
containers:
- name: fluentd
image: YOUR_FLUENTD_IMAGE_HERE
imagePullPolicy: Always
resources:
requests:
memory: ".5Gi"
cpu: "500m"
limits:
memory: ".5Gi"
cpu: "500m"
volumeMounts:
- name: config
mountPath: /fluentd/etc/fluent.conf
readOnly: true
subPath: fluent.conf
volumes:
- name: config
configMap:
name: fluent-conf
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-conf
namespace: default
data:
fluent.conf: |-
<source>
@type udp
tag all_section_logs
<parse>
@type json
</parse>
port 5160
message_length_limit 1MB
</source>

<match all_section_logs>
YOUR_LOG_DESTINATION_CONFIG_HERE
</match>

Our specific integration guides give exactly the code needed for YOUR_LOG_DESTINATION_CONFIG_HERE and YOUR_FLUENTD_IMAGE_HERE to make configuration easy.

Differences from Metric Collection‚Äč

It is worth noting that our metric collection guides are presented as a single deployment that scrapes metrics from all projects in an account. Whereas our log forwarders are presented as a deployment in each project to be logged.