6 Key Business Uses for Web Traffic Logs

There are multiple ways for businesses to use logs - from tracking user engagement, to following compliance guidelines, to monitoring suspicious activity. Frequently, however, businesses treat logs and metrics as an afterthought rather than as an integral part of their corporate strategy and development process. Furthermore, the volume of data available from logging can be overwhelming, particularly if you don’t have a clear goal in mind and established processes set up for its collection and analysis. Similarly, important data can be missing if the right procedures haven’t been set up to collect it. If logging is given careful thought as part of a larger plan, however, it can prove a highly valuable tool. The following six use cases are largely for web traffic logs. There are, of course, a wide range of other use case scenarios.

What You Can Learn From Web Traffic Logs

1. Track user engagement with your product: By analyzing data captured in your logs, you can watch how users interact with your application at a detailed level, noticing, for example, when they abandon transactions or using the logs to track aggregate use to work out the most popular features. These insights can then be exploited to tweak different aspects of the application to improve user engagement and increase revenue.
2. Troubleshoot problems: Logs can be helpful when troubleshooting application problems as you can use historic logs to compare real-time findings to baseline data. This can be useful in working out contributing factors and efficiently resolving them. Ongoing monitoring can also help you spot unusual patterns and identify problems before they have the chance to affect users.
3. Offer improved user support: When users approach companies for help on an issue, they are not always articulate or eloquent about exactly what the problem is or the steps they took that led up to it. By looking at the associated logs (if they are set up to track user clicks and requests), your help desk team can more rapidly diagnose what the problem is and take steps to resolve it.
4. Enable effective security monitoring: For effective use of logging for security purposes, it is important to regularly review your logs to both monitor real-time security concerns and track larger patterns and trends over longer durations. Make sure you have the appropriate processes in place to fully utilize logs to their maximum advantage.
5. Meet compliance requirements: Logs can provide a useful record of the kind of data that compliance standards and legal requirements demand you maintain. They can also show who has accessed that data and when.
6. Support data integration: By creating a central area in which all of an organization’s data is readily accessible, well-managed logs can be used strategically to support data integration.

Section Logs

As part of our goal to be the most DevOps-integrated Edge Compute Platform solution in the market, the Section platform supplies numerous detailed logs and metrics, giving valuable insight into how your websites and proxies are running. Every Section account includes these tools, letting developers gather baseline data before issuing any improvements, easily assess the impact of any changes made, and troubleshoot issues as they happen.

At Section, we work with three different types of logs:

• Grafana Metric Visualization
• Graphite Monitoring
• HTTP Logs

The Section Console Overview Page

Each environment within Section is provided with a distinct overview page, which provides a range of information, including:

• General information about the environment’s proxy stack - you can view all the proxies running in your environment, from the edge proxy to the LastProxy, as well as their current order;
• HTTP traffic summary - this is presented as an overview graph which offers views into three different metrics:
  • (i) the number of HTTP responses served per minute, offering insight into the overall volume of traffic hitting your site
  • (ii) the breakdown of error requests for the last hour, helping you detect any recent increases in errors served
  • (iii) the downstream response bandwidth served per minute, showing you the total amount of bandwidth sent to your customers
• Data from Varnish Cache (cache hit, pass and misses for the last hour and upstream request bandwidth served per minute) or Modsecurity (intercepts, audits and passes % for the last hour and upstream request bandwidth served per minute), depending on your choice of vendor;
• Errors that may have occurred between Section and the origin server in recent history;
• Links to access a more granular level of information, including HTTP logs and DNS records.

Grafana Dashboards and Metrics Visualization

The Grafana view offers a holistic way to visualize Graphite metrics in straightforward-to-understand visualizations. Section offers several different ways to view the metrics, starting with the Grafana view, which is an overall holistic level visualization tool using queries from Graphite. We also provide four separate dashboards, which can be viewed individually to gain different types of insight into your logs:

• Traffic Summary - This view will help you understand your website’s traffic from an overall perspective. It includes the number of HTTP requests served per minute and the corresponding bytes served per minute, in addition to a break down of all client side and server side errors.
• Varnish Cache Overview - This view will only be available if you have a Varnish Cache proxy set up. If you do, then these charts will be helpful in providing details about your application’s hit, miss and pass ratios. This will help you understand how much content you’re serving from the cache, allowing you to adjust it as needed.
• Varnish Cache Request Performance - This is a more detailed dashboard that supplies metrics on time to serve and bandwidth by hit type, content type and response time. It offers a good starting indication of the type of content to cache and provides an overview of how your website is responding to cache requests.
• Varnish Cache Hit Rates - This dashboard further breaks down the overall hit rates by their content type i.e. Images, HTML, CSS, JavaScript.

Graphite Monitoring

You can easily change the view from Grafana Dashboards to Graphite Metrics, which allows you to customize how you pull data using the same data source. You can then create your own customized dashboards for a more flexible working environment. You can view data within all the different environments, including your production environment, your development environment and any others. You can then put this data to use to create your own graphs, allowing you to easily change what is displayed and the look of each graph so that you can view statistics in the way that best suits you and your team, and fine-tune your configurations accordingly.

HTTP Logs

HTTP Logs give you a view of Kibana, which lets you dig deeper into your data and the trends you have identified in Graphite or Grafana. HTTP Logs lets you search through your logs for details within specific fields, for instance you can search using ‘scheme’ to determine whether a request was HTTP or HTTPS or ‘status’ to show the response code (200, 3XX, 4XX, etc.). These searches can then be turned into different graphs in the “Visualization” view, and saved under the “Dashboard” view for ease of use.