The ELK Stack, recently rebranded as the Elastic Stack, is a combination of several open-source tools: ElasticSearch, LogStash, and Kibana, that are combined to create a popular log management tool. The ELK Stack is an extremely useful set of tools for developers which has a wide range of uses. The most popular use case of the ELK Stack is getting increased visibility into how applications are working through an advanced log management.
Below is an explanation of the ELK Stack components and information on how Section uses the ELK Stack to give developers more visibility into how their content delivery configuration and associated reverse proxies (like Varnish Cache and ModSecurity) handle requests.
What makes up the ELK Stack?
ElasticSearch
ElasticSearch is a near-real time search engine that, as the name implies, is highly scalable and flexible. It centrally stores data so documents can be searched quickly, and allows for advanced queries so developers can get detailed analysis. ElasticSearch is based on the Lucene search engine, another open-source software, and built with RESTful APIs for simple deployment.
LogStash
LogStash is the data collection pipeline which sits in front of ElasticSearch to collect data inputs and pipe said data to a variety of different destinations - ElasticSearch being the destination for this data when utilizing the ELK Stack. LogStash supports a wide range of data types and sources (including web applications, hosting services, content delivery solutions, and web application firewalls or caching servers), and can collect them all at once so you have all the data you need immediately.
Kibana
Kibana visualizes ElasticSearch documents so it’s easy for developers to have immediate insight into the documents stored and how the system is operating. Kibana offers interactive diagrams that can visualize complex queries done through ElasticSearch, along with Geospatial data and timelines that show you different services are performing over time. Kibana also makes it easy for developers to create and save custom graphs that fit the needs of their specific applications.
ELK Stack vs Splunk
The ELK Stack has become popular with developers over legacy tools such as Splunk. This is mainly due to the increased visibility offered by the ELK Stack and its friendliness to DevOps teams - using the combined power of ElasticSearch, LogStash, and Kibana development and operations teams are able to quickly and easy collect, parse, and view data to troubleshoot and resolve complex problems.
As applications become more complex and utilize tools like bot blockers, WAFs, and caching proxies, the amount of data going into a log management system has grown, so having a fast and scalable system is important. This has also impacted price, making paid services like Splunk more expensive and more suitable to enterprise level companies. In addition, the open-source nature of the ELK Stack and the flexibility it offers is highly appealing to modern developers.
The ELK Stack is self-managed which can have some downsides, as teams will need to dedicate time to learning how to use it to its full potential. However, given its popularity there are numerous tutorials and guides to using the ELK Stack.
ELK Stack and Content Delivery
Content delivery solutions provide users with a huge amount of data representing the different proxies utilized in any content delivery solution - Section offers our users a wide range of reverse proxies including Varnish Cache, PageSpeed, and ModSecurity, and each of these tools comes with their own set of logs, in addition to the logs from Section’s global network of servers.
At Section we feel it’s imperative that users have access to fast, easily visualized logging tools built into the content delivery system for visibility across systems, and so we leverage the ELK Stack for this purpose. Through the ELK Stack Section users get real-time logs and metrics that can quickly be sorted through to identify and troubleshoot issues with caching, WAF, or other configurations. In the below video, you can see how to utilize Kibana within Section to query your logs.
DevOps-Integrated Content Delivery
To learn more about how Section supports DevOps workflows through ELK Stack log management, real time metrics through Graphite visualized in Grapaha, and a local developer PoP for testing, please contact us or request a demo of Section’s platform.
