Elastic Stack with Docker Compose
After working for several years as a network engineer, I developed an interest in cybersecurity, which eventually led me to shift my focus and begin working in this field. My first experience with a Security Information and Event Management (SIEM) system was with the Elastic Stack, though initially, I only worked with Elasticsearch and Kibana.
Starting my first adventure with Elasticsearch, I found it to be an incredibly user-friendly and intuitive solution. As I gained more experience and a deeper understanding of how these systems operate, I began to consider the possibilities of automation, particularly for use in lab environments or demonstration setups. This led me to explore Docker and Docker Compose, choosing to learn by doing, even though I had no prior knowledge in this area.
During my initial research, I noticed that there were no existing projects implementing a full SSL chain as recommended by the best practices in Elasticsearch’s documentation (https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html). My goal was to replicate a production-like environment as closely as possible, which inspired me to create a Docker Compose project for the Elastic Stack. The result of this effort is now available on GitHub (https://github.com/celeroon/docker-compose-elastic-stack).
As I mentioned earlier, ensuring encrypted connections was a priority for me. In my setup, it was essential to secure communication between various components of the Elastic Stack, including Elasticsearch, Kibana, Fleet Server, and Elastic Agents, as well as Metricbeat and Filebeat. I chose to install Metricbeat and Filebeat specifically for cluster health monitoring, as an alternative to using X-Pack Monitoring.
To test the available rules — both the pre-built ones provided by Elasticsearch and the community-driven Sigma rules (https://github.com/SigmaHQ/sigma) — I deployed a specific container. This container’s primary function was to download the pre-built rules (initially in a disabled state) and also to download, parse, and convert Sigma rules from Sigma syntax to Elastic Common Schema (ECS) format using Sigma tools. These converted rules were then uploaded to Kibana as NDJSON objects.
It’s important to note that enabling all the rules at once can generate a significant amount of noise, which might overwhelm the system with alerts. To manage this, I recommend first reviewing the information available about each rule. Alternatively, you can start by enabling only the critical rules if your goal is to test specific scenarios or functionalities.
Given my experience in networking and firewalls, and as a fan of FortiGate firewall solutions, I decided to implement a Logstash container to parse syslog data from FortiGate devices. This setup allowed me to utilize dashboards for monitoring and create rules based on FortiGate logs. I want to extend my gratitude to the developer of the FortiDragon (https://github.com/enotspe/fortinet-2-elasticsearch) project, which provided custom pipelines and impressive dashboards that greatly facilitated this process. Although the method of implementing the FortiDragon solution via Logstash is now deprecated, it still functions effectively.
The newer approach involves using Custom UDP Logs integration, which I plan to explore in the future. It would be interesting to test this method, and I might create another branch for it later on. In my most recent project, which I will discuss in a later section, I implemented a single-node Elastic Stack deployment using Ansible, incorporating the Custom UDP Logs integration. I plan to create a separate repository for this and update this article with those details. However, I encountered some issues when using an already functioning Elastic Stack deployed via Docker Compose in a nested virtualization environment. Specifically, I ran into trouble when trying to install the Fleet Server.
I also want to extend my sincere thanks to Evermight Systems (https://www.youtube.com/@evermightsystems) and Ali Younes https://www.youtube.com/@AliYounesGo4IT) for their invaluable resources that helped me understand how to establish secure connections for the Elastic Stack. Additionally, the official Elastic blog post (https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose) provided a solid foundation for building the Elastic Stack using Docker Compose.
