ExternalVendorCode/devilbox
1
0
Fork 0
mirror of https://github.com/cytopia/devilbox.git synced 2025-03-24 21:15:40 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #510 from cytopia/elk-stack

Browse Source 
ELK stack
This commit is contained in:
cytopia 2019-02-26 21:39:29 +01:00 committed by GitHub
commit 1b1004c87e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 464 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.devilbox/www/config.php
View File

@ -14,7 +14,7 @@ putenv('RES_OPTIONS=retrans:1 retry:1 timeout:1 attempts:1');
$DEVILBOX_VERSION = 'v0.15';
$DEVILBOX_DATE = '2019-02-17';
$DEVILBOX_DATE = '2019-02-23';
$DEVILBOX_API_PAGE = 'devilbox-api/status.json';
//

1
CHANGELOG.md
View File

@ -37,6 +37,7 @@ major versions.
- Alpine images where possible
- Docker Compose overwrite images:
- Blackfire
- ELK (Elastic Search, Logstash and Kibana)
- MailHog
- RabbitMQ
- Solr

17
README.md
View File

@ -60,12 +60,12 @@ Before updating the Devilbox ensure to always check for breaking changes until t
The Devilbox ships the following pre-configured Docker container in any available version.
| Web server | SQL | NoSQL | Queue & Search | Cache | Misc |
|------------|------------|-----------|----------------|---------|-----------|
| Apache | MariaDB | Memcached | RabbitMQ | Varnish | Bind |
| Nginx | MySQL | MongoDB | Solr | | Blackfire |
| | PerconaDB | Redis | | | MailHog |
| | PostgreSQL | | | | PHP |
| Accel | Frontend | SQL | NoSQL | Queue & Search | ELK | Misc |
|---------|------------|------------|-----------|----------------|---------------|-----------|
| Varnish | Apache | MariaDB | Memcached | RabbitMQ | ElasticSearch | Bind |
| | Nginx | MySQL | MongoDB | Solr | Logstash | Blackfire |
| | PHP | PerconaDB | Redis | | Kibana | MailHog |
| | | PostgreSQL | | | | |
> **Documentation:**
> [Available Container](https://devilbox.readthedocs.io/en/latest/readings/available-container.html)
@ -338,6 +338,7 @@ Additionally to the default stack, there are a variety of other services that ca
<thead>
<tr>
<th>Blackfire</th>
<th>ELK</th>
<th>MailHog</th>
<th>RabbitMQ</th>
<th>Solr</th>
@ -347,6 +348,7 @@ Additionally to the default stack, there are a variety of other services that ca
<tbody>
<tr>
<td><a target="_blank" title="Blackfire 1.8" href="https://github.com/blackfireio/docker">1.8</a></td>
<td><a target="_blank" title="ELK stack" href="https://www.docker.elastic.co">5.x.y</a></td>
<td><a target="_blank" title="MailHog v1.0.0" href="https://github.com/mailhog/MailHog">v1.0.0</a></td>
<td><a target="_blank" title="RabbitMQ 3.6" href="https://github.com/rabbitmq/rabbitmq-server">3.6</a></td>
<td><a target="_blank" title="Solr 5" href="https://github.com/apache/lucene-solr">5</a></td>
@ -354,6 +356,7 @@ Additionally to the default stack, there are a variety of other services that ca
</tr>
<tr>
<td>...</td>
<td><a target="_blank" title="ELK stack" href="https://www.docker.elastic.co">6.x.y</a></td>
<td><a target="_blank" title="MailHog latest" href="https://github.com/mailhog/MailHog">latest</a></td>
<td><a target="_blank" title="RabbitMQ 3.7" href="https://github.com/rabbitmq/rabbitmq-server">3.7</a></td>
<td><a target="_blank" title="Solr 6" href="https://github.com/apache/lucene-solr">6</a></td>
@ -361,6 +364,7 @@ Additionally to the default stack, there are a variety of other services that ca
</tr>
<tr>
<td><a target="_blank" title="Blackfire 1.18.0" href="https://github.com/blackfireio/docker">1.18.0</a></td>
<td><a target="_blank" title="ELK stack" href="https://www.docker.elastic.co">7.x.y</a></td>
<td></td>
<td><a target="_blank" title="RabbitMQ latest" href="https://github.com/rabbitmq/rabbitmq-server">latest</a></td>
<td><a target="_blank" title="Solr 7" href="https://github.com/apache/lucene-solr">7</a></td>
@ -370,6 +374,7 @@ Additionally to the default stack, there are a variety of other services that ca
<td><a target="_blank" title="Blackfire latest" href="https://github.com/blackfireio/docker">latest</a></td>
<td></td>
<td></td>
<td></td>
<td><a target="_blank" title="Solr latest" href="https://github.com/apache/lucene-solr">latest</a></td>
<td><a target="_blank" title="Varnish latest" href="https://github.com/devilbox/docker-varnish">latest</a></td>
</tr>

2
compose/README.md
View File

@ -1,3 +1,5 @@
# Docker Compose overwrites
This directory container various `docker-compose.override.yml` examples to be used with the Devilbox.
Note that those override files have their own environment variables that should be **appended** to `.env`

77
compose/docker-compose.override.yml-all
View File

@ -1,7 +1,14 @@
# vim: set ft=yaml:
version: '2.1'
###
### Services
###
services:
# -----------------------------------------------------------------------------------------------
# Blackfire
# -----------------------------------------------------------------------------------------------
blackfire:
image: blackfire/blackfire:${BLACKFIRE:-latest}
hostname: blackfire
@ -18,6 +25,9 @@ services:
- php
- httpd
# -----------------------------------------------------------------------------------------------
# MailHog
# -----------------------------------------------------------------------------------------------
mailhog:
image: mailhog/mailhog:${MAILHOG_SERVER:-latest}
hostname: mailhog
@ -31,6 +41,9 @@ services:
- php
- httpd
# -----------------------------------------------------------------------------------------------
# RabbitMQ
# -----------------------------------------------------------------------------------------------
rabbit:
image: rabbitmq:${RABBIT_SERVER:-management}
hostname: rabbit
@ -51,6 +64,9 @@ services:
- php
- httpd
# -----------------------------------------------------------------------------------------------
# Solr
# -----------------------------------------------------------------------------------------------
solr:
image: solr:${SOLR_SERVER:-latest}
hostname: solr
@ -70,6 +86,9 @@ services:
- php
- httpd
# -----------------------------------------------------------------------------------------------
# Varnish
# -----------------------------------------------------------------------------------------------
varnish:
image: devilbox/varnish:${VARNISH_SERVER:-6}
hostname: varnish
@ -91,5 +110,63 @@ services:
- php
- httpd
# -----------------------------------------------------------------------------------------------
# ELK Stack
# -----------------------------------------------------------------------------------------------
elastic:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:${ELK_SERVER:-6.6.1}
hostname: elastic
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_ELASTIC:-9200}:9200"
networks:
app_net:
ipv4_address: 172.16.238.240
environment:
- TZ=${TIMEZONE:-UTC}
- discovery.type=single-node
volumes:
- data-elastic:/usr/share/elasticsearch/data
logstash:
image: docker.elastic.co/logstash/logstash-oss:${ELK_SERVER:-6.6.1}
hostname: logstash
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_LOGSTASH:-9600}:9600"
networks:
app_net:
ipv4_address: 172.16.238.241
environment:
- TZ=${TIMEZONE:-UTC}
- LOG_LEVEL=info
- HTTP_HOST=0.0.0.0
- HTTP_PORT=9600
- CONFIG_RELOAD_AUTOMATIC=true
- CONFIG_RELOAD_INTERVAL=300
- DB_HOST=DATABASE-SERVER
volumes:
- data-logstash:/var/lib/logstash
depends_on:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana-oss:${ELK_SERVER:-6.6.1}
hostname: kibana
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_KIBANA:-5601}:5601"
networks:
app_net:
ipv4_address: 172.16.238.242
environment:
- TZ=${TIMEZONE:-UTC}
- ELASTICSEARCH_URL=http://elastic:9200
depends_on:
- elastic
###
### Volumes
###
volumes:
data-solr:
data-elastic:
data-logstash:

5
compose/docker-compose.override.yml-blackfire
View File

@ -1,6 +1,11 @@
# vim: set ft=yaml:
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# Blackfire
# -----------------------------------------------------------------------------------------------
blackfire:
image: blackfire/blackfire:${BLACKFIRE:-latest}
hostname: blackfire

62
compose/docker-compose.override.yml-elk Normal file
View File

@ -0,0 +1,62 @@
# vim: set ft=yaml:
# https://www.docker.elastic.co
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# ELK Stack
# -----------------------------------------------------------------------------------------------
elastic:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:${ELK_SERVER:-6.6.1}
hostname: elastic
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_ELASTIC:-9200}:9200"
networks:
app_net:
ipv4_address: 172.16.238.240
environment:
- TZ=${TIMEZONE:-UTC}
- discovery.type=single-node
volumes:
- data-elastic:/usr/share/elasticsearch/data
logstash:
image: docker.elastic.co/logstash/logstash-oss:${ELK_SERVER:-6.6.1}
hostname: logstash
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_LOGSTASH:-9600}:9600"
networks:
app_net:
ipv4_address: 172.16.238.241
environment:
- TZ=${TIMEZONE:-UTC}
- LOG_LEVEL=info
- HTTP_HOST=0.0.0.0
- HTTP_PORT=9600
- CONFIG_RELOAD_AUTOMATIC=true
- CONFIG_RELOAD_INTERVAL=300
- DB_HOST=DATABASE-SERVER
volumes:
- data-logstash:/var/lib/logstash
depends_on:
- elastic
kibana:
image: docker.elastic.co/kibana/kibana-oss:${ELK_SERVER:-6.6.1}
hostname: kibana
ports:
- "${LOCAL_LISTEN_ADDR}${HOST_PORT_ELK_KIBANA:-5601}:5601"
networks:
app_net:
ipv4_address: 172.16.238.242
environment:
- TZ=${TIMEZONE:-UTC}
- ELASTICSEARCH_URL=http://elastic:9200
depends_on:
- elastic
volumes:
data-elastic:
data-logstash:

5
compose/docker-compose.override.yml-mailhog
View File

@ -1,6 +1,11 @@
# vim: set ft=yaml:
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# MailHog
# -----------------------------------------------------------------------------------------------
mailhog:
image: mailhog/mailhog:${MAILHOG_SERVER:-latest}
hostname: mailhog

5
compose/docker-compose.override.yml-rabbitmq
View File

@ -1,6 +1,11 @@
# vim: set ft=yaml:
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# RabbitMQ
# -----------------------------------------------------------------------------------------------
rabbit:
image: rabbitmq:${RABBIT_SERVER:-management}
hostname: rabbit

5
compose/docker-compose.override.yml-solr
View File

@ -1,6 +1,11 @@
# vim: set ft=yaml:
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# Solr
# -----------------------------------------------------------------------------------------------
solr:
image: solr:${SOLR_SERVER:-latest}
hostname: solr

5
compose/docker-compose.override.yml-varnish
View File

@ -1,6 +1,11 @@
# vim: set ft=yaml:
version: '2.1'
services:
# -----------------------------------------------------------------------------------------------
# Varnish
# -----------------------------------------------------------------------------------------------
varnish:
image: devilbox/varnish:${VARNISH_SERVER:-6}
hostname: varnish

24
docs/_includes/links/documentation.rst
View File

@ -87,3 +87,27 @@
<a target="_blank" href="https://hub.docker.com/r/devilbox/varnish/">
Dockerhub: Varnish <img src="https://raw.githubusercontent.com/cytopia/icons/master/11x11/ext-link.png" />
</a>
.. |ext_lnk_elk_docker_registry| raw:: html
<a target="_blank" href="https://www.docker.elastic.co">
Elastic Docker: ELK <img src="https://raw.githubusercontent.com/cytopia/icons/master/11x11/ext-link.png" />
</a>
.. |ext_lnk_elk_elastic_github| raw:: html
<a target="_blank" href="https://github.com/elastic/elasticsearch">
GitHub: Elastic Search <img src="https://raw.githubusercontent.com/cytopia/icons/master/11x11/ext-link.png" />
</a>
.. |ext_lnk_elk_logstash_github| raw:: html
<a target="_blank" href="https://github.com/elastic/logstash">
GitHub: Logstash <img src="https://raw.githubusercontent.com/cytopia/icons/master/11x11/ext-link.png" />
</a>
.. |ext_lnk_elk_kibana_github| raw:: html
<a target="_blank" href="https://github.com/elastic/kibana">
GitHub: Kibana <img src="https://raw.githubusercontent.com/cytopia/icons/master/11x11/ext-link.png" />
</a>

6
docs/_includes/snippets/additional-container.rst
View File

@ -11,3 +11,9 @@
+---------------------------+-----------+-----------+----------------+
| Varnish | varnish | varnish | 172.16.238.230 |
+---------------------------+-----------+-----------+----------------+
| ELK: Elastic Search | elastic | elastic | 172.16.238.240 |
+---------------------------+-----------+-----------+----------------+
| ELK: Logstash | logstash | logstash | 172.16.238.241 |
+---------------------------+-----------+-----------+----------------+
| ELK: Kibana | kibana | kibana | 172.16.238.242 |
+---------------------------+-----------+-----------+----------------+

3
docs/_includes/snippets/docker-compose-override-tree-view.rst
View File

@ -10,12 +10,13 @@ However, each example also exists in its standalone file as shown below:
compose/
├── docker-compose.override.yml-all
├── docker-compose.override.yml-blackfire
├── docker-compose.override.yml-elk
├── docker-compose.override.yml-mailhog
├── docker-compose.override.yml-rabbitmq
├── docker-compose.override.yml-solr
├── docker-compose.override.yml-varnish
└── README.md
0 directories, 6 files
0 directories, 8 files
.. seealso:: :ref:`custom_container_enable_all_additional_container`

1
docs/custom-container/enable-all-container.rst
View File

@ -56,6 +56,7 @@ In order to fully customize each container, refer to their own documentation sec
.. seealso::
* :ref:`custom_container_enable_blackfire`
* :ref:`custom_container_enable_elk_stack`
* :ref:`custom_container_enable_mailhog`
* :ref:`custom_container_enable_rabbitmq`
* :ref:`custom_container_enable_solr`

251
docs/custom-container/enable-elk-stack.rst Normal file
View File

@ -0,0 +1,251 @@
.. include:: /_includes/all.rst
.. _custom_container_enable_elk_stack:
******************************
Enable and configure ELK Stack
******************************
This section will guide you through getting ELK stack (Elastic Search, Logstash and Kibana)
integrated into the Devilbox.
.. seealso::
* |ext_lnk_elk_elastic_github|
* |ext_lnk_elk_logstash_github|
* |ext_lnk_elk_kibana_github|
* |ext_lnk_elk_docker_registry|
* :ref:`custom_container_enable_all_additional_container`
* :ref:`docker_compose_override_yml_how_does_it_work`
**Table of Contents**
.. contents:: :local:
Overview
========
Available overwrites
--------------------
.. include:: /_includes/snippets/docker-compose-override-tree-view.rst
ELK settings
------------
In case of ELK stack, the file is ``compose/docker-compose.override.yml-elk``. This file
must be copied into the root of the Devilbox git directory.
+-----------------------+------------------------------------------------------------------------------------------------------+
| What | How and where |
+=======================+======================================================================================================+
| Example compose file | ``compose/docker-compose.override.yml-all`` or |br| ``compose/docker-compose.override.yml-elk`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
Elastic Search
^^^^^^^^^^^^^^
+-----------------------+------------------------------------------------------------------------------------------------------+
| What | How and where |
+=======================+======================================================================================================+
| Container IP address | ``172.16.238.240`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container host name | ``elastic`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container name | ``elastic`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Mount points | none |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Exposed port | ``9200`` (can be changed via ``.env``) |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Available at | ``http://localhost:9200`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Further configuration | ``.env`` vars |
+-----------------------+------------------------------------------------------------------------------------------------------+
Logstash
^^^^^^^^
+-----------------------+------------------------------------------------------------------------------------------------------+
| What | How and where |
+=======================+======================================================================================================+
| Container IP address | ``172.16.238.241`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container host name | ``logstash`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container name | ``logstash`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Mount points | none |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Exposed port | ``9600`` (can be changed via ``.env``) |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Available at | ``tcp://localhost:9600`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Further configuration | ``.env`` vars |
+-----------------------+------------------------------------------------------------------------------------------------------+
kibana
^^^^^^
+-----------------------+------------------------------------------------------------------------------------------------------+
| What | How and where |
+=======================+======================================================================================================+
| Container IP address | ``172.16.238.242`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container host name | ``kibana`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Container name | ``kibana`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Mount points | none |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Exposed port | ``9600`` (can be changed via ``.env``) |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Available at | ``tcp://localhost:9600`` |
+-----------------------+------------------------------------------------------------------------------------------------------+
| Further configuration | ``.env`` vars |
+-----------------------+------------------------------------------------------------------------------------------------------+
ELK env variables
-----------------
Additionally the following ``.env`` variables can be created for easy configuration:
Elastic Search
^^^^^^^^^^^^^^
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| Variable | Default value | Description |
+==============================+===============================================+======================================================================+
| ``HOST_PORT_ELK_ELASTIC`` | ``9200`` | Controls the host port on which Elastic Search will be available at. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``ELK_SERVER`` | ``6.6.1`` | Controls the ELK stack version to use. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``TIMEZONE`` | ``UTC`` | Timezone for Elastic Search (already available in ``.env``). |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
Logstash
^^^^^^^^
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| Variable | Default value | Description |
+==============================+===============================================+======================================================================+
| ``HOST_PORT_ELK_LOGSTASH`` | ``9600`` | Controls the host port on which Logstash will be available at. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``ELK_SERVER`` | ``6.6.1`` | Controls the ELK stack version to use. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``TIMEZONE`` | ``UTC`` | Timezone for Logstash (already available in ``.env``). |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
Kibana
^^^^^^
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| Variable | Default value | Description |
+==============================+===============================================+======================================================================+
| ``HOST_PORT_ELK_KIBANA`` | ``5601`` | Controls the host port on which Kibana will be available at. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``ELK_SERVER`` | ``6.6.1`` | Controls the ELK stack version to use. |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
| ``TIMEZONE`` | ``UTC`` | Timezone for Kibana (already available in ``.env``). |
+------------------------------+-----------------------------------------------+----------------------------------------------------------------------+
Instructions
============
1. Copy docker-compose.override.yml
-----------------------------------
Copy the ELK stack Docker Compose overwrite file into the root of the Devilbox git directory.
(It must be at the same level as the default ``docker-compose.yml`` file).
.. code-block:: bash
host> cp compose/docker-compose.override.yml-elk docker-compose.override.yml
.. seealso::
* :ref:`docker_compose_override_yml`
* :ref:`add_your_own_docker_image`
* :ref:`overwrite_existing_docker_image`
2. Adjust ``.env`` settings (optional)
--------------------------------------
The ELK stack is using sane defaults, which can be changed by adding variables to the ``.env`` file
and assigning custom values.
Add the following variables to ``.env`` and adjust them to your needs:
.. code-block:: bash
:caption: .env
# ELK stack general
# See here for all versions: https://www.docker.elastic.co/
#ELK_SERVER=6.1.4
#ELK_SERVER=6.2.4
#ELK_SERVER=6.3.2
#ELK_SERVER=6.4.3
#ELK_SERVER=6.5.4
ELK_SERVER=6.6.1
# Elastic Search settings
HOST_PORT_ELK_ELASTIC=9200
# Logstash settings
HOST_PORT_ELK_LOGSTASH=9600
# Kibana settings
HOST_PORT_ELK_KIBANA=5601
.. seealso:: :ref:`env_file`
3. Start the Devilbox
---------------------
The final step is to start the Devilbox with ELK stack.
Let's assume you want to start ``php``, ``httpd``, ``bind``, ``elastic``, ``logstash``, ``kibana``.
.. code-block:: bash
host> docker-compose up -d php httpd bind elastic logstash kibana
.. seealso:: :ref:`start_the_devilbox`
TL;DR
=====
For the lazy readers, here are all commands required to get you started.
Simply copy and paste the following block into your terminal from the root of your Devilbox git
directory:
.. code-block:: bash
# Copy compose-override.yml into place
cp compose/docker-compose.override.yml-elk docker-compose.override.yml
# Create .env variable
echo "# ELK stack general" >> .env
echo "# See here for all versions:" >> .env
echo "# https://www.docker.elastic.co/" >> .env
echo "#ELK_SERVER=6.1.4" >> .env
echo "#ELK_SERVER=6.2.4" >> .env
echo "#ELK_SERVER=6.3.2" >> .env
echo "#ELK_SERVER=6.4.3" >> .env
echo "#ELK_SERVER=6.5.4" >> .env
echo "ELK_SERVER=6.6.1" >> .env
echo "# Elastic Search settings" >> .env
echo "HOST_PORT_ELK_ELASTIC=9200" >> .env
echo "# Logstash settings" >> .env
echo "HOST_PORT_ELK_LOGSTASH=9600" >> .env
echo "# Kibana settings" >> .env
echo "HOST_PORT_ELK_KIBANA=5601" >> .env
# Start container
docker-compose up -d php httpd bind elastic logstash kibana

1
docs/index.rst
View File

@ -114,6 +114,7 @@ host is ready to be served with your custom domain.
custom-container/enable-all-container
custom-container/enable-blackfire
custom-container/enable-elk-stack
custom-container/enable-mailhog
custom-container/enable-rabbitmq
custom-container/enable-solr