We use ELK stack heavily in our production systems for log aggregation and monitoring.
Our daily log size generated is aroudn 100GB. Since, we do not intend to keep the log files in ELK for more than a month, it becomes important that we delete those indices to free up disk space.
Fortunately, Logstash creates a new index every day by default.
Thus, we can ask Curator to simply delete the indices x days old and which follow a particular naming pattern.
Simple use pip install elasticsearch-curator to install Curator on your machine. I prefer to install it on the Elasticsearch machine itself.
Create a file curator.yml with following contents.
blacklist: [‘elasticsearch’, ‘urllib3’]
Now, we need to define an action. i.e. What will curator do. There are many actions to choose from. Check the documentation for more information
For this dicussion, we will use Delete Indices as the action, since this is what we want to do.
Below is the sample action file delete_indices.yml , which will delete the logstash indices which are older than 10 days.
Delete indices older than 45 days (based on index name), for logstash-
prefixed indices. Ignore the error if the filter does not result in an
actionable list of indices (ignore_empty_list) and exit cleanly.
– filtertype: pattern
– filtertype: age
To run this action, simple use the command
curator ./delete_index.yml –config ./curator.yml –dry-run
2017-04-09 17:27:46,075 INFO Preparing Action ID: 1, “delete_indices”
2017-04-09 17:27:46,080 INFO Trying Action ID: 1, “delete_indices”: Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2017-04-09 17:27:46,538 INFO DRY-RUN MODE. No changes will be made.
2017-04-09 17:27:46,538 INFO (CLOSED) indices may be shown that may not be acted on by action “delete_indices”.
2017-04-09 17:27:46,538 INFO Action ID: 1, “delete_indices” completed.
2017-04-09 17:27:46,538 INFO Job completed.
The –dry-run mode will not actually delete the index. It can be used to test the output of the action.
If you want to schedule it in a cron, you can do so using crontab -e
00 8 * * * root curator /path/delete_index.yml –config /path/curator.yml
The above configuration will cleanup the indices older than 10 days everyday at 8 AM.