利用Curator清理Elasticsearch历史索引

发表于 | 分类于

curator 简介

Curator是一个用来管理Elasticsearch索引的工具，使用它可以管理需要删除或保留的索引数据。 当Elasticsearch作为ELK、EFK等日志收集方案的日志存储时，删除过期数据以释放存储空间显的格外重要，使用Curator可以删除旧的索引并优化系统。
curator官网地址：http://t.cn/RuwN0oM
Git地址：https://github.com/elastic/curator

安装

1
pip3 install elasticsearch-curator

配置

curator.yml

主要用来指定curator作为ES客户端的连接和日志配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
client:
  hosts:
    - 10.104.154.139
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: /usr/local/elasticsearch/logs/curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

actions定义在一个deleteLogs.yml文件中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: xxxxx_
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      exclude:

注意的地方有：

  • disable_action 要设置成False才是生效状态
  • filters.0.value 是匹配索引开头的字符串
  • filters.1.unit 单位
  • filters.1.unit_count 保留多少单位

运行

1
/usr/local/python3/bin/curator --config /usr/local/elasticsearch/config/curator.yml /usr/local/elasticsearch/config/deleteLogs.yml