www.colben.cn/content/post/elasticsearch.md

---
title: "Elasticsearch 笔记"
date: 2019-10-30T11:49:53+08:00
lastmod: 2019-10-30T11:49:53+08:00
tags: ["elasticsearch"]
categories: ["database"]
---

## 索引
- 查看某节点的全部索引
    ```bash
    curl http://127.0.0.1:9200/_cat/indices?v
    ```
- 新建 index
    ```bash
    curl -X PUT http://127.0.0.1:9200/index_name
    ```
- 删除 index
    ```bash
    curl -X DELETE http://127.0.0.1:9200/index_name
    ```

## 记录
- 新增记录(指定记录id)
    ```bash
    curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
    {
        "aa": "11",
        "bb": "22"
    }'
    ```
- 新增记录(不指定记录id)
    ```bash
    curl -X POST -H "Content-Type: application/json" http://127.0.0.1:9200/index_name -d '
    {
        "aa": "11",
        "bb": "22"
    }'
    ```
- 查看记录
    ```bash
    curl http://127.0.0.1:9200/index_name/doc_id?pretty=true
    ```
- 删除记录
    ```bash
    curl -X DELETE http://127.0.0.1:9200/index_name/doc_id
    ```
- 更新记录
    ```bash
    curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
    {
        "aa": "33",
        "bb": "44"
    }'
    ```

## 查询
- 查询所有记录
    ```bash
    curl http://127.0.0.1:9200/index_name/_search
    ```
- 查询匹配
    ```bash
    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
    {
        "query": {"match": {"key_name": "value_pattern"}}
    }'
    ```
- 从位置2(默认0)开始查询8(默认10)条记录
    ```bash
    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
    {
        "query": {"match": {"key_name": "value_pattern"}},
        "from": 2,
        "size": 8
    }'
    ```
- 逻辑 or 查询
    ```bash
    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
    {
        "query": {"match": {"key_name": "value_pattern_1 value_pattern_2"}}
    }'
    ```
- 逻辑 and 查询
    ```bash
    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
    {
        "query": {
            "bool": {
                "must": [
                    {"match": {"key_name": "value_pattern_1"}},
                    {"match": {"key_name": "value_pattern_2"}}
                ]
            }
        }
    }'
    ```
- 区间查询
    ```bash
    set -euo pipefail
    export START_TIME="$(date +%s -d $1)"
    export END_TIME="$(date +%s -d $2)"
    curl -s -H "Content-Type: application/json" -o result.txt \
        http://127.0.0.1:9200/wangmei_raw/_search?pretty -d @- <<EOF
    {
      "_source": [
        "spider_name",
        "spider_time",
        "media_name",
        "publish_time"
      ],
      "query": {
        "bool": {
          "filter": {
            "range": {
              "spider_time": {
                "gt": $START_TIME,
                "lte": $END_TIME
              }
            }
          }
        }
      },
      "size": 10000
    }
    EOF
    ```

## Kibana
- lucene 正则查询
    ```
    #查询包含10000-99999毫秒的 message 字段
    { "regexp": { "message": "[0-9]{5}ms" } }
    ```
- Dev tool 模拟 pipeline
    ```
    POST _ingest/pipeline/_simulate
    {
      "pipeline" : {
        "description": "",
        "processors": [
          {
            "grok": {
              "field": "message",
              "patterns": [
                "",
                ""
              ],
              "ignore_missing": true,
              "ignore_failure": true
            }
          }
        ]
      },
      "docs" : [
        {
          "_source": {
            "message": ""
          }
        },
        {
          "_source": {
            "message": ""
          }
        }
      ]
    }
    ```

## pipeline
- 查看已有的 pipeline
    ```bash
    curl http://127.0.0.1:9200/_ingest/pipeline?pretty=true
    # 指定 nginx-access
    curl http://127.0.0.1:9200/_ingest/pipeline/nginx-access?pretty=true
    ```
- [Mysql 慢查询日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/mysql-slow.json)
- [Secure 登陆日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/secure-login.json)
- [Nginx access 日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/nginx-access.json)

## filebeat
- [常用配置](https://git.colben.cn/colben/myfilebeat/raw/master/filebeat.yml)
- [收割 secure 日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/secure.yml)
- [收割 mysql 慢查询日志和错误日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/mysql.yml)
- [收割 nginx access 日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/nginx.yml)