update

content/post/elasticsearch.md

@@ -0,0 +1,187 @@
+---
+title: "Elasticsearch 笔记"
+date: 2019-10-30T11:49:53+08:00
+lastmod: 2019-10-30T11:49:53+08:00
+tags: ["elasticsearch"]
+categories: ["database"]
+---
+
+# 索引
+- 查看某节点的全部索引
+    ```bash
+    curl http://127.0.0.1:9200/_cat/indices?v
+    ```
+- 新建 index
+    ```bash
+    curl -X PUT http://127.0.0.1:9200/index_name
+    ```
+- 删除 index
+    ```bash
+    curl -X DELETE http://127.0.0.1:9200/index_name
+    ```
+
+# 记录
+- 新增记录(指定记录id)
+    ```bash
+    curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
+    {
+        "aa": "11",
+        "bb": "22"
+    }'
+    ```
+- 新增记录(不指定记录id)
+    ```bash
+    curl -X POST -H "Content-Type: application/json" http://127.0.0.1:9200/index_name -d '
+    {
+        "aa": "11",
+        "bb": "22"
+    }'
+    ```
+- 查看记录
+    ```bash
+    curl http://127.0.0.1:9200/index_name/doc_id?pretty=true
+    ```
+- 删除记录
+    ```bash
+    curl -X DELETE http://127.0.0.1:9200/index_name/doc_id
+    ```
+- 更新记录
+    ```bash
+    curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
+    {
+        "aa": "33",
+        "bb": "44"
+    }'
+    ```
+
+# 查询
+- 查询所有记录
+    ```bash
+    curl http://127.0.0.1:9200/index_name/_search
+    ```
+- 查询匹配
+    ```bash
+    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
+    {
+        "query": {"match": {"key_name": "value_pattern"}}
+    }'
+    ```
+- 从位置2(默认0)开始查询8(默认10)条记录
+    ```bash
+    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
+    {
+        "query": {"match": {"key_name": "value_pattern"}},
+        "from": 2,
+        "size": 8
+    }'
+    ```
+- 逻辑 or 查询
+    ```bash
+    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
+    {
+        "query": {"match": {"key_name": "value_pattern_1 value_pattern_2"}}
+    }'
+    ```
+- 逻辑 and 查询
+    ```bash
+    curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
+    {
+        "query": {
+            "bool": {
+                "must": [
+                    {"match": {"key_name": "value_pattern_1"}},
+                    {"match": {"key_name": "value_pattern_2"}}
+                ]
+            }
+        }
+    }'
+    ```
+- 区间查询
+    ```bash
+    set -euo pipefail
+    export START_TIME="$(date +%s -d $1)"
+    export END_TIME="$(date +%s -d $2)"
+    curl -s -H "Content-Type: application/json" -o result.txt \
+        http://127.0.0.1:9200/wangmei_raw/_search?pretty -d @- <<EOF
+    {
+      "_source": [
+        "spider_name",
+        "spider_time",
+        "media_name",
+        "publish_time"
+      ],
+      "query": {
+        "bool": {
+          "filter": {
+            "range": {
+              "spider_time": {
+                "gt": $START_TIME,
+                "lte": $END_TIME
+              }
+            }
+          }
+        }
+      },
+      "size": 10000
+    }
+    EOF
+    ```
+
+# Kibana
+- lucene 正则查询
+    ```
+    #查询包含10000-99999毫秒的 message 字段
+    { "regexp": { "message": "[0-9]{5}ms" } }
+    ```
+- Dev tool 模拟 pipeline
+    ```
+    POST _ingest/pipeline/_simulate
+    {
+      "pipeline" : {
+        "description": "",
+        "processors": [
+          {
+            "grok": {
+              "field": "message",
+              "patterns": [
+                "",
+                ""
+              ],
+              "ignore_missing": true,
+              "ignore_failure": true
+            }
+          }
+        ]
+      },
+      "docs" : [
+        {
+          "_source": {
+            "message": ""
+          }
+        },
+        {
+          "_source": {
+            "message": ""
+          }
+        }
+      ]
+    }
+    ```
+
+# pipeline
+- 查看已有的 pipeline
+    ```bash
+    curl http://127.0.0.1:9200/_ingest/pipeline?pretty=true
+    # 指定 nginx-access
+    curl http://127.0.0.1:9200/_ingest/pipeline/nginx-access?pretty=true
+    ```
+- [Mysql 慢查询日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/mysql-slow.json)
+- [Secure 登陆日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/secure-login.json)
+- [Nginx access 日志](https://git.colben.cn/colben/myfilebeat/raw/master/pipelines/nginx-access.json)
+
+# filebeat
+- [常用配置](https://git.colben.cn/colben/myfilebeat/raw/master/filebeat.yml)
+- [收割 secure 日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/secure.yml)
+- [收割 mysql 慢查询日志和错误日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/mysql.yml)
+- [收割 nginx access 日志](https://git.colben.cn/colben/myfilebeat/raw/master/prospectors.d/nginx.yml)
+