使用Elasticsearch + Kibana快速搭建监控看板

070

背景

Elasticsearch是分布式搜索和分析引擎，是常见的NOSQL数据库之一

Kibana可以帮助我们快速搜索和分析Elasticsearch中的数据，并使用图表来展示这些数据

使用Elasticsearch + Kibana，我们可以快速搭建监控看板/数据看板。笔者在生活中会买一些债券基金来增加收入，这里我们就演示如何快速搭建一个债券基金看板，来帮助我们分析债券基金的收益高低和风险高低

准备

创建index

Elasticsearch使用index来管理和存储数据，这里我们创建一个债券基金的index，包含了以下字段：

  • date：日期
  • name：债券基金名称
  • income：收益率，一般都是万分之几，这里为了方便统计，收益率我们统一乘以一万

打开Kibana，在侧边栏找到Management -> Dev Tools -> Console，创建index

PUT /fund
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date",
        "format": [
          "yyyy-MM-dd"
        ]
      },
      "name": {
        "type": "keyword"
      },
      "income": {
        "type": "integer"
      }
    }
  }
}
PUT /fund
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date",
        "format": [
          "yyyy-MM-dd"
        ]
      },
      "name": {
        "type": "keyword"
      },
      "income": {
        "type": "integer"
      }
    }
  }
}
PUT /fund
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date",
        "format": [
          "yyyy-MM-dd"
        ]
      },
      "name": {
        "type": "keyword"
      },
      "income": {
        "type": "integer"
      }
    }
  }
}

准备数据

这里准备了债券基金6月的部分收益，数据如下：

POST /_bulk
{"index":{"_index":"fund","_id":"007195_2023-06-15"}}
{"date":"2023-06-15","name":"长城短债债券C","income":2}
{"index":{"_index":"fund","_id":"016240_2023-06-15"}}
{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}
{"index":{"_index":"fund","_id":"016319_2023-06-15"}}
{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}
{"index":{"_index":"fund","_id":"003860_2023-06-15"}}
{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}
{"index":{"_index":"fund","_id":"016527_2023-06-15"}}
{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}
{"index":{"_index":"fund","_id":"016791_2023-06-15"}}
{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}
{"index":{"_index":"fund","_id":"007195_2023-06-14"}}
{"date":"2023-06-14","name":"长城短债债券C","income":4}
{"index":{"_index":"fund","_id":"016240_2023-06-14"}}
{"date":"2023-06-14","name":"泰信添鑫中短债债券C","income":2}
{"index":{"_index":"fund","_id":"016319_2023-06-14"}}
{"date":"2023-06-14","name":"东方臻裕债券C","income":4}
{"index":{"_index":"fund","_id":"003860_2023-06-14"}}
{"date":"2023-06-14","name":"招商招旭纯债债券C","income":4}
{"index":{"_index":"fund","_id":"016527_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫诚短债债券C","income":2}
{"index":{"_index":"fund","_id":"016791_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫利中短债债券C","income":3}
{"index":{"_index":"fund","_id":"007195_2023-06-13"}}
{"date":"2023-06-13","name":"长城短债债券C","income":2}
{"index":{"_index":"fund","_id":"016240_2023-06-13"}}
{"date":"2023-06-13","name":"泰信添鑫中短债债券C","income":1}
{"index":{"_index":"fund","_id":"016319_2023-06-13"}}
{"date":"2023-06-13","name":"东方臻裕债券C","income":5}
{"index":{"_index":"fund","_id":"003860_2023-06-13"}}
{"date":"2023-06-13","name":"招商招旭纯债债券C","income":5}
{"index":{"_index":"fund","_id":"016527_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫诚短债债券C","income":3}
{"index":{"_index":"fund","_id":"016791_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫利中短债债券C","income":5}
POST /_bulk

{"index":{"_index":"fund","_id":"007195_2023-06-15"}}

{"date":"2023-06-15","name":"长城短债债券C","income":2}

{"index":{"_index":"fund","_id":"016240_2023-06-15"}}

{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}

{"index":{"_index":"fund","_id":"016319_2023-06-15"}}

{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}

{"index":{"_index":"fund","_id":"003860_2023-06-15"}}

{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}

{"index":{"_index":"fund","_id":"016527_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}

{"index":{"_index":"fund","_id":"016791_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}

{"index":{"_index":"fund","_id":"007195_2023-06-14"}}
{"date":"2023-06-14","name":"长城短债债券C","income":4}
{"index":{"_index":"fund","_id":"016240_2023-06-14"}}
{"date":"2023-06-14","name":"泰信添鑫中短债债券C","income":2}
{"index":{"_index":"fund","_id":"016319_2023-06-14"}}
{"date":"2023-06-14","name":"东方臻裕债券C","income":4}
{"index":{"_index":"fund","_id":"003860_2023-06-14"}}
{"date":"2023-06-14","name":"招商招旭纯债债券C","income":4}
{"index":{"_index":"fund","_id":"016527_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫诚短债债券C","income":2}
{"index":{"_index":"fund","_id":"016791_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫利中短债债券C","income":3}
{"index":{"_index":"fund","_id":"007195_2023-06-13"}}
{"date":"2023-06-13","name":"长城短债债券C","income":2}
{"index":{"_index":"fund","_id":"016240_2023-06-13"}}
{"date":"2023-06-13","name":"泰信添鑫中短债债券C","income":1}
{"index":{"_index":"fund","_id":"016319_2023-06-13"}}
{"date":"2023-06-13","name":"东方臻裕债券C","income":5}
{"index":{"_index":"fund","_id":"003860_2023-06-13"}}
{"date":"2023-06-13","name":"招商招旭纯债债券C","income":5}
{"index":{"_index":"fund","_id":"016527_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫诚短债债券C","income":3}
{"index":{"_index":"fund","_id":"016791_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫利中短债债券C","income":5}
POST /_bulk

{"index":{"_index":"fund","_id":"007195_2023-06-15"}}

{"date":"2023-06-15","name":"长城短债债券C","income":2}

{"index":{"_index":"fund","_id":"016240_2023-06-15"}}

{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}

{"index":{"_index":"fund","_id":"016319_2023-06-15"}}

{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}

{"index":{"_index":"fund","_id":"003860_2023-06-15"}}

{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}

{"index":{"_index":"fund","_id":"016527_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}

{"index":{"_index":"fund","_id":"016791_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}

{"index":{"_index":"fund","_id":"007195_2023-06-14"}}
{"date":"2023-06-14","name":"长城短债债券C","income":4}
{"index":{"_index":"fund","_id":"016240_2023-06-14"}}
{"date":"2023-06-14","name":"泰信添鑫中短债债券C","income":2}
{"index":{"_index":"fund","_id":"016319_2023-06-14"}}
{"date":"2023-06-14","name":"东方臻裕债券C","income":4}
{"index":{"_index":"fund","_id":"003860_2023-06-14"}}
{"date":"2023-06-14","name":"招商招旭纯债债券C","income":4}
{"index":{"_index":"fund","_id":"016527_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫诚短债债券C","income":2}
{"index":{"_index":"fund","_id":"016791_2023-06-14"}}
{"date":"2023-06-14","name":"招商鑫利中短债债券C","income":3}
{"index":{"_index":"fund","_id":"007195_2023-06-13"}}
{"date":"2023-06-13","name":"长城短债债券C","income":2}
{"index":{"_index":"fund","_id":"016240_2023-06-13"}}
{"date":"2023-06-13","name":"泰信添鑫中短债债券C","income":1}
{"index":{"_index":"fund","_id":"016319_2023-06-13"}}
{"date":"2023-06-13","name":"东方臻裕债券C","income":5}
{"index":{"_index":"fund","_id":"003860_2023-06-13"}}
{"date":"2023-06-13","name":"招商招旭纯债债券C","income":5}
{"index":{"_index":"fund","_id":"016527_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫诚短债债券C","income":3}
{"index":{"_index":"fund","_id":"016791_2023-06-13"}}
{"date":"2023-06-13","name":"招商鑫利中短债债券C","income":5}

生成数据的脚本，我是使用golang编写的，代码如下：

package code
import (
   "encoding/json"
   "fmt"
   "testing"
   "time"
)
func TestGenJsonForFundIndex(t *testing.T) {
   // 可以自定义日期
   date := ""
   if len(date) == 0 {
      date = time.Now().Format("2006-01-02")
   }
   // 填写对应基金的最新收益
   fundNameToIncome := map[string]int64{
      "长城短债债券C":    2,
      "泰信添鑫中短债债券C": 1,
      "东方臻裕债券C":    -2,
      "招商招旭纯债债券C":  -3,
      "招商鑫诚短债债券C":  -1,
      "招商鑫利中短债债券C": -2,
   }
   for _, fundName := range fundNames {
      fundCode := fundNameToCode[fundName]
      fundIncome := fundNameToIncome[fundName]
      i := indexOperation{
         Index: basic{
            Index: "fund",
            ID:    fmt.Sprintf("%+v_%+v", fundCode, date),
         },
      }
      f := fundMapping{
         Date:   date,
         Name:   fundName,
         Income: fundIncome,
      }
      iData, _ := json.Marshal(i)
      fData, _ := json.Marshal(f)
      fmt.Println(string(iData))
      fmt.Println(string(fData))
   }
}
var (
   fundNames = []string{
      "长城短债债券C",
      "泰信添鑫中短债债券C",
      "东方臻裕债券C",
      "招商招旭纯债债券C",
      "招商鑫诚短债债券C",
      "招商鑫利中短债债券C",
   }
   fundNameToCode = map[string]string{
      "长城短债债券C":    "007195",
      "泰信添鑫中短债债券C": "016240",
      "东方臻裕债券C":    "016319",
      "招商招旭纯债债券C":  "003860",
      "招商鑫诚短债债券C":  "016527",
      "招商鑫利中短债债券C": "016791",
   }
)
func newIndexOperation(index string, id string) indexOperation {
   return indexOperation{
      Index: basic{
         Index: index,
         ID:    id,
      },
   }
}
type indexOperation struct {
   Index basic `json:"index"`
}
type basic struct {
   Index string `json:"_index"`
   ID    string `json:"_id"`
}
// 基金
type fundMapping struct {
   Date   string `json:"date"`
   Name   string `json:"name"`
   Income int64  `json:"income"`
}
package code

import (
   "encoding/json"
   "fmt"
   "testing"
   "time"
)

func TestGenJsonForFundIndex(t *testing.T) {
   // 可以自定义日期
   date := ""
   if len(date) == 0 {
      date = time.Now().Format("2006-01-02")
   }
   // 填写对应基金的最新收益
   fundNameToIncome := map[string]int64{
      "长城短债债券C":    2,
      "泰信添鑫中短债债券C": 1,
      "东方臻裕债券C":    -2,
      "招商招旭纯债债券C":  -3,
      "招商鑫诚短债债券C":  -1,
      "招商鑫利中短债债券C": -2,
   }

   for _, fundName := range fundNames {
      fundCode := fundNameToCode[fundName]
      fundIncome := fundNameToIncome[fundName]
      i := indexOperation{
         Index: basic{
            Index: "fund",
            ID:    fmt.Sprintf("%+v_%+v", fundCode, date),
         },
      }
      f := fundMapping{
         Date:   date,
         Name:   fundName,
         Income: fundIncome,
      }
      iData, _ := json.Marshal(i)
      fData, _ := json.Marshal(f)
      fmt.Println(string(iData))
      fmt.Println(string(fData))
   }
}

var (
   fundNames = []string{
      "长城短债债券C",
      "泰信添鑫中短债债券C",
      "东方臻裕债券C",
      "招商招旭纯债债券C",
      "招商鑫诚短债债券C",
      "招商鑫利中短债债券C",
   }
   fundNameToCode = map[string]string{
      "长城短债债券C":    "007195",
      "泰信添鑫中短债债券C": "016240",
      "东方臻裕债券C":    "016319",
      "招商招旭纯债债券C":  "003860",
      "招商鑫诚短债债券C":  "016527",
      "招商鑫利中短债债券C": "016791",
   }
)

func newIndexOperation(index string, id string) indexOperation {
   return indexOperation{
      Index: basic{
         Index: index,
         ID:    id,
      },
   }
}

type indexOperation struct {
   Index basic `json:"index"`
}

type basic struct {
   Index string `json:"_index"`
   ID    string `json:"_id"`
}

// 基金
type fundMapping struct {
   Date   string `json:"date"`
   Name   string `json:"name"`
   Income int64  `json:"income"`
}
package code

import (
   "encoding/json"
   "fmt"
   "testing"
   "time"
)

func TestGenJsonForFundIndex(t *testing.T) {
   // 可以自定义日期
   date := ""
   if len(date) == 0 {
      date = time.Now().Format("2006-01-02")
   }
   // 填写对应基金的最新收益
   fundNameToIncome := map[string]int64{
      "长城短债债券C":    2,
      "泰信添鑫中短债债券C": 1,
      "东方臻裕债券C":    -2,
      "招商招旭纯债债券C":  -3,
      "招商鑫诚短债债券C":  -1,
      "招商鑫利中短债债券C": -2,
   }

   for _, fundName := range fundNames {
      fundCode := fundNameToCode[fundName]
      fundIncome := fundNameToIncome[fundName]
      i := indexOperation{
         Index: basic{
            Index: "fund",
            ID:    fmt.Sprintf("%+v_%+v", fundCode, date),
         },
      }
      f := fundMapping{
         Date:   date,
         Name:   fundName,
         Income: fundIncome,
      }
      iData, _ := json.Marshal(i)
      fData, _ := json.Marshal(f)
      fmt.Println(string(iData))
      fmt.Println(string(fData))
   }
}

var (
   fundNames = []string{
      "长城短债债券C",
      "泰信添鑫中短债债券C",
      "东方臻裕债券C",
      "招商招旭纯债债券C",
      "招商鑫诚短债债券C",
      "招商鑫利中短债债券C",
   }
   fundNameToCode = map[string]string{
      "长城短债债券C":    "007195",
      "泰信添鑫中短债债券C": "016240",
      "东方臻裕债券C":    "016319",
      "招商招旭纯债债券C":  "003860",
      "招商鑫诚短债债券C":  "016527",
      "招商鑫利中短债债券C": "016791",
   }
)

func newIndexOperation(index string, id string) indexOperation {
   return indexOperation{
      Index: basic{
         Index: index,
         ID:    id,
      },
   }
}

type indexOperation struct {
   Index basic `json:"index"`
}

type basic struct {
   Index string `json:"_index"`
   ID    string `json:"_id"`
}

// 基金
type fundMapping struct {
   Date   string `json:"date"`
   Name   string `json:"name"`
   Income int64  `json:"income"`
}

写入数据

写入数据到index，这里我们可以使用_bulk进行批量写入

打开Kibana，在侧边栏找到Management -> Dev Tools -> Console，进行写入数据，这里由于数据太多，我只演示了写入部分数据，读者在实际操作时，可以把上面提供的所有数据都进行写入，这样才方便我们继续进行分析

POST /_bulk
{"index":{"_index":"fund","_id":"007195_2023-06-15"}}
{"date":"2023-06-15","name":"长城短债债券C","income":2}
{"index":{"_index":"fund","_id":"016240_2023-06-15"}}
{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}
{"index":{"_index":"fund","_id":"016319_2023-06-15"}}
{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}
{"index":{"_index":"fund","_id":"003860_2023-06-15"}}
{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}
{"index":{"_index":"fund","_id":"016527_2023-06-15"}}
{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}
{"index":{"_index":"fund","_id":"016791_2023-06-15"}}
{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}
POST /_bulk

{"index":{"_index":"fund","_id":"007195_2023-06-15"}}

{"date":"2023-06-15","name":"长城短债债券C","income":2}

{"index":{"_index":"fund","_id":"016240_2023-06-15"}}

{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}

{"index":{"_index":"fund","_id":"016319_2023-06-15"}}

{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}

{"index":{"_index":"fund","_id":"003860_2023-06-15"}}

{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}

{"index":{"_index":"fund","_id":"016527_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}

{"index":{"_index":"fund","_id":"016791_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}
POST /_bulk

{"index":{"_index":"fund","_id":"007195_2023-06-15"}}

{"date":"2023-06-15","name":"长城短债债券C","income":2}

{"index":{"_index":"fund","_id":"016240_2023-06-15"}}

{"date":"2023-06-15","name":"泰信添鑫中短债债券C","income":1}

{"index":{"_index":"fund","_id":"016319_2023-06-15"}}

{"date":"2023-06-15","name":"东方臻裕债券C","income":-2}

{"index":{"_index":"fund","_id":"003860_2023-06-15"}}

{"date":"2023-06-15","name":"招商招旭纯债债券C","income":-3}

{"index":{"_index":"fund","_id":"016527_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫诚短债债券C","income":-1}

{"index":{"_index":"fund","_id":"016791_2023-06-15"}}

{"date":"2023-06-15","name":"招商鑫利中短债债券C","income":-2}

Elasticsearch常见聚合

Elasticsearch提供了非常强大的聚合能力，使得我们可以简单，快速的进行各种数据分析

常见的聚合主要分为两大类：

  • Bucket aggregations
  • Metrics aggregations

Bucket aggregations

桶聚合，简单理解就是把我们查询出来的文档，按照一些条件来进行过滤，如果文档满足过滤条件，则把文档放到对应的桶中

流程图-9.jpg

常见的Bucket aggregations：

  • date_histogram：日期直方图
  • terms：条件聚合
  • filter：单个过滤
  • filters：多个过滤

Metrics aggregations

指标聚合，简单理解就是对特定文档进行计算操作

常见的Metrics aggregations：

  • avg aggregation：平均值
  • max aggregation：最大值
  • min aggregation：最小值
  • sum aggregation：求和
  • value_count aggregation：计数
  • cardinality aggregation：去重计数

Kibana搭建监控看板/数据看板

使用Kibana搭建监控看板/数据看板，主要有以下几步：

  1. 创建数据源
  2. 搭建dashboards

创建数据源

打开Kibana，在侧边栏找到Management -> Stack Management，找到Kibana -> Index Patterns，点击Create index pattern按钮

  • Name：名称能匹配到上面创建的fund index即可，可以使用通配符*，右边可以查看到匹配到的index
  • Timestamp field：选择index中的时间字段，没有也可以不选择

进入到刚刚新建的Index Patterns

选择date类型的字段，打开Set format，自定义date类型展示格式为YYYY-MM-DD

如何查看是否配置成功？

在侧边栏找到Analytics -> Discover，选择对应的Index Patterns，如果有数据即为配置成功

搭建dashboards

在侧边栏找到Analytics -> Dashboard，点击Create dashboard按钮，点击All types，选择Aggregation based

这里对应了很多基础的图表，这里我们以Horizontal bar来演示，其它的图表类型都是类似操作

选择Horizontal bar，下一步需要我们选择数据源，我们选择第一步配置好的数据源fund

选择好数据源后，我们就跳转到了创建图表的页面

这个页面看起来很复杂，下面我以 创建债券基金历史收益排行看板 为例子详细讲解一下

首先，最上面的一栏，可以进行查询，过滤，以及指定日期范围，比如这里我们希望查看过去30天的数据，则可以选择对应的日期范围

选择Bucket aggregations，在Buckets中，我们点击Add按钮，选择X-axis

  • Aggregation：Terms，每一支基金作为一个桶
  • Field：name，依据基金name作为过滤条件
  • Order by：Metric：Count，作为排序的指标聚合，这个我们下一步进行设置
  • Order：Descending，降序排序
  • Size：10，最多展示10支基金
  • Custom label：自定义X轴展示名称，比如：债券基金

点击Update按钮，即可查看刷新图表

选择Metrics aggregations，在Metrics中，我们作如下调整

  • Aggregation：Sum，计算收益率的总和
  • Field：income，收益率
  • Custom label：自定义Y轴展示名称，比如：收益率

点击Update按钮，即可查看刷新图表

到这一步 债券基金历史收益排行看板 已经搭建好了，剩下的Metrics & axes，Panel settings都是对图表进行一些样式的改变，这里我就不一一介绍了，感兴趣的读者可以自己去探索一下

点击右上角的Save and return按钮，就会保存并返回到dashboard看板

实际效果如下

© 版权声明
文章版权归作者所有，未经允许请勿转载，侵权请联系 admin@trc20.tw 删除。
THE END

# Elasticsearch# Kibana
喜欢就支持一下吧
点赞0

Warning: mysqli_query(): (HY000/3): Error writing file '/tmp/MYbNtyiU' (Errcode: 28 - No space left on device) in /www/wwwroot/583.cn/wp-includes/class-wpdb.php on line 2345
admin的头像-五八三
相关推荐
评论 抢沙发
头像
欢迎您留下宝贵的见解！
提交
头像

昵称

图形验证码
取消
昵称

请填写用户信息：

  • 昵称(必填)
  • 邮箱(必填)
代码

请输入代码：

确认
图片
文章目录
文章目录
热门推荐