# paoding-rose-elasticsearch **Repository Path**: fusheng_zhang/paoding-rose-elasticsearch ## Basic Information - **Project Name**: paoding-rose-elasticsearch - **Description**: 简化es的操作语法，支持sql查询es。支持动态sql 等等，具体请查看文档 - **Primary Language**: Java - **License**: Apache-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 1 - **Created**: 2022-02-24 - **Last Updated**: 2025-04-03 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # es version ```json { "name": "node-1", "cluster_name": "elasticsearch", "cluster_uuid": "44_cwRu_T9--yqHRXMcuSQ", "version": { "number": "7.4.2", "build_flavor": "default", "build_type": "tar", "build_hash": "2f90bbf7b93631e52bafb59b3b049cb44ec25e96", "build_date": "2019-10-28T20:40:44.881551Z", "build_snapshot": false, "lucene_version": "8.2.0", "minimum_wire_compatibility_version": "6.8.0", "minimum_index_compatibility_version": "6.0.0-beta1" }, "tagline": "You Know, for Search" } ``` # start ## 引入依赖 ```xml cn.zhangfusheng paoding-rose-elasticsearch 1.0 ``` ```yaml spring: elasticsearch: uris: - http://192.168.18.200:9201 ``` ## 启动类添加注解[EnablePaodingRoseElasticSearch] * EnablePaodingRoseElasticSearch属性描述 * value/scanReposityPackages: 要扫描的 cn.zhangfusheng.elasticsearch.repository.ElasticSearchRepository 的实现类的包 * mybatisXmlScanPath: mybatis xml的扫描路径,默认为classpath*:mapper/**/*.xml * mode: 暂无作用 ```java package com.paoding.rose.elasticsearch.demo2; import cn.zhangfusheng.elasticsearch.annotation.EnablePaodingRoseElasticSearch; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; /** * @author fusheng.zhang * @date 2022-03-02 11:44:13 */ @SpringBootApplication @EnablePaodingRoseElasticSearch public class Demo2StartApplication { public static void main(String[] args) { SpringApplication.run(Demo2StartApplication.class, args); } } ``` # create index and mapping ## index 和 mapping的描述 * index描述 [IndexDiscription] * value:索引名称,如果为空,则使用类名的全小写格式 * version: 索引版本号,建议从1开始,依次增加 * upgradeVersion: 索引升级编号,同version一起控制索引升级/更新/数据迁移 * createMappingBefore: 创建mapping前执行 * createMappingEnd: 创建mapping后执行 * mapping字段描述 [FieldMapping] * primaryId: 每一个mapping必须配置 FieldMapping.primaryId=true的字段,且只能存在一个 * routing: 描述字段为routing标识 * ignore: 忽略该字段 * type: 字段类型,对应es文档的字段类型,如果未配置,则按照默认解析,详情查看 AnalysisMapping.getEsType * geoPoints: FieldType.Geo_Point类型的补充 * alias: FieldType.Alias类型的补充 * patameters: 字段属性配置,对应es文档https://www.elastic.co/guide/en/elasticsearch/reference/7.4/mapping-params.html * 测试demo TbCompany 包含了绝大部分的用户,具体使用可以参考 ```java package com.paoding.rose.elasticsearch.demo.model; import cn.zhangfusheng.elasticsearch.annotation.document.IndexDescription; import cn.zhangfusheng.elasticsearch.annotation.document.field.FieldMapping; import cn.zhangfusheng.elasticsearch.annotation.document.field.MappingParameters; import cn.zhangfusheng.elasticsearch.annotation.document.field.parameters.Alias; import cn.zhangfusheng.elasticsearch.annotation.document.field.parameters.GeoPoint; import cn.zhangfusheng.elasticsearch.annotation.document.field.parameters.IndexPrefixes; import cn.zhangfusheng.elasticsearch.constant.enumeration.FieldType; import cn.zhangfusheng.elasticsearch.util.date.enumeration.DateFormat; import com.paoding.rose.elasticsearch.demo.cycle.CreateCompanyMappingAround; import com.paoding.rose.elasticsearch.demo.model.vo.CompanyField; import com.paoding.rose.elasticsearch.demo.model.vo.CompanyUser; import com.paoding.rose.elasticsearch.demo.model.vo.DateRange; import com.paoding.rose.elasticsearch.demo.model.vo.IntegerRangeDetail; import com.paoding.rose.elasticsearch.demo.model.vo.TagsField; import lombok.Data; import lombok.experimental.Accessors; import java.time.LocalDateTime; import java.util.Date; import java.util.List; /** * @author fusheng.zhang * @date 2022-02-28 13:37:11 */ @Data @Accessors(chain = true) @IndexDescription( // index: tbcompany_v1 value = "tbcompany", // 索引名称,默认为全小写的类名 version = 1, // 索引版本号, upgradeVersion = "2022-05-10 20:14:30",// 索引更新(创建.升级.更新)编号 createMappingBefore = CreateCompanyMappingAround.class, // 创建索引前执行 createMappingEnd = CreateCompanyMappingAround.class // 创建索引后执行 ) public class TbCompany { /** * 主键,必须使用primaryId=true,标注字段为主键 */ @FieldMapping(primaryId = true) private String rowId; /** * 利用 patameters 配置文档字段的属性,
* 具体查看es文档:https://www.elastic.co/guide/en/elasticsearch/reference/7.4/mapping-params.html */ @FieldMapping(mappingParameters = @MappingParameters(index_prefixes = @IndexPrefixes)) private String createUserName; /** * analyzer.copy_to.fields 的 Mapping Patameters的使用 */ @FieldMapping(mappingParameters = { @MappingParameters(analyzer = "keyword", copy_to = {"searchName", "searchField"}, fields = CompanyField.class) }) private String companyName; @FieldMapping(mappingParameters = @MappingParameters(copy_to = {"searchName", "searchField"})) private String addressName; @FieldMapping(mappingParameters = @MappingParameters(analyzer = "whitespace")) private String searchName; @FieldMapping(mappingParameters = @MappingParameters(analyzer = "whitespace")) private String searchField; private Integer sort; @FieldMapping(routing = true) private Integer group; /** * 地理 geo_point 字段的配置 */ @FieldMapping(type = FieldType.Geo_Point, geoPoint = @GeoPoint) private String location; /** * arrays 类型字段的配置使用 */ @FieldMapping(type = FieldType.Arrays, mappingParameters = @MappingParameters(properties = CompanyUser.class)) private List companyUsers; /** * object 类型字段的使用 */ @FieldMapping(type = FieldType.Object, mappingParameters = @MappingParameters(properties = CompanyUser.class)) private CompanyUser companyUser; /** * Nested 字段类型 */ @FieldMapping(type = FieldType.Nested) private CompanyUser companyUserNested; /** * Alias 字段类型 */ @FieldMapping(type = FieldType.Alias, alias = @Alias(path = "companyUser.userName")) private String companyUserNameAlias; @FieldMapping(mappingParameters = @MappingParameters(fields = TagsField.class)) private List tags; /** * range 字段类型 */ @FieldMapping(type = FieldType.Integer_Range) private IntegerRangeDetail integerRangeDetail; /** * date_range 字段类型 */ @FieldMapping(type = FieldType.Date_Range, mappingParameters = @MappingParameters(format = "uuuu-MM-dd'T'HH:mm:ss.SSS'Z'||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis")) private DateRange dateRange; private LocalDateTime defaultTime; private Date defaultDate; /** * 时间格式化 */ @FieldMapping(mappingParameters = @MappingParameters(format = DateFormat.YYYY_MM_DD_HH_MM_SS)) private LocalDateTime updateTime; @FieldMapping(mappingParameters = @MappingParameters(format = DateFormat.YYYY_MM_DD_HH_MM_SS)) private Date createTime; } ``` ## 创建 index 和 mapping * ElasticSearchRepository<>,接口必须继承该接口 ```java public interface TbCompanyRepository extends ElasticSearchRepository { } ``` ## 启动项目 观察日志 * 项目首次启动时,会创建版本控制的index和mapping * index: transfer_v1 * mapping ```json lines { "properties": { "rowId": { "type": "keyword" }, "transfer": { "type": "boolean" }, "indexName": { "type": "text" }, "upIndexName": { "type": "text" }, "className": { "type": "keyword" }, "version": { "type": "integer" }, "upgradeVersion": { "type": "text" }, "transferVersion": { "type": "keyword" }, "desc": { "type": "text" }, "createTime": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" } } } ``` * 创建 Entity 对应的index和mapping [entity对应的 Repository 必须被使用(注入)] * indexName:tbcompany_v1,version:1 * mapping ```json lines { "properties": { "rowId": { "type": "keyword" }, "createUserName": { "type": "text", "index_prefixes": { "max_chars": 5, "min_chars": 2 } }, "companyName": { "type": "text", "analyzer": "keyword", "copy_to": [ "searchName", "searchField" ], "fields": { "one": { "type": "text", "analyzer": "whitespace" }, "two": { "type": "text", "analyzer": "whitespace" } } }, "addressName": { "type": "text", "copy_to": [ "searchName", "searchField" ] }, "searchName": { "type": "text", "analyzer": "whitespace" }, "searchField": { "type": "text", "analyzer": "whitespace" }, "sort": { "type": "integer" }, "group": { "type": "text" }, "location": { "type": "geo_point" }, "companyUsers": { "properties": { "userName": { "type": "text" }, "age": { "type": "integer" } } }, "companyUser": { "properties": { "userName": { "type": "text" }, "age": { "type": "integer" } } }, "companyUserNested": { "type": "nested" }, "companyUserNameAlias": { "type": "alias", "path": "companyUser.userName" }, "tags": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, "integerRangeDetail": { "type": "integer_range" }, "dateRange": { "type": "date_range", "format": "uuuu-MM-dd'T'HH:mm:ss.SSS'Z'||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis" }, "defaultTime": { "type": "date" }, "defaultDate": { "type": "date" }, "updateTime": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" }, "createTime": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" } } } ``` # ElasticSearchRepository ## 默认方法使用 | 方法名 | 描述 | 参数 | |-------------|-----------------|---------------------| | insert | 新建一条数据 | Repository对应的Entity | | updateById | 更新一条数据 | Repository对应的Entity | | deleteById | 删除一条数据 | 主键 | | findById | 根据主键查询 | 主键 | | findOne | 根据查询条件只能查询到一条数据 | 可以自定义查询条件 | | findAll | 查询全部数据 | 可以自定义查询条件 | | findForPage | 分页查询 | 可以自定义查询条件,已经分页参数 | ## 注解的使用 * DslSearch注解 : 可通过该注解控制解析的语句 * DslSortOrder注解: 排序配置 ## 个别类 * EsRange 范围数据查询使用 * GeoDistance 地图数据查询 # jpa/动态sql/mybatis 查询 ## jpa 查询 > 不需要注解标注,具体用法可查看 JpaCompanyRepository * 普通查询 * 支持jpa的方法名查询语法 * 自定义关键字查询 * 例如 模糊查询 like,es的模糊查询有 match/match_phrase 等,我们可以通过注解[JpaSearch]指定查询字段使用的关键字 * between and 查询 * 范围查询: 对应字段对应的参数必须是 [JpaBetween] * 分页查询 [详见分页查询] * 明确无查询条件的查询,建议使用ElasticSearchRepository.findForPage方法, * 查询全部数据 * 查询全部数据默认最多查询到 1W 条数据,可以通过注解ElasticSearchConfig.trackTotalHits=true,查询全部数据 * 明确无查询条件的查询,建议使用ElasticSearchRepository.findAll(); ## mybatis 查询 > DslWithMybatis注解标注 ### 参数名 #### 默认参数名的使用 > Repository方法的参数列表未配置参数别名时,mybatis的xml文件获取参数名的格式为 arg_(index),index为参数的顺序,从1开始 ``` @DslWithMybatis(id = "searchById", nameSpace = "com.paoding.rose.elasticsearch.demo.model.TbCompany") TbCompany searchById(String rowId); arg_1 对应参数的 rowId ``` ```es cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : dynamic dsl:{"bool": {"must": [ {"term": {"rowId": {"value": "wrzoSIdZQUmCuCnbyJDWdw"}}} ]}} cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : result dsl:{"bool": {"must": [ {"term": {"rowId": {"value": "wrzoSIdZQUmCuCnbyJDWdw"}}} ]}} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"size":10000,"query":{"wrapper":{"query":"eyJib29sIjogeyJtdXN0IjogWyB7InRlcm0iOiB7InJvd0lkIjogeyJ2YWx1ZSI6ICJ3cnpvU0lkWlFVbUN1Q25ieUpEV2R3In19fSBdfX0="}},"track_total_hits":-1} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"from":0,"size":1000,"query":{"wrapper":{"query":"eyJib29sIjogeyJtdXN0IjogWyB7InRlcm0iOiB7InJvd0lkIjogeyJ2YWx1ZSI6ICJ3cnpvU0lkWlFVbUN1Q25ieUpEV2R3In19fSBdfX0="}},"track_total_hits":-1} ``` #### 自定义参数名 > 使用 DslParams 注解自定义参数名 ``` @DslWithMybatis(id = "queryById", nameSpace = "com.paoding.rose.elasticsearch.demo.model.TbCompany") TbCompany queryById(@DslParams("rowId") String rowId); ``` ```es > rowId 不为空 cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : dynamic dsl:{"bool": {"must": [ {"term": {"rowId": {"value": "wrzoSIdZQUmCuCnbyJDWdw"}}} ]}} cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : result dsl:{"bool": {"must": [ {"term": {"rowId": {"value": "wrzoSIdZQUmCuCnbyJDWdw"}}} ]}} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"size":10000,"query":{"wrapper":{"query":"eyJib29sIjogeyJtdXN0IjogWyB7InRlcm0iOiB7InJvd0lkIjogeyJ2YWx1ZSI6ICJ3cnpvU0lkWlFVbUN1Q25ieUpEV2R3In19fSBdfX0="}},"track_total_hits":-1} > rowId 为空 cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : dynamic dsl:{"bool": {"must": [ ]}} cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : result dsl:{"bool": {"must": [ ]}} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"size":10000,"query":{"wrapper":{"query":"eyJib29sIjogeyJtdXN0IjogWyBdfX0="}},"track_total_hits":-1} ``` ### 符号(#)($)的注意事项 * 使用(#) > 使用(#)时,[#(arg_1)]的前后可以加也可以不加双引号 ``` # 假设第一个参数为 张三 xml:{"value":#{arg_1}},解析结果:{"value":"张三"} xml:{"value":"#{arg_1}"},解析结果:{"value":"张三"} ``` * 使用($) > 使用($)时,[$(arg_1)]的前后必须加双引号,如果数据类型为int doublue float 等,也可以不加双引号 ``` # 假设第一个参数为 张三 xml:{"value":${arg_1}},解析结果:{"value":张三} xml:{"value":"${arg_1}"},解析结果:{"value":"张三"} ``` * 使用(#)($)的demo和日志 ``` @DslWithMybatis(id = "searchByIdInOrId", nameSpace = "com.paoding.rose.elasticsearch.demo.model.TbCompany") List searchByIdInOrId(@DslParams("rowIds") List rowIds, @DslParams("rowId") String rowId); cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : dynamic dsl:{"bool": { "should": [ {"terms": {"rowId": [ "CQXY0OibSXqrz-zsHU39Xg" , "O-m-W07qTM-dlDmWbP84AQ" ]}} ,{"term": {"rowId": {"value": ?}}} ]}} cn.zhangfusheng.elasticsearch.template.TemplateMybatisApi : result dsl:{"bool": { "should": [ {"terms": {"rowId": [ "CQXY0OibSXqrz-zsHU39Xg" , "O-m-W07qTM-dlDmWbP84AQ" ]}} ,{"term": {"rowId": {"value": "wrzoSIdZQUmCuCnbyJDWdw"}}} ]}} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"size":10000,"query":{"wrapper":{"query":"eyJib29sIjogeyAic2hvdWxkIjogWyB7InRlcm1zIjogeyJyb3dJZCI6IFsgIkNRWFkwT2liU1hxcnotenNIVTM5WGciICwgIk8tbS1XMDdxVE0tZGxEbVdiUDg0QVEiIF19fSAseyJ0ZXJtIjogeyJyb3dJZCI6IHsidmFsdWUiOiAid3J6b1NJZFpRVW1DdUNuYnlKRFdkdyJ9fX0gXX19"}},"track_total_hits":-1} ``` ## 动态sql 查询 > 必须用 DslWithSql 进行标注,主要语法文档 https://github.com/iamazy/elasticsearch-sql * 参数列表获取规则 * 使用 DslParams 注解标定参数名,可通过(:参数名)取值 * 参数为实体: 可通过(:参数名.属性)取值 * 未标定参数名,参数名为 (:arg_[index]) index:为参数顺序,从1开始 * 冒号(:) * 表示这是一个变量 * $ 表达式 * 使用($)从常量内读取值 ```java import cn.zhangfusheng.elasticsearch.repository.ElasticSearchRepository; public interface Demo extends ElasticSearchRepository { static final String INDEX_NAME = "tb_company"; /** * 解析结果为 select * from tb_company * @return */ @DslWithSql("select * from $INDEX_NAME") List searchAll(); } ``` * (##)双#表达式 * 表示后面的变量作字符串连接 ```java import cn.zhangfusheng.elasticsearch.annotation.dsl.DslParams; public interface Demo extends ElasticSearchRepository { /** * 假设 tableName=company * 解析结果为 select * from tb_company * @param tableName * @return */ @DslWithSql("select * from tb_##(:tableName)") List searchAll(@DslParams("tableName") String tableName); } ``` * (#if) 表达式 ``` @DslWithSql(value = "select * from $INDEX_NAME", conditions = { @SqlCondition("#if(:rowIds != null){rowId in (:rowIds)}") }) List findByIdIn(@DslParams("rowIds") List rowId); cn.zhangfusheng.elasticsearch.template.TemplateDynamicSqlApi : result sql:select * from tb_company where rowId in ('CQXY0OibSXqrz-zsHU39Xg','O-m-W07qTM-dlDmWbP84AQ') cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"from":0,"size":10000,"query":{"terms":{"rowId":["CQXY0OibSXqrz-zsHU39Xg","O-m-W07qTM-dlDmWbP84AQ"],"boost":1.0}},"_source":{"includes":[],"excludes":[]},"track_total_hits":-1} cn.zhangfusheng.elasticsearch.template.ElasticSearchTemplateApi : index:[tbcompany_v1],routing:null,queryJson:{"from":0,"size":1000,"query":{"terms":{"rowId":["CQXY0OibSXqrz-zsHU39Xg","O-m-W07qTM-dlDmWbP84AQ"],"boost":1.0}},"_source":{"includes":[],"excludes":[]},"track_total_hits":-1} ``` * (#for)表达式 * 注: 可以通过:_index获取当前索引,从1开始 ```java import cn.zhangfusheng.elasticsearch.repository.ElasticSearchRepository; public interface Demo extends ElasticSearchRepository { @DslWithSql(value = "select * from $INDEX_NAME", conditions = { @SqlCondition("#for(tbCompany in :tbCompanies){#if(:_index == 1){rowId = :tbCompany.rowId}}") }) List testFor(@DslParams("tbCompanies") List tbCompanies); } ``` # 动态字符串解析 ## 动态sql解析 > 利用表达式表述条件,解析表达式生成最终的sql,并转成查询的dsl语句 ## 动态query dsl解析 > 同mybatis的xml语法类似 ## 动态script解析 # 特殊用法 ## 分页查询 > 方法添加类型为 PageRequest的参数, 且返回值为 PageResponse,自动识别为分页查询 * 注: * 默认只查询前1W条的分页数据,如果想获取超过1W的总条数,可添加注解ElasticSearchConfig.trackTotalHits=true,返回正确的总条数 * 前1W条数据可以正常的跳页查询,超过1W条数据后,可以通过PageRequest.searchAfter参数,继续向后查询.前1W条数据也可以使用searchAfter查询 * 支持超过1W的跳页查询逻辑 ## 多索引查询以及返回值配置 > 使用 DSLIndex 注解,配置value属性,执行要查询的索引对应的Entity,配置returnType属性,控制返回值 > 多个索引对应的多个类内相同的字段类型必须一致 ## 动态sql/mybatis的方法的参数特点 > 可通过 DslParams.value 配置参数的别名 * 动态sql * 未配置参数别名,参数的获取格式为 (:arg_[index]) index从1开始 * 配置参数别名,则格式为 (:别名) * mybatis * 为配置参数别名,参数的获取格式为 (${arg_[index]}) index从1开始 * 配置别名,则用别名获取 (${别名}) * 如果参数为Object,可以通过点(.)的方式获取 (${别名}.id) ## 索引升级 > 当文档发生变化时,索引需要进行升级操作.即IndexDiscription注解相关配置的调整. * mapping删除字段时 * 无需进行任何处理 * mapping增加字段时 * 调整 IndexDiscription.upgradeVersion,项目启动后会自动调整Mapping * 未调整IndexDiscription.upgradeVersion,在进行数据修改时,es会自动根据数据进行文档字段的配置(不建议) * mapping 调整某个字段类型 * 必须调整 IndexDiscription.version,生成新版本的index和mapping * 调整version后,系统会自动的进行数据迁移 ## 数据迁移 > 将数据从某个索引迁移至当前索引.数据迁移提供两种方式,请查看枚举类 TransferType * 当IndexDiscription.version发生变化时会自动触发数据迁移 * 通过调整 IndexTransfer.upgradeVersion的值触发数据迁移 ## requestOptions > 自定义 request options ```java import cn.zhangfusheng.elasticsearch.request.PaodingRequestOptions; import org.elasticsearch.client.RequestOptions; // 新建类 实现 PaodingRequestOptions public class ProjectRequestOptions implements PaodingRequestOptions { @Override public RequestOptions options() { // 返回自定义的 requestOptions return null; } @Override public String requestOptions() { // 返回对应的唯一标识 return null; } } // 注入到spring 容器内,使用时可通过注解 ElasticSearchConfig.requestOptions 指定唯一标识,进而使用对应的requestOptions ```