Elasticsearch搜索一、索引生命周期管理
遇到问题描述
第一次上线需要使用kibana或者curl建索引以及为索引关联模板,增加运维人员操作的复杂度。
使用Elasticsearch官方提供的Java High Level REST Client 创建
policy
和绑定template
代码编写比较复杂,并且修改是不容易。使用Java Low Level REST Client ,使用
classpath:elastic/info-policy.json
和classpath:elastic/info-template.json
文件作为请求体,发送request
请求创建滚动规则并绑定索引模板。
收获
1. 读取classpath文件
final String filePath = ResourceUtils.getURL("classpath:elastic/info-policy.json").getFile(); File file = new File(filePath); if (!file.exists()) { log.info("policy file [{}] is not exit ?", filePath); return; } final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8")); putPolicyRequest.setEntity(new NStringEntity(String.join("", content), 复制代码
2. Elasticsearch提供两种方式的API
Java Low Level REST Client 允许用HTTP协议与ES集群通信,是面向用户的。低级客户端的特效包括:
最小化依赖
所有节点(nodes)的负载均衡
某个节点发生故障会自动转移到其他节点
对于失败的链接给予乘法,也就是失败次数越多,client下次尝试重新链接的时间就越长,持续练级
跟踪(tracing)请求和相应的记录
可以自动发现集群中的节点(cluster nodes)
Java high Level REST Client 基于低级客户端,面向开发者,开发者可以用管饭提供的各种API实现自己的功能
Java High Level Client 运行在Java Low Level REST 客户端上,面向开发人员,高级客户端提供API的具体方法,这些方法以请求对象(Request Object)为输入,返回响应对象(Response Object),每个API的方法都可以用同步或者异步的方式调用,异步方法返回的对象需要用监听器(listener)来接受。
Java High Level Client 依赖于ES核心工程(core project),因此与ES原生的Java API中的TransportClient有相同的输入和输出。由于8.0版本后,官方不在支持TransportClient,因此推荐把现有的用TransportClient写的代码移植到REST Client。目前,Java REST Client API中已经实现了原生API中的大部分方法,开发者可以在官方的github issue上提交暂时还未实现的方法。当然这些方法都可以用JSON请求Low Level REST Client手动实现。
3. 调整索引滚动策略刷新时间,默认为10m
PUT _cluster/settings { "transient": { "indices.lifecycle.poll_interval":"1s" } } 复制代码
动手实践
1. 建立spring-boot工程,使用最新依赖
<parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.6.2</version> <relativePath/> <!-- lookup parent from repository --> </parent> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.16</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies> 复制代码
2. application.yml配置elasticsearch
spring: elasticsearch: connection-timeout: 5s uris: http://localhost:9200 复制代码
3. 使用RestClient发送请求
通过RestHighLevelClient拿到RestClient客户端
RestClient restClient; @Autowired RestHighLevelClient restHighLevelClient; @Autowired private ElasticsearchOperations elasticsearchOperations; @BeforeEach public void setUp() { restClient = restHighLevelClient.getLowLevelClient(); } @AfterEach public void tearDown() { try { restClient.close(); } catch (IOException e) { e.printStackTrace(); } } 复制代码
创建
policy
检查
policy
是否存在,存在就直接返回如果
policy
不存在,读取classpath:elastic/info-policy.json
文件做为请求体,发送创建policy
的请求
@Test public void test_createPolicy() throws Exception { String policyName = "_ilm/policy/info-policy"; Request policyExitsRequest = new Request("GET", policyName); boolean policyExits = false; try { Response response = restClient.performRequest(policyExitsRequest); if (response.getStatusLine().getStatusCode() == 200) { log.info("policy is already exits"); policyExits = true; } } catch (ResponseException responseException) { log.info("policy is not exits [{}]", responseException.getMessage()); } if (policyExits) { return; } Request putPolicyRequest = new Request("PUT", policyName); final String filePath = ResourceUtils.getURL("classpath:elastic/info-policy.json").getFile(); File file = new File(filePath); if (!file.exists()) { log.info("policy file [{}] is not exit ?", filePath); return; } final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8")); putPolicyRequest.setEntity(new NStringEntity(String.join("", content), ContentType.APPLICATION_JSON)); Response putTemplateResponse = restClient.performRequest(putPolicyRequest); if (putTemplateResponse.getStatusLine().getStatusCode() == 200) { log.info("create policy success"); return; } } 复制代码
@Test public void test_createTemplate() throws Exception { String templateName = "_index_template/info-template"; Request templateExitsRequest = new Request("GET", templateName); boolean templateExits = false; try { Response response = restClient.performRequest(templateExitsRequest); if (response.getStatusLine().getStatusCode() == 200) { log.info("template is already exits"); templateExits = true; } } catch (ResponseException responseException) { log.info("template is not exits [{}]", responseException.getMessage()); } if (templateExits) { return; } Request putTemplateRequest = new Request("PUT", templateName); final String filePath = ResourceUtils.getURL("classpath:elastic/info-template.json").getFile(); File file = new File(filePath); if (!file.exists()) { log.info("template file [{}] is not exit ?", filePath); return; } final List<String> content = Files.readAllLines(file.toPath(), Charset.forName("UTF-8")); putTemplateRequest.setEntity(new NStringEntity(String.join("", content), ContentType.APPLICATION_JSON)); Response putTemplateResponse = restClient.performRequest(putTemplateRequest); if (putTemplateResponse.getStatusLine().getStatusCode() == 200) { log.info("create template success"); return; } } 复制代码
3. 使用ElasticsearchOperations写入数据到Elasticsearch,查看索引滚动策略是否生效
step1. 需要将索引滚动策略刷新时间调小一点,API
PUT _cluster/settings { "transient": { "indices.lifecycle.poll_interval":"1s" } } 复制代码
step2. 使用程序模拟数据
@Autowired private ElasticsearchOperations elasticsearchOperations; @BeforeEach public void setUp() { restClient = restHighLevelClient.getLowLevelClient(); } @AfterEach public void tearDown() { try { restClient.close(); } catch (IOException e) { e.printStackTrace(); } } @Test public void test_save() throws Exception { for (int i = 0; i < 1000; i++) { PersonDocument personDocument = new PersonDocument(); personDocument.setName("name-" + i); personDocument.setId(i); elasticsearchOperations.save(personDocument); TimeUnit.SECONDS.sleep(2); } }
作者:百炼000
链接:https://juejin.cn/post/7048187673910984741