Summary
This is Part 1 in a three-part series on a multi-cluster build of Elasticsearch (ES) with cross-cluster replication and search enablement. This post covers the build of the West cluster which is implemented in Kubernetes.
- Part 1: West Cluster Build
- Part 2: East Cluster Build
- Part 3: Replication + Search
Architecture
The West cluster is implemented as an Elastic Cloud on Kubernetes (ECK). I'm using Kind for the Kubernetes environment. This allows for a self-contained environment suitable for a capable laptop. Additionally, I use cloud-provider-kind to provide native load-balancer functionality.
Configuration
Kind/Cloud-Provider-Kind
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| kind create cluster | |
| docker run -d --rm --name cloud-provider-kind --network kind \ | |
| -v /var/run/docker.sock:/var/run/docker.sock registry.k8s.io/cloud-provider-kind/cloud-controller-manager:v0.6.0 |
ECK Operator
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| helm repo add elastic https://helm.elastic.co | |
| helm repo update elastic | |
| helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace |
Elasticsearch + Kibana
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| kind: Secret | |
| type: Opaque | |
| metadata: | |
| name: westcluster-es-elastic-user | |
| data: | |
| elastic: ZWxhc3RpYw== | |
| --- | |
| apiVersion: v1 | |
| kind: Secret | |
| metadata: | |
| name: eck-trial-license | |
| namespace: elastic-system | |
| labels: | |
| license.k8s.elastic.co/type: enterprise_trial | |
| annotations: | |
| elastic.co/eula: accepted | |
| --- | |
| apiVersion: elasticsearch.k8s.elastic.co/v1 | |
| kind: Elasticsearch | |
| metadata: | |
| name: westcluster | |
| spec: | |
| version: 8.17.2 | |
| nodeSets: | |
| - name: default | |
| count: 1 | |
| config: | |
| node.store.allow_mmap: false | |
| http: | |
| service: | |
| spec: | |
| type: LoadBalancer | |
| transport: | |
| service: | |
| spec: | |
| type: LoadBalancer |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: kibana.k8s.elastic.co/v1 | |
| kind: Kibana | |
| metadata: | |
| name: kibana | |
| spec: | |
| version: 8.17.2 | |
| count: 1 | |
| elasticsearchRef: | |
| name: westcluster | |
| http: | |
| service: | |
| spec: | |
| type: LoadBalancer |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| kubectl get secret westcluster-es-transport-certs-public -o jsonpath='{.data.ca\.crt}' | base64 --decode > west-ca.crt | |
| kubectl get secret westcluster-es-http-certs-public -o jsonpath='{.data.ca\.crt}' | base64 --decode > west-http-ca.crt | |
| WEST_ELASTIC_IP=$(kubectl get service westcluster-es-http -o=jsonpath='{.status.loadBalancer.ingress[0].ip}') | |
| WEST_KIBANA_IP=$(kubectl get service kibana-kb-http -o=jsonpath='{.status.loadBalancer.ingress[0].ip}') |
Indices
Two minimal indices are created with the REST API. These indices will used in a later post on cross-cluster replication and search.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| echo -e "\n*** Create west_ccr index ***" | |
| curl -s -k -u "elastic:elastic" "https://$WEST_ELASTIC_IP:9200/_bulk?pretty" \ | |
| -H "Content-Type: application/json" \ | |
| -d' | |
| { "index" : { "_index" : "west_ccr" } } | |
| {"name": "Snow Crash", "author": "Neal Stephenson", "release_date": "1992-06-01", "page_count": 470} | |
| { "index" : { "_index" : "west_ccr" } } | |
| {"name": "Revelation Space", "author": "Alastair Reynolds", "release_date": "2000-03-15", "page_count": 585} | |
| ' > /dev/null | |
| echo -e "\n*** Create west_ccs index ***" | |
| curl -s -k -u "elastic:elastic" "https://$WEST_ELASTIC_IP:9200/_bulk?pretty" \ | |
| -H "Content-Type: application/json" \ | |
| -d' | |
| { "index" : { "_index" : "west_ccs" } } | |
| {"name": "1984", "author": "George Orwell", "release_date": "1985-06-01", "page_count": 328} | |
| { "index" : { "_index" : "west_ccs" } } | |
| {"name": "Fahrenheit 451", "author": "Ray Bradbury", "release_date": "1953-10-15", "page_count": 227} | |
| ' > /dev/null |
