Elastic Cross-Cluster Ops - Replication + Search

Summary

This is the final post in a three-part series configuring Elasticsearch (ES) cross-cluster replication and search.  The first two posts set up two distinct ES clusters: one implemented in Kubernetes (ECK), and the other in Docker.

• Part 1: West Cluster Build
• Part 2: East Cluster Build
• Part 3: Replication + Search

Cross-cluster Replication

Architecture

Configuration

Networking

The two clusters (West and East) are both implemented in Docker.  Although West is a K8s implementation, the underlying architecture of Kind is in fact, Docker.  Each cluster is in its own Docker network.  For cross-cluster operations to function, we need to configure the linkage between these two Docker networks.  The Docker commands below do just that.

Remote Cluster Configuration

The West Cluster needs to be added to the East as a remote cluster.  The commands below do that and then wait for the remote configuration to complete.

East Follower Index

In this scenario, we are setting up a leader index on the West cluster (named west_ccr) and its corresponding follower index on the East (named east_ccr).  This will allow one-way replication from West to East.

Demo

At this point, true replication of the West index (west_ccr), including mappings (schema) has been accomplished.  This can be verified with a simple Nodejs Elasticsearch client application that is located in the src/javascript directory.

Cross-cluster Search

Architecture

Configuration

Remote Cluster Configuration

Similar to the prior exercise of establishing the West cluster as a remote cluster on the East, the East cluster now needs to be configured as a remote cluster on the West.

Demo

At this point, everything is in place to execute queries that span the two clusters.  A Python Elasticsearch client script is included in the src/python directory.  That script executes a search against the West ES endpoint that spans both the west_ccs index on West and the east_ccs on East.

Source

https://github.com/joeywhelan/elastic-crosscluster

Elastic Cross-Cluster Ops - East Cluster Build

Summary

This is Part 2 in the three-part series on Elasticsearch (ES) cross-cluster operations.  This ES cluster will be implemented in Docker.  Like the West Cluster, this configuration yields a single ES node and one Kibana node. 

• Part 1: West Cluster Build
• Part 2: East Cluster Build
• Part 3: Replication + Search

Architecture

Configuration

Docker-compose

For the most part, I used the reference docker-compose.  I added a bind mount such that I could add the West Cluster's transport CA to the trusted CAs for the East Cluster.

Index

Below a minimal index is built via the REST API.  This index is used in a demonstration of cross-cluster search in the next post.

Source

https://github.com/joeywhelan/elastic-crosscluster

Elastic Cross-Cluster Ops - West Cluster Build

Summary

This is Part 1 in a three-part series on a multi-cluster Elasticsearch (ES) build with cross-cluster replication and search enablement.  This post covers the build of the West cluster, which is implemented in Kubernetes.

• Part 1: West Cluster Build
• Part 2: East Cluster Build
• Part 3: Replication + Search

Architecture

The West cluster is implemented as an Elastic Cloud on Kubernetes (ECK). I'm using Kind for the Kubernetes environment, which allows for a self-contained environment suitable for a capable laptop. Additionally, I use cloud-provider-kind to provide native load-balancer functionality.

Configuration

Kind/Cloud-Provider-Kind

ECK Operator

Elasticsearch + Kibana

Indices

Two minimal indices are created with the REST API. These indices will used in a later post on cross-cluster replication and search.

Source

https://github.com/joeywhelan/elastic-crosscluster

Geospatial Search with Redis and Apache Pinot

Summary

I'll discuss Redis Enterprise and Apache Pinot setup in Docker for this post. 

• 3-node Redis environment
• 4-node Pinot environment
• 1M synthetic JSON records representing a user's geographic location
• Equivalent geospatial search queries for both environments

Architecture

Data Generation

Synthetic records are created with a Python function that utilizes the Faker library. They are saved to a file called locations.json.  A snippet of the data generator is below.

Redis Deployment

Docker

Three Redis nodes are created.

Cluster Build, DB creation, Index Build, Data Load

Nodes are joined into a Redis cluster. A single-shard database is then created, along with an index. Finally, the database is populated from a JSON file with Riot.

Redis Insight

Pinot Deployment

Docker

A 4-node Pinot cluster is created in docker-compose.  Pinot Controller definition is below.

Schema/Table Build, Data Load

Pinot Console

Queries

The query below finds the count of users within a polygon defined as the boundaries of the State of Colorado.

Redis

Pinot

Source

https://github.com/joeywhelan/redis-pinot-geo

Gloo Gateway on Kind

Summary

In this post, I'll cover a demo deployment of Gloo Gateway in a Kind K8s environment.

• 3-worker node Kind K8s environment
• 2 dummy REST microservices implemented in Nodejs via Express.js
• OSS deployment of Gloo Gateway providing ingress to the two microservices

Architecture

Deployment

Kind

Gloo Gateway

Microservices

Source

https://github.com/joeywhelan/gloo-kind

Redis RAG with Nvidia NeMoGuardrails

Summary

This post will cover the usage of guardrails in the context of an RAG application using Redis Stack as the vector store.  

• Nvidia's guardrail package is used for the railed implementation.
• Langchain LCEL is used for the non-railed implementation.
• Content from the online Redis vector search documentation is used for the RAG content
• GUI is implemented with Chainlit

Application Architecture

This bot is operating within a Chainlit app.  It has two modes of operation:  

• 'chain' - no guardrails
• 'rails' - NeMo guardrails in place for both user inputs and LLM outputs

Screenshots

Bot without rails

This first screenshot shows the bot operating with no guardrails.  It does just fine until an off-topic question is posed - then it cheerfully deviates from its purpose.

Bot with rails

Same series of questions here with guardrails enabled.  Note that it keeps the user on topic now.

Code Snippets

Non-railed chain (LCEL)

Railed with NeMO Guardrails

Source

https://github.com/redis-developer/llm-redisrail

Copyright  ©2024 Joey E Whelan, All rights reserved.

Change Data Capture w/Redis Enterprise

Summary

Redis Enterprise has the capability for continuous data integration with 3rd party data sources.  This capability is enabled via the Redis Data Integration (RDI) product.  With RDI, change data capture (CDC) can be achieved with all the major SQL databases for ingress.  Similarly, in the other direction, updates to Redis can be continuously written to 3rd party targets via the write-behind functionality of RDI.  

This post covers a demo-grade environment of Redis Enterprise + RDI with ingress and write-behind integrations with the following SQL databases:  Oracle, MS SQL, Postgres, and MySQL.  All components are containerized and run from a Docker environment.

Architecture

Ingress

Write-behind

Code Snippets

Docker Compose - Redis Enterprise Node

Docker Compose - Oracle Enterprise

Docker Compose - Debezium

RDI Ingress w/Prometheus Integration

Source

https://github.com/Redislabs-Solution-Architects/redisdi-docker

Copyright ©1993-2024 Joey E Whelan, All rights reserved.