Tech Tips: Change Data Capture w/Redis Enterprise

Summary

Redis Enterprise has the capability for continuous data integration with 3rd party data sources. This capability is enabled via the Redis Data Integration (RDI) product. With RDI, change data capture (CDC) can be achieved with all the major SQL databases for ingress. Similarly, in the other direction, updates to Redis can be continuously written to 3rd party targets via the write-behind functionality of RDI.

This post covers a demo-grade environment of Redis Enterprise + RDI with ingress and write-behind integrations with the following SQL databases: Oracle, MS SQL, Postgres, and MySQL. All components are containerized and run from a Docker environment.

Architecture

Ingress

Write-behind

Code Snippets

Docker Compose - Redis Enterprise Node

	#RE Cluster - Node 1
	re1:
	image: redislabs/redis:latest
	container_name: re1
	restart: unless-stopped
	tty: true
	cap_add:
	- sys_resource
	ports:
	- 12000
	- 8443
	- 9443
	- 8070
	profiles: ["mysql", "postgres", "sqlserver", "oracle_lm", "oracle_xs"]
	networks:
	re_cluster:
	ipv4_address: 192.168.20.2

view raw rdi-re.yml hosted with ❤ by GitHub

Docker Compose - Oracle Enterprise

	oracle_lm:
	image: container-registry.oracle.com/database/enterprise:latest
	container_name: oracle_lm
	ports:
	- 1521
	depends_on:
	- re1
	- re2
	- re3
	environment:
	- ORACLE_SID=ORCLCDB
	- ORACLE_EDITION=enterprise
	- ORACLE_PWD=Password1
	- INIT_SGA_SIZE=1024
	- INIT_PGA_SIZE=1024
	volumes:
	- $PWD/conf/oracle_lm/scripts:/opt/oracle/scripts/startup
	profiles: ["oracle_lm"]
	networks:
	- re_cluster

view raw rdi-oracleent.yml hosted with ❤ by GitHub

Docker Compose - Debezium

	debezium:
	build:
	context: $PWD/conf/debezium
	args:
	INSTANT_CLIENT: $INSTANT_CLIENT
	container_name: debezium
	volumes:
	- $PWD/conf/$SOURCE_DB/application.properties:/debezium/conf/application.properties
	profiles: ["debezium"]
	networks:
	- re_cluster

view raw rdi-debezium.yml hosted with ❤ by GitHub

RDI Ingress w/Prometheus Integration

	echo "* Build Redis DI DB for Ingress *"
	./redis-di create --silent --cluster-host 192.168.20.2 --cluster-api-port 9443 --cluster-user redis@redis.com \
	--cluster-password redis --rdi-port 13000 --rdi-password redis

	echo "* Deploy Redis DI for Ingress *"
	./redis-di deploy --dir ./conf/$SOURCE_DB/ingest --rdi-host 192.168.20.3 --rdi-port 13000 --rdi-password redis

	echo "* Start Debezium *"
	SOURCE_DB=$SOURCE_DB INSTANT_CLIENT=$INSTANT_CLIENT docker compose --profile debezium up -d

	echo "* Start Redis DI Monitor *"
	./redis-di monitor --rdi-host 192.168.20.3 --rdi-port 13000 --rdi-password redis &

view raw rdi-rdiingress.sh hosted with ❤ by GitHub

Source

https://github.com/Redislabs-Solution-Architects/redisdi-docker

Monday, January 15, 2024

Change Data Capture w/Redis Enterprise