Developer builds lightweight Kafka-free CDC pipeline in Go to stream Postgres changes
A developer has released Rift, an open-source Go binary that reads PostgreSQL Write-Ahead Log (WAL) changes and forwards them to destinations such as webhooks, Redis, or another Postgres database without requiring Kafka, Zookeeper, or Debezium. The tool uses logical replication via pglogrepl and pgx, supports optional JavaScript-based transformations through goja, and queues events to disk using BoltDB when a destination is unavailable. Rift also tracks schema changes by installing an event trigger in the source database that logs DDL operations like ALTER TABLE. During development, the author encountered a bug where inserts into the internal rift_ddl_log table were themselves treated as replication events, causing repeated sync errors against destination databases that lacked the table. The fix involved a helper function that filters out internal tables by checking both exact and suffix matches to handle inconsistent schema-qualified table names from the WAL decoder.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in