---

❯ How to add table to postgres connector debezium cdc stream?

Here's how to add a new table to an existing Debezium PostgreSQL connector without disrupting the running CDC stream:

1. Grant replication access to the new table

-- Set replica identity so UPDATE/DELETE events carry full row data
ALTER TABLE my_schema.new_table REPLICA IDENTITY FULL;

2. Add the table to the Postgres publication

If you're using the pgoutput plugin (the default/recommended one), Debezium reads from a publication, and tables not in it simply won't emit changes — this step is the one people most often forget.

ALTER PUBLICATION dbz_publication ADD TABLE my_schema.new_table;

(Skip this if publication.autocreate.mode=all_tables, since the publication already covers every table.)

3. Update the connector's table.include.list

Patch the connector config via the Kafka Connect REST API:

curl -X PUT -H "Content-Type: application/json" \
  http://localhost:8083/connectors/my-postgres-connector/config \
  -d '{
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    ...existing config...,
    "table.include.list": "my_schema.existing_table,my_schema.new_table"
  }'

Updating the config causes Kafka Connect to restart the connector task automatically — no need to manually stop/start it.

4. Get the existing data into the topic (snapshot)

Restarting the connector alone will not backfill the new table's existing rows — Debezium's snapshot.mode (e.g. initial, no_data) only controls what happens on the connector's very first startup, not on later config changes. For an already-running connector, use an incremental (ad-hoc) snapshot instead:

1. Make sure you have a signal table set up (signal.data.collection config) — if you don't already have one, create it:
CREATE TABLE debezium_signal (
  id VARCHAR(42) PRIMARY KEY,
  type VARCHAR(32) NOT NULL,
  data VARCHAR(2048) NULL
);
(and add it to the publication too, and set signal.data.collection in the connector config if not already set).

2. Trigger the snapshot for just the new table:
INSERT INTO debezium_signal (id, type, data)
VALUES ('snap-new-table-1', 'execute-snapshot',
  '{"data-collections": ["my_schema.new_table"], "type": "incremental"}');

This streams the existing rows as READ events without stopping ongoing CDC for other tables.

Order matters

Do step 1–2 (Postgres side) before step 3 (connector config), so no events are missed in the gap. If you skip the publication step, the connector will accept the config change silently but never emit any events for the new table — a very common silent failure.

---

Want me to tailor this to your actual connector config (e.g. what plugin — pgoutput vs decoderbufs, and whether you're on Debezium Server, Kafka Connect, or the embedded engine)?

✻ Brewed for 17s · done 7:13 AM

❯ what if i cannot create signal table in the source database?

If you don't have DDL permissions on the source database (or policy forbids adding new tables there), Debezium supports other signal channels that don't require a signal table at all:

Option 1: Kafka signal channel (most common workaround)

Send the ad-hoc snapshot signal as a message to a dedicated Kafka topic instead of inserting into a Postgres table.

Connector config:
{
  "signal.enabled.channels": "source,kafka",
  "signal.kafka.topic": "my-connector.signals",
  "signal.kafka.bootstrap.servers": "kafka:9092",
  "signal.kafka.poll.timeout.ms": "100"
}
(You can drop source from signal.enabled.channels entirely if you have no signal table at all.)

Then produce a message to that topic (key = connector name):

echo '{"id":"snap-new-table-1","type":"execute-snapshot","data":{"data-collections":["my_schema.new_table"],"type":"incremental"}}' | \
kafka-console-producer.sh --broker-list kafka:9092 --topic my-connector.signals --property "parse.key=false"

This requires no schema changes on the Postgres side — just Kafka topic access, which you almost always have as the connector owner.

Option 2: JMX signal channel

If you have JMX access to the Connect worker, you can invoke the signal as an MBean operation:

{ "signal.enabled.channels": "jmx" }

Then call the signal operation on the debezium.<connector-type>:type=management,context=signals,server=<name> MBean with the same JSON payload. Useful in locked-down environments where you can't produce Kafka messages either but do have management access.

Option 3: File signal channel

{ "signal.enabled.channels": "file", "signal.file": "/path/to/signals.jsonl" }
Debezium polls the file for new lines and appends processed signals. Works well for Debezium Server / single-node deployments where you control the filesystem.

Fallback: skip incremental snapshot entirely

If none of the above is workable and the new table is small, it's often simpler to just accept a full re-snapshot: temporarily set snapshot.mode to a value that forces snapshotting on next start (e.g. initial_only/always depending on connector version) after adding the table to table.include.list, restart the connector, then revert the mode. This re-reads all tables in the include list though, not just the new one, so it's only reasonable for small datasets or low-traffic windows.

Recommendation: go with the Kafka signal channel — it needs no source DB changes, works the same way operationally as the table-based signal, and is the officially supported alternative for exactly this scenario.

✻ Cooked for 21s · done 7:15 AM

❯ I think incremental signal requires source table

You're right to push back — let me verify this precisely rather than go from memory.

● Web Search("Debezium incremental snapshot Kafka signal channel requires signal.data.collection watermark")
Did 1 search in 6s
Allowed by auto mode classifier

● Web Search("Debezium "signal.data.collection" required incremental snapshot even with kafka signal channel")
Did 1 search in 7s
Allowed by auto mode classifier

● Fetch(https://debezium.io/documentation/reference/stable/configuration/signalling.html)
Received 0 bytes (403 Forbidden)
Allowed by auto mode classifier

● Web Search("Debezium Postgres connector "read-only incremental snapshot" replication slot without signal table")
Did 1 search in 6s
Allowed by auto mode classifier

● Web Search(""incremental.snapshot.watermarking.strategy" debezium postgres insert_delete")
Did 1 search in 7s
Allowed by auto mode classifier

You're correct, and I gave you wrong information — thanks for pushing back. I verified against the official docs: for the PostgreSQL connector specifically, incremental (ad-hoc) snapshots require the source signal channel to be enabled, because the watermarking algorithm works by writing low/high watermark rows into the signal table itself so they show up in the WAL stream Debezium is decoding. The Kafka/JMX/file channels can be used to trigger the signal, but the watermark writes still land in a Postgres table — so signal.data.collection pointing at a real table in the source database is mandatory for Postgres, not optional. There's no GTID-based "read-only incremental snapshot" mode for Postgres the way MySQL has, since Postgres has no GTID equivalent. My earlier claim that Kafka/JMX/file channels let you skip the source table entirely was wrong for this connector.

Given that, here are the actual options when you can't create a signal table:

Option 1: Ask for a narrower permission grant

You don't need broad DDL rights — just CREATE TABLE on one small table, ideally in a schema you already own or a dedicated debezium schema, not the application schema. This is a much smaller ask than what people usually assume "can't create tables" blocks, and it's the path of least resistance if there's any room to negotiate with the DBA.

Option 2: Manual consistent backfill using an exported replication snapshot

If truly no table can ever be created, you can replicate what Debezium's snapshot does internally, using only your existing replication privileges:

-- Creates a slot and returns a snapshot name valid until the slot's tx ends
SELECT * FROM pg_create_logical_replication_slot('tmp_backfill_slot', 'pgoutput');
Then in a separate session, before the slot's creating transaction commits:
BEGIN;
SET TRANSACTION SNAPSHOT '<snapshot_name_from_above>';
SELECT * FROM my_schema.new_table; -- bulk-export this, consistent as of slot creation LSN
COMMIT;
Load that export into Kafka yourself (batch job → producer, or a one-off JDBC source connector run). Then drop the temp slot, add my_schema.new_table to table.include.list on the real connector (with the publication updated as before), and restart it. Because the connector starts streaming from its current LSN — after the point your export snapshot was taken — you get no gap and no duplication, without ever touching table.include.list-based full re-snapshot or creating anything in the source schema. This only requires the REPLICATION privilege you already have for Debezium to work at all.

Option 3: Accept a full re-snapshot

Simplest but coarsest: bump snapshot.mode to force a fresh snapshot of the whole table.include.list on next restart. Fine for small datasets or a maintenance window, wasteful otherwise.

Sources:
- Sending signals to a Debezium connector
- Debezium connector for PostgreSQL
- Incremental Snapshots in Debezium

✻ Baked for 1m 31s · done 7:18 AM

❯ Maybe i can set type of download new table to not incremental in signal body?

● Web Search("Debezium ad-hoc "blocking" snapshot signal type execute-snapshot no signal table required")
Did 1 search in 6s

● Fetch(https://deepwiki.com/debezium/debezium/4.4-signaling-system)
Received 945.3KB (200 OK)

● Web Search("Debezium "ad hoc blocking snapshot" postgresql documentation "execute-snapshot" type blocking data-collections example")
Did 1 search in 6s

● Fetch(https://docs.redhat.com/en/documentation/red_hat_build_of_debezium/2.3.4/html/debezium_user_guide/debezium-connector-for-postgresql)
Received 0 bytes (403 Forbidden)

● Web Search("debezium jira "blocking snapshot" "signal.data.collection" not required kafka channel")
Did 1 search in 6s

● Web Search("debezium google groups "blocking snapshot" kafka signal channel without signal table postgres")
Did 1 search in 7s

Yes — that's actually the right move, and it sidesteps the signal-table requirement cleanly. Confirmed against real-world reports (a Postgres user running exactly this via the file channel, no signal table): a blocking snapshot pauses streaming and does a single consistent read, the same way the initial snapshot does — so there's no interleaving with concurrent WAL activity, which means there's nothing to watermark. That's precisely why incremental snapshots need writes into a source table and blocking snapshots don't.

Setup (no signal table anywhere)

{
  "signal.enabled.channels": "kafka",
  "signal.kafka.topic": "my-connector.signals",
  "signal.kafka.bootstrap.servers": "kafka:9092"
}
No signal.data.collection needed at all.

Trigger it

{"id":"backfill-new-table-1","type":"execute-snapshot","data":{"data-collections":["my_schema.new_table"],"type":"blocking"}}
Produce that as a message to my-connector.signals.

Trade-offs vs. incremental

- Streaming pauses for the whole connector while the blocking snapshot runs — CDC events for every table being captured queue up until it finishes, not just the new table. Fine for a small/quick table, more disruptive for a large one.
- Not chunked — it reads the table in one pass like an initial snapshot, so it holds a consistent-snapshot transaction open for the whole duration. For a very large table this can mean longer transactions and more resource pressure than incremental's chunked approach would.
- One known rough edge from real reports: if you fire the signal immediately after a connector restart, it can race the streaming phase starting up (worse with the file channel, but worth knowing generally) — safest to confirm the connector is actually streaming (check its status/metrics) before sending the signal, and retry if it's a no-op.

Full sequence for your case

1. ALTER TABLE my_schema.new_table REPLICA IDENTITY FULL;
2. ALTER PUBLICATION dbz_publication ADD TABLE my_schema.new_table;
3. Update table.include.list to include the new table, let the connector restart (this alone streams new changes going forward, but doesn't backfill existing rows).
4. Send the execute-snapshot / blocking signal above via the Kafka topic to backfill the existing rows.

This gets you the same end result as the incremental approach, with the one real cost being a brief pause of the entire stream instead of the new table doing its snapshot in the background — a reasonable trade if you genuinely can't get a signal table created.

Sources:
- Sending signals to a Debezium connector
- Debezium Engine - Blocking snapshots not producing data when no transactions present (Postgres)
- Debezium signaling and notifications - Part 1

✻ Churned for 1m 20s · done 7:22 AM