Loading...

Details

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 0.9.0.Beta2
Affects Version/s: 0.8.1.Final
Component/s: postgresql-connector
Labels:
None

SFDC Cases Counter:
SFDC Cases Links:

Description

On Debezium 0.8.1.Final, with postgres 9.6 / wal2json commit d2b7fef021c46e0d429f2c1768de361069e58696, we have encountered connector crashes when processing large db rows.

For example, consider the following table:

CREATE TABLE test_table (
    id integer PRIMARY KEY,
    x integer,
    data jsonb
);

Suppose we insert some db rows containing large blocks of json data:

INSERT INTO test_table (id, x, data) VALUES
    (1, 1, '{"key": "really long value UPpLXMEESxlYEdtVVTBvfrkyvIQmVzXGiPnOasKCxWRIZjCvlgpsXMwhCEjfkSGkOFvtzNuTgELcqndZtwPUKkzYBGxWkvvxTSPBADprPhHmoYYXQWEIGkCtwDxbBYFatblIBbtQHFwUNAWXazzOkkpVxdTfeBgLeWTYUMbrIofOdUxmvXuoAZHsSEXQbCvoCUkYrNQIypfWoHlmfIaAaQeXuqxOhUWzzyGEZjxHMqiECOtIMLJoxMuSkrFlOXdZmiXXrUKPCVmaYMJNhNpNeSVxIkrxrIgTvTLtwIHmDpBjqKcgPFbfCiVQKQaGKMhcVGJkUMYwStsysZnmMYGbtWCxrfzTqsVGVooPUcMfODkODpgGBkTErgBFRedYGQTIANYGpqJnJqALhbFtOPUuSecWXCdBbKTaPTYsXLcfpFtJYSRdYRmgrcPeyiXVWosRdgHnRJdKdZUOthOLEDcZwRUcSkzVgmZIRCleLFkVvyZZPLjUpyWzldiYhIZFSkRRnzLjCrnNGDLzhoQoYOVWnBjjtNJMaylIVAdriAppJCGUQTblabqyTkRFotyBQFNusmQevROsMUybfpmLfokBwJMTFguwQQfqDFpEdMsQdHooJoiQRYggBlPkAkIooGLLKrUZmkJxwXCsVtkOsjirOkPHzGuaFmKufjKwYtmuBxpvmZroUsakLTyLLgLlFoeWFunyDgrZqQJYJIMFZTFkwsuBfzFInECbylsNlIEnzPIpizslyMWyIlikLQotIBVlNWEUKHXDGQNAOIBonotAtlyQuxyGxidIAhJCkexbfyxrcCoKfBvCQTvAmoHABumkuBAIqAuQMEdKLuuqkBMNkuYCvbshyENYBOmwsTaCkmAVhKWMonNeqSTgnECvJGysLgKZjXdFWysgPEhtIiFsOwSHWHJCMXgzNkmZfIlQTWRBlLWCnebDPYZdjJavMwAenDomZlRdeQVwfHfGdQrLrAAfpsEyoHWeGrgYzUXkwySJFgJyBEfVfbUGJJcHzlvvihzEtDjElisqaWKVTIsmtdATUCYfyzjDICqgNjzgwyYMUwgmmvPGoduAJSuInRGYXAPcpiRaAhLwxBWDjVhoGLVHRFYNPRDmNYoxAoMFbReeIvNVkDmCNSCJkCRfzuvrsVIcPSmiydwPDyCYKPmXIIoJCxtsGEqahKMwzTqNIjvyoKIEDQhftYYuPvsNuscBpZUllaeEraeYdLAlxEQbGSUyLbFNpdCNLkauXcfNadqFCBglRZTLlGFIrxaNXYEWebpahPhvdRkBNRUvkbYtYTcsYMtCfUaOqauoiTdUvYWHuDlIzBzJUEGdZVPgrwfhsBkCsJsEmiBYHWtvGwHgkDbFuauwPCOwybxggAIvCzMVEFhkHRxmXuUsqYFKjCmCdugKJuELMWjKVnYoVPIGGJCSyfzcHpSxVLKRYEyLBmEfZjLZfwOXaBLPcqItkrGLlTYrXOkpQrqRmMzdZJTQNarAvfUqeUWoiwwgeNhavbPxmaUQSwxFxkFHcPzHYrZtCURabLxlDMuJalHgXmFxWaakScrimOGImiDIDyWRazlXksqoPiPJXWQsvxZXFvbegDoFpWToBbQbYcjIQUIlJqbhjsGNwotkpOnHLnOdREJObBOKFIHIWrmkVvYtOPbqkwHkJISTxEeVGoEpYxCVZLxHYIeVZEvYKIRAphrEjjPLXhzjkNmpDTFmygPxjdnlsMewIEzkaInXJNVLydCKCxXPKNNrbMdNsFUCLOsTckPLfdqgRVySKiWAuLarrIiyInxkrEhwEaJOPqOViEciiddjKGZMXtrxwzhazwMOdQYQPtfaVxmfEUPSqerUavxybImhlqMbsGHvVSxsjHcjDtUsZOiFdYslxcPznexSuQclPzimsyVpRsBETNbEeMswKfBsAWIHROWDqqcWDCxMAACJIWsZiCGShNMvTMjckercodsKTetIsRlUkrshhySxkaYecWzzXvdaNisaMXikHgDQGUsDThPalhcZRxOUFgvYVQBBxAGOKDqiQZbELEfCjMlHpgWXxkPTpowGfuQBmkbkaMKLRPqoCifhrrSEnEXThgeSstayWjCDzObzopQHfZjHJyfnUTnGVphEMmKAoaLGvYWTuVuNhrDeHrEEzuFJRTXOcpoXoupQthUuxBZMDfleWNsbEQiOVJxMrInMKWTjENJjNPTjYDqwODnAeOwCEpjnPZMuXFhTsKUxODUyjGxwMzGAFSUsFnxNIXakyuwcEObgTjINISNxpDjZFRUIyFvTRQOgRisKfCyIzjHvavYGMXtAfLQBbqVrUqaSOPrnRzYEcikWUDOACfLeSfakDeXmvrvujXqNYTseNhgwDDROMXbYoWWEFqrOumFxbMrvpUAAwSGTZcvwhAGtxcasynOdXljvhNSLZcJseJNlkLXaZplPutdjYmzeKypqhnvvygHXclITfPFFgFQjsskwOKVPTVECERyZtiyRORBGXtsffoZznUdJKtLDpLStTQaRWWCIEfMblYUZWEwJTQEunzsaVaSPwPXImAGbQvKHAPyThtdsHtzISXIdjLGSTpfEiftZfMFFfYtuJDjOloUIvfCYYzokGdasZRRqywPrGgZZkqDqcUlcQhClvfLxxfhpOQLxNbIgYZTXLmqEbVyorAjjSRQjtediVObrcMPCvuPaLYjVfyetSkauaaPaIBBFhRDcpWSSgCmlZZslVLwLtpsFZsFGlekfjsiOuAeKskESEKDTHecktYWTiVAdOSulfYJkOhUTIrPcCdGlxRJTqtiCWjPTqGmwYiljfhWAlbQBrfpJDgRvIvWVHPtsbirJaOeHSAAgbxZMiuOjpFLsPIwMxRULRAMbtopZXhDAulWYcqBYASWoUagHyjODqyngsXaHDQkknNrWYiqspDqLqePQINLByFIDJfwUxhKIseYcWWgDoLmOJdYjnOXfVeGeUGQHubvbfickuQsfAWqxnMXZTVoTWEja"}'),
    (2, 1, '{"key": "really long value YAqvPiJOvePFCdZnExgTdHBBcergSanZurWvMuAKHkrlOQMbcahAZRbnhNSNDzPbzSUYOHPkkBekvGJSljwupfqvJYeDxFrrAQuWLWrmTbeJXwEDIXPsFsllOZiRhoZZfZkywitwXjjXbctpFmvdWSHGfhDKHXAVTaAmKHixUrsffAIEHyOhABNbactkOyaPAESfeBNjLrplctFzbFGyncGPCLEuRGLRvHPJTxoLOflcpAhNXmBpKRlUzFAoqwcJtEVUDsMSKegeoZDrwKAyalowYeKehFrBYVhQwUShEpQVUmhgMMhyBKXgPGUXcvbmbzzMXJCOdldKevErxATquWFRyAyRnPqJpgGTEYWGpGrbsytMPCLUQcqypdppbstKkkXNHiSTlivUHdUacTKZgYlZBlyVNuXQMlNpreNFSlSZQeyUPUJWPRrjXZcVoOWqccrXHevlYlTfvUKjSumqkeEdDJbnisSdfAFsbnmFqUUSopSMQYCmqTzlNbSGNIkBpPgQGpekMpjzQgVxLRcrIMxLszziUVlaHhjuHuPJrRFFySUxGibqJfNShvYRNOCDwduqigIabefDLulCBqQarndukzHpusvzRNaAdaOCtjTkdYNyxIPrvYkovSATrdrggwqqmtyygawARgaSCQJJbqpUNPgZynUdUDQUCqGkVeXQpFXHgAdoiOoqyksAarhWMLTFaBcuCNPvSKObFRGgGpGRaZxcteXEUspEkvcAFPnMnEQLAQBNjZdAtpbeDCMqnwCukjZhewVRxXBIGltuJOmguNFLmxpNbShcWyfHuQWTfikbJViXZowbPCmiDTJAtPWPuGwygeFqvbNHOydsBfvLLAZotkBgvprQVPJXqikUEngnKzeDmHLsLhWuHaSsvGArGHOMFOwwDbxUZnwAYRzwUhhVLWrXUrBJqSgYbYorWLVORokAJVkymvRUSYuBDfWgfSAMOncsNJpQhUnMofETXVAvcOudRmAumqVHxYdYgVhXbbYvMpkICHdwLoTnqNhmFuKHASmIDGkTdTvrBSTseOzBRirkSrYbRUHPImDqAWdxzHQBBpfhBBtYGEewoYTgKUGjawZHvPUOmFgbvkAJOWIGpEtqfMeiwgAsXEaKrAglnUiDGvSbvrPUCThSlvzNNhxiwHLXNIzpHdVOITySjgoTdTfVwuIEmATuJWKAOVuHCsauEcruUhenSeMQUohbMNpqrhIQqsLOyiMJFDPYeKkqYYfaYxwsoFpJjpzUSnWCRNyFAKXfCoTUILbtQXQdmxWrdNkelgImclanHcVwBTVqcZAloPwSpeXAZQbTxSmxixESUmtNgBxTRIMaxqUMoQWBYxCNmXumybtcPTjlevxeBlVxJNcoCfuHoFsKfaAbnCDjFQmHnmyvGQCSnGpxnMvvpyveTjlanbUaEoNQVzvGtirroPVpcFhIMcsWXdXjfyeqNxdPKqDeFUjTVcdPpIklQeEyypfHIkYzdZWqOdzYImiZCaAiDMHzjgDJeSLdugoGcDKrdRFsNGsYvTXtrDbAiVqqqyDodrJRehnxUjTRcdMHwONvUBMuszdHuCCOAiaiYGKcfDQWnSCHwItTGSfJuZBSINipyUNEjMxNQqLbBAozILkZyGmyaltHIFtiSfsfXuOsfhnsJhqXEKNZveVDTTJNJVVovhPmjDXrsQayJVKUOTxdjPkjblfehLleQveAWMAPMOQGrTMrhKNtJdRFYcpIOWsrXJdJQZEfqxBVtuzbfOUEvwChOxHJGViQIayngcoaaWQhNDjUvNdaNjyPEHbhfMThOsxcrEGgATMOmdZYahdmnbVNjBGVyzFZQGWOmiAsJAuwSzLmdNBXlriIRYcHBGSZahIjDMJdfEYQFKfVoyLflVdJwzMVXmeVqVZgaZgcehQTQyKJQiuFBKWRseBRGGTOFfTsfQotIvyOrUpAEeNtiucRmpMyUkoYaTlwJumciEiJlHMHRsmwcwgDbPxVLnGuhxjxFmhogidgoDFqkKhTPQxVYdpTPChuugbcDUwqjBOItoADBQppUkwhGEHebqFAbuPcdcSTyvWLCWUhgCmhUnCEbuURbLOrVcUsDYdLQdoflcssuNhRCROXFqAxRlaBsAtrPLkSliDvDkVNJqQaqUYrTAGocwVZpJYslwiwwovrZyKtCLyGHQDhGjZPVDWyLufRmKebrTnkEPmmbGxVjrCQZIHQfTKjHzZbxErRdlXBkEfJlPDEQSzVGWacLFcnsDdEIzGFzmiEmAjmQquCIUvYTdcamrcJzwYDkTxMiyiAfGOpPnxWOrUzayhrzEJaXIXWWNKLTNEBcfdQjsMdenIHjqLhEhmQBWBsSDAuoIdOsjUikxqdRVSyjJcjHoXEsssrLTbTlCuCvMXaDJUdWkjKfvLRuFmdPFbolxrJSzWdYTtTHoLSWtVLjDIMZDXRtwDvlJYXRPCyUijGKBpBStaeGwPUZOVjdOFGdFJrOJwAXLlKdRTVGghpGJvxNlNvxQkejnHJKOyAMuiSkfPCHVQiumnKgguRXxruSuRHHGSWoStPLrORvsHcWQGXPOoLwfpgnUJaNnzkJEWMKwkfdYQCckzAjyxzLChdupSvgkldxxhPRJZEqsjpIztYNEBJWTJzeNkaCARWxtYhrwwGUysarjCjojQsubLTgSsLKzGYyYSwGooUYrhNvkuLCturXnVHFJnaHIUtFehaSRznlrDYXfuECSUvITJXzsOBqpDzkWSBnzrwQpaxWbMsSHPUzNZqRHGdzNXsPzNKkEsYjFeFdTolAAwmzVNpKWigDQhIQXtpDhHCPonSXbIsquykxguffcmsOWbUPGMTHteNUmLOGEnCgWctZaIDmXKrgPcQgqyeuUFpuJSflOwsPSCBVqSWyeNALjfbYPHWxYJPzBkTYPHVdhiBZGqZpPnNMSdSnXMNDQxsVuxescdIdbKiHxfCZeqjmUenSQpZbloFvYYbKiTEdwJrvXrHVmxv"}');

Now, start the connector.

After the initial snapshot completes, do some updates on the db rows:

UPDATE test_table
SET x = 2;
UPDATE test_table
SET x = 3;

The connector crashes:

2018-07-27 19:54:09,619 ERROR  Postgres|my_test_database|records-stream-producer  unexpected exception while streaming logical changes   [io.debezium.connector.postgresql.RecordsStreamProducer]
org.postgresql.util.PSQLException: ERROR: no known snapshots
  Where: slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
        at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
        at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
        at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
        at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
        at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:198)
        at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
        at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Errors are also visible in the db server logs:

2018-07-27T19:54:09.616547179Z ERROR:  no known snapshots
2018-07-27T19:54:09.616587252Z CONTEXT:  slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850

This is a rather contrived example, but we have also encountered this problem with other seemingly harmless rows in our database. We suspect that this is related to wal2json's handling of TOAST data. (The long strings I used in my example also happen to be highly incompressible.)

wal2json versions prior to ce82d7387a806bc8bacaeed80e0a0f1840733dce omitted unchanged TOAST columns from the change event. Newer versions include all changes, but this may have introduced a regression in Debezium, or in the wal2json plugin itself.

Attachments

Issue Links

causes

DBZ-1083 [debezium] ERROR: option \"include-unchanged-toast\" = \"0\" is unknown

Closed

is related to

DBZ-911 RecordsStreamProducer#columnValues() does not take into account unchanged TOASTed columns, refreshing table schemas unnecessarily

Closed

relates to

DBZ-911 RecordsStreamProducer#columnValues() does not take into account unchanged TOASTed columns, refreshing table schemas unnecessarily

Closed

"no known snapshots" error when DBs rows are large

Details

Description

Attachments

Issue Links

Activity

People

Dates