Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-842

"no known snapshots" error when DBs rows are large

    XMLWordPrintable

    Details

      Description

      On Debezium 0.8.1.Final, with postgres 9.6 / wal2json commit d2b7fef021c46e0d429f2c1768de361069e58696, we have encountered connector crashes when processing large db rows.

      For example, consider the following table:

      CREATE TABLE test_table (
          id integer PRIMARY KEY,
          x integer,
          data jsonb
      );
      

      Suppose we insert some db rows containing large blocks of json data:

      INSERT INTO test_table (id, x, data) VALUES
          (1, 1, '{"key": "really long value UPpLXMEESxlYEdtVVTBvfrkyvIQmVzXGiPnOasKCxWRIZjCvlgpsXMwhCEjfkSGkOFvtzNuTgELcqndZtwPUKkzYBGxWkvvxTSPBADprPhHmoYYXQWEIGkCtwDxbBYFatblIBbtQHFwUNAWXazzOkkpVxdTfeBgLeWTYUMbrIofOdUxmvXuoAZHsSEXQbCvoCUkYrNQIypfWoHlmfIaAaQeXuqxOhUWzzyGEZjxHMqiECOtIMLJoxMuSkrFlOXdZmiXXrUKPCVmaYMJNhNpNeSVxIkrxrIgTvTLtwIHmDpBjqKcgPFbfCiVQKQaGKMhcVGJkUMYwStsysZnmMYGbtWCxrfzTqsVGVooPUcMfODkODpgGBkTErgBFRedYGQTIANYGpqJnJqALhbFtOPUuSecWXCdBbKTaPTYsXLcfpFtJYSRdYRmgrcPeyiXVWosRdgHnRJdKdZUOthOLEDcZwRUcSkzVgmZIRCleLFkVvyZZPLjUpyWzldiYhIZFSkRRnzLjCrnNGDLzhoQoYOVWnBjjtNJMaylIVAdriAppJCGUQTblabqyTkRFotyBQFNusmQevROsMUybfpmLfokBwJMTFguwQQfqDFpEdMsQdHooJoiQRYggBlPkAkIooGLLKrUZmkJxwXCsVtkOsjirOkPHzGuaFmKufjKwYtmuBxpvmZroUsakLTyLLgLlFoeWFunyDgrZqQJYJIMFZTFkwsuBfzFInECbylsNlIEnzPIpizslyMWyIlikLQotIBVlNWEUKHXDGQNAOIBonotAtlyQuxyGxidIAhJCkexbfyxrcCoKfBvCQTvAmoHABumkuBAIqAuQMEdKLuuqkBMNkuYCvbshyENYBOmwsTaCkmAVhKWMonNeqSTgnECvJGysLgKZjXdFWysgPEhtIiFsOwSHWHJCMXgzNkmZfIlQTWRBlLWCnebDPYZdjJavMwAenDomZlRdeQVwfHfGdQrLrAAfpsEyoHWeGrgYzUXkwySJFgJyBEfVfbUGJJcHzlvvihzEtDjElisqaWKVTIsmtdATUCYfyzjDICqgNjzgwyYMUwgmmvPGoduAJSuInRGYXAPcpiRaAhLwxBWDjVhoGLVHRFYNPRDmNYoxAoMFbReeIvNVkDmCNSCJkCRfzuvrsVIcPSmiydwPDyCYKPmXIIoJCxtsGEqahKMwzTqNIjvyoKIEDQhftYYuPvsNuscBpZUllaeEraeYdLAlxEQbGSUyLbFNpdCNLkauXcfNadqFCBglRZTLlGFIrxaNXYEWebpahPhvdRkBNRUvkbYtYTcsYMtCfUaOqauoiTdUvYWHuDlIzBzJUEGdZVPgrwfhsBkCsJsEmiBYHWtvGwHgkDbFuauwPCOwybxggAIvCzMVEFhkHRxmXuUsqYFKjCmCdugKJuELMWjKVnYoVPIGGJCSyfzcHpSxVLKRYEyLBmEfZjLZfwOXaBLPcqItkrGLlTYrXOkpQrqRmMzdZJTQNarAvfUqeUWoiwwgeNhavbPxmaUQSwxFxkFHcPzHYrZtCURabLxlDMuJalHgXmFxWaakScrimOGImiDIDyWRazlXksqoPiPJXWQsvxZXFvbegDoFpWToBbQbYcjIQUIlJqbhjsGNwotkpOnHLnOdREJObBOKFIHIWrmkVvYtOPbqkwHkJISTxEeVGoEpYxCVZLxHYIeVZEvYKIRAphrEjjPLXhzjkNmpDTFmygPxjdnlsMewIEzkaInXJNVLydCKCxXPKNNrbMdNsFUCLOsTckPLfdqgRVySKiWAuLarrIiyInxkrEhwEaJOPqOViEciiddjKGZMXtrxwzhazwMOdQYQPtfaVxmfEUPSqerUavxybImhlqMbsGHvVSxsjHcjDtUsZOiFdYslxcPznexSuQclPzimsyVpRsBETNbEeMswKfBsAWIHROWDqqcWDCxMAACJIWsZiCGShNMvTMjckercodsKTetIsRlUkrshhySxkaYecWzzXvdaNisaMXikHgDQGUsDThPalhcZRxOUFgvYVQBBxAGOKDqiQZbELEfCjMlHpgWXxkPTpowGfuQBmkbkaMKLRPqoCifhrrSEnEXThgeSstayWjCDzObzopQHfZjHJyfnUTnGVphEMmKAoaLGvYWTuVuNhrDeHrEEzuFJRTXOcpoXoupQthUuxBZMDfleWNsbEQiOVJxMrInMKWTjENJjNPTjYDqwODnAeOwCEpjnPZMuXFhTsKUxODUyjGxwMzGAFSUsFnxNIXakyuwcEObgTjINISNxpDjZFRUIyFvTRQOgRisKfCyIzjHvavYGMXtAfLQBbqVrUqaSOPrnRzYEcikWUDOACfLeSfakDeXmvrvujXqNYTseNhgwDDROMXbYoWWEFqrOumFxbMrvpUAAwSGTZcvwhAGtxcasynOdXljvhNSLZcJseJNlkLXaZplPutdjYmzeKypqhnvvygHXclITfPFFgFQjsskwOKVPTVECERyZtiyRORBGXtsffoZznUdJKtLDpLStTQaRWWCIEfMblYUZWEwJTQEunzsaVaSPwPXImAGbQvKHAPyThtdsHtzISXIdjLGSTpfEiftZfMFFfYtuJDjOloUIvfCYYzokGdasZRRqywPrGgZZkqDqcUlcQhClvfLxxfhpOQLxNbIgYZTXLmqEbVyorAjjSRQjtediVObrcMPCvuPaLYjVfyetSkauaaPaIBBFhRDcpWSSgCmlZZslVLwLtpsFZsFGlekfjsiOuAeKskESEKDTHecktYWTiVAdOSulfYJkOhUTIrPcCdGlxRJTqtiCWjPTqGmwYiljfhWAlbQBrfpJDgRvIvWVHPtsbirJaOeHSAAgbxZMiuOjpFLsPIwMxRULRAMbtopZXhDAulWYcqBYASWoUagHyjODqyngsXaHDQkknNrWYiqspDqLqePQINLByFIDJfwUxhKIseYcWWgDoLmOJdYjnOXfVeGeUGQHubvbfickuQsfAWqxnMXZTVoTWEja"}'),
          (2, 1, '{"key": "really long value YAqvPiJOvePFCdZnExgTdHBBcergSanZurWvMuAKHkrlOQMbcahAZRbnhNSNDzPbzSUYOHPkkBekvGJSljwupfqvJYeDxFrrAQuWLWrmTbeJXwEDIXPsFsllOZiRhoZZfZkywitwXjjXbctpFmvdWSHGfhDKHXAVTaAmKHixUrsffAIEHyOhABNbactkOyaPAESfeBNjLrplctFzbFGyncGPCLEuRGLRvHPJTxoLOflcpAhNXmBpKRlUzFAoqwcJtEVUDsMSKegeoZDrwKAyalowYeKehFrBYVhQwUShEpQVUmhgMMhyBKXgPGUXcvbmbzzMXJCOdldKevErxATquWFRyAyRnPqJpgGTEYWGpGrbsytMPCLUQcqypdppbstKkkXNHiSTlivUHdUacTKZgYlZBlyVNuXQMlNpreNFSlSZQeyUPUJWPRrjXZcVoOWqccrXHevlYlTfvUKjSumqkeEdDJbnisSdfAFsbnmFqUUSopSMQYCmqTzlNbSGNIkBpPgQGpekMpjzQgVxLRcrIMxLszziUVlaHhjuHuPJrRFFySUxGibqJfNShvYRNOCDwduqigIabefDLulCBqQarndukzHpusvzRNaAdaOCtjTkdYNyxIPrvYkovSATrdrggwqqmtyygawARgaSCQJJbqpUNPgZynUdUDQUCqGkVeXQpFXHgAdoiOoqyksAarhWMLTFaBcuCNPvSKObFRGgGpGRaZxcteXEUspEkvcAFPnMnEQLAQBNjZdAtpbeDCMqnwCukjZhewVRxXBIGltuJOmguNFLmxpNbShcWyfHuQWTfikbJViXZowbPCmiDTJAtPWPuGwygeFqvbNHOydsBfvLLAZotkBgvprQVPJXqikUEngnKzeDmHLsLhWuHaSsvGArGHOMFOwwDbxUZnwAYRzwUhhVLWrXUrBJqSgYbYorWLVORokAJVkymvRUSYuBDfWgfSAMOncsNJpQhUnMofETXVAvcOudRmAumqVHxYdYgVhXbbYvMpkICHdwLoTnqNhmFuKHASmIDGkTdTvrBSTseOzBRirkSrYbRUHPImDqAWdxzHQBBpfhBBtYGEewoYTgKUGjawZHvPUOmFgbvkAJOWIGpEtqfMeiwgAsXEaKrAglnUiDGvSbvrPUCThSlvzNNhxiwHLXNIzpHdVOITySjgoTdTfVwuIEmATuJWKAOVuHCsauEcruUhenSeMQUohbMNpqrhIQqsLOyiMJFDPYeKkqYYfaYxwsoFpJjpzUSnWCRNyFAKXfCoTUILbtQXQdmxWrdNkelgImclanHcVwBTVqcZAloPwSpeXAZQbTxSmxixESUmtNgBxTRIMaxqUMoQWBYxCNmXumybtcPTjlevxeBlVxJNcoCfuHoFsKfaAbnCDjFQmHnmyvGQCSnGpxnMvvpyveTjlanbUaEoNQVzvGtirroPVpcFhIMcsWXdXjfyeqNxdPKqDeFUjTVcdPpIklQeEyypfHIkYzdZWqOdzYImiZCaAiDMHzjgDJeSLdugoGcDKrdRFsNGsYvTXtrDbAiVqqqyDodrJRehnxUjTRcdMHwONvUBMuszdHuCCOAiaiYGKcfDQWnSCHwItTGSfJuZBSINipyUNEjMxNQqLbBAozILkZyGmyaltHIFtiSfsfXuOsfhnsJhqXEKNZveVDTTJNJVVovhPmjDXrsQayJVKUOTxdjPkjblfehLleQveAWMAPMOQGrTMrhKNtJdRFYcpIOWsrXJdJQZEfqxBVtuzbfOUEvwChOxHJGViQIayngcoaaWQhNDjUvNdaNjyPEHbhfMThOsxcrEGgATMOmdZYahdmnbVNjBGVyzFZQGWOmiAsJAuwSzLmdNBXlriIRYcHBGSZahIjDMJdfEYQFKfVoyLflVdJwzMVXmeVqVZgaZgcehQTQyKJQiuFBKWRseBRGGTOFfTsfQotIvyOrUpAEeNtiucRmpMyUkoYaTlwJumciEiJlHMHRsmwcwgDbPxVLnGuhxjxFmhogidgoDFqkKhTPQxVYdpTPChuugbcDUwqjBOItoADBQppUkwhGEHebqFAbuPcdcSTyvWLCWUhgCmhUnCEbuURbLOrVcUsDYdLQdoflcssuNhRCROXFqAxRlaBsAtrPLkSliDvDkVNJqQaqUYrTAGocwVZpJYslwiwwovrZyKtCLyGHQDhGjZPVDWyLufRmKebrTnkEPmmbGxVjrCQZIHQfTKjHzZbxErRdlXBkEfJlPDEQSzVGWacLFcnsDdEIzGFzmiEmAjmQquCIUvYTdcamrcJzwYDkTxMiyiAfGOpPnxWOrUzayhrzEJaXIXWWNKLTNEBcfdQjsMdenIHjqLhEhmQBWBsSDAuoIdOsjUikxqdRVSyjJcjHoXEsssrLTbTlCuCvMXaDJUdWkjKfvLRuFmdPFbolxrJSzWdYTtTHoLSWtVLjDIMZDXRtwDvlJYXRPCyUijGKBpBStaeGwPUZOVjdOFGdFJrOJwAXLlKdRTVGghpGJvxNlNvxQkejnHJKOyAMuiSkfPCHVQiumnKgguRXxruSuRHHGSWoStPLrORvsHcWQGXPOoLwfpgnUJaNnzkJEWMKwkfdYQCckzAjyxzLChdupSvgkldxxhPRJZEqsjpIztYNEBJWTJzeNkaCARWxtYhrwwGUysarjCjojQsubLTgSsLKzGYyYSwGooUYrhNvkuLCturXnVHFJnaHIUtFehaSRznlrDYXfuECSUvITJXzsOBqpDzkWSBnzrwQpaxWbMsSHPUzNZqRHGdzNXsPzNKkEsYjFeFdTolAAwmzVNpKWigDQhIQXtpDhHCPonSXbIsquykxguffcmsOWbUPGMTHteNUmLOGEnCgWctZaIDmXKrgPcQgqyeuUFpuJSflOwsPSCBVqSWyeNALjfbYPHWxYJPzBkTYPHVdhiBZGqZpPnNMSdSnXMNDQxsVuxescdIdbKiHxfCZeqjmUenSQpZbloFvYYbKiTEdwJrvXrHVmxv"}');
      

      Now, start the connector.

      After the initial snapshot completes, do some updates on the db rows:

      UPDATE test_table
      SET x = 2;
      UPDATE test_table
      SET x = 3;
      

      The connector crashes:

      2018-07-27 19:54:09,619 ERROR  Postgres|my_test_database|records-stream-producer  unexpected exception while streaming logical changes   [io.debezium.connector.postgresql.RecordsStreamProducer]
      org.postgresql.util.PSQLException: ERROR: no known snapshots
        Where: slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
              at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
              at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
              at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
              at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
              at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
              at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
              at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:198)
              at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
              at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      Errors are also visible in the db server logs:

      2018-07-27T19:54:09.616547179Z ERROR:  no known snapshots
      2018-07-27T19:54:09.616587252Z CONTEXT:  slot "debezium", output plugin "wal2json", in the change callback, associated LSN 0/16D3850
      

      This is a rather contrived example, but we have also encountered this problem with other seemingly harmless rows in our database. We suspect that this is related to wal2json's handling of TOAST data. (The long strings I used in my example also happen to be highly incompressible.)

      wal2json versions prior to ce82d7387a806bc8bacaeed80e0a0f1840733dce omitted unchanged TOAST columns from the change event. Newer versions include all changes, but this may have introduced a regression in Debezium, or in the wal2json plugin itself.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  jpechanec Jiri Pechanec
                  Reporter:
                  csang Christopher Sang
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: