Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-638

wal2json cannot handle transactions bigger than 1Gb

    XMLWordPrintable

Details

    Description

      error stack trace –

      org.postgresql.util.PSQLException: ERROR: out of memory
      274663- Detail: Cannot enlarge string buffer containing 1073741685 bytes by 203 more bytes.
      274749- Where: slot "connect_prod_job_transaction", output plugin "wal2json", in the change callback, associated LSN A257/9069C888
      274874- at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
      274969- at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
      275062- at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
      275148- at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
      275223- at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
      275332- at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
      275438- at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
      275535- at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:189)
      275660- at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
      275765- at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
      275871- at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      275947- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      276008- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      276092- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      276176- at java.lang.Thread.run(Thread.java:748)

      please refer - https://github.com/eulerto/wal2json/issues/6

      please look at – https://github.com/eulerto/wal2json/blob/master/wal2json.c#L118

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            khushalvirmani Khushal Virmani (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: