Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-638

wal2json cannot handle transactions bigger than 1Gb

    Details

      Description

      error stack trace –

      org.postgresql.util.PSQLException: ERROR: out of memory
      274663- Detail: Cannot enlarge string buffer containing 1073741685 bytes by 203 more bytes.
      274749- Where: slot "connect_prod_job_transaction", output plugin "wal2json", in the change callback, associated LSN A257/9069C888
      274874- at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
      274969- at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1043)
      275062- at org.postgresql.core.v3.QueryExecutorImpl.readFromCopy(QueryExecutorImpl.java:962)
      275148- at org.postgresql.core.v3.CopyDualImpl.readFromCopy(CopyDualImpl.java:41)
      275223- at org.postgresql.core.v3.replication.V3PGReplicationStream.receiveNextData(V3PGReplicationStream.java:145)
      275332- at org.postgresql.core.v3.replication.V3PGReplicationStream.readInternal(V3PGReplicationStream.java:114)
      275438- at org.postgresql.core.v3.replication.V3PGReplicationStream.read(V3PGReplicationStream.java:60)
      275535- at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.read(PostgresReplicationConnection.java:189)
      275660- at io.debezium.connector.postgresql.RecordsStreamProducer.streamChanges(RecordsStreamProducer.java:119)
      275765- at io.debezium.connector.postgresql.RecordsStreamProducer.lambda$start$0(RecordsStreamProducer.java:105)
      275871- at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      275947- at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      276008- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      276092- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      276176- at java.lang.Thread.run(Thread.java:748)

      please refer - https://github.com/eulerto/wal2json/issues/6

      please look at – https://github.com/eulerto/wal2json/blob/master/wal2json.c#L118

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                khushalvirmani Khushal Virmani
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: