Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1311

Reading structure of captured tables is too slow

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • 0.9.5.Final
    • oracle-connector
    • None

    Description

      Hi,

      I have 3594 tables in my database schema:

      SELECT count(*)
FROM all_tables
WHERE owner='my_schema';
      COUNT(*)
      3594

      and I want to stream only one table.
      Snapshot step 5 takes several hours and falls after connector rebalancing:

      [2019-05-29T10:17:43,509][INFO][category=io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource] Snapshot step 5 - Reading structure of captured tables
...
[2019-05-29T12:23:43,016][ERROR][category=io.debezium.pipeline.ErrorHandler] Producer failure
java.lang.RuntimeException: java.sql.SQLRecoverableException: Closed Connection
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.rollbackTransaction(HistorizedRelationalSnapshotChangeEventSource.java:407)
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.execute(HistorizedRelationalSnapshotChangeEventSource.java:172)
	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:80)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLRecoverableException: Closed Connection
	at oracle.jdbc.driver.PhysicalConnection.getAutoCommit(PhysicalConnection.java:1828)
	at oracle.jdbc.driver.PhysicalConnection.rollback(PhysicalConnection.java:1953)
	at io.debezium.relational.HistorizedRelationalSnapshotChangeEventSource.rollbackTransaction(HistorizedRelationalSnapshotChangeEventSource.java:404)
	... 7 more

      Thread dump:

        java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.SocketDispatcher.read0(SocketDispatcher.java:-1)
	  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
	  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
	  at sun.nio.ch.IOUtil.read(IOUtil.java:245)
	  at sun.nio.ch.IOUtil.read(IOUtil.java:223)
	  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
	  at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
	  at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
	  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
	  at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
	  at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
	  at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
	  at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
	  at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
	  at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
	  at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
	  at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:213)
	  at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1009)
	  at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3353)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:736)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:692)
	  at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:406)
	  - locked <0xdcb> (a oracle.jdbc.driver.T4CConnection)
	  at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1003)
	  at io.debezium.connector.oracle.OracleConnection.readSchema(OracleConnection.java:101)

      The following code:

      io.debezium.jdbc.JdbcConnection.java
      try (ResultSet columnMetadata = metadata.getColumns(databaseCatalog, schemaNamePattern, null, null)) {
    while (columnMetadata.next()) {
        String catalogName = columnMetadata.getString(1);
        String schemaName = columnMetadata.getString(2);
        String tableName = columnMetadata.getString(3);
        TableId tableId = new TableId(catalogName, schemaName, tableName);

        ...
    }
}

      reads all columns from given database schema, it is very slow in my case.

      I suggest to read columns only for captured tables.

      Attachments

        Issue Links

          relates to

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-2575 Improve performance for very large postgres schemas

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              Unassigned Unassigned
              andrey.pustovetov@gmail.com Andrey Pustovetov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: