Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3880

Document awareness of Oracle database tuning

      We should consider adding a note/blurb in the Oracle documentation that in some way raises awareness that Oracle requires a bit of tuning in order to run the connector in an ideal performing condition.

      For example, Oracle XE images spin up with redo logs configured with 50MB each which is severely too small to store the data dictionary. For some Oracle EE environments with lots of tables and columns, logs may need to be sized upwards of 1GB to 2GB each depending on the size of the database's data dictionary.

      This is a bit tricky because it's not as simple as providing a raw number. There is a certain amount of nuance in considering the database's traffic volume, other integration points that may be involved such as DataGuard and GoldenGate, etc.

      The goal with this note is to make users aware that using Oracle isn't like other databases such as MySQL or PostgreSQL where you can spin up a database instance, start a connector, and it "just works". There is unfortunately a bit of care to making sure that Oracle LogMiner has the right space/resources to do its job effectively and efficiently.

      CC gmorling@redhat.com broldan@redhat.com

            [DBZ-3880] Document awareness of Oracle database tuning

            Released

            Debezium Builder added a comment - Released

            Merged to master (1.8) and 1.6; 1.7 backport to be done by jpechane upon the release.

            Chris Cranford added a comment - Merged to master (1.8) and 1.6; 1.7 backport to be done by jpechane upon the release.

            Chris Cranford added a comment - - edited

            Hi broldan@redhat.com, correct redo log groups.

            At a high level, redo is configured by setting up a certain number of groups and each group is assigned one or more log file members. When I refer to the size of the logs, this is a setting you define at the group level since all log files in a group are sized identically. You normally will see in production systems where a group will have multiple log files, which is called multiplexing. This is done so that in case the write to one disk fails for one log member and becomes corrupt, you have other identical copies of the logs on other disks that hopefully don't become corrupt; allowing the database to recover in case of a crash. So the topography of the redo looks something like this:

            + group1 (1GB)
                + log1a
                + log1b
            + group2 (1GB)
                + log2a
                + log2b
            ...
            + groupN (1GB)
                + logNa
                + logNb
            

            So assuming 5 groups each with 2 members, you would have 10GB of redo since each group effectively consumes 2GB of space for each log, in which each log member per group is stored on separate disks for disk failure protection.

            Chris Cranford added a comment - - edited Hi broldan@redhat.com , correct redo log groups. At a high level, redo is configured by setting up a certain number of groups and each group is assigned one or more log file members. When I refer to the size of the logs, this is a setting you define at the group level since all log files in a group are sized identically. You normally will see in production systems where a group will have multiple log files, which is called multiplexing. This is done so that in case the write to one disk fails for one log member and becomes corrupt, you have other identical copies of the logs on other disks that hopefully don't become corrupt; allowing the database to recover in case of a crash. So the topography of the redo looks something like this: + group1 (1GB) + log1a + log1b + group2 (1GB) + log2a + log2b ... + groupN (1GB) + logNa + logNb So assuming 5 groups each with 2 members, you would have 10GB of redo since each group effectively consumes 2GB of space for each log, in which each log member per group is stored on separate disks for disk failure protection.

            ccranfor@redhat.com When you mention groups in your earlier comment, are you referring to redo log groups? I wasn't sure whether you meant user groups or some sort of database groups, but in a search just now, I came across the concept of redo log groups.

            Robert Roldan added a comment - ccranfor@redhat.com When you mention groups in your earlier comment, are you referring to redo log groups? I wasn't sure whether you meant user groups or some sort of database groups, but in a search just now, I came across the concept of redo log groups .

            Ah, ok! Thanks.

            Robert Roldan added a comment - Ah, ok! Thanks.

            Hi broldan@redhat.com that was a typo on my part, it should have read pre-built Oracle containers and I just didn't catch it with grammarly.

            Chris Cranford added a comment - Hi broldan@redhat.com that was a typo on my part, it should have read pre-built Oracle containers and I just didn't catch it with grammarly.

            Ok, that makes sense. I'll tweak that a little and come up with something for you to review. I agree that we don't want to get into documenting Oracle processes.

            When you refer to pre-built connectors, are those the Integration points that you mentioned in the Jira description or something else?

            Robert Roldan added a comment - Ok, that makes sense. I'll tweak that a little and come up with something for you to review. I agree that we don't want to get into documenting Oracle processes. When you refer to pre-built connectors, are those the Integration points that you mentioned in the Jira description or something else?

            Hi broldan@redhat.com, we really can't make any real suggestions because the size of the redo logs can vary drastically. For example, in our test instance, I've configured them to 400MB each but in some of our users' production systems, they're configured between 1GB and 2GB each. Additionally, some environments have 3-5 groups and others have upwards of 12-16 groups; so you can see this varies.

            I could see something generic like this:

            In some databases that include many tables and columns, the data dictionary may not fit into a single redo log. Additionally, some pre-built Oracle connectors automatically use relatively small redo log sizes as well. In these situations, it's possible that excessive archive logs may be created and can lead to performance problems both with the connector and the database. Redo log size is critical for any Oracle environment and if you suspect the sizes need to be adjusted, consult with your DBA who will be able to verify and adjust them as needed.

            As can be shown here:
            http://www.nazmulhuda.info/redo-log-buffer-tuning

            The steps involved really are covered by Oracle training and their own documentation and online resources. I don't think we should try to convey this since its not our wheelhouse and where these thresholds should be differ from a low volume database to one that has lots of activity to whether its Standalone or RAC environments.

            Chris Cranford added a comment - Hi broldan@redhat.com , we really can't make any real suggestions because the size of the redo logs can vary drastically. For example, in our test instance, I've configured them to 400MB each but in some of our users' production systems, they're configured between 1GB and 2GB each. Additionally, some environments have 3-5 groups and others have upwards of 12-16 groups; so you can see this varies. I could see something generic like this: In some databases that include many tables and columns, the data dictionary may not fit into a single redo log. Additionally, some pre-built Oracle connectors automatically use relatively small redo log sizes as well. In these situations, it's possible that excessive archive logs may be created and can lead to performance problems both with the connector and the database. Redo log size is critical for any Oracle environment and if you suspect the sizes need to be adjusted, consult with your DBA who will be able to verify and adjust them as needed. As can be shown here: http://www.nazmulhuda.info/redo-log-buffer-tuning The steps involved really are covered by Oracle training and their own documentation and online resources. I don't think we should try to convey this since its not our wheelhouse and where these thresholds should be differ from a low volume database to one that has lots of activity to whether its Standalone or RAC environments.

            Robert Roldan added a comment - - edited

            ccranfor@redhat.com Not sure how to approach this. We could say something very general, like:

            In databases that include many tables and columns, the default size of the redo log might not be sufficient to store the data dictionary. If the connector fails to start, or if you experience [poor performance | <_some_symptom_> ] after you start the connector, consider adjusting the log size.

            But I'm not sure how helpful that is.

            Can we suggest a best practice for setting a starting size? Is 50MB for the redo logs always going to be insufficient?

            Or is this more an after-the-fact matter of informing users that if certain symptoms are manifest, it might be necessary to adjust the log configuration?

            Robert Roldan added a comment - - edited ccranfor@redhat.com Not sure how to approach this. We could say something very general, like: In databases that include many tables and columns, the default size of the redo log might not be sufficient to store the data dictionary. If the connector fails to start, or if you experience [poor performance | <_some_symptom_> ] after you start the connector, consider adjusting the log size. But I'm not sure how helpful that is. Can we suggest a best practice for setting a starting size? Is 50MB for the redo logs always going to be insufficient? Or is this more an after-the-fact matter of informing users that if certain symptoms are manifest, it might be necessary to adjust the log configuration?

              broldan@redhat.com Robert Roldan
              ccranfor@redhat.com Chris Cranford
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: