Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-4745

Avoid loading the same file over again when using 'org.jboss.teiid.resource-adapter.file'

    Details

    • Type: Enhancement
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 10.3
    • Component/s: Query Engine
    • Labels:
      None
    • Environment:
      • WildFly 10
      • Teiid Server 9.1.2
      • CentOs 7

      Description

      When using the file resource adapter org.jboss.teiid.resource-adapter.file, it should be possible that the system per se, by the means of a new property in the resource-adapter, or other option in the view definition, or other configuration elsewhere, to not to unnecessary load a file that is already loaded. Theoretically it should be possible, for example, to check the file date or the file checksum, to see if the file is already loaded by Teiid. If it is, then there is no need to parse the file again.

      This is important due to, specially, performance improvements. If there are a lot of files to be processed, and these files are big (200 MB or more), the system will be processing over and over again the same files if they are not updated for quite some time.

      If, for example, the user uses a Materialized View, then the view will be constantly being updated with the same data, which brings another performance penalty.

      Note: This could be extended to other resource-adapters with some sort of last data date for example, if the user whant to use this sort of mechanism. For example, if using a database like MySQL as source, then tthe user could define a last update table, and the system could do a check first against this table before tryng to load the data.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                shawkins Steven Hawkins
                Reporter:
                pringi Pedro Inácio
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: