Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-17796

CSV parameters are not updated in AtlasMap step

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • fuse-7.11-GA
    • fuse-7.10-GA
    • Fuse Online
    • None
    • False
    • False
    • % %
    • Todo
    • Hide

      When you edit CSV parameters in data-shape definition in some step, you need to add something (e.g. unused constant) into DataMapper in order to the atlasmapping is updated

      Show
      When you edit CSV parameters in data-shape definition in some step, you need to add something (e.g. unused constant) into DataMapper in order to the atlasmapping is updated

      I have created and published an integration (Webhook -> DB) which consumes CSV. I used this CSV parameters in the WebHook step: allowDuplicateHeaderNames=true, firstRecordAsHeader=true.
      integrationWithCsv-export-firstVersion.zip
      After that, I edited the webhook step, changed CSV parameters (allowDuplicateHeaderNames=false, added ignotreEmptyLines=true) and redeploy but the functionality didn't changed.
      integrationWithCsv-export-afterEditWebhook.zip
      When I compare the model.json files, the parameters are updated in the WebHook step, but not in the AtlasMap step.
      First version:

      ...
"parameters":{
  "allowDuplicateHeaderNames":"true",
  "firstRecordAsHeader":"true",
  "format":"Default"
},
"specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=true&firstRecordAsHeader=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"3 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[]},\"properties\":{\"property\":[]}}}"

      After edit webhook step:

      ...
"parameters":{
  "allowDuplicateHeaderNames":"false",
  "firstRecordAsHeader":"true",
  "format":"Default",
  "ignoreHeaderCase":"true"
},                "specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=true&firstRecordAsHeader=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"3 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[]},\"properties\":{\"property\":[]}}}"

      Also, when I open DataMapper step and save the integration, the atlasmapping is not updated. I have to do some changes in Data Mapper (e.g. create a dummy constant).
      integrationWithCsv-export-afterAddConstant.zip

      ...
"parameters":{
  "allowDuplicateHeaderNames":"false",
  "firstRecordAsHeader":"true",
  "format":"Default",
  "ignoreHeaderCase":"true"
},                "specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=false&firstRecordAsHeader=true&ignoreHeaderCase=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"4 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[{\"name\":\"dummy\",\"value\":\"dummy\",\"fieldType\":\"STRING\"}]},\"properties\":{\"property\":[]}}}"
                     },

      In my POV, the user should open the DataMapper after they edited the CSV parameters in the WebHook step to check that mapping is still valid, but after that, the mapping should update without any changes.

        1. integrationWithCsv-export-afterAddConstant.zip
          4 kB
        2. integrationWithCsv-export-afterEditWebhook.zip
          4 kB
        3. integrationWithCsv-export-firstVersion.zip
          4 kB

            zregvart@redhat.com Zoran Regvart
            mkralik@redhat.com Matej Kralik
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: