Customer Impact:

It is a supported feature to configure multiple endpoints in a single behaviorGroup. A user might want to do this if a single event trigger should send notifications to multiple webhooks. This bug if anyone of these webhooks timeout currently it seems that we are disabling the whole behaviorGroup even the endpoints that are working fine.

The customer impact would be if they have a single timing-out endpoint that could fail to receive notifications on other working endpoints

Technical details:

We are trying to send 4 kafka messages and we have registered 2 endpoints. One that will timeout (504) and another one that be successful (200)

Problem is that after encountering the first timeout, the service seems to stop working or the behavior group is disable. It says:

HTTP operation failed invoking https:\\\/\\\/notifications-webhooks-mock-service-cpt.apps.rhperfcluster.ptjz.p1.openshiftapps.com\\\/code\\\/success\\\/GoTJo with statusCode: 504

following is the entire log of engine service:

2024-05-01 21:52:31,973 INFO [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-0) SRMSG18256: Initialize record store for topic-partition 'platform.notifications.ingress-3' at position 4957.
2024-05-01 21:52:32,569 INFO [com.red.clo.not.eve.EventConsumer] (executor-thread-1) Processing received event [id=null, rh-message-id=null, orgId=12345, baet=console/integrations/integration-disabled]
2024-05-01 21:52:34,374 INFO [com.red.clo.not.pro.ConnectorSender] (executor-thread-1) Sending notification to connector [orgId=12345, eventId=6595fba8-6db1-4ec8-b082-5270bbe1f70f, connector=webhook, historyId=f66e542b-61e2-4ce9-b120-af0c22d3ac5f]
2024-05-01 21:52:34,775 INFO [com.red.clo.not.pro.ConnectorSender] (executor-thread-1) Sending notification to connector [orgId=12345, eventId=6595fba8-6db1-4ec8-b082-5270bbe1f70f, connector=webhook, historyId=85012cc2-e76b-448b-ab43-74679b5aca74]
2024-05-01 21:52:35,069 INFO [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-0) SRMSG18256: Initialize record store for topic-partition 'platform.notifications.ingress-2' at position 4185.
2024-05-01 21:52:35,172 INFO [com.red.clo.not.eve.EventConsumer] (executor-thread-1) Processing received event [id=null, rh-message-id=null, orgId=12345, baet=console/integrations/integration-disabled]
2024-05-01 21:52:35,377 INFO [com.red.clo.not.pro.ConnectorSender] (executor-thread-1) Sending notification to connector [orgId=12345, eventId=4c9d9aac-ca90-4692-90f6-1f50e55a77c8, connector=webhook, historyId=a065ec6d-2c34-40cc-a9a1-c0340048f5b3]
2024-05-01 21:52:35,571 INFO [com.red.clo.not.pro.ConnectorSender] (executor-thread-1) Sending notification to connector [orgId=12345, eventId=4c9d9aac-ca90-4692-90f6-1f50e55a77c8, connector=webhook, historyId=0075d88a-1691-4ca4-8a5e-93dc10e49557]
2024-05-01 21:54:09,122 INFO [io.sma.rea.mes.kafka] (vert.x-eventloop-thread-1) SRMSG18256: Initialize record store for topic-partition 'platform.notifications.fromcamel-2' at position 9320.
2024-05-01 21:54:09,127 INFO [com.red.clo.not.eve.ConnectorReceiver] (executor-thread-2) Processing return from camel: {"type":"com.redhat.console.notifications.history","specversion":"1.0","source":"webhook","id":"0075d88a-1691-4ca4-8a5e-93dc10e49557","time":"2024-05-01T21:54:08.022984663","data":"{\"successful\":false,\"duration\":92017,\"details\":{\"type\":\"com.redhat.console.notification.toCamel.webhook\",\"target\":\"https:\\\/\\\/notifications-webhooks-mock-service-cpt.apps.rhperfcluster.ptjz.p1.openshiftapps.com\\\/code\\\/success\\\/GoTJo\",\"outcome\":\"HTTP operation failed invoking https:\\\/\\\/notifications-webhooks-mock-service-cpt.apps.rhperfcluster.ptjz.p1.openshiftapps.com\\\/code\\\/success\\\/GoTJo with statusCode: 504\"}}"}