Details

    • Type: Feature Request Feature Request
    • Status: Closed Closed (View Workflow)
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Labels:
      None
    • Similar Issues:
      Show 10 results 

      Activity

      Hide
      Toby Crawley
      added a comment -

      We had a discussion on irc around error handling:

      <charliekilo> @tcrawley: question: are there plans to handle errors/exceptions within steps or are
                    they (errors) to be handled outside the pipeline?                             [16:16]
      <charliekilo> @tcrawley: let me re-phrase that, are there plans to put plumbing in place for
                    (easy) error handling within pipelines?                                       [16:18]
      <tcrawley> charliekilo: that's a good question - the pipeline uses listen under the hood. it may
                 be a good idea to add an :error-handler option to listen
      <jcrossley3> er...                                                                          [16:19]
      <jcrossley3> why is that a good idea?
      <tcrawley> and then allow you to set it on a per step basis, or globally
      <tcrawley> why is that a bad idea?
      <jcrossley3> because it's gunky                                                             [16:20]
      *** thorwil (~thorwil@p5B3DD2D8.dip.t-dialin.net) has quit: Remote host closed the connection
      <tcrawley> what would the preferred method be for handling listen errors? build it into each
                 listener?
      <tcrawley> that may make sense, since any errors that occur would be listener specific      [16:21]
      <jcrossley3> i'm not opposed to putting it on the pipeline (globally) but if i have to write the
                   listener fn anyway, why write the handler somewhere else?
      <tcrawley> and any non-listener specific errors should be handled by the retries and what not
      <jcrossley3> actually, the pipeline can't catch 'em anyway                                  [16:22]
      <jcrossley3> all it can guarantee is delivery, not consumption, which is between the broker and
                   the listener.
      <charliekilo> I am certainly no expert on event-driven workflows, but I liked how lamina
                    (http://bit.ly/R5pVp2) handled it
      <charliekilo> which is in process                                                           [16:23]
      <charliekilo> to be Mr. Obvious
      <tcrawley> ah, interesting. I took a look at lamina when I first started thinking about our
                 pipeline. I should revisit it                                                    [16:24]
      <jcrossley3> tcrawley: you could certainly wrap each listener inside a try/catch for the
                   :error-handler                                                                 [16:25]
      <jcrossley3> which wouldn't prevent each listener from doing its own handling as well
      <jcrossley3> but there is explicitly no notion of in-process                                [16:26]
      <tcrawley> yeah, hmm. I'll think about it                                                   [16:27]
      <jcrossley3> and that :error-handler might not fire for every type of error, e.g. hq is boinked
                   for some reason, and delivery isn't even attempted.                            [16:28]
      <tcrawley> right, there's only certain classes of errors it can handle
      <charliekilo> system (hq) errors are hard, but from a my (user) perspective two scenarios come to
                    mind: 1.) a pipleline with multiple (critical) steps in which I want to be
                    notifyiedif an error occurred and 2.) a pipleline in which a error 'could' trigger a
                    re-try of the whole pipeline (tx) or from a certain step (to be greedy).      [16:34]
      <charliekilo> just my 2 cents                                                               [16:38]
      <tcrawley> number 2 would be pretty tricky, since it implies a tx around the pipeline, which would
                 be tough
      <projectodd-ci> Project immutant-incremental build #613: FAILURE in 29 min:
                      https://projectodd.ci.cloudbees.com/job/immutant-incremental/613/           [16:39]
      <projectodd-ci> Tobias Crawley: First cut at a messaging pipeline [IMMUTANT-171]
      <jbossbot> jira [IMMUTANT-171] Add messaging pipeline [Open (Unresolved) Feature Request, Major,
                 Unassigned] https://issues.jboss.org/browse/IMMUTANT-171
      <jcrossley3> tcrawley: charliekilo: #2 from the failed step is the default behavior anyway.
                                                                                                  [16:40]
      <tcrawley> right, just that step though.
      <jcrossley3> a tx around the entire pipeline is impossible. only deliveries can participate in a
                   tx, not consumptions.                                                          [16:41]
      <jcrossley3> tcrawley: if that step then succeeds, wouldn't the subsequent ones be triggered as
                   well?
      <tcrawley> yes, but an error on step 2 can't rollback and retry step 1                      [16:42]
      <jcrossley3> it could certainly re-queue it                                                 [16:43]
      <tcrawley> true
      <jcrossley3> there is DLQ behavior built-in that we may be able to use as well
      <tcrawley> on a related note, what do you think about being able to do (def x (msg/start
                 "queue.foo")) (x "my message") ?                                                 [16:44]
      <tcrawley> (msg/start) would just return (partial publish "queue.foo")                      [16:45]
      <jcrossley3> tcrawley: i don't hate it
      <tcrawley> I like how lamina treats pipelines as functions
      <jcrossley3> tcrawley: i may need your help with IMMUTANT-184. i can't figure out why our integs
                   can schedule a job fine, but neither swank nor nrepl can. :(
      <jbossbot> jira [IMMUTANT-184] Can not schedule job from nREPL [Open (Unresolved) Bug, Major,
                 Unassigned] https://issues.jboss.org/browse/IMMUTANT-184
      <tcrawley> I was just about to take a look at that                                          [16:46]
      <charliekilo> if every step in the pipeline is a pure function, could you not keep track of the
                    process in the pipeline and re-try (similar to Clojure's dosync)?
      <tcrawley> charliekilo: since it's based on our messaging, it does that already
      <tcrawley> if the listen throws, the message will be put back on the queue and tried again (up to
                 10 times, by default)                                                            [16:47]
      <charliekilo> @tcrawley: please correct m eif I am wrong, but only retries the individual step
                    .. dosync executes the whole shebang (afaik)
      <jcrossley3> and if all those fail, the message is delivered to a DLQ (dead letter queue) that may
                   be acted on accordingly.
      *** lance|afk (~lanceball@redhat/jboss/lanceball) is now known as lanceball                 [16:48]
      <tcrawley> jcrossley3: are you using an anonymous function for the job handler? if so, can you try
                 a symbol that points to a fn already defined in the container?
      <jcrossley3> tcrawley: sure
      <tcrawley> charliekilo: you are correct - I thought that's what you meant. 
      <tcrawley> charliekilo: retrying the entire pipeline would be difficult, unless we stored each
                 message on entry until the pipeline completed.                                   [16:49]
      <jcrossley3> charliekilo: the trick there is that each stage may be run on a different peer, so
                   keeping up with that distributed state is tricky
      <charliekilo> @tcrawley: good point
      
      Show
      Toby Crawley
      added a comment - We had a discussion on irc around error handling: <charliekilo> @tcrawley: question: are there plans to handle errors/exceptions within steps or are they (errors) to be handled outside the pipeline? [16:16] <charliekilo> @tcrawley: let me re-phrase that, are there plans to put plumbing in place for (easy) error handling within pipelines? [16:18] <tcrawley> charliekilo: that's a good question - the pipeline uses listen under the hood. it may be a good idea to add an :error-handler option to listen <jcrossley3> er... [16:19] <jcrossley3> why is that a good idea? <tcrawley> and then allow you to set it on a per step basis, or globally <tcrawley> why is that a bad idea? <jcrossley3> because it's gunky [16:20] *** thorwil (~thorwil@p5B3DD2D8.dip.t-dialin.net) has quit: Remote host closed the connection <tcrawley> what would the preferred method be for handling listen errors? build it into each listener? <tcrawley> that may make sense, since any errors that occur would be listener specific [16:21] <jcrossley3> i'm not opposed to putting it on the pipeline (globally) but if i have to write the listener fn anyway, why write the handler somewhere else? <tcrawley> and any non-listener specific errors should be handled by the retries and what not <jcrossley3> actually, the pipeline can't catch 'em anyway [16:22] <jcrossley3> all it can guarantee is delivery, not consumption, which is between the broker and the listener. <charliekilo> I am certainly no expert on event-driven workflows, but I liked how lamina (http://bit.ly/R5pVp2) handled it <charliekilo> which is in process [16:23] <charliekilo> to be Mr. Obvious <tcrawley> ah, interesting. I took a look at lamina when I first started thinking about our pipeline. I should revisit it [16:24] <jcrossley3> tcrawley: you could certainly wrap each listener inside a try/catch for the :error-handler [16:25] <jcrossley3> which wouldn't prevent each listener from doing its own handling as well <jcrossley3> but there is explicitly no notion of in-process [16:26] <tcrawley> yeah, hmm. I'll think about it [16:27] <jcrossley3> and that :error-handler might not fire for every type of error, e.g. hq is boinked for some reason, and delivery isn't even attempted. [16:28] <tcrawley> right, there's only certain classes of errors it can handle <charliekilo> system (hq) errors are hard, but from a my (user) perspective two scenarios come to mind: 1.) a pipleline with multiple (critical) steps in which I want to be notifyiedif an error occurred and 2.) a pipleline in which a error 'could' trigger a re-try of the whole pipeline (tx) or from a certain step (to be greedy). [16:34] <charliekilo> just my 2 cents [16:38] <tcrawley> number 2 would be pretty tricky, since it implies a tx around the pipeline, which would be tough <projectodd-ci> Project immutant-incremental build #613: FAILURE in 29 min: https://projectodd.ci.cloudbees.com/job/immutant-incremental/613/ [16:39] <projectodd-ci> Tobias Crawley: First cut at a messaging pipeline [IMMUTANT-171] <jbossbot> jira [IMMUTANT-171] Add messaging pipeline [Open (Unresolved) Feature Request, Major, Unassigned] https://issues.jboss.org/browse/IMMUTANT-171 <jcrossley3> tcrawley: charliekilo: #2 from the failed step is the default behavior anyway. [16:40] <tcrawley> right, just that step though. <jcrossley3> a tx around the entire pipeline is impossible. only deliveries can participate in a tx, not consumptions. [16:41] <jcrossley3> tcrawley: if that step then succeeds, wouldn't the subsequent ones be triggered as well? <tcrawley> yes, but an error on step 2 can't rollback and retry step 1 [16:42] <jcrossley3> it could certainly re-queue it [16:43] <tcrawley> true <jcrossley3> there is DLQ behavior built-in that we may be able to use as well <tcrawley> on a related note, what do you think about being able to do (def x (msg/start "queue.foo")) (x "my message") ? [16:44] <tcrawley> (msg/start) would just return (partial publish "queue.foo") [16:45] <jcrossley3> tcrawley: i don't hate it <tcrawley> I like how lamina treats pipelines as functions <jcrossley3> tcrawley: i may need your help with IMMUTANT-184. i can't figure out why our integs can schedule a job fine, but neither swank nor nrepl can. :( <jbossbot> jira [IMMUTANT-184] Can not schedule job from nREPL [Open (Unresolved) Bug, Major, Unassigned] https://issues.jboss.org/browse/IMMUTANT-184 <tcrawley> I was just about to take a look at that [16:46] <charliekilo> if every step in the pipeline is a pure function, could you not keep track of the process in the pipeline and re-try (similar to Clojure's dosync)? <tcrawley> charliekilo: since it's based on our messaging, it does that already <tcrawley> if the listen throws, the message will be put back on the queue and tried again (up to 10 times, by default) [16:47] <charliekilo> @tcrawley: please correct m eif I am wrong, but only retries the individual step .. dosync executes the whole shebang (afaik) <jcrossley3> and if all those fail, the message is delivered to a DLQ (dead letter queue) that may be acted on accordingly. *** lance|afk (~lanceball@redhat/jboss/lanceball) is now known as lanceball [16:48] <tcrawley> jcrossley3: are you using an anonymous function for the job handler? if so, can you try a symbol that points to a fn already defined in the container? <jcrossley3> tcrawley: sure <tcrawley> charliekilo: you are correct - I thought that's what you meant. <tcrawley> charliekilo: retrying the entire pipeline would be difficult, unless we stored each message on entry until the pipeline completed. [16:49] <jcrossley3> charliekilo: the trick there is that each stage may be run on a different peer, so keeping up with that distributed state is tricky <charliekilo> @tcrawley: good point

        People

        • Assignee:
          Toby Crawley
          Reporter:
          Toby Crawley
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved: