Updated Design: msg.parts description of usage (markdown)

Dave Conway-Jones 2017-09-18 12:51:54 +01:00
parent 3378ec20c8
commit cc404e08d0
1 changed files with 14 additions and 2 deletions

@ -74,8 +74,7 @@ What happens when no messages arrive in "interval" mode ? Options:
1. send a blank msg, with the `parts` specified correctly as count of zero.
2. don't send anything.
- (HN) (1) seems better because it is convenient if there is a triggering message for processing messages over a cetetain period. As you pointed out offline, the functionality of chunk node is similar to split node. Integrating chunk functionality to split node seems to be prefered.
- (HN) (1) seems better because it is convenient if there is a triggering message for processing messages over a certain period. As you pointed out offline, the functionality of chunk node is similar to split node. Integrating chunk functionality to split node seems to be preferred.
### Filter node
@ -105,6 +104,7 @@ If the condition evaluates to true for an input message, the message is send to
- (HN) I also think yes.
- If any filter removes the "last" message in an incoming group - then no final msg with a "correct" `parts` will be left to be sent (ie `count` won't be set) - so we will need to send a blank msg with `parts.count` set correctly in this case. (as well as maybe sending to the second output if that port exists)
- (HN) Similar situation also happens on addition/removal of *any* message in the incoming group. But, the filter node can not distinguish between message delay and deletion without correct `parts`. (misunderstanding the comment?)
- (DCJ) No - this is only the same as "missing the last part" - as long as on the last message the index is same as count-1 (or greater) then the message (is currently) considered complete and gets sent. So it's only an issue on at the end. But is a real issue if it doesn't arrive.
### Sort node
@ -126,6 +126,8 @@ Sort key can be `payload` or any JavaScript expression for sorting messages, ele
- should the expression be javascript ? or jsonata ? or ...
- (HN) Similar to filter node, I think using JSONata is better.
(DCJ) - what happens if the last message never arrives ???? can't leave in memory forever...
### Merge node
A function that merges set of input messages into a single message.
@ -175,6 +177,9 @@ After applying reduce expression to the message sequence, expression specified i
- make head applicable to *Streaming mode* of split node (i.e. without `parts.count`).
- make whether or not to consider `parts` property selectable by option of switch node. (add third rule *checking all rules with message sequence*)
(DCJ) I'm starting to think the time sequence based modes (head tail) are actually different/confusing enough that they should still be a separate node - but that the generic expression filtering should be within the switch node. That way further sequence type modes could be considered more easily... for example - every odd numbered message, the 3rd to the 7th messages (sort of like a time based substring...). Both should be aware of `msg.parts`.
### Split node
- enhance to allow new mode for creating sequences from input messages as chunk node.
@ -189,6 +194,9 @@ After applying reduce expression to the message sequence, expression specified i
```
- messages are processed in order if they have `parts` property.
(DCJ) - Is this just to create an array ? If so a) what value should the resulting topic be ?
(I don't think Topic1+Topic2 is the answer). and b) should the "template" for which topics to include in the new array be an extra line in the "Send message after..." section of the existing options dialog ?
### Change node
- enhance to support reduce functionality of Map/Reduce node.
@ -196,6 +204,8 @@ After applying reduce expression to the message sequence, expression specified i
- reduce and fixup expression is only applied to message sequence (msg having `parts` property). Other messages are processed the same as current change node.
- for JSONata expression of reduce mode add special variable $I(index of message), $N(number of messages in a sequence), $A(accumulated value).
(DCJ) - are these extra variables needed as you can already get to all parts of the msg ? or are they in node context variables ? or are they just for convenience ?
### Sort node
- use JSONata for key expression.
@ -203,3 +213,5 @@ After applying reduce expression to the message sequence, expression specified i
### csv node and html node
- enhance to support `parts` property.
(DCJ - yes - see also Issue #1394 - which is about support on the input side of CSV. (but will need to be merged)