Updated Design: msg.parts description of usage (markdown)

Hiroyasu Nishiyama 2017-09-20 22:50:50 +09:00
parent cc404e08d0
commit 8fd4cfbaa9
1 changed files with 45 additions and 3 deletions

@ -106,6 +106,8 @@ If the condition evaluates to true for an input message, the message is send to
- (HN) Similar situation also happens on addition/removal of *any* message in the incoming group. But, the filter node can not distinguish between message delay and deletion without correct `parts`. (misunderstanding the comment?)
- (DCJ) No - this is only the same as "missing the last part" - as long as on the last message the index is same as count-1 (or greater) then the message (is currently) considered complete and gets sent. So it's only an issue on at the end. But is a real issue if it doesn't arrive.
- (HN) I understood. The issue on the arrival of last message is disscussed in the following item on sort node.
### Sort node
A function that sorts a sequence of messages or payload of array type. When paired with the split node, it will reorder the messages. The sorting order can be:
@ -128,6 +130,20 @@ Sort key can be `payload` or any JavaScript expression for sorting messages, ele
(DCJ) - what happens if the last message never arrives ???? can't leave in memory forever...
- (HN) I think there are two options:
1. Leave it to the user (i.e. consume memory forever), or
2. Set an upper limit on the length of a message sequence to wait. If the limit is exceeded, an exception is thrown.
The upper limit is:
a. set globally by settings.js, or
b. set by node setting panel, flow context, global context, or settings.js (referenced in this order).
I prefer option 2-b.
### Merge node
A function that merges set of input messages into a single message.
@ -173,12 +189,31 @@ After applying reduce expression to the message sequence, expression specified i
### Switch node
- enhance to consider `parts` property and to produce message sequence as filter node.
- allow head and tail as routing rule for message sequences.
- make head applicable to *Streaming mode* of split node (i.e. without `parts.count`).
- make whether or not to consider `parts` property selectable by option of switch node. (add third rule *checking all rules with message sequence*)
`<removed>`
- allow head and tail as routing rule for message sequences.
- make head applicable to *Streaming mode* of split node (i.e. without `parts.count`).
- make whether or not to consider `parts` property selectable by option of switch node. (add third rule *checking all rules with message sequence*)
`</removed>`
- (HN) These items are unnecessary because time sequence based modes are implemented in filter node.
(DCJ) I'm starting to think the time sequence based modes (head tail) are actually different/confusing enough that they should still be a separate node - but that the generic expression filtering should be within the switch node. That way further sequence type modes could be considered more easily... for example - every odd numbered message, the 3rd to the 7th messages (sort of like a time based substring...). Both should be aware of `msg.parts`.
- (HN) I agree with your proposal. So we enhance switch node to support parts property and add new filter mode with time sequence based filtering.
### Filter node [New]
- Allow filtering based on time-based sequence,
- Use rule specification interface similar to switch node,
- Support rules:
- head *N* - filter the first *N* msgs in a sequence,
- tail *N* - filter the last *N* msgs in a sequence,
- range *N*, *M* - filter from *N*-th to *M*-th msgs in a sequence,
- JSONata expression on `msg` with shorthand for `parts.index` (`$I`) and `parts.count` (`$N`),
- otherwise - no matching rule.
### Split node
@ -197,6 +232,9 @@ After applying reduce expression to the message sequence, expression specified i
(DCJ) - Is this just to create an array ? If so a) what value should the resulting topic be ?
(I don't think Topic1+Topic2 is the answer). and b) should the "template" for which topics to include in the new array be an extra line in the "Send message after..." section of the existing options dialog ?
- (HN) In the above example, I intended to represent message sequences by […] representation. Sorry for confusing representation. The purpose of message sequence concatenation is to generate a message sequence from more than one sequences. I think sequence concatenation is useful for creating multiple message sequence from one sequence then reconstructing a new sequence.
(e.g. filtering CSV records by multiple criteria, then joining to one CSV file)
### Change node
- enhance to support reduce functionality of Map/Reduce node.
@ -206,6 +244,8 @@ After applying reduce expression to the message sequence, expression specified i
(DCJ) - are these extra variables needed as you can already get to all parts of the msg ? or are they in node context variables ? or are they just for convenience ?
- (HN) `$I` is shorthand for `parts.index` so this variable is not necessarily required. `$N` is needed because `parts.count` may not exist for a message except the last one in a sequence. `$A` is also needed because the accumulated value of a reduce operation is not included in a message.
### Sort node
- use JSONata for key expression.
@ -215,3 +255,5 @@ After applying reduce expression to the message sequence, expression specified i
- enhance to support `parts` property.
(DCJ - yes - see also Issue #1394 - which is about support on the input side of CSV. (but will need to be merged)
- (HN) Thank you. I think this enhancement is useful.