Updated Design: msg.parts description of usage (markdown)

Design:-msg.parts---description-of-usage.md

@@ -27,6 +27,8 @@ The msg is currently considered complete if
 
 Having popped any pushed msg.parts.parts back up to msg.parts the completed `msg` is then output.
 
+---
+
 #### Discussion
 What the current behaviour allows is that the join can be manipulated by changing msg.parts.count (or msg.complete) on any of the intermediate parts. For example the join can be ended early by saying it is now complete - or extra items inserted by just adding to the the count so the existing join doesn't complete.
 
@@ -45,4 +47,101 @@ next | string | yes | `msg.id` of next message
 Either `parts.count` and `parts.index`, or `parts.next` can be used at the same time.
 Insertion or deletion of messages with list-like representation can be implemented by keeping smaller number of messages.
 
-Selection of old array-like representation or list-like representation should be selected by producer nodes. Conversion nodes betwee two prepresentations considered to be useful.
+Selection of old array-like representation or list-like representation should be selected by producer nodes. Conversion nodes between two representations considered to be useful.
+
+**DCJ comment** - looking at the list of proposed nodes - none currently would need this idea in order to be implemented successfully. Typically nodes will have to work in one of two ways.
+
+ 1. message sequence - one at a time. As each message arrives it is processed - the parts can then be manipulated - eg insert extra by incrementing index and count, etc. or
+ 2. message group - chunk at a time. The whole group needs to have arrived before being sorted / sliced etc - at which point the parts can be re-generated before sending on the burst.
+
+---
+
+## Proposed New Nodes
+
+### Chunk node
+
+A function that divides input messages into multiple sequences of messages. It can be used to group together a series of inputs into a "chunk" for further processing. An overlap can be specified so (for example) the last reading in one chunk will also be the first reading in the next chunk. 
+
+Messages grouping mode can be selected from:
+
+ - number of messages - group received messages into chunks of the same number of messages,
+ - interval in seconds - group received messages within specified interval.
+
+**Behaviour/Implementation thoughts**
+
+What happens when no messages arrive in "interval" mode ? Options:
+
+ 1. send a blank msg, with the `parts` specified correctly as count of zero.
+ 2. don't send anything.
+
+### Filter node
+
+A function that filters a sequence of messages. The type of the filtering condition can be:
+
+ - payload value - compare msg.payload with a constant value by operator(=,!=,<,<=,>,>=).
+ - regular expression - match with regular expression.
+ - filtering expression - JavaScript expression. On evaluation, variable msg is bound to the message, and $ is bound to msg.payload.
+ - head - only first N messages of a group are sent - just count messages as they arrive.
+ - tail - last N messages - would have to wait for "all" messages in a group to arrive before sending a burst of messages onwards... 
+
+If the condition evaluates to true for an input message, the message is send to output wire. Otherwise, the message is discarded.
+
+**Behaviour/Implementation thoughts**
+
+ - Should it have two outputs ? one for things that match and one for things that don't ? Would save having two nodes if both types of message are required. (DCJ - I think yes)
+ - Is javascript the right expression language to use (given we already have jsonata etc - is another one a good idea ?)
+ - Tail mode - is end only triggered by comparison to `count` ? I think yes.
+ - Head mode - if n messages don't arrive for that group - drop any currently accumulated ? send part msg ? eg if we ask for 10 - and a group of 5 arrive... does it get sent ? or dropped ? (DCJ - I think send)
+ - Apart from head and tail modes - this is similar to the switch node - should that be enhanced to handle parts instead ? (DCJ - I think no - but ?)
+ - In non head and tail mode this node could work on messages without `parts` also. Should that be supported ? (DCJ - I think yes)
+ - If any filter removes the "last" message in an incoming group - then no final msg with a "correct" `parts` will be left to be sent (ie `count` won't be set) - so we will need to send a blank msg with `parts.count` set correctly in this case. (as well as maybe sending to the second output if that port exists) 
+
+### Sort node
+
+A function that sorts a sequence of messages or payload of array type. When paired with the split node, it will reorder the messages. The sorting order can be:
+
+ - ascending,
+ - descending.
+
+For numbers, numerical ordering can be specified by a checkbox.
+
+Sort key can be `payload` or any JavaScript expression for sorting messages, element value or any JavaScript expression for sorting payload of array type.
+
+**Behaviour/Implementation**
+
+ - If "last" message doesn't arrive - just drop the group (when next group start to arrive) ?
+ - In order to sort correctly the whole sequence of messages needs to have arrived - so could (in theory) be a very large memory overhead, plus then once sorted all messages will be resent in correct order as a large burst. Is this acceptable ? The new parts properties can be regenerated just prior to sending.
+ - should the expression be javascript ? or jsonata ? or ...
+
+### Merge node
+
+A function that merges set of input messages into a single message.
+
+Input messages are merged in order specified by a field (called Topics).
+
+For example, if value of Topics is x,x,y, two input messages with topic x and one input message with topic y are merged into a new message.
+
+The merged message contains payload property and properties for each topic. The payload property represents ordered array of payload value of input messages for each topic. The property for each topic represents a payload value for single occurrence of topic or array of payload values for multiple occurrence of topic.
+
+This is merging messages without using `parts` - and creates a msg without `parts`.
+
+**Implementation**
+
+ - What happens if msg arrive out of order ? 
+ - or too many of one topic arrive before the correct number of another ? drop first ? drop last ?
+ - Should this be a mode of operation of the current join node ?
+
+### Map/Reduce node
+
+A function that map or reduce message payload values. Map and reduce operation can be selected by checkbox.
+
+If map option is selected, an expression specified by Map expression is applied to msg.payload.
+
+If reduce option is selected, an expression specified in Reduce expression field is applied to message sequence with initial value specified by Initial value field. Associative order of application of reduce expression is selected by a right reduce check box.
+
+After applying reduce expression to the message sequence, expression specified in Fixup exp is applied to the result.
+
+**Behaviour/Implementation**
+
+ - should the expressions be javascript or jsonata ?
+ - is the Map only mode - similar to change node ?