This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The completed document of this feature is Context Store API

Description

Currently context data is stored in the local memory of a Node.js instance running Node-RED.
The proposed feature allows Context to be stored in any external storage.

Use cases

Enable persistence of Context across restarts
Currently the contents of Context is just held in memory. When the runtime restarts, its contents is lost. Using this feature, you can persist contents by storing it in external storage (e.g. local file-system).
Enable shared context between instances
Currently Context is stored in local memory, so you can not share contents with other instances. By using this feature, you can share contents of Context with other instances by storing it in external storage (e.g. Redis).

API design

Just like the Storage API, this functionality is provided in plugin mechanism.
Users can use it by installing a plugin corresponding to the external storage you want to use.
Users can use multiple external storages for different purposes.

Usage

If you want to access the external storage, add the prefix(*) to the key of context data as follows.

(*)prefix(TBD) : #[identifier].

e.g. #redis.id

identifier is optional and defined in settings.js. see Configuration.

If prefix is not added, data in current local memory is accessed.

Discussion

NOL: I am concerned that using $ is too close to environment variable syntax. Having played with a few alternatives, I propose # as the prefix. Please let me know what you think.

HU: I agreed, a user may be confused using ''. `flow.set("$file.id", ("id"))I changed prefix from$to#` in design note.

how to use in function node

how to use in change node (or other nodes)

Configuration

The contextStorage property in settings.js can be used to identify a plugin to use.

settings.js

contextStorage : {
    file:{                        // identifer
        module:"localfilesystem", // module
        config:{                  // config
            dir:"/"
        }
    },
    redis:{
        module:"redis",
        config:{
            host:"172.0.0.1",
            port:6379,
            namespace:"nodered"
        }
    },
    default:{
        module:"mysql",
        config:{
            host:"172.0.0.1",
            port:3306,
            user:"root",
            password:"",
            database:"test"
        }
    }
    // you can use `file` as the `default`.
    // default: "file"
}

identifier
The identifier is used within a flow to identify which plugin a particular context request should use.
module
The node module used to provide context storage. Node-RED will provide a limited set of built-in modules - initially only memory and localfilesystem - which can be set here by name. 3rd-party context storage plugins would be provided here by require-ing them directly in the settings file and setting module to an instance of the plugin
config
The value set in the config is passed to the plugin. These items are defined by the plugin developer.

NOL (17/5): In our discussion this week, it was suggested it should be possible to have multiple instances of any storage module. For example a 'local redis' and a 'remote redis'. That means the API a ContextStorage plugin provides must support multiple instances existing - which is different to the existing Storage API which only allows one instance to exist. We need to define this API here and update the Pull Request to match.

`default` identifier

default is special identifier. If an undefined identifier is used, Node-RED will use the plugin defined as default. default can be used with #.key (also can be used with #default.key).

Discussion

DCJ: Should the default just have a name - and just point to the named section ? eg

redis: { module: "redis.... },
default: "redis",

So that you can swap default easily ?

NOL: Either/or - default could be an alias for another config, or a full options object.
HU: You're right. it is easily and simple. I changed above settings.js.

DCJ: Do we need to think about passwords in the clear in these settings ? Is the whole of settings object available to other nodes ?

NOL: The settings file contents is not accessible to the whole runtime - only the bits we choose to expose. How the user provides values to the file is up to them, such as env-var etc.

DCJ: Can the default also be existing in memory context if not specified ?

HU:
I think it is better to use existing memory context if default is not specified.
but I'm worry that a user may continue to use existing memory context unintentionally.
I think Node-RED should send warn to user. (but it will be too noisy...)

NOL: rather than 'if not specified', what if they explicitly set default to "memory" as a special identifier.

KN: I agree with supporting "memory" module. But I think that the case of 'if not specified' should happen when an existing user upgraded Node-RED. There are three options to handle such a case. I prefer the option 1.

Throw an error.

Store the context variable in memory.

Store the context variable in memory, and notify a user of the undefined module name.

I listed the actual cases below. The above handling is related to the case 3.

DCJ: And if entire contextStorage section missing (as-in an upgrade where user hasn't edited settings.js) then use memory.

KN: OK. I understand that we should distinguish the case between case 3 and the case that no contextStorage section exists. I added another scenario.

There are four different scenarios when importing a flow that uses context storage.

A user wants to run the flow as is by settings a context storage appropriately.
A user wants to always store context variable on the same storage (e.g. memory, file).
A user wants to receive an error if a context storage is used involuntarily.
A user, who upgraded the Node-RED from the previous version without knowing persistable context feature, wants to always store context variable on the memory.

To support all of the above cases, when the imported flow contains a persistable context:

if the identifier is defined, use the context storage.
if the identifier is not defined but default identifier is defined, use the default context storage.

Discussion

DCJ: In this case - what happens if the default doesn't support a feature (like 'run(...') ? We should warn at least if not exact match.

KN: Then we have no clue how to handle the instruction. So we should throw an error.

3. if neither the identifier nor a `default` identifier are not defined, throw an error.

Discussion

DCJ: Yes not running but ideally still able to edit/correct the error if possible.

KN: If a user does not specify default identifier, it means they declare that they want to receive an error instead of doing something. So I think that just throwing an error with an appropriate error message would be fine.

NOL: To be clear - this only applies if they have contextStorage in their settings, but no default config within it?

KN: Yes, that's right. I've added a table below to clarify the conditions.

4. if the entire contextStorage section is missing in settings.js, use `memory` module (same as the current context).

The table below shows the settting of settings.js for each case above.

Case	`contextStorage` object in settings.js	appropriate identifier in `contextStorage`	`default` in `contextStorage`	Action
1	Exists	Exists	Exists / Not exist	Use the specified module
2	Exists	Not exist	Exists	Use the `default` module
3	Exists	Not exist	Not exist	Throw an error
4	Not exist	N/A	N/A	Use `memory` module

Special case

#file
If a dot is not specified, it is treated as a normal key without persistable context.
#file.
If a key is omitted, throw an error.
#_.key
"_" is a reserved name. It is treated as a memory context.

Executing plugin specific function

Currently, get / set / keys is only methods to handle Context.
In Redis and others, an atomic API such as INCR can be used, but it can not be used in the current.
So we provide run method to execute plugin specific function .

run(command, key, value)

It executes plugin specific function passed in the argument command.

Argument	Description
command	The function name to execute
key	The key to pass 'command'
value	The value to pass 'command'

Returns a value that plugin specific function returns. If the plugin return nothing like a set, run will return undefined.

LocalFileSystem plugin

　　The LocalFileSystem plugin is one of the bundle plugins for Persistable Context.
　　This plugin stores context to Local File System.

Description of LocalFileSystem functionality

Directory Structure

$HOME/.node-red/contexts
├── global
│     └── global_context.json
├── <id of Flow 1>
│     ├── flow_context.json
│     ├── <id of Node a>.json
│     └── <id of Node b>.json
└── <id of Flow 2>
       ├── flow_context.json
       ├── <id of Node x>.json
       └── <id of Node y>.json

The plugin creates settings.userDir/contexts directory if config.dir is not passed.
global context is stored in global directory as global_context.json.
flow context is stored in <id of the flow> directory as flow_context.json.
local context is stored in <id of the flow it is in> directory as <id of the node>.json.

example

+--------+
+ Flow A +
+--------+----------------------------+
+                                     +
+　+----------+        +----------+   +
+  +timestamp +--------+ function +   +
+　+----------+        +----------+   +
+                                     +
+-------------------------------------+

The id of the Flow A is 8588e4b8.784b38
The id of the function node is 80d8039e.2b82
The function node stores data to local, flow and global context in local file system.
The following directories and files are created after the Flow A runs.

.node-red/contexts
├── global
│     └── global_context.json
└── 8588e4b8.784b38
       ├── flow_context.json
       └── 80d8039e.2b82.json

File Format

The plugin stores data as formatted JSON.

{
    "key": "value", 
    "num": 123456, 
    "array": [
        "1", 
        "2", 
        "3"
    ], 
    "object": {
        "foo": "bar"
    }
}

Considerations

Serializing/Deserializsing JavaScript Objects

Object can be stored in current memory context, but most external storage cannot handle Object as it is. So Node-RED or plugin should serialize/deserialize Object.

Discussion

HU: Persistable Context serializes/deserializes Objects using JSON.
If there are some problems, we will consider how solve those.

DCJ - Agreed for version 1

Directly accessing Object property

A user can directly access Object property in current memory context as following.

var onj = {"bar":1};
flow.set("foo", obj);
flow.get("foo.bar"); // return 1

Discussion

HU: Persistable Context do not support this.
If there are some problems, we will consider how solve those.

DCJ - Agreed for version 1.

Key name of default storage

In the case that file is not declared but default is declared on contextStorage, if a context data is specified like #file.count, Node-RED stores the key name in the default storage. In this case, there are two options for storing the key name.

#file.count (The whole name that a user specified)

Pros: Name confliction can be avoided.

Cons: The rule will become complicated especially when a user directly accesses to the context data (e.g. on redis).

count (The key name without storage name)

Pros: Store a key name in the same rule of the other context storage.

Cons: If a user uses a same key name for the different storages, name confliction will occur.

My proposal is as follows. Do you agree with this option?

Choose the option 2.
Encourage a unique key name by writing it on design note.

Discussion

NOL: agreed - fall back to the default store, and remove the store name from the key. I think having multiple storage plugins will be rare, so this is an very much an edge case.

How to get keys in the external storage

In Usage,

If you want to access the external storage, add the prefix(*) to the key of context data as follows.

Currently,keys() needs no arguments. How we get all keys in the external storage?

Adding a parameter to keys()
keys("#identifier") returns all keys in the specified storage.
If #identifier is null or undefined, keys() return all keys in the memory context.
If you use a LocalFileSystem plugin as #file, keys("#file") returns all keys in the LocalFileSystem context.
- Pros: You can get keys from the specified storage. You can also get keys from the memory context in the same way as current. i.e. flow.keys()
- Cons: keys() usage changes.
No arguments same as current
a. keys() returns all keys in all external storage.
If you use a LocalFileSystem plugin, keys() returns all keys in the memory and LocalFileSystem context.
If this is chosen, we have to discuss about returned keys format.
- Pros: You can use keys() with no arguments as before.
- Cons: You cannot get keys from the specified storage.
b. keys() returns all keys in the memory context.
Node-RED don't provide the interface of keys to plugin. So you cannot get all keys by keys().
If a plugin wants to returns keys, it will use run() to return keys.
- Pros: You can use keys() with no arguments as before.
- Cons: You cannot get keys from the external storage by keys()

Discussion

HU: I prefer number 1. I think that the cons is small impact same as set/get.

NOL: Agreed - number 1. The change to keys() syntax is only for persistable context users and just part of the new api it provides.

API Design

Previous Discussion

Problem: asynchronous access to context.

NOL: I can't remember if we discussed this previously. But looking at it now, this is a big problem we need to solve now before committing to any particular API. What we have planned will not work in an asynchronous way.

For any plugin that accesses data remotely, the get/set functions must be able to complete asynchronously. This is a problem as the current api we expose in the Function node is synchronous:

var value = flow.get("counter");

The plan to minimise the changes by just using the key name to identify if persistable context should be used or not won't work. We cannot change .get/.set to work asynchronously as that will break all existing users.

So we have to go back over the api design. 😿

The current proposal was chosen as it met lots of requirements:

it didn't add new objects/functions for a user to understand
it put the choice of store in to the key name - which means the TypedInput for flow/global just works

If we add a new top-level object for accessing the persistent store, we break all of those requirements; its more api for a user to understand and the TypedInput would need more options to cover it.

Having gone through various options on paper, I have reached the following proposal:

If the key passed to .get or .set is a persistable key (as defined above), then return a Promise and complete asynchronously.
Otherwise, complete synchronously as it does already.

It isn't ideal; returning different things under different circumstances is not very nice. But I simply can't see how else to handle it and meet all the requirements.

As an aside, when we drop support for Node 6, we can wrap the Function node as an async Function so the await keyword can be used:

var value = await flow.get("#.counter");

Which gets it back to looking more sensible.

HU: If the above proposal is realized, I think that the following count-up codes(Example A) will be changed like Example B to use persistable context in the Function Node. Is my understanding correct?

// Example A: use current context
var counter = flow.get("counter");
counter += 1;
flow.set("counter",counter);
msg.counter = counter;
return msg;

// Example B: use persistable context
flow.get("#.counter").then(function(counter){
    counter += 1;
    flow.set("#.counter", counter).then(function(){
        msg.counter = counter;
        return msg;
    });
});

nol: yes - I agree it is not very nice, but to support async access to persistable context, there's little choice. When we get to the point of only supporting Node 8 or later, we'll have the option of using async/await. At which point the example becomes:

// Example C: using persistable context with await
var counter = await flow.get("counter");
counter += 1;
await flow.set("counter",counter);
msg.counter = counter;
return msg;

If my understanding is correct, I think there are some issues.

Current .get and .set are synchronous, but these will be changed to be asynchronous.
It may be difficult to use and understand Promise for a user.

nol: my proposal was that .get/.set continue to return a value when accessing current context. But if their key is for persistable context (begins with a #) then they return a promise. I'm not entirely happy about returning different types of things. So happy to explore other options.

Our proposal are the following:

If the key passed to .getAsync or .setAsync is a persistable key, then return a Promise and complete asynchronously.
If the key passed to .get or .set is a persistable key, then return a value(not a Promise) and complete synchronously.
Otherwise, complete synchronously as it does already.

nol: there is no way to hide the fact persistable context uses async functions to get/set its values. You cannot use .get/set to return synchronously a value that is provided asynchronously without either using the await keyword in your Function, or if we added a compilation step in the Function node that used something like babel to transpile their code.

The following shows the relation of function call at .get and .getAsync

+++++++++++++++ flow.get('#.counter') +++++++++++ file.get('flow','counter') +++++++++++++
+             +---------------------->+         +--------------------------->+ LocalFile +
+function node+                       + context +                            + System    +
+             +<----------------------+         +<---------------------------+ Plugin    +
+++++++++++++++    return value(*1)   +++++++++++    return Promise<value>   +++++++++++++

*1: This value is synchronized in the context with await module like this.

+++++++++++++++ flow.getAsync('#.counter') +++++++++++ file.get('flow','counter') +++++++++++++
+             +--------------------------->+         +--------------------------->+ LocalFile +
+function node+                            + context +                            + System    +
+             +<---------------------------+         +<---------------------------+ Plugin    +
+++++++++++++++    return Promise<value>   +++++++++++    return Promise<value>   +++++++++++++

Options

Proposal A - overload get/set

.get/.set return synchronously for non-persistable keys
.get/.set return a Promise for persistable keys (key begins with #)

Pros

familiar get/set functions
existing TypedInput UI doesn't need changing

Cons

inconsistent return type from the same function call.

Proposal B - add getAsync/setSync

.get/.set can only be used for non-persistent context. Will throw an error if key begins with #.
.getAsync/.setAsync used for either normal context or persistable. Returns a Promise.

Pros

separate functions for persistable context - can be async
existing TypedInput UI doesn't need changing - the key still identifies persistable-vs-normal context

Cons

new functions to know about.

Proposal C - add new top-level object

Introduce a new top level object for persistable context access - all return Promises
```
 store.get/set
 store.flow.get/set
 store.global.get/set
```

Pros

clear separation of concept - operating on persistent store rather than volatile context

Cons

TypedInput UI would need new options to select persistent-vs-normal context

Conclusion

nol: looking back at my original proposal, (proposal A) I don't like returning different types for different keys. You have persuaded me that having separate getAsync/setAsync functions is cleaner (proposal B). The question I still have then, is what we call the functions. getAsync is an okay name - but I wonder if there is a better name that helps a user know its for accessing persistent context.

Friday 15th June: Proposal D (or is it E?)

nol: I have never been fully happy with the proposals. The mix of synchronous and asynchronous do not sit well and it has felt like we are picking the least-worst option. That is not a good place to start when adding a major new feature. My worries are:

get/set and getAsync/setAsync - unclear to the user about the difference.

the #foo.bar key syntax - too much for the user to get wrong to properly use the feature. This brings me to a new proposal for how to expose persistable context.

The goal is to have a single get/set pair of functions. The problem with the previous proposal was that those functions would sometimes return a value and sometimes return a Promise. That inconsistency would lead to confusion.

Use callbacks not promises

The solution in this proposal is to not use Promises, but to use normal Callbacks.

   // Synchronous access (as-is):
   var foo = flow.get('foo');
   flow.set('foo', 123);

   // Asynchronous access:
   flow.get('foo', function(err, value) {
   });

   flow.set('foo', 123, function(err) {
   });

That means all existing code continues to work - synchronous access is unchanged. To enable async access, the callback is added. Callbacks are much more user-friendly for novice programmers than Promises are (in my opinion)

In both cases, the callback's first argument is an error object - null if the operation was successful. This is a much cleaner way for a user to 'enable' asynchronous access.

To minimise callback nesting when multiple values need to be retrieved, it could support an array as the key argument:

   // Get multiple
   flow.get(['foo', 'bar'], function(err, fooValue, barValue) {
   });

Discussion

HN: According to Uchikawa-san's experience on updating core nodes, I think we need this kind of bulk operations.
But this may be the place to use Promise?

~~HU: I think too. For core nodes and contrib-nodes it might be good to be able to use Promise.~~

HN: How can we distinguish an error in foo and an error in bar? Should err (and values) be an array?

HU: ~~I think it is good to return an array.~~ ~~For example, if an error occured in foo , [fooError, barValue] is passed to the callback.~~
I thought that error-first callback style is common and better.

I think it is better to pass an array of values to the callback when many values need to be retrieved. If it is implemented, should we also implemented same it for synchronous access to memory context?

   // Get multiple
   flow.get(['foo', 'bar', ...], function(err, [fooValue, barValue, ...]) {
   });
   //In case of `set()`, if `key.length !== value.length`, throw an Error.
   flow.set(['foo', 'bar', ...], [fooValue, barValue, ...], function(err) {
   });

Supporting multiple context stores

The more I look at the persistable key name #store.key - I don't like it. There's too much going on in one string for a new user to understand.

We should also acknowledge that most users won't have multiple context stores.

To specify the context store (if not the default one), an optional extra argument can be provided to give the store name:

   // Asynchronous access to a named store:
   flow.get('foo', 'redis', function(err, value) {
   });

   flow.set('foo', 123, 'redis', function(err) {
   });

The rules then become:

no callback provided, no store provided : (sync access)use in-memory context
callback provided, no store provided : use default context (which could still be in-memory)
no callback provided, store provided : for a get - throw an error. for a set - allow it
callback provided, store provided : use specific store

Discussion

HU: In rule 3, if an error occurs in asynchronous process in set, the error cannot be caught.
How do we handle it?

Ignore the error.

Log the error.

Others

I prefer 1. I think that the caller should decide whether to handle the error or not.

I think that is a clearer thing to understand than trying to explain why keys now have a structure beyond just being the key to store the data under.

Exposing persistable context in the TypedInput

The benefit of the #.foo.bar syntax was that it could be used with the TypedInput elements with no changes. But this proposal moves away from that sort of syntax.

I am going to work on a design update for the TypedInput in line with this updated proposal. The goal will be to let the user select which store to use from a list.

Discussion

HN: My concern is how to handle an input specification of TypedInput from nodes implementation. Do we need a code for checking sync/async access for all TypedInput? Or, always use async access with callback?

HU: We need to consider how to handle persistable context in JSONata expression.

HU: I would like to make sure what API we should define for persistable context.
I think there are three APIs. the APIs are followings.

++++++++                            +++++++++++++++                            +++++++++++                              +++++++++++
+      + -------------------------> +             + -------------------------> +         + ---------------------------->+ Context +
+ User +    1.Function node API?    +Function node+      2.Context API?        + Context +  3.ContextStorage Plugin API + Storage +
+      + <------------------------- +             + <------------------------- +         +<---------------------------- + Plugin  +
++++++++                            +++++++++++++++                            +++++++++++                              +++++++++++

Please correct me if these API names are wrong.

Function node API
The Function node API is used by a user who edits Function node.
It will be defined based on Proposal D.
Context API
The Context API is used by the core or contrib nodes that need to use context.
So Node-RED contributor or contrib node developer may use it.
ContextStorage Plugin API
The ContextStorage Plugin API is used by the context object in Node-RED runtime.
So Node-RED contributor or plugin developer may use it.
It was defined based on bellow section.

I would like to discuss whether 2.Context API should be also defined based on Proposal D.
Because I think that it may be better to return Promise in the situation where the node handle persistable context.
If it only use Callback, code of the nodes may get complicated like Callback hell.
(Maybe it actually does not get complicated and I just don't know...)

So what do you think about the following context APIs?

   // Synchronous access (as-is):
   node.context().flow.get(key);
   node.context().flow.set(key, value);

   // Asynchronous access:
   node.context().flow.get(key, store); // return Promise<value>
   node.context().flow.set(key, value, store); // return Promise<>

NOL: There is also the RED.util.evaluateNodeProperty utility function to consider - (call it API 1.5 in your good summary of the APIs we're looking at). Most nodes that use the TypedInput will be using that function rather than accessing context directly.

The updates I've made to allow the Store to be provided in the TypedInput works by encoding the selection into the value - we have to do this because we can't rely on nodes adding another property to hold the store selection. So if the user sets a TypedInput to: [flow.][foo ][redis], the resulting property is set to #:redis::foo (The syntax may change again, but I've used a similar syntax to the previous proposal, but it can be a bit more complicated to reduce chance of clash as we don't require the user to know the syntax and type it themselves.)

This means RED.util.evaluateNodeProperty will still receive that property and need to do the right thing. For the same reasons as the Function node's get/set functions, we can't update that sometimes return a Promise - as that will break existing code in an unexpected way. I think this API should follow the optional callback approach we have for the Function node.

It currently has a signature of function evaluateNodeProperty(value, type, node, msg). It should be updated to take a 5th argument - callback. Then the same rules apply as for Function node.

For the node.context() API, I think we have the same problem; sometimes returning a Promise and sometimes returning a value. I think we have to follow the callback pattern.

The Context Plugin API is brand new - we're not extending an existing API. But it does need to provide both sync and async access. If we use promises for async access, then the api will need to provide separate get/set and getAsync/setAsync functions to distinguish them. The callback approach would be cleaner as it's back to a single get/set pair of functions.

*** Caching *** : most users will not be using persistable context because they want to have multiple node-red runtimes sharing a store. They will just want their data saved over restarts. I think we can do a lot to make it easier to adopt if we build in optional caching behaviour. This would be in the layer above the context plugin - if the plugin's configuration has cache=true set, then context will preload the contents into memory. It can then offer synchronous access to its contents (with saves/flushes being done in the background). That will allow all single-user systems to benefit from persistable context without having to modify their code at all.

end of Proposal D

ContextStorage Plugin API

A module that provides a ContextStorage implementation must export a single function that can be used to get new instance of the plugin:

module.exports = function(config) {
    // returns a new instance of ContextStorage using the provided config
}

Which means, for example, the localfilesystem plugin would be created by:

var filePlugin = require("./localfilesystem");
var filePluginInstance = filePlugin(config);

Although we haven't identified the API for within the Function node, here is my proposed API for a storage plugin - that uses promises on all of the functions to allow any of them to run asynchronously. It also introduces an open and close function to allow for any setup and tear down of database connections etc.

`ContextStorage.open()` ⇒ `Promise`

Called when the plugin is initialised. Allows it to perform any actions needed before it can start handling get/set requests.

Returns a promise which resolves when the plugin is ready.

`ContextStorage.close()` ⇒ `Promise`

Called when the plugin is no longer needed, for example, Node-RED is stopping.

Returns a promise which resolves when the plugin is closed.

`ContextStorage.get(scope,key)` ⇒ `Promise<value>`

Retrieves a value for a given key under a given scope. Note: scope/key have swapped compared to the current PR

`ContextStorage.set(scope,key,value)` ⇒ `Promise`

Sets a value for a given key under a given scope. Note: scope/key have swapped compared to the current PR

`ContextStorage.delete(scope)` ⇒ `Promise`

Deletes all context for a given scope.

Redis plugin

The Redis plugin holds context data in the Redis.

Data Structure

Node-RED                      Redis
+-------------------+         +-------------------------------+
| global context    |         | logical database              |
| +---------------+ |         | +---------------------------+ |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| | | key |value| | | <-----> | | | global:key      |value| | |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| +---------------+ |         | |                           | |
|                   |         | |                           | |
| flow context      |         | |                           | |
| +---------------+ |         | |                           | |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| | | key |value| | | <-----> | | | <flow's id>:key |value| | |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| +---------------+ |         | |                           | |
|                   |         | |                           | |
| node context      |         | |                           | |
| +---------------+ |         | |                           | |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| | | key |value| | | <-----> | | | <node's id>:key |value| | |
| | +-----+-----+ | |         | | +-----------------+-----+ | |
| +---------------+ |         | +---------------------------+ |
+-------------------+         +-------------------------------+

This plugin uses a Redis logical database for all context scope.
This plugin prefixes all used keys with context scope in order to identify the scope of the key.
- The keys of global context will be prefixed with global: .
  e.g. Set "foo" to hold "bar" in the global context -> Set "global:foo" to hold "bar" in the Redis logical database.
- The keys of flow context will be prefixed with <id of the flow>: .
  e.g. Set "foo" to hold "bar" in the flow context whose id is 8588e4b8.784b38 -> Set "8588e4b8.784b38:foo" to hold "bar" in the Redis.
- The keys of node context will be prefixed with <id of the node>: .
  e.g. Set "foo" to hold "bar" in the node context whose id is 80d8039e.2b82:8588e4b8.784b38 -> Set "80d8039e.2b82:8588e4b8.784b38:foo" to hold "bar" in the Redis.

Node-RED                                 Redis
+------------------------------+         +---------------------------------------------+
| global context               |         | logical database                            |
| +--------------------------+ |         | +-----------------------------------------+ |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | str    | "foo"       | | | <-----> | | | global:str    | "\"foo\""           | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | num    | 1           | | | <-----> | | | global:num    | "1"                 | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | nstr   | "10"        | | | <-----> | | | global:nstr   | "\"10\""            | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | bool   | false       | | | <-----> | | | global:bool   | "false"             | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | arr    | ["a","b"]   | | | <-----> | | | global:arr    | "[\"a\",\"b\"]"     | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| | | obj    | {foo,"bar"} | | | <-----> | | | global:obj    | "{\"foo\",\"bar\"}" | | |
| | +--------+-------------+ | |         | | +---------------+---------------------+ | |
| +--------------------------+ |         | +-----------------------------------------+ |
+------------------------------+         +---------------------------------------------+

This plugin converts a value of context to JSON before storing it as string type to the Redis.
After getting a value from the Redis, the plugin also converts the value to an object or a primitive value.

Other Redis client(e.g. redis-cli) can get the value stored by Node-RED like followings.

Node-RED

global.set("foo","bar","redis");
global.set("obj",{key:"value"},"redis");

redis-cli

redis> GET global:foo
"\"var\""
redis> GET global:obj
"{\"key\":\"value\"}"
redis>

Usage

This plugin will be provided as an npm package, not a built-in module.

Install the module

cd ~/.node-red
npm install node-red-context-store-redis  // tentative package name

Add a configuration in settings.js:

contextStorage: {
   redis: {
       module: require("node-red-context-store-redis"),
       config: {
           // see below options
       }
   }
}

Options

This plugin exposes some options defined in node_redis as itself options.

Options	Description
host	The IP address of the Redis server. `Default: "127.0.0.0.1"`
port	The port of the Redis server. `Default: 6379`
db	The Redis logical database to connect. `Default: 0`
prefix	If set, the string used to prefix all used keys.
password	If set, the plugin will run Redis AUTH command on connect. Note: the password will be sent as plaintext.
tls	An object containing options to pass to tls.connect to set up a TLS connection to the server.

see https://github.com/NodeRedis/node_redis#options-object-properties

Considerations

How to store a value of context to the Redis according to Redis data types

The plugin converts a value to JSON and store it as string type.
The plugin converts a value to Redis data type.
The plugin converts a value to custom data type with a Redis module (e.g. ReJSON).

At first, implement option 1. then consider if it should need option 2 or 3.

Description

Use cases

API design

Usage

how to use in function node

how to use in change node (or other nodes)

Configuration

default identifier

Special case

Executing plugin specific function

run(command, key, value)

LocalFileSystem plugin

Directory Structure

example

File Format

Considerations

Serializing/Deserializsing JavaScript Objects

Directly accessing Object property

Key name of default storage

How to get keys in the external storage

API Design

Problem: asynchronous access to context.

Options

Proposal A - overload get/set

Proposal B - add getAsync/setSync

Proposal C - add new top-level object

Conclusion

Friday 15th June: Proposal D (or is it E?)

Use callbacks not promises

Supporting multiple context stores

Exposing persistable context in the TypedInput

ContextStorage Plugin API

ContextStorage.open() ⇒ Promise

ContextStorage.close() ⇒ Promise

ContextStorage.get(scope,key) ⇒ Promise<value>

ContextStorage.set(scope,key,value) ⇒ Promise

ContextStorage.delete(scope) ⇒ Promise

Redis plugin

Data Structure

Usage

Options

Considerations

How to store a value of context to the Redis according to Redis data types

`default` identifier

`ContextStorage.open()` ⇒ `Promise`

`ContextStorage.close()` ⇒ `Promise`

`ContextStorage.get(scope,key)` ⇒ `Promise<value>`

`ContextStorage.set(scope,key,value)` ⇒ `Promise`

`ContextStorage.delete(scope)` ⇒ `Promise`