frodovdr
/
node-red
mirror of https://github.com/node-red/node-red.git
1
0
Fork 0
Code Issues Releases Wiki Activity

Created Flow testing (markdown)

Kazuki Nakanishi 2018-12-04 15:07:12 +00:00
parent fb13764c5b
commit 9f3dbef964
1 changed files with 119 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

119
Flow-testing.md Normal file

@ -0,0 +1,119 @@
_**This is a work in progress.**_
# Description
## Test target
Node-RED project already provides the tools for testing:
- [Runtime](https://github.com/node-red/node-red/wiki/Testing)
- [Node](https://github.com/node-red/node-red-node-test-helper)
- [Editor](https://github.com/node-red/node-red/wiki/UI-testing) (under development)
This design note focuses on realizing flow testing.
## Requirements
Basic requirements of this tool are as follows.
- Node-RED user can test flows without programming.
- Node-RED uesr can see the test results on the editor.
- It should test the original flow as is. If it tests a copied flow of the original one, sync problem will happen.
## Usecase
1. Node-RED user does regression testing for the existing flows when updating Node-RED or services that connect with Node-RED.
1. Node-RED automatically runs a test on Travis CI when pull request was posted.
# User interface
## Test node
There are two new nodes called test-in node and test-out node. Test-in node sends a mock message on behalf of the actual input node such as http-in node. Test-out node receives a message on behalf of the actual output node such as http-response node.
One test case consists of exactly one test-in node and one test-out node. One test case can cross tabs.
![Flow testing model](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/FlowTestingModel.png)
The requirements of test node are as follows.
- Do nothing when it is NOT in a testing mode.
- Appear only in a testing mode.
- Substitute input node and output node in a testing mode. (named as `shadowing`)
### test-in node
Test-in node specifies one or more test cases so that one test-in node can run multiple test cases.
![Testing in node properties](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/test-in_node_properties.png)
- Target
- Specifies a target node to be substituted by a test-in node.
- When you click the link on the right, all nodes except for test nodes will be shown.
- Cases
- Specifies test cases. You can add a test case by clicking `add` button.
- Each test case has `Label` and `Message` properties.
- Label
- Describes the name of the test case.
- Message
- Describes the properties of a message.
- Although it would be nice to specify each message property on GUI, simply specifying JSON data is a good starting point for the first version.
- Some properties such as `msg.req` of http-in node cannot be completely expressed with JSON. This is a fundamental restriction of flow testing.
### test-out node
Test-out node verifies whether the received message is exactly the expected value or not.
![Testing out node properties](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/test-out_node_properties.png)
- Cases
- Specifies one or more test-in nodes. In each test-in node, one or more test cases can be specified.
- Test in
- Specifies a test-in node corresponding to this test-out node.
- When you click the link on the right, all test-in nodes will be shown.
- Label
- Selects a label that the specific test-out node has.
- When you click the list, show all labels in the test-in node.
- Expected
- Describes the expected property value of a message.
- In the case that the test-out node receives a message without specifying the case here, the test will result in failure.
## Testing mode
As described in test node section, Node-RED needs to change the mode from production to testing, and vice versa.
### Difference of mode
Assume that the following flow is a testing target.
![Testing target flow](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/production_flow.png)
To test the above flow, change a mode to testing mode, and add test-in node and test-out node.
![Testing flow](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/testing_flow.png)
To express the substitution with testing nodes, there are several design options.
- Surround substituted nodes by a dotted line.
- Connect a wire between substituted node and test in/out node (when clicked).
### Switch the mode
A user can see only the original flow without test nodes if Node-RED provides the functionality switching a mode. This may not be necessary for the first version.
To realize the switch, the following implementation is necessary.
- Add a `test` tab on the sidebar
- Add a button to switch the mode on the `test` tab.
- Although changing the mode right after opening a `test` tab is an option, that cannot notify to runtime server.
## Sidebar
User can do the following actions on test sidebar.
- Click `test mode` button to enable testing mode.
- Choose test cases to run.
- Click `Run` button to start testing.
- See the result of each test case.
![Sidebar of flow testing](https://raw.githubusercontent.com/wiki/node-red-hitachi/node-red/images/flow_testing/flowtesting_sidebar.png)
# Development
## Flow data
Need to consider how to store the information of test nodes. There are two options.
Option 1 is to store into flow.json. Option 2 is to create a new file test.json.
- Option 1 (flow.json)
- Easy to share flow data including test data.
- Data will get larger even if a user does not need to run a test.
- Option 2 (test.json)
- Can divide production code and test code.
- Need a function for storage API to load a new file.
## Test driver
Needs additional discussion about which test driver should be used. If it is possible to show test results on the sidebar using Mocha, using Mocha would be good. Otherwise, we may need to implement test driver by ourselves.
# CLI
This flow testing needs to be run on the CLI for targeting to run automatically on Travis CI when receiving a pull request on GitHub.
When running a command like `grunt test-flow`, Node-RED runs flow testing then outputs each result such as the existing Mocha tests.