1
0
mirror of https://github.com/Oxalide/vsphere-influxdb-go.git synced 2023-10-10 13:36:51 +02:00
vsphere-influxdb-go/vendor/github.com/influxdata/influxdb/stress/v2/DESIGN.md

165 lines
5.7 KiB
Markdown
Raw Normal View History

2017-10-25 22:52:40 +02:00
# Influx Stress tool -> `v2`
The design of the new stress tool was designed to:
* have higher potential write throughput than previous version
* have more schema expressibility for testing different load profiles and professional services
* have more granular reporting to be better able to draw conclusions from tests
In service of these requirements we designed a language that looks a lot like `influxql` to give the new test commands. Instead of a configuration file, the new stress test takes a list of these `Statements`.
The tool has the following components:
* Parser - parses the configuration file and turns it into an `[]Statement`. All code related to the parser is in `v2/stressql/`. The parser was designed as per @benbjohnson's great article on [parsers in go](https://blog.gopheracademy.com/advent-2014/parsers-lexers/).
* Statements - perform operations on target instance or change test environment. All code related to statements is in `v2/statement/`. The following are the available statements:
- `EXEC` - Still a TODO, planned to run outside scripts from the config file.
- `GO` - Prepend to an `INSERT` or `QUERY` statement to run concurrently.
- `INFLUXQL` - All valid `influxql` will be passed directly to the targeted instance. Useful for setting up complex downsampling environments or just your testing environment.
- `INSERT` - Generates points following a template
- `QUERY` - Runs a given query or generates sample queries given a companion `INSERT` statement
- `SET` - Changes the test parameters. Defaults are listed in the `README.md`
- `WAIT` - Required after a `GO` statement. Blocks till all proceeding statements finish.
* Clients - The statement, results and InfluxDB clients. This code lives in `v2/stress_client`
- `StressTest` - The `Statement` client. Also contains the results client.
- `stressClient` - A performant InfluxDB client. Makes `GET /query` and `POST /write` requests. Forwards the results to the results client.
![Influx Stress Design](./influx_stress_v2.png)
### Statements
`Statement` is an interface defined in `v2/statement/statement.go`:
```go
type Statement interface {
Run(s *stressClient.StressTest)
Report(s *stressClient.StressTest) string
SetID(s string)
}
```
* `Run` prompts the statement to carry out it's instructions. See the run functions of the various statements listed above for more information.
* `Report` retrieves and collates all recorded test data from the reporting InfluxDB instance.
* `SetID` gives the statement an ID. Used in the parser. Each `statementID` is an 8 character random string used for reporting.
### `Statement` -> `StressTest`
`Statement`s send `Package`s (queries or writes to the target database) or `Directives` (for changing test state) through the `StressTest` to the `stressClient` where they are processed.
```go
// v2/stress_client/package.go
// T is Query or Write
// StatementID is for reporting
type Package struct {
T Type
Body []byte
StatementID string
Tracer *Tracer
}
// v2/stress_client/directive.go
// Property is test state variable to change
// Value is the new value
type Directive struct {
Property string
Value string
Tracer *Tracer
}
```
The `Tracer` on both of these packages contains a `sync.WaitGroup` that prevents `Statement`s from returning before all their operations are finished. This `WaitGroup` is incremented in the `Run()` of the statement and decremented in `*StressTest.resultsListen()` after results are recorded in the database. This is well documented with inline comments. `Tracer`s also carry optional tags for reporting purposes.
```go
// v2/stress_client/tracer.go
type Tracer struct {
Tags map[string]string
sync.WaitGroup
}
```
### `StressTest`
The `StressTest` is the client for the statements through the `*StressTest.SendPackage()` and `*StressTest.SendDirective()` functions. It also contains some test state and the `ResultsClient`.
```go
type StressTest struct {
TestID string
TestName string
Precision string
StartDate string
BatchSize int
sync.WaitGroup
sync.Mutex
packageChan chan<- Package
directiveChan chan<- Directive
ResultsChan chan Response
communes map[string]*commune
ResultsClient influx.Client
}
```
### Reporting Client
The `ResultsClient` turns raw responses from InfluxDB into properly tagged points containing any relevant information for storage in another InfluxDB instance. The code for creating those points lives in `v2/stress_client/reporting.go`
### InfluxDB Instance (reporting)
This is `localhost:8086` by default. The results are currently stored in the `_stressTest` database.
### `stressClient`
An InfluxDB client designed for speed. `stressClient` also holds most test state.
```go
// v2/stress_client/stress_client.go
type stressClient struct {
testID string
// State for the Stress Test
addresses []string
precision string
startDate string
database string
wdelay string
qdelay string
// Channels from statements
packageChan <-chan Package
directiveChan <-chan Directive
// Response channel
responseChan chan<- Response
// Concurrency utilities
sync.WaitGroup
sync.Mutex
// Concurrency Limit for Writes and Reads
wconc int
qconc int
// Manage Read and Write concurrency separately
wc *ConcurrencyLimiter
rc *ConcurrencyLimiter
}
```
Code for handling the write path is in `v2/stress_client/stress_client_write.go` while the query path is in `v2/stress_client/stress_client_query.go`.
### InfluxDB Instance (stress test target)
The InfluxDB which is being put under stress.
### response data
`Response`s carry points from `stressClient` to the `ResultsClient`.
```go
// v2/stress_client/response.go
type Response struct {
Point *influx.Point
Tracer *Tracer
}
```