chore: clean up all markdown lint errors in aggregator plugins (#10151)
This commit is contained in:
Joshua Powers
GitHub
parent
6ce4729813
commit
96e939a082
|
|
@ -3,7 +3,7 @@
|
|||
The BasicStats aggregator plugin give us count,diff,max,min,mean,non_negative_diff,sum,s2(variance), stdev for a set of values,
|
||||
emitting the aggregate every `period` seconds.
|
||||
|
||||
### Configuration:
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
# Keep the aggregate basicstats of each metric passing through.
|
||||
|
|
@ -20,32 +20,32 @@ emitting the aggregate every `period` seconds.
|
|||
```
|
||||
|
||||
- stats
|
||||
- If not specified, then `count`, `min`, `max`, `mean`, `stdev`, and `s2` are aggregated and pushed as fields. `sum`, `diff` and `non_negative_diff` are not aggregated by default to maintain backwards compatibility.
|
||||
- If empty array, no stats are aggregated
|
||||
- If not specified, then `count`, `min`, `max`, `mean`, `stdev`, and `s2` are aggregated and pushed as fields. `sum`, `diff` and `non_negative_diff` are not aggregated by default to maintain backwards compatibility.
|
||||
- If empty array, no stats are aggregated
|
||||
|
||||
### Measurements & Fields:
|
||||
## Measurements & Fields
|
||||
|
||||
- measurement1
|
||||
- field1_count
|
||||
- field1_diff (difference)
|
||||
- field1_rate (rate per second)
|
||||
- field1_max
|
||||
- field1_min
|
||||
- field1_mean
|
||||
- field1_non_negative_diff (non-negative difference)
|
||||
- field1_non_negative_rate (non-negative rate per second)
|
||||
- field1_sum
|
||||
- field1_s2 (variance)
|
||||
- field1_stdev (standard deviation)
|
||||
- field1_interval (interval in nanoseconds)
|
||||
- field1_count
|
||||
- field1_diff (difference)
|
||||
- field1_rate (rate per second)
|
||||
- field1_max
|
||||
- field1_min
|
||||
- field1_mean
|
||||
- field1_non_negative_diff (non-negative difference)
|
||||
- field1_non_negative_rate (non-negative rate per second)
|
||||
- field1_sum
|
||||
- field1_s2 (variance)
|
||||
- field1_stdev (standard deviation)
|
||||
- field1_interval (interval in nanoseconds)
|
||||
|
||||
### Tags:
|
||||
## Tags
|
||||
|
||||
No tags are applied by this aggregator.
|
||||
|
||||
### Example Output:
|
||||
## Example Output
|
||||
|
||||
```
|
||||
```shell
|
||||
$ telegraf --config telegraf.conf --quiet
|
||||
system,host=tars load1=1 1475583980000000000
|
||||
system,host=tars load1=1 1475583990000000000
|
||||
|
|
|
|||
|
|
@ -1,42 +1,47 @@
|
|||
# Derivative Aggregator Plugin
|
||||
|
||||
The Derivative Aggregator Plugin estimates the derivative for all fields of the
|
||||
aggregated metrics.
|
||||
|
||||
### Time Derivatives
|
||||
## Time Derivatives
|
||||
|
||||
In its default configuration it determines the first and last measurement of
|
||||
the period. From these measurements the time difference in seconds is
|
||||
calculated. This time difference is than used to divide the difference of each
|
||||
field using the following formula:
|
||||
```
|
||||
|
||||
```text
|
||||
field_last - field_first
|
||||
derivative = --------------------------
|
||||
time_difference
|
||||
```
|
||||
|
||||
For each field the derivative is emitted with a naming pattern
|
||||
`<fieldname>_rate`.
|
||||
|
||||
### Custom Derivation Variable
|
||||
## Custom Derivation Variable
|
||||
|
||||
The plugin supports to use a field of the aggregated measurements as derivation
|
||||
variable in the denominator. This variable is assumed to be a monotonically
|
||||
increasing value. In this feature the following formula is used:
|
||||
```
|
||||
|
||||
```text
|
||||
field_last - field_first
|
||||
derivative = --------------------------------
|
||||
variable_last - variable_first
|
||||
```
|
||||
|
||||
**Make sure the specified variable is not filtered and exists in the metrics passed to this aggregator!**
|
||||
|
||||
When using a custom derivation variable, you should change the `suffix` of the derivative name.
|
||||
When using a custom derivation variable, you should change the `suffix` of the derivative name.
|
||||
See the next section on [customizing the derivative name](#customize-the-derivative-name) for details.
|
||||
|
||||
### Customize the Derivative Name
|
||||
## Customize the Derivative Name
|
||||
|
||||
The derivatives generated by the aggregator are named `<fieldname>_rate`, i.e. they are composed of the field name and a suffix `_rate`.
|
||||
You can configure the suffix to be used by changing the `suffix` parameter.
|
||||
|
||||
### Roll-Over to next Period
|
||||
## Roll-Over to next Period
|
||||
|
||||
Calculating the derivative for a period requires at least two distinct measurements during that period.
|
||||
Whether those are available depends on the configuration of the aggregator `period` and the agent `interval`.
|
||||
|
|
@ -47,7 +52,7 @@ replace the roll-over metric. A main benefit of this roll-over is the ability to
|
|||
cope with multiple "quiet" periods, where no new measurement is pushed to the
|
||||
aggregator. The roll-over will take place at most `max_roll_over` times.
|
||||
|
||||
#### Example of Roll-Over
|
||||
### Example of Roll-Over
|
||||
|
||||
Let us assume we have an input plugin, that generates a measurement with a single metric "test" every 2 seconds.
|
||||
Let this metric increase the first 10 seconds from 0.0 to 10.0 and then decrease the next 10 seconds form 10.0 to 0.0:
|
||||
|
|
@ -111,18 +116,18 @@ To illustrate this, let us compare the derivatives for `period = "7s"`.
|
|||
| timestamp | value | `max_roll_over = 0` | `max_roll_over = 1` |
|
||||
|-----------|-------|-----------|--------------|
|
||||
| 0 | 0.0 |
|
||||
| 2 | 2.0 |
|
||||
| 4 | 4.0 |
|
||||
| 6 | 6.0 |
|
||||
| 2 | 2.0 |
|
||||
| 4 | 4.0 |
|
||||
| 6 | 6.0 |
|
||||
||| 1.0 | 1.0 |
|
||||
| 8 | 8.0 |
|
||||
| 10 | 10.0 |
|
||||
| 12 | 8.0 |
|
||||
||| 0.0 | 0.33... |
|
||||
| 14 | 6.0 |
|
||||
| 10 | 10.0 |
|
||||
| 12 | 8.0 |
|
||||
||| 0.0 | 0.33... |
|
||||
| 14 | 6.0 |
|
||||
| 16 | 4.0 |
|
||||
| 18 | 2.0 |
|
||||
| 20 | 0.0 |
|
||||
| 18 | 2.0 |
|
||||
| 20 | 0.0 |
|
||||
||| -1.0 | -1.0 |
|
||||
|
||||
The difference stems from the change of the value between periods, e.g. from 6.0 to 8.0 between first and second period.
|
||||
|
|
@ -130,7 +135,7 @@ Thoses changes are omitted with `max_roll_over = 0` but are respected with `max_
|
|||
That there are no more differences in the calculated derivatives is due to the example data, which has constant derivatives in during the first and last period, even when including the gap between the periods.
|
||||
Using `max_roll_over` with a value greater 0 may be important, if you need to detect changes between periods, e.g. when you have very few measurements in a period or quasi-constant metrics with only occasional changes.
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[aggregators.derivative]]
|
||||
|
|
@ -151,13 +156,14 @@ Using `max_roll_over` with a value greater 0 may be important, if you need to de
|
|||
period = "30s"
|
||||
```
|
||||
|
||||
### Tags:
|
||||
### Tags
|
||||
|
||||
No tags are applied by this aggregator.
|
||||
Existing tags are passed throug the aggregator untouched.
|
||||
|
||||
### Example Output
|
||||
## Example Output
|
||||
|
||||
```
|
||||
```text
|
||||
net bytes_recv=15409i,packets_recv=164i,bytes_sent=16649i,packets_sent=120i 1508843640000000000
|
||||
net bytes_recv=73987i,packets_recv=364i,bytes_sent=87328i,packets_sent=452i 1508843660000000000
|
||||
net bytes_recv_by_packets_recv=292.89 1508843660000000000
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ discrete time series such as procstat, cgroup, kubernetes etc.
|
|||
When a series has not been updated within the time defined in
|
||||
`series_timeout`, the last metric is emitted with the `_final` appended.
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[aggregators.final]]
|
||||
|
|
@ -25,20 +25,21 @@ When a series has not been updated within the time defined in
|
|||
series_timeout = "5m"
|
||||
```
|
||||
|
||||
### Metrics
|
||||
## Metrics
|
||||
|
||||
Measurement and tags are unchanged, fields are emitted with the suffix
|
||||
`_final`.
|
||||
|
||||
### Example Output
|
||||
## Example Output
|
||||
|
||||
```
|
||||
```text
|
||||
counter,host=bar i_final=3,j_final=6 1554281635115090133
|
||||
counter,host=foo i_final=3,j_final=6 1554281635112992012
|
||||
```
|
||||
|
||||
Original input:
|
||||
```
|
||||
|
||||
```text
|
||||
counter,host=bar i=1,j=4 1554281633101153300
|
||||
counter,host=foo i=1,j=4 1554281633099323601
|
||||
counter,host=bar i=2,j=5 1554281634107980073
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ By default bucket counts are not reset between periods and will be non-strictly
|
|||
increasing while Telegraf is running. This behavior can be changed by setting the
|
||||
`reset` parameter to true.
|
||||
|
||||
#### Design
|
||||
## Design
|
||||
|
||||
Each metric is passed to the aggregator and this aggregator searches
|
||||
histogram buckets for those fields, which have been specified in the
|
||||
|
|
@ -24,7 +24,7 @@ The algorithm of hit counting to buckets was implemented on the base
|
|||
of the algorithm which is implemented in the Prometheus
|
||||
[client](https://github.com/prometheus/client_golang/blob/master/prometheus/histogram.go).
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
# Configuration for aggregate histogram metrics
|
||||
|
|
@ -73,40 +73,39 @@ boundaries. Each float value defines the inclusive upper (right) bound of the b
|
|||
The `+Inf` bucket is added automatically and does not need to be defined.
|
||||
(For left boundaries, these specified bucket borders and `-Inf` will be used).
|
||||
|
||||
### Measurements & Fields:
|
||||
## Measurements & Fields
|
||||
|
||||
The postfix `bucket` will be added to each field key.
|
||||
|
||||
- measurement1
|
||||
- field1_bucket
|
||||
- field2_bucket
|
||||
- field1_bucket
|
||||
- field2_bucket
|
||||
|
||||
### Tags:
|
||||
### Tags
|
||||
|
||||
* `cumulative = true` (default):
|
||||
* `le`: Right bucket border. It means that the metric value is less than or
|
||||
- `cumulative = true` (default):
|
||||
- `le`: Right bucket border. It means that the metric value is less than or
|
||||
equal to the value of this tag. If a metric value is sorted into a bucket,
|
||||
it is also sorted into all larger buckets. As a result, the value of
|
||||
`<field>_bucket` is rising with rising `le` value. When `le` is `+Inf`,
|
||||
the bucket value is the count of all metrics, because all metric values are
|
||||
less than or equal to positive infinity.
|
||||
* `cumulative = false`:
|
||||
* `gt`: Left bucket border. It means that the metric value is greater than
|
||||
- `cumulative = false`:
|
||||
- `gt`: Left bucket border. It means that the metric value is greater than
|
||||
(and not equal to) the value of this tag.
|
||||
* `le`: Right bucket border. It means that the metric value is less than or
|
||||
- `le`: Right bucket border. It means that the metric value is less than or
|
||||
equal to the value of this tag.
|
||||
* As both `gt` and `le` are present, each metric is sorted in only exactly
|
||||
one bucket.
|
||||
- As both `gt` and `le` are present, each metric is sorted in only exactly
|
||||
one bucket.
|
||||
|
||||
|
||||
### Example Output:
|
||||
## Example Output
|
||||
|
||||
Let assume we have the buckets [0, 10, 50, 100] and the following field values
|
||||
for `usage_idle`: [50, 7, 99, 12]
|
||||
|
||||
With `cumulative = true`:
|
||||
|
||||
```
|
||||
```text
|
||||
cpu,cpu=cpu1,host=localhost,le=0.0 usage_idle_bucket=0i 1486998330000000000 # none
|
||||
cpu,cpu=cpu1,host=localhost,le=10.0 usage_idle_bucket=1i 1486998330000000000 # 7
|
||||
cpu,cpu=cpu1,host=localhost,le=50.0 usage_idle_bucket=2i 1486998330000000000 # 7, 12
|
||||
|
|
@ -116,7 +115,7 @@ cpu,cpu=cpu1,host=localhost,le=+Inf usage_idle_bucket=4i 1486998330000000000 #
|
|||
|
||||
With `cumulative = false`:
|
||||
|
||||
```
|
||||
```text
|
||||
cpu,cpu=cpu1,host=localhost,gt=-Inf,le=0.0 usage_idle_bucket=0i 1486998330000000000 # none
|
||||
cpu,cpu=cpu1,host=localhost,gt=0.0,le=10.0 usage_idle_bucket=1i 1486998330000000000 # 7
|
||||
cpu,cpu=cpu1,host=localhost,gt=10.0,le=50.0 usage_idle_bucket=1i 1486998330000000000 # 12
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Use this plugin when fields are split over multiple metrics, with the same
|
|||
measurement, tag set and timestamp. By merging into a single metric they can
|
||||
be handled more efficiently by the output.
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[aggregators.merge]]
|
||||
|
|
@ -16,7 +16,7 @@ be handled more efficiently by the output.
|
|||
drop_original = true
|
||||
```
|
||||
|
||||
### Example
|
||||
## Example
|
||||
|
||||
```diff
|
||||
- cpu,host=localhost usage_time=42 1567562620000000000
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
The minmax aggregator plugin aggregates min & max values of each field it sees,
|
||||
emitting the aggrate every `period` seconds.
|
||||
|
||||
### Configuration:
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
# Keep the aggregate min/max of each metric passing through.
|
||||
|
|
@ -16,19 +16,19 @@ emitting the aggrate every `period` seconds.
|
|||
drop_original = false
|
||||
```
|
||||
|
||||
### Measurements & Fields:
|
||||
## Measurements & Fields
|
||||
|
||||
- measurement1
|
||||
- field1_max
|
||||
- field1_min
|
||||
- field1_max
|
||||
- field1_min
|
||||
|
||||
### Tags:
|
||||
## Tags
|
||||
|
||||
No tags are applied by this aggregator.
|
||||
|
||||
### Example Output:
|
||||
## Example Output
|
||||
|
||||
```
|
||||
```shell
|
||||
$ telegraf --config telegraf.conf --quiet
|
||||
system,host=tars load1=1.72 1475583980000000000
|
||||
system,host=tars load1=1.6 1475583990000000000
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
The quantile aggregator plugin aggregates specified quantiles for each numeric field
|
||||
per metric it sees and emits the quantiles every `period`.
|
||||
|
||||
### Configuration
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[aggregators.quantile]]
|
||||
|
|
@ -33,8 +33,10 @@ per metric it sees and emits the quantiles every `period`.
|
|||
# compression = 100.0
|
||||
```
|
||||
|
||||
#### Algorithm types
|
||||
##### t-digest
|
||||
## Algorithm types
|
||||
|
||||
### t-digest
|
||||
|
||||
Proposed by [Dunning & Ertl (2019)][tdigest_paper] this type uses a
|
||||
special data-structure to cluster data. These clusters are later used
|
||||
to approximate the requested quantiles. The bounds of the approximation
|
||||
|
|
@ -47,7 +49,8 @@ where exact quantile calculation isn't required.
|
|||
|
||||
For implementation details see the underlying [golang library][tdigest_lib].
|
||||
|
||||
##### exact R7 and R8
|
||||
### exact R7 and R8
|
||||
|
||||
These algorithms compute quantiles as described in [Hyndman & Fan (1996)][hyndman_fan].
|
||||
The R7 variant is used in Excel and NumPy. The R8 variant is recommended
|
||||
by Hyndman & Fan due to its independence of the underlying sample distribution.
|
||||
|
|
@ -57,8 +60,8 @@ a lot of memory when used with a large number of series or a
|
|||
large number of samples. They are slower than the `t-digest`
|
||||
algorithm and are recommended only to be used with a small number of samples and series.
|
||||
|
||||
## Benchmark (linux/amd64)
|
||||
|
||||
#### Benchmark (linux/amd64)
|
||||
The benchmark was performed by adding 100 metrics with six numeric
|
||||
(and two non-numeric) fields to the aggregator and the derive the aggregation
|
||||
result.
|
||||
|
|
@ -72,7 +75,8 @@ result.
|
|||
| exact R7 | 100 | 7868816 ns/op |
|
||||
| exact R8 | 100 | 8099612 ns/op |
|
||||
|
||||
### Measurements
|
||||
## Measurements
|
||||
|
||||
Measurement names are passed trough this aggregator.
|
||||
|
||||
### Fields
|
||||
|
|
@ -82,6 +86,7 @@ fields are aggregated in the form `<fieldname>_<quantile*100>`. Other field
|
|||
types (e.g. boolean, string) are ignored and dropped from the output.
|
||||
|
||||
For example passing in the following metric as *input*:
|
||||
|
||||
- somemetric
|
||||
- average_response_ms (float64)
|
||||
- minimum_response_ms (float64)
|
||||
|
|
@ -89,7 +94,8 @@ For example passing in the following metric as *input*:
|
|||
- status (string)
|
||||
- ok (boolean)
|
||||
|
||||
and the default setting for `quantiles ` you get the following *output*
|
||||
and the default setting for `quantiles` you get the following *output*
|
||||
|
||||
- somemetric
|
||||
- average_response_ms_025 (float64)
|
||||
- average_response_ms_050 (float64)
|
||||
|
|
@ -110,18 +116,18 @@ Tags are passed through to the output by this aggregator.
|
|||
|
||||
### Example Output
|
||||
|
||||
```
|
||||
```text
|
||||
cpu,cpu=cpu-total,host=Hugin usage_user=10.814851731872487,usage_system=2.1679541490155687,usage_irq=1.046598554697342,usage_steal=0,usage_guest_nice=0,usage_idle=85.79616247197244,usage_nice=0,usage_iowait=0,usage_softirq=0.1744330924495688,usage_guest=0 1608288360000000000
|
||||
cpu,cpu=cpu-total,host=Hugin usage_guest=0,usage_system=2.1601016518428664,usage_iowait=0.02541296060990694,usage_irq=1.0165184243964942,usage_softirq=0.1778907242693666,usage_steal=0,usage_guest_nice=0,usage_user=9.275730622616953,usage_idle=87.34434561626493,usage_nice=0 1608288370000000000
|
||||
cpu,cpu=cpu-total,host=Hugin usage_idle=85.78199052131747,usage_nice=0,usage_irq=1.0476428036915637,usage_guest=0,usage_guest_nice=0,usage_system=1.995510102269591,usage_iowait=0,usage_softirq=0.1995510102269662,usage_steal=0,usage_user=10.975305562484735 1608288380000000000
|
||||
cpu,cpu=cpu-total,host=Hugin usage_guest_nice_075=0,usage_user_050=10.814851731872487,usage_guest_075=0,usage_steal_025=0,usage_irq_025=1.031558489546918,usage_irq_075=1.0471206791944527,usage_iowait_025=0,usage_guest_050=0,usage_guest_nice_050=0,usage_nice_075=0,usage_iowait_050=0,usage_system_050=2.1601016518428664,usage_irq_050=1.046598554697342,usage_guest_nice_025=0,usage_idle_050=85.79616247197244,usage_softirq_075=0.1887208672481664,usage_steal_075=0,usage_system_025=2.0778058770562287,usage_system_075=2.1640279004292173,usage_softirq_050=0.1778907242693666,usage_nice_050=0,usage_iowait_075=0.01270648030495347,usage_user_075=10.895078647178611,usage_nice_025=0,usage_steal_050=0,usage_user_025=10.04529117724472,usage_idle_025=85.78907649664495,usage_idle_075=86.57025404411868,usage_softirq_025=0.1761619083594677,usage_guest_025=0 1608288390000000000
|
||||
```
|
||||
|
||||
# References
|
||||
## References
|
||||
|
||||
- Dunning & Ertl: "Computing Extremely Accurate Quantiles Using t-Digests", arXiv:1902.04023 (2019) [pdf][tdigest_paper]
|
||||
- Hyndman & Fan: "Sample Quantiles in Statistical Packages", The American Statistician, vol. 50, pp. 361-365 (1996) [pdf][hyndman_fan]
|
||||
|
||||
|
||||
[tdigest_paper]: https://arxiv.org/abs/1902.04023
|
||||
[tdigest_lib]: https://github.com/caio/go-tdigest
|
||||
[hyndman_fan]: http://www.maths.usyd.edu.au/u/UG/SM/STAT3022/r/current/Misc/Sample%20Quantiles%20in%20Statistical%20Packages.pdf
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ Counting fields with a high number of potential values may produce significant
|
|||
amounts of new fields and memory usage, take care to only count fields with a
|
||||
limited set of values.
|
||||
|
||||
### Configuration:
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[aggregators.valuecounter]]
|
||||
|
|
@ -29,22 +29,23 @@ limited set of values.
|
|||
fields = ["status"]
|
||||
```
|
||||
|
||||
### Measurements & Fields:
|
||||
### Measurements & Fields
|
||||
|
||||
- measurement1
|
||||
- field_value1
|
||||
- field_value2
|
||||
- field_value1
|
||||
- field_value2
|
||||
|
||||
### Tags:
|
||||
### Tags
|
||||
|
||||
No tags are applied by this aggregator.
|
||||
|
||||
### Example Output:
|
||||
## Example Output
|
||||
|
||||
Example for parsing a HTTP access log.
|
||||
|
||||
telegraf.conf:
|
||||
```
|
||||
|
||||
```toml
|
||||
[[inputs.logparser]]
|
||||
files = ["/tmp/tst.log"]
|
||||
[inputs.logparser.grok]
|
||||
|
|
@ -57,13 +58,14 @@ telegraf.conf:
|
|||
```
|
||||
|
||||
/tmp/tst.log
|
||||
```
|
||||
|
||||
```text
|
||||
/some/path 200
|
||||
/some/path 401
|
||||
/some/path 200
|
||||
```
|
||||
|
||||
```
|
||||
```shell
|
||||
$ telegraf --config telegraf.conf --quiet
|
||||
|
||||
access,url=/some/path,path=/tmp/tst.log,host=localhost.localdomain response="200" 1511948755991487011
|
||||
|
|
|
|||
Loading…
Reference in New Issue