2021-03-19 03:30:24 +08:00
# Google BigQuery Output Plugin
2021-03-13 01:16:45 +08:00
2021-11-25 02:47:33 +08:00
This plugin writes to the [Google Cloud BigQuery ](https://cloud.google.com/bigquery ) and requires [authentication ](https://cloud.google.com/bigquery/docs/authentication )
2021-03-19 03:30:24 +08:00
with Google Cloud using either a service account or user credentials.
2021-03-13 01:16:45 +08:00
2021-03-19 03:30:24 +08:00
Be aware that this plugin accesses APIs that are [chargeable ](https://cloud.google.com/bigquery/pricing ) and might incur costs.
2021-03-13 01:16:45 +08:00
2021-11-25 02:47:33 +08:00
## Configuration
2021-03-13 01:16:45 +08:00
```toml
2022-04-08 05:55:03 +08:00
# Configuration for Google Cloud BigQuery to send entries
2021-03-13 01:16:45 +08:00
[[outputs.bigquery]]
2022-04-08 05:55:03 +08:00
## Credentials File
credentials_file = "/path/to/service/account/key.json"
2021-03-13 01:16:45 +08:00
2022-04-08 05:55:03 +08:00
## Google Cloud Platform Project
project = "my-gcp-project"
## The namespace for the metric descriptor
2021-03-13 01:16:45 +08:00
dataset = "telegraf"
## Timeout for BigQuery operations.
# timeout = "5s"
## Character to replace hyphens on Metric name
# replace_hyphen_to = "_"
```
2021-11-25 02:47:33 +08:00
2021-03-19 03:30:24 +08:00
Requires `project` to specify where BigQuery entries will be persisted.
Requires `dataset` to specify under which BigQuery dataset the corresponding metrics tables reside.
2021-11-25 02:47:33 +08:00
Each metric should have a corresponding table to BigQuery.
2021-03-19 03:30:24 +08:00
The schema of the table on BigQuery:
2021-11-25 02:47:33 +08:00
2021-03-19 03:30:24 +08:00
* Should contain the field `timestamp` which is the timestamp of a telegraph metrics
* Should contain the metric's tags with the same name and the column type should be set to string.
* Should contain the metric's fields with the same name and the column type should match the field type.
2021-03-13 01:16:45 +08:00
2021-11-25 02:47:33 +08:00
## Restrictions
2021-03-13 01:16:45 +08:00
Avoid hyphens on BigQuery tables, underlying SDK cannot handle streaming inserts to Table with hyphens.
In cases of metrics with hyphens please use the [Rename Processor Plugin ](https://github.com/influxdata/telegraf/tree/master/plugins/processors/rename ).
In case of a metric with hyphen by default hyphens shall be replaced with underscores (_).
This can be altered using the `replace_hyphen_to` configuration property.
Available data type options are:
2021-11-25 02:47:33 +08:00
2021-03-13 01:16:45 +08:00
* integer
* float or long
* string
* boolean
All field naming restrictions that apply to BigQuery should apply to the measurements to be imported.
Tables on BigQuery should be created beforehand and they are not created during persistence
2021-11-25 02:47:33 +08:00
Pay attention to the column `timestamp` since it is reserved upfront and cannot change.
2021-03-13 01:16:45 +08:00
If partitioning is required make sure it is applied beforehand.
