From 73f80f4adfb017c14a010e3bb9755c6b79e2697f Mon Sep 17 00:00:00 2001 From: Sven Rebhan <36194019+srebhan@users.noreply.github.com> Date: Fri, 1 Mar 2024 13:13:23 +0100 Subject: [PATCH] docs(specs): Add specification for custom-builder (#14901) Co-authored-by: Joshua Powers Co-authored-by: Dane Strandboge <136023093+DStrand1@users.noreply.github.com> --- docs/specs/tsd-002-custom-builder.md | 71 ++++++++++++++++++++++++++++ tools/custom_builder/README.md | 57 +++++++++++++++------- 2 files changed, 110 insertions(+), 18 deletions(-) create mode 100644 docs/specs/tsd-002-custom-builder.md diff --git a/docs/specs/tsd-002-custom-builder.md b/docs/specs/tsd-002-custom-builder.md new file mode 100644 index 000000000..3df714e96 --- /dev/null +++ b/docs/specs/tsd-002-custom-builder.md @@ -0,0 +1,71 @@ +# Telegraf Custom-Builder + +## Objective + +Provide a tool to build a customized, smaller version of Telegraf with only +the required plugins included. + +## Keywords + +tool, binary size, customization + +## Overview + +The Telegraf binary continues to grow as new plugins and features are added +and dependencies are updated. Users running on resource constrained systems +such as embedded-systems or inside containers might suffer from the growth. + +This document specifies a tool to build a smaller Telegraf binary tailored to +the plugins configured and actually used, removing unnecessary and unused +plugins. The implementation should be able to cope with configured parsers and +serializers including defaults for those plugin categories. Valid Telegraf +configuration files, including directories containing such files, are the input +to the customization process. + +The customization tool might not be available for older versions of Telegraf. +Furthermore, the degree of customization and thus the effective size reduction +might vary across versions. The tool must create a single static Telegraf +binary. Distribution packages or containers are *not* targeted. + +## Prior art + +[PR #5809](https://github.com/influxdata/telegraf/pull/5809) and +[telegraf-lite-builder](https://github.com/influxdata/telegraf/tree/telegraf-lite-builder/cmd/telegraf-lite-builder): + +- Uses docker +- Uses browser: + - Generates a webpage to pick what options you want. User chooses plugins; + does not take a config file + - Build a binary, then minifies by stripping and compressing that binary +- Does some steps that belong in makefile, not builder + - Special case for upx + - Makes gzip, zip, tar.gz +- Uses gopkg.in? +- Can also work from the command line + +[PR #8519](https://github.com/influxdata/telegraf/pull/8519) + +- User chooses plugins OR provides a config file + +[powers/telegraf-build](https://github.com/powersj/telegraf-build) + +- User chooses plugins OR provides a config file +- Currently kept in separate repo +- Undoes changes to all.go files + +[rawkode/bring-your-own-telegraf](https://github.com/rawkode/bring-your-own-telegraf) + +- Uses docker + +## Additional information + +You might be able to further reduce the binary size of Telegraf by removing +debugging information. This is done by adding `-w` and `-s` to the linker flags +before building `LDFLAGS="-w -s"`. + +However, please note that this removes information helpful for debugging issues +in Telegraf. + +Additionally, you can use a binary packer such as [UPX](https://upx.github.io/) +to reduce the required *disk* space. This compresses the binary and decompresses +it again at runtime. However, this does not reduce memory footprint at runtime. diff --git a/tools/custom_builder/README.md b/tools/custom_builder/README.md index 710b608c0..a8de796a3 100644 --- a/tools/custom_builder/README.md +++ b/tools/custom_builder/README.md @@ -4,6 +4,29 @@ Telegraf's `custom_builder` is a tool to select the plugins compiled into the Telegraf binary. By doing so, Telegraf can become smaller, saving both disk space and memory if only a sub-set of plugins is selected. +## Requirements + +For compiling the customized binary you need the +[Golang language](https://go.dev/) as well as the `make` build system. +The minimum required version of Golang can be found in the *Build From Source* +section of the `README.md` file of your version. Both the `go` and the `make` +command must be available in your path. + +## Downloading code + +The first step is to download the Telegraf repository for the version you are +planning to customize. In the example below, we want to use `v1.29.5` but you +might also use other versions or `master`. + +```shell +# git clone --branch v1.29.5 --single-branch https://github.com/influxdata/telegraf.git +... +# cd telegraf +``` + +Alternatively, you can download the source tarball or zip-archive of a +[Telegraf release](https://github.com/influxdata/telegraf/releases). + ## Building To build `custom_builder` run the following command: @@ -16,18 +39,17 @@ The resulting binary is located in the `tools/custom_builder` folder. ## Running -The easiest way of building a customized Telegraf is to use your -Telegraf configuration file(s). Assuming your configuration is -in `/etc/telegraf/telegraf.conf` you can run +The easiest way of building a customized Telegraf is to use your Telegraf +configuration file(s). Assuming your configuration is in +`/etc/telegraf/telegraf.conf` you can run ```shell # ./tools/custom_builder/custom_builder --config /etc/telegraf/telegraf.conf ``` -to build a Telegraf binary tailored to your configuration. -You can also specify a configuration directory similar to -Telegraf itself. To additionally use the configurations in -`/etc/telegraf/telegraf.d` run +to build a Telegraf binary tailored to your configuration. You can also specify +a configuration directory similar to Telegraf itself. To additionally use the +configurations in `/etc/telegraf/telegraf.d` run ```shell # ./tools/custom_builder/custom_builder \ @@ -35,8 +57,8 @@ Telegraf itself. To additionally use the configurations in --config-dir /etc/telegraf/telegraf.d ``` -Configurations can also be retrieved from remote locations just -like for Telegraf. +Configurations can also be retrieved from remote locations just like for +Telegraf. ```shell # ./tools/custom_builder/custom_builder --config http://myserver/telegraf.conf @@ -44,11 +66,10 @@ like for Telegraf. will download the configuration from `myserver`. -The `--config` and `--config-dir` option can be used multiple times. -In case you want to deploy Telegraf to multiple systems with -different configurations, simply specify the super-set of all -configurations you have. `custom_builder` will figure out the list -for you +The `--config` and `--config-dir` option can be used multiple times. In case +you want to deploy Telegraf to multiple systems with different configurations, +simply specify the super-set of all configurations you have. `custom_builder` +will figure out the list for you ```shell # ./tools/custom_builder/custom_builder \ @@ -74,8 +95,8 @@ To get more help run ## Notes -Please make sure to include all `parsers` you intend to use and check the -enabled-plugins list. +Please make sure to include all `parsers` and `serializers` you intend to use +and check the enabled-plugins list. -Additional plugins can potentially be enabled automatically due to -dependencies without being shown in the enabled-plugins list. +Additional plugins can potentially be enabled automatically due to dependencies +without being shown in the enabled-plugins list.