Fluent Bit is a lightweight and versatile log collector and forwarder designed to handle high-volume data streams from various sources. It is an open-source tool that offers a range of features, including filtering, buffering, and routing capabilities. Fluent Bit is widely used in cloud environments, such as Kubernetes and Docker, for collecting and forwarding logs.
In this guide, I will show you how to use Fluent Bit to send Docker logs to Yandex Cloud Logging. Yandex Cloud Logging is a managed logging service that allows you to store, search, and analyze logs from various sources in one centralized location.
By following this guide, you will learn how to set up and configure Fluent Bit to collect and forward Docker logs to Yandex Cloud Logging. We will cover the necessary steps to set up Fluent Bit and Yandex Cloud Logging, configure Fluent Bit to collect Docker logs, and forward them to Yandex Cloud Logging.
Firstly, install Docker
# Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo addgroup dev docker
Then install libssl from archive (cant be installed from apt-get)
# libssl
wget http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1-1ubuntu2.1~18.04.21_amd64.deb
sudo dpkg -i libssl1.1_1.1.1-1ubuntu2.1~18.04.21_amd64.deb
Install td-agent-bit
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
sudo sh -c 'echo "deb https://packages.fluentbit.io/ubuntu/focal focal main" >> /etc/apt/sources.list'
sudo apt-get update
sudo apt-get install td-agent-bit
sudo systemctl start td-agent-bit
sudo systemctl enable td-agent-bit
sudo systemctl status td-agent-bit
Install GOlang
wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
tar -xzf go1.17.6.linux-amd64.tar.gz
export PATH=$PWD/go/bin:$PATH
Install gcc
# gcc
sudo apt install build-essential
Clone, compile, install fluent-bit-plugin-yandex
# fluent-bit-plugin-yandex
git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git
cd fluent-bit-plugin-yandex/
export fluent_bit_version=1.8.6
export plugin_version=dev
CGO_ENABLED=1 go build -buildmode=c-shared \
-o ./yc-logging.so \
-ldflags "-X main.PluginVersion=${plugin_version}" \
-ldflags "-X main.FluentBitVersion=${fluent_bit_version}"
sudo cp yc-logging.so /usr/lib/td-agent-bit/yc-logging.so
sudo sh -c'echo "\tPath /usr/lib/td-agent-bit/yc-logging.so" >> /etc/td-agent-bit/plugins.conf'
/etc/td-agent-bit/td-agent-bit.conf:
[SERVICE]
log_level debug
[INPUT]
Name forward
Listen 0.0.0.0
port 24224
[OUTPUT]
Name yc-logging
Match *
resource_type logtest
folder_id <FOLDER_ID>
message_key log
level_key SEVERITY
default_level INFO
authorization instance-service-account
sudo systemctl restart td-agent-bit
Now you can try to run container: docker run --rm --log-driver=fluentd hello-world