What is a Logging Agent like Fluentd?
Logging Agent is a tool that is installed along with the running application and its role is to collect the logs from specified file paths, process them and finally send them to multiple destinations. There are many tools available for this purpose like Fluentd, Filebeat, Logstash, etc. Fluentd is an open-source Logging Agent and lets us unify data collection and consumption with various features. It has a wide variety of parsers available to collect different log types like JSON, Nginx, tomcat, etc. and we can also write our own parsers.
Today's Agenda
In this post, we will learn to create a Shell Script to Install and Configure our Fluentd agent. This script becomes really helpful in cases where we have multiple servers on which we need Fluentd to be installed. Also, it will detect the Ubuntu version and run the installation accordingly.
Prerequisite
This post has been prepared for the audience who :
- Have an access to a Linux based machine, we have used Ubuntu OS.
- Have a basic knowledge of Linux based systems.
- And finally, are eager to learn and try something new.
Let's get started
Step 1: Create a new file to store the shell script.
Create a file with name SetupLoggingAgent-Fluentd.sh and add the below script to it.
#!/bin/bash
if [ "${1}" == "-h" ] || [ "${1}" == "-help" ] || [ "${1}" == "" ]
then
echo "HELP SECTION FOR THIS SCRIPT!"
echo "This script needs user with root permissions to get executed."
echo -e "Fluentd Installation Script\n Purpose: \n This Script will perform the following functions:
-- Install Fluentd with required input parsers & output.
Usage:
FOR HELP:
bash SetupLoggingAgent.sh -help
bash SetupLoggingAgent.sh -h
bash SetupLoggingAgent.sh
TO BEGIN INSTALLATION:
bash SetupLoggingAgent.sh -start <Elasticsearch Endpoint or URL>"
exit 0
fi
# Verifying user for sudo permissions
if [ "${1}" == "-start" ] && [ "${2}" != "" ]
then
sudo -n true
if [ $? -ne 0 ]
then
echo "Permission Denied! Try executing this script with root user permissions or try with sudo."
echo "For more information about this file use -h or -help along with the above command."
exit 3
fi
set -e
create_config() {
## Create td-agent.conf file
cat > ./td-agent.conf << EOF
# tomcat access logs
<source>
type tail
format apache2
tag apache.access
path /var/log/apache2/access.log
</source>
# tomcat error logs
<source>
type tail
format /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\] \[pid (?<pid>[^\]]*)\] \[client (?<client>[^\]]*)\] (?<message>.*)$/
tag apache.error
path /var/log/apache2/error.log
</source>
# nginx access logs
<source>
type tail
format nginx
tag nginx.access
path /var/log/nginx/access.log
</source>
# nginx error logs
<source>
type tail
tag nginx.error
path /var/log/nginx/error.log
format multiline
format_firstline /^\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2} \[\w+\] (?<pid>\d+).(?<tid>\d+): /
format1 /^(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<log_level>\w+)\] (?<pid>\d+).(?<tid>\d+): (?<message>.*)/
multiline_flush_interval 3s
</source>
# json log type
<source>
@type tail
path /var/log/httpd-access.log #...or where you placed your Apache access log
pos_file /var/log/td-agent/httpd-access.log.pos # This is where you record file position
tag json.logs #fluentd tag!
format json # one JSON per line
time_key time_field # optional; default = time
</source>
<match apache.*>
@type elasticsearch
logstash_format true
host ESENDPOINT # elasticsearch endpoint
port 80
index_name tomcat-logs
type_name tomcat-logs
</match>
<match nginx.*>
@type elasticsearch
logstash_format true
host ESENDPOINT # elasticsearch endpoint
port 80
index_name nginx-logs
type_name nginx-logs
</match>
<match json.*>
@type elasticsearch
logstash_format true
host ESENDPOINT # elasticsearch endpoint
port 80
index_name json-logs
type_name json-logs
</match>
EOF
}
add_config() {
echo -e "Completed Installation of Fluentd\n Moving default configuration file of Fluentd to /tmp directory of your system\n Replacing default configuration file\n Restaring Fluentd service."
echo ${2}
ESEndpointVariable="${1}"
# Replace the Elasticsearch variable with actual endpoint
sudo sed -i "s/ESENDPOINT/$ESEndpointVariable/g" td-agent.conf
sudo mv /etc/td-agent/td-agent.conf /tmp
sudo cp ./td-agent.conf /etc/td-agent/td-agent.conf
# Run td-agent as root user
sed -i 's/User=td-agent/User=root/g' /lib/systemd/system/td-agent.service
sudo systemctl daemon-reload
sudo systemctl restart td-agent.service
sudo systemctl enable td-agent.service
echo -e "All Installations Completed. \nYou are ready to go!"
}
if [ "$(grep -Ei 'Ubuntu 18.04' /etc/*release)" ]
then
echo -e "It's a Ubuntu 18.04 LTS (Bionic Beaver) based system."
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh
create_config
add_config "${2}"
elif [ "$(grep -Ei 'Ubuntu 20.04' /etc/*release)" ]
then
echo -e "It's a Ubuntu 20.04 LTS (Focal Fossa) based system."
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-focal-td-agent4.sh | sh
create_config
add_config "${2}"
elif [ "$(grep -Ei 'Ubuntu 16.04' /etc/*release)" ]
then
echo -e "It's a Ubuntu 16.04 LTS (Xenial Xerus) based system."
curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent4.sh | sh
create_config
add_config "${2}"
else
echo "This script doesn't support this OS."
exit 3
fi
fi |
Step 2 : Run this Shell Script to begin Installation.
This Shell Script needs to be run as a root user.
To know how this shell script works, enter the below command:
bash SetupLoggingAgent-Fluentd.sh -help
Add the asked variables and run the Shell Script as the root user.
sudo bash SetupLoggingAgent-Fluentd.sh -start <Elasticsearch Endpoint or URL>
This script is designed to run only on Ubuntu-based machines.
That's all, you did a great job !!
If you face any issues or need any suggestions, please comment down below and hit the like button to appreciate the efforts.
Comments
Post a Comment