Fluent Bit is an open source and multi-platform for logs processing and distribution. Nowadays data comes from various sources and Fluent Bit can help you aggregate and process all your log data. Now, Manticore also supports the use of Fluent Bit as a processing pipeline. This allows the collected and transformed data to be sent to Manticore. tool Let’s examine a simple example of indexing , a standard log file of the Debian package manager. The log itself has a simple structure, as shown below: dpkg.log 2023-05-31 10:42:55 status triggers-awaited ca-certificates-java:all 20190405ubuntu1.1
2023-05-31 10:42:55 trigproc libc-bin:amd64 2.31-0ubuntu9.9 <none>
2023-05-31 10:42:55 status half-configured libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 status installed libc-bin:amd64 2.31-0ubuntu9.9
2023-05-31 10:42:55 trigproc systemd:amd64 245.4-4ubuntu3.21 <none> Configuration Here is an example of the Fluent configuration file one can use to work with Manticore: [SERVICE]
	flush    	1
	daemon   	On
	log_level	info

[INPUT]
	name tail
	path /var/log/dpkg.log
	inotify_watcher false
	read_from_head true

[OUTPUT]
	name es
	match *
	host 127.0.0.1
	port 9308
	index  dpkg_log Note that our example is meant to be run in Docker, so we start FluentBit in the daemon mode and with the INPUT inotify_watcher option disabled to avoid possible issues with the Docker environment which can lead to errors. Also, we assume that Manticore is started on the default http port 9308. Results Now you can just run Fluentbit using the config above. The data from the dpkg log will be passed to Manticore and properly indexed. Here is the resulting schema of the created table and an example of the inserted document: mysql> DESCRIBE dpkg_log;
+-------------+--------+----------------+
| Field       | Type   | Properties     |
+-------------+--------+----------------+
| id          | bigint |                |
| @timestamp  | text   | indexed stored |
| log         | text   | indexed stored |
+-------------+--------+----------------+

mysql> SELECT * FROM dpkg_log LIMIT 3\G
*************************** 1. row ***************************
id: 7856533729353662465
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 startup archives install
*************************** 2. row ***************************
id: 7856533729353662466
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 install base-passwd:amd64 <none> 3.5.47
*************************** 3. row ***************************
id: 7856533729353662467
@timestamp: 2023-08-04T15:09:21.191Z
log: 2023-06-05 14:03:04 status half-installed base-passwd:amd64 3.5.47 Conclusion The integration of Manticore with Fluent Bit provides a powerful and efficient solution for handling and indexing log data, making it more accessible and manageable for various applications. With this simple configuration and clear examples provided, even those new to these tools can quickly get started and benefit from the robust capabilities of Manticore and Fluent Bit working together. Whether you’re dealing with standard logs or more complex data sources, this collaboration simplifies the process and opens up new possibilities for effective data management. Also published . here

This writer has a vested interest be it monetary, business, or otherwise, with 1 or more of the products or companies mentioned within.

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Introducing Manticore Integration with Fluentbit

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 Principles of Proper Database Benchmarking

Centralized Logging for AWS ECS in New Relic using FluentBit

Improve System Insights: Build Your Own Log Visualization Pipeline

10 Things I Did To Increase CloudTrail Logs Security

44 Stories To Learn About Logging

5 Tips for Integrating Security into Development: Part 2

10 Principles of Proper Database Benchmarking

Centralized Logging for AWS ECS in New Relic using FluentBit

Improve System Insights: Build Your Own Log Visualization Pipeline

10 Things I Did To Increase CloudTrail Logs Security

44 Stories To Learn About Logging

5 Tips for Integrating Security into Development: Part 2

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps