使用Fluentd + MongoDB構建實時日志收集系統

機制圖解

安裝

• Fluentd with MongoDB Plugin
• MongoDB
• Apache (with the Combined Log Format)

gem install fluent-plugin-mongo

• Debian Package
• RPM Package
• MongoDB Downloads

配置

<source>
  type tail
  format apache
  path /var/log/apache2/access_log
  tag mongo.apache
</source>

1. type tail: tail方式是 Fluentd 內置的輸入方式，其原理是不停地從源文件中獲取新的日志。
2. format apache: 指定使用 Fluentd 內置的 Apache 日志解析器。
3. path /var/log/apache2/access_log: 指定日志文件位置。
4. tag mongo.apache: 指定tag，tag被用來對不同的日志進行分類

<match mongo.**>
  # plugin type
  type mongo
  # mongodb db + collection
  database apache
  collection access
  # mongodb host + port
  host localhost
  port 27017
  # interval
  flush_interval 10s
</match>

測試

$ ab -n 100 -c 10 http://localhost/

$ mongo
> use apache
> db.access.find()
{ "_id" : ObjectId("4ed1ed3a340765ce73000001"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:27Z") }
{ "_id" : ObjectId("4ed1ed3a340765ce73000002"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:34Z") }
{ "_id" : ObjectId("4ed1ed3a340765ce73000003"), "host" : "127.0.0.1", "user" : "-", "method" : "GET", "path" : "/", "code" : "200", "size" : "44", "time" : ISODate("2011-11-27T07:56:34Z") }