分布式文件系統mogilefs的簡單應用

簡介

MogileFS是一個開源的分布式文件存儲系統，由LiveJournal旗下的DangaInteractive公司開發。Danga團隊開發了包括Memcached、MogileFS、Perlbal等多個知名的開源項目。

特性

 1、應用層：用戶空間文件系統，無須特殊的核心組件

  2、無單點：（tracker, mogstore, database(MySQL)）

  3、自動文件復制：復制的最小單位不是文件，而class；

  4、傳輸中立，無特殊協議：可以通過NFS或HTTP進行通信；

   5、簡單的命名空間： 每個文件對應一個key：用于domain定義名稱空間

組成部分

MogileFS：

     tracker：追蹤元數據

     database：存儲元數據

     storage：存儲數據

 tracker：

        mogilefsd（守護進程）,職責：

         replication：節點之間的文件復制

         deletion：刪除文件

         queryworker：響應客戶請求的文件元數據訪問請求

         reaper：在存儲失敗后將文件復制請求重新放置于隊列中

         monitor：檢測主機和設備的健康狀態

database：

      存儲mogilefs的元數據，一般使用mysql，建議使用冗余方案保證可用性（MMM,MHA）

      mogilefs專門提供了數據結構管理工具mogdbsetup；

storage：

      mogstored（進程名），一個準備停當的mogstored節點可通過mogadm命令添加至現在的集群中，

      存儲節點需定義“設備dev”用作存儲空間；每個設備在當前集群中需通過一個唯一的DevID標識。

client：

            客戶端用于與mogilefs簡歷通信，完成數據存取；

安裝包：

MogileFS-Server-2.46-2.el6.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm

MogileFS-Utils-2.19-1.el6.noarch.rpm

perl-MogileFS-Client-1.14-1.el6.noarch.rpm

perl-Net-Netmask-1.9015-8.el6.noarch.rpm

perl-Perlbal-1.78-1.el6.noarch.rpm

架構：

1.安裝mogilefs和數據庫

# ls

MogileFS-Server-2.46-2.el6.noarch.rpm            perl-MogileFS-Client-1.14-1.el6.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm  perl-Net-Netmask-1.9015-8.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm  perl-Perlbal-1.78-1.el6.noarch.rpm

MogileFS-Utils-2.19-1.el6.noarch.rpm

# yum localinstall *.rpm  -y

# yum install mysql-server mysql -y

2.數據庫授權遠程訪問，并繼承授權權限

mysql> grant all on *.* to root@'192.168.95.%' identified by 'liaobin' with grant option;

mysql> flush privileges;

3.創建mofilefs的數據庫用戶以及授權

mysql> grant all on *.* to mguser@'192.168.95.%' identified by 'liaobin';

mysql> flush privileges;

4.用mogilefs自帶工具mogdbsetup初始化mogilefs數據庫

# mogdbsetup  --dbhost=192.168.95.30 --dbname=mogilefs --dbrootuser=root --dbrootpass=liaobin --dbuser=mguser --dbpass=liaobin --yes

查看初始化的數據庫

mysql> show databases;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mogilefs           |

| mysql              |

| test               |

+--------------------+

4 rows in set (0.01 sec)

5.修改tracker（mogilefsd）配置文件

# vim /etc/mogilefs/mogilefsd.conf

# Enable daemon mode to work in background and use syslog

daemonize = 1

# Where to store the pid of the daemon (must be the same in the init script)

pidfile = /var/run/mogilefsd/mogilefsd.pid

# Database connection information

db_dsn = DBI:mysql:mogilefs:host=192.168.95.30   #配置database的連接信息

db_user = mguser                                                            #配置mogilefs的數據庫用戶密碼

db_pass = liaobin

# IP:PORT to listen on for mogilefs client requests

listen = 0.0.0.0:7001                                          #配置監聽的地址和端口

# Optional, if you don't define the port above.

conf_port = 7001

# Number of query workers to start by default.

query_jobs = 10                                                               #生成多少個用于查詢的工作進程

# Number of delete workers to start by default.

delete_jobs = 1                                                                #生成多少個用于刪除的工作進程

# Number of replicate workers to start by default.

replicate_jobs = 5                                                           #生成多少個用于刪除的工作進程

# Number of reaper workers to start by default.

# (you don't usually need to increase this)

reaper_jobs = 1                                                              

# Number of fsck workers to start by default.

# (these can cause a lot of load when fsck'ing)

#fsck_jobs = 1

# Minimum amount of space to reserve in megabytes

# default: 100

# Consider setting this to be larger than the largest file you

# would normally be uploading.

#min_free_space = 200

# Number of seconds to wait for a storage node to respond.

# default: 2

# Keep this low, so busy storage nodes are quickly ignored.

#node_timeout = 2

# Number of seconds to wait to connect to a storage node.

# default: 2

# Keep this low so overloaded nodes get skipped.

#conn_timeout = 2

# Allow replication to use the secondary node get port,

# if you have apache or similar configured for GET's

#repl_use_get_port = 1

6.啟動mogilefsd服務

[root@localhost mgfs]# /etc/init.d/mogstored start

7.為每個節點新增加分區，作為存儲

# fdisk /dev/sda

.........

# partx -a /dev/sda

# mkfs.ext4 /dev/sda5

# mkdir /mgfs

# vim /etc/fstab

#新增如下條目

/dev/sda5                /mgfs                        ext4        defaults        0 0

# mount -a

# mount

/dev/sda5 on /mgfs type ext4 (rw)

8.三臺服務器分別創建存儲目錄，并修改權限

95.30：

# mkdir /mgfs/dev1

# chown -R /mgfs/dev1

95.34：

# mkdir /mgfs/dev2

# chown -R /mgfs/dev2

95.35：

# mkdir /mgfs/dev3

# chown -R /mgfs/dev3

9.修改store(mogstored)的配置文件

#vim /etc/mogilefs/mogstored.conf

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

#注意此處，寫到dev(1,2,3)的上級目錄就好

docroot = /mgfs

10.啟動mogstored服務

# /etc/init.d/mogstored start

11.用mogadm管理

創建存儲主機：

# mogadm --trackers=192.168.95.30:7001 host add store1 --ip=192.168.95.30

# mogadm --trackers=192.168.95.30:7001 host add store2 --ip=192.168.95.34

# mogadm --trackers=192.168.95.30:7001 host add store3 --ip=192.168.95.35

查看創建的主機：

# mogadm --trackers=192.168.95.30:7001 host list

創建存儲設備：

# mogadm --trackers=192.168.95.30:7001 device add store1 1 --status=alive

# mogadm --trackers=192.168.95.30:7001 device add store2 2 --status=alive

# mogadm --trackers=192.168.95.30:7001 device add store3 3 --status=alive

查看創建的存儲設備：

# mogadm --trackers=192.168.95.30:7001 device list

創建域domain：

mogadm --trackers=192.168.95.30:7001 domain add img

mogadm --trackers=192.168.95.30:7001 domain add conf

查看創建的域：

mogadm --trackers=192.168.95.30:7001 domain list

上傳文件測試：

mogupload --trackers=192.168.95.30:7001 --domain=img --key="1.jpg" --file="/root/1.jpg"

查看上傳的文件：

# mogfileinfo --trackers=192.168.95.30:7001 --domain=img --key="1.jpg"

- file: 1.jpg

     class:              default

  devcount:                    2

    domain:                  img

       fid:                    9

       key:                1.jpg

    length:                34892

 - http://192.168.95.35:7500/dev3/0/000/000/0000000009.fid

 - http://192.168.95.30:7500/dev1/0/000/000/0000000009.fid

用fid的url訪問測試：

正常訪問

nginx反向代理mogilefs

1.安裝nginx和mogilefs模塊

安裝好開發環境：

# yum groupinstall "Development tools" "Server Platform Development" -y

#  yum install pcre-devel -y

下載解壓nginx包和mogilefs包：

開始編譯安裝：

#cd nginx-1.6.2

# ./configure \

  --prefix=/usr/local/nginx \

  --sbin-path=/usr/local/nginx/sbin/nginx \

  --conf-path=/etc/nginx/nginx.conf \

  --error-log-path=/var/log/nginx/error.log \

  --http-log-path=/var/log/nginx/access.log \

  --pid-path=/var/run/nginx/nginx.pid  \

  --lock-path=/var/lock/nginx.lock \

  --user=nginx \

  --group=nginx \

  --with-http_ssl_module \

  --with-http_flv_module \

  --with-http_stub_status_module \

  --with-http_gzip_static_module \

  --http-client-body-temp-path=/var/tmp/nginx/client/ \

  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \

  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \

  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \

  --http-scgi-temp-path=/var/tmp/nginx/scgi \

  --with-pcre --add-module=../nginx_mogilefs_module-1.0.4

# make && make install

給nginx創建服務腳本

# vim /etc/init.d/nginx

#!/bin/sh

#

# nginx - this script starts and stops the nginx daemon

#

# chkconfig:   - 85 15

# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \

#               proxy and IMAP/POP3 proxy server

# processname: nginx

# config:      /etc/nginx/nginx.conf

# config:      /etc/sysconfig/nginx

# pidfile:     /var/run/nginx.pid

# Source function library.

. /etc/rc.d/init.d/functions

# Source networking configuration.

. /etc/sysconfig/network

# Check that networking is up.

[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/local/nginx/sbin/nginx"

prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {

   # make required directories

   user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`

   options=`$nginx -V 2>&1 | grep 'configure arguments:'`

   for opt in $options; do

       if [ `echo $opt | grep '.*-temp-path'` ]; then

           value=`echo $opt | cut -d "=" -f 2`

           if [ ! -d "$value" ]; then

               # echo "creating" $value

               mkdir -p $value && chown -R $user $value

           fi

       fi

   done

}

start() {

    [ -x $nginx ] || exit 5

    [ -f $NGINX_CONF_FILE ] || exit 6

    make_dirs

    echo -n $"Starting $prog: "

    daemon $nginx -c $NGINX_CONF_FILE

    retval=$?

    echo

    [ $retval -eq 0 ] && touch $lockfile

    return $retval

}

stop() {

    echo -n $"Stopping $prog: "

    killproc $prog -QUIT

    retval=$?

    echo

    [ $retval -eq 0 ] && rm -f $lockfile

    return $retval

}

restart() {

    configtest || return $?

    stop

    sleep 1

    start

}

reload() {

    configtest || return $?

    echo -n $"Reloading $prog: "

    killproc $nginx -HUP

    RETVAL=$?

    echo

}

force_reload() {

    restart

}

configtest() {

  $nginx -t -c $NGINX_CONF_FILE

}

rh_status() {

    status $prog

}

rh_status_q() {

    rh_status >/dev/null 2>&1

}

case "$1" in

    start)

        rh_status_q && exit 0

        $1

        ;;

    stop)

        rh_status_q || exit 0

        $1

        ;;

    restart|configtest)

        $1

        ;;

    reload)

        rh_status_q || exit 7

        $1

        ;;

    force-reload)

        force_reload

        ;;

    status)

        rh_status

        ;;

    condrestart|try-restart)

        rh_status_q || exit 0

            ;;

    *)

        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"

        exit 2

esac

給予執行權限：

# chmod +x /etc/init.d/nginx

創建nginx用戶和必須目錄：

# useradd -r nginx

# mkdir /var/tmp/nginx/client -pv

修改nginx配置文件讓其支持mogilefs反向代理（紅色字體為添加項）：

# vim /etc/nginx/nginx.conf

#user  nobody;

worker_processes  1;

#error_log  logs/error.log;

#error_log  logs/error.log  notice;

#error_log  logs/error.log  info;

#pid        logs/nginx.pid;

events {

    worker_connections  1024;

}

http {

    include       mime.types;

    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '

    #                  '$status $body_bytes_sent "$http_referer" '

    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;

    #tcp_nopush     on;

    #keepalive_timeout  0;

    keepalive_timeout  65;

    #gzip  on;

    upstream stores {

server 192.168.95.30:7001;

server 192.168.95.34:7001;

server 192.168.95.35:7001;

    }

    server {

        listen       80;

        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {

            root   html;

            index  index.html index.htm;

        }

location /img/ {

mogilefs_tracker stores;

mogilefs_domain img;

                 mogilefs_pass {

proxy_pass $mogilefs_path;

proxy_hide_header Content-Type;

proxy_buffering off;

}

}

location /upload/ {

mogilefs_tracker stores;

mogilefs_domain img;

mogilefs_methods PUT GET DELETE;

                 mogilefs_pass {

proxy_pass $mogilefs_path;

proxy_hide_header Content-Type;

proxy_buffering off;

}

}

        #error_page  404              /404.html;

}

用路徑訪問測試nginx代理：

再上傳一個文件測試：

# mogupload --trackers=192.168.95.30:7001 --domain=img --key="2.jpg" --file="/root/2.jpg"

測試刪除文件：

# curl -X DELETE http://192.168.95.34/upload/1.jpg