glusterfs+samba+ctdb 高可用集群

August 27, 2020 默认分类

环境

  • 配置hosts, 时间同步, 关闭selinux, 关闭防火墙

sysOS: centos 8.2
samba 4.12.5-1
gluster 7.7-1
ctdb 4.12.5

服务器

ip地址 hostname 硬盘 挂在路径
172.16.126.163 gfs01 /dev/sdb1 /export/gfsbrick
172.16.126.164 gfs02 /dev/sdb1 /export/gfsbrick
172.16.126.165 gfs03 /dev/sdb1 /export/gfsbrick

vip: 172.16.126.156

172.16.126.163 gfs01
172.16.126.164 gfs02
172.16.126.165 gfs03

配置转发

cat >> /etc/sysctl.conf <<EOF
net.ipv4.ip_forward = 1
EOF
sysctl -p

下载安装包并安装

防止有坑, 先到官网下载对应包及相关依赖

下载samba相关及ctdb 

mkdir ~/samba-412
cd ~/samba-412

下载

wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/c/ctdb-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/l/libsmbclient-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/l/libsmbclient-devel-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/l/libwbclient-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/l/libwbclient-devel-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/p/python3-samba-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-client-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-client-libs-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-common-4.12.5-1.el8.noarch.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-common-libs-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-common-tools-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-devel-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-krb5-printing-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-libs-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-pidl-4.12.5-1.el8.noarch.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-vfs-glusterfs-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-winbind-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-winbind-clients-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-winbind-krb5-locator-4.12.5-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/samba-412/Packages/s/samba-winbind-modules-4.12.5-1.el8.x86_64.rpm

安装

yum localinstall *.rpm -y

下载gluster及相关依赖 

mkdir ~/gluster-7
cd ~/gluster-7

下载

wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-api-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-api-devel-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-cli-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-client-xlators-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-cloudsync-plugins-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-coreutils-0.3.1-3.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-events-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-extra-xlators-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-fuse-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-geo-replication-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-libs-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-rdma-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-server-7.7-1.el8.x86_64.rpm
wget https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/g/glusterfs-thin-arbiter-7.7-1.el8.x86_64.rpm
https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-7/Packages/p/python3-gluster-7.7-1.el8.x86_64.rpm

分区和格式化GlusterFS存储驱动器

在每个GlusterFS节点上，在存储驱动器上格式化并创建文件系统。您可以使用任何分区方法。

# 格式化并在磁盘上创建主分区。  
parted /dev/sdb mklabel msdos
parted -a opt /dev/sdb mkpart primary xfs 0% 100%

在磁盘上创建一个文件系统。这里使用XFS

mkfs.xfs /dev/sdb1 -L gfsdisk

挂载GlusterFS存储驱动器(Bricks)

在每个节点上，为GlusterFS存储驱动器创建一个挂载点，这里称为Bricks。块是GlusterFS存储的基本单位

mkdir -p /export/gfsbrick

将GlusterFS存储驱动器安装在上面创建的bricks目录上 

mount /dev/sdb1 /export/gfsbrick/

要在系统启动时自动挂载驱动器，只需按照以下步骤更新/etc/fstab配置文件 

echo "/dev/sdb1 /export/gfsbrick/ xfs defaults 1 2" >> /etc/fstab

检查安装

df -hTP /dev/sdb1

在CentOS 8上运行GlusterFS

通过执行以下命令，启动并启用GlusterFS守护程序在系统启动时运行；

systemctl enable --now glusterd

检查GlusterFS守护程序的状态；

systemctl status glusterd

配置GlusterFS可信存储池

要配置GlusterFS存储集群，您首先需要创建一个trusted storage pool (TSP)基本上是存储服务器网络的。可以通过使用以下gluster peer probe命令将存储节点添加到TSP来创建TSP 。
为了探测其它节点，例如gfs02和gfs03 从gfs01在我们的情况下，运行下面的命令。相应地替换节点的名称。您可以从任何节点探测其他节点

# 添加节点gfs02
gluster peer probe gfs02
# 添加节点gfs03
gluster peer probe gfs03

要验证群集对等状态，请在任何节点上运行以下命令. 例如，来自节点1的gfs01 

gluster peer status

要列出TSP中的所有节点，例如来自GlusterFS节点02的文件，gfs01

gluster pool list

您可以添加更多服务或从GlusterFS存储池中删除服务器。
例如，要添加新服务器，您需要从池中已经存在的节点中对其进行探测。相应地替换服务器名称。

gluster peer probe server-name

从池中分离服务器

gluster peer detach server-name

设置GlusterFS存储卷

可以从上面创建的砖块创建GlusterFS存储卷。在这种情况下，我们创建了安装/export/gfsbrick在每个节点上的砖块。

GlusterFS存储卷类型

根据各种操作需求，在创建GlusterFS存储卷时需要考虑不同的配置设置。

  • Distributed: 文件分布在卷中的各个块中
  • Replicated: 文件在卷中的各个块之间复制。它确保了高存储可用性和可靠性
  • Distributed Replicated: 文件分布在卷中的所有复制砖中。确保高可靠性，可扩展性和改进的读取性能
  • Arbitrated Replicated: 文件将在副本集中的两个块之间复制，并且仅元数据被复制到第三个块。确保数据一致性
  • Dispersed: 文件分散在该卷的各个块中
  • Distributed Dispersed: 数据分布在分散的子卷中
    可以使用以下命令创建GlusterFS卷
gluster volume  create <NEW-VOLNAME> [stripe <COUNT>] [[replica <COUNT> [arbiter <COUNT>]]|[replica 2 thin-arbiter 1]] [disperse [<COUNT>]] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ...<TA-BRICK>

设置复制的GlusterFS存储卷

创建一个samba的用户组, 后续使用samba共享导出glusterfs时把samba用户加入到该组即可

groupadd samba

在每个群集节点上的brick为卷创建一个安装点。在我们的情况下，我们的brick挂载在/export/gfsbrick

mkdir /export/gfsbrick/gfsvol01
# 授权用户组
chown -R root.samba /export/gfsbrick/gfsvol01
# 只允许用户及用户组进行操作
chmod -R 770 /export/gfsbrick/gfsvol01

接下来，创建一个复制的存储卷。例如，创建具有三个名为的节点的复制的GlusterFS存储卷gfsvol01

gluster volume create gfsvol01 replica 3 transport tcp,rdma gfs01:/export/gfsbrick/gfsvol01 gfs02:/export/gfsbrick/gfsvol01 gfs03:/export/gfsbrick/gfsvol01

您可以从任何节点运行该命令，从而相应地替换其他对等方和关联的模块的名称
如果一切顺利，您应该得到如下输出:

volume create: gfsvol01: success: please start the volume to access data

之后，启动GlusterFS卷。用gfsvol01卷名替换

gluster volume start gfsvol01

验证GlusterFS卷

一旦创建并启动了卷，就可以通过在集群中的任何节点上运行以下命令来进行验证。

gluster volume info all
Volume Name: gfsvol01
Type: Replicate
Volume ID: 54a333fe-054c-4a44-b233-3471a441f822
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gfs01:/export/gfsbrick/gfsvol01
Brick2: gfs02:/export/gfsbrick/gfsvol01
Brick3: gfs03:/export/gfsbrick/gfsvol01
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: off
performance.client-io-threads: off
performance.stat-prefetch: off
server.allow-insecure: on
storage.batch-fsync-delay-usec: 0

至此gluster 卷配置完成

配置smaba对接gluster的gfsvol01

起动／启用 Samba 服务(每个节点)

systemctl start smb.service
systemctl enable smb.service
systemctl start nmb.service
systemctl enable nmb.service

修改ctdb默认samba模板(共享名为glusterfs卷名)

sed -i "s/gluster-//g" /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
sed -i "s/gluster-//g" /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh
sed -i "s/gluster-//g" /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh

配置/etc/samba/smb.conf(每个节点), 例如:

[gfsvol01]
comment = For samba share of volume gfsvol01
vfs objects = glusterfs
glusterfs:volume = gfsvol01
glusterfs:logfile = /var/log/samba/glusterfs-gfsvol01.%M.log
glusterfs:loglevel = 7
path = /
read only = no
kernel share modes = no
  • 注意: 使用 gluster volume set VOLNAME user.smb disable/enable 来生成

导出卷之前的性能调优

针对 Samba 设置 gfsvol01 GlusterFS 扇区：

gluster volume set gfsvol01 stat-prefetch off
volume set: success

防止root用户拥有root特权，而是为他们分配nfsnobody特权

gluster volume set q1autoops server.root-squash on
volume set: success

允许非安全访问

gluster volume set gfsvol01 server.allow-insecure on
volume set: success

确保锁定和IO一致性

gluster volume set gfsvol01 storage.batch-fsync-delay-usec 0
volume set: success

预读取

gluster volume set gfsvol01 performance.read-ahead on
volume set: success

io线程数大小

gluster volume set gfsvol01 performance.io-thread-count 64
volume set: success

缓存已经被读过的数据

gluster volume set gfsvol01 performance.io-cache on
volume set: success

优化读取小文件的性能

gluster volume set gfsvol01 performance.quick-read on
volume set: success

网络检测超时

gluster volume set gfsvol01 network.ping-timeout 10
volume set: success

设置预读取的数量

gluster volume set gfsvol01 performance.read-ahead-page-count 16
volume set: success

开启目录索引的自动愈合进程

gluster volume set gfsvol01 cluster.self-heal-daemon on
volume set: success

自动愈合的检测间隔

gluster volume set gfsvol01 cluster.heal-timeout 300
volume set: success

设置客户端进程和服务器进程同时处理的网路连接数

gluster volume set gfsvol01 client.event-threads 32
volume set: success
gluster volume set gfsvol01 server.event-threads 32
volume set: success

如果您使用的是gluster5或更高版本，请执行以下操作以提高性能

gluster volume set gfsvol01 group samba

在旧版本上，请执行以下操作

gluster volume set gfsvol01 features.cache-invalidation on
gluster volume set gfsvol01 features.cache-invalidation-timeout 600
gluster volume set gfsvol01 performance.cache-samba-metadata on
gluster volume set gfsvol01 performance.stat-prefetch on
gluster volume set gfsvol01 performance.cache-invalidation on
gluster volume set gfsvol01 performance.md-cache-timeout 600
gluster volume set gfsvol01 network.inode-lru-limit 200000
gluster volume set gfsvol01 performance.nl-cache on
gluster volume set gfsvol01 performance.nl-cache-timeout 600
gluster volume set gfsvol01 performance.readdir-ahead on
gluster volume set gfsvol01 performance.parallel-readdir on

调整gluster中的线程数以获得更好的性能
``bash
gluster volume set gfsvol01 client.event-threads 4
gluster volume set gfsvol01 server.event-threads 4


在每个节点的 /etc/glusterfs/glusterd.vol 内加入以下内容：
```bash
option rpc-auth-allow-insecure on

重新起动 glusterfs 服务：

systemctl restart glusterd.service

使用SMB挂载卷

定义一个新的 samba 用户(每个节点都需要添加)：

adduser sambauser -g samba -s /sbin/nologin
smbpasswd -a sambauser

重新起动 Samba

systemctl restart smb.service
systemctl restart nmb.service

配置CTDB(samba高可用)

配置ctdb使用的共享

在/export/gfsbrick/ctdb创建ctdb文件夹, 使集群中的主机可以共享配置文件

mkdir /export/gfsbrick/ctdb

接下来，创建一个复制的存储卷。例如，创建具有三个名为的节点的复制的GlusterFS存储卷ctdb
本卷将存放唯一一个零字节锁定文件(用于判断故障)，因此选择最小容量的砖

gluster volume create ctdb replica 3 transport tcp,rdma gfs01:/export/gfsbrick/ctdb gfs02:/export/gfsbrick/ctdb gfs03:/export/gfsbrick/ctdb

您可以从任何节点运行该命令，从而相应地替换其他对等方和关联的模块的名称
如果一切顺利，您应该得到如下输出:

volume create: ctdb: success: please start the volume to access data

由于ctdb卷存放唯一一个零字节锁定文件及ctdb配置文件, 限制其大小

gluster volume quota ctdb enable
gluster volume quota ctdb limit-usage / 2GB

修改ctdb默认模板, 使自动挂载

sed -i 's/META="all"/META="ctdb"/' /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
sed -i 's/META="all"/META="ctdb"/' /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh

在/etc/samba/smb.conf文件中添加以下内容到全局部分

    clustering = yes
    kernel share modes = no
    kernel oplocks = no
    map archive = no
    map hidden = no
    map read only = no
    map system = no
    store dos attributes = yes

之后，启动GlusterFS卷。用ctdb卷名替换

gluster volume start ctdb

验证GlusterFS卷

一旦创建并启动了卷，就可以通过在集群中的任何节点上运行以下命令来进行验证。

gluster volume info all
Volume Name: ctdb
Type: Replicate
Volume ID: fabe669e-2ac4-4b5c-b55e-ef4c8d92758a
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gfs01:/export/gfsbrick/ctdb
Brick2: gfs02:/export/gfsbrick/ctdb
Brick3: gfs03:/export/gfsbrick/ctdb
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off

Volume Name: gfsvol01
Type: Replicate
Volume ID: 54a333fe-054c-4a44-b233-3471a441f822
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gfs01:/export/gfsbrick/gfsvol01
Brick2: gfs02:/export/gfsbrick/gfsvol01
Brick3: gfs03:/export/gfsbrick/gfsvol01
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: off
performance.client-io-threads: off
performance.stat-prefetch: off
server.allow-insecure: on
storage.batch-fsync-delay-usec: 0

配置ctdb

将ctdb的配置文件/etc/sysconfig/ctdb, /etc/ctdb/public_addresses, /etc/ctdb/nodes也存放于/lock上，并为所有节点建立符号链接
ctdb文件

CTDB_RECOVERY_LOCK=/lock/lockfile
CTDB_PUBLIC_INTERFACE=eth0
CTDB_PUBLIC_ADDRESSES=/etc/ctdb/public_addresses
CTDB_MANAGES_SAMBA=yes
CTDB_MANAGES_WINBIND=yes
CTDB_MANAGES_NFS=no
CTDB_NODES=/etc/ctdb/nodes

public_addresses文件

172.16.125.30/24 eth0

nodes文件

172.16.125.27
172.16.125.28
172.16.125.29
ln -sf /gluster/lock/ctdb /etc/sysconfig/ctdb
ln -sf /gluster/lock/public_addresses /etc/ctdb/public_addresses
ln -sf /gluster/lock/nodes /etc/ctdb/nodes

配置系统启动项, CTDB自动管理smb服务

systemctl disable smb
systemctl stop smb

修改ctdb启动项, 等待glusterd.service和gluster-lock.mount正常后启动

sed -i '/After/ s/$/ glusterd.service gluster-lock.mount/' /usr/lib/systemd/system/ctdb.service
systemctl daemon-reload

修改ctdb服务, 意外停止自动启动

sed -i "s/Restart=no/Restart=always/g" /usr/lib/systemd/system/ctdb.service
systemctl daemon-reload
systemctl daemon-reload
systemctl enable ctdb
systemctl start ctdb

查看服务状态

ctdb status
ctdb ip
ctdb ping -n all

测试

mount -t cifs -o username=sambauser,password=sambauser //172.16.125.30/gluster-gfsvol01 /mnt/

已知问题

  1. gluster>=3.x.x 不支持直接暴露nfs
  2. peer probe: failed: Peer uuid (host node2) is same as local uuid
    删除 /var/lib/glusterd/glusterd.info, 重启glusterd

参考:

添加新评论