colben/www.colben.cn
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
colben 2024-04-17 20:44:50 +08:00
parent bc68d63a8f
commit c549c384a7
1 changed files with 69 additions and 93 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

162
content/post/minio.md
View File

@ -1,7 +1,7 @@
--- ---
title: "Minio 笔记" title: "Minio 笔记"
date: 2021-10-17T00:48:00+08:00 date: 2021-10-17T00:48:00+08:00
lastmod: 2021-10-17T00:48:00+08:00 lastmod: 2024-04-17T20:08:00+08:00
keywords: [] keywords: []
tags: ["minio"] tags: ["minio"]
categories: ["storage"] categories: ["storage"]
@ -65,38 +65,80 @@ categories: ["storage"]
## 部署多节点分布式存储 ## 部署多节点分布式存储
### 环境 ### 环境
自定义域名 | 网卡 IP | 硬盘槽 | 数据分区 自定义域名 | 网卡 IP | 数据盘
---- | ---- | ---- | ---- ---- | ---- | ----
X1.X.X | eth0: 192.168.1.11<br>eth1:10.0.0.11 | 8 个 | /dev/sdb1, /dev/sdc1<br>/dev/sdd1, /dev/sde1 X1.X.X | eth0: 192.168.1.11<br>eth1:10.0.0.11 | /dev/sdb, /dev/sdc<br>/dev/sdd, /dev/sde
X2.X.X | eth0: 192.168.1.12<br>eth1:10.0.0.12 | 8 个 | /dev/sdb1, /dev/sdc1<br>/dev/sdd1, /dev/sde1 X2.X.X | eth0: 192.168.1.12<br>eth1:10.0.0.12 | /dev/sdb, /dev/sdc<br>/dev/sdd, /dev/sde
X3.X.X | eth0: 192.168.1.13<br>eth1:10.0.0.13 | 8 个 | /dev/sdb1, /dev/sdc1<br>/dev/sdd1, /dev/sde1 X3.X.X | eth0: 192.168.1.13<br>eth1:10.0.0.13 | /dev/sdb, /dev/sdc<br>/dev/sdd, /dev/sde
X4.X.X | eth0: 192.168.1.14<br>eth1:10.0.0.14 | 8 个 | /dev/sdb1, /dev/sdc1<br>/dev/sdd1, /dev/sde1 X4.X.X | eth0: 192.168.1.14<br>eth1:10.0.0.14 | /dev/sdb, /dev/sdc<br>/dev/sdd, /dev/sde
- eth0 用于接收 api 请求 - eth0 用于接收 api 请求
- eth1 用于集群内部通信 - eth1 用于集群内部通信
- 每台服务器有 8 个硬盘槽，这里先各用 4 个组成 server pool，运行 minio 集群 - 操作系统：RockyLinux 9.3
- 每台服务器剩下的 4 个硬盘槽用于后面的扩容操作
### 挂载数据分区 ### 挂载数据盘
- 在全部服务器上执行如下操作 - 在全部服务器上执行如下操作
- 挂载硬盘 - 格式化硬盘，设置卷标
```bash ```bash
mkdir -p /mnt/minio{1,2,3,4} mkfs.xfs -f /dev/sdb -L MINIO1
mount /dev/sdb1 /mnt/minio1 mkfs.xfs -f /dev/sdc -L MINIO2
mount /dev/sdc1 /mnt/minio2 mkfs.xfs -f /dev/sdd -L MINIO3
mount /dev/sdd1 /mnt/minio3 mkfs.xfs -f /dev/sde -L MINIO4
mount /dev/sde1 /mnt/minio4
``` ```
- 建议在 fstab 中配置开机自动挂载 - 在 /etc/fstab 中增加挂载配置
```
LABEL=MINIO1 /mnt/minio1 xfs defaults,noatime 0 2
LABEL=MINIO2 /mnt/minio2 xfs defaults,noatime 0 2
LABEL=MINIO3 /mnt/minio3 xfs defaults,noatime 0 2
LABEL=MINIO4 /mnt/minio4 xfs defaults,noatime 0 2
```
- 创建挂载目录，挂载数据盘
```bash
systemctl daemon-reload
mkdir -p /mnt/minio{1,2,3,4}
mount -a
```
- 禁用 xfs 错误重试
```bash
for i in sdb sdc sdd sde; do
echo 0 > /sys/fs/xfs/$i/error/metadata/EIO/max_retries
echo 0 > /sys/fs/xfs/$i/error/metadata/ENOSPC/max_retries
echo 0 > /sys/fs/xfs/$i/error/metadata/default/max_retries
done
```
- 为 /etc/rc.d/rc.local 增加可执行权限
```bash
chmod 0755 /etc/rc.d/rc.local
```
- 编辑 /etc/rc.d/rc.local，增加“禁用 xfs 错误重试”的脚本
```bash
for i in sdb sdc sdd sde; do
echo 0 > /sys/fs/xfs/$i/error/metadata/EIO/max_retries
echo 0 > /sys/fs/xfs/$i/error/metadata/ENOSPC/max_retries
echo 0 > /sys/fs/xfs/$i/error/metadata/default/max_retries
done
```
### 安装 minio ### 安装 minio
- 在全部服务器上执行如下操作 - 在全部服务器上执行如下操作
- 下载 - 下载 minio rpm 包，安装
```bash
curl -L -o minio.rpm https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20240406052602.0.0-1.x86_64.rpm
rpm -Uvh minio.rpm
groupadd -r minio-user
useradd -r -g minio-user minio-user
chown minio-user:minio-user /mnt/minio{1..4}
```
- 下载 minio 客户端工具 mc
```bash ```bash
curl -L -o /usr/local/bin/minio https://dl.min.io/server/minio/release/linux-amd64/minio
curl -L -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc curl -L -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
chmod 0755 /usr/local/bin/{minio,mc} chmod 0755 /usr/local/bin/mc
``` ```
- 修改 hosts，增加自定义域名解析，**注意这里是集群通信，要用 eth1 网卡 ip** - 修改 hosts，增加自定义域名解析，**注意这里是集群通信，要用 eth1 网卡 ip**
@ -107,87 +149,21 @@ X4.X.X | eth0: 192.168.1.14<br>eth1:10.0.0.14 | 8 个 | /dev/sdb1, /dev/sd
10.0.0.14 X4.X.X 10.0.0.14 X4.X.X
``` ```
- 创建 /etc/systemd/system/minio.service，内容如下 - 创建 /etc/default/minio，内容如下
```ini ```ini
[Unit] MINIO_VOLUMES="http://X{1...4}:9000/mnt/minio{1...4}"
Description=MinIO MINIO_OPTS="--console-address :9001"
After=network.service MINIO_ROOT_USER=minio-admin
Wants=network.service MINIO_ROOT_PASSWORD=1234567890
[Service]
Environment=MINIO_ROOT_USER=XXX
Environment=MINIO_ROOT_PASSWORD=XXXXXXXX
ExecStart=/usr/local/bin/minio server \
--address ":9000" \
--console-address ":9010" \
http://X{1...4}.X.X:9000/mnt/minio{1...4}
#Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
``` ```
### 启动 minio ### 启动 minio
- 在全部服务器上执行如下操作 - 在全部服务器上执行如下操作
- 配置开机自动启动 - 启动 minio，并配置开机自动启动
```bash ```bash
systemctl daemon-reload systemctl daemon-reload
systemctl enable minio
```
- 启动 minio
```bash
systemctl start monio systemctl start monio
``` systemctl enable minio
## 扩容多节点分布式存储
### 加装硬盘
- 在全部服务器上执行如下操作
- 在每台服务器上个增加四个新硬盘，每个硬盘创建一个分区，格式化
- 挂载新增的硬盘分区
```bash
mkdir -p /mnt/minio{5,6,7,8}
mount /dev/sdf1 /mnt/minio5
mount /dev/sdg1 /mnt/minio6
mount /dev/sdh1 /mnt/minio7
mount /dev/sdi1 /mnt/minio8
```
- 建议在 fstab 中配置开机自动挂载
### 扩容
- 在全部服务器上执行如下操作
- 停止 minio 服务
```bash
systemctl stop minio
```
- 修改 /etc/systemd/system/minio.service，增加新的 server pool，内容如下
```ini
[Unit]
Description=MinIO
After=network.service
Wants=network.service
[Service]
Environment=MINIO_ROOT_USER=XXX
Environment=MINIO_ROOT_PASSWORD=XXXXXXXX
ExecStart=/usr/local/bin/minio server \
--address ":9000" \
--console-address ":9010" \
http://X{1...4}.X.X:9000/mnt/minio{1...4} \
http://X{1...4}.X.X:9000/mnt/minio{5...8}
#Restart=on-failure
RestartSec=10s
[Install]
WantedBy=multi-user.target
```
- 启动 minio 服务
```bash
systemctl start minio
``` ```
## 客户端常用操作 ## 客户端常用操作