Myaql 无法重启 “InnoDB: Error number 28 means 'No space left on device'” 错误解决办法

Docker 容器 Mysql 在服务器正常运行一段时间后，莫名的down后就一直无法重启，查看日志出现以下错误提示：

...
db-dz | 2021-03-13T22:21:24.844381Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
db-dz | 2021-03-13T22:21:24.845663Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
db-dz | 2021-03-13T22:21:24.846437Z 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
db-dz | 2021-03-13T22:21:24.847059Z 0 [ERROR] InnoDB: Write to file ./ibtmp1failed at offset 0, 1048576 bytes should have been written, only 0 were writtf this size. Check also that the disk is not full or a disk quota exceeded.
db-dz | 2021-03-13T22:21:24.847614Z 0 [ERROR] InnoDB: Error number 28 means 'No space left on device'
db-dz | 2021-03-13T22:21:24.847955Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operatin
db-dz | 2021-03-13T22:21:24.848383Z 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
db-dz | 2021-03-13T22:21:24.848773Z 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
db-dz | 2021-03-13T22:21:24.849159Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
db-dz | 2021-03-13T22:21:25.351333Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
db-dz | 2021-03-13T22:21:25.352365Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
db-dz | 2021-03-13T22:21:25.352789Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
db-dz | 2021-03-13T22:21:25.353229Z 0 [ERROR] Failed to initialize builtin plugins.
db-dz | 2021-03-13T22:21:25.353663Z 0 [ERROR] Aborting
db-dz | 
db-dz | 2021-03-13T22:21:25.354244Z 0 [Note] Binlog end
db-dz | 2021-03-13T22:21:25.354741Z 0 [Note] Shutting down plugin 'CSV'
db-dz | 2021-03-13T22:21:25.356762Z 0 [Note] mysqld: Shutdown complete
db-dz | 

关键的提示是没空间写入：

Error number 28 means 'No space left on device'

既然提示没空间了，那就 df -h看下使用情况：

root@localhost:~/discuz# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            480M     0  480M   0% /dev
tmpfs            99M   12M   88M  12% /run
/dev/sda         25G   24G     0 100% /
tmpfs           494M     0  494M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           494M     0  494M   0% /sys/fs/cgroup
overlay          25G   24G     0 100% /var/lib/docker/overlay2/4a1169686897ab35274b9a790e731307ff0a7b17f7da872a2cc801921c1157cb/merged
tmpfs            99M     0   99M   0% /run/user/0

果然 根目录/ overlay占用了 24个G，直接满盘。

通常Overlay2 占满分以下几种情况

1. 空间被其他大文件塞满
2. Overlay2 同级目录下，containers里有大量的日志文件，以“-json.log结尾”

一、 第一种情况删除大文件即可，与docker 无关

1、进入根目录：
cd /
2、使用命令 ： du -sh * 查看根目录下每个文件夹的大小
du -sh *
3、进入大文件夹，使用命令 du -h --max-depth=1
一步步找出文件并删除

root@localhost:~# du -h --max-depth=1
52K    ./traefik2
8.0K    ./.gnupg
12K    ./.local
1.9G    ./webdatas
41M    ./newDockerDiscuz
4.3G    ./DockerDiscuz
13G    ./discuz
8.0K    ./.ssh
19G    .

二、第二种情况是docker输出大量日志

1. 进入 container 目录
cd /var/lib/docker/containers 
2. 查找大文件 du * -sh 或 du -h --max-depth=1
3. 删除日志文件 .json结尾

清出空间即可启动Mysql。

PS 设置docker 限制容器日志大小

#1. 创建/etc/docker/daemon.json如果已经存在则不用创建
touch /etc/docker/daemon.json
#2. 加入代码 vi /etc/docker/daemon.json
"log-driver": "json-file",
"log-opts": {
    "max-size": "10m",    
    "max-file": "3"    
    }
}
3. 重启
systemctl  daemon-reload
systemctl  restart docker

执行 docker info 发现

Logging Driver: json-file
Cgroup Driver: cgroupfs