分析關于Mysql容器啟動失敗恢復的問題

昨天重啟完NAS后，今天早上發現NAS又死機了。只能再次強制關機，重啟。
啟動docker容器之前，我把mysql容器的內存調整了一下，調整成了512M。 然后就悲劇的發現啟動不了了。后來發現，啟動不了不是我調內存的關系。

查看日志，顯示如下:

2020-12-27T02:43:53.375776Z 0 mysqld: [Warning] World-writable config file '/etc/mysql/my.cnf' is ignored.
2020-12-27T02:43:53.375776Z 0 mysqld: Error on realpath() on '/var/lib/mysql-files' (Error 2 – No such file or directory)
2020-12-27T02:43:53.375776Z 0 [ERROR] [MY-010095] [Server] Failed to access directory for –secure-file-priv. Please make sure that directory exists and is accessible by MySQL Server. Supplied value : /var/lib/mysql-files
2020-12-27T02:43:53.376005Z 0 [ERROR] [MY-010119] [Server] Aborting

百度了下，解決辦法是重新run一個mysql容器，并且在之前的配置上加上-v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/。

現在問題來了，之前run mysql容器時，我并沒有記錄下完整的run 命令。還好，通過rekcod工具可以查看。更喜的是，這個工具可以通過docker來運行。因為我的NAS的系統不是發行版的linux，npm，yum好多工具都沒法安裝。

關于rekcod的使用可以參考linuxea:如何復現查看docker run參數命令

簡單來說，用法如下：

1 docker 安裝rekcod

$ docker pull nexdrew/rekcod $ alias rekcod="docker run --rm -v /var/run/docker.sock:/var/run/docker.sock nexdrew/rekcod"

2 使用方法：

[root@TNAS-012664 ~]# rekcod mysql ==>mysql是我的容器名 docker run --name mysql --runtime runc -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql -p 3306:3306/tcp --net bridge --restart no -h 39964e9e508a --expose 3306/tcp --expose 33060/tcp -e 'MYSQL_ROOT_PASSWORD=123456' -e 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' -e 'GOSU_VERSION=1.12' -e 'MYSQL_MAJOR=5.7' -e 'MYSQL_VERSION=5.7.30-1debian10' -d -t -i --entrypoint "docker-entrypoint.sh" mysql 'mysqld'

可以看到我的數據庫文件都是放在/mnt/md0/User/wzp/home/www/mysql/目錄下，我查看了下該目錄文件，數據應該是沒有丟失的。

于是，我新建了mysql2 容器，命令如下
docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 --name mysql2 -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql -v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/ mysql

然后，進入mysql2，發現沒法連接數據庫
docker exec -it mysql2 /bin/bash

查看mysql 狀態

root@0e83698acbfb:/# mysqld status mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/docker.cnf' is ignored. mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/mysql.cnf' is ignored. 2020-12-27T02:39:41.865252Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release. 2020-12-27T02:39:41.865455Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20) starting as process 105 2020-12-27T02:39:41.871715Z 0 [ERROR] [MY-010123] [Server] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root! 2020-12-27T02:39:41.872541Z 0 [ERROR] [MY-010119] [Server] Aborting 2020-12-27T02:39:41.872776Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.20)  MySQL Community Server - GPL. root@0e83698acbfb:/# Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!^C

用以下方法以root方式啟動mysqld

root@0e83698acbfb:/# mysqld --user=root mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/docker.cnf' is ignored. mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/mysql.cnf' is ignored. 2020-12-27T02:40:57.169719Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release. 2020-12-27T02:40:57.169896Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20) starting as process 116 2020-12-27T02:40:57.184807Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2020-12-27T02:40:57.241048Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11 2020-12-27T02:40:58.241783Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11 2020-12-27T02:40:59.242983Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11 2020-12-27T02:41:00.244290Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11 2020-12-27T02:41:01.245762Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11 2020-12-27T02:41:02.247539Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11

屏幕一直打印Unable to lock ./ibdata1 error: 11，只能CTRL-C強制中斷。百度了下，有兩個方式來解決該問題

1 參考mysql之 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
該文章分析，導致該error主要是以下兩個原因，很可惜，文章中的方法并不能解決我的問題。

1. 磁盤空間目錄不足
2. ibdata1 文件被其他的進程占用

2 參考mysqld報InnoDB: Unable to lock ./ibdata1 error: 11這篇文章完美解決了我的問題。主要進行了以下操作。

2.1 首先，進入我的mysql數據庫所在目錄
cd /mnt/md0/User/wzp/home/www/mysql

2.2 然后將ibdata1等文件重命名后，再cp回來。具體為什么要這么做，我也不太明白。

[root@TNAS-012664 www]# cd /mnt/md0/User/wzp/home/www/mysql/ [root@TNAS-012664 mysql]# ls  auto.cnf          binlog.index      client-key.pem   dci            '#ib_16384_0.dblwr'   ib_logfile0     mautic               nianbao    performance_schema   robot             shangbiao          undo_001         zeng  baike             ca-key.pem        company_works    dianzicaipiao  '#ib_16384_1.dblwr'   ib_logfile1     mysql                niuwan     private_key.pem      sara_wiki         sjzt_ry6           undo_002  bigdatapaltfrom   ca.pem            copyrightdata    gs_data         ib_buffer_pool      '#innodb_temp'   mysql.ibd            pachong    public_key.pem       server-cert.pem   sys                wenshu  binlog.000001     client-cert.pem   db_huayun        hy_dci_admin    ibdata1              integrate       mysql_upgrade_info   pachong2   qianliu_wiki         server-key.pem    ucenter_huayunyy   yuanqixiaoshuo [root@TNAS-012664 mysql]# mv ibdata1 ibdata1.bak [root@TNAS-012664 mysql]# mv ib_logfile0 ib_logfile0.bak [root@TNAS-012664 mysql]# mv ib_logfile1 ib_logfile1.bak [root@TNAS-012664 mysql]# cp -a ibdata1.bak ibdata1 [root@TNAS-012664 mysql]# cp -a ib_logfile0.bak ib_logfile0 [root@TNAS-012664 mysql]# cp -a ib_logfile1.bak ib_logfile1

2.3 然后我刪除之前的mysql2 container，用下面的命令再次run一個mysql2容器，發現就可以連接mysql數據庫了。

[root@TNAS-012664 ~]# docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456  --name mysql2 -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql  -v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/    mysql 1e031247ea46e82f6205db68e7fb1b55389c87e5e2cb13517f9e1ac17d514509 [root@TNAS-012664 ~]# docker exec -it mysql2 /bin/bash root@1e031247ea46:/# mysql -uroot -p mysql: [Warning] World-writable config file '/etc/mysql/my.cnf' is ignored. Enter password: Welcome to the MySQL monitor.  Commands end with ; or g. Your MySQL connection id is 8 Server version: 8.0.20 MySQL Community Server - GPL  Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.  Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.  Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.  mysql> show databases;

以為mysql容器好了就萬事大吉了？ No，too naive! 還有一個小插曲等著我呢。這邊剛把mysql容器解決好，突然又發現我的wiki容器莫名其妙沒了。不過好在，數據庫文件在，一切還能恢復如初。

1 用以下的命令新建兩個wiki

docker run --name sarawiki --link mysql2:mysql -p 8086:80 -d sarawang85/mediawiki:1.0.0 docker run --name qianliuwiki --link mysql2:mysql -p 8083:80 -d sarawang85/mediawiki:1.0.0

2 然后把備份好的LocalSettings.php 文件和Logo圖片copy 進容器，搞定。

PS，如果在docker gui界面編輯stop狀態的容器，比如設置內存限制，實際是會新建一個同名的容器。因為我發現我這樣做了后，訪問我的wiki，又提示我重新安裝。

再次PS，mysql 備份得立即提上日程。mysql 內存限制不能單單通過容器來限制，my.cnf也得做響應更改。