k8s中修复mongodb启动失败

file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open,

知本知至

598人浏览 · 2024-04-15 11:45:59

知本知至 · 2024-04-15 11:45:59 发布

背景

同事反馈 dev环境的yapi不能登录，看了一下是同事两年前用helm搭建的。单副本使用。
排查发现是后端数据库mongodb数据库挂掉。

root@dev-k8s-master03:~# kubectl get svc 
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
mongodb       ClusterIP   10.101.87.7     <none>        27017/TCP        2y39d
yapi-svc      NodePort    10.108.180.80   <none>        3000:31781/TCP   2y39d

故障现象

 2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory

 2024-04-12T09:52:26.257+0000 I CONTROL  [initandlisten] options: { config: "/opt/bitnami/mongodb/conf/mongodb.conf", net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: "/opt/bitnami/mongodb/tmp" } }, processManagement: { fork: false, pidFilePath: "/opt/bitnami/mongodb/tmp/mongodb.pid" }, security: { authorization: "enabled" }, setParameter: { enableLocalhostAuthBypass: "false" }, storage: { dbPath: "/bitnami/mongodb/data/db", directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/opt/bitnami/mongodb/logs/mongodb.log", quiet: false, verbosity: 0 } }

 2024-04-12T09:52:26.258+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.

 2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

 2024-04-12T09:52:26.259+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.

 2024-04-12T09:52:26.259+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.365+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.376+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

 2024-04-12T09:52:26.386+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

 2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.

 2024-04-12T09:52:26.390+0000 W STORAGE  [initandlisten] WiredTiger metadata corruption detected

 2024-04-12T09:52:26.390+0000 F STORAGE  [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

 2024-04-12T09:52:26.390+0000 F -        [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701

 2024-04-12T09:52:26.390+0000 F -        [initandlisten]

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] 4096 MB of memory available to the process out of 64321 MB total system memory

2024-04-12T09:52:26.257+0000 I CONTROL [initandlisten] options: { config: “/opt/bitnami/mongodb/conf/mongodb.conf”, net: { bindIpAll: true, ipv6: false, port: 27017, unixDomainSocket: { enabled: true, pathPrefix: “/opt/bitnami/mongodb/tmp” } }, processManagement: { fork: false, pidFilePath: “/opt/bitnami/mongodb/tmp/mongodb.pid” }, security: { authorization: “enabled” }, setParameter: { enableLocalhostAuthBypass: “false” }, storage: { dbPath: “/bitnami/mongodb/data/db”, directoryPerDB: false, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, logRotate: “reopen”, path: “/opt/bitnami/mongodb/logs/mongodb.log”, quiet: false, verbosity: 0 } }

2024-04-12T09:52:26.258+0000 W STORAGE [initandlisten] Detected unclean shutdown - /bitnami/mongodb/data/db/mongod.lock is not empty.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.

2024-04-12T09:52:26.259+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.

2024-04-12T09:52:26.259+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1536M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:365352][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:365392][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:365398][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.365+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:365403][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:376001][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:376056][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:376067][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.376+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:376077][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712915546:386587][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata Raw: [1712915546:386638][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 577: WiredTiger has failed to open its metadata

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk Raw: [1712915546:386645][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 581: This may be due to the database files being encrypted, being from an older version or due to corruption on disk

2024-04-12T09:52:26.386+0000 E STORAGE [initandlisten] WiredTiger error (0) [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options Raw: [1712915546:386652][1:0x7fc65c8f1080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 585: You should confirm that you have opened the database with the correct options including all encryption and compression options

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.

2024-04-12T09:52:26.390+0000 W STORAGE [initandlisten] WiredTiger metadata corruption detected

2024-04-12T09:52:26.390+0000 F STORAGE [initandlisten] Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair

2024-04-12T09:52:26.390+0000 F - [initandlisten] Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 701

2024-04-12T09:52:26.390+0000 F - [initandlisten]

分析&思路

看前面的E级别的log信息，存储引擎读取root page失败。unable to read root page from file:WiredTiger.wt

找到这个mongo挂载的nfs存储，先将数据库文件做一个备份。

单副本的mongo并且一直不断重启，网上搜到的使用mogod --repair执行修复根本不能进入到容器中执行的。

在同一个pod中使用跟mongo同样的镜像做一个二次封装，命名为mongodebug 把这个镜像放在mongo的pod中，使得这个pod存在两个容器，然后挂载同一个共享存储，进入到mongodebug执行mongod命令进行修复

实施

必要条件

挂载持久卷存储的容器路径/bitnami/mongodb/data/db

制作mongodebug容器镜像

目的：需要一个可执行的mongd命令

拉镜像，运行，找到容器启动脚本，覆盖掉。
防止读取同一份配置文件不能启动

root@master01:~# docker run -it bitnami/mongodb:4.0.14-debian-9-r24 /bin/bash 
 10:11:48.80 
 10:11:48.80 Welcome to the Bitnami mongodb container
 10:11:48.81 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
 10:11:48.81 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
 10:11:48.81 Send us your feedback at containers@bitnami.com
 10:11:48.81 

I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ 
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  var

I have no name!@7b9781de0eed:/$ cat entrypoint.sh 
#!/bin/bash
# shellcheck disable=SC1091

set -o errexit
set -o nounset
set -o pipefail


# Load libraries
. /libbitnami.sh
. /libmongodb.sh

# Load MongoDB env. variables
eval "$(mongodb_env)"

print_welcome_page

if [[ "$*" = "/run.sh" ]]; then
    info "** Starting MongoDB setup **"
    /setup.sh
    info "** MongoDB setup finished! **"
fi

echo ""
exec "$@"

I have no name!@7b9781de0eed:/$ cat run.sh 
#!/bin/bash
# shellcheck disable=SC1091

set -o errexit
set -o nounset
set -o pipefail

. /libmongodb.sh
. /libos.sh

# Load MongoDB env. variables
eval "$(mongodb_env)"

cmd=$(command -v mongod)

flags=("--config=$MONGODB_CONF_FILE")

if [[ -n "${MONGODB_EXTRA_FLAGS:-}" ]]; then
    read -r -a extra_flags <<< "$MONGODB_EXTRA_FLAGS"
    flags+=("${extra_flags[@]}")
fi

info "** Starting MongoDB **"
if am_i_root; then
    exec gosu "$MONGODB_DAEMON_USER" "${cmd}" "${flags[@]}"
else
    exec "${cmd}" "${flags[@]}"
fi
I have no name!@7b9781de0eed:/$ cmd=$(command -v mongod)
I have no name!@7b9781de0eed:/$ echo $cmd
/opt/bitnami/mongodb/bin/mongod
I have no name!@7b9781de0eed:/$ echo $MONGODB_CONF_FILE
/opt/bitnami/mongodb/conf/mongodb.conf
I have no name!@7b9781de0eed:/$ ls
bin  bitnami  boot  dev  entrypoint.sh	etc  home  lib	lib64  libbitnami.sh  libcomponent.sh  libfile.sh  libfs.sh  liblog.sh	libmongodb.sh  libnet.sh  libos.sh  libservice.sh  libvalidations.sh  libversion.sh  licenses  media  mnt  opt	postunpack.sh  proc  root  run	run.sh	sbin  setup.sh	srv  sys  tmp  usr  var
I have no name!@7b9781de0eed:/$ ps -ef 
UID          PID    PPID  C STIME TTY          TIME CMD
1001           1       0  0 10:11 pts/0    00:00:00 /bin/bash
1001          25       1  0 10:25 pts/0    00:00:00 ps -ef
I have no name!@7b9781de0eed:/$ exit
exit

dockerfile文件

root@master01:/data/mongodb# cat Dockerfile 
FROM bitnami/mongodb:4.0.14-debian-9-r24
USER root
RUN echo "tail -f /etc/hosts" > /setup.sh
RUN echo "tail -f /etc/hosts" > /run.sh

构建镜像&推送到一个具有公共读的镜像仓库中

root@master01:/data/mongodb# docker build -t xxx/xxx/mongodb-debug:v1 .

测试

root@master01:/data/mongodb# docker run -it xxx/xxx/mongodb-debug:v1 /bin/bash 
 11:15:17.58 
 11:15:17.58 Welcome to the Bitnami mongodb container
 11:15:17.58 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
 11:15:17.58 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
 11:15:17.58 Send us your feedback at containers@bitnami.com
 11:15:17.58 

root@8af93d798c63:/# ls /bitnami/
mongodb
root@8af93d798c63:/# cd bitnami/
root@8af93d798c63:/bitnami# ls
mongodb

编辑部署文件

编辑mongodb的deployment文件，将自定义的mongodb镜像做为第二个容器镜像，并配置pvc

kubectl edit deployment mongodb

      containers:
        - name: mongodbdebug
          image: xxx/istio_test/mongodb-debug:v1
          volumeMounts:
          - name: data
            mountPath: /bitnami/mongodb

        - name: mongodb
          image: xxx/xxx/mongodb:4.0.14-debian-9-r24

进入自定义的容器内部执行修复

root@dev-k8s-master03:/data# kubectl exec -it mongodb-7f97f799d9-qz82v -c mongodbdebug -- bash 
root@mongodb-7f97f799d9-qz82v:/# ls bitnami/mongodb/data/db/
WiredTiger			      collection-11--5099670552419067160.wt  collection-4-5162049135609455324.wt    index-1--8902867768542019650.wt   index-2--5099670552419067160.wt	index-30--5099670552419067160.wt  index-44--5099670552419067160.wt  index-56--5099670552419067160.wt  index-7-5162049135609455324.w
.........

root@mongodb-7f97f799d9-qz82v:/bitnami/mongodb/data/db# mongod --dbpath /bitnami/mongodb/data/db --repair

2024-04-12T11:23:29.167+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] MongoDB starting : pid=40 port=27017 dbpath=/bitnami/mongodb/data/db 64-bit host=mongodb-7f97f799d9-qz82v
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] db version v4.0.14
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] git version: 1622021384533dade8b3c89ed3ecd80e1142c132
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] modules: none
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] build environment:
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distmod: debian92
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     distarch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2024-04-12T11:23:29.169+0000 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/bitnami/mongodb/data/db" } }
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] Detected data files in /bitnami/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2024-04-12T11:23:29.175+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=31648M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2024-04-12T11:23:29.284+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error Raw: [1712921009:284845][40:0x7f7ccdc7d080], file:WiredTiger.wt, connection: __wt_btree_tree_open, 571: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
省略日志.......
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten] Modifications made by repair:
2024-04-12T11:23:34.236+0000 W STORAGE  [initandlisten]   WiredTiger metadata salvaged
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] finished checking dbs
2024-04-12T11:23:34.236+0000 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Shutting down session sweeper thread
2024-04-12T11:23:34.238+0000 I STORAGE  [initandlisten] Finished shutting down session sweeper thread
2024-04-12T11:23:34.378+0000 I STORAGE  [initandlisten] shutdown: removing fs lock...
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] now exiting
2024-04-12T11:23:34.380+0000 I CONTROL  [initandlisten] shutting down with code:0

新的报错

mongodb提示读取数据没权限，看了一下是由于构建镜像时候用了root用户执行命令，导致修改后的文件的所有者是root，chown xxx mongodb数据库文件即可

验证

root@dev-k8s-master03:/data# kubectl get pod 
NAME                           READY   STATUS             RESTARTS   AGE
mongodb-68d99d8cf9-wlcn2       1/1     Running            0          2d16h
yapi-pod-6f889c4b68-6xgqc      1/1     Running            0          2d18h

yapi也能正常登录了。

refer

mongodb社区
https://www.mongodb.com/community/forums/t/my-mongodb-not-working-properly-wt-error-non-specific-wiredtiger-error/145434/4

官网修复意外断电导致的mongo报错
https://www.mongodb.com/zh-cn/docs/manual/tutorial/recover-data-following-unexpected-shutdown/

mongodb镜像层
https://hub.docker.com/layers/bitnami/mongodb/4.0.14-debian-9-r24/images/sha256-fd839bcba360b2f470d003706f21ad4b8b1306be0b74e3ab936ff69b72f2e5d3?context=explore

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub