Blog

Helm chart -> bitnami/mongodb

Issue -> Cluster master was not responding due to some upgrade in resources.

PVC was intact in AWS.

Attached it to a EC2 instance and mount the volume.

I used my existing node instance to attach the old PVC. It worked like charm. SSH into instance. If you are not able to SSH then option is to create one new instance for this purpose.

Ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

Now download data to your machine using SCP.

scp -i ~/.ssh/id_rsa -r ubuntu@13.36.230.153:/mongodb /home/robomex/Desktop/mybackaws/

Here was the worst part of my recovery. Kind of nightmare and was awake till 3:17 AM to resolve this.

It happens that in my k8s I was using MongoDB 4.4.3 and locally I had 4.0.17 version. This really busted my head to figure out whats going wrong.

Golden rule: PLEASE READ LOGS ONE BY ONE.

In logs there were 3 hints for me. First one related to “Operation not permitted”, second one in “Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade.” and the last being mongo lock file state repair.

Keep multiple backup in your local machine while doing this.

Method 1:

So I tried first with --repair flag and run,

mongod --dbpath=/home/robomex/Desktop/mybackaws/data/db --repair

Now run the server. Make sure no other mongo server is running. Stop any using sudo service mongod stop.

mongod --dbpath=/home/robomex/Desktop/mybackaws/data/db

Open compass and connect with default config.

You are safe if you see your data here.

Still not ?

Lets move to what I was doing wrong. This step worked best for me. Reason was that data was not encrypted and I could restore the state well with repair flag.

Check your Mongo version. Make sure both are same and repeat method 1.

Data restoration:

First dump the data,

mongodump --db <db_name> --gzip --archive=/home/robomex/Desktop/mybackaws/

Now push the data back to your new cluster or any mongo server.

mongorestore --db <db_name> --uri="<uri>" --gzip --archive=/home/robomex/Desktop/mybackaws/archive.gz

This was a saviour.

More methods ?

2. Use existing PVC with a new helm chart. “existingClaim” find name using kubectl get pvc. This is possible if your cluster is intact. My case it was completely down and kube was not running.