Backing up longhorn volumes to MINIO/S3 on CIVO K3S
Civo K3s is the World's first managed K3s solution, you can sign up to join the BETA program below and get $70 of credit each month! There is liternally nothing stopping you! Well apart from you clicking the link....
This guide will show you how to setup backups of your persistant volumes to an S3 compatible backup destination, in this example I will be using MINIO but you could quite easily setup an Amazon S3 bucket. Setting up the S3 bucket is beyond the scope of this post but there are plenty of guides out there.
Setting up MINIO is pretty straight forward, you can follow this excellent guide from Alejandro @ Civo to get up and running. You will also find it in the Civo kubernetes marketplace, a one click installation!
Once you have MINIO setup there are 3 important things you need to have a record of:
- The URL to reach your MINIO server
- The aws_access_key_id
- The aws_secret_access_key
Make a note of these as they will be needed in a sec.
You may already have Longhorn setup, if not and you are using Civo K3s this is as easy as going to the marketplace and installing the app. After a minute you should see all the Longhorn pods up and running.
The first thing we need to do is store your MINIO connection information in a Kubernetes secret. To do this we need to convert each value to BASE64, replacing the values below with your URL, access key and secret:
Your MINIO URL should look something like: http://minio.somedomain.com:9000
echo -n MINIO_URL | base64 echo -n aws_access_key_id | base64 echo -n aws_secret_access_key | base64
You will see something like the following:
TUlOSU9fVVJM YXdzX2FjY2Vzc19rZXlfaWQ= YXdzX3NlY3JldF9hY2Nlc3Nfa2V5
Once you have these values we can generate the secret (make sure you replace the data values with your BASE64 ones):
cat <<EOF >>aws_secret.yml apiVersion: v1 kind: Secret metadata: name: aws-secret namespace: longhorn-system type: Opaque data: AWS_ACCESS_KEY_ID: TUlOSU9fVVJM AWS_SECRET_ACCESS_KEY: YXdzX2FjY2Vzc19rZXlfaWQ= AWS_ENDPOINTS: YXdzX3NlY3JldF9hY2Nlc3Nfa2V5 EOF
We can now apply the manifest to create the secret:
kubectl apply -f aws_secret.yml
You can check this has been created by running:
kubectl get secrets -n longhorn-system
NAME TYPE DATA AGE longhorn-service-account-token-9spgn kubernetes.io/service-account-token 3 30d default-token-szgv7 kubernetes.io/service-account-token 3 30d aws-secret Opaque 3 30d
Setting up the backup is pretty straight forward and intuitive so i'm not going to go overbaord with the instructions! Let's create a simple PVC which will in turn create the PV and the volume in Longhorn. If you already have a volume, you can skip this bit.
cat <<EOF >>volume.yml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pv-claim labels: type: longhorn spec: storageClassName: longhorn accessModes: - ReadWriteOnce resources: requests: storage: 5Gi EOF
Now apply the volume:
kubectl apply -f volume.yml
Once setup, you will need to access the Longhorn UI to configure the backups. As there is no authentication built into the UI out of the box, I would recommend you don't expose this to the outside world and rather use kubectl port-forward (Obviously change the local port from 8081 if needed):
kubectl port-forward svc/longhorn-frontend -n longhorn-system 8081:80
You can then use your local browser with the address:
All being well you will be presented with the longhorn dashboard which will show the health of your volumes:
If you already have running containers using volumes then these should show as healthy, as you can see here, as I have only just created this volume on a new cluster, therefore it's showing as "detached". We can easily attach this volume by selecting the volume and clicking "attach". You can attach this volume to the master.
Next we need to configure the backup destination, navigate to settings -> general:
Scroll down to the section about backup. You will now fill in the details (changing these as required):
Make sure you save the configuration. Next we can test a backup, select volumes from the menu and then click on the volume you want to backup:
You can click create backup and add any labels if you wish. If this is a new volume it will be completed very quickly, you can check by hovering over the snapshot:
Once this shows 100%, it should be visable from the backup tab:
You can also double check your Minio bucket:
Restoring a volume from backup
If you want to restore a volume from backup, this is pretty straight forward. From the backup menu select the volume you want to restore then on the next screen select which backup you want to restore:
You can then complete the details as required:
You can then see the restored volume in the Volumes screen:
Once you have the volume available you can attach to a node and use as you wish.
I hope you found this guide useful, any questions please give me a shout on twitter.