Backing up longhorn volumes to MINIO/S3 on CIVO K3S

Civo K3s is the World's first managed K3s solution, you can sign up to join the BETA program below and get $70 of credit each month! There is liternally nothing stopping you! Well apart from you clicking the link....

Introduction

This guide will show you how to setup backups of your persistant volumes to an S3 compatible backup destination, in this example I will be using MINIO but you could quite easily setup an Amazon S3 bucket. Setting up the S3 bucket is beyond the scope of this post but there are plenty of guides out there.

MINIO Setup

Setting up MINIO is pretty straight forward, you can follow this excellent guide from Alejandro @ Civo to get up and running. You will also find it in the Civo kubernetes marketplace, a one click installation!

First Steps

Once you have MINIO setup there are 3 important things you need to have a record of:

• The URL to reach your MINIO server
• The aws_access_key_id
• The aws_secret_access_key

Make a note of these as they will be needed in a sec.

Deploying Longhorn

You may already have Longhorn setup, if not and you are using Civo K3s this is as easy as going to the marketplace and installing the app. After a minute you should see all the Longhorn pods up and running.

The first thing we need to do is store your MINIO connection information in a Kubernetes secret. To do this we need to convert each value to BASE64, replacing the values below with your URL, access key and secret:

Your MINIO URL should look something like: http://minio.somedomain.com:9000

echo -n MINIO_URL | base64
echo -n aws_access_key_id | base64
echo -n aws_secret_access_key | base64

You will see something like the following:

TUlOSU9fVVJM
YXdzX2FjY2Vzc19rZXlfaWQ=
YXdzX3NlY3JldF9hY2Nlc3Nfa2V5

Once you have these values we can generate the secret (make sure you replace the data values with your BASE64 ones):

cat <<EOF >>aws_secret.yml
apiVersion: v1
kind: Secret
metadata:
  name: aws-secret
  namespace: longhorn-system
type: Opaque
data:
  AWS_ACCESS_KEY_ID: TUlOSU9fVVJM
  AWS_SECRET_ACCESS_KEY: YXdzX2FjY2Vzc19rZXlfaWQ=
  AWS_ENDPOINTS: YXdzX3NlY3JldF9hY2Nlc3Nfa2V5
EOF

We can now apply the manifest to create the secret:

kubectl apply -f aws_secret.yml

You can check this has been created by running:

kubectl get secrets -n longhorn-system

NAME                                   TYPE                                  DATA   AGE
longhorn-service-account-token-9spgn   kubernetes.io/service-account-token   3      30d
default-token-szgv7                    kubernetes.io/service-account-token   3      30d
aws-secret                             Opaque                                3      30d

Setting up the backup is pretty straight forward and intuitive so i'm not going to go overbaord with the instructions! Let's create a simple PVC which will in turn create the PV and the volume in Longhorn. If you already have a volume, you can skip this bit.

cat <<EOF >>volume.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-claim  
  labels:
    type: longhorn
spec:
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
EOF

Now apply the volume:

kubectl apply -f volume.yml

Configuring Longhorn

Once setup, you will need to access the Longhorn UI to configure the backups. As there is no authentication built into the UI out of the box, I would recommend you don't expose this to the outside world and rather use kubectl port-forward (Obviously change the local port from 8081 if needed):

kubectl port-forward svc/longhorn-frontend -n longhorn-system 8081:80

You can then use your local browser with the address:

http://localhost:8081

All being well you will be presented with the longhorn dashboard which will show the health of your volumes:

If you already have running containers using volumes then these should show as healthy, as you can see here, as I have only just created this volume on a new cluster, therefore it's showing as "detached". We can easily attach this volume by selecting the volume and clicking "attach". You can attach this volume to the master.

Next we need to configure the backup destination, navigate to settings -> general:

Scroll down to the section about backup. You will now fill in the details (changing these as required):

For example:

Make sure you save the configuration. Next we can test a backup, select volumes from the menu and then click on the volume you want to backup:

You can click create backup and add any labels if you wish. If this is a new volume it will be completed very quickly, you can check by hovering over the snapshot:

Once this shows 100%, it should be visable from the backup tab:

You can also double check your Minio bucket:

Restoring a volume from backup

If you want to restore a volume from backup, this is pretty straight forward. From the backup menu select the volume you want to restore then on the next screen select which backup you want to restore:

You can then complete the details as required:

You can then see the restored volume in the Volumes screen:

Once you have the volume available you can attach to a node and use as you wish.

I hope you found this guide useful, any questions please give me a shout on twitter.