Encrypted Linux Backup with Google Drive and Duplicity
When it comes to protecting your important data it’s best to have at least 3 copies of your things backed up. One should be air-gapped (not actively connected) and another be offsite (not at the same building). The air gap will protect against malicious code that would encrypt all your files and force you to pay bitcoin to get the decryption key, and offsite in case of fire/flood/quake/etc.
I’ve been using duplicity to backup files from my main system which runs Linux. Duplicity offers many secure ways to back up files. And it will both encrypt them and version control them like git. This is great because the data you store on the server will never be exposed and you have no worries if the data is stolen. People won’t be able to read the files.
My personal favorite place to backup files is with Google Drive. Google Drive will give you 15Gb free and for just $1.99 you can have 100Gb of storage space. That’s more than enough for me for working documents and my home directory. Well worth the $2.
Since Google phased out their OAuth1 access the newer OAuth2 takes a lot more setup to get working. Most current online blogs have recommend a helper application for your helper application… you get the idea. Here I will give you step by step directions to save you the time and trouble of figuring out the right way to do it.
For this I’ll be using Ubuntu 14LTS. A couple things you’ll need to have from the package manager are build-essential, gcc, libpopt-dev, libpopt0. And you’ll be downloading the latest librsync from Github, the gzipped tar file for duplicity 0.7.07.1, and PyDrive.
sudo apt-get install libpopt-dev libpopt0
git clone [email protected]:librsync/librsync.git cd librsync cmake . make all check sudo make install
wget https://code.launchpad.net/duplicity/0.7-series/0.7.07.1/+download/duplicity-0.7.07.1.tar.gz gunzip duplicity-0.7.07.1.tar.gz tar -xf duplicity-0.7.07.1.tar cd duplicity-0.7.07.1 sudo python setup.py install
For this you need to have Python’s pip installer.
pip install PyDrive
For this to work you need to create Google API Credentials at console.developers.google.com. At the API Manager page you’ll need to use the drop down menu at the top and click Create a project.... Give it a name and continue. Under Google APIs enter Google Drive in the search, click on Google Drive API and then click Enable. Then on the left side menu click Credentials. Click Create credentials. Then click OAuth client ID. It will complain about not having a name so follow the link and give it a name under Product name shown to users. Then click Save. For the Application Type select Other, then type in a name and click Create. You will now have your credentials show to you.
In your Linux console go ahead and make a directory for your configuration files.
cd ~ mkdir .duplicity cd .duplicity touch ignore touch credentials
Open the credentials file and use the following as the template.
client_config_backend: settings client_config: client_id: xxx.apps.googleusercontent.com client_secret: yyyy save_credentials: True save_credentials_backend: file save_credentials_file: gdrive.cache get_refresh_token: True
Where the two indented lines are; that’s where you want to put in the credentials that Google gave you for your script. That it for setup except for one thing; an executable bash script to start your backup.
For the executable script I’ve added the folder ~/bin to my user’s PATH and so any executable files and there can be run from anywhere else. Lets write a startup script named gbackup and put the following in.
#!/bin/bash export GOOGLE_DRIVE_SETTINGS=~/.duplicity/credentials duplicity --exclude-filelist ~/.duplicity/ignore ~/ gdocs://[email protected]/backup
And change the file to executable with chmod +x ~/bin/gbackup and you’re good to go! For any directories that you want to exclude just put them each in their own line in the ~/.duplicity/ignore file. And the /backup at the end is the name of the folder you’re choosing to write to in your Google Drive.
For this to work you’ll need to have your GnuPG encryption key setup as this is how it will encrypt everything before uploading. Howto do this is available in the Ubuntu help docs help.ubuntu.com/community/GnuPrivacyGuardHowto
The documentation for this was not quite complete and so getting the credentials to work took a long time to figure out. The other helper that some recommend is called duply (it’s like a credential manager for duplicity). You should make it a practice to backup regularly. With duplicity the initial backup will be a full one and then the next time you run it it will only have to upload any changes. So now you can worry less with your data safely tucked away offsite.
The one thing you may have to consider is how you will unencrypt everything if a fire destroys your GPG keys and your software for backup/retrieval. For that case I’d recommend keeping a backup install of Ubuntu, a disc/usb stick of these installable files above, and an AES encrypted copy of your private/public GPG keys offsite. The GPG keys are the most important piece, but 10 years from now who knows if these useful tools will even be available for download? If you think ahead you can save yourself from a lot of trouble and pain later.
As always I hope you found this educational and enjoyable! Please feel free to comment, share, subscribe to my RSS Feed, and follow me on twitter@6ftdan!
-Daniel P. Clark
Image by J P via the Creative Commons Attribution-NonCommercial 2.0 Generic License
August 27, 2016 - 3:23 pm
Thanks for the instructions Daniel!
I’ve used the same setup with duplicity + gdrive last year. However, instead of using Google API Credentials with pydrive, I set up 2FA, and used App Passwords with the native duplicity gdocs backend, which worked fine with the standard duplicity (0.6.23) found in the 14.04 LTS repos.
I believe this setup still works, as I assume Google hasn’t changed much about its API this year.
August 28, 2016 - 11:08 am
Hmm, this setup no longer works with the gdata backend (e.g. gdata+gdocs://) because Google apparently shut down its ClientLogin in June 2015 according to my cron logs. However, using PyDrive backend instead does work.
Daniel P. Clark
August 29, 2016 - 1:18 pm
There is something you can do to get it to work. I’ve done it but I’ll have to check and see what it was.
… You need to set it up with PyDrive for the new OAuth requirement.
Daniel P. Clark
August 29, 2016 - 1:38 pm
Yes. Mine had stopped working since then as well. That’s what this blog post solves. This was the change after Google updated their OAuth. This should solve your problem.
December 21, 2016 - 9:42 am
This is more reliable than what you have posted
git clone https://github.com/librsync/librsync.git