Loading...
 

History: TRIM and Syncthing for automated offsite backups

Preview of version: 47

Use case: I have some Tiki instances on ClearOS and I want automatic backups of different Tiki instances to various local computers for safekeeping. These are high level instructions as to how to combine TRIM and Syncthing to do so. For a hosting company and its customer, this is the optimal solution: This permits a customer to delegate the hosting / management of an application while obtain automatic backups of their application and data.

At a high level:

  1. TRIM (installed on the same server) will make incremental and automated backups of each site to a distinct folder
  2. Syncthing will copy each folder to one or many devices.

Benefits / Features

  • Automated (unattended): put TRIM backups on a cron job
  • Incremental while being somewhat efficient for disk spaceThis could be improved quite a bit. Volunteers welcome! : TRIM has a retention cycle: keep last 7 days, last 4 weeks and once per month forever
  • Remote backup to any number of devices: Syncthing is designed for peer-to-backups
  • Managed via a web interface (Syncthing)
  • Web interface can be protected by 2-factor authentication (in addition to password)
  • Data is encrypted in transport by Syncthing
  • Cross-platform: Syncthing supports GNU/Linux, Windows, OSX, Android, *BSD, Solaris
  • Devices can be behind the firewall, and without a fixed IP address. Ex.: Syncthing on your laptop
  • Throttling: Can set Incoming and Outgoing Rate Limit (KiB/s)
  • Backup Tiki, the database and even files that are outside the web root: TRIM checks the Tiki database to know which directories to fetch, if any.
  • Designed to cope with a Ransomware attack (see recipe below)
  • Flexible multi-site design: Each Tiki is backed up in own folder providing flexibility on what to send where.
  • Easy to restore: backups are in a format that can be restored by TRIM with the make restore command.
  • Avoid disk full issues: Syncthing will halt before filling up 100% of your disk. (TRIM still needs this added). TODO: Add alerting to both

ClearOS as web server

Install each Tiki as a distinct website using sub-domains: How to set up websites on ClearOS

Good:

  • example.org
  • example.com
  • projecta.example.org
  • projectb.example.org


Bad:

  • example.org/projecta/
  • example.org/projectb/


Setup TRIM

Install TRIM in /var/local/trim as per https://doc.tiki.org/TRIM#How_to_download_and_use

Use TRIM to make instances, or adopt them

Configure TRIM to backup your Tiki instances

https://doc.tiki.org/TRIM#make_backup

The TRIM archive folders have the following pattern:

  • /var/local/trim/backup/archive/1-example.org/
  • /var/local/trim/backup/archive/2-example.com/
  • etc.


Check that your backup is OK, and move on to the next step

Set up automatic backups

TRIM will make the backups and also has a retention cycle (keep last 7 days, last 4 weeks and once per month forever)

Of all the instances

https://doc.tiki.org/TRIM#To_setup_automated_backups

For just one of the instances

[+]

Install Syncthing

On the server


You can install many instances of Syncthing, but there is not yet a good way to segment permissions. So it's better to create one master Syncthing user. Ex.: tikisyncthingbackup and manage all the backups from that account. Do not give that passwords to customers.

https://example.org:81/
u: tikisyncthingbackup
p: superstrongpassword

then visit https://example.org:81/syncthing/ (it may ask login again in basic auth)

On your laptop

Setup Syncthing


Make sure Syncthing automatically restarts when you reboot so that you have unattended automatic backups. Just reboot to see if it's OK. If not, you need to look up the documentation for your Operating System.

Configuring Syncthing for your Tiki archive folder

You archives will be in something like /var/local/trim/backup/archive/1-example.org/: so sync that folder with your local computer.

Setting up Syncthing for backups

Syncthing by default is set to sync multiple folders so they get to the exact same set of files and folders. So if you delete a file in one folder, the deletion is propagated. In the contexts of backups, this could cause issues. Examples:

  • Ransomware on any device: Then, all the devices would receive the corrupted data which replaces the valid data.
  • One of the devices runs out of space, and its user decides to clear out old backups.


The solution is:

  1. On the device sending the backups: set Folder Type to "Send Only"
  2. On the device(s) receiving the backups: set File Versioning. Recommended setting is "Staggered"

Because the master server is set to "Send Only", there is no point in setting to "Staggered" (It doesn't receive any data, and thus doesn't need versioning).
Syncthing Backups Send Only Setting
Syncthing Backup Receiving Staggered

Troubleshooting

  • If sync is not working as expect, check the Syncthing folder interface for errors.

History

Advanced
Information Version
Marc Laporte 54
View
Marc Laporte 53
View
Marc Laporte 52
View
Marc Laporte 51
View
Marc Laporte 50
View
Marc Laporte 49
View
Marc Laporte 48
View
Marc Laporte 47
View
Marc Laporte 46
View
Marc Laporte 45
View
Marc Laporte 44
View
Marc Laporte Nicer with images :-) 43
View
Marc Laporte 42
View
Marc Laporte 41
View
Marc Laporte 40
View
Marc Laporte 39
View
Marc Laporte 38
View
Marc Laporte 37
View
Marc Laporte 36
View
Marc Laporte 35
View
Marc Laporte 34
View
Marc Laporte 33
View
Marc Laporte 32
View
Marc Laporte 31
View
Marc Laporte 30
View
Marc Laporte 29
View
Marc Laporte Benefits / Features 28
View
Marc Laporte 27
View
Marc Laporte 26
View
Marc Laporte 25
View
Marc Laporte cleaning up 24
View
Marc Laporte 23
View
Marc Laporte 22
View
Marc Laporte 21
View
Marc Laporte 20
View
Marc Laporte 19
View
Marc Laporte 18
View
Marc Laporte 17
View
Marc Laporte 16
View
Marc Laporte 15
View
Marc Laporte will be simpler 14
View
Marc Laporte Better domain name 13
View
Marc Laporte 12
View
Marc Laporte 11
View
Marc Laporte 10
View
Marc Laporte Code Plugin modified by editor. 9
View
Marc Laporte Removing to avoid confusion 8
View
Marc Laporte 7
View
luis.lucio 6
View
luis.lucio 5
View
  • «
  • 1 (current)
  • 2