I have backups on a backup hard drive and also synced to B2, but I am thinking about backing up to some format to put in the cupboard.
The issue I see is that if I don’t have a catastrophic failure and instead just accidentally delete some files one day while organising and don’t realise, at some point the oldest backup state is removed and the files are gone.
The other thing is if I get hit by a bus and no one can work out how to decrypt a backup or whatever.
So I’m thinking of a plain old unencrypted copy of photos etc that anyone could find and use. Bonus points if I can just do a new CD or whatever each year with additions.
I have about 700GB of photos and videos which is the main content I’m concerned about. Do people use DVDs for this or is there something bigger? I am adding 60GB or more each year, would be nice to do one annual addition or something like that.
You must log in or # to comment.
deleted by creator
I have 3 2 1 but I want the equivalent of a suitcase of photos in the cupboard. No family member is gonna be cleaning out my house as they move me to a rest home and stumble upon my Borg backup in B2 object storage. And if they do they won’t have the key. I want something a bit closer to physical.
I think an extra drive for cold storage is a good idea. My main backups are automated, this one I can add any new files done in the last year once a year, then back in the cupboard. I just need to make sure I’m rotating the drives so I don’t have the same one in storage for 50 years, and instead buy new ones every 5 years or so.
If this is your fear, why not just have a will or something that specifically describes what to do and where to go?
I have such a document, but it’s not quite the same. I’m just as worried about my dumb ass losing the borg key and all data along with it…
I’m thinking a clearly labelled hard drive with instructions, rotating the hard drive with a new one every 5 years or so.
Exactly. I have a document for my SO that describes what to do if I pass (where the money is, how the WiFi is set up, various important accounts, etc). It’s not a will (nothing about who gets what, though that’s assumed by the state to be my SO, or my kids equally if we pass together), just a document that explains the stuff I handle.
I have not used them myself, but M-DISC sounds like what you’re looking for. There are a few other alternatives listed on that Wikipedia article, too.
Thanks, I think the risk here is that there may not be hardware to read it.
From the suggestions here I’m thinking a hard drive with USB connection would be best. It won’t last 50 years but instead I’d replace it every 5 years or so. I’d use an error resistant file system and plug it in each year to add the new files.
This way I also get the chance to move it to newer technology in future instead of a new hard drive. It would then only need to survive for some period of time after I last replaced it, so there’s a good chance of it remaining readable for most of my life.
Thanks, I think the risk here is that there may not be hardware to read it.
M-Disc DVDs are readable by ordinary DVD drives. So you could simply put a USB DVD drive alongside those backup M-Discs on the shelf.
Yeah that’s an idea. It does seem like I’d need a lot of disks though. And I don’t actually have a disk reader or writer at all at the moment.
I usually use a dehydrator for ~3 days on my drives to make them shelf stable. So far I haven’t had any issues.
Tape. Amazon glacier if you’re okay with that.
And regular test restores. An untested backup is not a backup.
But when considering what I need to back up, I usually overestimate how much I or other people will care if it’s lost. Family photos are great, but what are the odds of someone saying “damn I wish we still had two dozen photos of that one barbecue?”
I have cloud with B2, I’m looking for cupboard storage that a random family member can pull out and browse through after I get put in a resthome (only half joking).
Is home tape storage feasible (and good for this use-case)?
In terms of what to backup, I’m running on the assumption that technology will be able to autofilter the good stuff at some point, no need to put much effort in now haha.
In your scenario, I’d be looking at ZFS or BTRFS for your live data, especially when taking photos into account. They’ll self-repair files that may run into decay issues, which I’ve seen a lot of with photos in all formats. Since you already keep off-site backups, I’d then just keep an extra drive around that you snapshot to from time to time.
So my offsites are an incremental backup, but at some point the oldest version is gone. I am keen for a completely separate, long term snapshot of what I had that could be thrown in a cupboard, and any random family member clearing my house out as I get moved into a rest home at 108 can go through the photos and find a good one to put on my headstone.
I am also keen for protection against doing something dumb and losing everything (like losing my hard drive and finding out for some reason I can’t access my backups because I lost the encryption key because I put it in bitwarden and they shut down years ago and I never moved the key over because I forgot it was stored there).
ZFS and BTRFS both provide that functionality. Have a look into the features.
So the drive doesn’t need to be hot, I can just plug in once a year and it auto-repairs?
No, the “live” filesystems will repair themselves when they detect problems. They keep revisions of your data, and run checksums constantly. When they find a file has inadvertently changed without access, it will restore said files. Think of it like Mac “Time Machine”, but it’s just the filesystem . You can restore stuff from points in time when needed.
Just read up on it.
Yeah I will read up on it, thanks for the tip!
USB hard drive? If we’re talking about a cold backup that’s easy to access a USB drive is reliable and easy.
Yeah based on the suggestions so far this seems like the best option. Just need to make sure I have a way to verify no files are corrupted (and if they are, which ones), and remember to swap it out for a new one every 5 years or so or each time I need a new drive.
This was a recent point of discussion on the 2.5 Admins podcast (https://2.5admins.com/2-5-admins-228/). Some good discussion on there.
My own thought is the best way to handle your family-member-finding-your-old-photos problem is the analog way: make some prints. It’s absolutely idiot proof, the methodology of keeping paper goods is well understood, and the technology is platform independent.
Yeah I really want a digital copy, but for redundancy physical photos sounds like a good idea. Maybe I should do the hard drive rotation thing mentioned in other comments, but each year when I’m updating I could print 100 photos as well. Have a suitcase or container where I keep the hard drives and the physical photos. Then I’d have a decent chance of all photos making it in digital form, plus a bunch of photos just in case the hard drives fail. The hardest part will be going back through the previous years finding an printing 100 photos from each year, but after that I don’t have to get through too many each year.
A hybrid is probably a good way forward. I had a career as a photographer for a while and I learned from that: going through 1000 photos takes very little time, but going through 10,000 takes an eternity. If you can star or mark your obviously important photos as you go along, it’ll take very little to print them at the end of the year.
I believe M-Disc to be the best consumer grade, optical solution out there. If you want to go commercial grade you are looking for LTO tapes, but your costs begin rising exponentially. If M-Disc claims are to be believed, they should last well longer than your requirement and be able to handle your data footprint using multiple, but not an unreasonable amount of discs.
No matter which solution you choose, if you are targeting multiple decades, you must save not only the media, but ideally the drive, computer and software used to archive. There is no guarantee that any of the existing technology will be relevant or backward compatible across several decades.
Thanks! I think this is probably a big risk of not being able to find the hardware to play it.
Through other conversations I think the answer is to instead get a normal drive, USB connection, and every few years replace the drive and copy the data to the new drive, using an error resistant file system and something like rsync that validates that the files arrived correctly.
As technology changes, I’d move the files as needed onto the more modern media.
ZFS with automatic snapshots and scrubbing. This will keep as many and as old snapshots as your like. It’ll ensure the files don’t rot. It’ll ensure the media doesn’t die, so long as you have enough redundancy and you replace disks as they die. This is what I’d trust for long term storage because I think I understand how and why it works. It should last as long as I feed it disks. If I delete something, I should be able to restore it from a snapshot. The hardware doesn’t need to be anything fancy. Just a Pi 4/5 with a couple of WD Elements would be fine. Could add more disks for more redudnancy. I’m running 2-disk residency.
You don’t have to touch the software if it’s not exposed to the Internet. Whatever works today on it will work 20 years from now, so long as the hardware works. A couple of spare Pis, SD cards and power supplies should let it last for decades.
I’d rather cold storage but am thinking of looking along these lines, ZFS or btrfs on a standard HDD that I add files on to once a year and replace the disk every few years.
I have a standard backup setup I just want something that is more point in time and not connected to all the automation, in case I automatically delete everything.
Yup, turn it on, let it do a scrub, then turn it off. I’d still use redudnancy though. Not merely to cover the case of the drive failing, but also to cover the bit rot use case. It’s exceedingly unlikely bits to rot at the exact same spot on two or more disks. When ZFS finds a checksum mismatch during a scrub (which indicates bit rot), it’ll be able to trivially recover the data from the drive where the checksum matches. It’ll then rewrite the rotten part.
Would that be two disks under a type of RAID or does ZFS have something?
With 2 disks that would be type mirror in ZFS-speak, completely built-in. Equivalent to RAID1 in terms of hardware fault tolerance.
You could do a 3-disk mirror or n-disk mirror really. The RAID5/6 rough equivalents are called RAIDzN where N is the number of disk failures they tolerate. E g. RAIDz1, RAIDz2, etc. You probably want a mirror unless you need more space than a single disk provides.
Ah thanks, that gives me something to research.
This is why I do my first-level of backups with
rsnapshot. It backs up to the plain filesystem using rsync and uses hard-links to de-dup between backups. No special filesystem, no encryption, restore is just an ‘rsync’ away.Yes my issue is that I seem to be replacing a drive somewhere every couple of years. I am keen for something that can be stored in a cupboard for years, preferably a decent chance at lasting decades.
Ah - I gotcha. That’s some terrible luck with drives.
I mean, there are a lot of drives. Two laptops with a drive each. A desktop/server with three drives, and a spare laptop used for Kodi is the current setup. I’m not counting but I think it’s three drives, one laptop, and one mobo since I started self-hosting perhaps 5 or 6 years ago.
The drives themselves, one was still under warranty, one was probably 3 or 4 years old, and the last was probably 6 or 8 and was in an old laptop and well used.
I think some of the drives have had a hard live while I messed around self-hosting, especially during my phase of trying out photo solutions.
Thinking about this, the only ancient information we are still able to access is painted or edged on stone or clay. How about some sort huge wall with thousands of QRcode like engravings?
Reminds me of project Silica. Media historically was more durable (stone/ ink and cloth paper, etc) but had a low data density. As density increased, so did fragility
Haha like
spoilers for the three body problem series
at the the end of the third three body problem book where they need to write something that will last for millions of years so they carve a message in huge letters into the rock
But I’m a millennial so if you think I own a huge amount of land you would be wrong 😛
I decided instead to use ZFS. Better protection than just letting something sit there. Your backups are only as good as your restores. So, if you are not testing your restores, those backups may be useless anyway.
ZFS with snapshots, replicated to another ZFS box. The replicated data also stores the snapshots and they are read-only. I have snapshots running every hour.
I have full confidence that my data is safe and recoverable.
With that said, you could always use M-disk.
I have automated backups including to cloud, but I want a separated manual system that cannot get erased if I mess something up (accidentally sync a delete, lose encryption key, forget to pay cloud bill). I have 3 2 1 but it’s all automated and backups are eventually replaced, if it’s not a critical failure I won’t necessarily know I’ve lost something.
Basically, I specifically want cold storage, and not cloud. I will only add, not delete from it. And I don’t want it encrypted.
Based on other conversations I’m planning on using duel disks mirrored, zfs, annual updates and disk checks with disks rotated out every 5 years (unless failing/failed). Handling the need for layman retrival of data by including instructions with the hard drives.
That sounds like a really good idea. You basically get the best of everything.
The cool thing about ZFS is the pool information is stored on the disks themselves. You can just plug them in and import the pools.
Basically, I specifically want cold storage, and not cloud. I will only add, not delete from it. And I don’t want it encrypted.
I have a client with a photographic studio. To give you an estimate, his data is around 14TB of mostly camera pictures with approximately 20 years or history and the owner believe it or not, relies on multiple external hard drives for cold storage, he has a 2TB Seagate thats like 2011-2012 old which still works.
To put in a cupboard tho, M disc is your best bet.
I’m thinking of using a HDD and keeping it at work, which is climate controlled. I’d bring it back every few months to sync the latest.
Since it’s constantly being used, I’m pretty confident it’ll be usable as a backup if my NAS fails, so it only needs to be “shelf stable” for a few months at a time. If you’re retired or something, a safe deposit box at your local bank should do the trick.
I have a cloud backup, so this isn’t about a critical loss of data. It’s about an accessible copy that isn’t encrypted and a layman could get the data off.
If my house burns down and I lose the copy, I can restore the data from the cloud backup (so long as I’m not in the house when it burns).
