Someone building their first NAS joined our Discord server recently, and they are in an extremely common conundrum. They are planning to run TrueNAS, but they don’t have enough room in their budget to fill their server with giant hard disks. He isn’t the first person to encounter this conundrum.

The Jonsbo N1 and the Topton N5105 at Brian's House

I wish I could just give you a simple answer that you could run away with, but it just isn’t that simple. When you don’t have an unlimited budget, you often need to make compromises. In this instance, there are at least a handful of sensible compromises to choose from, but each has different pros and cons.

A ZFS vdev can only grow upward and not outward

If you build a 4-disk RAID-Z device, then it will always be a 4-disk RAID-Z device. You can replace all four disks one at a time with larger disks, and then you will have a larger RAID-Z device. It will still have four disks.

You can add four more disks to your server, create a new RAID-Z device with those new disks, and use this new vdev to make your existing ZFS filesystem bigger.

The bummer here is that you are now dedicating the capacity of another entire disk to parity. You also have to have four drive bays and SATA ports free in your server, and you have to buy four new disks at the same time.

Other RAID systems allow you to add one disk at a time. Wouldn’t it be nice to be able to turn your 4-disk RAID-Z into a 5-disk RAID-Z? The OpenZFS team has been working on this for a few years now. I don’t know when it will be available in TrueNAS, but expanding your RAID-Z or RAID-Z2 could be a possibility by the time you need to add storage. I wouldn’t be willing to bet on that, but it is certainly possible!

If you could add disks to an existing RAID-Z vdev, then you would only be using one disk’s worth of storage on parity by the time you got up to an 8-disk RAID-Z.

Why is ZFS like this?!

ZFS was created by Sun Microsystems, and Sun used to be in the business of selling gigantic enterprise servers. Big businesses with datacenters almost always fill every available drive bay when they purchase a new server. They will never need to add a disk to their RAID-Z2.

Customers didn’t need the feature, and Sun was almost definitely happier selling more disks. I used to be one of their customers. Sun charged an awful lot more for the same SCSI disk that Compaq would also sell you, and Compaq was already charging an awful lot more than Seagate.

What are the options if I can’t afford to fill my server with the largest drives?

There do happen to be a lot of good options, but there is a problem. Compromises often have pitfalls. If you clicked on this blog searching for an answer to this question, then there is a really good chance that you don’t already know what potential downsides are involved with the other options, and you might not have a good understanding of how much risk may be involved if you change your strategy.

It is easy to just install something like TrueNAS, set up a big, honkin’ RAID-Z2, and be done with it. You may be spending more money that you need to on disks, but you aren’t wasting that money. If you don’t have the experience to understand some of the risks of doing things another way, then that extra money you spend is helping you rely on someone else’s expertise.

I will do my best to call out any pitfalls!

RAID is not a backup!

This is repeated often, so I will try to keep this to a paragraph or two.

You can install TrueNAS, set up a RAID-Z2, and you can survive up to two disk failures. That is great, but so many other things can go wrong. You might accidentally delete your wedding photos. You might be attacked by ransomware that encrypts all your files. Your SATA controller hardware or driver might have a bug, and it winds up writing random data to all your disks.

There are a lot of things TrueNAS with RAID-Z2 won’t protect you from. If you are just torrenting TV shows, then maybe you don’t need a backup. You may not even really need to pay for redundancy at all! Maybe you are storing the video of your baby’s first steps, and you better have backups.

RAID prevents downtime. Backups help you recover from disasters.

Skip the RAID. Rely on backups.

This is the route I am taking now. I only have one small RAID left in my life, and it will probably go away next time I upgrade that piece of hardware. Mechanical hard drives are now much bigger than I need, and drives you can buy today are big enough to hold all my data for at least the next several years.

I have a 14-terabyte USB drive attached to a Raspberry Pi, and that is running a Dropbox-like software package called Seafile. All my data is synced to this Seafile server as quickly as possible. Seafile keeps 90 days’ worth of history, and that server is at my friend Brian Moses’s house. This is my off-site backup, and it is about a 15-minute drive from here.

My Seafile Pi

Seafile encrypts my data before uploading to the server, and the hard drive attached to the Raspberry Pi is encrypted using LUKS.

I have a 14-terabyte USB hard drive attached to my Homelab server. My NAS virtual machine stores all its data on that external drive, and it runs a Seafile client to pull down any new data as quickly as it possibly can. The NAS storage is using BTRFS, and I have a reasonable snapshot schedule there.

I also have a 12-terabyte SATA hard disk in my desktop computer. I am running a Seafile client here as well. My laptop has much less storage, so it only syncs a subset of my data using Seafile.

There is no RAID involved here, but it almost feels like there is. I have three disks with full copies of my current data. One of those disks is at a different physical location. The most important data is also synced to my laptop. My data is pretty well protected.

This is important. If your data is valuable to you, you need to have a good backup plan, and at least one of your backups needs to be somewhere outside of your home. Whether you decide to put a RAID in your NAS or not, you need backups.

Use RAID-Z1 or RAID-Z2, but buy smaller disks

Maybe your budget would allow you to fill every available drive bay with the smallest hard disks you can find. The sweet spot for hard disk pricing falls in the 12- to 18-terabyte range. Smaller or larger disks will cost a few extra dollars per terabyte.

You can buy a pair of 18-terabyte disks for about $540, put them in a mirror, and wind up having 18 terabytes of usable storage. One disk can fail, and you won’t lose data.

Stack of hard disks

You could instead buy five 6-terabyte disks for around $480, put them in a RAID-Z2, and still have the same 18 terabytes of usable storage. In this configuration, two disks can fail before you lose data.

You could instead put those five 6-terabyte disks in a RAID-Z1, and you would have 24 terabytes of usable storage. The downside here is that you can only lose one disk before your data is toast.

Why five disks? That’s how many 3.5” hard-drive bays the Jonsbo N1 case has, and that is the case Brian used in the 2023 DIY NAS!

Please don’t use RAID-Z1 or RAID 5 without a good backup!

The spec sheet for every hard disk lists how often you will encounter a single-bit read error. The likelihood seems infinitesimally small, but when we started building arrays with disks over around 100 gigabytes, the odds of hitting at least one of those errors started approaching 100%. Our disks are 100 times larger now.

The bigger the disks get, the longer it takes to rebuild onto a fresh disk. When you rebuild an array, you will be reading every byte from the remaining disks. You are giving your drives a workout at exactly the time when you can’t afford to lose another disk.

I have not been bitten by this yet. I once had a RAID 5 at home built out of 320-gigabyte hard disks. I was lucky. The second drive failed roughly two days after I replaced the first failed drive. That was a close call. I haven’t built a RAID 5 array since then.

Forget about TrueNAS

This is where I start to run out of newbie-friendly advice. I am intimately aware of how Linux’s logical volume manager (LVM) and all of the common Linux filesystems work. My own NAS is not newbie-friendly. My servers are all manually configured Debian or Ubuntu installs.

I know for sure that openmediavault uses LVM along with your choice of various filesystems supported by the Linux kernel. What I don’t know is how easy it is to configure this stuff.

Linux’s RAID layer lets you do all sorts of useful stuff. You can run a RAID 10 with only two disks, and you can grow that RAID 10 device by a single disk at time! Linux also lets you add a single disk at a time to your RAID 5 or RAID 6 arrays.

That means you could start with two 18-terabyte disks in a RAID 10, which will leave you with 18 terabytes of usable storage. When you fill that up, you can add a third disk to bring your usable storage up to 27 terabytes.

You could start with three 6-terabyte disks in a RAID 5 to get yourself 12 terabytes of usable storage. Every time you add a new 6-terabyte disk, your usable storage will go up by 6 terabytes.

What is the catch? You can use a battle-tested filesystem like EXT4 on top of that RAID, but then you will be missing out on ZFS’s checksumming and all the features that come with it. If you want the checksumming, you can run Btrfs, but Btrfs hasn’t been a stable filesystem for as long as EXT4 or even ZFS.

The other catch is that you have to think about it. You can put some extra thought and effort into this, or you can consider the extra expense of filling up your server’s drive bays all at once as the cost of not having to worry about making any bad choices.

You can check out alternative RAID-like solutions!

Unraid is popular. It costs money to use their filesystem, but Unraid OS is also more than just a filesystem. Unraid OS wants to be a little homelab-in-a-box. TrueNAS Scale seems to want to fill the same niche. I have no first-hand experience here, but the vibe I get on our Discord server is that Unraid is doing better at the job.

The Unraid filesystem lets you mix and match disk of various sizes while still having redundancy. I am not a fan of Unraid’s filesystem. I usually make a weird face whenever anyone says Unraid. I would probably have to write 1,500 words to properly express how I feel about Unraid.

Another choice is using Btrfs’s built-in RAID system. I hear that its RAID 1 and RAID 10 implementations are stable and reliable. It might be safe to use Btrfs’s built-in RAID 5 and RAID 6, but you have to make sure you store your Btrfs metadata using RAID 1. Please don’t take my word on that. My source for that is just a couple of Reddit threads!

You already have plans for regular backups, right? As long as you have good backups, then you won’t have any trouble recovering if one of these goes sideways.

Maybe you don’t need a dedicated NAS server!

People stop by the Butter, What?! Discord server to tell us about the parts they have chosen for their build, and they ask if they have chosen fast enough hardware for a NAS. We explain that 15-year old hardware can saturate gigabit Ethernet, and that it is nearly impossible to spec out a NAS that is too weak for home use.

Then they explain that they need to transcode video for Jellyfin or Plex. Servers exist on a spectrum. They don’t really want to build a NAS. They want to build a streaming server that is also a NAS. They are asking if they have picked out a good truck for hauling lumber, but they left out that they really need it to run fast laps at the Nürburgring.

NAS stands for network-attached storage. Sharing a USB flash drive that is plugged into your home router would qualify it as a NAS. It may not be an exciting NAS, but it is still a NAS.

Maybe the right choice for you is to spec out a Jellyfin streaming server and just stick one big SATA disk in there to hold the media. Maybe a better choice would be to build a virtual machine host for your homelab that just happens to have one big SATA disk to attach to a NAS virtual machine.

Do you want to build a file server that can also transcode video, or do you want to build a streaming server that also serves files? You might still arrive at the same destination either way, but sometimes it helps to look at things from a different perspective!

I want to build a NAS anyway!

Some folks have a lot of computers running at home because they have to. Some folks have a lot of computers running just because it is a fun hobby. Most of us are somewhere in between.

Server rack in your home office

If you think it will be fun to pick up a 17” rack and fill it with a stack of servers, switches, and routers, then I absolutely think you should do that. Especially if you think it will be fun and interesting!

That is what I used to do for a living, except with lots of server and network racks. I don’t want any of that at home. I want my gear at home to be relatively quiet and low power. I want to squeeze about as much as I can out of my spending.

Conclusion

I am excited that you are working on taking control of your own data. There isn’t a shortage of good reasons to avoid uploading all your data to services like Google Drive or Dropbox, but these services provide value that isn’t always immediately obvious. One of the most important things they bring to the table is durability.

Don't burn down your NAS!

Your home NAS with a big RAID-Z2 array is probably fairly durable, but there are so many unfortunate things that can take out that server with no warning: fire, flooding, a power surge, or a configuration mistake. Dropbox and Google Drive store multiple copies of your data, and they seem to have a good backup plan.

If you have the extra cash to spend, and you want to make sure you will be able to watch Manimal, Sledge Hammer, or Automan even while suffering from a disk failure, then you should definitely build a NAS with a nice RAID. If you are preserving irreplaceable video footage from your wedding, then you ought to think about off-site backups before worrying about the uptime of your storage server!