I have a 2x1TB RAID-1 array on one of my computers. It holds a backup.
It's starting to become too small, not because it's shrinking, but
because I'm getting more stuff. So, I need to do something that ends up
with a larger array using 3, maybe 4 disks. It'd be nice if it supports
disks of disparate sizes (and actually uses the extra space), so I can
upgrade by attrition.

I am by no means an expert at mdadm. Heck, I'm barely competent at it.
So I have no particular attachment to it. My friend uses btrfs to make
a (for me) massive array, some 6-8 disks and probably 40 TiB of space.
But it seems he spends a lot of time on administrivia, balancing the
array and whatnot. Maybe that's because it's so large? Dunno. I've
heard there are other filesystems that do similar things, but I'm not
familiar with them. Any recommendations?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* Eben King <eben@gmx.us> [25-03/21=Fr 15:32 -0400]:

I have a 2x1TB RAID-1 array on one of my computers. It holds
a backup. It's starting to become too small, not because it's
shrinking, but because I'm getting more stuff. So, I need to
do something that ends up with a larger array using 3, maybe 4
disks. It'd be nice if it supports disks of disparate sizes (and
actually uses the extra space), so I can upgrade by attrition.

I am by no means an expert at mdadm. Heck, I'm barely competent at
it. So I have no particular attachment to it. My friend uses btrfs
to make a (for me) massive array, some 6-8 disks and probably 40 TiB
of space. But it seems he spends a lot of time on administrivia,
balancing the array and whatnot. Maybe that's because it's so
large? Dunno. I've heard there are other filesystems that do
similar things, but I'm not familiar with them. Any recommendations?

LVM is used for this kind of thing. You can install LVM over RAID
(which is better than installing RAID over LVM):

https://serverfault.com/questions/217666/what-is-better-lvm-on-raid-or-raid-on-lvm

This gives a reasonable procedure to follow:

https://tomlankhorst.nl/setup-lvm-raid-array-mdadm-linux

However, note that if your filesystems aren't already part of
LVM, you'll either need to clobber them (trusting backups for
restoration, which is slow as well as scary) or shrink your
existing partition(s) and use a new one for LVM, planning
to extend that new partition by later adding more drives.

Hmm, what consumer-level motherboard supports
more than 2 NVMe drives? Or are you using SATA?

If you need vastly more information and have many hours
available for leisurely reading, you might want to look at <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/
html/configuring_and_managing_logical_volumes/
configuring-raid-logical-volumes_configuring-and-managing-logical-volumes>.

For dealing with humongous multiline URLs like that, I sometimes find
smush() {
tr -d '[:space:]' <<< "$*";
echo
}
to be helpful.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On Fri, Mar 21, 2025 at 03:32:39PM -0400, Eben King wrote:

I need to do something that ends up with a larger array using 3, maybe
4 disks. It'd be nice if it supports disks of disparate sizes (and
actually uses the extra space), so I can upgrade by attrition.

It depends on how wedded you are to the idea of making use of more of
the caoacity of mismatched (capacity) devices.

md and zfs are both fine choices on Linux but in the case of mirrors
will only give you mirrors of the smallest device up until you have
both sides of the mirror with the larger device, then you can grow it
in place to the full capacity.

So for example, with your existing 2x1TB if you add 2x8TB or something,
you could:

1. Break your 1TB mirror
2. Add an 8TB drive as the other half of your mirror
3. Resync your mirror
4. Use removed 1TB drive and remaining 8TB drive to make another (1TB) mirror 5. LVM the two mirrors together to get 2TB usable space
6. At later date replace 1TB drives with more 8TB ones and grow it all
in place to two mirrors of 8TB usable each.

Or start again with ZFS and make the equivalent stripe of two mirror
vdevs that again is only two sets of 1TB until the small drives are
replaced and it can be grown in place.

Now, if you immediately want to obtain more capacity you could do it by
making separate arrays, i.e. keep the 1TB mirror and make an 8TB mirror
for a total 9TB usable. Obviously it's a pain having to manually decide
where to put data.

If you absolutely want the capacity and you absolutely must have it in
one array, you could do btrfs. It is great for making redundant use of
mixed size drives. With a "raid1" profile and the drive sizes described
you would get 9TB usable. The performance won't be predictable though
because you won't be able to control where the extents for the files go.

You would also be able to replace the smaller drives later on to get
higher usable capacity.

Here is a capacity calculator:

https://carfax.org.uk/btrfs-usage/?c=2&slo=1&shi=1&p=0&dg=1&d=8000&d=8000&d=1000&d=1000

On Fri, Mar 21, 2025 at 05:11:19PM -0400, Boyan Penkov wrote:

If someone who has a firm grasp of the current state of btrfs vs bcachefs
for "power users who still need to get stuff other than 'administrativia' done" would speak up, I'd be glad for some perspective as well...

bcachefs is not ready for production use and its upstream author
specifically has asked Debian users not to use it as they want only
people who use the latest userland tools to test it out and report
bugs.

https://lore.kernel.org/all/36xhap5tafvm4boiy3acu5kxlhkvnp32wp3oknbfbkxbdkeq7r@galecvidi3bn/
https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/

Thanks,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Will Mengarini wrote:

Hmm, what consumer-level motherboard supports
more than 2 NVMe drives? Or are you using SATA?

Pretty much all of them, assuming you have a PCIe slot available
and a cheap adapter card. 1, 2, and 4 M.2 slots on a card are
all easy to find.

(Cheap: $45 for a 4 x M.2 PCIe to PCIe x16 slot card. $10 for
single adapters.)

-dsr-

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 3/21/25 12:32, Eben King wrote:

I have a 2x1TB RAID-1 array on one of my computers.  It holds a backup.
It's starting to become too small, not because it's shrinking, but
because I'm getting more stuff.  So, I need to do something that ends up with a larger array using 3, maybe 4 disks.  It'd be nice if it supports disks of disparate sizes (and actually uses the extra space), so I can upgrade by attrition.

I am by no means an expert at mdadm.  Heck, I'm barely competent at it.
So I have no particular attachment to it.  My friend uses btrfs to make
a (for me) massive array, some 6-8 disks and probably 40 TiB of space.
But it seems he spends a lot of time on administrivia, balancing the
array and whatnot.  Maybe that's because it's so large?  Dunno.  I've heard there are other filesystems that do similar things, but I'm not familiar with them.  Any recommendations?

Years ago, I built an LVM JBOD array from different size disks. It
worked, until one of the disks started failing. Basically, I had all of
the storage space but three times the chance of failure. I migrated my
data to a single large HDD. Eventually, an md mirror of 2 @ HDD's.


About 5 years ago, I built a ZFS pool with a mirror of 2 @ 3 TB HDD's. Performance was as expected. I soon added 40 GB of unused space from
the OS SSD as a read cache. Performance was enough for 1 GbE LAN.


About 1 year ago, the ZFS pool was getting full. So, I added another
mirror of 2 @ 3 TB HDD's. Without re-balancing, most reads came from
the old mirror (as expected) and most writes went to the new mirror (as
I learned). I thought filesystem churn would gradually balance the two mirrors, but this did not happen. I now believe that the root cause was
old snapshots and insufficient pruning and/or churn.


I backed up the data, destroyed the pool, installed two 180 GB SSD's,
created a new pool with two data mirrors of 2 @ 3 TB HDD's and one
special mirror of 2 @ 180 GB SSD's, and restored the data. Performance
should be enough for 2.5 GbE.


Be warned that ZFS has a non-trivial learning curve. ZFS is extremely flexible, so TIMTOWTDI, strategies for success can be inobvious, and
mistakes can be costly. Start by buying and reading Lucas "FreeBSD
Mastery: ZFS":

https://mwl.io/nonfiction/os#fmzfs


David

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Dan Ritter <dsr@randomstring.org> writes:

(Cheap: $45 for a 4 x M.2 PCIe to PCIe x16 slot card. $10 for
single adapters.)

The last part I can agree with but I recently paid something like $80
for a TXB122 card which is just a 2x NVMe m.2 to PCIe x4 adapter card
with the ASM2812 bridge chip. A card like this one: https://www.ebay.com/itm/256199958489

So, where do you get bigger adapter cards for those prices?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Eben King <eben@gmx.us> writes:

I have a 2x1TB RAID-1 array on one of my computers. It holds a backup.
It's starting to become too small, not because it's shrinking, but
because I'm getting more stuff. So, I need to do something that ends up
with a larger array using 3, maybe 4 disks. It'd be nice if it supports disks of disparate sizes (and actually uses the extra space), so I can upgrade by attrition.

If you want to keep it simple, maybe stick with RAID-1 with just bigger
disks. As I recall, mdadm can help the migration along since it lets you increase the individual drive size in the array.

Don't need to bring LVM or btrfs or other administrivia time sinks into
it.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Anssi Saari wrote:

Dan Ritter <dsr@randomstring.org> writes:

(Cheap: $45 for a 4 x M.2 PCIe to PCIe x16 slot card. $10 for
single adapters.)

The last part I can agree with but I recently paid something like $80
for a TXB122 card which is just a 2x NVMe m.2 to PCIe x4 adapter card
with the ASM2812 bridge chip. A card like this one: https://www.ebay.com/itm/256199958489

So, where do you get bigger adapter cards for those prices?

Prices via Newegg 24 hours ago. Here's a 4x for $28: https://www.newegg.com/p/3C6-00SN-00045

Note that if you need a bridge chip for PCIe bifurcation, those cards are more expensive, though you probably still overpaid a bit. Prices
fluctuate a lot.

-dsr-

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 3/21/25 18:06, Will Mengarini wrote:

* Eben King <eben@gmx.us> [25-03/21=Fr 15:32 -0400]:

I have a 2x1TB RAID-1 array on one of my computers. It holds
a backup. It's starting to become too small, not because it's
shrinking, but because I'm getting more stuff. So, I need to
do something that ends up with a larger array using 3, maybe 4
disks. It'd be nice if it supports disks of disparate sizes (and
actually uses the extra space), so I can upgrade by attrition.

I am by no means an expert at mdadm. Heck, I'm barely competent at
it. So I have no particular attachment to it. My friend uses btrfs
to make a (for me) massive array, some 6-8 disks and probably 40 TiB
of space. But it seems he spends a lot of time on administrivia,
balancing the array and whatnot. Maybe that's because it's so
large? Dunno. I've heard there are other filesystems that do
similar things, but I'm not familiar with them. Any recommendations?

LVM is used for this kind of thing. You can install LVM over RAID
(which is better than installing RAID over LVM):

https://serverfault.com/questions/217666/what-is-better-lvm-on-raid-or-raid-on-lvm

This gives a reasonable procedure to follow:

https://tomlankhorst.nl/setup-lvm-raid-array-mdadm-linux

However, note that if your filesystems aren't already part of
LVM, you'll either need to clobber them (trusting backups for
restoration, which is slow as well as scary) or shrink your
existing partition(s) and use a new one for LVM, planning
to extend that new partition by later adding more drives.

Hmm, what consumer-level motherboard supports
more than 2 NVMe drives?

That mobo has no M.2 slots. Technically I could get a PCIe M.2 card if
I found an exceptional deal on M.2 drives.

Or are you using SATA?

SATA. Spinning-rust even, but that'll change when 1-2 TB SSDs reach
price parity. Heck, they may already be there now that I know about SMR
vs CMR.

If you need vastly more information and have many hours
available for leisurely reading, you might want to look at <https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/
html/configuring_and_managing_logical_volumes/
configuring-raid-logical-volumes_configuring-and-managing-logical-volumes>.

Thanks. After de-duping and removing some old backups, the array has
some 136 GiB (14%) free, so while the end is in sight it's no longer threatening ominously.

For dealing with humongous multiline URLs like that, I sometimes find
smush() {
tr -d '[:space:]' <<< "$*";
echo
}
to be helpful.

tinyurl or similar is also good.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Dan Ritter <dsr@randomstring.org> writes:

Prices via Newegg 24 hours ago. Here's a 4x for $28: https://www.newegg.com/p/3C6-00SN-00045

Of course, now I understand. My problem is I don't have a spare x16 slot
to put the card into, just an x4. Hence the bridge chip needed. Next
build, definitely going for a motherboard that has two x16 slots.

Note that if you need a bridge chip for PCIe bifurcation, those cards are more
expensive, though you probably still overpaid a bit. Prices
fluctuate a lot.

Shrug. It was the cheapest I could find at the time so I'm fine with it.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)