This is my attempt to cut through the hype and uncertainty to find a storage subsystem that works. I compared XFS and EXT4 under Linux with ZFS under OpenSolaris. The machine is a 2200MHz Athlon 64 with 1GB of memory and 8 400GB disks attached to an Areca ARC-1220 SATA controller. Linux is 2.6.22.5 and OpenSolaris is 5.11 snv_70 "Nevada".

Aside from the different kernels and filesystems, I tested internal and external journal devices and software and hardware RAIDs. Software RAIDs are "raid-10 near2" with 6 disks on Linux. On Solaris the zpool is created with three mirrors of two disks each. Hardware RAIDs use the Areca's RAID-10 for both Linux and Solaris. Drive caches are disabled throughout, but the battery-backed cache on the controller is enabled when using hardware RAID. If an external journal is used, it is on a single disk with write caching disabled. The system is installed on the remaining disk, which is not involved in the testing.

XFS and ZFS filesystems were created with default parameters. EXT4 filesystems are created with -b 4096, to force 4096 block size when using the external journal. All filesystems were mounted with atime disabled. EXT4 is mounted with extents.

The benchmark uses Bonnie++ 1.03a, randomio 1.3, FFSB 5.2.1, postmark 1.51, dd, tar, and cpio with a tarball of Linux 2.6.22.5. FFSB and randomio were patched to build on Solaris, and to use the Solaris equivalent of Linux's O_DIRECT. The test proceeded as follows:

1. bonnie++
2. copy linux kernel tarball to filesystem under test
3. untar it
4. dd if=/dev/zero of=bigfile bs=1048576 count=4096
5. randomio bigfile 10 .25 .01 2048 60 1
6. find linux-2.6.22.5 | cpio -pdm linux
7. postmark 1, 10, and 100
8. ffsb read, write, mixed, and huge
9. tar -cf linux.tar linux

The results are below. EXT4 is fast for metadata operations, tar, untar, cpio, and postmark. EXT4 is much faster than the others under FFSB. EXT4 with hardware RAID and external journal device is ludicrously fast. EXT4 seems to have a bad interaction with software RAID, probably because mkfs fails to query the RAID layout when setting the filesystem parameters.

ZFS has excellent performance on metadata tests. ZFS has very bad sequential transfer with hardware RAID and appalling sequential transfer with software RAID. ZFS can copy the linux kernel source code in only 3 seconds! ZFS has equal latency for read and write requests under mixed loads, which is good.

XFS has good sequential transfer under Bonnie++. Oddly XFS has better sequential reads when using an external journal, which makes little sense. Is noatime broken on XFS? XFS is very slow on all the metadata tests. XFS takes the RAID layout into consideration and it performs well on randomio with hardware or software RAID.

The Areca controller seems to have some problem with large requests. The "huge" FFSB test does reads and writes in blocks of 1MiB. The hardware RAID is only half as fast as the software RAID in this test. To check that there's nothing weird about the Areca driver for Solaris, I spot-checked sequential transfer using dd, and rates were in excess of 150MB/s.