Measuring Disk Performance using bonnie++

Over the past few years I've added any number of RAID units to our systems. The machines run the stable version of Debian Linux with one exception (at least right now). I started using bonnie++ (version 1.03) just to get a basic idea of how the various disks performed. All drives are 7200 RPM unless otherwise noted. These are my results. I've made no great effort to make conditions exactly identical for direct comparison, but rather the attempt was just to get an idea of how well some combination of machine and disk can perform out of the box. No optimizations have been done for any of these measurements. You can draw your own conclusions.

RAID Units

DataMax SCSI attach, Datamax 8200, 8x250GB IDE drives (http://www.dnfstorage.com)
SIVY SCSI attach, SIVY SA-7230, 12x400GB SATA drives (http://www.maxtronic.com/)
JetStor SCSI attach, JetStor 416S, 16x750GB SATA drives (http://www.acnc.com/)
iSCSI iSCSI attach, Infortrend A16E iSCSI, 1G RAM, 16x750GB SATA drives (http://infortrend.com/)
MD1000 SAS attach, Dell MD1000, 15x750GB SATA drives (http://www.dell.com/)
J4400 SAS attach, Sun J4400, 24x750GB SATA drives (http://www.sun.com/storage/disk_systems/expansion/4400/)

Sun Gear

V20Z SunFire V20Z, 2xAMD Opteron Processor 1210@1.8GHz processors, 12GB memory, Linux i686, XFS file system
V440 SunFire V440, 4xUltraSPARC IIIi@1.6GHz processors, 8GB memory, Solaris 8
V40z SunFire V40z, 4xAMD Opteron Processor 848@2.2Ghz processors, 12GB memory, Linux i686, XFS file system
X4500 SunFire X4500, 2xAMD dual-core AMD Opteron Processors 290@2.8Ghz, 16GB memory, 48x1TB SATA drives (JBOD), 6 internal SATA controllers, Linux x86_64, XFS and ZFS-FUSE file systems

           Version  1.03                 ------Sequential Output------   --Sequential  Input-  --Random-  
                                         -Per Chr-  --Block--  -Rewrite-  -Per Chr-  --Block--  --Seeks-- 
Machine   (disk type)              Size  K/sec %CP  K/sec %CP  K/sec %CP  K/sec %CP  K/sec %CP   /sec %CP
V20Z      (int. SCSI)                8G  23808  98  46284  17  18638   4  23023  81  55739   7  358.2   0
V20Z-a    (int. 10K SCSI)           24G  48250  92  44976   9  27568   4  42529  93  74166   6  410.6   0
V20Z-b    (ext. DataMax, RAID 5)     8G  14930  62  14655   5  13751   3  21564  76  59799   8  305.1   0
V20Z-c    (ext. JetStor 3, RAID 5)  24G  52456  98 127882  25  45031  11  34622  86 120788  14  387.9   0
V20Z-c    (ext. JetStor 3, RAID 10) 24G  52767  99  96015  18  29781   7  33038  82  98870  11  282.7   0
V20Z-c    (ext. JetStor 3, RAID 10) 24G  51469  97 109988  21  35997   8  21279  57  90992  11  283.9   0
V20Z-d    (ext. JetStor 3, RAID 5)  24G  52586  98 101945  21  34940   8  32388  87  86455  10  278.1   0
V20Z-d    (ext. JetStor 3, RAID 10) 24G  49727  93  73764  14  26474   6  28012  77  61675   8  379.7   0
V20Z-e    (ext. Sun J4400 RAID 5)    5G  38992  94 212198  50  73426  29  44046  98 381799  45  533.0   2


V40z      (int. SCSI)               23G  39827  88  55703  21  27903   8  16638  46  65353   9  200.2   0

V440      (int. SCSI)               16G  32918  97  57152  38  23289  26  34112  99  96554  41  244.7   2

X4500-a   (int. 1TB drive,ufs)      32G  73104  77  65851  33  20754  10  64572  65  81220  17  245.2   0
X4500-b   (int. 42x1TB drive,raidz) 32G 128589  99 514613  90 313714  76 112114  96 647724  68 5095.0  15
X4500-c   (int. 1TB drive,xfs)      32G  34695  57  39427   7  24592   6  48373  67  81152   6  262.7   0
X4500-d   (int. 4x1TB drive, LVM)   32G  58416  98 105942  23  38810  13  57046  96  70084   7  566.3   0
X4500-e   (int. 3x1TB drives, MD5)  32G  40877  68  38997   5  32820   6  57870  97 137544  14  380.6   0
X4500-f   (int. 8x1TB drives,raidz) 32G  38759  67  40561   8  20864   6  50810  85  55545   5  284.9   0

X3650-a   (int. 1TB drive,ufs)      32G  73104  77  65851  33  20754  10  64572  65  81220  17  245.2   0

Notes on Variations

V20Z-a uses internal; Sun SCSI controller on 10K rpm SCSI drive
V20Z-b has an older IDE unit, DNF (40MB transfer rate)
V20Z-c has a new SATA unit Jetstor (160MB transfer rate), internal Sun controller
V20Z-d same as V20Z but with 2nd Sun SCSI controller
V20Z-e used IBM HBA with Sun J4400. Poor result is because drive was not fully initialized.
X4500-a with SATA drives (JBOD). Using 1x1TB drive, x86/Solaris, UFS file system
X4500-b with 42x1TB SATA drive, x86/Solaris 10, ZFS raidz file system
X4500-c with SATA drives (JBOD). Using 1x1TB drive, amd64/Linux, XFS file system
X4500-d with SATA drives (JBOD). Using 4x1TB LVM drive, amd64/Linux, XFS file system
X4500-e with SATA drives (JBOD). Using 3x1TB MD RAID5 drive, amd64/Linux, XFS file system
X4500-f with SATA drives (JBOD). Using 8x1TB ZFS-Fuse raidz, amd64/Linux, ZFS file system

Dell Gear

Optiplex my old (2003) Dell desktop: 2xIntel@1.8GHz processors, 1.5GB memory, Linux i686, EXT3 file system
PE2600 4+ year old Dell PowerEdge 2600, 2xIntel@2.3 GHz processors, 6GB memory, Linux i686, EXT3 file system
PE6600 3+ year old Dell PowerEdge 6600, 4xIntel@2.8 GHz processors, 16GB memory, Linux i686, XFS file system
PE1850 Dell PowerEdge 1850, 2xIntel@3.0GHz processors, 4GB memory, Linux i686, EXT3 file system
PE1950 Dell PowerEdge 1950, 2xIntel quad core 5130@2GHz processors, 8GB memory, Linux i686, XFS file system
PE2950 Dell PowerEdge 2950, 2xIntel quad core 5345@2.33GHz processors, 16GB memory, Linux i686, XFS file system

           Version  1.03                 ------Sequential Output------   --Sequential  Input-  --Random-  
                                         -Per Chr-  --Block--  -Rewrite-  -Per Chr-  --Block--  --Seeks-- 
Optiplex  (int. IDE)                 3G  14821  98  30234  29  14333   7  14859  78  37262  10  160.9   0
Optiplex  (ext. DataMax, RAID 5)    .5G  14283  98  25341  27  11053  11  13597  81  59574  40  126.8   4

PE2600    (int. SCSI RAID 5)         8G  20217  99  71042  53  28436  13  20019  79  50729   9  621.0   2
PE2600    (ext. JetStor 3, RAID 10) 12G  29746  90  38619  18  13838   4  24300  69  31246   4  350.1   0
LVM PE2600(ext. JetStor 3, RAID 10) 12G  29706  92  38102  20  15972   5  22465  68  32390   5  254.6   0

PE6600    (int. SCSI RAID 5)        31G  20763  96  30596  23  23139  12  21983  78  73694  26  481.9   2
PE6600    (ext. SIVY, RAID 10)      31G  21946  99 107887  76  44115  20  24189  85  85119  21  129.1   0

PE1850    (ext. DataMax, RAID 5)  6584M   9783  21  10865   2   4304   1   9336  18   9203   1  290.1   0
PE1850-a  (ext. SIVY, RAID 5)     6584M  45972  97 120022  31  56724  15  48934  95 206050  34  284.6   0
PE1850-b  (ext. SIVY, RAID 5)     6584M  38649  82  40177  10  18893   5  36644  71  37623   6  415.3   0
PE1850-c  (ext. iSCSI RAID 5)        4G  43502  94  92347  26  37463   9  42271  97  92138   9  573.2   1
PE1850-d  (int. 300GB drive)      6576M  32869  70  57852  15  28523   8  46290  92  73628  14  248.8   0

PE1950    (int. SCSI, RAID 5)       16G  44037  97  52119  12  25645   5  38379  76  64303   5  173.8   0
PE1950    (ext. J4400 RAID 5)       16G  50016  99 214244  48  48833  12  49120  89 200283  20  358.1   1

PE2950    (int. SCSI RAID 5)        32G  37405  92  86949  11  40448   6  49650  76 140540  15  359.5   0
PE2950-a  (ext. MD1000 RAID 5)      32G  39941  99 143734  21  62375  11  41062  98 176976  20  139.2   0
PE2950-b  (int. 1.5TB)              63G  34986  42  93516  13  48320  12  84551  94 126340  11  315.5   0
PE2950-c  (ext. J4400 RAID 5)       16G  70120  99 248505  43  37435   6  38047  51  79344   6  424.3   0
PE2950-d  (ext. J4400 RAID 5)       32G  74831  99 207711  25  68454  13  77245  96 256411  16  457.0   0

R610      (int. 250GB 2inch)        63G  48525  47  48340   4  26601   5  60109  57  65073   4  226.1   0

Notes on Variations

PE6600 has SIVY attached to Adaptec 39160 adapter
PE2600 has SIVY attached to Adaptec 39160 adapter. Difference between PE2600 and LVM PE2600 is no LVM and with LVM
PE1850-a has SIVY attached to Adaptec 39160 adapter
PE1850-b has SIVY attached to LSI 21320 adapter
PE1850-c has Infortrend iSCSI on active 1Gbit LAN
PE1850-d is a single refurbished 300GB SCSI drive
PE2950-a has MD1000 attached via PERC/5e adapter
PE2950-b has MD1000 attached via PERC/5i adapter
PE2950-c has Sun J4400 attached via PERC/5e adapter. Note PE2950-a and PE2950-c use identical hardware, only memory and external array changed.
PE2950-d has Sun J4400 attached via Promise 8654 HBA

IBM Gear

x3650 IBM x-Series 3650, 2xIntel quad core 5440@2.83GHz processors, 32GB memory, Linux i686, XFS file system

           Version  1.03                 ------Sequential Output------   --Sequential  Input-  --Random- 
                                         -Per Chr-  --Block--  -Rewrite-  -Per Chr-  --Block--  --Seeks--
x3650-a   (int. SCSI RAID 5)        63G  70055  99  99329  12  78592   9  68331  90 274257  18  362.8   0
x3650-b   (ext. J4400 RAID 5)       63G  24229  27  22575   2  16139   3  39687  43  55273   4  318.4   0
x3650-c   (ext. J4400 RAID 5)       63G  73399  99 196737  24  90048  18  74464  98 359047   42 346.4   0

Notes on Variations

x3650-a has a Jetstor attached to Adaptec 39160 adapter
x3650-b has Sun J4400 attached via IBM HBA. Poor result is because drive was not fully initialized.
x3650-c has Sun J4400 attached via Adaptec 3805 HBA