OpenGFS_Performance (Nov 20 2003)


Contact the author.


This document contains results of some performance tests run with various
configurations.  Not much analysis here, just some data.


Contributed by Brian, January 2003.
----------------------------------

I (Brian) have done a few tests on the storage devices that I have tried out.

If you have tested OpenGFS on something else or you have different numbers,
let me know.  I will run more tests as time allows.

Here's a link to a Cisco webpage about Fibre Channel versus iSCSI perfomance:
http://www.cisco.com/warp/public/cc/pd/rt/5420/prodlit/csubl_br.htm

Media	   Max Read
-----      --------
ieee1394    12MB/s
iSCSI        9MB/s

The ieee1394 setup was an old WD310100 10GB UDMA33 in an ADS Pyro firewire
drive box (uses the oxford ieee1394->ide bridge chip) connected between a
P2 450 and a Dual P2 450 with ieee1394 (Duh!?!)

The iSCSI setup was an Athlon XP 1800+ with an ide raid setup with 2xWD800JB
acting as the target, and 2 K6-2 266's as the "clients" on fast ethernet (not
switched).



Contributed by Brian, April-June 2003.
----------------------------------
Results from running OpenGFS on OSDL machines.

---2.4.19-1 drive-1 subpool

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
cl038          496M  6209  62 10723  13  6471   8  8678  82 18728   9 210.7   3
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   479  19  1811  25   323   4  1126  26 +++++ +++   233   4
cl038,496M,6209,62,10723,13,6471,8,8678,82,18728,9,210.7,3,16,479,19,1811,25,323,4,1126,26,+++++,+++,233,4


---2.4.20-8 drives-8 subpools
cl049,5000M,6000,59,10612,12,7364,14,9827,92,23422,13,148.2,1,16,439,20,+++++,+++,257,6,897,27,+++++,+++,247,4

---2.4.20-8 drives-8 subpools-noatime
cl049,5000M,5950,59,10501,12,7373,14,9858,93,21995,11,111.2,2,16,105,4,2556,27,274,4,1010,30,+++++,+++,229,5
cl049,5000M,5956,59,10490,12,7358,14,9866,93,22017,11,111.0,2,16,100,4,2635,31,276,5,896,27,+++++,+++,215,4


---2.4.20-8 drives-8 subpools-noatime-w/shared_writ_mmap
cl049,5000M,5984,59,10467,12,7393,14,9905,93,21962,11,110.3,2,16,100,4,2992,26,275,6,940,29,2208,25,213,4

2003/05/31
The below tests were all run on a single node setup all on the same exact drive
just mounted with different diralloc methods, without restarting inbetween.
Machine only has 512MB ram so the tests couldn't be totally cached

DAMETH_RANDOM
spyder,1G,6087,65,14476,12,8277,6,9602,92,18033,5,175.4,1,16,776,21,+++++,+++,113,1,593,14,9486,63,115,1
spyder,1G,6069,65,14500,12,8015,6,9531,91,16600,4,175.7,1,16,848,21,18115,70,112,1,567,14,+++++,+++,112,1
spyder,1G,6081,65,14286,11,7999,6,9535,91,16581,5,176.1,1,16,72,1,8858,70,117,1,680,18,+++++,+++,115,1

DAMETH_SINGLE
spyder,1G,6092,65,14027,11,7909,5,9475,90,16177,5,171.6,0,16,829,22,+++++,+++,111,1,629 ,17,18969,74 ,112,1
spyder,1G,5773,62,13947,11,7856,6,9468,90,16240,4,158.0,0,16,801,21,12045,65 ,113,1,570 ,16,6450 ,66 ,113,1
spyder,1G,5747,62,13939,11,7863,6,9456,90,16247,4,160.2,1,16,825,22,+++++,+++,111,1,585 ,15,+++++,+++,111,1
spyder,1G,6031,64,13410,10,7769,5,9411,90,16009,5,137.0,0,16,73 ,1 ,3127 ,31 ,119,1,1670,28,+++++,+++,77 ,1
spyder,1G,5799,62,13346,10,7782,6,9457,90,16035,4,136.1,1,16,72 ,1 ,6992 ,68 ,117,1,1733,31,+++++,+++,75 ,1
spyder,1G,5795,62,13471,11,7810,6,9469,90,16039,4,138.4,0,16,72 ,1 ,2417 ,24 ,123,1,1717,29,+++++,+++,75 ,1
spyder,1G,5797,62,13421,10,7741,5,9382,90,16031,5,136.1,0,16,71 ,1 ,7196 ,65 ,122,1,1661,30,+++++,+++,76 ,1

DAMETH_RROBIN
spyder,1G,6064,65,14048,11,7869,6,9456,90,16277,5,157.9,0,16,738,19,+++++,+++,113,1,613,16,27336,70 ,113,1
spyder,1G,5785,62,13866,11,7845,5,9457,90,16278,4,150.1,0,16,70 ,1 ,7442 ,70 ,121,1,593,17,+++++,+++,115,1
spyder,1G,5666,61,13561,10,7857,6,9479,90,16311,4,153.1,0,16,850,22,+++++,+++,107,1,638,18,8755 ,66 ,112,1
spyder,1G,6058,65,13296,11,7776,5,9429,90,15923,4,148.8,0,16,804,21,+++++,+++,112,1,649,17,+++++,+++,114,1
spyder,1G,5668,61,13410,11,7753,5,9412,90,15965,4,149.5,1,16,802,20,+++++,+++,114,1,624,17,25736,67 ,113,1
spyder,1G,5865,63,13278,10,7799,6,9396,90,15923,5,140.5,0,16,796,20,+++++,+++,113,1,575,15,15162,70 ,113,1
spyder,1G,5678,61,13228,10,7741,5,9440,90,15957,4,140.8,0,16,72 ,1 ,7735 ,67 ,121,1,594,15,21861,70 ,114,1
avg-rr,1G,5826,63,13527,11,7806,5,9438,90,16091,4,148.7,0,16,798,20,7589 ,69 ,114,1,612,16,24978,69 ,113,1

IOzone

[root@cl049 ogfs]# /tmp/osdl.project/src/current/iozone -a
        Iozone: Performance Test of File I/O
                Version $Revision: 1.2 $
                Compiled for 32 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg.

        Run began: Sun Dec  1 21:34:33 2002

        Auto Mode
        Command line used: /tmp/osdl.project/src/current/iozone -a
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read   write    read rewrite    read   fwrite frewrite   fread  freread
              64       4   70639    7300   438279   853341  576300    7235  405107  118520  609573     6737     7266  363648   711267
              64       8  103904    5955   571343   984256  752479    6836  484972  167109  695243     6347     6652  453818   864412
              64      16  130337    5119   516183   954871  790194    7036  537908  203787  615714     5818     6234  453961   780235
              64      32  135023    4653   378811   955665  852868    6400  496249  185515  546785     5462     5839  420897   711217
              64      64  133322    5183   503988   842400  646591    5129  457156  132216  361484     4129     4175  447404   727384
             128       4   75250   12967   412876   595289  547043   11525  376475  133478  324896    12160    11339  358582   561369
             128       8  111114   13911   428142   684517  646625   13100  419707  203506  785068    11125     9291  403726   652994
             128      16  143491   11234   486738   753140  761822   13448  481250  270616  757304    11029    10892  422474   646477
             128      32  155349   12498   494204   743934  731262   12380  494105  280693  723401     7815     9904  391490   735579
             128      64  145948   10208   439928   612545  711139   10243  438304  201592  533439     7823     8263  347793   540102
             128     128  151308    7834   343134   406317  357521    7744  278252  144793  315285     6237     6271  337683   379872
             256       4   72644   23961   375892   486741  490441   24353    8054  144468  326101    20940    19595  341765   410240
             256       8  105918   22387   410256   515150  568945   24471  381553  218614  337740    20023    15518  383799   455514
             256      16  124270   24151   448308   533358  599471   25147  442880  327375  820576    21485    18637  420399   430232
             256      32  142302   22178   468019   574001  668308   23247  468016  363641  870801    19485    15512  429572   508977
             256      64  137266   19420   361082   421076  436076   19929  367312  293593  526743    15014    15662  344993   403793
             256     128  138980   15085   325685   353600  361054   15056  322018  193796  360578    12450    11805  309918   337738
             256     256  129226    9142   240613   242886  234644    9228  247112  137188  248051     9959    10411  231471   243115
             512       4   68193   36165   362608   385538  400921   39251  340205  149051  329259    17829     8198  336409   355560
             512       8   97672   41819   401253   423121  480737   43656  389656  241281  375908    33143    31395  379538   405371
             512      16  119318   37042   440610   449509  515628   44591  428469  363643  427744    35811    33909  422106   461240
             512      32  127999   41454   452707   492775  548181   42384  455132  439846  949956    33236    31439  434261   477166
             512      64  128225   34527   380967   398758  483476   36988  396902  338403  858995    29882    33339  341777   394758
             512     128  132744   25194   306592   310491  291399   29904  307329  221257  363908    24982    24813  289105   300123
             512     256  137043   22093   236054   238250  235620   21739  241508  164311  241169    19538    20068  237035   233255
             512     512  124452   13986   209917   212890  211220   13366  211831  134208  212007    12114    12420  212979   213596
            1024       4   66614   54159   356176   387872  376752   55205  332679  150587  337842    42307    33641  339297   359924
            1024       8   94535   62526   394304   409768  438919   64333  391731  249150  386275    47577    27823  385100   406662
            1024      16  112862   66030   439115   470586  487630   49566  439665  385685  423132    53358    53693  423850   457144
            1024      32  119166   63784   460443   489728  485536   56841  465661  512011  464389    50448    39510  431162   480087
            1024      64  119920   57322   386421   403776  406020   41960  378283  455113  577870    47313    44564  368607   390534
            1024     128  125814   50000   289187   301802  304403   49247  303681  264732  394300    36089    14826  271835   295443
            1024     256  130979   32583   227204   234699  230889   32308  231623  181913  240655    26246    26461  223728   232619
            1024     512  134188   23941   194787   201101  200429   26744  198642  154286  200547    20670    21626  197187   198528
            1024    1024  126638   15174   190440   192917  192589   14627  189558  129292  190829    13859    13798  186588   191366
            2048       4   50755   55872   369338   378627  348242   62385  340256  151771  341052    43909    55945  345537   357164
            2048       8   84586   76615   396671   424634  413735   26010  395826  252901  393545    59083    76738  390843   406266
            2048      16   99191   88997   447943   457250  467047   91277  435470  401808  440336    58218    81818  426671   456937
            2048      32  103980   89074   465456   476501  492423   90519  458681  553510  468976    64202    24399  458269   465879
            2048      64  109156   79636   373519   385108  392638   81736  378838  596210  372903    62215    68073  372094   381876
            2048     128  113771   72464   299371   304040  306815   73473  296766  283265  411907    45099    60737  294086   300648
            2048     256  117816   61210   231359   233524  232042   49524  231517  194104  232330    47584    51383  228750   233336
            2048     512   78772   46895   192753   195233  193408   43736  194676  165467  193408    35245    40644  190387   193793
            2048    1024  116735   28708   182385   184437  183676   29993  184057  149816  182709    28829    30050  181287   184107
            2048    2048  114798   16989   182093   183908  183513   16208  183874  108486  182923    13199    14949  181641   183331
            4096       4   61502   57703   367916   384675  347294   76312  344026  152545  339580    53164    63481  346062   359961
            4096       8   84526   97889   406548   422877  403865   97077  395750  253999  394037    68222    72926  391884   407320
            4096      16  101153  109806   446136   466462  457091  109912  445268  404344  445263    78093    80723  434034   457809
            4096      32  104746  101905   464135   487850  485826  110565  471180  559867  469560    80430    80506  458060   482222
            4096      64  107182  102185   379295   392824  388137  104007  384350  425469  383950    72946    73170  187924   387475
            4096     128  111686   91220   244479   255297  243446   63230  261408  245607  269244    60599    61741  291882   298693
            4096     256  101905   87355   229635   233110  232251   77743  231988  196714  236721    56881    48168  227138   231988
            4096     512  119222   71910   189454   191285  190564   68661  190352  170702  191840    53873    46714  188287   190715
            4096    1024  120772   31664   178521   180544  180297   49016  180266  158575  180250    43357    37746  178280   179916
            4096    2048  122447   31297   178040   161935   86455   24351  180051  125217  179224    30867    25484  176918   178981
            4096    4096  121962   17387   178140   180162  179719   16948  179846  120818  179854    15156    14063  177808   179917
            8192       4   60654   71288   373877   388873  345843   77357  350040  153150  342775    52592    67428  350325   364364
            8192       8   78414   82122   412258   426599  112113  105630  400979  257278  397941    73330    84482  397264   412841
            8192      16   98270  101691   450752   470778  453071  121183  449787  409089  449516    83249   100133  440856   460482
            8192      32  100706   98423   467953   489628  482620  125246  474650  572789  475092    87855    87950  462693   141387
            8192      64  105371   95532   380316   394301  389336  120267  386452  369207  387897    81793    84805  378383   391026
            8192     128   65573  116312   295697   302243  298684  113058  299942  300711  300459    73590    43490  292571   300271
            8192     256  114912   84730   229551   232767  231857  108216  232701  202511  232219    70578    65147  228030   231020
            8192     512  118698   79801   188534   190578  189520   96975  190175  174613  190458    67917    63334  187928   190436
            8192    1024  121381   62061   176612   178502  177820   54702  177986  163418  177743    57339    53285  176024   177939
            8192    2048  121235   42225   175718   177343  176925   51101  177323  155127  178017    41571    38996  175173   177089
            8192    4096  123173   27414   175331   176933  176700   31192  176605  143266  167600    27391    27551  175507   177101
            8192    8192  123330   16064   139840   177481  177020   17390  177423  118134  177101    13619    17199  175526   177335
           16384       4   58606   80935   377354   388348  343537   83842  351753  154295  344788    41948    79999  353675   364348
           16384       8   80118   99275   412737   426933  399249  110881  401479  257537  397834    74343    94421  399609   413559
           16384      16   97551  113164   453159   468435  450976  127035  452359  411502  450023    84913   115626  443613   459180
           16384      32  105184  123122   470466   489878  480624  130424  477055  586022  477278    58422    81635  465785   483403
           16384      64  102065  108354   386971   399055  396725  127010  391064  456964  396016    85330    97816  385406   397419
           16384     128  110607   73110   297290   302678  300590  125438  301132  280600  301248    76568    89967  294968   301481
           16384     256  111983  118746   229243   231746  230666  124189  232097  203361  139065    74116    81592  228056   231320
           16384     512   89690  102994   187425   189408  189392  105850  189563  175613  189152    73818    76347  186901   189469
           16384    1024  116262   85108   175694   177244  177110   95558  176770  165337  177812    34607    69596  175108   176988
           16384    2048  121589   72747   174602   176331  176092   74320  176696  162017  177112    51320    56940  174578   176065
           16384    4096  123581   45067   174835   176490  176675   45971  176748  155003  176715    27245    40806  174917   176485
           16384    8192  109915   29375   174615   176257  176001   33155  176088  127774  176299    31819    30732  175010   176380
           16384   16384  123021   17000   174919   176599  176557   18532  176160  119753  176557    13776    18239  174515   175914
           32768      64   98618  130164   386137   396981  394382   83139  393704  404054  392600    28940   105264  382138   392930
           32768     128   27759  128980   297226   303098  300800  133460  300861  311426  300905    36502    96051  227794   300783
           32768     256   33089  124091   229146   232423  231317  124285  231584  207993  231456    42238    93240  227937   231187
           32768     512   30152  120931   187183   189110  189097  114012  189009  144011  188953    64096    85713  186847   188981
           32768    1024   31469  116976   174430   176897  176349  110822  176760  166916  176590    75837    83294  175085   176807
           32768    2048   36482   93870   174485   176010  175874   96576  175818   66270  175556    43103    72725  173771   175349
           32768    4096   39641   71409   174367   144549  175805   69214  175908  162294  176080    47885    59084  174172   175818
           32768    8192   46661   48011   148323   175683  175562   53008  175755  115698  175619    21044    38553  174232   175925
           32768   16384   55833   32164   174268   176016  150172   30515  176158  140927  175977    28744    23955  173642   175900
           65536      64   68086  136417   388023   398824  393820  138480  394149  339043  394388    75280   113639  384842   395145
           65536     128   41283  137695   297999   304148  301091  139943  301847  281009  301631    56044   100601  295047   300614
           65536     256   45937  137421   228811   231827  231041  141224  231743  204874  231616    56649    96817  228076   231384
           65536     512   54178  120339   157413   188901  188539  141603  188759  176545  147982    80005    95478  186989   188685
           65536    1024   51149  130750   174951   176618  176578  135816  176306  168764  176369    79922    93992  174742   176272
           65536    2048   46902  116600   173837   175550  175321  107414  175446   79300  175663    69118    88330  161549   175707
           65536    4096   53272   97768   174096   175553  175680  103211  175529   55197  175639    71456    77574  173983   175611
           65536    8192   66119   68829   174012   175477  149423   85412  175465   58839  162234    37890    61443  173882   175726
           65536   16384   82632   53202   174041   175646  175725   55085  175622   87228  151509    25471    42154  174083   175738
          131072      64   95429  138969   388988   399485  395999  132944  395552  363733  394739    78121   116849  387751   397869
          131072     128   66718  137134   283145   303978  300613   85944  301870  263998  301664    82523   102859  295774   301179
          131072     256   65918  144249   229386   232428  220037  143171  231438  202640  231023    77083    95431  219368   231200
          131072     512   98849  144080   187014   188929  188818  148790  182807  177080  188796    79701    98147  181279   187869
          131072    1024   96076  141096   174883   176474  176315  147678  176456  166924  176436    79362    98061  174789   176356
          131072    2048   81811  127134   168863   169202  175559  141299  175588  127555  175573    82772    94488  173811   175467
          131072    4096  107518  114333   173922   164209  175497  130260  175517  125186  175413    79942    80393  173877   175431
          131072    8192  111901   98853   167655   170455  175559  113333  169235  156974  175443    48530    75527  167784   175518
          131072   16384  107110   61933   174101   175687  175729   86208  175608  152998  175680    41663    60572  173808   165628
          262144      64   13020  112700    42747   394455  393223  124052  389338  470243  393477    19506   113815  385381   394901
          262144     128   15371  133769   298147   301697  300951  143124  298755  308377  301854    19473    58529  295764   298555
          262144     256   20189   74484   227287   232317  229878  148490  230370  204816  231653    20036    60497  228493   230148
          262144     512   20630   76719   185969   188965  187726  146623  188940  176540  186789    20298    60244  185737   188629
          262144    1024   19955   74451   174773   176450  175275  132601  175237  168741  176344    20045    61698  174600   175410
          262144    2048   20964   72563   173925   174659  174180  131313  175481  166014  175328    19150    67785  173985   175327
          262144    4096   20765   66684   173942   175495  173607  142211  173601  166563  172731    18834    58338  173921   175298
          262144    8192   20816   66239   174071   175670  173863  116763  174608  166340  174454    18056    77315  174024   175515
          262144   16384   20418   99403   173984   175421  174730  104308  175450  161454  175540    17342    65884  173659   175516
          524288      64   12963   15093    22037    24270   17824  132031   10041  581512   12881    11142    13853   20989    22032
          524288     128   13595   19545    21568    23149   29501  133515   18420  291665   19196    11351    14196   20515    22792
          524288     256   13370   33251    22248    22686   37552  134541   23054  204853   23615    11324    14258   20572    22033
          524288     512   13405   33429    21823    22157   43202  140909   23402  176111   25303    11009    14075   19772    22400
          524288    1024   13417   29700    21507    25063   47239  137215   26206  167976   27702    10962    14170   20494    22064
          524288    2048   13250   27352    22660    22344   47140  135819   25911  167212   28402    11832    14004   19979    22673
          524288    4096   13318   23560    20811    22956   49508  139333   28701  167150   29863    10867    14283   20522    21508
          524288    8192   13518   23477    21198    22639   38236  135270   23791  166918   28509    11090    14007   19839    21725
          524288   16384   13713   13689    19946    21716   43479  124958   26747  165479   29265    11328    13573   19962    21719

iozone test complete.
SourceForge Logo