FIO IOPS metrics



  • The below are from a hypervisor guest that is being replaced. The hypervisor is being replaced this year.

    fio random-read-test.fio
    random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
    fio-3.1
    Starting 1 process
    Jobs: 1 (f=1): [r(1)][100.0%][r=2978KiB/s,w=0KiB/s][r=744,w=0 IOPS][eta 00m:00s]
    random-read: (groupid=0, jobs=1): err= 0: pid=6882: Wed Jan  2 08:45:49 2019
       read: IOPS=363, BW=1456KiB/s (1491kB/s)(128MiB/90025msec)
    	clat (usec): min=110, max=21204, avg=2742.12, stdev=1538.32
    	 lat (usec): min=110, max=21204, avg=2742.58, stdev=1538.32
    	clat percentiles (usec):
    	 |  1.00th=[  215],  5.00th=[  239], 10.00th=[  306], 20.00th=[ 1401],
    	 | 30.00th=[ 1860], 40.00th=[ 2311], 50.00th=[ 2769], 60.00th=[ 3228],
    	 | 70.00th=[ 3687], 80.00th=[ 4113], 90.00th=[ 4621], 95.00th=[ 4883],
    	 | 99.00th=[ 5145], 99.50th=[ 7570], 99.90th=[12387], 99.95th=[13042],
    	 | 99.99th=[16319]
       bw (  KiB/s): min=  927, max= 3064, per=100.00%, avg=1455.89, stdev=267.33, samples=180
       iops        : min=  231, max=  766, avg=363.89, stdev=66.86, samples=180
      lat (usec)   : 250=6.85%, 500=5.17%, 750=0.44%, 1000=0.72%
      lat (msec)   : 2=19.96%, 4=43.70%, 10=22.81%, 20=0.35%, 50=0.01%
      cpu          : usr=0.32%, sys=2.42%, ctx=32770, majf=0, minf=30
      IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=32768,0,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=1
    
    Run status group 0 (all jobs):
       READ: bw=1456KiB/s (1491kB/s), 1456KiB/s-1456KiB/s (1491kB/s-1491kB/s), io=128MiB (134MB), run=90025-90025msec
    
    Disk stats (read/write):
    	dm-0: ios=32644/40, merge=0/0, ticks=87796/1416, in_queue=89209, util=97.74%, aggrios=32768/39, aggrmerge=0/1, aggrticks=87823/1391, aggrin_queue=89191, aggrutil=97.55%
      sda: ios=32768/39, merge=0/1, ticks=87823/1391, in_queue=89191, util=97.55%
    

    fio random-read-test-aio.fio
    random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
    fio-3.1
    Starting 1 process
    Jobs: 1 (f=1): [r(1)][97.2%][r=4840KiB/s,w=0KiB/s][r=1210,w=0 IOPS][eta 00m:01s]
    random-read: (groupid=0, jobs=1): err= 0: pid=7599: Wed Jan  2 11:03:12 2019
       read: IOPS=920, BW=3683KiB/s (3771kB/s)(128MiB/35592msec)
    	slat (usec): min=12, max=205, avg=29.12, stdev= 9.00
    	clat (usec): min=81, max=170612, avg=8651.10, stdev=10195.54
    	 lat (usec): min=99, max=170637, avg=8681.18, stdev=10195.54
    	clat percentiles (usec):
    	 |  1.00th=[   178],  5.00th=[   408], 10.00th=[  1336], 20.00th=[  2474],
    	 | 30.00th=[  3392], 40.00th=[  4228], 50.00th=[  5211], 60.00th=[  6783],
    	 | 70.00th=[  8848], 80.00th=[ 12387], 90.00th=[ 19792], 95.00th=[ 28443],
    	 | 99.00th=[ 50070], 99.50th=[ 61604], 99.90th=[ 86508], 99.95th=[ 96994],
    	 | 99.99th=[122160]
       bw (  KiB/s): min= 3008, max= 6112, per=99.69%, avg=3670.51, stdev=455.13, samples=71
       iops        : min=  752, max= 1528, avg=917.55, stdev=113.80, samples=71
      lat (usec)   : 100=0.01%, 250=3.36%, 500=2.62%, 750=1.14%, 1000=0.81%
      lat (msec)   : 2=7.06%, 4=22.39%, 10=36.69%, 20=16.08%, 50=8.83%
      lat (msec)   : 100=0.97%, 250=0.04%
      cpu          : usr=1.07%, sys=3.82%, ctx=31460, majf=0, minf=38
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=32768,0,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=8
    
    Run status group 0 (all jobs):
       READ: bw=3683KiB/s (3771kB/s), 3683KiB/s-3683KiB/s (3771kB/s-3771kB/s), io=128MiB (134MB), run=35592-35592msec
    
    Disk stats (read/write):
    	dm-0: ios=32380/33, merge=0/0, ticks=280932/2716, in_queue=283735, util=99.74%, aggrios=32768/29, aggrmerge=0/4, aggrticks=282528/2188, aggrin_queue=284690, aggrutil=99.71%
      sda: ios=32768/29, merge=0/4, ticks=282528/2188, in_queue=284690, util=99.71%
    

    fio four-threads-ranio.fio
    bgwriter: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
    queryA: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=1
    queryB: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=1
    bgupdater: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
    fio-3.1
    Starting 4 processes
    bgwriter: Laying out IO file (1 file / 256MiB)
    queryA: Laying out IO file (1 file / 256MiB)
    queryB: Laying out IO file (1 file / 256MiB)
    bgupdater: Laying out IO file (1 file / 32MiB)
    Jobs: 3 (f=3): [_(1),r(2),m(1)][34.4%][r=1876KiB/s,w=296KiB/s][r=469,w=74 IOPS][                                                                                                                                                             Jobs: 3 (f=3): [_(1),r(2),m(1)][34.7%][r=1965KiB/s,w=284KiB/s][r=491,w=71 IOPS][                                                                                                                                                             Jobs: 3 (f=3): [_(1),r(2),m(1)][34.9%][r=1848KiB/s,w=324KiB/s][r=462,w=81 IOPS][                                                                                                                                                             Jobs: 3 (f=3): [_(1),r(2),m(1)][35.2%][r=1889KiB/s,w=268KiB/s][r=472,w=67 IOPS][                                                                                                                                                             Jobs: 3 (f=3): [_(1),r(2),m(1)][35.5%][r=2048KiB/s,w=280KiB/s][r=512,w=70 IOPS][                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][35.9%][r=1909KiB/s,w=56KiB/s][r=477,w=14 IOPS][e                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][36.2%][r=1680KiB/s,w=0KiB/s][r=420,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][36.5%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][36.8%][r=1515KiB/s,w=0KiB/s][r=378,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][37.1%][r=1564KiB/s,w=0KiB/s][r=391,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][37.4%][r=1572KiB/s,w=0KiB/s][r=393,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][37.7%][r=1529KiB/s,w=0KiB/s][r=382,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][38.0%][r=1520KiB/s,w=0KiB/s][r=380,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][38.3%][r=1548KiB/s,w=0KiB/s][r=387,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][38.6%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][38.9%][r=1672KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][39.2%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][39.5%][r=1593KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][39.8%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][40.1%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][40.4%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][40.8%][r=2012KiB/s,w=0KiB/s][r=503,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][41.2%][r=2044KiB/s,w=0KiB/s][r=511,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][41.6%][r=1591KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][41.8%][r=1516KiB/s,w=0KiB/s][r=379,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][42.1%][r=1616KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][42.4%][r=1541KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][42.7%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][43.0%][r=1616KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][43.4%][r=1609KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][43.7%][r=1524KiB/s,w=0KiB/s][r=381,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][44.0%][r=1584KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][44.3%][r=1557KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][44.6%][r=1640KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][44.9%][r=1532KiB/s,w=0KiB/s][r=383,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][45.2%][r=1617KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][45.5%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][45.8%][r=1596KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][46.1%][r=1485KiB/s,w=0KiB/s][r=371,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][46.4%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][46.7%][r=1628KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][47.0%][r=1585KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][47.3%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][47.6%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][47.9%][r=1548KiB/s,w=0KiB/s][r=387,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][48.2%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][48.5%][r=1565KiB/s,w=0KiB/s][r=391,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][48.8%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][49.2%][r=1688KiB/s,w=0KiB/s][r=422,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][49.5%][r=1603KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][49.8%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][50.1%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][50.4%][r=1657KiB/s,w=0KiB/s][r=414,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][50.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][51.1%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][51.4%][r=1673KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][51.7%][r=1617KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][52.0%][r=1644KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][52.4%][r=1653KiB/s,w=0KiB/s][r=413,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][52.7%][r=1408KiB/s,w=0KiB/s][r=352,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][52.9%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][53.2%][r=1629KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][53.5%][r=1541KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][53.9%][r=1648KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][54.2%][r=1673KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][54.4%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][54.7%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][55.0%][r=1613KiB/s,w=0KiB/s][r=403,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][55.4%][r=1576KiB/s,w=0KiB/s][r=394,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][55.7%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][56.0%][r=1553KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][56.3%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][56.6%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][56.9%][r=1637KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][57.2%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][57.5%][r=1596KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][57.8%][r=1561KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][58.1%][r=1532KiB/s,w=0KiB/s][r=383,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][58.4%][r=1637KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][58.7%][r=1524KiB/s,w=0KiB/s][r=381,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][59.0%][r=1600KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][59.3%][r=1645KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][59.6%][r=1552KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][59.9%][r=1576KiB/s,w=0KiB/s][r=394,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][60.2%][r=1497KiB/s,w=0KiB/s][r=374,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][60.5%][r=1276KiB/s,w=0KiB/s][r=319,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][60.8%][r=1628KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][61.1%][r=1685KiB/s,w=0KiB/s][r=421,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][61.4%][r=1600KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][61.7%][r=1660KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][62.1%][r=1597KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][62.4%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][62.6%][r=1640KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][62.9%][r=1605KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][63.2%][r=1660KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][63.5%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][63.9%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][64.2%][r=1669KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][64.5%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][64.8%][r=1584KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][65.1%][r=1529KiB/s,w=0KiB/s][r=382,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][65.3%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][65.7%][r=1625KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][66.0%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][66.3%][r=1560KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][66.7%][r=1553KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][66.9%][r=1648KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][67.2%][r=1536KiB/s,w=0KiB/s][r=384,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][67.5%][r=1661KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][67.9%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][68.1%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][68.4%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][68.8%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][69.1%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][69.3%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][69.6%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][70.0%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][70.3%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][70.5%][r=1621KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][70.9%][r=1480KiB/s,w=0KiB/s][r=370,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][71.2%][r=1716KiB/s,w=0KiB/s][r=429,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][71.4%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][71.8%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][72.1%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][72.3%][r=1585KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][72.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][72.9%][r=1552KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][73.3%][r=1521KiB/s,w=0KiB/s][r=380,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][73.6%][r=1593KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][73.8%][r=1557KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][74.2%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][74.4%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][74.7%][r=1569KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][75.1%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][75.3%][r=1560KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][75.7%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][75.9%][r=1573KiB/s,w=0KiB/s][r=393,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][76.2%][r=1469KiB/s,w=0KiB/s][r=367,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][76.5%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][76.8%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][77.2%][r=1641KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][77.4%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][77.8%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][78.0%][r=1641KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][78.4%][r=1684KiB/s,w=0KiB/s][r=421,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][78.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][78.9%][r=1645KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][79.3%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][79.5%][r=1696KiB/s,w=0KiB/s][r=424,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][79.9%][r=1665KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][80.1%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][80.5%][r=1700KiB/s,w=0KiB/s][r=425,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][80.9%][r=1713KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][81.1%][r=1728KiB/s,w=0KiB/s][r=432,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][81.5%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][81.7%][r=1621KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][82.1%][r=1712KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][82.5%][r=1728KiB/s,w=0KiB/s][r=432,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][82.9%][r=1681KiB/s,w=0KiB/s][r=420,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][83.1%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][83.5%][r=1712KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][83.7%][r=1724KiB/s,w=0KiB/s][r=431,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][84.1%][r=1653KiB/s,w=0KiB/s][r=413,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][84.3%][r=1800KiB/s,w=0KiB/s][r=450,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][84.7%][r=1736KiB/s,w=0KiB/s][r=434,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][84.9%][r=1761KiB/s,w=0KiB/s][r=440,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][85.3%][r=1832KiB/s,w=0KiB/s][r=458,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][85.7%][r=1788KiB/s,w=0KiB/s][r=447,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][85.9%][r=1889KiB/s,w=0KiB/s][r=472,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][86.3%][r=1824KiB/s,w=0KiB/s][r=456,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][86.6%][r=1820KiB/s,w=0KiB/s][r=455,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][87.0%][r=1837KiB/s,w=0KiB/s][r=459,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][87.4%][r=1912KiB/s,w=0KiB/s][r=478,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][87.6%][r=1876KiB/s,w=0KiB/s][r=469,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][88.0%][r=1869KiB/s,w=0KiB/s][r=467,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][88.4%][r=2012KiB/s,w=0KiB/s][r=503,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][88.6%][r=1900KiB/s,w=0KiB/s][r=475,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][89.0%][r=2010KiB/s,w=0KiB/s][r=502,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][89.5%][r=2036KiB/s,w=0KiB/s][r=509,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][89.7%][r=2032KiB/s,w=0KiB/s][r=508,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][90.1%][r=2058KiB/s,w=0KiB/s][r=514,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][90.5%][r=2180KiB/s,w=0KiB/s][r=545,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][90.7%][r=2124KiB/s,w=0KiB/s][r=531,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][91.2%][r=2154KiB/s,w=0KiB/s][r=538,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][91.6%][r=2576KiB/s,w=0KiB/s][r=644,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][92.0%][r=2216KiB/s,w=0KiB/s][r=554,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][92.4%][r=2410KiB/s,w=0KiB/s][r=602,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][92.9%][r=2488KiB/s,w=0KiB/s][r=622,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][93.1%][r=2564KiB/s,w=0KiB/s][r=641,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][93.5%][r=2774KiB/s,w=0KiB/s][r=693,w=0 IOPS][eta                                                                                                                                                             Jobs: 2 (f=2): [_(1),r(2),_(1)][93.9%][r=2942KiB/s,w=0KiB/s][r=735,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][94.6%][r=2208KiB/s,w=0KiB/s][r=552,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][95.2%][r=2006KiB/s,w=0KiB/s][r=501,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][96.1%][r=2244KiB/s,w=0KiB/s][r=561,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][97.2%][r=2568KiB/s,w=0KiB/s][r=642,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][98.1%][r=2518KiB/s,w=0KiB/s][r=629,w=0 IOPS][eta                                                                                                                                                             Jobs: 1 (f=1): [_(2),r(1),_(1)][99.3%][r=3196KiB/s,w=0KiB/s][r=799,w=0 IOPS][eta                                                                                                                                                              00m:03s]
    bgwriter: (groupid=0, jobs=1): err= 0: pid=7629: Wed Jan  2 11:13:37 2019
      write: IOPS=312, BW=1250KiB/s (1280kB/s)(256MiB/209721msec)
    	slat (usec): min=11, max=794, avg=39.49, stdev=13.49
    	clat (msec): min=2, max=12285, avg=102.36, stdev=289.05
    	 lat (msec): min=2, max=12285, avg=102.40, stdev=289.05
    	clat percentiles (msec):
    	 |  1.00th=[    6],  5.00th=[   11], 10.00th=[   16], 20.00th=[   26],
    	 | 30.00th=[   36], 40.00th=[   47], 50.00th=[   61], 60.00th=[   77],
    	 | 70.00th=[   99], 80.00th=[  129], 90.00th=[  186], 95.00th=[  245],
    	 | 99.00th=[  443], 99.50th=[ 1720], 99.90th=[ 4212], 99.95th=[ 5604],
    	 | 99.99th=[ 7416]
       bw (  KiB/s): min=   24, max= 1640, per=100.00%, avg=1249.65, stdev=521.69, s                                                                                                                                                             amples=419
       iops        : min=    6, max=  410, avg=312.34, stdev=130.40, samples=419
      lat (msec)   : 4=0.25%, 10=4.11%, 20=9.73%, 50=28.36%, 100=28.42%
      lat (msec)   : 250=24.37%, 500=3.96%, 750=0.21%, 1000=0.03%, 2000=0.14%
      lat (msec)   : >=2000=0.42%
      cpu          : usr=0.32%, sys=1.33%, ctx=64621, majf=0, minf=25
      IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=0,65536,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=32
    queryA: (groupid=0, jobs=1): err= 0: pid=7630: Wed Jan  2 11:13:37 2019
       read: IOPS=156, BW=626KiB/s (641kB/s)(256MiB/419038msec)
    	clat (usec): min=113, max=297077, avg=6386.23, stdev=7067.73
    	 lat (usec): min=116, max=297078, avg=6386.64, stdev=7067.73
    	clat percentiles (usec):
    	 |  1.00th=[   196],  5.00th=[  1074], 10.00th=[  1598], 20.00th=[  2442],
    	 | 30.00th=[  3130], 40.00th=[  3752], 50.00th=[  4424], 60.00th=[  5276],
    	 | 70.00th=[  6783], 80.00th=[  8586], 90.00th=[ 12387], 95.00th=[ 17695],
    	 | 99.00th=[ 36439], 99.50th=[ 46400], 99.90th=[ 67634], 99.95th=[ 77071],
    	 | 99.99th=[104334]
       bw (  KiB/s): min=  128, max= 1973, per=46.26%, avg=587.91, stdev=256.81, sam                                                                                                                                                             ples=838
       iops        : min=   32, max=  493, avg=146.79, stdev=64.08, samples=838
      lat (usec)   : 250=3.19%, 500=1.14%, 750=0.26%, 1000=0.24%
      lat (msec)   : 2=9.72%, 4=29.27%, 10=42.03%, 20=10.04%, 50=3.72%
      lat (msec)   : 100=0.39%, 250=0.01%, 500=0.01%
      cpu          : usr=0.23%, sys=0.88%, ctx=65537, majf=65536, minf=36
      IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=65536,0,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=1
    queryB: (groupid=0, jobs=1): err= 0: pid=7631: Wed Jan  2 11:13:37 2019
       read: IOPS=154, BW=616KiB/s (631kB/s)(256MiB/425296msec)
    	clat (usec): min=95, max=294727, avg=6479.55, stdev=7250.47
    	 lat (usec): min=96, max=294727, avg=6479.96, stdev=7250.47
    	clat percentiles (usec):
    	 |  1.00th=[   206],  5.00th=[   930], 10.00th=[  1614], 20.00th=[  2474],
    	 | 30.00th=[  3130], 40.00th=[  3785], 50.00th=[  4490], 60.00th=[  5342],
    	 | 70.00th=[  6849], 80.00th=[  8717], 90.00th=[ 12518], 95.00th=[ 18220],
    	 | 99.00th=[ 36963], 99.50th=[ 45876], 99.90th=[ 68682], 99.95th=[ 83362],
    	 | 99.99th=[130548]
       bw (  KiB/s): min=  112, max= 3706, per=45.37%, avg=576.65, stdev=301.11, sam                                                                                                                                                             ples=850
       iops        : min=   28, max=  926, avg=143.98, stdev=75.16, samples=850
      lat (usec)   : 100=0.01%, 250=3.73%, 500=0.98%, 750=0.15%, 1000=0.20%
      lat (msec)   : 2=9.20%, 4=28.93%, 10=42.19%, 20=10.34%, 50=3.89%
      lat (msec)   : 100=0.36%, 250=0.02%, 500=0.01%
      cpu          : usr=0.26%, sys=0.87%, ctx=65541, majf=65536, minf=36
      IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=65536,0,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=1
    bgupdater: (groupid=0, jobs=1): err= 0: pid=7632: Wed Jan  2 11:13:37 2019
       read: IOPS=17, BW=69.8KiB/s (71.5kB/s)(15.9MiB/232986msec)
    	slat (usec): min=14, max=144, avg=43.42, stdev=14.67
    	clat (usec): min=92, max=181899, avg=8734.17, stdev=10918.07
    	 lat (usec): min=175, max=181944, avg=8778.72, stdev=10917.02
    	clat percentiles (usec):
    	 |  1.00th=[   161],  5.00th=[   306], 10.00th=[  1778], 20.00th=[  2769],
    	 | 30.00th=[  3556], 40.00th=[  4293], 50.00th=[  5145], 60.00th=[  6718],
    	 | 70.00th=[  8979], 80.00th=[ 12518], 90.00th=[ 19268], 95.00th=[ 27132],
    	 | 99.00th=[ 51643], 99.50th=[ 63701], 99.90th=[111674], 99.95th=[147850],
    	 | 99.99th=[181404]
       bw (  KiB/s): min=    8, max=  425, per=5.94%, avg=75.50, stdev=82.12, sample                                                                                                                                                             s=428
       iops        : min=    2, max=  106, avg=18.86, stdev=20.52, samples=428
      write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(16.1MiB/232986msec)
    	slat (usec): min=14, max=850, avg=47.04, stdev=19.35
    	clat (usec): min=1754, max=745718, avg=47818.37, stdev=79232.60
    	 lat (usec): min=1800, max=745766, avg=47866.56, stdev=79229.96
    	clat percentiles (msec):
    	 |  1.00th=[    4],  5.00th=[    5], 10.00th=[    6], 20.00th=[    7],
    	 | 30.00th=[   10], 40.00th=[   13], 50.00th=[   17], 60.00th=[   25],
    	 | 70.00th=[   39], 80.00th=[   65], 90.00th=[  130], 95.00th=[  207],
    	 | 99.00th=[  414], 99.50th=[  502], 99.90th=[  617], 99.95th=[  659],
    	 | 99.99th=[  743]
       bw (  KiB/s): min=    8, max=  368, per=5.98%, avg=71.43, stdev=78.72, sample                                                                                                                                                             s=458
       iops        : min=    2, max=   92, avg=17.84, stdev=19.67, samples=458
      lat (usec)   : 100=0.01%, 250=2.23%, 500=0.73%, 750=0.09%, 1000=0.17%
      lat (msec)   : 2=2.42%, 4=14.28%, 10=33.84%, 20=18.58%, 50=14.79%
      lat (msec)   : 100=6.10%, 250=5.00%, 500=1.49%, 750=0.26%
      cpu          : usr=0.20%, sys=0.19%, ctx=8195, majf=0, minf=27
      IO depths    : 1=99.8%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.0%, >=64=0.0%
    	 submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    	 issued rwt: total=4067,4125,0, short=0,0,0, dropped=0,0,0
    	 latency   : target=0, window=0, percentile=100.00%, depth=16
    
    Run status group 0 (all jobs):
       READ: bw=1271KiB/s (1302kB/s), 69.8KiB/s-626KiB/s (71.5kB/s-641kB/s), io=528M                                                                                                                                                             iB (554MB), run=232986-425296msec
      WRITE: bw=1196KiB/s (1225kB/s), 70.8KiB/s-1250KiB/s (72.5kB/s-1280kB/s), io=27                                                                                                                                                             2MiB (285MB), run=209721-232986msec
    
    Disk stats (read/write):
    	dm-0: ios=135129/69955, merge=0/0, ticks=870964/6969822, in_queue=7840785, u                                                                                                                                                             til=99.89%, aggrios=135140/69947, aggrmerge=0/8, aggrticks=870647/6968590, aggri                                                                                                                                                             n_queue=7839132, aggrutil=99.91%
      sda: ios=135140/69947, merge=0/8, ticks=870647/6968590, in_queue=7839132, util                                                                                                                                                             =99.91%

Log in to reply