FIO IOPS metrics
-
The below are from a hypervisor guest that is being replaced. The hypervisor is being replaced this year.
fio random-read-test.fio random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.1 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=2978KiB/s,w=0KiB/s][r=744,w=0 IOPS][eta 00m:00s] random-read: (groupid=0, jobs=1): err= 0: pid=6882: Wed Jan 2 08:45:49 2019 read: IOPS=363, BW=1456KiB/s (1491kB/s)(128MiB/90025msec) clat (usec): min=110, max=21204, avg=2742.12, stdev=1538.32 lat (usec): min=110, max=21204, avg=2742.58, stdev=1538.32 clat percentiles (usec): | 1.00th=[ 215], 5.00th=[ 239], 10.00th=[ 306], 20.00th=[ 1401], | 30.00th=[ 1860], 40.00th=[ 2311], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 3687], 80.00th=[ 4113], 90.00th=[ 4621], 95.00th=[ 4883], | 99.00th=[ 5145], 99.50th=[ 7570], 99.90th=[12387], 99.95th=[13042], | 99.99th=[16319] bw ( KiB/s): min= 927, max= 3064, per=100.00%, avg=1455.89, stdev=267.33, samples=180 iops : min= 231, max= 766, avg=363.89, stdev=66.86, samples=180 lat (usec) : 250=6.85%, 500=5.17%, 750=0.44%, 1000=0.72% lat (msec) : 2=19.96%, 4=43.70%, 10=22.81%, 20=0.35%, 50=0.01% cpu : usr=0.32%, sys=2.42%, ctx=32770, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32768,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1456KiB/s (1491kB/s), 1456KiB/s-1456KiB/s (1491kB/s-1491kB/s), io=128MiB (134MB), run=90025-90025msec Disk stats (read/write): dm-0: ios=32644/40, merge=0/0, ticks=87796/1416, in_queue=89209, util=97.74%, aggrios=32768/39, aggrmerge=0/1, aggrticks=87823/1391, aggrin_queue=89191, aggrutil=97.55% sda: ios=32768/39, merge=0/1, ticks=87823/1391, in_queue=89191, util=97.55%
fio random-read-test-aio.fio random-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 fio-3.1 Starting 1 process Jobs: 1 (f=1): [r(1)][97.2%][r=4840KiB/s,w=0KiB/s][r=1210,w=0 IOPS][eta 00m:01s] random-read: (groupid=0, jobs=1): err= 0: pid=7599: Wed Jan 2 11:03:12 2019 read: IOPS=920, BW=3683KiB/s (3771kB/s)(128MiB/35592msec) slat (usec): min=12, max=205, avg=29.12, stdev= 9.00 clat (usec): min=81, max=170612, avg=8651.10, stdev=10195.54 lat (usec): min=99, max=170637, avg=8681.18, stdev=10195.54 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 408], 10.00th=[ 1336], 20.00th=[ 2474], | 30.00th=[ 3392], 40.00th=[ 4228], 50.00th=[ 5211], 60.00th=[ 6783], | 70.00th=[ 8848], 80.00th=[ 12387], 90.00th=[ 19792], 95.00th=[ 28443], | 99.00th=[ 50070], 99.50th=[ 61604], 99.90th=[ 86508], 99.95th=[ 96994], | 99.99th=[122160] bw ( KiB/s): min= 3008, max= 6112, per=99.69%, avg=3670.51, stdev=455.13, samples=71 iops : min= 752, max= 1528, avg=917.55, stdev=113.80, samples=71 lat (usec) : 100=0.01%, 250=3.36%, 500=2.62%, 750=1.14%, 1000=0.81% lat (msec) : 2=7.06%, 4=22.39%, 10=36.69%, 20=16.08%, 50=8.83% lat (msec) : 100=0.97%, 250=0.04% cpu : usr=1.07%, sys=3.82%, ctx=31460, majf=0, minf=38 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32768,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): READ: bw=3683KiB/s (3771kB/s), 3683KiB/s-3683KiB/s (3771kB/s-3771kB/s), io=128MiB (134MB), run=35592-35592msec Disk stats (read/write): dm-0: ios=32380/33, merge=0/0, ticks=280932/2716, in_queue=283735, util=99.74%, aggrios=32768/29, aggrmerge=0/4, aggrticks=282528/2188, aggrin_queue=284690, aggrutil=99.71% sda: ios=32768/29, merge=0/4, ticks=282528/2188, in_queue=284690, util=99.71%
fio four-threads-ranio.fio bgwriter: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32 queryA: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=1 queryB: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=mmap, iodepth=1 bgupdater: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 fio-3.1 Starting 4 processes bgwriter: Laying out IO file (1 file / 256MiB) queryA: Laying out IO file (1 file / 256MiB) queryB: Laying out IO file (1 file / 256MiB) bgupdater: Laying out IO file (1 file / 32MiB) Jobs: 3 (f=3): [_(1),r(2),m(1)][34.4%][r=1876KiB/s,w=296KiB/s][r=469,w=74 IOPS][ Jobs: 3 (f=3): [_(1),r(2),m(1)][34.7%][r=1965KiB/s,w=284KiB/s][r=491,w=71 IOPS][ Jobs: 3 (f=3): [_(1),r(2),m(1)][34.9%][r=1848KiB/s,w=324KiB/s][r=462,w=81 IOPS][ Jobs: 3 (f=3): [_(1),r(2),m(1)][35.2%][r=1889KiB/s,w=268KiB/s][r=472,w=67 IOPS][ Jobs: 3 (f=3): [_(1),r(2),m(1)][35.5%][r=2048KiB/s,w=280KiB/s][r=512,w=70 IOPS][ Jobs: 2 (f=2): [_(1),r(2),_(1)][35.9%][r=1909KiB/s,w=56KiB/s][r=477,w=14 IOPS][e Jobs: 2 (f=2): [_(1),r(2),_(1)][36.2%][r=1680KiB/s,w=0KiB/s][r=420,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][36.5%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][36.8%][r=1515KiB/s,w=0KiB/s][r=378,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][37.1%][r=1564KiB/s,w=0KiB/s][r=391,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][37.4%][r=1572KiB/s,w=0KiB/s][r=393,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][37.7%][r=1529KiB/s,w=0KiB/s][r=382,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][38.0%][r=1520KiB/s,w=0KiB/s][r=380,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][38.3%][r=1548KiB/s,w=0KiB/s][r=387,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][38.6%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][38.9%][r=1672KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][39.2%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][39.5%][r=1593KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][39.8%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][40.1%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][40.4%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][40.8%][r=2012KiB/s,w=0KiB/s][r=503,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][41.2%][r=2044KiB/s,w=0KiB/s][r=511,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][41.6%][r=1591KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][41.8%][r=1516KiB/s,w=0KiB/s][r=379,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][42.1%][r=1616KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][42.4%][r=1541KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][42.7%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][43.0%][r=1616KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][43.4%][r=1609KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][43.7%][r=1524KiB/s,w=0KiB/s][r=381,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][44.0%][r=1584KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][44.3%][r=1557KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][44.6%][r=1640KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][44.9%][r=1532KiB/s,w=0KiB/s][r=383,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][45.2%][r=1617KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][45.5%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][45.8%][r=1596KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][46.1%][r=1485KiB/s,w=0KiB/s][r=371,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][46.4%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][46.7%][r=1628KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][47.0%][r=1585KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][47.3%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][47.6%][r=1649KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][47.9%][r=1548KiB/s,w=0KiB/s][r=387,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][48.2%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][48.5%][r=1565KiB/s,w=0KiB/s][r=391,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][48.8%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][49.2%][r=1688KiB/s,w=0KiB/s][r=422,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][49.5%][r=1603KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][49.8%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][50.1%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][50.4%][r=1657KiB/s,w=0KiB/s][r=414,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][50.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][51.1%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][51.4%][r=1673KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][51.7%][r=1617KiB/s,w=0KiB/s][r=404,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][52.0%][r=1644KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][52.4%][r=1653KiB/s,w=0KiB/s][r=413,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][52.7%][r=1408KiB/s,w=0KiB/s][r=352,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][52.9%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][53.2%][r=1629KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][53.5%][r=1541KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][53.9%][r=1648KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][54.2%][r=1673KiB/s,w=0KiB/s][r=418,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][54.4%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][54.7%][r=1604KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][55.0%][r=1613KiB/s,w=0KiB/s][r=403,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][55.4%][r=1576KiB/s,w=0KiB/s][r=394,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][55.7%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][56.0%][r=1553KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][56.3%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][56.6%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][56.9%][r=1637KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][57.2%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][57.5%][r=1596KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][57.8%][r=1561KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][58.1%][r=1532KiB/s,w=0KiB/s][r=383,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][58.4%][r=1637KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][58.7%][r=1524KiB/s,w=0KiB/s][r=381,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][59.0%][r=1600KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][59.3%][r=1645KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][59.6%][r=1552KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][59.9%][r=1576KiB/s,w=0KiB/s][r=394,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][60.2%][r=1497KiB/s,w=0KiB/s][r=374,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][60.5%][r=1276KiB/s,w=0KiB/s][r=319,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][60.8%][r=1628KiB/s,w=0KiB/s][r=407,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][61.1%][r=1685KiB/s,w=0KiB/s][r=421,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][61.4%][r=1600KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][61.7%][r=1660KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][62.1%][r=1597KiB/s,w=0KiB/s][r=399,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][62.4%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][62.6%][r=1640KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][62.9%][r=1605KiB/s,w=0KiB/s][r=401,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][63.2%][r=1660KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][63.5%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][63.9%][r=1556KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][64.2%][r=1669KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][64.5%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][64.8%][r=1584KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][65.1%][r=1529KiB/s,w=0KiB/s][r=382,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][65.3%][r=1540KiB/s,w=0KiB/s][r=385,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][65.7%][r=1625KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][66.0%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][66.3%][r=1560KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][66.7%][r=1553KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][66.9%][r=1648KiB/s,w=0KiB/s][r=412,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][67.2%][r=1536KiB/s,w=0KiB/s][r=384,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][67.5%][r=1661KiB/s,w=0KiB/s][r=415,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][67.9%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][68.1%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][68.4%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][68.8%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][69.1%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][69.3%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][69.6%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][70.0%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][70.3%][r=1620KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][70.5%][r=1621KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][70.9%][r=1480KiB/s,w=0KiB/s][r=370,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][71.2%][r=1716KiB/s,w=0KiB/s][r=429,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][71.4%][r=1601KiB/s,w=0KiB/s][r=400,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][71.8%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][72.1%][r=1592KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][72.3%][r=1585KiB/s,w=0KiB/s][r=396,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][72.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][72.9%][r=1552KiB/s,w=0KiB/s][r=388,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][73.3%][r=1521KiB/s,w=0KiB/s][r=380,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][73.6%][r=1593KiB/s,w=0KiB/s][r=398,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][73.8%][r=1557KiB/s,w=0KiB/s][r=389,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][74.2%][r=1568KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][74.4%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][74.7%][r=1569KiB/s,w=0KiB/s][r=392,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][75.1%][r=1632KiB/s,w=0KiB/s][r=408,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][75.3%][r=1560KiB/s,w=0KiB/s][r=390,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][75.7%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][75.9%][r=1573KiB/s,w=0KiB/s][r=393,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][76.2%][r=1469KiB/s,w=0KiB/s][r=367,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][76.5%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][76.8%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][77.2%][r=1641KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][77.4%][r=1668KiB/s,w=0KiB/s][r=417,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][77.8%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][78.0%][r=1641KiB/s,w=0KiB/s][r=410,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][78.4%][r=1684KiB/s,w=0KiB/s][r=421,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][78.7%][r=1608KiB/s,w=0KiB/s][r=402,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][78.9%][r=1645KiB/s,w=0KiB/s][r=411,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][79.3%][r=1664KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][79.5%][r=1696KiB/s,w=0KiB/s][r=424,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][79.9%][r=1665KiB/s,w=0KiB/s][r=416,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][80.1%][r=1588KiB/s,w=0KiB/s][r=397,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][80.5%][r=1700KiB/s,w=0KiB/s][r=425,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][80.9%][r=1713KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][81.1%][r=1728KiB/s,w=0KiB/s][r=432,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][81.5%][r=1636KiB/s,w=0KiB/s][r=409,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][81.7%][r=1621KiB/s,w=0KiB/s][r=405,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][82.1%][r=1712KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][82.5%][r=1728KiB/s,w=0KiB/s][r=432,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][82.9%][r=1681KiB/s,w=0KiB/s][r=420,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][83.1%][r=1624KiB/s,w=0KiB/s][r=406,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][83.5%][r=1712KiB/s,w=0KiB/s][r=428,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][83.7%][r=1724KiB/s,w=0KiB/s][r=431,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][84.1%][r=1653KiB/s,w=0KiB/s][r=413,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][84.3%][r=1800KiB/s,w=0KiB/s][r=450,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][84.7%][r=1736KiB/s,w=0KiB/s][r=434,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][84.9%][r=1761KiB/s,w=0KiB/s][r=440,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][85.3%][r=1832KiB/s,w=0KiB/s][r=458,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][85.7%][r=1788KiB/s,w=0KiB/s][r=447,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][85.9%][r=1889KiB/s,w=0KiB/s][r=472,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][86.3%][r=1824KiB/s,w=0KiB/s][r=456,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][86.6%][r=1820KiB/s,w=0KiB/s][r=455,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][87.0%][r=1837KiB/s,w=0KiB/s][r=459,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][87.4%][r=1912KiB/s,w=0KiB/s][r=478,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][87.6%][r=1876KiB/s,w=0KiB/s][r=469,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][88.0%][r=1869KiB/s,w=0KiB/s][r=467,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][88.4%][r=2012KiB/s,w=0KiB/s][r=503,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][88.6%][r=1900KiB/s,w=0KiB/s][r=475,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][89.0%][r=2010KiB/s,w=0KiB/s][r=502,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][89.5%][r=2036KiB/s,w=0KiB/s][r=509,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][89.7%][r=2032KiB/s,w=0KiB/s][r=508,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][90.1%][r=2058KiB/s,w=0KiB/s][r=514,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][90.5%][r=2180KiB/s,w=0KiB/s][r=545,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][90.7%][r=2124KiB/s,w=0KiB/s][r=531,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][91.2%][r=2154KiB/s,w=0KiB/s][r=538,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][91.6%][r=2576KiB/s,w=0KiB/s][r=644,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][92.0%][r=2216KiB/s,w=0KiB/s][r=554,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][92.4%][r=2410KiB/s,w=0KiB/s][r=602,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][92.9%][r=2488KiB/s,w=0KiB/s][r=622,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][93.1%][r=2564KiB/s,w=0KiB/s][r=641,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][93.5%][r=2774KiB/s,w=0KiB/s][r=693,w=0 IOPS][eta Jobs: 2 (f=2): [_(1),r(2),_(1)][93.9%][r=2942KiB/s,w=0KiB/s][r=735,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][94.6%][r=2208KiB/s,w=0KiB/s][r=552,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][95.2%][r=2006KiB/s,w=0KiB/s][r=501,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][96.1%][r=2244KiB/s,w=0KiB/s][r=561,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][97.2%][r=2568KiB/s,w=0KiB/s][r=642,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][98.1%][r=2518KiB/s,w=0KiB/s][r=629,w=0 IOPS][eta Jobs: 1 (f=1): [_(2),r(1),_(1)][99.3%][r=3196KiB/s,w=0KiB/s][r=799,w=0 IOPS][eta 00m:03s] bgwriter: (groupid=0, jobs=1): err= 0: pid=7629: Wed Jan 2 11:13:37 2019 write: IOPS=312, BW=1250KiB/s (1280kB/s)(256MiB/209721msec) slat (usec): min=11, max=794, avg=39.49, stdev=13.49 clat (msec): min=2, max=12285, avg=102.36, stdev=289.05 lat (msec): min=2, max=12285, avg=102.40, stdev=289.05 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 11], 10.00th=[ 16], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 47], 50.00th=[ 61], 60.00th=[ 77], | 70.00th=[ 99], 80.00th=[ 129], 90.00th=[ 186], 95.00th=[ 245], | 99.00th=[ 443], 99.50th=[ 1720], 99.90th=[ 4212], 99.95th=[ 5604], | 99.99th=[ 7416] bw ( KiB/s): min= 24, max= 1640, per=100.00%, avg=1249.65, stdev=521.69, s amples=419 iops : min= 6, max= 410, avg=312.34, stdev=130.40, samples=419 lat (msec) : 4=0.25%, 10=4.11%, 20=9.73%, 50=28.36%, 100=28.42% lat (msec) : 250=24.37%, 500=3.96%, 750=0.21%, 1000=0.03%, 2000=0.14% lat (msec) : >=2000=0.42% cpu : usr=0.32%, sys=1.33%, ctx=64621, majf=0, minf=25 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwt: total=0,65536,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 queryA: (groupid=0, jobs=1): err= 0: pid=7630: Wed Jan 2 11:13:37 2019 read: IOPS=156, BW=626KiB/s (641kB/s)(256MiB/419038msec) clat (usec): min=113, max=297077, avg=6386.23, stdev=7067.73 lat (usec): min=116, max=297078, avg=6386.64, stdev=7067.73 clat percentiles (usec): | 1.00th=[ 196], 5.00th=[ 1074], 10.00th=[ 1598], 20.00th=[ 2442], | 30.00th=[ 3130], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6783], 80.00th=[ 8586], 90.00th=[ 12387], 95.00th=[ 17695], | 99.00th=[ 36439], 99.50th=[ 46400], 99.90th=[ 67634], 99.95th=[ 77071], | 99.99th=[104334] bw ( KiB/s): min= 128, max= 1973, per=46.26%, avg=587.91, stdev=256.81, sam ples=838 iops : min= 32, max= 493, avg=146.79, stdev=64.08, samples=838 lat (usec) : 250=3.19%, 500=1.14%, 750=0.26%, 1000=0.24% lat (msec) : 2=9.72%, 4=29.27%, 10=42.03%, 20=10.04%, 50=3.72% lat (msec) : 100=0.39%, 250=0.01%, 500=0.01% cpu : usr=0.23%, sys=0.88%, ctx=65537, majf=65536, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=65536,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 queryB: (groupid=0, jobs=1): err= 0: pid=7631: Wed Jan 2 11:13:37 2019 read: IOPS=154, BW=616KiB/s (631kB/s)(256MiB/425296msec) clat (usec): min=95, max=294727, avg=6479.55, stdev=7250.47 lat (usec): min=96, max=294727, avg=6479.96, stdev=7250.47 clat percentiles (usec): | 1.00th=[ 206], 5.00th=[ 930], 10.00th=[ 1614], 20.00th=[ 2474], | 30.00th=[ 3130], 40.00th=[ 3785], 50.00th=[ 4490], 60.00th=[ 5342], | 70.00th=[ 6849], 80.00th=[ 8717], 90.00th=[ 12518], 95.00th=[ 18220], | 99.00th=[ 36963], 99.50th=[ 45876], 99.90th=[ 68682], 99.95th=[ 83362], | 99.99th=[130548] bw ( KiB/s): min= 112, max= 3706, per=45.37%, avg=576.65, stdev=301.11, sam ples=850 iops : min= 28, max= 926, avg=143.98, stdev=75.16, samples=850 lat (usec) : 100=0.01%, 250=3.73%, 500=0.98%, 750=0.15%, 1000=0.20% lat (msec) : 2=9.20%, 4=28.93%, 10=42.19%, 20=10.34%, 50=3.89% lat (msec) : 100=0.36%, 250=0.02%, 500=0.01% cpu : usr=0.26%, sys=0.87%, ctx=65541, majf=65536, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=65536,0,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 bgupdater: (groupid=0, jobs=1): err= 0: pid=7632: Wed Jan 2 11:13:37 2019 read: IOPS=17, BW=69.8KiB/s (71.5kB/s)(15.9MiB/232986msec) slat (usec): min=14, max=144, avg=43.42, stdev=14.67 clat (usec): min=92, max=181899, avg=8734.17, stdev=10918.07 lat (usec): min=175, max=181944, avg=8778.72, stdev=10917.02 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 306], 10.00th=[ 1778], 20.00th=[ 2769], | 30.00th=[ 3556], 40.00th=[ 4293], 50.00th=[ 5145], 60.00th=[ 6718], | 70.00th=[ 8979], 80.00th=[ 12518], 90.00th=[ 19268], 95.00th=[ 27132], | 99.00th=[ 51643], 99.50th=[ 63701], 99.90th=[111674], 99.95th=[147850], | 99.99th=[181404] bw ( KiB/s): min= 8, max= 425, per=5.94%, avg=75.50, stdev=82.12, sample s=428 iops : min= 2, max= 106, avg=18.86, stdev=20.52, samples=428 write: IOPS=17, BW=70.8KiB/s (72.5kB/s)(16.1MiB/232986msec) slat (usec): min=14, max=850, avg=47.04, stdev=19.35 clat (usec): min=1754, max=745718, avg=47818.37, stdev=79232.60 lat (usec): min=1800, max=745766, avg=47866.56, stdev=79229.96 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 10], 40.00th=[ 13], 50.00th=[ 17], 60.00th=[ 25], | 70.00th=[ 39], 80.00th=[ 65], 90.00th=[ 130], 95.00th=[ 207], | 99.00th=[ 414], 99.50th=[ 502], 99.90th=[ 617], 99.95th=[ 659], | 99.99th=[ 743] bw ( KiB/s): min= 8, max= 368, per=5.98%, avg=71.43, stdev=78.72, sample s=458 iops : min= 2, max= 92, avg=17.84, stdev=19.67, samples=458 lat (usec) : 100=0.01%, 250=2.23%, 500=0.73%, 750=0.09%, 1000=0.17% lat (msec) : 2=2.42%, 4=14.28%, 10=33.84%, 20=18.58%, 50=14.79% lat (msec) : 100=6.10%, 250=5.00%, 500=1.49%, 750=0.26% cpu : usr=0.20%, sys=0.19%, ctx=8195, majf=0, minf=27 IO depths : 1=99.8%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4067,4125,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1271KiB/s (1302kB/s), 69.8KiB/s-626KiB/s (71.5kB/s-641kB/s), io=528M iB (554MB), run=232986-425296msec WRITE: bw=1196KiB/s (1225kB/s), 70.8KiB/s-1250KiB/s (72.5kB/s-1280kB/s), io=27 2MiB (285MB), run=209721-232986msec Disk stats (read/write): dm-0: ios=135129/69955, merge=0/0, ticks=870964/6969822, in_queue=7840785, u til=99.89%, aggrios=135140/69947, aggrmerge=0/8, aggrticks=870647/6968590, aggri n_queue=7839132, aggrutil=99.91% sda: ios=135140/69947, merge=0/8, ticks=870647/6968590, in_queue=7839132, util =99.91%
