Hardware changes to my Ceph Nautilus Cluster

Table of Contents

I made some hardware changes to my Ceph Nautilus cluster.

picture of my 4 F5-422 cluster on a shelf

While previously I used one SSD shared between the operating system and Ceph, I did want a clean setup where the OS and Ceph are on separate devices.

Dedicated 120 GB SSD for OS

I bought myself four Crucial BX500 SSD 120GB for about 27€ a piece plus shipping.

Additionally, I recycled 4 HDDs from machines around the lab that I had not booted in a year or more.

Now the storage setup of each F5-422 is as follows;

bay type use
1 120 GB SATA SSD Operating System
2 500 GB SATA SSD 3x Ceph BlueStore DB
3 1 TB or larger HDD Ceph BlueStore data
4 1 TB or larger HDD Ceph BlueStore data
5 1 TB or larger HDD Ceph BlueStore data

The mix of HDD sizes I managed to scrounge in my home-lab plus one 155GiB SSD slice for each of the three OSDs per node show up as follows in the dashboard.

Screenshot of the OSD page in dashboard showing 12 OSDs total, 3 per node.

While this wild mix of HDD sizes is not ideal, recycling drives that I already owned is good enough for the home-lab and incurred no addition cost.

Re-install the Nodes

Originally I had the OS (including /boot and /boot/efi) on the 500GB SSD. While the PV used for the OS I could have moved at runtime, I simply did not want the hassle of moving the boot bits (cattle, not pets and all that jazz).

Since I have kickstart files in place for the F5-422 nodes, and I do all my configuration with Ansible, it was a lot less hassle for me to re-install each node.

By doing this one machine at at time, I obviously had no downtime on my Ceph.

Steps taken;

  1. remove OSDs of one node from Cluster, in my case that was 2 per node.
  2. power node down.
  3. rearrange storage devices.
  4. install the node using kickstart
  5. sgdisk --zap-all the previously used devies (bays 2 through 5) because I also wanted to switch away from me having top set up LVM for Ceph use and that (understandably) only grabs empty devices.
  6. run my configuration playbooks.
  7. use ceph-ansible.

repeat with the next node until all nodes done.

Yes, I did wait while the cluster was self healing the removal of OSDs and subsequent addition of more OSDs. But for me that was preferable to interactively spending time on each node to move the OS from one SSD to another.

Used host_vars

Since I already had host specific host_vars, I decided to also test how ceph-ansible deals with a mix of lvm_volumes: and devices:.

It deals absolutely fine with this.

example of lvm_volumes: (click to expand)
[...]
lvm_volumes:
  - data:         LV_HDD_bay2_OSD_data
    data_vg:      VG_HDD_bay2_Ceph
    db:           LV_SSD_bay1_OSD_db_01
    db_vg:        VG_SSD_bay1_Ceph
  - data:         LV_HDD_bay3_OSD_data
    data_vg:      VG_HDD_bay3_Ceph
    db:           LV_SSD_bay1_OSD_db_02
    db_vg:        VG_SSD_bay1_Ceph
[...]
example of devices: (again, click to expand)
[...]
# avoid using half arsed names like sda, sdb, etc
# mapping is:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1
devices:
  - /dev/disk/by-path/pci-0000:00:12.0-ata-2
  - /dev/disk/by-path/pci-0000:03:00.0-ata-1
  - /dev/disk/by-path/pci-0000:03:00.0-ata-2
  - /dev/disk/by-path/pci-0000:04:00.0-ata-1

See section OSD Scenario in the ceph-ansible documentation for more information.

This ends up set up by ceph-ansible, as expected, with the 500GB SSD being divided into 3 identical parts (click to expand an lsblk output on one of the nodes).
[root@f5-422-02 ~]# lsblk
NAME                                    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                       8:0    0 111.8G  0 disk  
├─sda1                                    8:1    0   512M  0 part  /boot/efi
├─sda2                                    8:2    0     1G  0 part  /boot
└─sda3                                    8:3    0 110.3G  0 part  
  ├─VG_OS_SSD_bay1-LV_root              253:0    0     4G  0 lvm   /
  ├─VG_OS_SSD_bay1-LV_swap              253:1    0     4G  0 lvm   [SWAP]
  ├─VG_OS_SSD_bay1-LV_home              253:2    0     1G  0 lvm   /home
  ├─VG_OS_SSD_bay1-LV_containers        253:3    0     8G  0 lvm   /var/lib/containers
  ├─VG_OS_SSD_bay1-LV_var_crash         253:4    0    15G  0 lvm   /var/crash
  ├─VG_OS_SSD_bay1-LV_var_log           253:5    0     4G  0 lvm   /var/log
  └─VG_OS_SSD_bay1-LV_var               253:6    0     4G  0 lvm   /var
sdb                                       8:16   0 465.8G  0 disk  
├─ceph--block--dbs--a3488d07--c12f[...] 253:8    0 155.3G  0 lvm   
│ └─gptTuV-yI5F-CT03-HHIo-GVpi-v5v[...] 253:10   0 155.2G  0 crypt 
├─ceph--block--dbs--a3488d07--c12f[...] 253:12   0 155.3G  0 lvm   
│ └─lydAIm-olLm-Ml22-p0as-XTe2-5Gp[...] 253:14   0 155.2G  0 crypt 
└─ceph--block--dbs--a3488d07--c12f[...] 253:16   0 155.3G  0 lvm   
  └─qYVMiU-9EHg-nRje-5XoO-rFi2-dtW[...] 253:18   0 155.2G  0 crypt 
sdd                                       8:48   0   1.8T  0 disk  
└─ceph--block--fce5b416--a642--48b[...] 253:7    0   1.8T  0 lvm   
  └─Nsx79i-aRR9-2GeJ-i6jX-6ekE-Far[...] 253:9    0   1.8T  0 crypt 
sde                                       8:64   0   1.8T  0 disk  
└─ceph--block--0da07e49--e9d2--49a[...] 253:11   0   1.8T  0 lvm   
  └─vDb5md-6n6O-4jMj-J5ns-dnFx-NRP[...] 253:13   0   1.8T  0 crypt 
sdf                                       8:80   0 931.5G  0 disk  
└─ceph--block--65a4c7bb--0247--471[...] 253:15   0 931.5G  0 lvm   
  └─NWElgJ-3rQh-iD4b-7x9e-Gmkr-ude[...] 253:17   0 931.5G  0 crypt 

First column pruned a bit to increase readability.

Used Ceph Installation Method

It goes without saying that this whole procedure was without any Ceph cluster downtime.

While the manual Ceph method would have been my fallback, I wanted to do this change with ceph-ansible.

I started with node number 4. That runs OSDs and grafana, but no MON nor MGR and tried infrastructure-playbooks/add-osd.yml with the needed inventory change (documented, amongst others, in the add-osd.yml itself), because I was unsure how ceph-ansible would deal with a mix of lvm_volumes: and devices:.

Everything went fine when I added 3 OSDs running on node 4 the add-osds.yml way.

After that I switched to using site-container.yml to give that a try with the mix.

That also went absolutely fine.

In the future, now that I know all is fine with a mix, I would only use site-container.yml Simply because that ensures that the co-located non-MON containers (e.g. a MON or a grafana) are put back in place ASAP.

Performance Baseline with rados bench, pool with 16 PGs

Like previously, I gathered a performance baseline based on the instructions at 6.2. Performance baseline of the RHCS 4 Administration Guide.

Since I now had a total of 12 OSDs and with my existing 10 pools, I was below 30 PGs per OSD, I picked 16 PGs for my testbench pool by resizing the existing one from 8 to 16 PGs and waiting for all PGs to be active+clean).

All rados bench tests were run on f5-422-01.

Sequential Write Test, pool with 16 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1023
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        42        26   103.977       104    0.398447    0.365326
    2      16        83        67   133.891       164    0.239482    0.414529
    3      16       123       107   142.583       160    0.542042    0.413781
    4      16       151       135   134.934       112    0.136264     0.41478
    5      16       195       179    143.14       176     0.31506    0.425895
    6      16       242       226   150.611       188    0.136594    0.408965
    7      16       284       268   153.088       168    0.386108    0.405365
    8      16       314       298   148.951       120   0.0920322    0.409887
    9      16       358       342   151.953       176    0.413527    0.412807
   10      16       398       382   152.754       160    0.320931    0.409935
   11      16       439       423   153.772       164    0.225462    0.408766
   12      16       470       454    151.29       124    0.182687     0.41236
   13      16       502       486   149.497       128    0.270364    0.418863
   14      16       531       515   147.103       116    0.120548    0.422944
   15      16       577       561   149.559       184    0.261647    0.422054
   16      16       618       602    150.46       164    0.237889    0.417494
   17      16       653       637   149.844       140    0.111806    0.419136
   18      16       689       673   149.518       144    0.538778    0.419554
   19      16       726       710   149.437       148    0.237639    0.423924
2020-06-08 11:59:12.552308 min lat: 0.0794906 max lat: 1.44304 avg lat: 0.42214
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       762       746   149.164       144     0.17261     0.42214
   21      16       797       781   148.726       140     1.04316     0.42337
   22      16       831       815   148.147       136    0.774272    0.426569
   23      16       863       847    147.27       128    0.128176    0.425961
   24      16       893       877   146.133       120     0.34782     0.43013
   25      16       921       905   144.766       112    0.805176    0.431506
   26      16       958       942   144.889       148     0.32231    0.435559
   27      16       992       976   144.559       136   0.0900235    0.437707
   28      16      1035      1019   145.538       172    0.402015    0.436377
   29      16      1075      1059   146.035       160    0.362278    0.435442
   30      16      1113      1097   146.229       152    0.708746    0.432798
   31      16      1150      1134   146.285       148    0.537088    0.433644
   32      16      1188      1172   146.462       152    0.111633    0.432276
   33      16      1209      1193   144.569        84    0.187649    0.433327
   34      16      1243      1227   144.301       136    0.516067     0.43945
   35      16      1269      1253   143.149       104    0.218054    0.439513
   36      16      1307      1291   143.394       152    0.582905    0.442032
   37      16      1348      1332   143.949       164    0.301545    0.440825
   38      16      1374      1358   142.897       104     1.56705    0.442952
   39      16      1408      1392    142.72       136    0.154026    0.444163
2020-06-08 11:59:32.561544 min lat: 0.0794906 max lat: 2.23993 avg lat: 0.444688
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1441      1425    142.45       132     0.41124    0.444688
   41      16      1475      1459   142.292       136    0.154866    0.445385
   42      16      1514      1498   142.618       156     0.26497    0.446216
   43      16      1551      1535   142.742       148    0.246734    0.444359
   44      16      1587      1571   142.762       144   0.0868129    0.444821
   45      16      1622      1606     142.7       140    0.414572    0.445094
   46      16      1654      1638    142.38       128    0.506686    0.446723
   47      16      1693      1677   142.667       156    0.447949    0.445872
   48      16      1728      1712   142.607       140    0.274668    0.445592
   49      16      1772      1756   143.288       176    0.137756    0.444681
   50      16      1811      1795   143.541       156     0.76994    0.444292
   51      16      1844      1828   143.314       132    0.384643    0.444032
   52      16      1888      1872   143.941       176    0.388899    0.442767
   53      16      1923      1907   143.866       140    0.258826    0.441891
   54      16      1965      1949   144.299       168    0.466725    0.441493
   55      16      2002      1986   144.366       148    0.286507    0.441327
   56      16      2036      2020   144.216       136    0.406687    0.440509
   57      16      2066      2050   143.791       120    0.478132    0.441727
   58      16      2109      2093   144.277       172    0.286874    0.441564
   59      16      2152      2136   144.745       172      0.4606    0.440057
2020-06-08 11:59:52.575672 min lat: 0.0794906 max lat: 2.23993 avg lat: 0.440594
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2180      2164   144.199       112     0.61586    0.440594
   61      16      2230      2214   145.113       200    0.339176      0.4399
   62      16      2273      2257   145.546       172      0.1729    0.437923
   63      16      2311      2295   145.648       152    0.111251    0.437601
   64      16      2350      2334   145.809       156    0.115738    0.435915
   65      16      2383      2367   145.596       132    0.129876    0.435851
   66      16      2417      2401   145.451       136       0.385    0.435713
   67      16      2439      2423   144.593        88   0.0921996    0.437128
   68      16      2468      2452   144.166       116     2.40533    0.439414
   69      16      2495      2479   143.642       108    0.133776    0.440936
   70      16      2521      2505   143.075       104    0.162519    0.441913
   71      16      2550      2534   142.694       116    0.193849    0.442939
   72      16      2578      2562   142.267       112    0.120453    0.442734
   73      16      2620      2604   142.619       168    0.180627    0.446909
   74      16      2648      2632   142.205       112    0.104623    0.446433
   75      16      2688      2672   142.441       160    0.263871    0.446968
   76      16      2724      2708   142.462       144    0.282413    0.446169
   77      16      2760      2744   142.481       144    0.259337    0.446333
   78      16      2802      2786   142.806       168    0.213389    0.446877
   79      16      2847      2831   143.277       180    0.316799      0.4456
2020-06-08 12:00:12.583506 min lat: 0.0765629 max lat: 2.92381 avg lat: 0.444675
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2886      2870   143.435       156    0.120977    0.444675
   81      16      2909      2893     142.8        92    0.156527    0.445901
   82      16      2942      2926   142.668       132    0.581131    0.446634
   83      16      2981      2965   142.829       156    0.550399    0.446677
   84      16      3028      3012   143.366       188    0.166803    0.445309
   85      16      3061      3045   143.232       132    0.129409    0.444698
   86      16      3107      3091   143.706       184    0.260706     0.44461
   87      16      3144      3128   143.755       148   0.0969723    0.443777
   88      16      3179      3163   143.709       140    0.399811    0.444042
   89      16      3218      3202   143.846       156    0.368254    0.443821
   90      16      3251      3235   143.715       132    0.679004    0.443631
   91      16      3292      3276   143.937       164    0.498053    0.443424
   92      16      3330      3314   144.023       152     0.62485    0.443343
   93      16      3369      3353   144.151       156    0.104437    0.442384
   94      16      3400      3384   143.935       124   0.0818116    0.442889
   95      16      3430      3414   143.683       120   0.0899581    0.443341
   96      16      3464      3448   143.598       136    0.780032    0.444225
   97      16      3501      3485   143.644       148    0.256891     0.44381
   98      16      3536      3520   143.606       140    0.198576    0.444417
   99      16      3571      3555   143.569       140    0.236379    0.444209
2020-06-08 12:00:32.593968 min lat: 0.0753806 max lat: 2.92381 avg lat: 0.44355
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3615      3599   143.893       176    0.696873     0.44355
  101      16      3642      3626   143.538       108    0.626366    0.443691
  102      16      3683      3667   143.738       164    0.190347     0.44365
  103      16      3717      3701   143.662       136    0.765361    0.443916
  104      16      3759      3743   143.896       168    0.276769    0.443753
  105      16      3791      3775   143.744       128    0.118761    0.443968
  106      16      3830      3814   143.859       156    0.139814    0.443565
  107      16      3871      3855   144.042       164    0.371427    0.443257
  108      16      3910      3894   144.152       156    0.143957    0.442597
  109      16      3951      3935   144.333       164    0.205618    0.442368
  110      16      3988      3972   144.366       148     1.04951    0.442508
  111      16      4034      4018   144.723       184   0.0987925    0.441425
  112      16      4066      4050   144.566       128    0.533442    0.441813
  113      16      4108      4092   144.773       168    0.279903    0.440983
  114      16      4141      4125    144.66       132    0.657968    0.441282
  115      16      4176      4160    144.62       140    0.268767    0.441315
  116      16      4208      4192   144.476       128    0.465917    0.442303
  117      15      4244      4229   144.506       148    0.318879     0.44183
  118      16      4275      4259   144.298       120    0.408339    0.442327
  119      16      4314      4298   144.396       156    0.308683    0.441775
2020-06-08 12:00:52.611705 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.442011
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4353      4337   144.489       156     1.04169    0.442011
  121      16      4381      4365   144.221       112    0.899546    0.442104
  122      16      4405      4389   143.823        96      1.1861    0.442601
  123      16      4439      4423   143.759       136    0.171213    0.443698
  124      16      4468      4452   143.535       116    0.123565    0.444382
  125      16      4503      4487   143.504       140    0.151779    0.445298
  126      16      4540      4524   143.538       148    0.826155    0.444937
  127      16      4577      4561   143.573       148    0.296185    0.444777
  128      16      4613      4597   143.576       144    0.311355    0.444352
  129      16      4640      4624     143.3       108    0.843498    0.444817
  130      16      4674      4658   143.244       136    0.122587    0.445197
  131      16      4709      4693   143.216       140    0.195114    0.445781
  132      16      4751      4735   143.404       168    0.197907    0.445371
  133      16      4784      4768   143.318       132    0.256699    0.445172
  134      16      4818      4802   143.263       136    0.171877    0.445363
  135      16      4860      4844   143.446       168    0.198978    0.445519
  136      16      4891      4875   143.303       124    0.305737    0.445381
  137      16      4929      4913   143.366       152    0.427361    0.445782
  138      16      4964      4948   143.341       140    0.211179    0.445654
  139      16      4996      4980    143.23       128    0.241285    0.445271
2020-06-08 12:01:12.624783 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.445793
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5026      5010   143.064       120     1.03791    0.445793
  141      16      5056      5040     142.9       120    0.137966    0.446919
  142      16      5080      5064    142.57        96     1.53823    0.446994
  143      16      5119      5103   142.664       156     0.51641    0.447947
  144      16      5163      5147   142.895       176    0.268755    0.447237
  145      16      5205      5189   143.067       168    0.124783    0.446577
  146      16      5246      5230    143.21       164    0.317081    0.446294
  147      16      5285      5269   143.297       156    0.290308    0.446103
  148      16      5332      5316   143.599       188    0.366505    0.445152
  149      16      5372      5356   143.709       160    0.214472     0.44444
  150      16      5419      5403   144.003       188    0.137074    0.443797
  151      16      5459      5443   144.109       160    0.490304    0.443261
  152      16      5504      5488   144.344       180    0.434736    0.442887
  153      16      5546      5530   144.498       168    0.278205    0.442198
  154      16      5588      5572    144.65       168    0.175283    0.441773
  155      16      5631      5615   144.826       172    0.242373    0.441296
  156      16      5673      5657   144.975       168    0.303418    0.440525
  157      16      5717      5701   145.172       176    0.267878    0.440493
  158      16      5759      5743   145.316       168    0.218735    0.439766
  159      16      5800      5784   145.434       164    0.318679    0.439357
2020-06-08 12:01:32.630641 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.438659
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5847      5831   145.699       188    0.381478    0.438659
  161      16      5892      5876   145.912       180    0.129163    0.437905
  162      16      5928      5912     145.9       144    0.412485    0.437835
  163      16      5966      5950   145.937       152    0.400137    0.437772
  164      16      6006      5990   146.023       160    0.216359    0.437808
  165      16      6044      6028   146.059       152    0.175006    0.437535
  166      16      6085      6069   146.165       164    0.266665    0.437137
  167      16      6124      6108   146.224       156    0.143355    0.436973
  168      16      6162      6146   146.258       152    0.264271    0.436783
  169      16      6202      6186   146.339       160    0.817793    0.436957
  170      16      6235      6219   146.253       132    0.132424    0.436679
  171      16      6287      6271   146.614       208     0.34244    0.436127
  172      16      6329      6313   146.738       168    0.392364    0.435427
  173      16      6379      6363   147.045       200    0.212298    0.434666
  174      16      6422      6406   147.188       172      0.3763    0.433958
  175      16      6451      6435    147.01       116    0.272379    0.434327
  176      16      6491      6475   147.084       160    0.241195    0.434484
  177      16      6532      6516   147.179       164   0.0907898    0.433779
  178      16      6567      6551   147.138       140    0.114962    0.433723
  179      16      6609      6593   147.255       168    0.187057    0.434074
2020-06-08 12:01:52.638675 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.433549
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      6656      6640   147.481       188    0.292246    0.433549
  181      16      6703      6687   147.704       188    0.302311    0.432886
  182      16      6749      6733   147.904       184     0.37474    0.432322
  183      16      6792      6776   148.035       172    0.361231     0.43173
  184      16      6838      6822    148.23       184    0.133267    0.431371
  185      16      6885      6869   148.445       188     0.29473    0.430569
  186      16      6931      6915   148.636       184    0.261117    0.429924
  187      16      6965      6949   148.568       136     1.01155    0.429947
  188      16      6991      6975   148.331       104    0.410788     0.43045
  189      16      7028      7012   148.329       148    0.144746    0.430771
  190      16      7068      7052   148.391       160    0.129355     0.43056
  191      15      7106      7091    148.43       156    0.516433    0.430598
  192      16      7142      7126   148.386       140    0.228968    0.430469
  193      16      7182      7166   148.446       160    0.191215    0.430637
  194      16      7226      7210   148.588       176    0.265485    0.430225
  195      16      7269      7253   148.706       172    0.238815    0.430121
  196      16      7319      7303   148.967       200    0.299365     0.42922
  197      16      7365      7349   149.145       184    0.135801    0.428615
  198      16      7410      7394     149.3       180    0.279442    0.428367
  199      16      7455      7439   149.454       180    0.485155    0.427842
2020-06-08 12:02:12.645137 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.427369
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      7495      7479   149.507       160    0.124383    0.427369
  201      16      7543      7527   149.718       192    0.307045    0.427205
  202      16      7588      7572   149.868       180    0.157108    0.426854
  203      16      7624      7608   149.839       144    0.215977    0.426455
  204      16      7666      7650   149.927       168    0.103162    0.426126
  205      16      7709      7693   150.035       172    0.161674    0.425948
  206      16      7759      7743   150.277       200    0.610101    0.425483
  207      16      7800      7784   150.342       164    0.401227     0.42529
  208      16      7843      7827   150.446       172    0.463575    0.425082
  209      16      7894      7878     150.7       204    0.302787    0.424372
  210      16      7943      7927   150.915       196    0.309337    0.423608
  211      16      7996      7980   151.203       212    0.249553    0.422875
  212      16      8038      8022   151.282       168    0.359746    0.422442
  213      16      8084      8068   151.436       184    0.429353    0.422246
  214      16      8129      8113   151.566       180    0.276603    0.421692
  215      16      8176      8160   151.735       188    0.458162    0.421328
  216      16      8218      8202   151.811       168    0.241697     0.42092
  217      16      8265      8249   151.977       188     0.33623    0.420796
  218      16      8303      8287   151.977       152    0.404618    0.420554
  219      16      8343      8327   152.013       160    0.616621    0.420138
2020-06-08 12:02:32.662702 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.420531
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      8385      8369   152.084       168    0.311759    0.420531
  221      16      8432      8416   152.246       188     0.30745    0.420057
  222      16      8482      8466    152.46       200   0.0896688    0.419295
  223      16      8523      8507   152.512       164    0.527969    0.419227
  224      16      8573      8557   152.724       200    0.243354    0.418652
  225      16      8625      8609   152.969       208     0.13521     0.41811
  226      16      8667      8651   153.036       168    0.798107     0.41786
  227      16      8697      8681    152.89       120    0.121405     0.41768
  228      16      8734      8718   152.868       148     1.35738    0.418116
  229      16      8784      8768   153.074       200    0.459701    0.417857
  230      16      8823      8807   153.086       156     0.56197    0.417524
  231      16      8871      8855   153.255       192    0.433494    0.417145
  232      16      8921      8905   153.456       200    0.607592    0.416747
  233      16      8965      8949   153.553       176    0.111828    0.416236
  234      16      9000      8984   153.495       140    0.804002    0.416245
  235      16      9035      9019   153.437       140    0.966972    0.416357
  236      16      9071      9055   153.396       144    0.253087     0.41683
  237      16      9120      9104   153.576       196    0.157833    0.416397
  238      16      9152      9136   153.468       128    0.129639    0.416469
  239      16      9198      9182   153.596       184    0.165864    0.416347
2020-06-08 12:02:52.669071 min lat: 0.0722708 max lat: 2.92381 avg lat: 0.416047
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      9243      9227   153.705       180    0.103726    0.416047
  241      16      9294      9278   153.914       204     0.11835    0.415359
  242      16      9340      9324   154.038       184    0.363781    0.415079
  243      16      9376      9360   153.997       144    0.141759    0.414889
  244      16      9410      9394   153.923       136    0.264552    0.415212
  245      16      9438      9422    153.75       112     1.19538     0.41518
  246      16      9469      9453   153.629       124    0.188027    0.415619
  247      16      9490      9474   153.347        84    0.093494       0.416
  248      16      9518      9502    153.18       112   0.0887564    0.416562
  249      16      9544      9528   152.983       104    0.195166    0.417013
  250      16      9568      9552   152.754        96    0.181726     0.41786
  251      16      9582      9566   152.369        56    0.137892    0.418165
  252      16      9608      9592   152.177       104    0.195848    0.419046
  253      16      9633      9617   151.971       100    0.145371    0.420078
  254      16      9661      9645   151.813       112    0.293228    0.420621
  255      16      9695      9679   151.751       136    0.159858     0.42052
  256      16      9723      9707   151.595       112    0.166974     0.42118
  257      16      9745      9729   151.348        88   0.0940269    0.421655
  258      16      9763      9747    151.04        72    0.118476     0.42253
  259      16      9784      9768   150.781        84    0.134013     0.42298
2020-06-08 12:03:12.678681 min lat: 0.0722708 max lat: 3.23771 avg lat: 0.423818
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      9799      9783   150.432        60    0.123094    0.423818
  261      16      9826      9810   150.269       108    0.695075    0.425154
  262      16      9866      9850   150.306       160    0.348291    0.425046
  263      16      9882      9866   149.977        64     1.42374    0.425621
  264      16      9911      9895   149.848       116      2.4134    0.425976
  265      16      9929      9913   149.555        72   0.0979419    0.426662
  266      16      9962      9946   149.488       132    0.209607    0.427224
  267      16      9991      9975   149.362       116    0.225419    0.427444
  268      16     10018     10002   149.208       108     1.18886    0.427815
  269      16     10053     10037   149.174       140    0.216152     0.42824
  270      16     10068     10052   148.843        60    0.210188    0.428728
  271      16     10101     10085   148.781       132     0.19895    0.428999
  272      16     10114     10098   148.425        52    0.122905    0.429609
  273      16     10132     10116   148.145        72   0.0935088     0.43025
  274      16     10160     10144   148.013       112    0.147505    0.430672
  275      16     10176     10160   147.708        64     1.49305    0.431699
  276      16     10194     10178   147.434        72    0.724972    0.432737
  277      16     10212     10196   147.161        72     2.36926    0.433687
  278      16     10233     10217   146.934        84    0.143773    0.434311
  279      16     10253     10237   146.694        80     1.52916    0.435166
2020-06-08 12:03:32.687564 min lat: 0.0722708 max lat: 3.23771 avg lat: 0.435724
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     10271     10255   146.427        72    0.161734    0.435724
  281      16     10290     10274   146.176        76    0.125682    0.436299
  282      16     10312     10296    145.97        88    0.129635    0.436866
  283      16     10342     10326   145.878       120     0.15821    0.437504
  284      16     10377     10361   145.857       140   0.0928508    0.437496
  285      16     10406     10390   145.752       116    0.518276    0.438129
  286      16     10422     10406   145.467        64     1.07445     0.43872
  287      16     10440     10424   145.211        72    0.776983    0.439414
  288      16     10469     10453   145.109       116    0.111772    0.439781
  289      16     10486     10470   144.842        68     1.64157    0.440535
  290      16     10502     10486   144.563        64    0.136105    0.440971
  291      16     10540     10524   144.588       152    0.246552    0.442327
  292      16     10576     10560   144.586       144    0.312766     0.44208
  293      16     10610     10594   144.557       136    0.149812    0.441935
  294      16     10627     10611   144.296        68     1.81328    0.442429
  295      16     10645     10629   144.051        72     2.88195    0.443139
  296      16     10672     10656   143.929       108     0.09646    0.443584
  297      16     10698     10682   143.795       104    0.169175    0.443832
  298      16     10723     10707   143.648       100   0.0914411    0.444542
  299      16     10744     10728   143.448        84    0.187771    0.444777
2020-06-08 12:03:52.693662 min lat: 0.0722708 max lat: 3.378 avg lat: 0.445679
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     10775     10759   143.383       124    0.133518    0.445679
  301      16     10808     10792   143.346       132    0.185371    0.445421
  302      16     10831     10815   143.175        92     1.07333    0.446164
  303      16     10853     10837   142.993        88     1.42305    0.446246
  304      16     10878     10862   142.852       100    0.886884    0.446938
  305      16     10920     10904   142.934       168    0.101993    0.447436
  306      16     10956     10940   142.937       144    0.122725     0.44727
  307      16     10989     10973   142.901       132    0.169972    0.447055
  308      16     11013     10997   142.748        96    0.139724    0.447154
  309      16     11037     11021   142.597        96     0.12336    0.447716
  310      16     11060     11044   142.434        92    0.289583    0.448132
  311      16     11091     11075   142.375       124   0.0838391    0.448213
  312      16     11104     11088   142.085        52    0.152349    0.448905
  313      16     11128     11112   141.938        96    0.168615    0.449489
  314      16     11151     11135   141.779        92    0.181976    0.450049
  315      16     11170     11154    141.57        76   0.0941329    0.450461
  316      16     11202     11186   141.526       128    0.117752    0.451235
  317      16     11230     11214   141.433       112    0.178836    0.451296
  318      16     11241     11225   141.126        44     1.07644    0.451933
  319      16     11261     11245   140.935        80    0.124781    0.452948
2020-06-08 12:04:12.702569 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.453411
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     11294     11278   140.907       132   0.0905571    0.453411
  321      16     11318     11302   140.767        96     1.35392    0.453578
  322      16     11347     11331    140.69       116   0.0921295    0.453859
  323      16     11372     11356   140.564       100     1.87001    0.454322
  324      16     11386     11370   140.303        56    0.166954    0.454588
  325      16     11421     11405   140.302       140    0.102514    0.455089
  326      16     11446     11430   140.178       100     0.14884    0.455715
  327      16     11458     11442   139.896        48     1.09856    0.456165
  328      16     11485     11469   139.799       108    0.163835    0.456908
  329      16     11509     11493   139.666        96    0.112958    0.457008
  330      16     11523     11507   139.412        56    0.109407    0.457764
  331      16     11553     11537   139.353       120    0.148788    0.458412
  332      16     11572     11556   139.163        76   0.0888086    0.458897
  333      16     11590     11574   138.961        72     1.99026    0.459604
  334      16     11611     11595   138.796        84    0.159798     0.46014
  335      16     11622     11606   138.513        44    0.126609    0.460395
  336      16     11652     11636   138.458       120     1.67195    0.461612
  337      16     11679     11663   138.368       108     0.18286    0.461683
  338      16     11710     11694   138.325       124   0.0989499     0.46173
  339      16     11740     11724   138.271       120    0.128565    0.462053
2020-06-08 12:04:32.707720 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.462351
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     11760     11744     138.1        80     2.54891    0.462351
  341      16     11808     11792   138.258       192    0.334997    0.462584
  342      16     11865     11849    138.52       228    0.376299    0.461876
  343      16     11902     11886   138.547       148    0.686378    0.461656
  344      16     11948     11932   138.679       184    0.296212    0.461112
  345      16     11996     11980   138.834       192    0.264821     0.46061
  346      16     12046     12030    139.01       200    0.370486    0.460037
  347      16     12095     12079   139.174       196    0.209926    0.459683
  348      16     12146     12130   139.361       204    0.254113    0.459082
  349      16     12201     12185   139.591       220    0.355664    0.458304
  350      16     12250     12234   139.752       196    0.220042    0.457672
  351      16     12300     12284   139.924       200    0.432428    0.457187
  352      16     12352     12336   140.117       208    0.258861     0.45658
  353      16     12398     12382   140.241       184    0.275295    0.456105
  354      16     12445     12429   140.376       188    0.233265    0.455581
  355      16     12496     12480   140.554       204    0.160331    0.455161
  356      16     12537     12521   140.619       164    0.336082    0.454759
  357      16     12581     12565   140.718       176    0.310762    0.454478
  358      16     12637     12621   140.951       224    0.274925    0.453871
  359      16     12685     12669   141.093       192    0.232541    0.453355
2020-06-08 12:04:52.715753 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.452734
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     12739     12723   141.301       216    0.225254    0.452734
  361      16     12784     12768   141.407       180    0.148216    0.452396
  362      16     12825     12809    141.47       164    0.258651    0.452135
  363      16     12867     12851   141.543       168    0.269325    0.451757
  364      16     12913     12897   141.659       184    0.238692    0.451572
  365      16     12964     12948    141.83       204    0.252388    0.451034
  366      16     13012     12996   141.966       192    0.131394    0.450524
  367      16     13058     13042    142.08       184    0.272433     0.45026
  368      16     13105     13089   142.205       188    0.268804     0.44981
  369      16     13141     13125   142.209       144    0.311773     0.44929
  370      16     13179     13163   142.234       152     2.03279    0.449493
  371      16     13223     13207   142.325       176    0.227359     0.44935
  372      16     13269     13253   142.437       184    0.179598    0.448779
  373      16     13307     13291   142.462       152    0.318101    0.448404
  374      16     13351     13335   142.552       176     0.66818    0.448633
  375      16     13390     13374   142.587       156    0.223599     0.44857
  376      16     13435     13419   142.687       180    0.233914    0.448252
  377      16     13482     13466   142.807       188    0.277388    0.447959
  378      16     13529     13513   142.926       188    0.602233    0.447456
  379      16     13576     13560   143.045       188     0.20597    0.447141
2020-06-08 12:05:12.729810 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.446799
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     13625     13609   143.184       196    0.350819    0.446799
  381      16     13676     13660   143.343       204    0.377371    0.446217
  382      16     13719     13703   143.418       172    0.376005    0.445957
  383      16     13774     13758   143.618       220    0.202868     0.44556
  384      16     13812     13796    143.64       152    0.335153    0.445335
  385      16     13853     13837   143.692       164    0.195561    0.444999
  386      16     13899     13883   143.794       184    0.198264    0.444808
  387      16     13956     13940   144.012       228    0.127742    0.444229
  388      16     13997     13981   144.063       164    0.677126    0.444023
  389      16     14040     14024   144.135       172    0.708282    0.443725
  390      16     14091     14075   144.288       204     0.30224    0.443349
  391      16     14141     14125   144.431       200    0.215943    0.442911
  392      16     14179     14163    144.45       152    0.196804    0.442711
  393      16     14211     14195   144.408       128   0.0901027    0.442863
  394      16     14254     14238   144.478       172    0.610329    0.442763
  395      16     14293     14277   144.507       156    0.584145    0.442646
  396      16     14339     14323   144.606       184    0.410331    0.442333
  397      16     14382     14366   144.675       172    0.347563    0.442056
  398      16     14436     14420   144.854       216    0.148742     0.44166
  399      16     14485     14469   144.982       196    0.504047    0.441184
2020-06-08 12:05:32.741202 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.440707
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     14536     14520    145.13       204    0.354395    0.440707
  401      16     14585     14569   145.256       196    0.349962    0.440394
  402      16     14630     14614   145.342       180    0.797574    0.440063
  403      16     14681     14665   145.487       204    0.426952    0.439734
  404      16     14723     14707   145.543       168    0.428204     0.43952
  405      16     14775     14759   145.697       208    0.239253    0.439079
  406      16     14826     14810   145.841       204    0.480806    0.438641
  407      16     14881     14865   146.022       220    0.317419     0.43815
  408      16     14931     14915   146.154       200    0.210176    0.437732
  409      16     14982     14966   146.295       204    0.293815    0.437273
  410      16     15023     15007   146.339       164    0.611661    0.437085
  411      16     15075     15059   146.488       208    0.128427    0.436685
  412      16     15127     15111   146.638       208    0.132936    0.436226
  413      16     15178     15162   146.776       204    0.290908    0.435812
  414      16     15231     15215   146.933       212    0.358994    0.435405
  415      16     15272     15256   146.974       164    0.365472    0.435181
  416      16     15314     15298   147.024       168    0.335647    0.435094
  417      16     15366     15350   147.171       208     0.33835    0.434725
  418      16     15414     15398   147.278       192    0.462802    0.434373
  419      16     15462     15446   147.384       192    0.453619    0.433979
2020-06-08 12:05:52.751372 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.433664
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     15513     15497   147.519       204    0.391907    0.433664
  421      16     15564     15548   147.653       204    0.384983    0.433258
  422      16     15611     15595   147.748       188    0.246389    0.432973
  423      16     15656     15640   147.824       180    0.093805    0.432728
  424      16     15702     15686    147.91       184    0.179644    0.432476
  425      16     15752     15736   148.031       200    0.319919    0.432128
  426      16     15805     15789    148.18       212    0.175002     0.43168
  427      16     15851     15835   148.263       184    0.290861    0.431402
  428      16     15903     15887   148.403       208    0.318378    0.431115
  429      16     15946     15930   148.458       172    0.206864     0.43086
  430      16     16000     15984   148.615       216    0.417185    0.430481
  431      16     16044     16028   148.678       176    0.348322    0.430173
  432      16     16097     16081   148.824       212    0.255571    0.429839
  433      16     16150     16134    148.97       212    0.488645    0.429347
  434      16     16200     16184   149.088       200    0.229537    0.429113
  435      16     16254     16238   149.239       216    0.429418    0.428716
  436      16     16300     16284   149.319       184    0.364776    0.428328
  437      16     16350     16334   149.435       200    0.243105    0.428091
  438      16     16400     16384    149.55       200    0.267639    0.427729
  439      16     16447     16431   149.638       188    0.590509    0.427413
2020-06-08 12:06:12.768851 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.427493
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     16483     16467   149.625       144    0.468702    0.427493
  441      16     16524     16508   149.657       164    0.251888    0.427409
  442      16     16570     16554   149.735       184    0.205337    0.427203
  443      16     16623     16607   149.875       212    0.162406    0.426895
  444      16     16669     16653   149.951       184    0.286025    0.426638
  445      15     16718     16703   150.064       200    0.417447    0.426309
  446      16     16773     16757   150.211       216    0.189652    0.425987
  447      16     16818     16802   150.278       180    0.153314    0.425695
  448      16     16864     16848   150.353       184    0.297534    0.425524
  449      16     16904     16888   150.374       160     0.14118    0.425322
  450      16     16960     16944   150.538       224   0.0969851    0.424962
  451      16     16995     16979   150.514       140    0.146842    0.424915
  452      16     17047     17031   150.641       208    0.310798    0.424657
  453      16     17092     17076   150.706       180    0.417551    0.424498
  454      16     17132     17116   150.726       160    0.465241    0.424403
  455      16     17190     17174   150.905       232    0.245149    0.423994
  456      16     17236     17220   150.977       184    0.291126    0.423743
  457      16     17281     17265   151.037       180    0.454404     0.42348
  458      16     17326     17310     151.1       180    0.178635    0.423288
  459      16     17375     17359   151.197       196    0.223157    0.423067
2020-06-08 12:06:32.788643 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.422829
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     17425     17409   151.303       200    0.219531    0.422829
  461      16     17477     17461   151.426       208    0.339728    0.422485
  462      16     17532     17516   151.574       220    0.367407    0.422087
  463      16     17577     17561   151.635       180    0.267091    0.421897
  464      16     17626     17610    151.73       196    0.291679    0.421667
  465      16     17673     17657   151.808       188    0.334067    0.421415
  466      16     17710     17694     151.8       148    0.163953    0.421337
  467      16     17765     17749   151.946       220    0.222007    0.421052
  468      16     17812     17796   152.023       188    0.553244    0.420795
  469      16     17862     17846   152.125       200    0.224579    0.420571
  470      16     17918     17902   152.278       224    0.327208    0.420123
  471      16     17968     17952   152.379       200    0.261327    0.419853
  472      16     18020     18004   152.497       208    0.164646    0.419458
  473      16     18069     18053   152.589       196    0.258609    0.419252
  474      16     18127     18111   152.756       232    0.280153    0.418806
  475      16     18176     18160   152.847       196    0.468567    0.418589
  476      16     18211     18195    152.82       140    0.634876    0.418565
  477      16     18257     18241   152.885       184    0.135715    0.418466
  478      16     18312     18296   153.024       220    0.351938     0.41815
  479      16     18362     18346   153.121       200    0.383693    0.417802
2020-06-08 12:06:52.806381 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.417531
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     18411     18395   153.209       196   0.0827548    0.417531
  481      16     18456     18440   153.265       180    0.280516    0.417473
  482      16     18504     18488   153.345       192    0.223016    0.417164
  483      16     18552     18536   153.425       192      0.2243     0.41698
  484      16     18601     18585   153.513       196    0.403077    0.416713
  485      16     18649     18633   153.592       192    0.164898    0.416485
  486      16     18702     18686   153.712       212    0.454405    0.416235
  487      16     18753     18737   153.815       204     0.55738    0.415955
  488      16     18800     18784   153.885       188    0.326559     0.41571
  489      16     18845     18829   153.938       180    0.307519    0.415588
  490      16     18885     18869   153.951       160    0.163752    0.415423
  491      16     18939     18923   154.077       216    0.206451    0.415218
  492      16     18994     18978   154.211       220    0.448214    0.414921
  493      16     19033     19017   154.213       156    0.326946    0.414701
  494      16     19082     19066   154.296       196    0.337582    0.414674
  495      16     19135     19119   154.413       212    0.249238    0.414306
  496      16     19180     19164   154.464       180    0.105208    0.414089
  497      16     19227     19211   154.532       188    0.101409    0.413962
  498      16     19276     19260   154.614       196    0.294092    0.413753
  499      16     19328     19312   154.721       208    0.213221    0.413482
2020-06-08 12:07:12.820907 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.413285
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     19376     19360   154.795       192    0.531453    0.413285
  501      16     19422     19406   154.854       184    0.248493    0.413083
  502      16     19477     19461   154.983       220    0.657029    0.412738
  503      16     19524     19508   155.048       188    0.413768    0.412669
  504      16     19573     19557    155.13       196    0.313754    0.412422
  505      16     19616     19600   155.163       172    0.553231    0.412282
  506      16     19662     19646    155.22       184    0.142738    0.412156
  507      16     19709     19693   155.282       188    0.230748    0.412015
  508      16     19759     19743   155.369       200   0.0998474    0.411784
  509      16     19804     19788   155.417       180    0.378392    0.411674
  510      16     19849     19833   155.465       180    0.112066    0.411457
  511      16     19896     19880   155.529       188    0.237953    0.411342
  512      16     19947     19931   155.624       204    0.352576    0.411094
  513      16     19987     19971   155.632       160    0.257366    0.411005
  514      16     20040     20024   155.742       212    0.296339    0.410809
  515      16     20090     20074   155.827       200    0.384746    0.410599
  516      16     20144     20128   155.944       216    0.308672     0.41026
  517      16     20193     20177   156.021       196    0.236316    0.410056
  518      16     20245     20229   156.121       208    0.340904    0.409771
  519      16     20292     20276   156.182       188    0.806175    0.409652
2020-06-08 12:07:32.840163 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.409371
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     20344     20328   156.281       208    0.243567    0.409371
  521      16     20399     20383   156.403       220    0.328701    0.409084
  522      16     20445     20429   156.456       184    0.230827    0.408857
  523      16     20488     20472   156.485       172    0.454214    0.408771
  524      16     20540     20524   156.584       208    0.230132    0.408558
  525      16     20585     20569   156.628       180    0.141949    0.408427
  526      16     20626     20610   156.642       164    0.211831    0.408297
  527      16     20675     20659   156.717       196    0.386993    0.408298
  528      16     20723     20707   156.783       192    0.322823    0.408065
  529      16     20765     20749   156.804       168    0.238553    0.407967
  530      16     20816     20800   156.893       204    0.176812    0.407788
  531      16     20866     20850   156.974       200    0.363665    0.407535
  532      16     20911     20895   157.017       180    0.369616    0.407365
  533      16     20959     20943   157.083       192    0.400751    0.407259
  534      16     21008     20992   157.154       196    0.455853    0.407108
  535      16     21048     21032    157.16       160    0.764347    0.407024
  536      16     21082     21066    157.12       136    0.171308    0.407108
  537      16     21132     21116     157.2       200    0.130863     0.40698
  538      16     21179     21163   157.257       188    0.101887    0.406867
  539      16     21228     21212   157.329       196    0.168948    0.406631
2020-06-08 12:07:52.854944 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.406513
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     21274     21258   157.377       184    0.442846    0.406513
  541      16     21322     21306   157.441       192    0.332705     0.40635
  542      16     21375     21359   157.541       212    0.440169     0.40609
  543      16     21422     21406   157.598       188    0.385983    0.405945
  544      16     21475     21459   157.697       212    0.305759    0.405751
  545      16     21527     21511    157.79       208    0.176825    0.405499
  546      16     21570     21554   157.816       172     0.53299    0.405342
  547      16     21622     21606   157.907       208    0.321296    0.405171
  548      16     21675     21659   158.006       212    0.548881    0.404927
  549      16     21721     21705   158.053       184    0.828538    0.404788
  550      15     21774     21759   158.158       216    0.509562     0.40455
  551      16     21835     21819   158.307       240    0.363137    0.404145
  552      16     21888     21872   158.402       212    0.310273    0.403918
  553      16     21929     21913   158.412       164    0.225615    0.403842
  554      16     21984     21968   158.523       220    0.237284    0.403607
  555      16     22030     22014   158.569       184    0.218398    0.403441
  556      16     22083     22067   158.665       212    0.376688    0.403269
  557      16     22132     22116   158.732       196    0.153964    0.403055
  558      16     22185     22169   158.828       212    0.136437     0.40283
  559      16     22238     22222   158.923       212    0.474256    0.402602
2020-06-08 12:08:12.864001 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.402452
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     22283     22267    158.96       180    0.462368    0.402452
  561      16     22335     22319   159.047       208     0.33058    0.402278
  562      16     22376     22360   159.056       164    0.670445    0.402127
  563      16     22423     22407   159.107       188    0.102584    0.402078
  564      16     22467     22451   159.137       176    0.334776    0.401971
  565      16     22517     22501   159.209       200    0.287939    0.401841
  566      16     22570     22554   159.303       212    0.336027    0.401625
  567      16     22611     22595   159.311       164    0.282087    0.401573
  568      16     22664     22648   159.403       212    0.316354    0.401382
  569      16     22711     22695   159.454       188     0.36714    0.401212
  570      16     22752     22736   159.462       164     0.64866    0.401213
  571      16     22796     22780    159.49       176   0.0893633    0.401081
  572      16     22845     22829   159.554       196    0.217594     0.40101
  573      16     22897     22881   159.639       208    0.395147    0.400811
  574      16     22950     22934    159.73       212    0.421703    0.400534
  575      16     22996     22980   159.772       184    0.230524    0.400469
  576      16     23044     23028   159.828       192    0.106761    0.400302
  577      16     23092     23076   159.883       192    0.136746    0.400093
  578      16     23138     23122   159.925       184    0.260966    0.400044
  579      16     23184     23168   159.967       184     0.48356     0.39993
2020-06-08 12:08:32.868653 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.399762
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     23233     23217   160.029       196    0.100009    0.399762
  581      16     23285     23269   160.111       208    0.158893    0.399638
  582      16     23328     23312   160.131       172    0.252311    0.399499
  583      16     23384     23368   160.241       224    0.116848     0.39925
  584      16     23438     23422   160.336       216    0.258759    0.399074
  585      16     23482     23466   160.363       176    0.312108    0.398946
  586      16     23532     23516    160.43       200    0.107516    0.398786
  587      16     23589     23573   160.545       228    0.221537    0.398577
  588      16     23632     23616   160.564       172    0.535554    0.398405
  589      16     23678     23662   160.604       184    0.647506    0.398369
  590      16     23725     23709    160.65       188   0.0864281    0.398182
  591      16     23776     23760   160.724       204    0.532072     0.39804
  592      16     23831     23815   160.824       220    0.234719    0.397821
  593      16     23880     23864   160.882       196    0.262867    0.397658
  594      16     23935     23919   160.982       220    0.416923    0.397486
  595      16     23980     23964   161.014       180    0.184396    0.397362
  596      16     24019     24003   161.005       156    0.216402    0.397268
  597      16     24069     24053   161.071       200    0.373137     0.39722
  598      16     24123     24107   161.162       216    0.325188    0.396974
  599      16     24172     24156    161.22       196    0.238673    0.396852
2020-06-08 12:08:52.876561 min lat: 0.0703938 max lat: 4.09319 avg lat: 0.396759
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     24206     24190   161.178       136    0.263172    0.396759
Total time run:         600.336
Total writes made:      24206
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     161.283
Stddev Bandwidth:       40.5339
Max bandwidth (MB/sec): 240
Min bandwidth (MB/sec): 44
Average IOPS:           40
Stddev IOPS:            10.1335
Max IOPS:               60
Min IOPS:               11
Average Latency(s):     0.396815
Stddev Latency(s):      0.332784
Max latency(s):         4.09319
Min latency(s):         0.0703938

Total time run:         600.336
Total writes made:      24206
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     161.283
Stddev Bandwidth:       40.5339
Max bandwidth (MB/sec): 240
Min bandwidth (MB/sec): 44
Average IOPS:           40
Stddev IOPS:            10.1335
Max IOPS:               60
Min IOPS:               11
Average Latency(s):     0.396815
Stddev Latency(s):      0.332784
Max latency(s):         4.09319
Min latency(s):         0.0703938

Sequential Read Test, pool with 16 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      16        16         0         0         0           -           0
    1      16        61        45   179.725       180    0.653734    0.265785
    2      16       102        86   171.736       164     0.26818    0.304166
    3      16       141       125   166.455       156     0.62637    0.346581
    4      16       191       175   174.807       200    0.653689    0.333282
    5      16       236       220   175.837       180   0.0471615    0.341651
    6      16       281       265   176.454       180     0.68578    0.338637
    7      16       326       310   176.947       180    0.327634    0.340565
    8      16       377       361   180.322       204   0.0924538    0.335444
    9      15       424       409   181.595       192   0.0621164    0.338367
   10      16       475       459   183.404       200   0.0580188    0.330934
   11      16       517       501   182.002       168    0.305636    0.334249
   12      16       560       544   181.168       172   0.0857937    0.338028
   13      16       597       581   178.603       148    0.328407    0.345288
   14      16       642       626   178.701       180   0.0691258    0.344505
   15      16       685       669   178.253       172   0.0807547    0.348703
   16      16       723       707   176.612       152    0.200282    0.349929
   17      16       766       750   176.326       172    0.154271     0.35287
   18      16       818       802   178.045       208   0.0766331    0.349761
   19      16       862       846   177.936       176     0.44736    0.350352
2020-06-08 12:21:19.424208 min lat: 0.0249483 max lat: 1.60795 avg lat: 0.350601
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       908       892   178.182       184    0.249175    0.350601
   21      16       953       937   178.267       180    0.292387    0.348578
   22      16       994       978    177.61       164    0.059223    0.350062
   23      16      1032      1016   176.497       152     0.87098    0.352958
   24      16      1070      1054   175.477       152    0.114091    0.354365
   25      15      1103      1088   173.859       136    0.100794    0.357305
   26      16      1147      1131   173.775       172    0.296744    0.358744
   27      16      1188      1172     173.4       164   0.0501677    0.359326
   28      15      1223      1208   172.349       144    0.662945    0.362405
   29      16      1265      1249   171.949       164     1.23023    0.361509
   30      16      1312      1296   172.467       188    0.338225    0.360919
   31      16      1352      1336   172.065       160    0.138675    0.362541
   32      16      1396      1380   172.187       176    0.229331    0.363456
   33      16      1436      1420   171.818       160   0.0771103    0.363231
   34      16      1477      1461   171.587       164    0.382998    0.363294
   35      16      1524      1508   172.055       188    0.106015    0.362061
   36      16      1564      1548   171.714       160    0.289089    0.362784
   37      15      1604      1589   171.496       164   0.0598599    0.363486
   38      16      1642      1626   170.878       148     1.15266    0.366261
   39      16      1675      1659   169.882       132    0.255769    0.368106
2020-06-08 12:21:39.463237 min lat: 0.0248126 max lat: 1.69083 avg lat: 0.367704
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      15      1726      1711   170.829       208    0.545817    0.367704
   41      16      1769      1753   170.608       168   0.0535616    0.368004
   42      15      1810      1795   170.541       168    0.373448    0.367582
   43      16      1855      1839   170.667       176    0.553561     0.36781
   44      16      1897      1881   170.601       168    0.612187    0.368522
   45      16      1940      1924   170.631       172    0.270757    0.368138
   46      16      1981      1965   170.475       164    0.748291    0.368665
   47      16      2022      2006   170.335       164    0.800102    0.367622
   48      16      2059      2043    169.87       148    0.166018    0.368026
   49      16      2101      2085   169.831       168    0.131844    0.369179
   50      16      2149      2133   170.272       192   0.0691953    0.369083
   51      16      2193      2177   170.384       176      1.1028    0.368662
   52      16      2224      2208   169.493       124   0.0501345    0.368457
   53      16      2254      2238   168.552       120     1.97205    0.368903
   54      16      2300      2284   168.837       184    0.780315    0.372975
   55      16      2339      2323   168.604       156   0.0482763    0.372959
   56      16      2386      2370   168.947       188      0.1946     0.37259
   57      16      2426      2410   168.789       160    0.885615    0.372047
   58      16      2467      2451   168.707       164    0.176191    0.372209
   59      16      2510      2494   168.761       172   0.0975932    0.372901
2020-06-08 12:21:59.513225 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.372011
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2557      2541    169.08       188    0.258079    0.372011
   61      16      2599      2583   169.062       168    0.283663    0.371939
   62      16      2641      2625   169.045       168    0.154265    0.371796
   63      16      2682      2666   168.964       164    0.121294    0.371245
   64      16      2719      2703   168.635       148    0.210276    0.372871
   65      16      2766      2750   168.932       188    0.301155    0.372831
   66      16      2814      2798    169.28       192   0.0746804    0.371501
   67      16      2857      2841    169.32       172    0.196694    0.372003
   68      15      2905      2890     169.7       196    0.186451    0.370624
   69      16      2946      2930    169.56       160    0.282274    0.371114
   70      16      2992      2976   169.765       184     1.14253    0.370176
   71      16      3031      3015   169.571       156    0.228478    0.371012
   72      16      3078      3062   169.826       188    0.209222    0.370815
   73      15      3130      3115   170.398       212    0.179196    0.369642
   74      16      3173      3157   170.366       168    0.958304    0.369307
   75      16      3208      3192   169.961       140     1.01501     0.36968
   76      16      3251      3235   169.988       172   0.0769253    0.370113
   77      16      3290      3274   169.806       156     0.40686    0.370771
   78      16      3341      3325   170.243       204    0.143182    0.369888
   79      16      3378      3362   169.962       148    0.944513    0.370147
2020-06-08 12:22:19.524533 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.369985
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      3420      3404   169.935       168    0.807407    0.369985
   81      16      3463      3447    169.96       172    0.493126    0.370948
   82      16      3504      3488   169.887       164     1.13408    0.370608
   83      15      3545      3530   169.853       168     0.24172    0.370816
   84      16      3589      3573   169.875       172    0.521379    0.370599
   85      15      3632      3617   169.946       176   0.0960671    0.370405
   86      16      3676      3660   169.962       172    0.102477    0.370182
   87      16      3721      3705   170.077       180    0.516006     0.37001
   88      16      3764      3748   170.099       172    0.180662    0.370361
   89      16      3811      3795   170.299       188    0.353325    0.369694
   90      16      3861      3845   170.629       200      0.6795    0.369402
   91      16      3903      3887   170.599       168    0.474503    0.369444
   92      16      3946      3930   170.614       172    0.407425    0.369108
   93      16      3988      3972   170.586       168    0.106546    0.368901
   94      15      4030      4015   170.592       172    0.334292    0.369065
   95      16      4079      4063   170.817       192   0.0831617    0.368644
   96      16      4116      4100   170.579       148    0.531145      0.3694
   97      16      4155      4139   170.421       156    0.888093     0.36943
   98      16      4202      4186     170.6       188    0.429109     0.36934
   99      16      4239      4223   170.372       148    0.597653    0.369292
2020-06-08 12:22:39.549519 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.36881
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      4292      4276   170.784       212    0.382936     0.36881
  101      16      4332      4316   170.677       160   0.0284035     0.36907
  102      16      4376      4360   170.729       176     0.57246    0.369122
  103      16      4419      4403   170.741       172    0.280385    0.369219
  104      16      4462      4446   170.753       172    0.401163    0.369185
  105      16      4504      4488   170.726       168    0.779487    0.368732
  106      16      4540      4524   170.474       144     1.48542    0.368711
  107      16      4587      4571   170.638       188    0.189971    0.369469
  108      16      4630      4614    170.65       172   0.0716492    0.369319
  109      16      4672      4656   170.626       168    0.169196     0.36955
  110      16      4716      4700   170.674       176   0.0684362    0.369269
  111      16      4760      4744   170.722       176    0.833629    0.369114
  112      16      4799      4783    170.59       156    0.668671    0.369513
  113      16      4841      4825   170.567       168    0.168062    0.369518
  114      16      4886      4870    170.65       180    0.316925    0.369232
  115      15      4932      4917     170.8       188    0.137072    0.369261
  116      16      4973      4957   170.705       160   0.0846937    0.369009
  117      16      5016      5000    170.71       172     0.52219    0.368954
  118      16      5062      5046   170.821       184    0.398768    0.369099
  119      15      5092      5077   170.425       124   0.0661291    0.368983
2020-06-08 12:22:59.560617 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.369774
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      5132      5116   170.305       156   0.0683798    0.369774
  121      16      5174      5158   170.285       168    0.154458    0.370022
  122      16      5208      5192   170.004       136     1.02461    0.370329
  123      16      5235      5219     169.5       108    0.129999    0.371054
  124      16      5280      5264    169.58       180    0.225555    0.371786
  125      15      5316      5301   169.377       148    0.241433    0.371444
  126      16      5356      5340    169.27       156     0.22375    0.371694
  127      16      5394      5378   169.134       152      1.0247     0.37258
  128      16      5433      5417    169.03       156     1.55902     0.37207
  129      16      5478      5462   169.112       180    0.043229    0.372815
  130      16      5526      5510   169.279       192     0.23312    0.372621
  131      16      5566      5550   169.207       160   0.0453516    0.372683
  132      16      5608      5592   169.198       168   0.0749937    0.372598
  133      16      5651      5635   169.219       172    0.908214    0.372527
  134      16      5692      5676   169.172       164     0.12826     0.37275
  135      16      5739      5723   169.302       188   0.0424444    0.372397
  136      16      5786      5770   169.438       188    0.207119    0.372037
  137      16      5828      5812    169.42       168   0.0628769     0.37193
  138      16      5864      5848   169.236       144     1.41336     0.37205
  139      16      5909      5893   169.313       180     1.58894    0.372159
2020-06-08 12:23:19.621113 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.37275
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5943      5927   169.076       136   0.0688151     0.37275
  141      16      5975      5959   168.784       128    0.254856     0.37341
  142      16      6015      5999   168.721       160     0.13998    0.373737
  143      15      6059      6044   168.797       180    0.324826    0.373979
  144      16      6111      6095   169.038       204   0.0586259    0.373278
  145      16      6151      6135   168.973       160    0.176567     0.37326
  146      15      6195      6180   169.044       180    0.960333    0.373222
  147      16      6237      6221    169.01       164   0.0696781    0.373076
  148      16      6277      6261   168.946       160     1.31551    0.373172
  149      16      6319      6303   168.939       168     1.06384    0.373341
  150      16      6360      6344   168.905       164     1.07922    0.373454
  151      16      6397      6381   168.766       148   0.0775141    0.373405
  152      16      6437      6421   168.707       160    0.888969     0.37367
  153      16      6478      6462   168.676       164   0.0954357    0.373791
  154      16      6521      6505   168.679       172     1.08875    0.373622
  155      16      6564      6548   168.696       172    0.106057    0.373667
  156      16      6613      6597   168.871       196    0.247057    0.373366
  157      16      6655      6639   168.865       168    0.152871    0.373457
  158      16      6695      6679   168.809       160     0.32451    0.373523
  159      16      6736      6720    168.74       164    0.797271    0.373767
2020-06-08 12:23:39.698033 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.373872
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      6776      6760   168.686       160    0.175902    0.373872
  161      16      6820      6804   168.726       176    0.430652    0.373903
  162      15      6870      6855   168.941       204    0.345304    0.373512
  163      15      6916      6901   169.033       184   0.0927639    0.373159
  164      15      6961      6946    169.09       180    0.301668    0.373161
  165      16      7014      6998   169.324       208     0.39952    0.372602
  166      16      7059      7043   169.388       180    0.272027     0.37244
  167      16      7106      7090   169.498       188    0.346263    0.372241
  168      15      7153      7138    169.63       192    0.303535    0.372102
  169      16      7200      7184   169.715       184    0.792886    0.371806
  170      16      7237      7221   169.587       148    0.397384    0.371751
  171      16      7280      7264   169.601       172    0.100315    0.371644
  172      16      7317      7301   169.475       148    0.642609    0.371243
  173      16      7354      7338   169.351       148    0.151381     0.37173
  174      16      7397      7381   169.366       172    0.299631    0.372604
  175      16      7443      7427   169.449       184    0.908767    0.372409
  176      16      7483      7467   169.396       160    0.385455    0.372517
  177      16      7525      7509   169.385       168    0.407758     0.37254
  178      15      7567      7552   169.398       172    0.273753    0.372301
  179      16      7617      7601    169.54       196    0.110554    0.372117
2020-06-08 12:23:59.734115 min lat: 0.0230592 max lat: 2.08667 avg lat: 0.371832
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      7662      7646   169.596       180      0.2817    0.371832
  181      16      7705      7689   169.609       172    0.269791     0.37186
  182      16      7745      7729   169.556       160   0.0923176    0.372007
  183      16      7784      7768   169.482       156    0.631796    0.372254
  184      16      7820      7804   169.343       144    0.713916    0.372468
  185      16      7861      7845   169.314       164    0.319438    0.372665
  186      15      7907      7892   169.412       188    0.298928    0.372459
  187      16      7949      7933   169.383       164    0.111257    0.372334
  188      16      7992      7976   169.397       172    0.111834    0.372168
  189      16      8032      8016   169.347       160   0.0932603    0.372001
  190      16      8070      8054   169.256       152     1.72287    0.372328
  191      16      8092      8076   168.831        88    0.049547    0.372895
  192      16      8119      8103   168.514       108   0.0920323    0.373235
  193      16      8156      8140   168.408       148   0.0856176    0.374086
  194      16      8199      8183   168.426       172    0.116559    0.374232
  195      16      8237      8221   168.337       152    0.215625    0.374518
  196      16      8278      8262   168.315       164   0.0801257    0.374161
  197      16      8320      8304   168.312       168    0.233133    0.374362
  198      16      8358      8342    168.23       152    0.211434    0.374398
  199      16      8392      8376   168.068       136   0.0805567    0.374965
2020-06-08 12:24:19.749046 min lat: 0.0230592 max lat: 2.31143 avg lat: 0.375312
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      8440      8424   168.186       192    0.893368    0.375312
  201      16      8484      8468   168.221       176    0.306854    0.375168
  202      16      8524      8508   168.144       160     0.13247    0.374973
  203      16      8560      8544   168.025       144    0.148057    0.375034
  204      16      8591      8575    167.81       124     0.10823    0.375202
  205      16      8624      8608   167.635       132   0.0907727    0.375628
  206      16      8660      8644   167.519       144   0.0740024    0.375771
  207      16      8703      8687   167.541       172    0.292113     0.37628
  208      16      8750      8734   167.639       188    0.120207    0.376575
  209      16      8790      8774   167.601       160    0.300057    0.376465
  210      15      8838      8823   167.733       196    0.259945    0.376447
  211      16      8889      8873   167.885       200    0.297212     0.37591
  212      15      8936      8921   167.989       192   0.0924405    0.375739
  213      15      8981      8966   168.044       180    0.756622    0.375544
  214      16      9024      9008   168.043       168    0.742799    0.375558
  215      16      9069      9053   168.099       180   0.0969489    0.375406
  216      15      9114      9099   168.172       184    0.507433     0.37537
  217      16      9159      9143   168.208       176    0.604682    0.375247
  218      16      9195      9179   168.097       144    0.075566     0.37535
  219      16      9233      9217    168.02       152    0.459101    0.375446
2020-06-08 12:24:39.826809 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.375683
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      9274      9258   168.001       164    0.709525    0.375683
  221      16      9316      9300   168.001       168    0.105815    0.375641
  222      15      9356      9341   167.982       164    0.103179    0.375719
  223      16      9402      9386   168.036       180   0.0976973    0.375358
  224      15      9446      9431   168.087       180    0.220347    0.375707
  225      15      9492      9477   168.157       184     1.01907    0.375363
  226      16      9531      9515   168.086       152    0.376302    0.375362
  227      16      9574      9558   168.103       172    0.937817    0.375517
  228      16      9613      9597    168.05       156    0.660098    0.375498
  229      15      9661      9646   168.163       196    0.818207    0.375485
  230      16      9705      9689   168.179       172   0.0882443    0.375519
  231      16      9755      9739   168.315       200    0.171719    0.375227
  232      16      9802      9786     168.4       188    0.136964    0.374891
  233      16      9844      9828   168.398       168    0.237665    0.374939
  234      16      9887      9871   168.413       172    0.295812    0.374974
  235      16      9932      9916   168.462       180   0.0674042    0.374769
  236      16      9976      9960   168.494       176    0.132963    0.374399
  237      16     10019     10003   168.509       172   0.0382068    0.374833
  238      16     10062     10046   168.523       172    0.753091    0.374687
  239      16     10106     10090   168.546       176    0.428319    0.374695
2020-06-08 12:24:59.860167 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374344
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16     10155     10139    168.66       196   0.0943221    0.374344
  241      15     10197     10182   168.647       172     0.20105    0.374382
  242      16     10234     10218   168.545       144     1.05015    0.374511
  243      15     10279     10264   168.608       184   0.0601367    0.374424
  244      16     10322     10306   168.605       168    0.367138    0.374263
  245      16     10363     10347   168.586       164   0.0847016    0.374395
  246      16     10401     10385   168.518       152    0.124863    0.374697
  247      16     10437     10421   168.417       144    0.127071    0.374713
  248      16     10486     10470   168.528       196   0.0701429    0.374565
  249      16     10529     10513   168.541       172    0.159608    0.374656
  250      16     10567     10551   168.474       152    0.450094     0.37483
  251      16     10606     10590   168.424       156    0.646874    0.374853
  252      16     10650     10634   168.454       176     1.07673    0.374889
  253      16     10682     10666   168.293       128    0.872949    0.375043
  254      16     10727     10711   168.339       180   0.0961278    0.374819
  255      16     10771     10755   168.369       176   0.0948567    0.374893
  256      16     10812     10796   168.352       164   0.0991479    0.374972
  257      16     10848     10832   168.257       144     1.24371      0.3752
  258      16     10893     10877   168.303       180    0.657484     0.37519
  259      16     10935     10919   168.301       168    0.100677    0.375137
2020-06-08 12:25:19.917240 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.375227
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16     10981     10965   168.357       184   0.0463872    0.375227
  261      15     11025     11010   168.401       180      0.3973    0.375052
  262      16     11072     11056   168.459       184    0.338258    0.374912
  263      16     11117     11101   168.502       180    0.595439    0.374865
  264      16     11154     11138   168.424       148     0.29759    0.374847
  265      16     11191     11175   168.346       148    0.157838    0.375024
  266      16     11232     11216    168.33       164   0.0774951    0.375107
  267      16     11284     11268   168.475       208   0.0569242    0.374752
  268      16     11319     11303   168.369       140    0.111172    0.374761
  269      16     11355     11339   168.277       144    0.500632    0.375068
  270      16     11397     11381   168.276       168    0.218784    0.375335
  271      16     11443     11427   168.334       184   0.0822048     0.37517
  272      16     11490     11474   168.406       188    0.361562    0.374994
  273      16     11536     11520   168.463       184    0.499848    0.374917
  274      16     11584     11568   168.548       192    0.405607    0.374757
  275      16     11632     11616   168.633       192    0.921353     0.37453
  276      15     11672     11657   168.615       164    0.471223    0.374392
  277      16     11713     11697   168.583       160    0.920939    0.374607
  278      16     11755     11739   168.581       168     0.10323    0.374535
  279      16     11796     11780   168.564       164   0.0690183    0.374355
2020-06-08 12:25:39.937611 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374501
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     11839     11823   168.576       172   0.0729308    0.374501
  281      16     11878     11862   168.531       156     0.62186    0.374631
  282      16     11921     11905   168.543       172    0.173199    0.374627
  283      16     11962     11946   168.521       164    0.613569    0.374548
  284      15     12002     11987   168.505       164    0.409955     0.37471
  285      16     12045     12029   168.503       168    0.507504    0.374697
  286      16     12085     12069    168.46       160    0.142108    0.374831
  287      16     12131     12115   168.499       184    0.423421    0.374676
  288      16     12181     12165   168.608       200    0.349587    0.374549
  289      16     12234     12218   168.757       212    0.434835    0.374189
  290      16     12276     12260   168.755       168   0.0687152     0.37401
  291      16     12320     12304    168.78       176   0.0388994    0.374132
  292      16     12361     12345   168.763       164    0.926596    0.374063
  293      15     12410     12395   168.869       200    0.491036    0.373951
  294      16     12449     12433   168.811       152     0.35548    0.373944
  295      16     12494     12478   168.846       180     0.14995    0.373931
  296      16     12537     12521   168.856       172    0.100294    0.373855
  297      16     12580     12564   168.867       172   0.0870731    0.373873
  298      16     12624     12608    168.89       176    0.179845    0.373846
  299      16     12672     12656   168.966       192    0.158471    0.373711
2020-06-08 12:26:00.010689 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373596
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     12721     12705   169.056       196    0.090572    0.373596
  301      16     12767     12751   169.092       184    0.058073    0.373406
  302      16     12810     12794   169.095       172     1.22799    0.373267
  303      16     12854     12838   169.118       176    0.280225     0.37343
  304      16     12897     12881   169.127       172    0.271161    0.373415
  305      15     12939     12924   169.135       172    0.788942    0.373355
  306      15     12977     12962   169.075       152    0.732941    0.373458
  307      15     13019     13004   169.048       168   0.0439113     0.37342
  308      16     13066     13050   169.095       184     1.24987    0.373496
  309      16     13110     13094   169.118       176    0.284408    0.373332
  310      16     13148     13132   169.062       152    0.228812    0.373455
  311      16     13191     13175   169.072       172    0.499808     0.37354
  312      16     13240     13224   169.158       196    0.165439    0.373414
  313      16     13284     13268    169.18       176   0.0820455    0.373294
  314      16     13332     13316   169.252       192    0.525569    0.373007
  315      16     13372     13356   169.222       160   0.0941317     0.37301
  316      16     13416     13400   169.243       176     0.19094    0.372979
  317      16     13455     13439   169.201       156    0.368214    0.373151
  318      16     13494     13478   169.159       156     1.40004    0.373129
  319      15     13537     13522   169.178       176   0.0604933    0.373176
2020-06-08 12:26:20.110826 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373125
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      15     13578     13563   169.162       164    0.149702    0.373125
  321      16     13624     13608   169.195       180    0.371208    0.373004
  322      16     13672     13656   169.259       192    0.197198    0.372899
  323      16     13716     13700   169.279       176    0.701214    0.372868
  324      16     13755     13739   169.238       156    0.854059    0.373033
  325      16     13799     13783   169.259       176   0.0956063    0.372985
  326      16     13839     13823    169.23       160     1.27962    0.372944
  327      15     13879     13864   169.213       164    0.026284    0.372841
  328      16     13927     13911   169.271       188   0.0643772    0.372945
  329      16     13962     13946    169.18       140    0.302032    0.373101
  330      16     14001     13985   169.136       156     1.51292    0.373174
  331      16     14037     14021   169.059       144    0.470913    0.373425
  332      16     14085     14069   169.128       192    0.921336    0.373352
  333      16     14123     14107   169.076       152    0.534653    0.373415
  334      15     14162     14147   169.047       160     1.42114    0.373338
  335      16     14206     14190   169.056       172     0.99175    0.373491
  336      16     14248     14232   169.053       168     1.03147    0.373408
  337      16     14297     14281   169.132       196    0.194408     0.37337
  338      15     14335     14320   169.092       156    0.104599    0.373343
  339      16     14375     14359   169.053       156     1.12634    0.373467
2020-06-08 12:26:40.153864 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373552
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      15     14416     14401   169.049       168    0.315563    0.373552
  341      16     14465     14449   169.115       192    0.284339    0.373345
  342      15     14503     14488   169.072       156    0.190026      0.3734
  343      16     14548     14532   169.093       176    0.329976    0.373403
  344      16     14588     14572   169.048       160    0.868697    0.373524
  345      16     14626     14610   168.998       152    0.197223    0.373709
  346      16     14671     14655   169.029       180   0.0527044    0.373635
  347      15     14714     14699   169.048       176    0.264045    0.373598
  348      16     14759     14743   169.066       176    0.287995    0.373565
  349      16     14808     14792   169.143       196   0.0734263    0.373333
  350      16     14850     14834   169.139       168    0.845664    0.373225
  351      16     14892     14876   169.136       168    0.347596    0.373362
  352      16     14933     14917   169.121       164    0.553563    0.373443
  353      16     14975     14959   169.118       168    0.388754    0.373438
  354      16     15011     14995   169.046       144     1.08782     0.37342
  355      16     15052     15036   169.023       164    0.230196    0.373354
  356      16     15101     15085   169.081       196    0.331586    0.373464
  357      16     15149     15133   169.131       192    0.155034    0.373396
  358      16     15191     15175   169.128       168    0.950143    0.373327
  359      16     15239     15223   169.191       192    0.254462    0.373304
2020-06-08 12:27:00.300879 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373238
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     15281     15265   169.188       168    0.222089    0.373238
  361      16     15322     15306   169.172       164    0.165509    0.373221
  362      16     15366     15350   169.189       176    0.630044    0.373239
  363      16     15414     15398   169.251       192      1.1549    0.373111
  364      16     15454     15438   169.222       160     0.33455    0.373049
  365      16     15494     15478   169.197       160   0.0976162    0.373173
  366      16     15531     15515   169.139       148    0.400027    0.373355
  367      16     15575     15559   169.158       176    0.890982    0.373261
  368      16     15610     15594   169.079       140    0.334188    0.373346
  369      16     15648     15632   169.033       152    0.696061    0.373407
  370      16     15690     15674    169.03       168    0.301545    0.373403
  371      16     15734     15718   169.048       176    0.297896     0.37346
  372      16     15773     15757   169.013       156    0.102745    0.373609
  373      16     15811     15795   168.959       152      1.2442    0.373519
  374      16     15850     15834   168.924       156    0.266913    0.373633
  375      16     15887     15871   168.868       148    0.154025    0.373804
  376      16     15923     15907   168.801       144     1.56844    0.373869
  377      16     15967     15951    168.82       176    0.438185    0.374101
  378      16     16009     15993   168.818       168    0.434633    0.374083
  379      16     16058     16042   168.889       196    0.288303    0.373905
2020-06-08 12:27:20.341162 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373764
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     16106     16090    168.95       192    0.102851    0.373764
  381      16     16143     16127   168.895       148     0.59253    0.373856
  382      16     16187     16171   168.913       176   0.0711049    0.373791
  383      16     16233     16217   168.953       184    0.307945    0.373758
  384      16     16274     16258    168.94       164    0.997311    0.373781
  385      16     16315     16299   168.927       164   0.0534854     0.37376
  386      15     16357     16342   168.934       172     1.30778    0.373766
  387      16     16402     16386   168.952       176   0.0273739    0.373742
  388      16     16446     16430    168.97       176   0.0823523    0.373643
  389      16     16486     16470   168.947       160    0.236351    0.373601
  390      16     16529     16513   168.954       172   0.0866406    0.373601
  391      16     16573     16557   168.971       176   0.0804077    0.373588
  392      16     16605     16589   168.867       128    0.659212    0.373787
  393      16     16649     16633   168.882       176    0.324744    0.373891
  394      16     16691     16675   168.879       168    0.109391    0.373982
  395      16     16735     16719    168.88       176    0.371198    0.373865
  396      16     16781     16765   168.918       184    0.250705     0.37384
  397      16     16823     16807   168.915       168    0.218924     0.37385
  398      15     16866     16851   168.932       176    0.284068    0.373831
  399      16     16911     16895    168.95       176    0.371854    0.373711
2020-06-08 12:27:40.400625 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373788
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     16948     16932   168.897       148   0.0891069    0.373788
  401      16     16980     16964   168.796       128    0.062622    0.374021
  402      16     17024     17008   168.813       176   0.0428555    0.373992
  403      16     17069     17053    168.84       180    0.357778     0.37399
  404      16     17111     17095   168.836       168    0.156089    0.373952
  405      15     17156     17141   168.873       184   0.0988409    0.373801
  406      16     17194     17178   168.822       148    0.101354    0.373972
  407      16     17240     17224   168.859       184    0.480111    0.373999
  408      16     17280     17264   168.837       160    0.334585    0.374053
  409      16     17329     17313   168.904       196     0.45556    0.373887
  410      16     17368     17352   168.871       156     1.04888    0.373839
  411      16     17411     17395   168.877       172   0.0679636    0.373787
  412      16     17450     17434   168.846       156     1.07292    0.373792
  413      16     17487     17471   168.796       148     1.17226    0.373957
  414      16     17530     17514   168.803       172     1.16831    0.374032
  415      16     17575     17559    168.83       180   0.0991521    0.373938
  416      16     17602     17586   168.683       108     1.48129    0.373972
  417      16     17634     17618   168.586       128   0.0961447    0.374313
  418      16     17677     17661   168.593       172    0.285313    0.374612
  419      15     17719     17704     168.6       172    0.807376    0.374583
2020-06-08 12:28:00.423428 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374479
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     17765     17749   168.627       180     0.11501    0.374479
  421      16     17805     17789   168.607       160     1.00096    0.374543
  422      16     17847     17831   168.605       168    0.112573     0.37464
  423      16     17893     17877   168.641       184    0.460584    0.374563
  424      16     17940     17924   168.687       188   0.0542812    0.374299
  425      16     17982     17966   168.683       168    0.147957    0.374244
  426      16     18022     18006   168.663       160   0.0830238    0.374335
  427      16     18071     18055   168.727       196    0.094898    0.374253
  428      16     18115     18099   168.743       176     1.00264    0.374132
  429      16     18161     18145   168.779       184    0.260795    0.374049
  430      16     18190     18174   168.656       116    0.439204    0.374323
  431      15     18236     18221   168.701       188   0.0688787     0.37434
  432      16     18283     18267   168.736       184     0.10751    0.374236
  433      16     18331     18315   168.789       192    0.304596    0.374142
  434      15     18372     18357   168.786       168   0.0558009    0.374132
  435      16     18413     18397   168.766       160     1.01258    0.374165
  436      16     18456     18440   168.766       172    0.443794    0.374169
  437      16     18491     18475   168.698       140    0.382729    0.374223
  438      16     18534     18518   168.691       172    0.462145    0.374213
  439      15     18582     18567   168.753       196    0.271437    0.374163
2020-06-08 12:28:20.501147 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374149
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      15     18623     18608   168.741       164    0.090205    0.374149
  441      16     18665     18649    168.73       164    0.593713    0.374186
  442      16     18709     18693   168.746       176    0.379424    0.374092
  443      16     18750     18734   168.733       164   0.0899103    0.374166
  444      15     18786     18771   168.678       148    0.611534    0.374258
  445      16     18832     18816   168.703       180    0.158367    0.374262
  446      16     18885     18869     168.8       212     0.80647    0.374105
  447      15     18923     18908   168.771       156    0.099704     0.37403
  448      15     18967     18952   168.784       176   0.0231707    0.374053
  449      16     19014     18998   168.818       184   0.0553393    0.373923
  450      16     19046     19030   168.721       128     0.09116    0.374001
  451      16     19096     19080    168.79       200    0.031907    0.373984
  452      16     19134     19118   168.753       152   0.0739179    0.374068
  453      16     19169     19153   168.689       140    0.232571    0.374115
  454      16     19213     19197   168.705       176    0.418458    0.374043
  455      16     19254     19238   168.694       164    0.277625    0.374144
  456      16     19302     19286   168.743       192    0.200266     0.37418
  457      16     19358     19342   168.863       224   0.0717209    0.373924
  458      16     19402     19386   168.878       176    0.126194    0.373794
  459      16     19442     19426   168.859       160    0.315356     0.37374
2020-06-08 12:28:40.572353 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373873
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     19485     19469   168.865       172     1.10045    0.373873
  461      16     19525     19509   168.837       160    0.447995    0.373842
  462      16     19564     19548   168.809       156    0.439457    0.373913
  463      16     19600     19584   168.755       144   0.0422886    0.374122
  464      16     19642     19626   168.753       168    0.275161    0.374125
  465      16     19690     19674   168.802       192   0.0649647    0.374009
  466      16     19741     19725   168.876       204    0.101011    0.373934
  467      16     19794     19778   168.968       212    0.100933    0.373578
  468      15     19831     19816   168.931       152    0.070962    0.373754
  469      16     19876     19860   168.945       176   0.0472632    0.373631
  470      16     19918     19902   168.939       168    0.272227    0.373537
  471      16     19957     19941   168.911       156     1.47847    0.373679
  472      16     20003     19987   168.943       184   0.0821059    0.373639
  473      16     20044     20028   168.932       164     1.12177     0.37375
  474      16     20088     20072   168.942       176   0.0653919    0.373706
  475      16     20129     20113   168.932       164    0.803102    0.373695
  476      16     20171     20155   168.929       168    0.424077     0.37381
  477      16     20217     20201    168.96       184     0.35976    0.373763
  478      16     20255     20239   168.925       152   0.0528211    0.373753
  479      16     20300     20284   168.948       180   0.0582426    0.373685
2020-06-08 12:29:00.652451 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373749
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     20341     20325   168.934       164     1.11283    0.373749
  481      16     20380     20364   168.907       156    0.585802    0.373812
  482      16     20422     20406   168.905       168   0.0464579    0.373784
  483      16     20465     20449   168.912       172    0.626965    0.373749
  484      16     20504     20488   168.869       156    0.123267    0.373827
  485      16     20553     20537   168.925       196    0.315675    0.373704
  486      16     20599     20583   168.955       184     1.24376    0.373673
  487      15     20640     20625   168.953       168   0.0696686    0.373654
  488      16     20689     20673       169       192   0.0469364     0.37358
  489      16     20728     20712   168.973       156   0.0309123    0.373647
  490      16     20769     20753   168.963       164    0.113886     0.37361
  491      16     20816     20800   169.002       188      1.5052    0.373531
  492      16     20856     20840   168.983       160   0.0744769    0.373628
  493      16     20896     20880   168.964       160   0.0434707    0.373596
  494      16     20936     20920   168.946       160    0.089939    0.373704
  495      16     20972     20956   168.895       144    0.318724    0.373779
  496      15     21009     20994   168.859       152   0.0729679     0.37393
  497      16     21049     21033   168.834       156   0.0760474    0.373911
  498      16     21089     21073   168.816       160   0.0967915    0.373925
  499      16     21125     21109   168.766       144    0.235737    0.373987
2020-06-08 12:29:20.713789 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.374068
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     21170     21154   168.788       180     0.70099    0.374068
  501      16     21209     21193   168.763       156   0.0473979    0.374112
  502      16     21249     21233   168.745       160   0.0695431    0.374115
  503      16     21287     21271   168.706       152    0.175888    0.374247
  504      16     21337     21321   168.768       200    0.275681    0.374105
  505      16     21371     21355   168.699       136    0.820066    0.374221
  506      16     21414     21398   168.705       172    0.395828    0.374157
  507      16     21452     21436   168.673       152   0.0561993    0.374271
  508      16     21505     21489    168.74       212    0.109654     0.37416
  509      15     21549     21534   168.762       180   0.0586498    0.374031
  510      16     21590     21574   168.744       160    0.472577    0.374101
  511      16     21630     21614   168.727       160    0.479988    0.374137
  512      16     21669     21653   168.702       156    0.250771    0.374201
  513      16     21701     21685   168.623       128   0.0358874    0.374358
  514      16     21750     21734   168.676       196    0.947301    0.374273
  515      16     21795     21779   168.696       180   0.0787879    0.374205
  516      15     21837     21822   168.702       172   0.0434811    0.374192
  517      16     21888     21872   168.762       200    0.390348    0.374104
  518      16     21932     21916   168.775       176   0.0632455    0.373985
  519      16     21977     21961   168.797       180   0.0785645     0.37388
2020-06-08 12:29:40.812520 min lat: 0.0230592 max lat: 2.43145 avg lat: 0.373921
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     22028     22012   168.864       204   0.0770314    0.373921
  521      15     22077     22062   168.923       200   0.0860754    0.373745
  522      16     22127     22111   168.974       196   0.0544964    0.373598
  523      16     22172     22156   168.995       180    0.103899    0.373551
  524      16     22211     22195   168.971       156    0.266032    0.373505
  525      16     22260     22244   169.021       196    0.598512    0.373534
  526      16     22311     22295   169.087       204     0.13551    0.373386
  527      16     22355     22339     169.1       176     1.12599    0.373324
  528      16     22398     22382   169.105       172    0.103698    0.373311
  529      16     22436     22420   169.072       152   0.0599454    0.373317
  530      16     22471     22455   169.018       140    0.652483    0.373402
  531      16     22516     22500   169.038       180   0.0859822    0.373477
  532      15     22553     22538   169.005       152     0.12833    0.373531
  533      16     22590     22574   168.957       144    0.270461    0.373631
  534      16     22636     22620   168.985       184    0.101993    0.373594
  535      16     22676     22660   168.968       160    0.720174    0.373609
  536      16     22711     22695   168.914       140   0.0571462    0.373761
  537      16     22751     22735   168.897       160   0.0430687     0.37376
  538      16     22788     22772   168.858       148   0.0985609    0.373844
  539      16     22837     22821   168.907       196   0.0666636    0.373742
2020-06-08 12:30:00.866806 min lat: 0.0230414 max lat: 2.43145 avg lat: 0.373761
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      15     22880     22865   168.911       176    0.728356    0.373761
  541      16     22931     22915   168.969       200    0.425171    0.373605
  542      16     22968     22952    168.93       148     1.00427    0.373727
  543      15     23012     22997    168.95       180    0.623734    0.373699
  544      16     23059     23043   168.977       184    0.559206    0.373701
  545      16     23105     23089   169.005       184   0.0540311    0.373539
  546      16     23140     23124   168.952       140    0.406238    0.373631
  547      16     23187     23171   168.978       188    0.208259    0.373622
  548      16     23225     23209   168.947       152    0.310856    0.373648
  549      16     23267     23251   168.945       168   0.0596425    0.373595
  550      16     23314     23298    168.98       188    0.360297    0.373586
  551      15     23364     23349   169.041       204    0.394656    0.373417
  552      16     23412     23396   169.075       188    0.392099    0.373443
  553      16     23449     23433   169.037       148    0.316882    0.373461
  554      16     23487     23471   169.003       152     1.37468    0.373478
  555      16     23526     23510   168.979       156    0.952755    0.373527
  556      16     23570     23554   168.992       176    0.699079    0.373596
  557      16     23617     23601   169.026       188    0.105293    0.373513
  558      16     23655     23639   168.995       152    0.046516    0.373511
  559      16     23691     23675   168.951       144    0.490041    0.373625
2020-06-08 12:30:20.919319 min lat: 0.0230414 max lat: 2.43145 avg lat: 0.373597
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     23736     23720    168.97       180     1.10394    0.373597
  561      16     23783     23767   169.004       188    0.100392    0.373506
  562      16     23825     23809   169.002       168    0.149469    0.373554
  563      16     23871     23855   169.029       184   0.0914753    0.373522
  564      16     23915     23899   169.041       176    0.854179    0.373463
  565      15     23952     23937   169.011       152    0.581261    0.373546
  566      16     23997     23981   169.023       176    0.157163    0.373489
  567      16     24039     24023   169.021       168   0.0542004    0.373521
  568      15     24082     24067   169.031       176    0.350725    0.373455
  569      16     24117     24101   168.973       136    0.544972    0.373547
  570      15     24161     24146   168.992       180   0.0691027    0.373577
  571      15     24203     24188    168.99       168    0.365866    0.373523
Total time run:       573.097
Total reads made:     24206
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   168.949
Average IOPS:         42
Stddev IOPS:          4.60581
Max IOPS:             56
Min IOPS:             22
Average Latency(s):   0.373816
Max latency(s):       2.43145
Min latency(s):       0.0230414

Total time run:       573.097
Total reads made:     24206
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   168.949
Average IOPS:         42
Stddev IOPS:          4.60581
Max IOPS:             56
Min IOPS:             22
Average Latency(s):   0.373816
Max latency(s):       2.43145
Min latency(s):       0.0230414

Random Read Test, pool with 16 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        54        38   151.941       152   0.0560878    0.301558
    2      15        93        78   155.961       160   0.0508687    0.334311
    3      16       139       123   163.965       180   0.0970996    0.344061
    4      16       182       166   165.969       172   0.0827236    0.351308
    5      16       228       212    169.57       184    0.104305    0.347256
    6      16       265       249   165.922       148    0.793498    0.354993
    7      16       310       294   167.929       180    0.877642    0.361335
    8      16       347       331   165.436       148    0.606384      0.3668
    9      16       395       379   168.384       192   0.0695003    0.359143
   10      16       440       424   169.542       180    0.127387    0.352231
   11      16       486       470   170.854       184    0.208886    0.364528
   12      16       532       516   171.947       184    0.100045    0.358566
   13      16       579       563    173.18       188    0.707554    0.359704
   14      15       620       605   172.809       168    0.294718       0.358
   15      15       664       649    173.02       176   0.0703784    0.354072
   16      16       710       694   173.455       180   0.0691815    0.355399
   17      16       749       733   172.427       156    0.631627     0.35975
   18      16       787       771   171.291       152    0.102011    0.362699
   19      16       831       815   171.538       176    0.194053    0.363742
2020-06-08 12:34:24.728113 min lat: 0.00855444 max lat: 1.74713 avg lat: 0.362494
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       877       861    172.16       184    0.778364    0.362494
   21      16       921       905   172.337       176   0.0567624    0.362528
   22      16       974       958   174.132       212    0.153594    0.359249
   23      16      1016      1000   173.856       168    0.256497    0.358638
   24      16      1054      1038   172.945       152    0.212232    0.361255
   25      16      1093      1077   172.266       156   0.0840251    0.361163
   26      15      1134      1119   172.101       168    0.410222    0.362375
   27      16      1175      1159   171.652       160    0.886293    0.362714
   28      16      1213      1197   170.949       152     0.15387     0.36495
   29      16      1261      1245   171.674       192    0.656081    0.364763
   30      16      1298      1282   170.885       148     1.39393    0.364236
   31      16      1348      1332   171.823       200    0.385381    0.362314
   32      16      1385      1369   171.078       148   0.0715221    0.365919
   33      16      1430      1414   171.345       180    0.216946    0.365676
   34      16      1471      1455   171.126       164   0.0916466    0.365462
   35      16      1513      1497   171.035       168    0.170297    0.365562
   36      15      1553      1538   170.839       164    0.568043    0.367149
   37      16      1595      1579   170.618       164    0.203232    0.367024
   38      16      1637      1621   170.549       168     0.10333    0.366906
   39      15      1673      1658   169.966       148     1.01188    0.368589
2020-06-08 12:34:44.743051 min lat: 0.00855444 max lat: 1.74713 avg lat: 0.369317
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      15      1717      1702   170.117       176    0.237889    0.369317
   41      15      1761      1746   170.258       176     0.21963    0.368777
   42      16      1807      1791   170.484       180    0.716421    0.368889
   43      16      1851      1835   170.611       176    0.492626    0.368246
   44      16      1893      1877    170.55       168    0.868193    0.368495
   45      16      1933      1917   170.315       160    0.658381    0.368697
   46      16      1971      1955   169.916       152    0.884089    0.369789
   47      16      2016      2000    170.13       180   0.0755125    0.368881
   48      16      2060      2044   170.251       176    0.105925     0.36842
   49      15      2105      2090   170.531       184      1.4859    0.368157
   50      15      2146      2131   170.397       164    0.295478    0.368033
   51      16      2191      2175   170.507       176     0.13361    0.368038
   52      16      2236      2220   170.689       180   0.0560121     0.36767
   53      16      2281      2265   170.859       180   0.0925873    0.368093
   54      16      2324      2308   170.879       172    0.165669    0.367542
   55      16      2374      2358   171.408       200     1.51535    0.367335
   56      15      2414      2399   171.276       164    0.437569    0.366908
   57      15      2454      2439   171.078       160    0.682976    0.367406
   58      16      2499      2483    171.16       176    0.475668    0.367909
   59      16      2545      2529   171.377       184    0.390914    0.367125
2020-06-08 12:35:04.751358 min lat: 0.00855444 max lat: 1.78207 avg lat: 0.366872
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2589      2573   171.454       176    0.172929    0.366872
   61      16      2629      2613   171.265       160   0.0765162    0.366603
   62      16      2670      2654   171.148       164     0.21768    0.366755
   63      15      2707      2692   170.844       152     1.07602     0.36782
   64      16      2748      2732   170.674       160    0.288796    0.368158
   65      16      2785      2769   170.325       148    0.392669    0.368874
   66      16      2828      2812    170.35       172     0.14012    0.369346
   67      16      2874      2858   170.553       184   0.0623601    0.368788
   68      16      2923      2907   170.927       196    0.078374    0.368413
   69      15      2966      2951       171       176   0.0939366    0.367736
   70      15      3004      2989   170.728       152    0.785511    0.368216
   71      16      3050      3034   170.858       180    0.322062     0.36865
   72      15      3088      3073   170.647       156   0.0633324    0.368374
   73      16      3130      3114   170.556       164    0.318614    0.368543
   74      16      3175      3159   170.683       180   0.0571033    0.368064
   75      15      3215      3200   170.594       164    0.450791    0.368223
   76      16      3264      3248   170.873       192     0.28818    0.368458
   77      16      3311      3295   171.093       188    0.539617    0.368093
   78      16      3352      3336   171.002       164    0.943884    0.367812
   79      16      3392      3376   170.862       160   0.0803677    0.367359
2020-06-08 12:35:24.757886 min lat: 0.00855444 max lat: 1.78207 avg lat: 0.367158
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      3438      3422   171.026       184    0.153218    0.367158
   81      16      3489      3473   171.433       204   0.0991558    0.367021
   82      16      3537      3521   171.683       192    0.417421    0.367092
   83      15      3582      3567   171.831       184     0.13382    0.366716
   84      15      3618      3603   171.498       144    0.211952    0.367122
   85      16      3663      3647   171.551       176    0.190954    0.366645
   86      16      3701      3685   171.323       152     1.22431    0.367533
   87      15      3747      3732   171.514       188    0.326937    0.367436
   88      16      3795      3779   171.701       188    0.242663    0.366849
   89      15      3839      3824   171.794       180    0.822329    0.366739
   90      16      3878      3862   171.574       152    0.225071    0.366788
   91      16      3919      3903   171.491       164     1.09846     0.36668
   92      16      3959      3943   171.366       160    0.220888    0.367126
   93      16      4003      3987   171.415       176    0.635968    0.367697
   94      16      4051      4035   171.634       192     0.22924    0.367152
   95      16      4093      4077   171.595       168    0.228047    0.367052
   96      15      4138      4123   171.724       184    0.151202    0.366646
   97      15      4177      4162   171.562       156    0.393589    0.366684
   98      15      4221      4206   171.607       176    0.606391    0.367675
   99      15      4268      4253   171.709       188    0.531056    0.367072
2020-06-08 12:35:44.798371 min lat: 0.00855444 max lat: 1.84405 avg lat: 0.366413
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      4312      4296   171.711       172      0.1059    0.366413
  101      16      4356      4340   171.751       176     1.02397    0.366625
  102      16      4406      4390   172.028       200    0.160712    0.366399
  103      16      4444      4428   171.826       152      1.1547    0.366429
  104      16      4485      4469   171.751       164    0.101778    0.366449
  105      16      4530      4514   171.829       180    0.280699    0.366633
  106      15      4572      4557    171.83       172    0.282829    0.366736
  107      15      4617      4602   171.906       180    0.706571    0.366768
  108      15      4668      4653   172.201       204    0.105208    0.366122
  109      15      4706      4691   172.013       152    0.210905     0.36567
  110      16      4742      4726   171.722       140    0.129857    0.366329
  111      16      4785      4769   171.722       172    0.228014    0.366765
  112      16      4831      4815    171.83       184    0.146259    0.366835
  113      15      4875      4860   171.898       180    0.235777    0.366896
  114      16      4916      4900   171.794       160    0.706492    0.366817
  115      16      4960      4944    171.83       176    0.580098    0.366866
  116      16      5005      4989     171.9       180    0.151648    0.366682
  117      16      5051      5035   172.003       184    0.158664    0.366572
  118      16      5091      5075   171.901       160    0.198732    0.366619
  119      16      5129      5113   171.734       152    0.659346    0.366775
2020-06-08 12:36:04.814978 min lat: 0.00855444 max lat: 1.84405 avg lat: 0.367176
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      5167      5151   171.569       152    0.199474    0.367176
  121      15      5208      5193   171.539       168    0.466834    0.367381
  122      16      5251      5235    171.51       168    0.599882    0.367319
  123      15      5291      5276   171.449       164    0.102313    0.367412
  124      16      5332      5316   171.357       160   0.0881955    0.367465
  125      16      5370      5354   171.202       152    0.101228    0.367561
  126      16      5412      5396   171.176       168    0.348982    0.367732
  127      16      5449      5433   170.993       148   0.0733555     0.36795
  128      15      5491      5476   170.997       172    0.516966      0.3681
  129      15      5530      5515   170.881       156   0.0965617    0.368581
  130      15      5570      5555   170.797       160    0.133512    0.368795
  131      16      5607      5591   170.592       144   0.0667129    0.369345
  132      16      5641      5625    170.33       136    0.111097    0.370059
  133      15      5675      5660   170.102       140   0.0685105    0.370463
  134      16      5720      5704   170.146       176   0.0961603    0.370605
  135      16      5767      5751   170.278       188    0.660571    0.370415
  136      15      5813      5798   170.408       188      0.4856    0.370251
  137      15      5854      5839   170.361       164   0.0978587    0.370041
  138      16      5892      5876   170.199       148   0.0902359    0.370212
  139      16      5933      5917   170.154       164    0.188452    0.370492
2020-06-08 12:36:24.821230 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.370804
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      5974      5958    170.11       164    0.503391    0.370804
  141      16      6017      6001   170.123       172    0.322609    0.370546
  142      16      6059      6043   170.108       168    0.136321    0.370721
  143      16      6110      6094   170.345       204     0.49758      0.3703
  144      16      6156      6140   170.439       184    0.105002    0.369923
  145      15      6203      6188   170.588       192   0.0709188    0.369495
  146      16      6249      6233   170.652       180   0.0909963    0.369342
  147      15      6293      6278   170.715       180    0.104871    0.369187
  148      16      6333      6317   170.616       156   0.0888819    0.369408
  149      15      6376      6361   170.652       176    0.357895    0.369484
  150      16      6422      6406   170.714       180   0.0842686    0.369438
  151      16      6468      6452   170.801       184    0.088993    0.369056
  152      16      6514      6498   170.888       184    0.106771    0.368836
  153      16      6552      6536   170.764       152    0.274565    0.368838
  154      16      6595      6579    170.77       172   0.0927322     0.36896
  155      16      6634      6618   170.675       156   0.0970653     0.36905
  156      16      6673      6657   170.581       156   0.0825291    0.369279
  157      16      6716      6700    170.59       172    0.281331    0.369147
  158      16      6758      6742   170.573       168    0.178415     0.36942
  159      15      6798      6783   170.532       164   0.0685451    0.369779
2020-06-08 12:36:44.826489 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369803
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      6837      6821   170.415       152     1.03978    0.369803
  161      16      6877      6861    170.35       160    0.299515    0.370076
  162      15      6921      6906    170.41       180    0.479073    0.370047
  163      16      6955      6939   170.174       132    0.970325    0.370485
  164      16      6991      6975   170.015       144     1.25103     0.37083
  165      15      7038      7023   170.147       192    0.412588     0.37058
  166      16      7084      7068   170.206       180    0.166015    0.370487
  167      16      7136      7120   170.432       208   0.0960646    0.370014
  168      16      7176      7160    170.37       160     1.20631    0.370032
  169      15      7217      7202   170.355       168   0.0725855    0.370052
  170      16      7261      7245   170.365       172   0.0880765    0.369988
  171      16      7301      7285   170.304       160   0.0939977    0.370118
  172      16      7338      7322   170.175       148   0.0937779    0.370366
  173      16      7373      7357       170       140    0.285505    0.370597
  174      16      7414      7398   169.964       164    0.509076    0.370912
  175      16      7462      7446    170.09       192    0.123406    0.370835
  176      16      7504      7488   170.077       168   0.0673462    0.370791
  177      15      7537      7522   169.885       136    0.359524    0.370841
  178      16      7582      7566   169.916       176   0.0573193    0.371038
  179      15      7617      7602    169.77       144    0.161006    0.371041
2020-06-08 12:37:04.836065 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.371552
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      7653      7637   169.605       140     1.48986    0.371552
  181      16      7689      7673   169.463       144    0.165271    0.371909
  182      16      7736      7720   169.565       188    0.087052    0.371983
  183      16      7779      7763   169.578       172    0.755047    0.372088
  184      16      7833      7817   169.829       216    0.175646    0.371405
  185      16      7881      7865   169.949       192   0.0892331     0.37119
  186      16      7921      7905   169.896       160    0.299585    0.371403
  187      16      7967      7951   169.971       184   0.0684797    0.371202
  188      16      8009      7993    169.96       168    0.292215    0.371086
  189      16      8053      8037   169.992       176    0.614826    0.371233
  190      15      8086      8071   169.813       136   0.0720735     0.37129
  191      16      8126      8110   169.741       156    0.320248    0.371545
  192      16      8166      8150    169.69       160    0.549814    0.371717
  193      15      8211      8196   169.764       184    0.425214    0.371778
  194      16      8254      8238   169.755       168   0.0870834    0.371445
  195      15      8300      8285   169.848       188    0.337432    0.371548
  196      16      8346      8330     169.9       180     0.19846    0.370976
  197      16      8384      8368   169.809       152     1.16759    0.371254
  198      16      8425      8409    169.77       164    0.328002    0.371196
  199      16      8471      8455   169.842       184       0.522    0.371561
2020-06-08 12:37:24.851318 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.371114
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      15      8515      8500   169.891       180    0.369488    0.371114
  201      16      8556      8540   169.841       160    0.229863    0.371443
  202      16      8599      8583   169.852       172    0.378308    0.371379
  203      16      8648      8632    169.98       196    0.321516    0.371269
  204      16      8692      8676    170.01       176    0.106048    0.371112
  205      16      8731      8715   169.941       156    0.821938    0.371155
  206      16      8775      8759    169.97       176    0.128704    0.370892
  207      16      8820      8804   170.018       180     1.33029    0.371066
  208      16      8861      8845   169.986       164    0.706841    0.371111
  209      16      8909      8893   170.092       192    0.500026    0.370966
  210      15      8947      8932   170.024       156    0.460491    0.370982
  211      16      8992      8976   170.051       176   0.0817317    0.370814
  212      16      9034      9018   170.041       168    0.121517    0.370766
  213      16      9072      9056   169.956       152   0.0494781    0.370849
  214      16      9116      9100   169.984       176    0.337138    0.370895
  215      16      9155      9139   169.919       156    0.250411    0.371265
  216      16      9198      9182   169.928       172   0.0511419    0.370894
  217      16      9245      9229   170.011       188   0.0505669    0.371062
  218      16      9289      9273   170.038       176    0.241485    0.371116
  219      16      9332      9316   170.047       172    0.420921    0.371013
2020-06-08 12:37:44.869174 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.370747
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      15      9380      9365    170.16       196    0.123874    0.370747
  221      16      9430      9414   170.269       196    0.727918    0.370537
  222      16      9478      9462   170.353       192    0.151264    0.370298
  223      16      9517      9501   170.288       156     1.33769     0.37031
  224      16      9561      9545   170.314       176   0.0647897     0.37037
  225      16      9613      9597   170.481       208   0.0624257     0.36997
  226      16      9658      9642   170.522       180    0.517904    0.370019
  227      16      9697      9681   170.458       156   0.0834134    0.370062
  228      16      9738      9722    170.43       164     0.33671    0.370219
  229      15      9785      9770   170.524       192    0.589027     0.37014
  230      15      9828      9813    170.53       172   0.0972365    0.370032
  231      15      9876      9861   170.623       192    0.161094    0.369796
  232      16      9918      9902   170.594       164    0.934149    0.369721
  233      15      9957      9942   170.546       160    0.537636    0.369896
  234      16     10003      9987   170.587       180    0.842218    0.369962
  235      16     10042     10026   170.524       156    0.136153     0.36994
  236      16     10085     10069   170.531       172    0.110852     0.36995
  237      15     10127     10112   170.537       172    0.585616     0.36997
  238      16     10168     10152   170.492       160    0.165433    0.370122
  239      16     10215     10199   170.565       188    0.491289    0.369969
2020-06-08 12:38:04.905117 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369971
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      15     10252     10237   170.488       152    0.450304    0.369971
  241      15     10296     10281   170.509       176    0.074617    0.370017
  242      16     10340     10324   170.514       172   0.0748047    0.369973
  243      15     10384     10369   170.553       180    0.190594     0.36987
  244      16     10435     10419    170.67       200    0.687906    0.369674
  245      15     10486     10471   170.821       208    0.346122    0.369289
  246      16     10518     10502    170.63       124   0.0958972    0.369579
  247      16     10561     10545   170.635       172    0.477452    0.369753
  248      16     10600     10584   170.575       156    0.149365    0.369825
  249      15     10647     10632    170.66       192    0.587592    0.369778
  250      15     10691     10676   170.681       176   0.0887679    0.369776
  251      16     10738     10722   170.734       184     0.43434    0.369488
  252      15     10775     10760   170.659       152    0.289411    0.369328
  253      16     10821     10805   170.695       180     1.10484    0.369712
  254      15     10864     10849    170.71       176    0.614384     0.36973
  255      16     10906     10890   170.684       164    0.360487    0.369725
  256      16     10946     10930   170.642       160    0.691751    0.369749
  257      16     10990     10974   170.663       176    0.102485    0.369666
  258      16     11033     11017   170.666       172    0.078748    0.369672
  259      16     11077     11061   170.687       176    0.205638    0.369744
2020-06-08 12:38:24.935612 min lat: 0.00855444 max lat: 2.04466 avg lat: 0.369638
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16     11118     11102   170.661       164    0.280352    0.369638
  261      15     11157     11142    170.62       160     1.23471    0.369661
  262      16     11199     11183   170.594       164   0.0719021    0.369878
  263      16     11238     11222   170.538       156   0.0719583    0.369864
  264      16     11285     11269   170.604       188    0.372989    0.369713
  265      16     11329     11313   170.624       176    0.205712    0.369717
  266      16     11377     11361   170.704       192    0.217287    0.369723
  267      16     11425     11409   170.783       192    0.104386    0.369514
  268      16     11468     11452   170.785       172   0.0754446    0.369326
  269      16     11509     11493    170.76       164    0.201825    0.369348
  270      16     11547     11531    170.69       152     1.57529     0.36947
  271      16     11597     11581   170.798       200    0.405929    0.369466
  272      15     11645     11630    170.89       196   0.0776648    0.369341
  273      16     11691     11675   170.923       180   0.0870909    0.369237
  274      15     11733     11718   170.926       172    0.103877    0.369162
  275      16     11778     11762   170.944       176    0.340553    0.369209
  276      16     11825     11809   171.005       188    0.498878    0.369047
  277      15     11873     11858   171.095       196    0.389793    0.368885
  278      16     11910     11894   170.998       144    0.184223    0.369059
  279      16     11952     11936   170.987       168    0.880617    0.368982
2020-06-08 12:38:44.950293 min lat: 0.00803674 max lat: 2.04466 avg lat: 0.368991
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16     11997     11981   171.019       180    0.124486    0.368991
  281      16     12048     12032   171.136       204    0.241506    0.368816
  282      15     12087     12072   171.096       160    0.268891     0.36891
  283      16     12129     12113    171.07       164     1.00581    0.368959
  284      16     12173     12157   171.087       176    0.531792     0.36892
  285      16     12212     12196   171.034       156     1.16607    0.368898
  286      16     12254     12238   171.024       168   0.0710417    0.368941
  287      16     12295     12279   170.999       164   0.0871388    0.368994
  288      16     12338     12322   171.003       172    0.591573    0.369082
  289      16     12375     12359   170.923       148    0.495751     0.36923
  290      15     12417     12402   170.926       172   0.0411501     0.36914
  291      16     12460     12444   170.916       168    0.101774    0.369164
  292      16     12498     12482   170.851       152    0.199923    0.369323
  293      16     12546     12530   170.923       192    0.443601    0.369198
  294      16     12585     12569   170.872       156     0.32418    0.369049
  295      16     12625     12609   170.835       160    0.205517    0.369366
  296      16     12671     12655    170.88       184    0.147168    0.369341
  297      16     12722     12706   170.991       204    0.120022    0.368844
  298      16     12768     12752   171.034       184     0.97924    0.368894
  299      15     12812     12797   171.064       180    0.116044    0.368897
2020-06-08 12:39:04.956275 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.36888
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     12856     12840   171.067       172    0.519146     0.36888
  301      16     12901     12885   171.097       180    0.402145    0.368828
  302      16     12943     12927   171.086       168   0.0975264    0.368752
  303      16     12990     12974   171.142       188   0.0440586    0.368661
  304      16     13036     13020   171.184       184    0.186594     0.36858
  305      15     13081     13066   171.226       184    0.816621     0.36862
  306      16     13129     13113   171.281       188   0.0939368    0.368523
  307      16     13167     13151   171.218       152    0.071984    0.368553
  308      16     13205     13189   171.156       152   0.0115672    0.368622
  309      16     13242     13226   171.081       148   0.0681405     0.36879
  310      16     13285     13269   171.084       172    0.296127    0.368752
  311      15     13328     13313   171.099       176     1.01723    0.368844
  312      16     13367     13351   171.038       152     0.12673    0.368946
  313      15     13406     13391   171.003       160    0.544955     0.36898
  314      15     13449     13434   171.006       172    0.120477    0.368944
  315      16     13497     13481    171.06       188    0.119845    0.368942
  316      16     13541     13525   171.075       176   0.0955662    0.368919
  317      16     13586     13570   171.103       180    0.828479    0.368862
  318      16     13634     13618   171.169       192    0.135091    0.368797
  319      16     13674     13658   171.134       160    0.261829    0.368743
2020-06-08 12:39:24.959038 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368694
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     13724     13708   171.224       200    0.443875    0.368694
  321      15     13763     13748   171.189       160    0.318633    0.368698
  322      16     13807     13791   171.191       172   0.0734651    0.368665
  323      16     13854     13838   171.243       188    0.329117    0.368621
  324      16     13900     13884   171.282       184    0.346575    0.368538
  325      15     13941     13926   171.272       168     0.10977    0.368579
  326      16     13984     13968   171.262       168    0.461086    0.368565
  327      16     14033     14017   171.337       196    0.617285    0.368431
  328      16     14081     14065     171.4       192    0.234995    0.368327
  329      15     14122     14107   171.371       168    0.261684    0.368319
  330      16     14167     14151   171.385       176    0.264592    0.368219
  331      16     14208     14192   171.362       164    0.326276    0.368272
  332      16     14254     14238   171.399       184    0.106693    0.368265
  333      16     14298     14282   171.413       176    0.526763    0.368131
  334      16     14337     14321   171.366       156    0.121332    0.368159
  335      16     14375     14359   171.307       152    0.652309    0.368182
  336      16     14414     14398   171.261       156    0.457002    0.368495
  337      16     14458     14442   171.274       176   0.0491926    0.368411
  338      16     14503     14487   171.299       180   0.0676485    0.368358
  339      16     14539     14523   171.219       144   0.0824609    0.368449
2020-06-08 12:39:45.008426 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368397
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     14579     14563   171.186       160   0.0587123    0.368397
  341      16     14620     14604   171.165       164    0.164267    0.368729
  342      16     14664     14648   171.178       176     1.46672    0.368628
  343      15     14701     14686   171.122       152     0.19675    0.368726
  344      16     14744     14728   171.113       168    0.579119    0.368835
  345      16     14786     14770   171.103       168   0.0777964    0.368897
  346      16     14835     14819   171.175       196    0.145912    0.368732
  347      16     14882     14866   171.222       188    0.108431    0.368658
  348      16     14924     14908   171.213       168    0.344849    0.368635
  349      16     14968     14952   171.226       176    0.170138    0.368617
  350      16     15013     14997    171.25       180    0.577516      0.3686
  351      16     15047     15031   171.148       136    0.279342    0.368771
  352      15     15088     15073   171.139       168    0.216839    0.368739
  353      16     15129     15113   171.107       160     0.26602    0.368819
  354      16     15169     15153   171.076       160   0.0670473    0.368949
  355      16     15206     15190   171.011       148     0.10684    0.369033
  356      16     15250     15234   171.025       176   0.0900062    0.369047
  357      16     15293     15277   171.026       172    0.496779    0.369027
  358      16     15338     15322   171.051       180   0.0877171     0.36904
  359      16     15375     15359   170.987       148    0.081768    0.369064
2020-06-08 12:40:05.025289 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.369087
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     15419     15403   171.001       176     1.18147    0.369087
  361      16     15454     15438   170.915       140    0.304616    0.369171
  362      16     15494     15478   170.885       160    0.085328    0.369151
  363      16     15543     15527   170.953       196    0.156983    0.369239
  364      15     15582     15567   170.923       160    0.647775    0.369236
  365      16     15627     15611   170.937       176    0.085923    0.369237
  366      16     15674     15658   170.983       188    0.112202    0.369109
  367      16     15727     15711   171.095       212     0.12493    0.368917
  368      16     15773     15757    171.13       184    0.200318    0.368847
  369      15     15818     15803   171.165       184    0.194294    0.368766
  370      16     15862     15846   171.167       172     0.18507    0.368782
  371      16     15910     15894   171.223       192    0.510724    0.368781
  372      16     15951     15935   171.204       164    0.321037    0.368631
  373      16     15991     15975   171.174       160     0.04023      0.3685
  374      16     16031     16015   171.144       160   0.0743827    0.368522
  375      16     16071     16055   171.114       160    0.100989    0.368641
  376      16     16111     16095   171.084       160   0.0698934    0.368633
  377      16     16148     16132   171.023       148     1.56425    0.368877
  378      16     16193     16177   171.047       180    0.076014    0.369106
  379      15     16229     16214   170.986       148    0.104242    0.369005
2020-06-08 12:40:25.030381 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.369126
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      15     16273     16258   170.999       176    0.266939    0.369126
  381      16     16316     16300   170.991       168    0.651709    0.369243
  382      15     16366     16351   171.077       204    0.244598    0.369095
  383      15     16413     16398   171.121       188    0.661911    0.368918
  384      16     16453     16437   171.082       156      0.1239    0.368912
  385      15     16493     16478   171.062       164    0.205343    0.369012
  386      16     16537     16521   171.065       172     0.92265     0.36905
  387      16     16578     16562   171.046       164   0.0943781    0.369027
  388      15     16629     16614   171.141       208    0.288907    0.368991
  389      16     16671     16655   171.123       164    0.283482    0.368878
  390      16     16714     16698   171.125       172    0.846183    0.369016
  391      16     16751     16735   171.066       148   0.0405144    0.368932
  392      15     16794     16779   171.078       176   0.0665434    0.368977
  393      16     16837     16821   171.069       168    0.109843    0.368986
  394      16     16882     16866   171.091       180    0.274312    0.369024
  395      16     16929     16913   171.133       188    0.813605    0.368956
  396      16     16974     16958   171.155       180    0.196726    0.368841
  397      15     17017     17002   171.167       176    0.828185    0.368902
  398      16     17066     17050   171.219       192    0.621207    0.368727
  399      16     17109     17093   171.221       172    0.264891    0.368679
2020-06-08 12:40:45.044451 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368777
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      15     17147     17132   171.183       156   0.0894283    0.368777
  401      16     17193     17177   171.205       180    0.632196     0.36874
  402      16     17240     17224   171.246       188     0.48601    0.368732
  403      16     17284     17268   171.258       176    0.303091    0.368619
  404      16     17327     17311   171.259       172     1.13456    0.368504
  405      16     17371     17355   171.271       176    0.196824     0.36845
  406      15     17415     17400   171.292       180    0.484843    0.368536
  407      16     17461     17445   171.314       180   0.0772191    0.368493
  408      16     17505     17489   171.325       176     1.28317    0.368537
  409      16     17548     17532   171.326       172     0.45095    0.368494
  410      16     17588     17572   171.298       160    0.929039    0.368493
  411      16     17632     17616    171.31       176     1.12283    0.368465
  412      16     17671     17655   171.273       156   0.0630831    0.368423
  413      16     17717     17701   171.303       184    0.311614    0.368483
  414      16     17760     17744   171.305       172     1.12869    0.368439
  415      15     17809     17794   171.374       200    0.202476    0.368366
  416      15     17854     17839   171.395       180     1.43189    0.368289
  417      16     17895     17879   171.367       160    0.071285    0.368411
  418      16     17942     17926   171.407       188    0.334598    0.368354
  419      16     17991     17975   171.465       196    0.238768    0.368236
2020-06-08 12:41:05.050839 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368204
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     18034     18018   171.466       172    0.986549    0.368204
  421      15     18078     18063   171.487       180     1.46089    0.368135
  422      16     18127     18111   171.535       192    0.386874    0.367998
  423      15     18165     18150   171.498       156    0.300668    0.368097
  424      16     18211     18195   171.518       180    0.351673    0.368059
  425      16     18254     18238   171.519       172    0.738602    0.368047
  426      16     18298     18282   171.529       176    0.100675    0.368054
  427      16     18341     18325   171.529       172    0.291243    0.368056
  428      16     18381     18365   171.502       160    0.705053    0.368095
  429      16     18426     18410   171.522       180    0.238712    0.368041
  430      15     18468     18453   171.523       172    0.479224    0.368042
  431      16     18513     18497   171.533       176    0.121838    0.367902
  432      15     18555     18540   171.534       172    0.143614    0.367975
  433      16     18602     18586   171.562       184    0.130385    0.368027
  434      16     18643     18627   171.545       164    0.340189    0.368091
  435      16     18687     18671   171.554       176    0.651568    0.368042
  436      15     18728     18713   171.546       168     0.24549    0.368076
  437      16     18778     18762   171.602       196    0.132161    0.367919
  438      16     18819     18803   171.584       164   0.0591813    0.367831
  439      16     18860     18844   171.566       164    0.297181    0.367922
2020-06-08 12:41:25.064246 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368035
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     18902     18886   171.558       168    0.743189    0.368035
  441      16     18948     18932   171.586       184    0.219826    0.368001
  442      16     18995     18979   171.623       188     0.47599    0.367938
  443      16     19040     19024   171.641       180    0.535307    0.367893
  444      16     19082     19066   171.633       168    0.685933    0.367873
  445      16     19130     19114   171.677       192   0.0607425    0.367814
  446      16     19178     19162   171.712       192    0.256125    0.367747
  447      16     19220     19204   171.704       168    0.723128    0.367719
  448      15     19253     19238   171.624       136   0.0871409    0.367759
  449      16     19299     19283   171.643       180   0.0977956    0.367739
  450      15     19345     19330   171.679       188    0.350577    0.367832
  451      16     19393     19377   171.715       188    0.443752    0.367754
  452      16     19440     19424   171.751       188    0.055133    0.367597
  453      15     19484     19469   171.769       180    0.332569    0.367572
  454      16     19526     19510   171.752       164    0.336732    0.367549
  455      16     19567     19551   171.734       164   0.0777378    0.367588
  456      15     19610     19595   171.744       176    0.684707    0.367607
  457      16     19653     19637   171.735       168    0.595498    0.367618
  458      16     19698     19682   171.752       180    0.546961    0.367603
  459      16     19742     19726    171.76       176    0.154392    0.367519
2020-06-08 12:41:45.107477 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.367502
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     19788     19772   171.787       184   0.0965505    0.367502
  461      15     19830     19815   171.787       172     0.69698    0.367524
  462      16     19878     19862   171.822       188    0.293698    0.367446
  463      16     19926     19910   171.866       192    0.721074    0.367413
  464      16     19971     19955   171.883       180   0.0823879    0.367297
  465      16     20012     19996   171.866       164    0.193197     0.36731
  466      16     20051     20035   171.831       156    0.427847    0.367267
  467      16     20092     20076   171.814       164    0.334002    0.367453
  468      16     20133     20117   171.797       164   0.0732855    0.367419
  469      16     20177     20161   171.806       176    0.250304    0.367468
  470      16     20214     20198   171.755       148   0.0578775    0.367529
  471      16     20252     20236   171.713       152   0.0606962    0.367566
  472      15     20296     20281   171.731       180   0.0706551    0.367523
  473      16     20340     20324   171.731       172   0.0995366    0.367534
  474      16     20377     20361   171.681       148    0.105159    0.367619
  475      16     20420     20404   171.682       172    0.193995    0.367719
  476      16     20462     20446   171.674       168    0.745976    0.367802
  477      16     20500     20484   171.633       152    0.100294    0.367803
  478      16     20541     20525   171.616       164     1.04113    0.367899
  479      16     20583     20567   171.609       168    0.300737    0.367753
2020-06-08 12:42:05.116361 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.367749
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     20626     20610    171.61       172   0.0737056    0.367749
  481      16     20671     20655   171.627       180    0.161571    0.367761
  482      16     20712     20696   171.611       164    0.250212    0.367869
  483      16     20754     20738   171.604       168   0.0801082    0.367947
  484      16     20797     20781   171.604       172    0.080388    0.367923
  485      16     20841     20825   171.613       176   0.0886142    0.367931
  486      16     20881     20865   171.589       160    0.354113    0.367922
  487      16     20923     20907   171.582       168    0.259697    0.367951
  488      16     20968     20952   171.599       180    0.312084    0.367889
  489      16     21013     20997   171.616       180   0.0996561    0.367802
  490      16     21053     21037   171.592       160   0.0765918    0.367855
  491      16     21098     21082    171.61       180    0.461261    0.367967
  492      16     21142     21126   171.618       176    0.298625    0.367939
  493      16     21182     21166   171.595       160   0.0972208    0.367882
  494      16     21223     21207   171.579       164    0.194985     0.36794
  495      16     21262     21246   171.548       156    0.100899    0.368001
  496      16     21300     21284   171.508       152    0.684771    0.368061
  497      16     21342     21326   171.501       168   0.0776132    0.368097
  498      16     21383     21367   171.486       164    0.971859    0.368162
  499      16     21428     21412   171.503       180   0.0725294    0.368133
2020-06-08 12:42:25.120038 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368206
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      15     21463     21448   171.448       144   0.0502814    0.368206
  501      16     21507     21491   171.449       172    0.283424    0.368146
  502      15     21543     21528   171.402       148    0.507856    0.368198
  503      15     21587     21572   171.411       176    0.950412    0.368274
  504      16     21627     21611   171.381       156     1.53861     0.36832
  505      16     21670     21654   171.382       172    0.138185    0.368333
  506      16     21711     21695   171.367       164   0.0794395    0.368323
  507      16     21751     21735   171.344       160    0.197224    0.368437
  508      15     21790     21775   171.322       160    0.285314    0.368379
  509      16     21833     21817   171.315       168    0.272573    0.368378
  510      16     21875     21859   171.309       168     1.69653    0.368503
  511      16     21920     21904   171.326       180     0.11077    0.368537
  512      16     21972     21956   171.397       208    0.278276     0.36838
  513      16     22016     22000   171.406       176    0.250833    0.368378
  514      15     22059     22044   171.414       176    0.304413    0.368413
  515      16     22102     22086   171.408       168   0.0509173    0.368366
  516      16     22138     22122   171.355       144   0.0969523     0.36837
  517      16     22180     22164   171.348       168     1.59564    0.368439
  518      16     22223     22207   171.349       172     0.71206     0.36854
  519      15     22266     22251   171.357       176    0.607223      0.3685
2020-06-08 12:42:45.129420 min lat: 0.00736974 max lat: 2.04466 avg lat: 0.368498
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     22308     22292   171.343       164    0.801967    0.368498
  521      15     22352     22337   171.357       180    0.313468    0.368539
  522      16     22401     22385   171.396       192    0.155938    0.368349
  523      16     22435     22419   171.328       136     1.59721    0.368336
  524      16     22464     22448   171.223       116     1.73026    0.368489
  525      16     22501     22485   171.178       148      1.8747    0.368573
  526      16     22543     22527   171.172       168     0.18821    0.368636
  527      16     22586     22570   171.173       172   0.0725106    0.368654
  528      16     22625     22609   171.145       156    0.444407    0.368929
  529      16     22672     22656   171.176       188   0.0658022    0.368885
  530      16     22713     22697   171.162       164   0.0653625    0.368855
  531      16     22751     22735   171.126       152     1.05794    0.368961
  532      15     22797     22782   171.158       188     1.01297    0.368923
  533      16     22843     22827   171.175       180    0.331542    0.368947
  534      16     22890     22874   171.206       188    0.112522    0.368779
  535      16     22934     22918   171.215       176    0.295483    0.368704
  536      15     22975     22960   171.209       168    0.330568    0.368747
  537      16     23013     22997   171.165       148    0.399563    0.368815
  538      16     23050     23034   171.122       148   0.0617131    0.369026
  539      16     23095     23079   171.139       180    0.224041    0.368934
2020-06-08 12:43:05.147687 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.368984
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      15     23133     23118    171.11       156    0.323369    0.368984
  541      16     23177     23161   171.112       172    0.256919    0.369023
  542      16     23222     23206   171.128       180   0.0765498    0.369039
  543      16     23259     23243   171.085       148    0.754259    0.369051
  544      15     23300     23285    171.08       168    0.323543    0.369055
  545      16     23347     23331   171.103       184    0.432705    0.369011
  546      15     23388     23373   171.098       168     1.00259    0.368974
  547      16     23428     23412    171.07       156     1.37922    0.369058
  548      16     23470     23454   171.064       168     0.91654    0.369091
  549      15     23506     23491   171.022       148    0.825622    0.369162
  550      16     23544     23528    170.98       148   0.0766564    0.369265
  551      15     23588     23573   170.996       180     1.33703    0.369289
  552      16     23632     23616   170.997       172    0.301987    0.369187
  553      16     23674     23658   170.992       168    0.108623    0.369226
  554      16     23713     23697   170.963       156    0.234901    0.369327
  555      16     23762     23746   171.009       196    0.238057    0.369289
  556      16     23793     23777   170.924       124   0.0873343     0.36924
  557      16     23829     23813   170.876       144     1.57494    0.369429
  558      16     23866     23850   170.835       148    0.182317     0.36952
  559      16     23914     23898   170.872       192    0.187263    0.369548
2020-06-08 12:43:25.160659 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369533
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     23952     23936   170.838       152    0.364742    0.369533
  561      16     23996     23980   170.847       176    0.214645     0.36954
  562      15     24035     24020   170.828       160    0.340947    0.369606
  563      16     24076     24060   170.808       160    0.654762    0.369707
  564      16     24120     24104   170.817       176    0.367582    0.369676
  565      16     24163     24147   170.819       172    0.387184    0.369667
  566      16     24204     24188   170.807       164    0.279247    0.369648
  567      16     24245     24229   170.795       164     1.05249     0.36973
  568      16     24288     24272   170.797       172    0.198967    0.369741
  569      16     24332     24316   170.806       176    0.136252    0.369715
  570      15     24366     24351   170.752       140   0.0584609    0.369714
  571      16     24409     24393   170.747       168    0.740716    0.369819
  572      16     24452     24436   170.749       172    0.101809    0.369823
  573      16     24496     24480   170.758       176   0.0578064    0.369746
  574      16     24537     24521   170.746       164    0.998901    0.369775
  575      16     24576     24560    170.72       156    0.207606     0.36985
  576      16     24618     24602   170.715       168    0.212732    0.369869
  577      16     24665     24649   170.745       188    0.949637    0.369771
  578      15     24712     24697   170.773       192    0.923029    0.369766
  579      16     24753     24737   170.755       160    0.651137    0.369768
2020-06-08 12:43:45.198519 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369777
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     24796     24780   170.757       172   0.0584573    0.369777
  581      15     24845     24830   170.806       200    0.364453    0.369687
  582      15     24891     24876   170.827       184    0.205766    0.369647
  583      16     24932     24916   170.809       160   0.0839604     0.36966
  584      16     24974     24958   170.804       168   0.0906737    0.369679
  585      15     25009     24994   170.758       144   0.0812008    0.369732
  586      15     25054     25039   170.774       180   0.0987196    0.369754
  587      15     25091     25076   170.735       148   0.0988818    0.369777
  588      15     25131     25116   170.716       160     0.14241     0.36976
  589      16     25178     25162   170.738       184   0.0643666    0.369812
  590      16     25221     25205   170.741       172     0.18708    0.369836
  591      16     25269     25253   170.776       192    0.743014    0.369722
  592      16     25314     25298   170.791       180    0.263785    0.369716
  593      16     25361     25345    170.82       188    0.376313    0.369688
  594      16     25401     25385   170.802       160    0.125557    0.369671
  595      16     25446     25430   170.817       180    0.087672    0.369677
  596      15     25495     25480   170.866       200    0.060978    0.369554
  597      16     25544     25528   170.902       192     0.22937    0.369446
  598      15     25578     25563    170.85       140     1.47206    0.369474
  599      16     25622     25606   170.852       172    0.964434    0.369568
2020-06-08 12:44:05.214016 min lat: 0.00736974 max lat: 2.12878 avg lat: 0.369611
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      15     25640     25625   170.694        76    0.988185    0.369611
Total time run:       600.544
Total reads made:     25640
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   170.779
Average IOPS:         42
Stddev IOPS:          3.97991
Max IOPS:             54
Min IOPS:             19
Average Latency(s):   0.369894
Max latency(s):       2.12878
Min latency(s):       0.00736974

Total time run:       600.544
Total reads made:     25640
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   170.779
Average IOPS:         42
Stddev IOPS:          3.97991
Max IOPS:             54
Min IOPS:             19
Average Latency(s):   0.369894
Max latency(s):       2.12878
Min latency(s):       0.00736974

Cleanup, pool with 16 PGs

podman exec ceph-mon-f5-422-01 rados -p testbench cleanup

Performance Baseline with rados bench, pool with 256 PGs

Since in an earlier blog post I tested with 8 and 256, I wanted to have at least one test run with the same number of PGs. Long term though I will align myself to the numbers suggested by Ceph Nautilus’ PG autoscaler.

I went on to create a single, replica 3, pool testbench2 with 256 PGs. I set the application type to rgw.

All rados bench tests were run on f5-422-01.

Sequential Write Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 write --no-cleanup | tee --append $(date +%F)-256PGs-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1349
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        36        20   79.8409        80    0.321672    0.474287
    2      16        53        37    73.919        68    0.623514    0.598543
    3      16        72        56   74.6075        76     2.21294    0.721355
    4      16        91        75   74.9514        76     0.85585    0.710295
    5      16       106        90   71.9595        60    0.656597    0.728313
    6      16       125       109   72.6089        76    0.290908    0.763383
    7      16       141       125   71.3779        64    0.901804    0.800385
    8      16       158       142   70.9531        68    0.518742    0.831229
    9      16       179       163   72.4008        84    0.704042    0.832311
   10      16       194       178    71.156        60     1.54382    0.839197
   11      16       210       194   70.5041        64     1.97506    0.854411
   12      16       232       216   71.9601        88    0.983213     0.84574
   13      16       252       236   72.5772        80      0.6913    0.842717
   14      16       271       255   72.8205        76    0.791158    0.840037
   15      16       291       275    73.298        80    0.812229    0.837232
   16      16       310       294   73.4662        76    0.921979    0.839894
   17      16       326       310   72.9056        64    0.529278    0.839648
   18      16       348       332   73.7432        88    0.598223    0.834106
   19      16       363       347   73.0193        60    0.667751    0.833339
2020-06-08 20:11:17.984346 min lat: 0.0802415 max lat: 3.10168 avg lat: 0.839968
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       378       362    72.368        60    0.450386    0.839968
   21      16       396       380   72.3482        72    0.139175     0.86046
   22      16       408       392   71.2415        48    0.418451     0.86067
   23      16       423       407   70.7523        60     2.20081    0.874599
   24      16       445       429   71.4697        88      1.2708    0.866653
   25      16       471       455   72.7696       104    0.285421    0.860298
   26      16       486       470   72.2781        60    0.389528    0.861504
   27      16       499       483   71.5249        52    0.410962    0.866005
   28      16       515       499   71.2558        64    0.429008    0.877484
   29      16       529       513   70.7292        56    0.285012    0.879612
   30      16       541       525   69.9714        48    0.457178    0.883154
   31      16       552       536   69.1307        44    0.545492    0.890488
   32      16       560       544   67.9705        32      2.8067    0.896042
   33      16       572       556   67.3653        48     3.29071    0.905777
   34      16       586       570   67.0308        56    0.294248    0.914702
   35      16       603       587   67.0582        68     1.75803    0.938793
   36      16       621       605   67.1951        72    0.532758    0.936639
   37      16       645       629    67.973        96    0.604943    0.925118
   38      16       667       651   68.4918        88    0.558411    0.924037
   39      16       678       662   67.8639        44       2.005    0.924458
2020-06-08 20:11:37.994972 min lat: 0.0802415 max lat: 4.29548 avg lat: 0.9253
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       697       681   68.0669        76     1.08916      0.9253
   41      16       719       703   68.5525        88    0.570717    0.922517
   42      16       737       721   68.6338        72     1.31116    0.922207
   43      16       757       741   68.8978        80    0.526462    0.916848
   44      16       774       758   68.8771        68     1.93632    0.916822
   45      16       797       781     69.39        92    0.134226     0.91297
   46      16       810       794   69.0119        52     1.02052    0.912372
   47      16       831       815   69.3304        84     0.15731    0.909389
   48      16       848       832   69.3021        68    0.360588    0.910789
   49      16       863       847   69.1121        60    0.659684    0.913781
   50      16       887       871   69.6491        96    0.133689    0.912219
   51      16       905       889   69.6949        72    0.855513    0.909605
   52      16       921       905    69.585        64     1.17666    0.908347
   53      16       938       922   69.5548        68     1.38649     0.91053
   54      16       951       935   69.2281        52     0.55436    0.911495
   55      16       971       955   69.4206        80    0.219227    0.908762
   56      16       987       971   69.3236        64    0.301743    0.907939
   57      16      1005       989   69.3701        72    0.501604    0.911605
   58      16      1026      1010   69.6175        84    0.986777    0.912436
   59      16      1047      1031   69.8609        84    0.559188    0.908551
2020-06-08 20:11:58.007277 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.905304
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1063      1047   69.7631        64    0.758972    0.905304
   61      16      1079      1063   69.6684        64     1.48431    0.907042
   62      16      1093      1077   69.4478        56     1.04793    0.909227
   63      16      1109      1093   69.3592        64    0.517822    0.908047
   64      16      1126      1110   69.3379        68    0.138691    0.908729
   65      16      1145      1129    69.439        76     0.81336    0.914216
   66      15      1163      1148    69.538        76    0.927726    0.911847
   67      16      1186      1170   69.8131        88     1.01725    0.909855
   68      16      1205      1189   69.9038        76    0.662465    0.907191
   69      16      1219      1203   69.7022        56    0.609877    0.908125
   70      16      1240      1224   69.9062        84    0.708919    0.909375
   71      16      1261      1245   70.1043        84     1.41813    0.909731
   72      16      1280      1264   70.1859        76    0.693473    0.905951
   73      16      1298      1282   70.2106        72    0.135294    0.903533
   74      16      1316      1300   70.2346        72     0.96985     0.90298
   75      16      1337      1321   70.4179        84     1.08314    0.902772
   76      16      1360      1344   70.7016        92    0.328514    0.899776
   77      16      1379      1363   70.7701        76    0.242521    0.897653
   78      16      1401      1385   70.9906        88    0.458992    0.895658
   79      16      1418      1402   70.9526        68    0.263757    0.892196
2020-06-08 20:12:18.014418 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.892981
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      1435      1419   70.9155        68     1.08344    0.892981
   81      16      1450      1434   70.7806        60     0.30134    0.893305
   82      16      1469      1453   70.8442        76    0.848352    0.893995
   83      16      1491      1475   71.0499        88     0.38473    0.894693
   84      16      1504      1488   70.8231        52   0.0967975    0.893998
   85      16      1521      1505   70.7897        68    0.873757    0.895943
   86      16      1536      1520   70.6642        60    0.529404    0.897625
   87      16      1555      1539   70.7254        76    0.529974    0.899846
   88      16      1573      1557   70.7395        72    0.366885    0.899557
   89      16      1589      1573   70.6637        64    0.154215    0.898277
   90      16      1611      1595   70.8561        88    0.150982    0.895677
   91      16      1632      1616   71.0002        84     0.54568    0.894606
   92      16      1647      1631   70.8806        60    0.504008     0.89485
   93      16      1667      1651   70.9784        80     1.06263    0.894491
   94      16      1680      1664   70.7765        52    0.454223    0.894676
   95      16      1706      1690    71.126       104     0.19763    0.894489
   96      16      1723      1707   71.0933        68    0.403824    0.893514
   97      16      1744      1728   71.2261        84    0.573909    0.893502
   98      16      1759      1743   71.1106        60    0.968286     0.89342
   99      16      1775      1759   71.0387        64     2.57467     0.89396
2020-06-08 20:12:38.020321 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.8947
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      1790      1774   70.9282        60    0.271104      0.8947
  101      16      1803      1787   70.7407        52    0.526812    0.896171
  102      16      1821      1805   70.7529        72     0.45184     0.89595
  103      16      1830      1814   70.4154        36    0.507727    0.896412
  104      16      1846      1830   70.3536        64     0.13647    0.899743
  105      16      1861      1845   70.2549        60    0.520945    0.899306
  106      16      1877      1861   70.1958        64     3.94324    0.901647
  107      16      1893      1877   70.1378        64    0.835055    0.904566
  108      16      1914      1898    70.266        84    0.620073    0.904139
  109      16      1930      1914   70.2085        64    0.653299      0.9056
  110      16      1944      1928   70.0791        56     1.85829    0.907273
  111      16      1963      1947   70.1324        76     1.55756    0.907764
  112      16      1985      1969   70.2917        88    0.645438    0.906213
  113      16      2006      1990   70.4129        84    0.524514    0.904945
  114      16      2026      2010   70.4968        80    0.813488    0.904036
  115      16      2053      2037   70.8227       108    0.959682    0.900399
  116      16      2069      2053   70.7637        64    0.459798    0.898856
  117      16      2099      2083   71.1842       120    0.371983    0.897921
  118      16      2121      2105   71.3266        88    0.594193    0.894294
  119      16      2139      2123    71.332        72     1.12007    0.893198
2020-06-08 20:12:58.024585 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.892195
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      2158      2142   71.3708        76     1.58431    0.892195
  121      16      2177      2161   71.4088        76    0.477455    0.892128
  122      16      2196      2180   71.4462        76    0.571804    0.892119
  123      16      2210      2194   71.3199        56    0.689722    0.891802
  124      16      2226      2210   71.2585        64    0.603326    0.893514
  125      16      2240      2224   71.1364        56     1.53439    0.894229
  126      16      2260      2244   71.2066        80    0.426871     0.89444
  127      16      2281      2265   71.3072        84    0.357074    0.892664
  128      16      2303      2287   71.4374        88    0.597109    0.891852
  129      16      2322      2306   71.4727        76    0.460509    0.889652
  130      16      2345      2329   71.6303        92    0.615189    0.890176
  131      16      2366      2350   71.7227        84      2.0324    0.888586
  132      16      2388      2372   71.8458        88     1.24972     0.88716
  133      16      2408      2392    71.907        80    0.488024     0.88656
  134      16      2428      2412   71.9672        80    0.930692    0.885833
  135      16      2447      2431    71.997        76    0.323277    0.884211
  136      16      2473      2457   72.2321       104    0.394564     0.88333
  137      16      2499      2483   72.4638       104    0.195229    0.879484
  138      16      2532      2516   72.8949       132    0.521735     0.87598
  139      16      2552      2536   72.9459        80    0.613578    0.874899
2020-06-08 20:13:18.037765 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.870502
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      2584      2568   73.3388       128    0.810856    0.870502
  141      16      2611      2595   73.5836       108   0.0896614    0.866676
  142      16      2631      2615   73.6287        80    0.152923    0.865296
  143      16      2656      2640   73.8129       100    0.569885     0.86478
  144      16      2677      2661   73.8836        84    0.816704    0.862674
  145      16      2693      2677   73.8154        64    0.710956    0.862682
  146      16      2714      2698    73.885        84    0.534859    0.862624
  147      16      2730      2714   73.8177        64     2.11317    0.862827
  148      16      2750      2734   73.8573        80    0.140814    0.862759
  149      16      2779      2763   74.1361       116    0.597553    0.861577
  150      16      2800      2784   74.2017        84    0.294405    0.859901
  151      16      2813      2797   74.0547        52    0.551044    0.859506
  152      16      2827      2811   73.9359        56    0.179322     0.86001
  153      16      2841      2825   73.8186        56    0.537953    0.862982
  154      16      2851      2835    73.599        40   0.0894164    0.862833
  155      16      2870      2854   73.6144        76    0.146518    0.863878
  156      16      2893      2877   73.7321        92    0.667938    0.864718
  157      16      2909      2893   73.6694        64     0.35911    0.864521
  158      16      2931      2915   73.7593        88     1.99247    0.864606
  159      16      2954      2938   73.8738        92    0.470108    0.864151
2020-06-08 20:13:38.059453 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.863755
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      2971      2955   73.8363        68    0.913875    0.863755
  161      16      2985      2969   73.7255        56    0.122776     0.86388
  162      16      3015      2999   74.0109       120    0.292877    0.862976
  163      16      3034      3018   74.0229        76     1.14578    0.862273
  164      16      3052      3036   74.0105        72    0.194916    0.861431
  165      16      3068      3052   73.9484        64     2.15639    0.861781
  166      16      3083      3067   73.8643        60     2.59581    0.861478
  167      16      3097      3081   73.7573        56    0.217251    0.861189
  168      16      3112      3096   73.6754        60    0.744343    0.862415
  169      16      3131      3115    73.689        76    0.432271    0.862675
  170      16      3149      3133    73.679        72    0.132586    0.865311
  171      16      3163      3147   73.5756        56    0.311371    0.864491
  172      16      3185      3169   73.6593        88     2.54587    0.866844
  173      16      3203      3187   73.6497        72    0.597465     0.86665
  174      16      3213      3197   73.4563        40     1.66117     0.86732
  175      16      3230      3214    73.425        68    0.525784    0.868341
  176      16      3251      3235    73.485        84    0.195678    0.868587
  177      16      3273      3257   73.5669        88    0.318552    0.866793
  178      16      3294      3278   73.6254        84    0.456156    0.866243
  179      16      3316      3300   73.7056        88     0.90386     0.86562
2020-06-08 20:13:58.066526 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.866679
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      3330      3314   73.6072        56    0.772631    0.866679
  181      16      3349      3333   73.6204        76    0.506055    0.867013
  182      16      3370      3354   73.6773        84    0.862603    0.866022
  183      16      3389      3373   73.6899        76    0.126265    0.865657
  184      16      3413      3397    73.811        96    0.472542    0.864694
  185      16      3438      3422   73.9525       100    0.548133    0.863556
  186      16      3459      3443   74.0064        84    0.142547    0.862001
  187      16      3482      3466   74.1025        92    0.197975     0.86119
  188      16      3504      3488   74.1763        88     1.21617    0.860449
  189      16      3520      3504   74.1224        64    0.617482    0.860191
  190      16      3538      3522   74.1111        72     1.42981    0.860781
  191      16      3555      3539   74.0791        68     0.67103    0.860066
  192      16      3575      3559   74.1098        80    0.209799    0.859894
  193      16      3595      3579   74.1403        80    0.102169    0.860217
  194      16      3608      3592   74.0256        52    0.551272    0.860418
  195      16      3623      3607   73.9537        60    0.549342    0.860765
  196      16      3641      3625   73.9436        72    0.284283    0.862124
  197      16      3658      3642   73.9134        68      1.3556    0.863595
  198      16      3672      3656   73.8229        56    0.118254    0.863518
  199      16      3685      3669   73.7133        52    0.442957     0.86486
2020-06-08 20:14:18.071479 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.865218
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      3709      3693   73.8246        96    0.764446    0.865218
  201      16      3729      3713   73.8552        80     0.10531     0.86423
  202      16      3753      3737   73.9646        96     1.16193    0.863313
  203      16      3776      3760   74.0533        92     1.35557    0.862287
  204      16      3793      3777   74.0236        68    0.362237     0.86228
  205      16      3818      3802   74.1502       100     1.95675    0.860927
  206      16      3843      3827   74.2756       100    0.699881    0.859939
  207      16      3862      3846   74.2838        76    0.373804    0.858611
  208      16      3887      3871   74.4073       100    0.468576    0.858524
  209      16      3904      3888   74.3767        68    0.850498    0.858089
  210      16      3921      3905   74.3459        68    0.122704     0.85862
  211      16      3946      3930   74.4671       100    0.688795    0.857776
  212      16      3965      3949   74.4743        76    0.887289     0.85747
  213      16      3986      3970   74.5189        84     0.10099    0.857015
  214      16      4007      3991   74.5631        84    0.652765     0.85672
  215      16      4023      4007   74.5131        64     1.24319    0.856411
  216      16      4031      4015   74.3162        32    0.123267    0.856949
  217      16      4051      4035    74.342        80     1.76581     0.85828
  218      16      4069      4053   74.3312        72     1.24225    0.858736
  219      16      4087      4071   74.3205        72     0.36971      0.8589
2020-06-08 20:14:38.080841 min lat: 0.0730331 max lat: 4.29548 avg lat: 0.858769
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      4109      4093   74.3826        88    0.421784    0.858769
  221      16      4131      4115   74.4441        88    0.643271    0.858154
  222      16      4150      4134    74.451        76     1.15955    0.857912
  223      16      4166      4150   74.4041        64    0.364356    0.857335
  224      16      4184      4168   74.3933        72    0.556197    0.857229
  225      16      4201      4185   74.3648        68     2.87592    0.857676
  226      16      4220      4204    74.372        76     0.11732    0.857482
  227      16      4234      4218    74.291        56    0.542402    0.857934
  228      16      4250      4234   74.2458        64    0.994756    0.857719
  229      16      4260      4244   74.0962        40    0.460497    0.859127
  230      16      4276      4260   74.0523        64    0.166571    0.859328
  231      16      4290      4274   73.9742        56    0.633513    0.859575
  232      16      4302      4286   73.8622        48     4.20751      0.8606
  233      16      4319      4303    73.837        68    0.704034    0.862291
  234      16      4337      4321   73.8289        72    0.767082    0.863067
  235      16      4355      4339   73.8211        72    0.307592    0.864738
  236      16      4369      4353   73.7455        56     1.06873    0.865095
  237      16      4393      4377   73.8393        96    0.640935    0.865275
  238      16      4410      4394   73.8148        68    0.217787     0.86458
  239      16      4430      4414   73.8406        80    0.112997    0.864407
2020-06-08 20:14:58.085368 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.86433
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      4448      4432   73.8329        72    0.750785     0.86433
  241      16      4466      4450   73.8252        72    0.980823     0.86425
  242      16      4485      4469   73.8327        76    0.330765    0.863928
  243      16      4496      4480   73.7099        44     2.53958    0.863934
  244      16      4517      4501    73.752        84    0.300727    0.865453
  245      16      4528      4512   73.6304        44     1.32415    0.866626
  246      16      4550      4534   73.6887        88    0.683801    0.867046
  247      16      4565      4549   73.6333        60    0.559875    0.866727
  248      16      4586      4570    73.675        84     1.58628    0.867045
  249      16      4610      4594   73.7646        96    0.619608    0.865653
  250      16      4631      4615   73.8054        84    0.893729    0.865241
  251      16      4643      4627   73.7026        48     1.36248    0.865516
  252      16      4668      4652   73.8068       100    0.111955    0.865293
  253      16      4682      4666   73.7364        56     1.03799    0.865374
  254      16      4698      4682    73.698        64    0.118631    0.865129
  255      16      4712      4696   73.6286        56     2.77725    0.865131
  256      16      4734      4718   73.6846        88    0.200313    0.866035
  257      16      4747      4731   73.6002        52    0.377505    0.867007
  258      16      4767      4751   73.6249        80    0.333156    0.866838
  259      16      4782      4766   73.5723        60    0.391211    0.867606
2020-06-08 20:15:18.095119 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.867525
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      4801      4785   73.5815        76    0.801623    0.867525
  261      16      4828      4812   73.7132       108     1.46074    0.867074
  262      15      4849      4834   73.7675        88   0.0946217      0.8662
  263      16      4869      4853    73.776        76    0.485207    0.865938
  264      16      4887      4871   73.7692        72     1.02176    0.865994
  265      16      4896      4880   73.6267        36     1.02482    0.866309
  266      16      4916      4900   73.6506        80     1.75639    0.867135
  267      16      4930      4914   73.5844        56    0.793195    0.866968
  268      16      4949      4933   73.5934        76    0.269814    0.866848
  269      16      4965      4949   73.5576        64     0.47214    0.867088
  270      16      4984      4968   73.5666        76   0.0818159    0.867103
  271      16      5003      4987   73.5755        76    0.355008    0.867918
  272      16      5022      5006   73.5843        76     1.37336     0.86837
  273      16      5034      5018   73.4906        48     1.08197    0.868069
  274      16      5050      5034   73.4558        64    0.574166    0.868039
  275      16      5063      5047   73.3775        52    0.410347    0.868049
  276      16      5080      5064   73.3579        68    0.462378    0.868981
  277      16      5097      5081   73.3386        68    0.595854    0.869527
  278      16      5117      5101   73.3624        80    0.144244    0.869842
  279      16      5133      5117   73.3289        64    0.152577    0.869983
2020-06-08 20:15:38.101770 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.869712
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      5153      5137   73.3526        80    0.265636    0.869712
  281      16      5181      5165     73.49       112    0.100195     0.86907
  282      16      5200      5184   73.4989        76    0.531119    0.868996
  283      16      5210      5194   73.3806        40    0.164295    0.869019
  284      16      5230      5214   73.4038        80    0.501973    0.869291
  285      16      5248      5232   73.3979        72    0.139468    0.869976
  286      16      5265      5249    73.379        68    0.805504    0.869721
  287      16      5280      5264   73.3324        60    0.837478    0.870432
  288      16      5289      5273   73.2027        36     1.06794    0.870923
  289      16      5305      5289   73.1708        64     2.13368    0.872454
  290      16      5324      5308   73.1804        76    0.284596    0.872336
  291      16      5340      5324   73.1487        64     0.72313    0.872466
  292      16      5358      5342   73.1447        72     0.31993    0.872824
  293      16      5375      5359   73.1271        68    0.252697    0.873302
  294      16      5391      5375    73.096        64    0.106871    0.873439
  295      16      5409      5393   73.0922        72     0.59772    0.872936
  296      16      5422      5406   73.0209        52    0.109273    0.873116
  297      16      5439      5423   73.0033        68    0.444732    0.873416
  298      16      5454      5438   72.9597        60     0.32053    0.874756
  299      16      5476      5460   73.0099        88     1.80855    0.875165
2020-06-08 20:15:58.113911 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.8745
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      5493      5477    72.993        68    0.442914      0.8745
  301      16      5514      5498   73.0292        84    0.454339    0.874582
  302      16      5532      5516   73.0258        72     2.05633    0.874407
  303      16      5549      5533   73.0092        68      1.9747    0.874562
  304      16      5565      5549   72.9795        64    0.556228    0.874017
  305      16      5583      5567   72.9763        72    0.584978    0.874186
  306      16      5605      5589   73.0252        88     2.75141    0.874373
  307      16      5625      5609   73.0478        80    0.394376    0.873885
  308      16      5642      5626   73.0314        68    0.687721     0.87404
  309      16      5660      5644   73.0279        72    0.835038    0.873733
  310      16      5682      5666   73.0762        88    0.481862    0.874148
  311      16      5700      5684   73.0726        72   0.0981046    0.873689
  312      16      5717      5701   73.0563        68    0.173722    0.873277
  313      16      5734      5718   73.0397        68    0.454973    0.873389
  314      16      5751      5735   73.0236        68    0.584949    0.874248
  315      16      5769      5753   73.0203        72    0.451557    0.874648
  316      16      5784      5768   72.9791        60    0.379275     0.87474
  317      16      5801      5785   72.9633        68      0.2345    0.875049
  318      16      5826      5810   73.0483       100    0.633942    0.875274
  319      16      5842      5826   73.0199        64    0.526023    0.875175
2020-06-08 20:16:18.121757 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.875051
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      5857      5841   72.9791        60    0.869822    0.875051
  321      16      5877      5861    73.001        80     2.01968    0.875335
  322      16      5890      5874   72.9357        52    0.500115     0.87541
  323      16      5912      5896   72.9823        88    0.476156    0.875176
  324      16      5930      5914   72.9793        72     3.12918     0.87563
  325      16      5950      5934   73.0008        80    0.417863    0.875449
  326      16      5971      5955   73.0345        84    0.387265    0.874447
  327      16      5994      5978   73.0924        92    0.912747    0.873835
  328      16      6011      5995   73.0769        68    0.340952    0.873547
  329      16      6035      6019   73.1465        96     1.76545    0.873406
  330      16      6055      6039   73.1672        80    0.439994    0.872812
  331      16      6073      6057   73.1636        72    0.182007    0.872341
  332      16      6092      6076   73.1721        76    0.539628    0.872316
  333      16      6111      6095   73.1806        76    0.397292    0.872843
  334      16      6127      6111    73.153        64    0.686426    0.872999
  335      16      6150      6134   73.2092        92     1.55672    0.873199
  336      16      6167      6151   73.1937        68    0.745112    0.872449
  337      16      6188      6172   73.2257        84    0.416634    0.872576
  338      16      6213      6197   73.3048       100    0.221025    0.871792
  339      16      6237      6221   73.3717        96    0.380544     0.87119
2020-06-08 20:16:38.125389 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.870645
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      6255      6239   73.3677        72    0.237674    0.870645
  341      16      6273      6257   73.3636        72     1.91148    0.870875
  342      16      6285      6269   73.2894        48    0.625473    0.870964
  343      16      6304      6288   73.2972        76    0.684181    0.871232
  344      16      6320      6304   73.2702        64     1.31585    0.871307
  345      16      6337      6321   73.2548        68    0.533962    0.871754
  346      16      6354      6338   73.2396        68    0.567822    0.871976
  347      16      6370      6354    73.213        64    0.282414    0.872345
  348      16      6392      6376   73.2553        88      1.0019    0.872403
  349      16      6413      6397   73.2861        84    0.834793     0.87182
  350      16      6434      6418   73.3166        84    0.690901    0.871459
  351      16      6456      6440   73.3584        88     1.84694    0.871126
  352      16      6470      6454   73.3091        56    0.764582    0.871519
  353      16      6484      6468     73.26        56    0.369115    0.871565
  354      16      6500      6484   73.2338        64     2.27152    0.871695
  355      16      6518      6502   73.2303        72    0.420901    0.872285
  356      16      6536      6520   73.2268        72     0.72114    0.872575
  357      16      6556      6540   73.2434        80     2.14448    0.872497
  358      16      6573      6557   73.2288        68     1.11625    0.872674
  359      16      6588      6572   73.1919        60    0.364302    0.872606
2020-06-08 20:16:58.141106 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.873117
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      6604      6588   73.1663        64    0.174063    0.873117
  361      16      6623      6607   73.1742        76    0.862532    0.873303
  362      16      6646      6630   73.2261        92     0.89393    0.872871
  363      16      6665      6649   73.2335        76    0.837989    0.872608
  364      16      6685      6669    73.252        80    0.781318    0.872458
  365      16      6706      6690   73.2814        84      1.0914    0.872143
  366      16      6723      6707   73.2669        68    0.451284    0.871841
  367      16      6738      6722   73.2307        60    0.506314    0.872117
  368      16      6759      6743   73.2599        84    0.625485    0.872373
  369      16      6775      6759   73.2348        64    0.687533     0.87251
  370      16      6790      6774   73.1986        60     0.67488    0.872351
  371      16      6807      6791   73.1846        68    0.488116    0.872876
  372      16      6826      6810   73.1921        76     1.86775     0.87267
  373      16      6849      6833   73.2425        92    0.102388    0.872601
  374      16      6870      6854   73.2712        84    0.963751    0.872613
  375      16      6892      6876   73.3104        88    0.738477    0.871927
  376      16      6911      6895   73.3176        76    0.712372    0.871598
  377      16      6929      6913    73.314        72    0.500957    0.871591
  378      16      6947      6931   73.3105        72    0.308952    0.871554
  379      16      6968      6952   73.3387        84    0.469417    0.871198
2020-06-08 20:17:18.147871 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.870984
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16      6989      6973   73.3667        84    0.438869    0.870984
  381      16      7008      6992   73.3736        76    0.980196    0.871126
  382      16      7025      7009   73.3595        68    0.302758    0.870969
  383      16      7048      7032   73.4078        92    0.579218    0.871022
  384      16      7065      7049   73.3937        68    0.215526     0.87051
  385      16      7081      7065   73.3693        64     0.15136    0.870739
  386      16      7108      7092   73.4589       108    0.933536    0.870274
  387      16      7128      7112   73.4758        80     1.34933    0.870011
  388      16      7147      7131   73.4822        76    0.278081    0.869447
  389      16      7165      7149   73.4784        72   0.0909752    0.869213
  390      16      7184      7168   73.4848        76     1.68477     0.86937
  391      16      7196      7180   73.4196        48     2.09545    0.869853
  392      16      7212      7196   73.3956        64     1.25778    0.870143
  393      16      7224      7208   73.3304        48     1.51674    0.870445
  394      16      7243      7227   73.3372        76    0.511055    0.870731
  395      16      7259      7243   73.3135        64      1.0424    0.870905
  396      16      7278      7262   73.3203        76    0.144077    0.871276
  397      16      7297      7281    73.327        76     0.53167    0.870933
  398      16      7316      7300   73.3325        76    0.261177    0.870826
  399      16      7329      7313    73.279        52    0.800692    0.871398
2020-06-08 20:17:38.162475 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.871241
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16      7347      7331   73.2758        72    0.440267    0.871241
  401      16      7365      7349   73.2721        72     0.41204    0.871827
  402      16      7381      7365    73.249        64     0.34712    0.871989
  403      16      7401      7385   73.2657        80    0.446505     0.87246
  404      16      7415      7399   73.2229        56     1.57592    0.872653
  405      16      7434      7418   73.2298        76     1.14035    0.872554
  406      16      7458      7442   73.2858        96    0.937192    0.872384
  407      16      7477      7461   73.2923        76     1.24865    0.871886
  408      16      7491      7475   73.2499        56    0.689652    0.872057
  409      16      7519      7503   73.3446       112    0.105469    0.871771
  410      16      7535      7519   73.3218        64    0.592535    0.871449
  411      16      7558      7542   73.3659        92    0.490002    0.870978
  412      16      7572      7556   73.3237        56    0.613341    0.871017
  413      16      7586      7570   73.2818        56   0.0930081    0.870963
  414      16      7598      7582   73.2207        48    0.336964    0.871426
  415      16      7616      7600   73.2177        72    0.630183    0.872075
  416      16      7635      7619   73.2244        76    0.362077    0.872057
  417      16      7651      7635   73.2022        64     1.79705    0.872563
  418      16      7665      7649   73.1611        56    0.524832    0.872746
  419      16      7674      7658   73.0724        36     2.70833    0.873783
2020-06-08 20:17:58.176349 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.874201
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16      7692      7676   73.0698        72    0.606834    0.874201
  421      16      7707      7691   73.0388        60    0.530377    0.874531
  422      16      7729      7713   73.0742        88      1.5095    0.874335
  423      16      7753      7737   73.1283        96    0.729968    0.874409
  424      16      7771      7755   73.1256        72    0.899083    0.874033
  425      16      7788      7772   73.1135        68     1.56701    0.873923
  426      16      7811      7795   73.1578        92    0.146697     0.87367
  427      16      7826      7810    73.127        60    0.525777     0.87344
  428      16      7843      7827    73.115        68    0.833314    0.873993
  429      16      7858      7842   73.0844        60    0.749745    0.873882
  430      16      7877      7861    73.091        76     2.57347    0.873901
  431      16      7895      7879   73.0884        72    0.480756     0.87408
  432      16      7905      7889   73.0119        40    0.667566    0.874432
  433      16      7921      7905   72.9908        64    0.271584    0.874489
  434      16      7938      7922   72.9793        68     0.57887    0.874595
  435      16      7956      7940    72.977        72    0.788886    0.875303
  436      16      7971      7955   72.9472        60    0.991915    0.875371
  437      16      7991      7975   72.9633        80    0.781064    0.875913
  438      16      8014      7998   73.0068        92    0.531112    0.875517
  439      16      8031      8015   72.9953        68      1.0933    0.875622
2020-06-08 20:18:18.181905 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.875376
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16      8053      8037   73.0294        88     1.48891    0.875376
  441      16      8069      8053   73.0089        64    0.667919    0.875279
  442      16      8090      8074   73.0337        84    0.515497    0.875288
  443      16      8111      8095   73.0584        84    0.969861    0.875129
  444      16      8131      8115    73.074        80    0.586855    0.874649
  445      16      8150      8134   73.0806        76    0.911212    0.874691
  446      16      8162      8146   73.0243        48     1.47094    0.875018
  447      16      8182      8166   73.0399        80    0.984956    0.875177
  448      16      8200      8184   73.0375        72    0.511365     0.87497
  449      16      8221      8205   73.0619        84    0.466792    0.874861
  450      16      8241      8225   73.0773        80    0.498444    0.874711
  451      16      8261      8245   73.0926        80     1.12324    0.874394
  452      16      8277      8261   73.0725        64    0.358167    0.874631
  453      16      8292      8276   73.0434        60     1.55083    0.874833
  454      16      8317      8301   73.1027       100     1.08791    0.874758
  455      16      8335      8319   73.0998        72    0.485704     0.87463
  456      16      8364      8348   73.1937       116    0.106496    0.873635
  457      16      8379      8363   73.1648        60      0.6704    0.873633
  458      16      8391      8375   73.1098        48    0.142305    0.873526
  459      16      8405      8389   73.0726        56     2.03715    0.874052
2020-06-08 20:18:38.190489 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.874502
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16      8415      8399   73.0007        40     3.17999    0.874502
  461      16      8430      8414   72.9724        60     1.38506    0.875866
  462      16      8449      8433    72.979        76    0.138435    0.876087
  463      16      8471      8455   73.0114        88    0.982731    0.875894
  464      16      8489      8473   73.0092        72     1.12901    0.875441
  465      16      8511      8495   73.0408        88    0.116214    0.875281
  466      16      8526      8510   73.0128        60     1.47453    0.875189
  467      16      8546      8530   73.0277        80    0.578646    0.874975
  468      16      8562      8546   73.0084        64     1.86771    0.874965
  469      16      8583      8567   73.0318        84    0.244918    0.874656
  470      16      8599      8583   73.0126        64    0.275093    0.874632
  471      16      8619      8603   73.0274        80    0.128726    0.874655
  472      16      8634      8618   72.9998        60    0.572616    0.874788
  473      16      8647      8631   72.9554        52    0.479875    0.875331
  474      16      8659      8643   72.9027        48    0.688491    0.875632
  475      16      8675      8659    72.884        64     2.74441    0.876297
  476      16      8690      8674   72.8569        60   0.0918844    0.876375
  477      16      8706      8690   72.8383        64    0.498907    0.876704
  478      16      8721      8705   72.8114        60    0.751913    0.877124
  479      16      8739      8723   72.8097        72     3.47103    0.877329
2020-06-08 20:18:58.197532 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.87719
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16      8755      8739   72.7913        64    0.111138     0.87719
  481      16      8766      8750   72.7315        44    0.506961    0.877573
  482      16      8783      8767   72.7216        68    0.504694    0.877913
  483      16      8794      8778   72.6621        44    0.489185    0.878579
  484      16      8810      8794   72.6442        64    0.177624     0.87876
  485      16      8825      8809   72.6181        60     1.07506    0.879152
  486      16      8838      8822   72.5757        52    0.656139     0.87968
  487      16      8854      8838    72.558        64    0.673322    0.880395
  488      15      8869      8854   72.5405        64    0.509493    0.880211
  489      16      8883      8867   72.4984        52     3.05994    0.880934
  490      16      8893      8877   72.4321        40     1.04804    0.881329
  491      16      8910      8894   72.4231        68    0.863421    0.881913
  492      16      8926      8910    72.406        64     1.97329    0.882478
  493      16      8941      8925   72.3808        60    0.448451    0.882659
  494      16      8955      8939   72.3476        56    0.461017    0.882659
  495      16      8970      8954   72.3227        60    0.642146    0.882912
  496      16      8988      8972    72.322        72     1.11537    0.883378
  497      16      9004      8988   72.3052        64    0.750038    0.883875
  498      16      9026      9010   72.3367        88    0.223701    0.883509
  499      16      9047      9031     72.36        84    0.901783    0.883695
2020-06-08 20:19:18.202528 min lat: 0.0720062 max lat: 4.4007 avg lat: 0.883166
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16      9073      9057   72.4231       104    0.445671    0.883166
  501      16      9090      9074   72.4143        68    0.344235    0.882752
  502      16      9114      9098   72.4612        96    0.574177    0.882186
  503      16      9132      9116   72.4603        72    0.296478    0.881961
  504      16      9146      9130   72.4276        56     2.85507    0.882088
  505      16      9165      9149   72.4347        76     0.61285    0.882188
  506      16      9185      9169   72.4496        80    0.568072    0.882126
  507      16      9195      9179   72.3855        40    0.288035      0.8821
  508      16      9215      9199   72.4005        80    0.444345    0.882319
  509      16      9236      9220   72.4232        84    0.443104    0.882563
  510      16      9258      9242   72.4537        88    0.512261    0.882188
  511      16      9274      9258   72.4372        64     2.43589    0.882312
  512      16      9284      9268   72.3738        40    0.431717    0.882272
  513      16      9294      9278   72.3107        40    0.362419    0.882814
  514      16      9308      9292    72.279        56    0.652186    0.883425
  515      16      9330      9314   72.3095        88    0.361627    0.883462
  516      16      9348      9332   72.3089        72     1.91057    0.883873
  517      16      9364      9348   72.2928        64    0.236557    0.883988
  518      16      9380      9364   72.2767        64     0.75314    0.884531
  519      16      9396      9380   72.2608        64    0.466132    0.884371
2020-06-08 20:19:38.206381 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.88435
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16      9417      9401   72.2833        84    0.510479     0.88435
  521      16      9437      9421    72.298        80    0.775545    0.884331
  522      16      9454      9438   72.2898        68     1.57618    0.884393
  523      16      9467      9451    72.251        52    0.653131    0.884449
  524      16      9489      9473    72.281        88     0.48567    0.884436
  525      16      9520      9504   72.3794       124    0.477646    0.883514
  526      16      9539      9523   72.3863        76    0.778714    0.882993
  527      16      9563      9547    72.431        96     1.65646    0.882646
  528      16      9584      9568   72.4529        84    0.101928    0.882272
  529      16      9602      9586    72.452        72    0.705437    0.881983
  530      16      9627      9611   72.5038       100    0.643886    0.882033
  531      16      9643      9627   72.4878        64    0.640231    0.882064
  532      16      9663      9647   72.5019        80    0.450927    0.881662
  533      16      9686      9670   72.5384        92    0.722139    0.881466
  534      16      9702      9686   72.5224        64    0.493813    0.881443
  535      16      9719      9703   72.5139        68    0.893322    0.881658
  536      16      9738      9722   72.5204        76    0.416984    0.881512
  537      16      9755      9739   72.5117        68     1.93052    0.881418
  538      16      9768      9752   72.4736        52     2.15586    0.881727
  539      16      9781      9765   72.4356        52    0.869417    0.881877
2020-06-08 20:19:58.213102 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.882211
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16      9800      9784   72.4422        76    0.359025    0.882211
  541      16      9810      9794   72.3822        40    0.580788    0.882555
  542      16      9836      9820   72.4405       104    0.693178    0.882941
  543      16      9854      9838   72.4397        72    0.982478      0.8828
  544      16      9872      9856   72.4388        72    0.546233    0.882561
  545      16      9888      9872   72.4233        64    0.183194    0.882458
  546      16      9911      9895   72.4591        92    0.518778    0.882679
  547      16      9926      9910   72.4363        60    0.673578    0.882286
  548      16      9943      9927   72.4282        68     1.47013    0.882573
  549      16      9963      9947   72.4419        80    0.470011      0.8824
  550      16      9980      9964   72.4338        68     1.13835    0.882455
  551      16      9997      9981   72.4257        68    0.364902    0.882334
  552      16     10015      9999   72.4249        72    0.375624    0.882508
  553      16     10033     10017   72.4241        72    0.293664    0.882372
  554      16     10048     10032   72.4017        60     2.33343    0.882648
  555      16     10063     10047   72.3792        60     2.31065     0.88295
  556      16     10081     10065   72.3785        72     0.66063    0.883219
  557      16     10096     10080   72.3563        60     1.07249    0.883129
  558      16     10114     10098   72.3556        72    0.750912    0.883415
  559      16     10131     10115   72.3478        68    0.653086    0.883261
2020-06-08 20:20:18.219006 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.883564
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     10150     10134   72.3543        76    0.512078    0.883564
  561      16     10172     10156   72.3821        88    0.545524    0.883269
  562      16     10188     10172   72.3672        64    0.630823    0.883461
  563      16     10203     10187   72.3452        60    0.575679    0.883473
  564      16     10221     10205   72.3445        72   0.0998445    0.883611
  565      16     10241     10225   72.3559        80    0.723494    0.883808
  566      16     10259     10243   72.3552        72    0.372775    0.883602
  567      16     10270     10254   72.3052        44    0.978835    0.883669
  568      16     10290     10274   72.3187        80     1.58439    0.883981
  569      16     10314     10298   72.3603        96     1.12667    0.883854
  570      16     10335     10319   72.3807        84    0.476324    0.883195
  571      16     10351     10335    72.366        64    0.804836    0.882921
  572      16     10363     10347   72.3234        48    0.642962    0.882924
  573      16     10383     10367   72.3368        80    0.523085    0.883201
  574      16     10399     10383   72.3222        64    0.776479    0.883045
  575      16     10416     10400   72.3147        68    0.318009     0.88346
  576      16     10436     10420    72.328        80    0.624059    0.883577
  577      16     10455     10439   72.3343        76    0.789692    0.883085
  578      16     10477     10461   72.3614        88    0.462701    0.883291
  579      16     10491     10475   72.3332        56    0.299997    0.883183
2020-06-08 20:20:38.239770 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.883197
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     10513     10497   72.3601        88    0.468421    0.883197
  581      16     10526     10510   72.3251        52    0.516525    0.883276
  582      16     10542     10526   72.3108        64    0.150456    0.883449
  583      16     10561     10545   72.3171        76    0.152284    0.883633
  584      16     10582     10566    72.337        84    0.903022    0.883638
  585      16     10602     10586   72.3501        80      1.3925    0.883683
  586      16     10624     10608   72.3767        88    0.584053    0.883096
  587      16     10648     10632   72.4169        96    0.817164    0.882926
  588      16     10668     10652   72.4298        80    0.501111    0.882698
  589      16     10688     10672   72.4426        80    0.508018    0.882566
  590      16     10721     10705   72.5434       132    0.569683    0.882076
  591      16     10751     10735   72.6237       120    0.113236    0.880791
  592      16     10782     10766   72.7104       124    0.189927    0.879653
  593      16     10812     10796   72.7901       120    0.201906    0.878778
  594      16     10852     10836   72.9368       160    0.602972    0.877035
  595      16     10888     10872   73.0562       144    0.497497    0.875504
  596      16     10923     10907   73.1684       140     0.33945    0.874248
  597      16     10956     10940   73.2669       132    0.418792    0.873231
  598      16     10985     10969   73.3383       116    0.124343    0.872056
  599      16     11022     11006   73.4629       148    0.500349    0.870872
2020-06-08 20:20:58.244444 min lat: 0.0718277 max lat: 4.4007 avg lat: 0.869454
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     11055     11039   73.5604       132    0.552603    0.869454
Total time run:         600.488
Total writes made:      11055
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     73.6401
Stddev Bandwidth:       17.4025
Max bandwidth (MB/sec): 160
Min bandwidth (MB/sec): 32
Average IOPS:           18
Stddev IOPS:            4.35062
Max IOPS:               40
Min IOPS:               8
Average Latency(s):     0.869061
Stddev Latency(s):      0.692347
Max latency(s):         4.4007
Min latency(s):         0.0718277

Total time run:         600.488
Total writes made:      11055
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     73.6401
Stddev Bandwidth:       17.4025
Max bandwidth (MB/sec): 160
Min bandwidth (MB/sec): 32
Average IOPS:           18
Stddev IOPS:            4.35062
Max IOPS:               40
Min IOPS:               8
Average Latency(s):     0.869061
Stddev Latency(s):      0.692347
Max latency(s):         4.4007
Min latency(s):         0.0718277

Sequential Read Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 seq | tee --append $(date +%F)-256PGs-rados-test-read-seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        52        36   143.952       144     0.10248    0.257708
    2      16        80        64    127.97       112    0.395358    0.388466
    3      16       112        96   127.975       128    0.332911    0.416507
    4      16       141       125   124.848       116   0.0534434    0.447449
    5      16       174       158   126.272       132   0.0459037    0.452606
    6      16       203       187   124.559       116   0.0904399    0.466416
    7      16       235       219   125.047       128    0.717821    0.475117
    8      16       260       244   121.916       100     1.31779     0.49151
    9      16       289       273   121.257       116    0.316016    0.494611
   10      16       320       304    121.53       124     0.62921    0.497556
   11      16       355       339   123.179       140     1.35594     0.49199
   12      15       388       373   124.227       136    0.209883    0.493579
   13      16       421       405   124.385       128   0.0983189    0.489222
   14      16       452       436   124.357       124    0.248015    0.488435
   15      16       483       467   124.332       124   0.0657991    0.493576
   16      16       512       496   123.811       116    0.407897    0.498002
   17      15       540       525    123.33       116    0.223065    0.498863
   18      16       569       553   122.701       112   0.0935895    0.500051
   19      16       598       582   122.348       116   0.0550563    0.502108
2020-06-08 20:23:55.835704 min lat: 0.0229199 max lat: 1.96882 avg lat: 0.501858
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       630       614   122.629       128     0.27236    0.501858
   21      16       662       646   122.884       128     1.93863    0.501786
   22      16       691       675    122.57       116      0.5895    0.506386
   23      16       721       705   122.458       120    0.318985    0.505303
   24      15       754       739   122.989       136     1.26615    0.505394
   25      16       783       767   122.546       112     1.67834    0.504546
   26      16       813       797   122.447       120    0.342159    0.507667
   27      16       847       831   122.948       136     1.39683    0.505411
   28      15       876       861   122.813       120    0.316757    0.504973
   29      16       903       887   122.161       104    0.487501    0.507258
   30      15       933       918   122.211       124    0.387951    0.510148
   31      16       962       946   121.882       112    0.196314    0.510662
   32      16       994       978   122.072       128     1.81746    0.511042
   33      16      1023      1007   121.888       116     1.22816    0.511654
   34      16      1054      1038   121.949       124     0.47151    0.510387
   35      16      1082      1066   121.665       112    0.325294    0.510798
   36      16      1121      1105   122.617       156     1.09879    0.510156
   37      16      1155      1139   122.978       136   0.0704184    0.508363
   38      16      1183      1167   122.689       112    0.274135    0.509345
   39      16      1214      1198   122.722       124    0.333837    0.509944
2020-06-08 20:24:15.898027 min lat: 0.0229199 max lat: 2.16235 avg lat: 0.510367
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1242      1226   122.324       112     1.55592    0.510367
   41      16      1268      1252   121.878       104    0.233863    0.511122
   42      16      1299      1283   121.927       124   0.0489042    0.514665
   43      16      1329      1313   121.882       120     1.22462     0.51466
   44      16      1363      1347   122.202       136    0.868778    0.514214
   45      16      1390      1374   121.886       108    0.552348     0.51362
   46      16      1425      1409   122.279       140   0.0486978    0.511113
   47      16      1453      1437    122.06       112   0.0799922    0.512212
   48      16      1487      1471   122.349       136    0.380298    0.510939
   49      16      1521      1505   122.627       136    0.250579    0.510254
   50      15      1547      1532   122.334       108    0.726027    0.513268
   51      16      1576      1560    122.13       112    0.221502    0.513779
   52      16      1608      1592   122.242       128     0.43852    0.512775
   53      16      1642      1626     122.5       136    0.404703    0.511807
   54      16      1674      1658     122.6       128    0.274548    0.511684
   55      16      1707      1691    122.77       132    0.320383    0.510371
   56      16      1739      1723   122.863       128    0.214707    0.510048
   57      16      1769      1753   122.812       120     1.60844    0.509381
   58      16      1797      1781   122.625       112    0.198468    0.510414
   59      16      1828      1812   122.645       124     1.29003    0.511771
2020-06-08 20:24:35.906436 min lat: 0.0229199 max lat: 2.16235 avg lat: 0.511011
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1859      1843   122.665       124   0.0929808    0.511011
   61      16      1890      1874   122.685       124     0.16983    0.510338
   62      16      1924      1908   122.899       136     1.95647    0.511738
   63      16      1954      1938   122.851       120    0.174255    0.510767
   64      16      1987      1971   122.989       132     0.84841    0.511279
   65      15      2012      1997   122.679       104   0.0474931    0.512642
   66      16      2045      2029    122.76       128    0.310809    0.512644
   67      15      2075      2060   122.738       124    0.573777    0.512099
   68      16      2105      2089   122.639       116   0.0646949    0.511945
   69      16      2138      2122   122.774       132   0.0624156    0.512316
   70      16      2169      2153   122.791       124     1.36978    0.511709
   71      15      2198      2183   122.703       120   0.0569101    0.512025
   72      16      2232      2216   122.832       132    0.153078    0.511764
   73      16      2262      2246   122.793       120    0.199818    0.511774
   74      16      2293      2277   122.809       124    0.339496     0.51146
   75      16      2324      2308   122.824       124    0.048679     0.51081
   76      16      2353      2337   122.735       116    0.312595    0.511596
   77      16      2384      2368   122.751       124     2.03753    0.511888
   78      16      2413      2397   122.622       116    0.955439    0.511969
   79      16      2443      2427   122.588       120    0.299719     0.51355
2020-06-08 20:24:56.000431 min lat: 0.0227928 max lat: 2.36116 avg lat: 0.513632
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2473      2457   122.555       120    0.261607    0.513632
   81      16      2512      2496   122.967       156    0.242555    0.511745
   82      16      2545      2529   123.076       132     1.67602    0.510647
   83      16      2579      2563   123.231       136     0.35015    0.510781
   84      16      2610      2594   123.239       124    0.311045    0.511449
   85      16      2641      2625   123.248       124    0.292918     0.51041
   86      16      2668      2652    123.07       108     1.74807    0.511303
   87      16      2698      2682   123.035       120    0.226664      0.5109
   88      16      2727      2711   122.955       116   0.0801858     0.51106
   89      16      2763      2747   123.191       144     1.41941    0.510999
   90      16      2794      2778     123.2       124    0.275317     0.51053
   91      16      2824      2808   123.164       120     1.44233    0.510651
   92      16      2856      2840   123.217       128      0.2362    0.510411
   93      16      2888      2872   123.265       128      1.8657    0.509986
   94      16      2921      2905   123.356       132   0.0872072    0.509621
   95      15      2958      2943   123.628       152   0.0598051    0.508228
   96      16      2992      2976   123.715       132    0.107187    0.508602
   97      16      3020      3004   123.594       112    0.583915    0.508965
   98      16      3050      3034   123.557       120     1.37677    0.509671
   99      16      3080      3064    123.52       120     1.66099    0.509464
2020-06-08 20:25:16.031657 min lat: 0.0227928 max lat: 2.36752 avg lat: 0.509699
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3110      3094   123.484       120      1.6026    0.509699
  101      16      3144      3128   123.578       136   0.0654255    0.509263
  102      16      3175      3159   123.582       124    0.329964    0.509369
  103      16      3200      3184   123.353       100     0.21294    0.509977
  104      16      3231      3215   123.357       124     1.71388    0.510178
  105      16      3264      3248   123.439       132     1.72292    0.510106
  106      16      3289      3273   123.219       100   0.0469407    0.509794
  107      16      3324      3308   123.375       140   0.0559577    0.510508
  108      16      3354      3338   123.344       120     1.71473    0.510917
  109      16      3389      3373   123.496       140    0.256227    0.509952
  110      16      3422      3406   123.573       132   0.0522499    0.510271
  111      16      3454      3438   123.613       128   0.0741706    0.510096
  112      16      3485      3469   123.615       124    0.346869    0.510085
  113      16      3510      3494   123.406       100    0.408131    0.510194
  114      16      3537      3521   123.271       108    0.072818    0.511152
  115      16      3573      3557    123.41       144   0.0773894    0.509976
  116      16      3603      3587   123.381       120     1.61408    0.510598
  117      15      3637      3622   123.514       140   0.0639511    0.510057
  118      16      3671      3655   123.585       132     1.19071    0.510168
  119      16      3701      3685   123.555       120    0.258496    0.509793
2020-06-08 20:25:36.108114 min lat: 0.0227928 max lat: 2.36752 avg lat: 0.508928
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      15      3731      3716   123.558       124   0.0414287    0.508928
  121      15      3763      3748   123.594       128   0.0556417    0.509653
  122      16      3795      3779   123.592       124   0.0562003    0.510296
  123      16      3828      3812    123.66       132   0.0570671    0.510063
  124      16      3859      3843   123.663       124    0.262134    0.508943
  125      15      3888      3873   123.632       120     2.94952    0.509582
  126      16      3914      3898   123.445       100     0.42958    0.510385
  127      16      3943      3927   123.386       116     1.12855    0.511349
  128      16      3971      3955   123.297       112     1.13273     0.51139
  129      16      3998      3982   123.179       108     1.12062     0.51201
  130      16      4033      4017   123.305       140   0.0689844    0.511214
  131      16      4066      4050   123.371       132    0.297624    0.510419
  132      15      4095      4080   123.343       120   0.0498298    0.511205
  133      16      4126      4110   123.318       120    0.705154    0.511123
  134      16      4154      4138   123.233       112   0.0619335    0.511476
  135      16      4182      4166    123.15       112    0.154478    0.511353
  136      16      4212      4196   123.127       120     1.46591    0.512014
  137      16      4238      4222   122.988       104    0.470556    0.512629
  138      15      4269      4254   123.018       128    0.109019    0.512289
  139      16      4303      4287   123.082       132    0.210095    0.511941
2020-06-08 20:25:56.129699 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.511053
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4340      4324    123.26       148   0.0884521    0.511053
  141      16      4367      4351   123.151       108     1.55959    0.511664
  142      16      4394      4378   123.045       108     0.19392    0.511737
  143      16      4426      4410   123.079       128     1.48088    0.511844
  144      16      4455      4439    123.03       116    0.443785    0.512202
  145      16      4486      4470   123.037       124   0.0610501    0.512105
  146      16      4522      4506    123.18       144     0.64723    0.511634
  147      16      4552      4536   123.158       120    0.164321    0.511933
  148      16      4579      4563   123.056       108    0.217635    0.511822
  149      16      4614      4598   123.169       140     2.19509    0.511897
  150      15      4643      4628   123.146       120    0.209133    0.511937
  151      16      4675      4659   123.152       124    0.575137    0.512007
  152      16      4706      4690   123.157       124   0.0768242    0.512139
  153      16      4736      4720   123.136       120   0.0726477    0.511735
  154      15      4766      4751    123.14       124    0.283298    0.511607
  155      16      4801      4785   123.222       136    0.964164    0.512191
  156      16      4833      4817   123.253       128    0.325812    0.512042
  157      16      4860      4844   123.156       108    0.297552    0.512269
  158      16      4889      4873    123.11       116     0.46431    0.512605
  159      16      4916      4900   123.015       108    0.511602    0.513101
2020-06-08 20:26:16.144374 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.513083
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      4943      4927   122.916       108     1.23488    0.513083
  161      16      4974      4958   122.923       124    0.673927    0.512792
  162      16      5006      4990   122.945       128     1.60016    0.513208
  163      16      5036      5020   122.927       120    0.306099    0.513255
  164      16      5066      5050   122.909       120    0.066428    0.512953
  165      16      5095      5079   122.867       116    0.389608    0.512774
  166      16      5126      5110   122.865       124    0.302484    0.512145
  167      16      5163      5147   123.015       148    0.247665    0.512363
  168      16      5196      5180   123.068       132     1.96385    0.512461
  169      16      5227      5211   123.073       124    0.171269    0.512254
  170      16      5260      5244   123.125       132   0.0228844    0.512298
  171      16      5287      5271   123.036       108     1.03666    0.512897
  172      16      5318      5302   123.041       124    0.104699    0.512502
  173      16      5349      5333   123.047       124    0.263161    0.512221
  174      16      5379      5363   123.012       120    0.210632    0.512855
  175      16      5410      5394   123.017       124    0.289337    0.512695
  176      15      5442      5427   123.066       132    0.353725    0.512443
  177      16      5470      5454   122.981       108    0.215974    0.512398
  178      15      5502      5487   123.031       132     1.60185    0.513026
  179      16      5531      5515   122.969       112     1.55787    0.512736
2020-06-08 20:26:36.202581 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.512982
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5562      5546   122.975       124   0.0278812    0.512982
  181      15      5592      5577   122.979       124    0.153238     0.51253
  182      16      5626      5610   123.028       132     1.34885    0.512791
  183      16      5656      5640   123.012       120    0.446493    0.512759
  184      16      5683      5667   122.914       108    0.276121    0.513173
  185      16      5712      5696   122.876       116     0.11302    0.513048
  186      16      5744      5728   122.903       128    0.163667    0.512958
  187      16      5773      5757   122.865       116    0.246265    0.513342
  188      16      5802      5786   122.827       116      1.2992    0.513934
  189      16      5828      5812   122.727       104    0.411384    0.513875
  190      16      5858      5842   122.712       120    0.226328    0.513802
  191      16      5891      5875    122.76       132     1.93024    0.514104
  192      15      5918      5903   122.686       112    0.334509    0.514099
  193      16      5955      5939   122.796       144    0.232033    0.513861
  194      16      5991      5975   122.905       144     1.82599    0.513048
  195      16      6023      6007   122.931       128    0.185962    0.513075
  196      16      6057      6041   122.998       136    0.388207    0.512848
  197      16      6089      6073   123.004       128      1.9637    0.512527
  198      15      6118      6103   122.987       120     2.11486    0.513053
  199      16      6147      6131   122.932       112    0.319751    0.513075
2020-06-08 20:26:56.300622 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.513492
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6179      6163   122.957       128    0.452565    0.513492
  201      16      6213      6197   123.022       136    0.623237    0.512988
  202      16      6242      6226   122.987       116     0.22832    0.512766
  203      15      6270      6255   122.953       116     1.76105    0.513175
  204      16      6301      6285   122.938       120   0.0488869    0.513091
  205      16      6333      6317   122.962       128    0.176727    0.513104
  206      16      6365      6349   122.986       128    0.474029     0.51275
  207      16      6396      6380    122.99       124    0.358129    0.513158
  208      16      6429      6413   123.033       132    0.165151    0.512857
  209      16      6459      6443   123.019       120    0.227616    0.512817
  210      16      6490      6474   123.023       124    0.193078    0.513148
  211      16      6517      6501   122.951       108    0.295597    0.512844
  212      16      6545      6529     122.9       112   0.0520579    0.513129
  213      15      6577      6562   122.942       132   0.0428166    0.513279
  214      16      6609      6593   122.947       124   0.0898623    0.513379
  215      16      6638      6622   122.914       116    0.351055    0.513152
  216      16      6664      6648   122.827       104    0.457336    0.513379
  217      16      6694      6678   122.814       120    0.757288    0.514233
  218      16      6723      6707   122.783       116   0.0415485    0.514156
  219      16      6747      6731    122.66        96    0.079386    0.514509
2020-06-08 20:27:16.311787 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515177
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      6775      6759    122.61       112    0.460664    0.515177
  221      16      6802      6786   122.544       108   0.0552546    0.515491
  222      16      6834      6818   122.568       128   0.0598628    0.515325
  223      16      6858      6842   122.449        96    0.663182    0.515606
  224      16      6890      6874    122.46       128    0.769975    0.515737
  225      16      6920      6904   122.448       120    0.752443    0.515702
  226      16      6948      6932   122.402       112    0.597319    0.515866
  227      16      6980      6964   122.426       128   0.0529803    0.515565
  228      16      7011      6995   122.432       124   0.0843282    0.515687
  229      15      7044      7029   122.486       136   0.0415242    0.515231
  230      16      7070      7054   122.388       100    0.300856    0.515405
  231      16      7103      7087    122.43       132    0.275401    0.515418
  232      16      7132      7116   122.402       116    0.426996    0.515655
  233      16      7165      7149   122.443       132   0.0698307    0.515385
  234      16      7195      7179   122.433       120      1.3029    0.515821
  235      16      7225      7209   122.422       120    0.736337    0.515593
  236      16      7256      7240   122.422       124   0.0908626    0.515584
  237      16      7286      7270   122.396       120    0.250061    0.515574
  238      16      7315      7299   122.369       116   0.0436245    0.515432
  239      16      7347      7331   122.391       128    0.479437    0.515564
2020-06-08 20:27:36.400361 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515349
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7379      7363   122.414       128    0.073707    0.515349
  241      16      7411      7395   122.437       128    0.176264    0.515559
  242      16      7437      7421   122.361       104     1.60691    0.515857
  243      16      7466      7450   122.335       116    0.392598    0.515885
  244      16      7501      7485   122.407       140    0.126986    0.515987
  245      16      7532      7516   122.413       124    0.260484    0.515732
  246      16      7564      7548   122.436       128     1.38074    0.515799
  247      16      7591      7575   122.377       108    0.390964    0.515951
  248      16      7620      7604   122.352       116    0.232344    0.515948
  249      16      7650      7634   122.342       120      1.2388    0.516239
  250      15      7681      7666   122.365       128    0.130807    0.516233
  251      16      7718      7702   122.451       144    0.039761    0.515476
  252      16      7747      7731   122.425       116    0.286396    0.515473
  253      16      7777      7761   122.415       120     1.76953    0.515806
  254      16      7811      7795   122.468       136    0.156431    0.515406
  255      16      7839      7823   122.427       112      1.6787    0.515651
  256      16      7867      7851   122.387       112     0.26108    0.515656
  257      15      7895      7880   122.361       116    0.229928    0.516206
  258      15      7924      7909   122.336       116   0.0847597    0.516009
  259      16      7954      7938   122.311       116    0.410862    0.516013
2020-06-08 20:27:56.409067 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.515997
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      15      7981      7966   122.271       112    0.406446    0.515997
  261      16      8019      8003   122.368       148    0.190577    0.516212
  262      16      8047      8031   122.329       112    0.260786    0.516008
  263      16      8077      8061    122.32       120     1.77423    0.516061
  264      16      8108      8092   122.326       124    0.320483    0.516016
  265      16      8137      8121   122.302       116    0.480798    0.516348
  266      16      8165      8149   122.264       112   0.0402743    0.516797
  267      16      8191      8175   122.195       104    0.383156     0.51675
  268      15      8219      8204   122.171       116    0.254408    0.517035
  269      16      8250      8234   122.163       120    0.100691    0.516921
  270      16      8280      8264   122.155       120   0.0998158    0.516964
  271      16      8313      8297   122.191       132    0.551078    0.516838
  272      16      8341      8325   122.153       112     1.91776    0.516964
  273      16      8371      8355   122.145       120    0.702243    0.517167
  274      16      8403      8387   122.167       128   0.0521393     0.51697
  275      16      8431      8415    122.13       112    0.170423    0.516868
  276      16      8461      8445   122.122       120    0.301784     0.51688
  277      16      8489      8473   122.085       112   0.0697051    0.516845
  278      16      8519      8503   122.078       120     0.22144    0.517237
  279      16      8549      8533    122.07       120     1.50557    0.517312
2020-06-08 20:28:16.417187 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.51703
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      8584      8568   122.134       140   0.0833517     0.51703
  281      16      8614      8598   122.127       120    0.268542    0.516842
  282      16      8646      8630   122.147       128     1.61622    0.517074
  283      16      8673      8657   122.097       108    0.306155    0.517013
  284      15      8699      8684   122.047       108    0.187822    0.517219
  285      16      8726      8710   121.983       104     1.71585    0.517776
  286      16      8758      8742   121.996       128     1.38552    0.517483
  287      16      8788      8772   121.989       120    0.584332    0.517755
  288      16      8814      8798   121.926       104     0.28312    0.517649
  289      16      8844      8828   121.918       120      1.8311    0.518047
  290      16      8871      8855    121.87       108    0.271836    0.517879
  291      16      8898      8882   121.822       108    0.237387    0.518344
  292      16      8933      8917   121.884       140     0.19795    0.518179
  293      16      8961      8945   121.849       112   0.0720802    0.518151
  294      16      8992      8976   121.856       124    0.377498    0.518005
  295      15      9023      9008   121.877       128    0.336213    0.518273
  296      16      9055      9039   121.883       124    0.421593    0.518064
  297      16      9088      9072   121.915       132    0.281163    0.517804
  298      16      9121      9105   121.949       132    0.264226    0.517848
  299      16      9155      9139   121.995       136   0.0452371    0.517625
2020-06-08 20:28:36.458626 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.517897
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9183      9167   121.962       112    0.796513    0.517897
  301      16      9212      9196   121.942       116    0.341818    0.517728
  302      16      9247      9231   122.002       140   0.0415116    0.517609
  303      16      9274      9258   121.956       108     0.28618    0.517839
  304      16      9304      9288   121.949       120    0.197714    0.517763
  305      16      9335      9319   121.956       124   0.0481885    0.517615
  306      16      9367      9351   121.972       128    0.461866    0.517715
  307      16      9398      9382   121.978       124     1.74955    0.517827
  308      16      9427      9411   121.959       116    0.603792    0.517872
  309      16      9458      9442   121.965       124    0.360782    0.517917
  310      16      9494      9478   122.036       144   0.0545681     0.51755
  311      15      9531      9516   122.131       152     1.58507    0.517139
  312      16      9563      9547   122.137       124    0.243046    0.517091
  313      16      9593      9577    122.13       120    0.328936     0.51697
  314      16      9625      9609   122.148       128     2.09227    0.517021
  315      16      9656      9640   122.154       124    0.232122    0.517195
  316      16      9686      9670   122.147       120     1.36692    0.517148
  317      16      9712      9696    122.09       104     1.64755    0.517278
  318      16      9744      9728   122.109       128    0.224697    0.517303
  319      16      9776      9760   122.127       128     0.20737    0.517126
2020-06-08 20:28:56.498477 min lat: 0.0227928 max lat: 2.94952 avg lat: 0.517035
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      15      9807      9792   122.136       128    0.236348    0.517035
  321      16      9835      9819   122.092       108     1.66787     0.51705
  322      16      9861      9845   122.036       104    0.236375    0.517296
  323      16      9895      9879   122.079       136    0.555548    0.517181
  324      16      9922      9906   122.036       108     1.50661    0.517683
  325      16      9946      9930   121.956        96    0.427905    0.517881
  326      16      9975      9959   121.938       116     0.56027    0.518102
  327      16     10005      9989   121.932       120    0.304654     0.51816
  328      16     10035     10019   121.926       120    0.608291    0.518218
  329      16     10065     10049    121.92       120    0.256241    0.518068
  330      16     10090     10074   121.853       100    0.339154    0.518257
  331      16     10121     10105    121.86       124     0.26115    0.518408
  332      16     10150     10134   121.842       116    0.569751    0.518405
  333      16     10180     10164   121.836       120    0.831929    0.518518
  334      16     10213     10197   121.867       132    0.267892    0.518477
  335      16     10243     10227   121.861       120    0.258912     0.51845
  336      16     10275     10259   121.877       128    0.299752    0.518122
  337      16     10305     10289   121.871       120    0.291985    0.518174
  338      16     10337     10321   121.889       128   0.0579847     0.51824
  339      16     10366     10350   121.859       116    0.202518    0.518286
2020-06-08 20:29:16.545616 min lat: 0.0227802 max lat: 2.94952 avg lat: 0.518194
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     10396     10380   121.853       120    0.229475    0.518194
  341      16     10425     10409   121.836       116   0.0529861    0.518389
  342      15     10459     10444   121.889       140    0.316579    0.518168
  343      16     10492     10476   121.906       128   0.0581969    0.518135
  344      16     10523     10507   121.912       124    0.244396    0.518129
  345      16     10551     10535   121.884       112     1.77602    0.518198
  346      16     10580     10564   121.867       116    0.601367    0.518402
  347      16     10611     10595   121.873       124    0.414638    0.518427
  348      16     10644     10628   121.902       132   0.0398358    0.518336
  349      16     10674     10658   121.896       120    0.287942     0.51827
  350      16     10704     10688   121.891       120   0.0742775    0.518127
  351      16     10732     10716   121.862       112   0.0462632     0.51825
  352      16     10759     10743   121.815       108    0.437449    0.518427
  353      16     10793     10777   121.855       136     1.12846    0.518502
  354      16     10828     10812   121.906       140     1.18425    0.518289
  355      16     10853     10837   121.845       100   0.0665273    0.518143
  356      16     10884     10868   121.842       124    0.323274    0.518262
  357      16     10909     10893   121.781       100    0.719393    0.518434
  358      15     10935     10920   121.741       108     0.59949    0.518959
  359      16     10965     10949   121.725       116    0.789253       0.519
2020-06-08 20:29:36.601742 min lat: 0.0227802 max lat: 2.94952 avg lat: 0.518776
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     10999     10983   121.765       136    0.246673    0.518776
  361      16     11032     11016   121.793       132    0.274904    0.518609
  362      15     11055     11040   121.722        96   0.0521564    0.518555
Total time run:       363.124
Total reads made:     11055
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   121.777
Average IOPS:         30
Stddev IOPS:          2.75726
Max IOPS:             39
Min IOPS:             24
Average Latency(s):   0.519379
Max latency(s):       2.94952
Min latency(s):       0.0227471

Total time run:       363.124
Total reads made:     11055
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   121.777
Average IOPS:         30
Stddev IOPS:          2.75726
Max IOPS:             39
Min IOPS:             24
Average Latency(s):   0.519379
Max latency(s):       2.94952
Min latency(s):       0.0227471

Random Read Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench2 600 rand | tee --append $(date +%F)
  -rados-test-read-rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        51        35   139.943       140   0.0586091    0.277962
    2      16        81        65   129.966       120    0.066807    0.375681
    3      16       113        97   129.305       128    0.268188    0.402898
    4      16       153       137   136.964       160     1.44037    0.407198
    5      16       182       166   132.741       116   0.0480781    0.419728
    6      16       210       194   129.142       112   0.0466207    0.451808
    7      16       244       228   130.118       136    0.829422    0.455489
    8      16       271       255   127.354       108   0.0613913    0.467553
    9      16       305       289   128.311       136    0.516398    0.463554
   10      16       335       319   127.479       120    0.355922    0.466055
   11      16       366       350   127.161       124    0.293341    0.473895
   12      15       396       381   126.895       124    0.580237     0.48482
   13      16       427       411    126.34       120    0.570833    0.483577
   14      16       461       445   127.028       136     1.52073    0.482284
   15      16       489       473   126.026       112    0.536009    0.483434
   16      15       521       506   126.398       132    0.776196    0.486821
   17      16       552       536    126.02       120    0.085827    0.486352
   18      16       581       565   125.456       116    0.256079    0.488214
   19      15       611       596   125.373       124    0.701293    0.495113
2020-06-08 20:37:28.247285 min lat: 0.00767023 max lat: 2.2389 avg lat: 0.495997
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       640       624   124.703       112     1.56505    0.495997
   21      16       674       658    125.24       136    0.425942    0.495574
   22      16       707       691   125.546       132     1.47021    0.495525
   23      15       734       719   124.903       112    0.314597    0.497891
   24      16       766       750   124.865       124     1.09931     0.49645
   25      16       796       780   124.669       120    0.273213    0.495906
   26      16       825       809   124.335       116    0.264268    0.496062
   27      16       858       842   124.618       132    0.444402    0.495716
   28      16       894       878   125.304       144    0.280718    0.495399
   29      15       924       909   125.258       124    0.407136    0.495202
   30      16       956       940   125.216       124     1.20638    0.496961
   31      16       988       972   125.305       128    0.582289     0.49698
   32      16      1020      1004   125.388       128    0.459654     0.49613
   33      16      1050      1034   125.225       120     0.35612    0.496381
   34      16      1081      1065   125.188       124    0.305698    0.495892
   35      16      1111      1095   124.907       120    0.362349    0.495583
   36      16      1141      1125   124.771       120    0.279651    0.497179
   37      16      1170      1154   124.534       116    0.269168    0.500527
   38      16      1204      1188   124.823       136    0.368712    0.498541
   39      16      1239      1223   125.211       140    0.324084    0.498661
2020-06-08 20:37:48.302970 min lat: 0.0075863 max lat: 2.44289 avg lat: 0.499018
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1266      1250   124.778       108    0.425744    0.499018
   41      15      1298      1283   124.953       132   0.0877296    0.502065
   42      16      1326      1310   124.547       108    0.436453    0.501906
   43      16      1355      1339   124.345       116   0.0865259    0.503146
   44      15      1388      1373   124.589       136    0.282347    0.502295
   45      16      1420      1404   124.575       124     2.15099    0.501313
   46      16      1449      1433   124.389       116    0.184875    0.502487
   47      16      1478      1462    124.21       116    0.275427    0.501894
   48      16      1509      1493   124.206       124   0.0566634    0.503259
   49      16      1540      1524   124.201       124     1.95947     0.50452
   50      16      1571      1555   124.197       124    0.186263     0.50492
   51      16      1602      1586   124.192       124    0.599994    0.504414
   52      16      1633      1617   124.188       124     0.25071    0.503623
   53      16      1672      1656   124.787       156    0.100314    0.502715
   54      16      1698      1682   124.403       104   0.0575364    0.505972
   55      15      1727      1712   124.323       120    0.776987    0.506102
   56      16      1763      1747   124.602       140     1.28794    0.504847
   57      16      1792      1776   124.451       116   0.0664667    0.504545
   58      16      1819      1803   124.167       108       1.718    0.505636
   59      16      1847      1831   123.961       112    0.390346    0.506569
2020-06-08 20:38:08.314952 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.506042
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      15      1881      1866   124.228       140   0.0079828    0.506042
   61      16      1915      1899   124.355       132    0.159932    0.505287
   62      15      1945      1930   124.349       124    0.213707    0.504482
   63      16      1977      1961   124.342       124   0.0752758    0.505421
   64      16      2014      1998   124.711       148   0.0680605    0.504238
   65      16      2041      2025   124.454       108     0.27941    0.503864
   66      15      2070      2055   124.386       120    0.551604    0.504859
   67      16      2103      2087    124.44       128     1.41295    0.506116
   68      16      2131      2115   124.257       112    0.790664    0.506181
   69      16      2161      2145   124.192       120    0.293269    0.505521
   70      15      2190      2175   124.132       120   0.0441665    0.506671
   71      16      2219      2203   123.961       112    0.454395    0.507879
   72      16      2244      2228   123.628       100     0.56955    0.509076
   73      16      2277      2261   123.742       132   0.0787766     0.50831
   74      16      2304      2288    123.53       108   0.0784757    0.508522
   75      16      2342      2326   123.909       152   0.0994228    0.508304
   76      16      2373      2357    123.91       124       0.206     0.50693
   77      16      2400      2384   123.703       108    0.499882    0.506759
   78      15      2427      2412   123.553       112    0.180056    0.509061
   79      16      2462      2446    123.71       136   0.0776175     0.50797
2020-06-08 20:38:28.320191 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.50781
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2495      2479   123.813       132      1.9827     0.50781
   81      16      2528      2512   123.914       132   0.0470868    0.507869
   82      16      2554      2538   123.671       104     1.09973    0.509034
   83      16      2586      2570   123.721       128    0.471985    0.508107
   84      16      2616      2600   123.676       120      2.2733    0.508633
   85      16      2647      2631    123.68       124   0.0991686    0.508681
   86      16      2679      2663    123.73       128   0.0486071    0.508748
   87      16      2707      2691   123.595       112   0.0563499    0.508083
   88      16      2740      2724    123.69       132    0.329987    0.509494
   89      16      2770      2754   123.648       120    0.191511    0.508935
   90      16      2800      2784   123.608       120     1.20885    0.509202
   91      16      2832      2816   123.656       128    0.239141    0.508898
   92      16      2864      2848   123.703       128   0.0446796     0.50945
   93      16      2898      2882   123.835       136   0.0763327     0.50761
   94      16      2930      2914   123.879       128    0.510066    0.508343
   95      16      2964      2948   124.006       136    0.210589    0.507625
   96      16      2991      2975   123.839       108     1.18171    0.508812
   97      16      3020      3004   123.758       116    0.333442    0.508956
   98      16      3054      3038   123.883       136   0.0905627    0.508407
   99      16      3084      3068   123.843       120    0.379993      0.5087
2020-06-08 20:38:48.325116 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508697
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3118      3102   123.964       136     0.23994    0.508697
  101      16      3149      3133   123.964       124     0.21101    0.508078
  102      16      3180      3164   123.965       124     0.37649    0.509156
  103      15      3207      3192   123.848       112    0.217731     0.50909
  104      16      3239      3223    123.85       124   0.0523112    0.508872
  105      16      3274      3258   124.003       140   0.0490936    0.507924
  106      16      3302      3286    123.89       112    0.407885    0.508503
  107      15      3331      3316   123.853       120    0.286567    0.508713
  108      16      3364      3348   123.892       128     1.77374    0.509035
  109      16      3392      3376   123.782       112   0.0684978    0.509142
  110      16      3424      3408    123.82       128   0.0692926    0.509873
  111      16      3455      3439   123.822       124     1.54867    0.508961
  112      16      3488      3472   123.895       132    0.715173    0.508953
  113      16      3519      3503   123.895       124    0.642609    0.508741
  114      16      3553      3537   124.001       136    0.514705    0.508665
  115      16      3586      3570   124.069       132    0.226899    0.508361
  116      16      3613      3597   123.931       108    0.228104     0.50851
  117      16      3646      3630   123.999       132     1.63747    0.508497
  118      16      3675      3659    123.93       116    0.271013    0.508356
  119      16      3705      3689   123.897       120   0.0547901    0.507972
2020-06-08 20:39:08.331750 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508951
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      15      3739      3724    124.03       140    0.232311    0.508951
  121      16      3770      3754   123.997       120     1.67157    0.508966
  122      16      3796      3780   123.833       104     0.20622    0.508606
  123      16      3825      3809   123.769       116     2.27719    0.509577
  124      16      3859      3843   123.867       136   0.0685277    0.509121
  125      15      3888      3873   123.835       120    0.272679    0.509094
  126      16      3919      3903   123.805       120     1.47513    0.509965
  127      15      3949      3934   123.806       124  0.00774259    0.509008
  128      16      3979      3963   123.745       116   0.0644373    0.509208
  129      16      4013      3997   123.839       136   0.0466334    0.509202
  130      16      4042      4026   123.779       116    0.311963      0.5092
  131      15      4073      4058   123.811       128     1.58711    0.509654
  132      16      4104      4088   123.782       120    0.244707    0.509497
  133      16      4131      4115   123.663       108   0.0986581    0.510015
  134      16      4161      4145   123.636       120    0.324041    0.510727
  135      16      4194      4178   123.697       132    0.324672    0.510124
  136      15      4223      4208    123.67       120      2.0338     0.51089
  137      16      4256      4240   123.702       128   0.0555726    0.510462
  138      16      4287      4271   123.704       124    0.383916    0.510635
  139      16      4319      4303   123.734       128    0.284673    0.510007
2020-06-08 20:39:28.336477 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.510037
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4350      4334   123.736       124    0.375594    0.510037
  141      16      4380      4364   123.709       120    0.175716    0.509952
  142      16      4410      4394   123.683       120   0.0568104    0.510282
  143      16      4444      4428   123.769       136   0.0601803    0.509923
  144      16      4475      4459   123.771       124    0.305176    0.509981
  145      16      4506      4490   123.772       124     0.28875    0.509653
  146      16      4539      4523   123.828       132     1.11541    0.510107
  147      15      4573      4558   123.938       140    0.318495    0.509141
  148      15      4607      4592   124.019       136     1.52266    0.509268
  149      16      4639      4623   124.019       124     1.42369    0.509024
  150      16      4667      4651   123.939       112    0.157892    0.508891
  151      16      4698      4682   123.939       124    0.317812      0.5092
  152      15      4727      4712   123.913       120     1.73716    0.509819
  153      16      4757      4741   123.861       116    0.399952    0.509763
  154      16      4794      4778   124.018       148    0.405619    0.509288
  155      16      4825      4809   124.018       124    0.699032    0.509096
  156      16      4859      4843   124.094       136   0.0835734    0.508968
  157      16      4887      4871   124.017       112   0.0672621    0.508836
  158      16      4918      4902   124.017       124    0.245538    0.509377
  159      16      4958      4942   124.243       160    0.104857    0.508502
2020-06-08 20:39:48.339445 min lat: 0.0075863 max lat: 2.66352 avg lat: 0.508473
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      4986      4970   124.166       112    0.207302    0.508473
  161      16      5015      4999   124.116       116   0.0557762    0.508803
  162      16      5048      5032   124.164       132   0.0650655    0.508544
  163      15      5074      5059   124.065       108    0.211816    0.508679
  164      16      5107      5091   124.089       128   0.0534329    0.509043
  165      16      5145      5129   124.258       152    0.206019    0.508435
  166      16      5174      5158   124.208       116   0.0635734    0.508493
  167      16      5202      5186   124.135       112    0.321672    0.508552
  168      16      5232      5216    124.11       120     1.85804    0.508933
  169      16      5263      5247   124.109       124   0.0518495    0.508773
  170      16      5294      5278   124.109       124    0.400471    0.508858
  171      16      5324      5308   124.084       120    0.590401    0.509171
  172      16      5352      5336   124.014       112    0.389228    0.509427
  173      16      5381      5365   123.968       116    0.226304    0.509386
  174      16      5417      5401   124.083       144    0.228978    0.509089
  175      16      5445      5429   124.014       112     0.58449    0.509205
  176      16      5476      5460   124.013       124     1.33933    0.509297
  177      16      5505      5489   123.968       116   0.0111495    0.509446
  178      16      5535      5519   123.946       120     0.30085    0.509571
  179      15      5564      5549   123.923       120    0.185957    0.509627
2020-06-08 20:40:08.342729 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.509528
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5597      5581   123.946       128    0.275075    0.509528
  181      15      5629      5614    123.99       132    0.398652    0.509065
  182      16      5662      5646   124.012       128    0.279213    0.509617
  183      16      5695      5679   124.056       132    0.315781    0.509399
  184      16      5728      5712   124.099       132   0.0618818      0.5089
  185      16      5757      5741   124.055       116    0.170632    0.509219
  186      16      5789      5773   124.075       128    0.052908     0.50922
  187      16      5822      5806   124.117       132   0.0700102    0.509207
  188      16      5852      5836   124.095       120     1.34414    0.508999
  189      16      5882      5866   124.073       120    0.340822    0.508673
  190      15      5916      5901   124.156       140    0.587977    0.508722
  191      16      5946      5930   124.113       116    0.173819     0.50917
  192      16      5983      5967   124.238       148    0.408126    0.508085
  193      16      6017      6001   124.298       136     2.29032    0.507992
  194      15      6046      6031   124.276       120     1.46933     0.50861
  195      16      6079      6063   124.295       128    0.335119     0.50832
  196      16      6106      6090   124.211       108      1.4968    0.508679
  197      16      6136      6120    124.19       120    0.246017    0.508745
  198      16      6168      6152   124.209       128   0.0817655    0.508565
  199      16      6195      6179   124.128       108      1.5363    0.509042
2020-06-08 20:40:28.350671 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.508874
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6225      6209   124.106       120    0.222804    0.508874
  201      16      6257      6241   124.123       128     0.53307    0.509389
  202      16      6284      6268   124.043       108    0.374498    0.509456
  203      16      6314      6298   124.023       120   0.0111701    0.509575
  204      16      6341      6325   123.945       108    0.265543    0.509497
  205      16      6373      6357   123.965       128   0.0546058    0.509751
  206      15      6404      6389   123.984       128    0.394781    0.509861
  207      15      6432      6417   123.926       112     1.78949    0.510346
  208      16      6468      6452   124.003       140    0.193595     0.50978
  209      16      6500      6484   124.022       128    0.289926    0.509269
  210      16      6532      6516    124.04       128     2.12999    0.509346
  211      16      6560      6544   123.983       112    0.160417    0.509588
  212      16      6587      6571   123.907       108    0.189348    0.509916
  213      16      6620      6604   123.945       132     1.54803    0.509682
  214      16      6651      6635   123.945       124    0.267665    0.509534
  215      16      6682      6666   123.946       124   0.0804845    0.509949
  216      16      6712      6696   123.927       120      1.4027     0.50989
  217      16      6744      6728   123.946       128    0.207887    0.509583
  218      16      6776      6760   123.963       128     1.45251    0.509927
  219      16      6808      6792   123.981       128   0.0771457    0.509744
2020-06-08 20:40:48.361940 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.509848
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      6841      6825   124.018       132     1.40176    0.509848
  221      16      6871      6855   123.999       120    0.153146    0.509766
  222      15      6905      6890   124.071       140    0.236409    0.509499
  223      16      6937      6921    124.07       124     2.17101    0.509445
  224      16      6972      6956   124.141       140     1.75282    0.509013
  225      16      7001      6985   124.105       116    0.051531    0.509253
  226      15      7030      7015   124.087       120    0.420543    0.509651
  227      16      7065      7049   124.139       136    0.423939    0.509544
  228      16      7102      7086   124.244       148    0.353998    0.508848
  229      16      7131      7115   124.206       116    0.301212    0.508678
  230      16      7160      7144    124.17       116   0.0625969    0.508972
  231      16      7191      7175   124.153       124    0.332617    0.509179
  232      15      7223      7208   124.187       132     1.13588    0.509092
  233      16      7252      7236   124.134       112    0.174538    0.508877
  234      16      7281      7265   124.099       116    0.213757     0.50902
  235      16      7310      7294   124.065       116     1.64307    0.509362
  236      16      7334      7318   123.946        96    0.339064     0.50974
  237      16      7366      7350   123.963       128    0.167864    0.510063
  238      16      7396      7380   123.946       120    0.238383    0.509722
  239      16      7424      7408   123.896       112    0.370573    0.510142
2020-06-08 20:41:08.400492 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510074
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      15      7456      7441    123.93       132    0.085039    0.510074
  241      16      7491      7475    123.98       136   0.0534243    0.509853
  242      16      7526      7510   124.046       140     1.07023    0.509888
  243      16      7554      7538   123.996       112    0.307925     0.50968
  244      16      7583      7567   123.963       116    0.373594    0.509595
  245      16      7611      7595   123.914       112    0.363044    0.510341
  246      16      7639      7623   123.866       112     1.42921    0.510401
  247      16      7671      7655   123.882       128    0.485589    0.510211
  248      16      7701      7685   123.866       120    0.168639    0.510337
  249      16      7727      7711   123.787       104     1.37117    0.510641
  250      16      7759      7743   123.803       128    0.248629    0.510484
  251      16      7791      7775    123.82       128     1.42661    0.510676
  252      16      7827      7811     123.9       144    0.502747    0.510141
  253      15      7854      7839   123.853       112   0.0930292    0.510181
  254      16      7885      7869   123.838       120    0.339748    0.510154
  255      16      7918      7902    123.87       132     0.20261    0.510449
  256      16      7948      7932   123.854       120   0.0505014    0.510476
  257      16      7978      7962   123.839       120     0.29542    0.510559
  258      16      8005      7989   123.778       108      1.5502    0.510944
  259      16      8036      8020   123.779       124   0.0426524     0.51089
2020-06-08 20:41:28.407660 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510816
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      15      8068      8053   123.809       132     1.58499    0.510816
  261      16      8098      8082   123.779       116    0.302931    0.510664
  262      16      8128      8112   123.764       120    0.460322    0.510907
  263      16      8158      8142    123.75       120    0.297061      0.5111
  264      16      8187      8171    123.72       116   0.0733328    0.511164
  265      16      8225      8209   123.827       152   0.0473569    0.510416
  266      16      8252      8236   123.767       108     2.20866    0.510748
  267      16      8286      8270   123.813       136   0.0572092    0.510573
  268      16      8315      8299   123.784       116     1.75067    0.510689
  269      16      8345      8329   123.769       120     1.65631    0.510747
  270      16      8382      8366   123.858       148   0.0979424    0.510329
  271      16      8411      8395   123.829       116   0.0590657    0.510407
  272      16      8437      8421   123.757       104    0.275211    0.510897
  273      15      8467      8452   123.757       124    0.297085    0.510831
  274      16      8499      8483   123.757       124   0.0824411    0.510844
  275      16      8531      8515   123.772       128   0.0639115    0.510776
  276      16      8563      8547   123.787       128    0.010436    0.510743
  277      16      8591      8575   123.745       112    0.232358     0.51088
  278      16      8623      8607    123.76       128    0.427809    0.510844
  279      16      8659      8643   123.832       144    0.103848    0.510587
2020-06-08 20:41:48.418451 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.51031
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      8695      8679   123.903       144    0.139037     0.51031
  281      16      8730      8714    123.96       140   0.0542311    0.510136
  282      16      8761      8745    123.96       124    0.511144    0.510336
  283      16      8791      8775   123.946       120   0.0513455    0.510252
  284      16      8823      8807    123.96       128    0.221567    0.510231
  285      16      8852      8836   123.932       116   0.0483402    0.510023
  286      16      8877      8861   123.849       100       1.614    0.510573
  287      15      8910      8895   123.891       136    0.510751     0.51047
  288      16      8940      8924   123.864       116     1.17024    0.510512
  289      16      8966      8950   123.795       104     1.43761    0.510837
  290      16      8996      8980   123.782       120     0.20469    0.510744
  291      15      9025      9010   123.769       120       1.318    0.510886
  292      16      9057      9041   123.769       124    0.276344    0.510791
  293      16      9085      9069   123.729       112    0.343372    0.510957
  294      15      9115      9100   123.729       124    0.235895    0.511052
  295      16      9145      9129   123.703       116   0.0550538     0.51104
  296      16      9176      9160   123.704       124    0.292128    0.511164
  297      15      9204      9189   123.678       116     1.50093    0.511552
  298      16      9233      9217   123.639       112     1.03965    0.511498
  299      16      9265      9249   123.653       128    0.126052    0.511238
2020-06-08 20:42:08.424242 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511379
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9294      9278   123.627       116    0.220644    0.511379
  301      16      9325      9309   123.628       124    0.291652    0.511307
  302      16      9355      9339   123.616       120   0.0227457      0.5116
  303      16      9387      9371   123.631       128    0.628228     0.51152
  304      16      9414      9398   123.579       108    0.052724    0.511771
  305      16      9448      9432    123.62       136    0.404111    0.511363
  306      15      9481      9466    123.66       136      1.4108    0.511454
  307      16      9512      9496   123.648       120    0.232413    0.511393
  308      15      9547      9532   123.713       144    0.274889    0.511283
  309      16      9580      9564   123.726       128   0.0500387    0.510885
  310      15      9612      9597   123.753       132    0.273428     0.51064
  311      16      9648      9632   123.805       140   0.0607635    0.510788
  312      16      9678      9662   123.793       120   0.0657142     0.51075
  313      15      9711      9696   123.831       136     0.29422    0.510858
  314      16      9740      9724   123.793       112    0.167771    0.510908
  315      16      9772      9756   123.807       128     1.47696    0.510824
  316      16      9801      9785   123.782       116    0.262162    0.510938
  317      16      9832      9816   123.783       124    0.262578    0.510694
  318      16      9862      9846   123.771       120     1.92793    0.511048
  319      16      9893      9877   123.771       124   0.0111321    0.510698
2020-06-08 20:42:28.433528 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510866
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      9928      9912   123.822       140   0.0747197    0.510866
  321      16      9959      9943   123.822       124   0.0622337    0.510632
  322      16      9990      9974   123.823       124     1.15184    0.510831
  323      16     10018     10002   123.786       112    0.283216    0.510686
  324      15     10052     10037   123.836       140   0.0570723    0.510882
  325      16     10084     10068   123.837       124    0.180581    0.510689
  326      16     10114     10098   123.825       120     1.57316    0.510687
  327      16     10142     10126   123.787       112   0.0700195    0.510686
  328      16     10175     10159   123.812       132    0.078846    0.510876
  329      16     10211     10195   123.874       144     1.66505    0.510516
  330      15     10238     10223   123.838       112    0.237626    0.510647
  331      16     10267     10251   123.802       112    0.242313    0.510869
  332      16     10299     10283   123.814       128     1.59151    0.511055
  333      16     10333     10317   123.851       136    0.307842     0.51065
  334      16     10365     10349   123.863       128     1.69979    0.510759
  335      16     10396     10380   123.864       124     1.42013     0.51076
  336      16     10426     10410   123.852       120    0.291663    0.510681
  337      16     10456     10440   123.841       120    0.305409    0.510878
  338      16     10485     10469   123.817       116    0.799237    0.510997
  339      16     10519     10503   123.852       136    0.054971    0.511053
2020-06-08 20:42:48.443454 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510834
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     10552     10536   123.876       132    0.312648    0.510834
  341      16     10581     10565   123.853       116    0.342335    0.510661
  342      16     10614     10598   123.876       132   0.0851834    0.510513
  343      15     10644     10629   123.877       124    0.425976    0.510556
  344      16     10677     10661   123.889       128    0.333656    0.510431
  345      16     10710     10694   123.912       132    0.237909     0.51058
  346      16     10738     10722   123.878       112    0.267664    0.510516
  347      16     10766     10750   123.843       112   0.0632882     0.51055
  348      15     10793     10778   123.809       112    0.721677    0.510889
  349      16     10827     10811   123.833       132   0.0511126    0.510809
  350      16     10860     10844   123.856       132   0.0960083    0.510583
  351      16     10891     10875   123.856       124    0.272481    0.510424
  352      15     10923     10908   123.879       132    0.523735    0.510421
  353      16     10955     10939    123.88       124    0.203108    0.510501
  354      16     10987     10971   123.891       128    0.345238    0.510592
  355      16     11020     11004   123.914       132    0.307542    0.510327
  356      16     11051     11035   123.914       124   0.0802598    0.510384
  357      16     11080     11064   123.892       116      1.3456    0.510543
  358      16     11109     11093    123.87       116    0.518924    0.510555
  359      16     11144     11128   123.915       140    0.726929    0.510407
2020-06-08 20:43:08.446750 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510504
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11175     11159   123.915       124    0.544431    0.510504
  361      16     11204     11188   123.893       116    0.823161    0.510755
  362      16     11232     11216    123.86       112   0.0609196    0.510956
  363      16     11263     11247    123.86       124     1.03556    0.510885
  364      16     11292     11276   123.839       116    0.323328    0.510859
  365      16     11326     11310   123.872       136   0.0742619    0.510547
  366      16     11362     11346   123.927       144   0.0560132    0.510614
  367      16     11393     11377   123.927       124    0.294645    0.510476
  368      16     11423     11407   123.916       120    0.431146    0.510378
  369      16     11457     11441   123.948       136    0.544814    0.510194
  370      16     11482     11466   123.884       100     2.14508    0.510682
  371      16     11515     11499   123.905       132    0.647952    0.510804
  372      16     11541     11525   123.851       104    0.977412    0.511017
  373      16     11572     11556   123.851       124   0.0453287    0.511044
  374      16     11602     11586    123.84       120    0.245995    0.510803
  375      16     11633     11617   123.841       124    0.243285    0.510837
  376      16     11660     11644   123.799       108   0.0433665    0.511015
  377      15     11685     11670   123.746       104     1.48479    0.511142
  378      16     11722     11706   123.799       144   0.0460282    0.511031
  379      16     11757     11741   123.842       140   0.0804824    0.510634
2020-06-08 20:43:28.456555 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510991
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     11787     11771   123.832       120    0.272876    0.510991
  381      15     11816     11801   123.822       120    0.395054    0.511066
  382      16     11849     11833   123.833       128   0.0641325    0.510832
  383      16     11881     11865   123.844       128     1.67749    0.510731
  384      15     11912     11897   123.854       128   0.0629009     0.51084
  385      16     11940     11924   123.813       108    0.192493    0.510849
  386      16     11974     11958   123.845       136   0.0761082    0.510637
  387      16     12004     11988   123.835       120    0.228055    0.510834
  388      16     12032     12016   123.804       112    0.754459    0.511176
  389      16     12063     12047   123.805       124     1.38822    0.511204
  390      16     12093     12077   123.795       120   0.0464053    0.511294
  391      15     12120     12105   123.765       112    0.827743    0.511319
  392      16     12151     12135   123.755       120    0.632207    0.511411
  393      16     12188     12172   123.817       148    0.204327    0.510969
  394      16     12218     12202   123.807       120     1.64355    0.511059
  395      16     12251     12235   123.826       132    0.328473    0.510752
  396      16     12286     12270   123.866       140    0.525551    0.510663
  397      16     12315     12299   123.847       116     1.74368    0.510904
  398      16     12350     12334   123.887       140    0.215078    0.510593
  399      16     12381     12365   123.887       124     0.34144    0.510754
2020-06-08 20:43:48.465409 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.510466
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12415     12399   123.918       136    0.257447    0.510466
  401      16     12446     12430   123.918       124      1.9718    0.510434
  402      15     12474     12459   123.898       116    0.253661    0.510514
  403      16     12504     12488   123.878       116    0.194725    0.510584
  404      16     12530     12514   123.829       104     2.13664    0.510738
  405      15     12557     12542     123.8       112     0.38411    0.510892
  406      15     12591     12576    123.83       136    0.264805    0.510878
  407      16     12617     12601   123.771       100    0.254247    0.511037
  408      16     12646     12630   123.752       116    0.254397    0.511287
  409      16     12674     12658   123.723       112   0.0555876    0.511232
  410      16     12703     12687   123.704       116    0.191088    0.511346
  411      16     12732     12716   123.686       116    0.594941     0.51133
  412      16     12758     12742   123.638       104    0.297959    0.511442
  413      16     12792     12776   123.668       136    0.292224    0.511596
  414      16     12824     12808   123.678       128    0.213199    0.511473
  415      16     12850     12834   123.631       104    0.214737    0.511679
  416      16     12881     12865   123.632       124    0.204426     0.51165
  417      16     12917     12901    123.68       144    0.128506    0.511486
  418      15     12944     12929   123.652       112   0.0426699    0.511671
  419      16     12974     12958   123.634       116   0.0489743    0.511653
2020-06-08 20:44:08.469140 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511459
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13007     12991   123.654       132    0.329233    0.511459
  421      16     13035     13019   123.626       112   0.0821875    0.511665
  422      16     13068     13052   123.646       132     0.45616    0.511607
  423      16     13097     13081   123.628       116   0.0570792    0.511798
  424      16     13129     13113   123.638       128    0.755786    0.511739
  425      16     13158     13142    123.62       116    0.416511    0.511909
  426      16     13198     13182   123.697       160   0.0623024    0.511689
  427      16     13228     13212   123.689       120     0.42818    0.511579
  428      15     13261     13246   123.716       136     1.13032    0.511563
  429      16     13291     13275   123.698       116    0.303452     0.51161
  430      16     13321     13305    123.69       120    0.042945    0.511613
  431      16     13352     13336    123.69       124     1.67741    0.511533
  432      15     13380     13365   123.673       116    0.344221    0.511665
  433      16     13412     13396   123.673       124    0.347792    0.511613
  434      16     13441     13425   123.655       116    0.045106    0.511641
  435      16     13475     13459   123.684       136   0.0460147     0.51156
  436      16     13503     13487   123.657       112    0.250473    0.511563
  437      16     13536     13520   123.676       132   0.0093678     0.51146
  438      16     13568     13552   123.686       128    0.196691    0.511626
  439      16     13602     13586   123.714       136    0.145612    0.511335
2020-06-08 20:44:28.506419 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.511612
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      15     13629     13614   123.686       112     1.34441    0.511612
  441      16     13661     13645   123.687       124    0.378196    0.511486
  442      15     13692     13677   123.697       128   0.0570218    0.511661
  443      16     13721     13705    123.67       112     1.17798     0.51175
  444      16     13751     13735   123.662       120    0.792439    0.511746
  445      16     13783     13767   123.671       128   0.0360382     0.51176
  446      16     13806     13790     123.6        92     1.49323    0.511914
  447      16     13838     13822   123.609       128    0.407204    0.511809
  448      16     13863     13847   123.556       100    0.251138    0.511931
  449      16     13892     13876   123.539       116   0.0844853    0.511986
  450      15     13921     13906   123.531       120    0.352163    0.512236
  451      16     13950     13934   123.505       112   0.0181103    0.512367
  452      16     13979     13963   123.489       116    0.176074    0.512193
  453      16     14009     13993   123.481       120     2.31226    0.512233
  454      16     14040     14024   123.482       124    0.258509    0.512215
  455      16     14070     14054   123.474       120     1.97507    0.512309
  456      16     14099     14083   123.458       116     1.51515    0.512425
  457      16     14126     14110   123.424       108     1.89717    0.512474
  458      16     14161     14145    123.46       140   0.0106847    0.512576
  459      16     14193     14177    123.47       128   0.0684864    0.512439
2020-06-08 20:44:48.518700 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512432
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14226     14210   123.488       132     1.73628    0.512432
  461      16     14257     14241   123.489       124     1.28408    0.512418
  462      15     14287     14272    123.49       124    0.233685    0.512345
  463      16     14321     14305   123.508       132     1.66653    0.512237
  464      16     14354     14338   123.527       132     1.36833    0.512191
  465      16     14391     14375   123.579       148   0.0596241    0.512001
  466      16     14421     14405   123.571       120    0.189901    0.511978
  467      15     14448     14433   123.547       112    0.173545    0.511998
  468      16     14477     14461   123.522       112   0.0812387     0.51206
  469      16     14514     14498   123.574       148    0.141564    0.511996
  470      16     14543     14527   123.558       116   0.0714603    0.512027
  471      16     14571     14555   123.533       112    0.112975    0.512121
  472      16     14597     14581   123.492       104     1.32184    0.512311
  473      16     14627     14611   123.484       120   0.0777479    0.512374
  474      16     14661     14645   123.511       136     1.45962    0.512317
  475      16     14689     14673   123.487       112    0.167458    0.512192
  476      16     14718     14702   123.471       116     2.28247    0.512498
  477      15     14748     14733   123.472       124    0.283824    0.512558
  478      15     14780     14765   123.481       128    0.329042    0.512501
  479      16     14810     14794   123.466       116     1.24327    0.512461
2020-06-08 20:45:08.524332 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512639
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      15     14837     14822   123.441       112    0.490039    0.512639
  481      15     14866     14851   123.425       116   0.0658814    0.512787
  482      16     14895     14879   123.402       112    0.397216    0.512757
  483      16     14927     14911   123.411       128    0.299788    0.512714
  484      15     14963     14948   123.462       148     0.20779    0.512583
  485      16     15000     14984   123.504       144    0.244432    0.512344
  486      16     15030     15014   123.497       120     1.49644    0.512378
  487      16     15062     15046   123.506       128   0.0931012    0.512297
  488      16     15092     15076   123.499       120    0.126937    0.512349
  489      16     15125     15109   123.516       132     1.58117    0.512278
  490      15     15152     15137   123.492       112    0.305903    0.512468
  491      15     15185     15170   123.509       132    0.306541    0.512361
  492      16     15220     15204   123.534       136     1.46515     0.51238
  493      16     15250     15234   123.527       120     1.44872    0.512218
  494      16     15277     15261   123.495       108    0.214639      0.5125
  495      16     15312     15296   123.528       140    0.290033    0.512335
  496      15     15339     15324   123.504       112     1.51602    0.512473
  497      16     15368     15352   123.481       112    0.244055    0.512459
  498      16     15397     15381   123.466       116     1.69339    0.512489
  499      15     15428     15413   123.475       128    0.435628    0.512387
2020-06-08 20:45:28.538352 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512573
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     15455     15439   123.436       104     2.12661    0.512573
  501      16     15486     15470   123.437       124    0.074086    0.512603
  502      16     15519     15503   123.454       132    0.379324    0.512493
  503      16     15550     15534   123.455       124     1.79179    0.512662
  504      16     15584     15568   123.479       136    0.550992    0.512538
  505      16     15611     15595   123.448       108    0.423585    0.512483
  506      16     15643     15627   123.457       128    0.266231    0.512626
  507      15     15671     15656   123.442       116    0.236596    0.512636
  508      16     15707     15691   123.475       140    0.277762    0.512599
  509      15     15740     15725   123.498       136   0.0762671     0.51243
  510      16     15773     15757   123.506       128     1.55159    0.512327
  511      16     15802     15786   123.492       116    0.228607    0.512305
  512      16     15835     15819   123.508       132     1.72701    0.512333
  513      16     15864     15848   123.494       116   0.0790003    0.512417
  514      16     15896     15880   123.502       128    0.483712    0.512423
  515      16     15928     15912   123.511       128     1.58394    0.512415
  516      16     15957     15941   123.496       116    0.198625    0.512344
  517      16     15988     15972   123.497       124     1.78059    0.512348
  518      16     16019     16003   123.498       124   0.0429903    0.512494
  519      16     16052     16036   123.514       132     0.26881     0.51225
2020-06-08 20:45:48.556328 min lat: 0.00726443 max lat: 2.66352 avg lat: 0.512329
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16087     16071   123.546       140     1.25423    0.512329
  521      15     16116     16101   123.539       120    0.155699     0.51224
  522      15     16143     16128   123.509       108     0.91247    0.512328
  523      16     16177     16161   123.525       132     1.61275    0.512304
  524      16     16207     16191   123.519       120     1.59436    0.512343
  525      16     16234     16218   123.489       108    0.285469     0.51244
  526      16     16265     16249    123.49       124   0.0631698    0.512357
  527      16     16296     16280   123.491       124    0.676844    0.512588
  528      16     16329     16313   123.506       132   0.0671672    0.512477
  529      16     16364     16348   123.537       140    0.486114    0.512255
  530      16     16392     16376   123.515       112   0.0638656    0.512293
  531      16     16420     16404   123.493       112   0.0655929     0.51254
  532      16     16450     16434   123.486       120    0.803907    0.512571
  533      16     16477     16461   123.457       108   0.0450396    0.512646
  534      15     16503     16488   123.428       108    0.322278    0.512734
  535      16     16534     16518   123.421       120    0.266941    0.512612
  536      16     16566     16550   123.429       128    0.238308    0.512608
  537      16     16594     16578   123.408       112    0.237731    0.512712
  538      16     16621     16605   123.379       108    0.231232    0.512884
  539      16     16649     16633   123.357       112    0.216255    0.512844
2020-06-08 20:46:08.575554 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.512956
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     16678     16662   123.344       116    0.346213    0.512956
  541      16     16707     16691    123.33       116    0.274218    0.513066
  542      16     16741     16725   123.353       136   0.0876994    0.513019
  543      16     16774     16758   123.369       132     1.21974    0.512968
  544      16     16804     16788   123.363       120    0.405559    0.512912
  545      16     16839     16823   123.394       140   0.0606176    0.512845
  546      16     16870     16854   123.395       124   0.0749025    0.512826
  547      16     16903     16887   123.406       132     1.70951    0.512732
  548      16     16933     16917   123.399       120  0.00824786    0.512624
  549      16     16963     16947   123.393       120    0.165414    0.512828
  550      15     16988     16973   123.358       104    0.189045     0.51294
  551      16     17013     16997   123.308        96    0.566199    0.513122
  552      16     17037     17021   123.259        96    0.205699     0.51339
  553      15     17066     17051   123.253       120     1.50583    0.513442
  554      16     17095     17079   123.232       112     1.95111    0.513388
  555      16     17127     17111   123.241       128   0.0519989     0.51348
  556      16     17159     17143   123.249       128      1.0562    0.513629
  557      16     17188     17172   123.236       116    0.321593    0.513379
  558      16     17220     17204   123.245       128     0.34435    0.513274
  559      15     17254     17239   123.275       140   0.0733466    0.513406
2020-06-08 20:46:28.600176 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.513256
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     17288     17272    123.29       132    0.127781    0.513256
  561      16     17316     17300    123.27       112    0.339301    0.513318
  562      16     17343     17327   123.243       108    0.236572    0.513485
  563      16     17372     17356    123.23       116      1.6081    0.513577
  564      16     17403     17387   123.231       124   0.0666848    0.513504
  565      16     17433     17417   123.226       120     1.62159    0.513606
  566      16     17458     17442   123.185       100    0.610262    0.513778
  567      16     17490     17474   123.193       128   0.0960859    0.513776
  568      16     17522     17506   123.201       128    0.691232    0.513809
  569      16     17553     17537   123.203       124    0.516746     0.51385
  570      15     17581     17566    123.19       116     1.38389    0.513848
  571      16     17614     17598   123.198       128   0.0420619    0.513756
  572      16     17642     17626   123.179       112    0.438171    0.513841
  573      16     17671     17655   123.166       116    0.952635     0.51396
  574      16     17697     17681   123.133       104   0.0575533    0.513836
  575      16     17727     17711   123.127       120    0.297035    0.513906
  576      16     17755     17739   123.108       112    0.270716    0.514065
  577      15     17785     17770    123.11       124     1.35757    0.514109
  578      15     17814     17799   123.097       116    0.256755    0.513921
  579      15     17848     17833   123.119       136   0.0650893    0.514007
2020-06-08 20:46:48.604263 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.514024
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     17880     17864   123.121       124   0.0475618    0.514024
  581      16     17909     17893   123.109       116    0.367496    0.514028
  582      16     17936     17920   123.083       108   0.0784211    0.513969
  583      16     17967     17951   123.084       124    0.203588    0.514001
  584      15     17999     17984   123.099       132   0.0760753    0.514197
  585      15     18027     18012    123.08       112     1.54829    0.514226
  586      16     18058     18042   123.075       120   0.0488779    0.514229
  587      16     18088     18072    123.07       120    0.788804    0.514299
  588      16     18120     18104   123.078       128   0.0492432    0.514288
  589      16     18148     18132   123.059       112    0.439958    0.514217
  590      16     18183     18167   123.088       140    0.263807    0.514094
  591      16     18211     18195   123.069       112    0.795766     0.51436
  592      16     18243     18227   123.078       128   0.0553352    0.514339
  593      16     18271     18255   123.059       112     1.40097    0.514367
  594      15     18300     18285   123.054       120    0.314329    0.514379
  595      16     18334     18318   123.069       132   0.0830795    0.514141
  596      16     18363     18347   123.057       116    0.343694    0.514137
  597      16     18391     18375   123.038       112      1.8729    0.514305
  598      16     18419     18403    123.02       112    0.166918    0.514431
  599      16     18448     18432   123.008       116    0.323109    0.514472
2020-06-08 20:47:08.607204 min lat: 0.00726443 max lat: 3.03968 avg lat: 0.514576
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      14     18466     18452   122.936        80     1.91121    0.514576
Total time run:       600.536
Total reads made:     18466
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   122.997
Average IOPS:         30
Stddev IOPS:          2.7656
Max IOPS:             40
Min IOPS:             20
Average Latency(s):   0.514993
Max latency(s):       3.03968
Min latency(s):       0.00726443

Total time run:       600.536
Total reads made:     18466
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   122.997
Average IOPS:         30
Stddev IOPS:          2.7656
Max IOPS:             40
Min IOPS:             20
Average Latency(s):   0.514993
Max latency(s):       3.03968
Min latency(s):       0.00726443

Cleanup Pool with 256 PGs and remove it

Note that I could also have simply deleted the pool testbench2 from within the Ceph Dashboard.

[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 rados -p testbench2 cleanup
Removed 11055 objects
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench2 testbench2 --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
Error: non zero exit code: 1: OCI runtime error
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=true
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'true' 
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'true' 
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'true' 
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench2 testbench2 --yes-i-really-really-mean-it
pool 'testbench2' removed
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=false
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'false' 
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'false' 
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'false' 
[root@f5-422-01 ~]# 

Quick libvirt and Ceph RBD Smoke Test

Since I already have an RBD pool libvirt-test that is configured for my Fedora32 hypervisor and Nautilus can scale PGs up and down as well as auto-scale, I ran my first tests withe the current 16 PGs.

This is not ideal though, since my last tests were with the EPYC based hypervisor which has a 10 Gig network connection, but I broke something in my RBD setup on that box and have not fixed it yet :-/

FIXME: re-test with 256 PGs and VM on EPYC for comparison with earlier results.

root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-test
 Name                        Path
---------------------------------------------------------------------
 20200608-fio-testing-1TiB   libvirt-test/20200608-fio-testing-1TiB

Prepare Test VM

NOTE: The hypervisor used is not my main one, this one only has a single 1 Gbit network connection. So even with 0 overhead and no other traffic, it simply can not reach the 140MiB/s I saw in the earlier test. And since in previous tests there was no huge difference between a 1st and a 2nd run, as a quick smoke test, I only ran each test once.

As last time, I attached the 1 TiB RBD (20200608-fio-testing-1TiB) to a VM (CentOS 8) as a virtio block device, formatted it with XFS, mounted it and ran fio.

But since the VM in question only had 4 GiB RAM and I wanted this smoke test done quickly, I opted to do my fio tests with 16 GiB as size, not 160 like last time.

VM side of taking the RBD image in use (click to expand)
[root@guest13 ~]# dmesg | tail -n 4
[15125.785114] pcieport 0000:00:02.6:   bridge window [mem 0xfc000000-0xfc1fffff]
[15125.787750] pcieport 0000:00:02.6:   bridge window [mem 0xfde00000-0xfdffffff 64bit pref]
[15125.791450] virtio-pci 0000:07:00.0: enabling device (0000 -> 0002)
[15125.797291] virtio_blk virtio5: [vdb] 2147483648 512-byte logical blocks (1.10 TB/1.00 TiB)
[root@guest13 ~]# lsblk
NAME                MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda                 252:0    0  20G  0 disk 
├─vda1              252:1    0   1G  0 part /boot
└─vda2              252:2    0  19G  0 part 
  ├─cl_guest13-root 253:0    0  17G  0 lvm  /
  └─cl_guest13-swap 253:1    0   2G  0 lvm  [SWAP]
vdb                 252:16   0   1T  0 disk 
[root@guest13 ~]# mkfs.xfs /dev/vdb
meta-data=/dev/vdb               isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@guest13 ~]# mount /dev/vdb /mnt/
[root@guest13 ~]# df -h /mnt/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb        1.0T  7.2G 1017G   1% /mnt
[root@guest13 ~]# mkdir /mnt/fiotest/
[root@guest13 ~]# uname -a # I think the yused kickstart is wuithout package upgrades in %post
Linux guest13.internal.pcfe.net 4.18.0-147.el8.x86_64 #1 SMP Wed Dec 4 21:51:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@guest13 ~]# dnf upgrade # yes it was, let's upgrade
[reboot and mount again]
[root@guest13 ~]# uname -a
Linux guest13.internal.pcfe.net 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Sequential Read, 4k Blocksize

  • READ: bw=85.8MiB/s
  • iops: avg=21940.29
fio --name=readtest --rw=read --size=16g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7                                                                                                  
Starting 1 process                                                                                                          
readtest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=101MiB/s,w=0KiB/s][r=25.9k,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=4207: Mon Jun  8 22:14:32 2020
   read: IOPS=21.0k, BW=85.8MiB/s (89.9MB/s)(16.0GiB/191065msec)
    clat (nsec): min=1843, max=328591k, avg=41103.33, stdev=1721413.61
     lat (usec): min=2, max=328591, avg=42.15, stdev=1721.41
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    4], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[   10], 99.50th=[   18], 99.90th=[ 1004], 99.95th=[25822],
     | 99.99th=[91751]
   bw (  KiB/s): min=16384, max=123819, per=99.95%, avg=87761.29, stdev=17705.34, samples=382
   iops        : min= 4096, max=30954, avg=21940.29, stdev=4426.32, samples=382
  lat (usec)   : 2=0.07%, 4=81.84%, 10=17.18%, 20=0.59%, 50=0.19%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.03%
  lat (msec)   : 100=0.02%, 250=0.01%, 500=0.01%
  cpu          : usr=9.13%, sys=9.14%, ctx=4482, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4194304,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=85.8MiB/s (89.9MB/s), 85.8MiB/s-85.8MiB/s (89.9MB/s-89.9MB/s), io=16.0GiB (17.2GB), run=191065-191065msec

Disk stats (read/write):
  vdb: ios=16473/8, merge=0/1, ticks=2094592/785, in_queue=2087094, util=6.09%

Sequential Read, 4m Blocksize

  • READ: bw=89.6MiB/s
  • iops: avg=22.34
fio --name=readtest --rw=read --size=16g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=112MiB/s,w=0KiB/s][r=28,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=4210: Mon Jun  8 22:17:36 2020
   read: IOPS=22, BW=89.6MiB/s (93.9MB/s)(16.0GiB/182944msec)
    clat (usec): min=882, max=613961, avg=44528.03, stdev=51858.65
     lat (usec): min=883, max=613963, avg=44530.35, stdev=51858.37
    clat percentiles (usec):
     |  1.00th=[  1106],  5.00th=[  1303], 10.00th=[  1369], 20.00th=[  1614],
     | 30.00th=[  3458], 40.00th=[  9634], 50.00th=[ 25035], 60.00th=[ 42206],
     | 70.00th=[ 61080], 80.00th=[ 86508], 90.00th=[121111], 95.00th=[145753],
     | 99.00th=[200279], 99.50th=[225444], 99.90th=[295699], 99.95th=[354419],
     | 99.99th=[616563]
   bw (  KiB/s): min=32768, max=131072, per=100.00%, avg=91833.11, stdev=17359.10, samples=364
   iops        : min=    8, max=   32, avg=22.34, stdev= 4.24, samples=364
  lat (usec)   : 1000=0.22%
  lat (msec)   : 2=22.12%, 4=11.43%, 10=6.52%, 20=6.45%, 50=17.26%
  lat (msec)   : 100=20.65%, 250=15.06%, 500=0.27%, 750=0.02%
  cpu          : usr=0.04%, sys=6.24%, ctx=4269, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4096,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=89.6MiB/s (93.9MB/s), 89.6MiB/s-89.6MiB/s (93.9MB/s-93.9MB/s), io=16.0GiB (17.2GB), run=182944-182944msec

Disk stats (read/write):
  vdb: ios=16536/0, merge=2/0, ticks=2140652/0, in_queue=2132410, util=7.00%

Sequential Write, 4k Blocksize

  • WRITE: bw=79.9MiB/s
  • iops: avg=20440.18
fio --name=writetest --rw=write --size=16g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=87.2MiB/s][r=0,w=22.3k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4213: Mon Jun  8 22:21:01 2020
  write: IOPS=20.5k, BW=79.9MiB/s (83.8MB/s)(16.0GiB/204930msec)
    clat (usec): min=2, max=44243, avg=43.39, stdev=559.10
     lat (usec): min=3, max=44244, avg=44.73, stdev=559.10
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    5], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[    7], 60.00th=[    7],
     | 70.00th=[    7], 80.00th=[    8], 90.00th=[    8], 95.00th=[   10],
     | 99.00th=[   25], 99.50th=[   50], 99.90th=[ 9372], 99.95th=[11076],
     | 99.99th=[12125]
   bw (  KiB/s): min=35840, max=705232, per=99.87%, avg=81760.88, stdev=44608.86, samples=409
   iops        : min= 8960, max=176308, avg=20440.18, stdev=11152.19, samples=409
  lat (usec)   : 4=5.33%, 10=90.85%, 20=2.13%, 50=1.19%, 100=0.04%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.38%, 20=0.08%, 50=0.01%
  cpu          : usr=10.63%, sys=14.15%, ctx=19217, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4194304,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=79.9MiB/s (83.8MB/s), 79.9MiB/s-79.9MiB/s (83.8MB/s-83.8MB/s), io=16.0GiB (17.2GB), run=204930-204930msec

Disk stats (read/write):
  vdb: ios=0/1613, merge=0/4, ticks=0/38770165, in_queue=38762165, util=1.30%

Sequential Write, 4m Blocksize

  • WRITE: bw=77.8MiB/s
  • iops: avg=19.95
fio --name=writetest --rw=write --size=16g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=76.1MiB/s][r=0,w=19 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4219: Mon Jun  8 22:24:32 2020
  write: IOPS=19, BW=77.8MiB/s (81.6MB/s)(16.0GiB/210545msec)
    clat (usec): min=1754, max=109764, avg=49735.40, stdev=16393.01
     lat (usec): min=1791, max=109914, avg=49904.31, stdev=16394.86
    clat percentiles (usec):
     |  1.00th=[  1778],  5.00th=[ 21103], 10.00th=[ 30016], 20.00th=[ 38536],
     | 30.00th=[ 44827], 40.00th=[ 47973], 50.00th=[ 52167], 60.00th=[ 53740],
     | 70.00th=[ 55837], 80.00th=[ 62653], 90.00th=[ 69731], 95.00th=[ 71828],
     | 99.00th=[ 83362], 99.50th=[ 87557], 99.90th=[ 99091], 99.95th=[108528],
     | 99.99th=[109577]
   bw (  KiB/s): min=40960, max=827392, per=100.00%, avg=81999.56, stdev=47883.25, samples=409
   iops        : min=   10, max=  202, avg=19.95, stdev=11.69, samples=409
  lat (msec)   : 2=2.61%, 4=1.51%, 10=0.07%, 20=0.66%, 50=42.80%
  lat (msec)   : 100=52.25%, 250=0.10%
  cpu          : usr=0.41%, sys=6.64%, ctx=22460, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4096,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=77.8MiB/s (81.6MB/s), 77.8MiB/s-77.8MiB/s (81.6MB/s-81.6MB/s), io=16.0GiB (17.2GB), run=210545-210545msec

Disk stats (read/write):
  vdb: ios=0/16660, merge=0/24, ticks=0/39547058, in_queue=39538722, util=1.36%

Sequential, 80% Read, 20% Write, 4k Blocksize

  • READ: bw=81.9MiB/s
  • iops: avg=20959.49
  • WRITE: bw=20.5MiB/s
  • iops: avg=5239.82
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=16g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 16384MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=96.1MiB/s,w=24.6MiB/s][r=24.6k,w=6306 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=4394: Mon Jun  8 22:30:11 2020
   read: IOPS=20.0k, BW=81.9MiB/s (85.9MB/s)(12.8GiB/160073msec)
    clat (nsec): min=1866, max=439393k, avg=41066.81, stdev=1806675.83
     lat (usec): min=2, max=439393, avg=42.06, stdev=1806.81
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    4],
     | 70.00th=[    4], 80.00th=[    4], 90.00th=[    5], 95.00th=[    5],
     | 99.00th=[    9], 99.50th=[   19], 99.90th=[  971], 99.95th=[24511],
     | 99.99th=[91751]
   bw (  KiB/s): min=16384, max=122900, per=99.99%, avg=83838.07, stdev=20691.10, samples=320
   iops        : min= 4096, max=30725, avg=20959.49, stdev=5172.77, samples=320
  write: IOPS=5240, BW=20.5MiB/s (21.5MB/s)(3277MiB/160073msec)
    clat (usec): min=2, max=1532, avg= 5.59, stdev= 3.83
     lat (usec): min=3, max=1533, avg= 6.62, stdev= 4.11
    clat percentiles (nsec):
     |  1.00th=[ 3408],  5.00th=[ 3536], 10.00th=[ 3728], 20.00th=[ 3920],
     | 30.00th=[ 4080], 40.00th=[ 4512], 50.00th=[ 5024], 60.00th=[ 5536],
     | 70.00th=[ 5984], 80.00th=[ 6560], 90.00th=[ 7328], 95.00th=[ 8896],
     | 99.00th=[18560], 99.50th=[24192], 99.90th=[40192], 99.95th=[51456],
     | 99.99th=[80384]
   bw (  KiB/s): min= 4384, max=30780, per=99.99%, avg=20959.44, stdev=5163.88, samples=320
   iops        : min= 1096, max= 7695, avg=5239.82, stdev=1290.97, samples=320
  lat (usec)   : 2=0.10%, 4=74.33%, 10=24.20%, 20=0.89%, 50=0.38%
  lat (usec)   : 100=0.03%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  cpu          : usr=10.41%, sys=11.11%, ctx=3742, majf=0, minf=14
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3355445,838859,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=81.9MiB/s (85.9MB/s), 81.9MiB/s-81.9MiB/s (85.9MB/s-85.9MB/s), io=12.8GiB (13.7GB), run=160073-160073msec
  WRITE: bw=20.5MiB/s (21.5MB/s), 20.5MiB/s-20.5MiB/s (21.5MB/s-21.5MB/s), io=3277MiB (3436MB), run=160073-160073msec

Disk stats (read/write):
  vdb: ios=13249/3272, merge=0/3, ticks=1751978/464437, in_queue=2208132, util=6.93%

Sequential, 80% Read, 20% Write, 4m Blocksize

  • READ: bw=85.4MiB/s
  • iops: avg=21.34
  • WRITE: bw=22.3MiB/s
  • iops: avg=6.21
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=16g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=60.0MiB/s,w=40.0MiB/s][r=15,w=10 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=4397: Mon Jun  8 22:32:44 2020
   read: IOPS=21, BW=85.4MiB/s (89.6MB/s)(12.7GiB/152077msec)
    clat (usec): min=824, max=541502, avg=45766.26, stdev=57657.21
     lat (usec): min=825, max=541504, avg=45768.34, stdev=57657.00
    clat percentiles (usec):
     |  1.00th=[  1020],  5.00th=[  1237], 10.00th=[  1319], 20.00th=[  1598],
     | 30.00th=[  3458], 40.00th=[  9503], 50.00th=[ 24511], 60.00th=[ 41681],
     | 70.00th=[ 60031], 80.00th=[ 85459], 90.00th=[121111], 95.00th=[149947],
     | 99.00th=[263193], 99.50th=[308282], 99.90th=[408945], 99.95th=[480248],
     | 99.99th=[541066]
   bw (  KiB/s): min=24526, max=131072, per=100.00%, avg=87759.99, stdev=24141.96, samples=303
   iops        : min=    5, max=   32, avg=21.34, stdev= 5.89, samples=303
  write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(3392MiB/152077msec)
    clat (usec): min=1789, max=6071, avg=2933.39, stdev=590.52
     lat (usec): min=1885, max=6340, avg=3080.36, stdev=600.38
    clat percentiles (usec):
     |  1.00th=[ 1893],  5.00th=[ 2147], 10.00th=[ 2278], 20.00th=[ 2343],
     | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3064], 60.00th=[ 3195],
     | 70.00th=[ 3261], 80.00th=[ 3326], 90.00th=[ 3523], 95.00th=[ 3818],
     | 99.00th=[ 4817], 99.50th=[ 5342], 99.90th=[ 6063], 99.95th=[ 6063],
     | 99.99th=[ 6063]
   bw (  KiB/s): min= 8126, max=81920, per=100.00%, avg=25816.79, stdev=16308.32, samples=269
   iops        : min=    1, max=   20, avg= 6.21, stdev= 4.00, samples=269
  lat (usec)   : 1000=0.44%
  lat (msec)   : 2=17.48%, 4=28.66%, 10=5.96%, 20=5.35%, 50=14.33%
  lat (msec)   : 100=15.84%, 250=10.99%, 500=0.93%, 750=0.02%
  cpu          : usr=0.13%, sys=7.44%, ctx=3522, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3248,848,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=85.4MiB/s (89.6MB/s), 85.4MiB/s-85.4MiB/s (89.6MB/s-89.6MB/s), io=12.7GiB (13.6GB), run=152077-152077msec
  WRITE: bw=22.3MiB/s (23.4MB/s), 22.3MiB/s-22.3MiB/s (23.4MB/s-23.4MB/s), io=3392MiB (3557MB), run=152077-152077msec

Disk stats (read/write):
  vdb: ios=13120/3226, merge=1/0, ticks=1781440/780190, in_queue=2553537, util=7.56%

sequential write, 2300 bytes, fsync

Normally this test is not part of my baseline but a colleague inquired, so pasting here.

  • WRITE: bw=206KiB/s
  • iops: avg=93.01
fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7                                                                       
Starting 1 process                                                             
etcdtest: Laying out IO file (1 file / 22MiB)                                  
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=217KiB/s][r=0,w=97 IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=4402: Mon Jun  8 22:34:34 2020
  write: IOPS=91, BW=206KiB/s (211kB/s)(21.0MiB/109322msec)
    clat (usec): min=12, max=526, avg=26.99, stdev=11.56
     lat (usec): min=13, max=528, avg=29.14, stdev=11.65                                                             
    clat percentiles (usec):                                                                                         
     |  1.00th=[   18],  5.00th=[   19], 10.00th=[   20], 20.00th=[   23],
     | 30.00th=[   24], 40.00th=[   24], 50.00th=[   25], 60.00th=[   27],
     | 70.00th=[   30], 80.00th=[   31], 90.00th=[   34], 95.00th=[   39],          
     | 99.00th=[   58], 99.50th=[   67], 99.90th=[   82], 99.95th=[   88],                               
     | 99.99th=[  515]                                                                                                      
   bw (  KiB/s): min=    4, max=  247, per=100.00%, avg=208.53, stdev=31.13, samples=215
   iops        : min=    2, max=  110, avg=93.01, stdev=13.84, samples=215
  lat (usec)   : 20=11.57%, 50=86.65%, 100=1.74%, 500=0.02%, 750=0.02%             
  fsync/fdatasync/sync_file_range:                                                
    sync (msec): min=4, max=1145, avg=10.86, stdev=19.30     
    sync percentiles (msec):                                      
     |  1.00th=[    6],  5.00th=[    6], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    7], 40.00th=[    7], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   13], 90.00th=[   14], 95.00th=[   18],    
     | 99.00th=[   34], 99.50th=[   38], 99.90th=[   71], 99.95th=[  105],    
     | 99.99th=[  986]                                                        
  cpu          : usr=0.18%, sys=0.69%, ctx=35703, majf=0, minf=12             
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%               
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
                                                              
Run status group 0 (all jobs):
  WRITE: bw=206KiB/s (211kB/s), 206KiB/s-206KiB/s (211kB/s-211kB/s), io=21.0MiB (23.1MB), run=109322-109322msec
                                                                          
Disk stats (read/write):                                                  
  vdb: ios=0/25943, merge=0/15, ticks=0/259443, in_queue=248278, util=14.06%

sequential write, 2300 bytes, NO fsync

Since it seems silly to me to call fsync after 2300 bytes, same test without that option to have a comparison.

  • WRITE: bw=100MiB/s
  • iops: avg=45595.99
fio --rw=write --ioengine=sync  --size=16g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7                                                                   
Starting 1 process         
etcdtest: Laying out IO file (1 file / 16384MiB)                  
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=102MiB/s][r=0,w=46.6k IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=4411: Mon Jun  8 22:37:18 2020
  write: IOPS=45.6k, BW=100MiB/s (105MB/s)(15.0GiB/163747msec)                
    clat (nsec): min=1570, max=44945k, avg=17722.52, stdev=351436.48           
     lat (usec): min=2, max=44946, avg=18.75, stdev=351.47                     
    clat percentiles (nsec):                                      
     |  1.00th=[    1624],  5.00th=[    2160], 10.00th=[    2512],
     | 20.00th=[    2832], 30.00th=[    3152], 40.00th=[    3408],
     | 50.00th=[    4576], 60.00th=[    5216], 70.00th=[    5664],
     | 80.00th=[    6048], 90.00th=[    6624], 95.00th=[    7648],                                                   
     | 99.00th=[   14784], 99.50th=[   19584], 99.90th=[ 7766016],                                                   
     | 99.95th=[10027008], 99.99th=[11599872]
   bw (  KiB/s): min=71583, max=447138, per=99.96%, avg=102412.57, stdev=35742.40, samples=327
   iops        : min=31870, max=199074, avg=45595.99, stdev=15913.14, samples=327
  lat (usec)   : 2=3.02%, 4=41.94%, 10=53.04%, 20=1.53%, 50=0.32%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.10%, 20=0.05%, 50=0.01%
  cpu          : usr=18.24%, sys=21.96%, ctx=11258, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7469508,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=15.0GiB (17.2GB), run=163747-163747msec

Disk stats (read/write):
  vdb: ios=0/15883, merge=0/4, ticks=0/30558816, in_queue=30550869, util=1.59%

New Shelves for Cluster

In late August 2020 I finally moved the 4 cubes off the desk they were hogging and onto a wall.

I went from this

picture of my 4 F5-422 nodes all drive bays filled

to this

picture of my 4 F5-422 cluster on a shelf