Four TerraMaster F5-422 with Ceph

Table of Contents

After installing RHEL 8, I started building a Ceph Nautilus cluster with my four TerraMaster F5-422 nodes.

picture of my 4 F5-422 nodes in their temporary location

I was made aware of these cheap, yet equipped with one 10G PCIe NIC and 5 SATA bays, machines in in c’t 2 / 2020 (caveat, paywall)

While my ODROID-HC2 Ceph cluster served well for ceph-ansible training, I wanted something more beefy for a while now.

I first bought one and ran some tests

I’ll try to avoid having x86 VMs running on my hypervisor that are needed for cluster operation. That way I will be able take the hypervisor down without affecting my Ceph storage. But I’ll have to see how the small NAS boxes perform with colocation.

Hardware Bought

Since Ceph needs a minimum of three nodes to be even remotely useful, I bought four TerraMaster F5-422 nodes. As they come with 10G copper interfaces, I recently added a 10G capable UniFi Switch 16 XG to my homelab (and promptly had to modify the fans).

This cluster will replace my ODROID-HC2 based cluster. I have enough old disks that I can recycle when tearing down the ODROID-HC2 cluster.

Shopping List

In case you plan a similar setup;

  • 1 UniFi Switch 16 XG
  • 2 Noctua NF-A4x20 FLX
  • 4 TerraMaster F5-422
  • 4 SO-DIMMs, 8GB
  • 4 SATA SSD, 500GB
  • 2 DAC cables
  • 8 Cat 5e cables
  • 4 Cat 6 cables
  • 1 4x HDMI, USB keyboard & mouse switch

HDD are old ones that were previously used in the now decommissioned cluster.

Hardware Notes

Since I paid only some 560€ per F5-422 (plus some for RAM and storage), I fully expect some downsides. But not quite as many as I had with my ODROID-HC2 boards.

So far I have not had any unexpected major ones.

Downsides

General Downsides

With only 6 PCIe lanes on the CPU and seemingly the Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) only being attached with 2 lanes I only got between 5.4 Gbits/sec and 5.8 Gbits/sec in initial testing.

With 3 drives up and running, only 2 of the drive LEDs in the chassis light up.

Ceph Relevant Downsides

While I get more than 1 Gbit/s, it’s by no means full 10 Gbit/s.\

Install RHEL8 x86_64

See here for my installation notes. Especially the Firmware settings, the rest is a standard install with all non standard steps listed again in this post.

Log In Via ssh

My kickstart dumps an ssh pubkey for the root user, so on the Ansible control node, as user, I

  • verify the ssh key I plan to use for ansible is loaded in the ssh-agent
  • verify I can log in to a node as root
ssh-add -l
[...]
ssh root@f5-422-01

Check Ansible Inventory

Since I have both group_vars and host_vars that apply to these hosts, I check of all values is in order.

ansible-inventory -i ~/work/git/HouseNet/ansible/inventories/ceph-F5-422-cluster.ini --list all
ansible-inventory -i ~/work/git/HouseNet/ansible/inventories/ceph-F5-422-cluster.ini --graph all

Verify Ansible Connectivity

On the Ansible control node, as user (-e ansible_user=root is only needed until terramaster-initial-setup-centos8.yml created the ansible user)

ansible -i ../inventories/ceph-F5-422-cluster.ini all -m ping -e ansible_user=root

Initially Configure the F5-422

Create an ansible user with permission to do password-less sudo. I do this via Playbook.

Running My Initial Setup Playbook

terramaster-initial-setup-rhel8.yml

takes care of

  • creating an ansible user
  • basic-security-setup
  • local settings (timezone etc)
  • ensuring EPEL is disabled
  • disabling console blanking (normally there is no screen connected, but if I do connect one I want the display not blanked)

As user, on my F31 x96_64 workstation.

ansible-playbook terramaster-initial-setup-rhel8.yml
full playbook terramaster-initial-setup-rhel8.yml
---
# sets up a RHEL 8 minimal install to be ready for ceph-ansible
#
# this is for my home setup, not for production!
- hosts:
  - TerraMaster_boxes

  become: false

  roles:
    - pcfe.user_owner
    - pcfe.basic-security-setup
    - pcfe.housenet
    #- pcfe.checkmk
    # note though that check-mk-agent is in EPEL but RHCS 4 docs say EPEL needs to be off
    # for now I manually installed the rpm with
    # dnf install http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/check-mk-agent-1.6.0p5-1.noarch.rpm

  # no need for double indirect if you are OK with checking in ak details into git
  # this is OK to do if you use an in-house Satellite server and your security policies allow it
  # this is not a good idea if you register your systems directly to redhat.com and cannot guarantee that your git remains private
  vars_files:
    - "vars/subscription-manager-autoattach-ak-secrets.yml"
  vars:
    ansible_user: root
    user_owner: ansible
    common_timezone: Europe/Berlin
    rhsm_activationkey: "{{ vaulted_rhsm_activationkey }}"
    rhsm_org_id: "{{ vaulted_rhsm_org_id }}"
    rhsm_pool_ids: "{{ vaulted_rhsm_pool_ids }}"

  pre_tasks:
    # work around identical UUID on each host as per https://theforeman.org/plugins/katello/nightly/troubleshooting/content_hosts.html
    # The TerraMaster F5-422 boxes all have the same system-uuid :-(
    # RHSM does not like that, so override to avoid both
    # "HTTP error (409 - Conflict): Request failed due to concurrent modification, please re-try.\n"
    # and all 4 boxes overriding each other in insights inventory
    - name: "RHSM | ensure uuid override is derived from fqdn, F5-422 all have idential uuid in DMI"
      copy:
        dest: "/etc/rhsm/facts/uuid_override.facts"
        owner: "root"
        group: "root"
        mode: 0644
        content: |
          {"dmi.system.uuid": "{{ ansible_fqdn | to_uuid }}"}


    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "RHSM on the RHEL8 boxes"
      block:
        - name: "RHSM | ensure system is registered with my activation key"
          redhat_subscription:
            activationkey:  "{{ rhsm_activationkey }}"
            org_id:         "{{ rhsm_org_id }}"
            syspurpose:
              usage: "Development/Test"
              role: ""
              service_level_agreement: "Self-Support"

        - name: "RHSM | ensure RHEL8 BaseOS repos needed for Ceph are enabled"
          rhsm_repository:
            name:
              - rhel-8-for-x86_64-baseos-rpms
              - rhel-8-for-x86_64-appstream-rpms
            state: enabled
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"
      tags: do_subsmgr_register
    - name: "RHSM on the RHEL7 boxes"
      block:
        - name: "RHSM | ensure system is registered with my activation key"
          redhat_subscription:
            activationkey:  "{{ rhsm_activationkey }}"
            org_id:         "{{ rhsm_org_id }}"
        - name: "RHSM | disable all repositories, next task will enable needed repos"
          rhsm_repository:
            name: '*'
            state: disabled
        - name: "RHSM | ensure RHEL7 BaseOS repos needed for Ceph are enabled"
          rhsm_repository:
            name:
              - rhel-7-server-rpms
              - rhel-7-server-extras-rpms
            state: enabled
      when: ansible_distribution == "RedHat" and ansible_distribution_major_version == "7"
      tags: do_subsmgr_register

  tasks:
    # !!!
    #
    # if I ever enable EPEL, then I MUST exclude
    # - ansible
    # - ceph
    # in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
    #
    # exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
    #
    # !!!
    #
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "REPOS | ensure EPEL is disabled"
      yum_repository:
        name: epel
        state: absent

    # start by enabling time sync, RHSM operations will fail on too large time delta
    # note that this uses chronyd, not ntpd.
    - name: "CHRONYD | ensure chrony is installed"
      package:
        name:       chrony
        state:      present
    - name:         "CHRONYD | ensure chrony-wait is enabled"
      service:
        name:       chrony-wait
        enabled:    true
    - name:         "CHRONYD | ensure chronyd is enabled and running"
      service:
        name:       chronyd
        enabled:    true
        state:      started
    
    # enable persistent journal
    - name: "JOURNAL | ensure persistent logging for the systemd journal is possible"
      file:
        path: /var/log/journal
        state: directory
        owner: root
        group: systemd-journal
        mode: u=rwx,g=rs,o=rx

    # 2.10. Enabling Password-less SSH for Ansible
    - name: "SUDO | enable passwordless sudo for ansible user"
      copy:
        dest: /etc/sudoers.d/ansible
        content: |
          ansible   ALL=NOPASSWD:   ALL
        owner: root
        group: root
        mode: 0440

    # Install prerequisites
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/installation_guide/requirements-for-installing-rhcs#enabling-the-red-hat-ceph-storage-repositories-install
    - name: "package | ensure prerequisites needed in addition to minimal install are present"
      package:
        name:
          - yum-utils
          - vim
        state: present

    # I admit, the regexp is a search engine hit
    - name: "GRUB | ensure console blanking is disabled in defaults file"
      lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: yes
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'

(click the triangle to expand)

Perform Hardware Specific Setup

Running My Hardware Specific Playbook

terramaster-general-setup.yml

Besides running some of my roles, it takes care of

  • ensuring Wake on LAN is set up for the 1Gig PCIe network interface
  • set up watchdog, rngd, tuned
  • ensuring cockbit is installed but disabled
  • ensuring libvirtd is not running
  • ensuring kdump is on
  • ensuring I can monitor the host with my Check MK instance
  • ensuring consoleblank=0 is set in my default boot options
  • ensuring I can kickstart the node again via a target in grub
  • ensuring the system has the latest updates applied

As user, on my F31 x96_64 workstation.

ansible-playbook terramaster-general-setup.yml
full playbook terramaster-general-setup-rhel8.yml
---
- hosts:
  - TerraMaster_boxes
  become: true
  roles:
    - linux-system-roles.network
    - pcfe.user_owner
    - pcfe.basic-security-setup
    - pcfe.housenet
    - pcfe.comfort
#    - pcfe.checkmk

  handlers:
    - name: grub2-mkconfig | run
      command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

  tasks:
    # Install some tools
    - name: "PACKAGE | tool installation"
      package:
        name:
          - pciutils
          - usbutils
          - fio
          - powertop
          - tuned
          - tuned-utils
          - numactl
          - mailx
          - teamd
          - NetworkManager-team
          - iperf3
          - tcpdump
          - hwloc-gui
        state: present
        update_cache: no

    # linux-system-roles.network sets static network config (from host_vars)
    # but I want the static hostname nailed down too
    - name: "set hostname"
      hostname:
        name:          "{{ ansible_fqdn }}"
        use:           systemd

    # enable WOL manually until https://github.com/linux-system-roles/network/issues/150 is fixed
    - name: "ensure Wake On LAN is enable for the upper network card"
      lineinfile:
        path:         /etc/sysconfig/network-scripts/ifcfg-1G_PCIe
        create:       false
        regexp:       '^ETHTOOL_OPTS= '
        insertafter:  '^TYPE=Ethernet'
        line:         'ETHTOOL_OPTS="wol g"'

    # enable watchdog
    # it's a Jun 22 13:12:09 localhost kernel: sbsa-gwdt e0bb0000.gwdt: Initialized with 10s timeout @ 250000000 Hz, action=0.
    - name: "WATCHDOG | ensure kernel module wdat_wdt has correct options configured"
      lineinfile:
        path:         /etc/modprobe.d/wdat_wdt.conf
        create:       true
        regexp:       '^options '
        insertafter:  '^#options'
        line:         'options wdat_wdt nowayout=0'

    # configure both watchdog.service and systemd watchdog, but only use the latter
    - name: "PACKAGE | ensure watchdog package is installed"
      package:
        name:         watchdog
        state:        present
        update_cache: no
    - name: "WATCHDOG | ensure correct watchdog-device is used by watchdog.service"
      lineinfile:
        path:         /etc/watchdog.conf
        regexp:       '^watchdog-device'
        insertafter:  '^#watchdog-device'
        line:         'watchdog-device = /dev/watchdog'
    - name: "WATCHDOG | ensure timeout is set to 30 seconds for watchdog.service"
      lineinfile:
        path:         /etc/watchdog.conf
        regexp:       '^watchdog-timeout'
        insertafter:  '^#watchdog-timeout'
        line:         'watchdog-timeout = 30'
    # Using systemd watchdog rather than watchdog.service
    - name: "WATCHDOG | ensure watchdog.service is disabled"
      systemd:
        name:         watchdog.service
        state:        stopped
        enabled:      false
    # configure systemd watchdog
    # c.f. http://0pointer.de/blog/projects/watchdog.html
    - name: "SYSTEMD | ensure systemd watchdog is enabled"
      lineinfile:
        path:         /etc/systemd/system.conf
        regexp:       '^RuntimeWatchdogSec'
        insertafter:  'EOF'
        line:         'RuntimeWatchdogSec=30'
    - name: "SYSTEMD | ensure systemd shutdown watchdog is enabled"
      lineinfile:
        path:         /etc/systemd/system.conf
        regexp:       '^ShutdownWatchdogSec'
        insertafter:  'EOF'
        line:         'ShutdownWatchdogSec=30'

    # install and enable rngd
    - name: "PACKAGE | ensure rng-tools package is installed"
      package:
        name:         rng-tools
        state:        present
        update_cache: no
    - name: "RNGD | ensure rngd.service is enabled and started"
      systemd:
        name:         rngd.service
        state:        started
        enabled:      true

    - name: "TUNED | ensure tuned.service is enabled and running"
      systemd:
        name:           tuned.service
        state:          started
        enabled:        true
    - block:
      - name: "TUNED | check which tuned profile is active"
        shell:          tuned-adm active
        register:       tuned_active_profile
        ignore_errors:  yes
        changed_when:   no
      - name: "TUNED | activate tuned profile throughput-performance if not yet enabled"
        shell:          tuned-adm profile throughput-performance
        when:           "tuned_active_profile.stdout.find('Current active profile: throughput-performance') != 0"

    # install cockpit, but disabled for now
    - name: "COCKPIT | ensure packages for https://cockpit-project.org/ are installed"
      package:
        name:
          - cockpit
          - cockpit-selinux
          - cockpit-machines
          - cockpit-kdump
          - cockpit-storaged
          - cockpit-system
        state: present
        update_cache: no
    - name: "COCKPIT | ensure cockpit.socket is stopped and disabled"
      systemd:
        name:       cockpit.socket
        state:      stopped
        enabled:    False
    - name: "COCKPIT | ensure firewalld forbids service cockpit in zone public"
      firewalld:
        service:    cockpit
        zone:       public
        permanent:  True
        state:      disabled
        immediate:  True

    # disable libvirtd
    - name: "Ensure libvirtd.service is disabled and stopped"
      systemd:
        name:         libvirtd.service
        state:        stopped
        enabled:      False

    # enable kdump.service since kickstart now creates a sufficiently large /var/crash
    # alternatively, you could set up netdump
    - name: "Ensure kdump.service is enabled and started"
      systemd:
        name:         kdump.service
        state:        started
        enabled:      True

    # podman
    - name: "PACKAGE | ensure podman is installed"
      package:
        name:
          - podman
          - podman-docker
        state: present

    # setroubleshoot, see also https://danwalsh.livejournal.com/20931.html
    - name: "PACKAGE | ensure setroubleshoot for headless server is installed"
      package:
        name:
          - setroubleshoot-server
          - setroubleshoot-plugins
        state: present

    - name: "MONITORING | ensure packages for monitoring are installed"
      package:
        name:
          - smartmontools
          - hdparm
          - check-mk-agent
        state: present

    - name: "MONITORING | ensure firewalld permits 6556 in zone public for check-mk-agent"
      firewalld:
        port:       6556/tcp
        permanent:  True
        state:      enabled
        immediate:  True
        zone:       public
    - name: "MONITORING | ensure tarsnap cache is in fileinfo"
      lineinfile:
        path: /etc/check_mk/fileinfo.cfg
        line: "/usr/local/tarsnap-cache/cache"
        create: yes
    - name: "MONITORING | ensure entropy_avail plugin for Check_MK is present"
      template:
        src:        templates/check-mk-agent-plugin-entropy_avail.j2
        dest:       /usr/lib/check_mk_agent/plugins/entropy_avail
        mode:       0755
        group:      root
        owner:      root
    - name: "MONITORING | plugins from running CEE instance"
      get_url:
        url: "http://check-mk.internal.pcfe.net/HouseNet/check_mk/agents/plugins/{{ item }}"
        dest: "/usr/lib/check_mk_agent/plugins/{{ item }}"
        mode: "0755"
      loop:
        - smart
        - lvm
    - name: "MONITORING | ensure check_mk.socket is started and enabled"
      systemd:
        name:       check_mk.socket
        state:      started
        enabled:    True

    # I admit, the regexp is a search engine hit
    # maybe using grubby(8) would be more readable
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel#what-is-grubby_configuring-kernel-command-line-parameters
    # - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sec-Making_Persistent_Changes_to_a_GRUB_2_Menu_Using_the_grubby_Tool
    - name: "GRUB | ensure console blanking is disabled in defaults file plus handler"
      lineinfile:
        state: present
        dest: /etc/default/grub
        backrefs: yes
        regexp: '^(GRUB_CMDLINE_LINUX=(?!.* consoleblank)\"[^\"]+)(\".*)'
        line: '\1 consoleblank=0\2'
      notify: grub2-mkconfig | run

    # Since I do not manage to get these F5-422 to PXE boot, add an entry into grub
    # so that I can kickstart the box after this without fiddling with a USB stick
    - name: "GRUB | ensure initrd for RHEL 8.2 kickstart is present"
      get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.2/Server/x86_64/os/images/pxeboot/initrd.img"
        dest: "/boot/initrd-kickstart-rhel82.img"
        mode: "0600"
    - name: "GRUB | ensure kernel for RHEL 8.2 kickstart is present"
      get_url:
        url: "ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.2/Server/x86_64/os/images/pxeboot/vmlinuz"
        dest: "/boot/vmlinuz-kickstart-rhel82"
        mode: "0755"
    - name: "GRUB | ensure kickstarting RHEL 8.2 entry is present"
      copy:
        dest: "/etc/grub.d/11_RHEL82_kickstart"
        owner: "root"
        group: "root"
        mode: 0755
        content: |
          #!/bin/sh
          exec tail -n +3 $0
          # This file provides an easy way to add custom menu entries.  Simply type the
          # menu entries you want to add after this comment.  Be careful not to change
          # the 'exec tail' line above.
          menuentry "WARNING Kickstart this box with RHEL 8.2 as a F5-422 ceph node WARNING" {
              linuxefi /vmlinuz-kickstart-rhel82 ip=dhcp inst.repo=ftp://fileserver.internal.pcfe.net/pub/redhat/RHEL/RHEL-8.2/Server/x86_64/os inst.ks=ftp://fileserver.internal.pcfe.net/pub/kickstart/RHEL82-TerraMaster-F5-422-ks.cfg
              initrdefi /initrd-kickstart-rhel82.img
          }
      notify: grub2-mkconfig | run


    # upgrade the box
    - name: "package | ensure all updates are applied"
      package:
        update_cache: yes
        name: '*'
        state: latest
      tags: apply_errata

(click the triangle to expand)

Test Watchdog by Forcing Crash

Verify that the watchdog works as expected.

As root, on a F5-422:

echo '1' > /proc/sys/kernel/sysrq
echo 'c' > /proc/sysrq-trigger

As expected, I get an oops output on the console acter the c and, as previously tested, the box reboots itself after 30 seconds.

Planned Ceph Rollout

I had planned to roll out Red Hat Ceph Storage 4 containerised on theses RHEL 8.1 nodes after I ran into this bug on CentOS Stream.

My Ceph-Specific Playbook

Simply follows the steps from 2. Requirements for Installing Red Hat Ceph Storage in the Red Hat Ceph Storage 4 (aka Nautilus) docs.

As user, on my F31 x96_64 workstation.

ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini ceph-nautilus-F4-422-cluster-prepare-x86.yml
full playbook ceph-nautilus-F4-422-cluster-prepare-x86.yml
- hosts:
  - TerraMaster_boxes
  - ceph-ansible
  - ganesha
  become: true
  roles:
    - pcfe.user_owner
    - pcfe.basic-security-setup
    - pcfe.housenet
    - pcfe.comfort
    #- pcfe.checkmk
    # note though that check-mk-agent is in EPEL and RHCS 4 docs say EPEL needs to be off
    # for now I manually installed the rpm from the Rawb Edition server
  vars:
    ansible_user: ansible
    common_timezone: Europe/Berlin

    # RHSM pre-task moves to `terramaster-initial-setup-rhel8.yml`

  tasks:
    # !!!
    #
    # if I ever enable EPEL, then I MUST exclude
    # - ansible
    # - ceph
    # in the EPEL repo files to ensure no newer versions of those packages are pulled in from EPEL
    #
    # exclude = *ceph* nfs-ganesha-rgw rbd-mirror *ansible*
    #
    # !!!
    - name: "REPOS | ensure EPEL is disabled"
      yum_repository:
        name: epel
        state: absent

    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#requirements-for-installing-rhcs

    # Ansible el7 control node, also runs the playbook that installs Ceph Dashboard on the host ceph-dashboard
    - name: "Prepare the el7 Ansible administration node"
      block:
        - name: "RHSM | ensure all repos for ceph-ansible RHEL7 usage are enabled"
          rhsm_repository:
            name:
              - rhel-7-server-rhceph-4-tools-rpms
              - rhel-7-server-ansible-2.8-rpms
            state: enabled
        - name: "PACKAGE | ensure packages for admin node are present"
          package:
            name:
              - ceph-ansible
              - mlocate
            state: present
      when:
        - ansible_hostname == 'ceph-ansible'
        - ansible_distribution == "RedHat" and ansible_distribution_major_version == "7"

    # Ansible el8 control node, also runs the playbook that installs Ceph Dashboard on the host ceph-dashboard
    - name: "Prepare the el8 Ansible administration node"
      block:
        - name: "RHSM | ensure all repos for ceph-ansible RHEL8 usage are enabled"
          rhsm_repository:
            name:
              - rhceph-4-tools-for-rhel-8-x86_64-rpms
              - ansible-2.8-for-rhel-8-x86_64-rpms
            state: enabled
        - name: "PACKAGE | ensure packages for admin node are present"
          package:
            name:
              - ceph-ansible
              - mlocate
            state: present
      when:
        - ansible_hostname == 'ceph-ansible'
        - ansible_distribution == "RedHat" and ansible_distribution_major_version == "8"

    # # dashboard
    # # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/administration_guide/monitoring#installing-the-red-hat-ceph-storage-dashboard-management
    # - name: "Prepare the Ansible Dashboard node"
    #   block:
    #     - name: "RHSM | ensure all repos for ceph-dashboard usage are enabled"
    #       rhsm_repository:
    #         name:
    #           - rhel-7-server-rhceph-4-tools-rpms
    #           - rhel-7-server-ansible-2.8-rpms
    #         state: enabled
    #   when: ansible_hostname == 'ceph-dashboard'

    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html-single/installation_guide/index#configuring-a-firewall-for-red-hat-ceph-storage-install
    # remember to configure_firewall: True
    - name:         "FIREWALLD | ensure firewall is enabled and running"
      service:
        name:       firewalld
        enabled:    true
        state:      started

    # NFS Ganesha
    # https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/installation_guide/installing-red-hat-ceph-storage-using-ansible#installing-the-nfs-ganesha-gateway-install
    - name: "GANESHA prep | ensure rpcbind and nfs-utils are present"
      package:
        name:
          - rpcbind
          - nfs-utils
        state: present
      when: ansible_hostname == 'ganesha'
    - name:         "GANESHA prep | ensure rpcbind is enabled and running"
      service:
        name:       rpcbind
        enabled:    true
        state:      started
      when: ansible_hostname == 'ganesha'
    - name:         "GANESHA prep | ensure nfs-server.service is disabled and stopped"
      service:
        name:       nfs-server.service
        enabled:    false
        state:      stopped
      when: ansible_hostname == 'ganesha'

    - name: "dump VCS controlled files onto Ansible administration node"
      block:
        # Assuming daemonized deployment.. this probably has something to do with it: https://github.com/ceph/ceph-ansible/blob/v4.0.23/roles/ceph-config/tasks/main.yml#L140
        # else containerized:
        # In which case it's dropped in /etc/ceph : https://github.com/ceph/ceph-ansible/blob/v4.0.23/roles/ceph-config/tasks/main.yml#L163
        # - name: "FILE | ensure ~ansible/ceph-ansible-keys/ exists"
        #   file:
        #     owner:  ansible
        #     group:  ansible
        #     mode:   0700
        #     path:   /home/ansible/ceph-ansible-keys
        #     state:  directory
        - name: 'FILE | ensure /usr/share/ceph-ansible/host_vars/ exists'
          file:
            owner:  ansible
            group:  ansible
            mode:   0755
            path:   /usr/share/ceph-ansible/host_vars
            state:  directory
        - name: 'FILE | ensure a symbolic link to the /usr/share/ceph-ansible/host_vars directory in the /etc/ansible/ directory exists'
          file:
            src:    /usr/share/ceph-ansible/host_vars
            dest:   /etc/ansible/host_vars
            state:  link
        - name: 'FILE | ensure a symbolic link to the /usr/share/ceph-ansible/group_vars directory in the /etc/ansible/ directory exists'
          file:
            src:    /usr/share/ceph-ansible/group_vars
            dest:   /etc/ansible/group_vars
            state:  link
        - name: "FILE | Create a symbolic link to the /usr/share/ceph-ansible/group_vars directory in the /etc/ansible/ directory"
          file:
            src:    /usr/share/ceph-ansible/group_vars
            dest:   /etc/ansible/group_vars
            state:  link
        - name: "VIMRC | on yaml files, autoindent tabstop=2 shiftwidth=2 expandtab"
          lineinfile:
            create:       true
            insertafter:  EOF
            line:         autocmd FileType yaml setlocal autoindent tabstop=2 shiftwidth=2 expandtab
            mode:         0644
            path:         "~{{ ansible_user }}/.vimrc"
            regexp:       ^autocmd FileType yaml
            state:        present
        - name: "FILE | copy my files to ceph-ansible control node"
          copy:
            src: '{{ item.src }}'
            dest: '{{ item.dest }}'
            owner: ansible
            group: ansible
            mode: 0644
          with_items:
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/ceph-F5-422-cluster.ini', dest: '/etc/ansible/hosts' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-all.yml', dest: '/usr/share/ceph-ansible/group_vars/all.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-osds.yml', dest: '/usr/share/ceph-ansible/group_vars/osds.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-mdss.yml', dest: '/usr/share/ceph-ansible/group_vars/mdss.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-rgws.yml', dest: '/usr/share/ceph-ansible/group_vars/rgws.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-nfss.yml', dest: '/usr/share/ceph-ansible/group_vars/nfss.yml' }
            # - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-mgrs.yml', dest: '/usr/share/ceph-ansible/group_vars/mgrs.yml' }
            # - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph-F5-422-clients.yml', dest: '/usr/share/ceph-ansible/group_vars/clients.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/pcfe.net/ceph-HouseNet-site-container.yml', dest: '/usr/share/ceph-ansible/site-container.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/t7910.yml', dest: '/usr/share/ceph-ansible/host_vars/t7910.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/karhu.yml', dest: '/usr/share/ceph-ansible/host_vars/karhu.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/f5-422-01.yml', dest: '/usr/share/ceph-ansible/host_vars/f5-422-01.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/f5-422-02.yml', dest: '/usr/share/ceph-ansible/host_vars/f5-422-02.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/f5-422-03.yml', dest: '/usr/share/ceph-ansible/host_vars/f5-422-03.yml' }
            - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/f5-422-04.yml', dest: '/usr/share/ceph-ansible/host_vars/f5-422-04.yml' }
        #     - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/overdrive-1000.yml', dest: '/usr/share/ceph-ansible/host_vars/overdrive-1000.yml' }
        #     - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/group_vars/ceph_fedora_nodes.yml', dest: '/usr/share/cephmetrics-ansible/group_vars/ceph_fedora_nodes.yml' }
        #     - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/openshift-master.yml', dest: '/usr/share/ceph-ansible/host_vars/openshift-master.yml' }
        #     - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/openshift-router.yml', dest: '/usr/share/ceph-ansible/host_vars/openshift-router.yml' }
        #     - { src: '/home/pcfe/work/git/HouseNet/ansible/inventories/host_vars/openshift-node-01.yml', dest: '/usr/share/ceph-ansible/host_vars/openshift-node-01.yml' }
      when: ansible_hostname == 'ceph-ansible'

(click the triangle to expand)

old Prepare SSD and HDDs

I used to share one 500 GB SSD between the OS (60GB) and as multiple db for the HDDs with

lvm_volumes:
  - data:         LV_HDD_bay2_OSD_data
    data_vg:      VG_HDD_bay2_Ceph
    db:           LV_SSD_bay1_OSD_db_01
    db_vg:        VG_SSD_bay1_Ceph
  - data:         LV_HDD_bay3_OSD_data
    data_vg:      VG_HDD_bay3_Ceph
    db:           LV_SSD_bay1_OSD_db_02
    db_vg:        VG_SSD_bay1_Ceph

For that one has to pre create the VGs and LVs oneself.

As user, on my F31 x96_64 workstation I would run.

ansible-playbook -i ../inventories/ceph-F5-422-cluster.ini terramaster-prep-disks.yml
# sets partitions on my TerraMaster F5-422 boxes
# OS is installed in VG_SSD_bay1_OS (of 65 GBiB) on the SSD in bay 1
# HDDs should be unused until this is run
- hosts:
  - TerraMaster_boxes
  become: yes
  # to avoid using half arsed names like sda, sdb, etc
  vars:
    storage_bay1: /dev/disk/by-path/pci-0000:00:12.0-ata-1
    storage_bay2: /dev/disk/by-path/pci-0000:00:12.0-ata-2
    storage_bay3: /dev/disk/by-path/pci-0000:03:00.0-ata-1
    storage_bay4: /dev/disk/by-path/pci-0000:03:00.0-ata-2
    storage_bay5: /dev/disk/by-path/pci-0000:04:00.0-ata-1

  # inspired by https://www.epilis.gr/en/blog/2017/08/09/extending-root-fs-whole-farm/
  tasks:
    - name: "PARTITIONS | get partition information of OS device in bay 1"
      parted:
        device: "{{ storage_bay1 }}"
      register: bay1_info
    - debug: var=bay1_info
    # evtl set fact fuer die Partition?
    - block:
      - name: "PARTITIONS | if more than 160 MiB space left on the SSD, then create a new partition after {{bay1_info.partitions[-1].end}}KiB"
        parted:
         part_start: "{{ bay1_info.partitions[-1].end }}KiB"
         device: "{{ storage_bay1 }}"
         number: "{{ bay1_info.partitions[-1].num + 1 }}"
         flags: [ lvm ]
         label: gpt
         state: present
      - name: "PARTITIONS | partprobe after change to OS device in bay 1"
        command: partprobe
      - name: "LVM | create VG_SSD_bay1_Ceph using PV {{ storage_bay1 }}-part{{ bay1_info.partitions[-1].num + 1 }}"
        lvg:
          vg:   VG_SSD_bay1_Ceph
          pvs:  "{{ storage_bay1 }}-part{{ bay1_info.partitions[-1].num + 1 }}"
      when: (bay1_info.partitions[-1].end + 163840) < bay1_info.disk.size

    - name: "PARTITIONS | get partition information of the device in bay 2"
      parted:
        device: "{{ storage_bay2 }}"
      register: bay2_info
    - debug: var=bay2_info.partitions
    - block:
      - name: "PARTITIONS | if no partitions on the device in bay 2, then create one covering whole disk"
        parted:
         part_start: "0%"
         part_end: "100%"
         device: "{{ storage_bay2 }}"
         number: 1
         flags: [ lvm ]
         label: gpt
         state: present
      - name: "PARTITIONS | partprobe after change to device in bay 2"
        command: partprobe
      - name: "LVM | create VG_Ceph_HDD_bay2 using PV {{ storage_bay2 }}-part1"
        lvg:
          vg:   VG_HDD_bay2_Ceph
          pvs:  "{{ storage_bay2 }}-part1"
      - name: "LVM | create LV_HDD_bay2_OSD_data in VG_HDD_bay2_Ceph"
        lvol:
          vg:   VG_HDD_bay2_Ceph
          lv:   LV_HDD_bay2_OSD_data
          size: 100%FREE
      - name: "LVM | create LV_SSD_bay1_OSD_db_01 in VG_SSD_bay1_Ceph"
        lvol:
          vg:   VG_SSD_bay1_Ceph
          lv:   LV_SSD_bay1_OSD_db_01
          size: 100G
          # VG_SSD_bay1_Ceph is currently 414.26g
          # I have 4 slots left for HDD, so just make it 100G
          # 80G is roughly 4% of 2TB
          # 40G is roughly 4% of 1TB
      - name: "PARTITIONS | get partition information of the HDD in bay 2 again"
        parted:
          device: "{{ storage_bay2 }}"
        register: bay2_info
      when: not bay2_info.partitions|length
    - debug: var=bay2_info.partitions


    - name: "PARTITIONS | get partition information of the HDD in bay 3"
      parted:
        device: "{{ storage_bay3 }}"
      register: bay3_info
    - debug: var=bay3_info.partitions
    - block:
      - name: "PARTITIONS | if no partitions on the device in bay 3, then create one covering whole disk"
        parted:
         part_start: "0%"
         part_end: "100%"
         device: "{{ storage_bay3 }}"
         number: 1
         flags: [ lvm ]
         label: gpt
         state: present
      - name: "PARTITIONS | partprobe after change to device in bay 3"
        command: partprobe
      - name: "LVM | create VG_HDD_bay3_Ceph using PV {{ storage_bay3 }}-part1"
        lvg:
          vg: VG_HDD_bay3_Ceph
          pvs: "{{ storage_bay3 }}-part1"
      - name: "LVM | create LV_HDD_bay3_OSD_data in VG_HDD_bay3_Ceph"
        lvol:
          vg:   VG_HDD_bay3_Ceph
          lv:   LV_HDD_bay3_OSD_data
          size: 100%FREE
      - name: "LVM | create LV_SSD_bay1_OSD_db_02 in VG_SSD_bay1_Ceph"
        lvol:
          vg:   VG_SSD_bay1_Ceph
          lv:   LV_SSD_bay1_OSD_db_02
          size: 100G
      - name: "PARTITIONS | get partition information of the device in bay 3 again"
        parted:
          device: "{{ storage_bay3 }}"
        register: bay3_info
      when: not bay3_info.partitions|length
    - debug: var=bay3_info.partitions

new dedicated 120 GB SSD for OS

In early June 2020 I wanted to try a deployment with devices, so I did not run …-prep-disks.yml (see https://docs.ceph.com/ceph-ansible/master/osds/scenarios.html for details) and have in my host_vars the following

# avoid using half arsed names like sda, sdb, etc
# mapping is:
# bay 1 /dev/disk/by-path/pci-0000:00:12.0-ata-1
# bay 2 /dev/disk/by-path/pci-0000:00:12.0-ata-2
# bay 3 /dev/disk/by-path/pci-0000:03:00.0-ata-1
# bay 4 /dev/disk/by-path/pci-0000:03:00.0-ata-2
# bay 5 /dev/disk/by-path/pci-0000:04:00.0-ata-1
dmcrypt: True
osd_objectstore:  bluestore
osd_scenario:     lvm
devices:
  - /dev/disk/by-path/pci-0000:00:12.0-ata-2
  - /dev/disk/by-path/pci-0000:03:00.0-ata-1
  - /dev/disk/by-path/pci-0000:03:00.0-ata-2
  - /dev/disk/by-path/pci-0000:04:00.0-ata-1

re-install nodes while Ceph actively in use

My aim was to do invasive maintenance (OS needs to be on another SSD than it currently is on) one each node in turn, and using Ceph’s self-healing to avoid making my CephFS and RBD pools unavailable while doing so.

Since it was quicker for me to kickstart the nodes again to get the OS onto the new 120 GB SSDs, I simply followed the remove OSDs procedure for one node, installed it anew, used sgdisk --zap-all … on the 500 GB SSD and the HDDs, and for the new nodes ran infrastructure-playbooks/add-osd.yml after following the instruction in the playbook as a first test with node 04 (besides OSDs it only ran grafana).

After this went fine (although the moving back and forth of data while Ceph heals itself of course takes time) I plan to do the other nodes with site-container.yml.

At this moment node 03 is being emptied of it’s data and will be re-installed tonight or tomorrow.

Run ceph-ansible

After the …cluster-prepare-x86.yml Playbook dumped inventory, host_vars and group_vars onto the Ansible control node. Alternatively, create those files manually as per docs. Remember to back them up if you use the maual approach.

As user ansible, on the RHEL8 VM ceph-ansible.internal.pcfe.net

ansible-playbook site-container.yml

Read the Docs WRT Containerized Ceph

Be sure to read https://github.com/ceph/ceph-container/blob/master/src/daemon/README.md

ceph -s with containers

[root@f5-422-01 ~]# systemctl status ceph-mon@f5-422-01.service 
● ceph-mon@f5-422-01.service - Ceph Monitor
   Loaded: loaded (/etc/systemd/system/ceph-mon@.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-03-08 03:00:21 CET; 28min ago
 Main PID: 28143 (podman)
    Tasks: 12 (limit: 26213)
   Memory: 42.9M
   CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@f5-422-01.service
           └─28143 /usr/bin/podman run --rm --name ceph-mon-f5-422-01 --memory=11694m --cpus=1 -v /var/lib/ceph:/var/>

Mar 08 03:28:45 f5-422-01.internal.pcfe.net podman[28143]: cluster 2020-03-08 03:28:42.049124 mgr.f5-422-02 (mgr.4696>
Mar 08 03:28:45 f5-422-01.internal.pcfe.net podman[28143]: cluster 2020-03-08 03:28:44.050143 mgr.f5-422-02 (mgr.4696>
Mar 08 03:28:50 f5-422-01.internal.pcfe.net podman[28143]: debug 2020-03-08 03:28:50.264 7fbc2c34b700  1 mon.f5-422-0>
Mar 08 03:28:55 f5-422-01.internal.pcfe.net podman[28143]: debug 2020-03-08 03:28:55.265 7fbc2c34b700  1 mon.f5-422-0>
[root@f5-422-01 ~]# podman ps
CONTAINER ID  IMAGE                                 COMMAND  CREATED         STATUS             PORTS  NAMES
4f44a896ab6d  quay.io/rhceph-dev/rhceph:latest-4.3           26 minutes ago  Up 26 minutes ago         ceph-osd-4
a8a08bac84a5  quay.io/rhceph-dev/rhceph:latest-4.3           26 minutes ago  Up 26 minutes ago         ceph-osd-0
c2c546eb2e48  quay.io/rhceph-dev/rhceph:latest-4.3           29 minutes ago  Up 29 minutes ago         ceph-mgr-f5-422-01
414e1f62efa8  quay.io/rhceph-dev/rhceph:latest-4.3           31 minutes ago  Up 31 minutes ago         ceph-mon-f5-422-01
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph -s
  cluster:
    id:     385fff79-9c77-4e09-9280-07993eb33988
    health: HEALTH_WARN
            too few PGs per OSD (9 < min 30)
 
  services:
    mon: 3 daemons, quorum f5-422-01,f5-422-02,f5-422-03 (age 29m)
    mgr: f5-422-02(active, since 18m), standbys: f5-422-01, f5-422-03
    osd: 8 osds: 8 up (since 24m), 8 in (since 24m)
 
  data:
    pools:   3 pools, 24 pgs
    objects: 0 objects, 0 B
    usage:   648 GiB used, 11 TiB / 12 TiB avail
    pgs:     24 active+clean

ceph -w with containers

[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph -w

Not sure yet, but this has given my trouble with the rolling upgrade twice now and IIRC I only tried rolling_upgrade.yml two times on this cluster.

see also https://github.com/containers/libpod/issues/3226

try this instead

[root@f5-422-01 ~]# journalctl -f --full -u ceph-mon@f5-422-01.service

Workaround for RHBZ #1793564

Until Bug 1793564 is fixed, work around as follows after ansible-playbook infrastructure-playbooks/rolling_update.yml finished;

[root@f5-422-01 ~]# podman exec --interactive --tty ceph-mon-f5-422-01 ceph -s
  cluster:
    id:     385fff79-9c77-4e09-9280-07993eb33988
    health: HEALTH_WARN
            norebalance flag(s) set
[...]
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd unset norebalance
norebalance is unset

Performance Baseline, pool with 8 PGs

I gathered a performance baseline based on the instructions at 6.2. Performance baseline of the RHCS 4 Administration Guide.

Performance Measurements with rados bench, pool with 8 PGs

All rados bench tests were run on f5-422-01.

Sequential Write Test, pool with 8 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_1206
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        36        20   79.9845        80    0.723583    0.576026
    2      16        61        45   89.9843       100    0.584442    0.606205
    3      16        85        69   91.9845        96    0.531685    0.619067
    4      16       112        96   95.9664       108    0.574702    0.633199
    5      16       130       114   91.1704        72    0.594669    0.633379
    6      16       155       139   92.6393       100        0.66    0.666516
    7      16       180       164   93.6886       100    0.155607    0.652325
    8      16       199       183   91.4749        76    0.544948    0.662251
    9      16       221       205   91.0849        88    0.559885     0.67024
   10      16       242       226   90.3741        84    0.132288     0.67139
   11      16       270       254   92.3382       112     0.92944    0.675088
   12      16       290       274   91.3083        80    0.118037    0.673343
   13      16       314       298   91.6678        96    0.316335    0.675676
   14      16       345       329   93.9754       124    0.527899    0.667295
   15      16       369       353   94.1091        96   0.0945891    0.665796
   16      16       394       378   94.4758       100    0.616406    0.668414
   17      16       415       399   93.8588        84    0.484051    0.668813
   18      16       442       426   94.6434       108    0.305182    0.661079
   19      16       470       454    95.556       112    0.215421    0.657006
2020-03-08 03:40:56.073907 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.662172
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       490       474   94.7773        80    0.684891    0.662172
   21      16       512       496   94.4475        88    0.562806    0.664983
   22      16       535       519   94.3261        92    0.457757    0.668154
   23      16       563       547   95.0915       112    0.561656    0.663839
   24      16       586       570   94.9612        92    0.347066    0.661026
   25      16       613       597    95.482       108    0.727905    0.661553
   26      16       643       627   96.4242       120    0.347148    0.656599
   27      16       666       650   96.2598        92    0.326133    0.656999
   28      16       689       673    96.107        92    0.317507    0.655335
   29      16       716       700   96.5156       108    0.429798    0.654834
   30      16       740       724   96.4979        96    0.589708    0.657404
   31      16       760       744   95.9647        80    0.586795    0.656781
   32      16       787       771     96.34       108    0.715718    0.659763
   33      16       808       792   95.9655        84    0.688204    0.657657
   34      16       832       816   95.9658        96    0.533666    0.660013
   35      16       855       839   95.8521        92    0.777156    0.659789
   36      16       881       865   96.0779       104    0.603781     0.65963
   37      16       909       893   96.5074       112    0.273226    0.658766
   38      16       930       914   96.1777        84    0.566452    0.656826
   39      16       958       942   96.5829       112    0.543521    0.657998
2020-03-08 03:41:16.082412 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.660309
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16       978       962    96.168        80    0.983711    0.660309
   41      16      1003       987   96.2603       100    0.837931    0.660551
   42      16      1025      1009   96.0633        88     0.32828    0.662518
   43      16      1049      1033   96.0615        96     1.02893     0.66174
   44      16      1069      1053   95.6961        80    0.248816    0.663268
   45      16      1087      1071   95.1692        72      1.0122    0.664249
   46      16      1110      1094   95.0999        92    0.199519     0.66612
   47      16      1141      1125   95.7142       124     0.10855    0.665186
   48      16      1163      1147   95.5527        88    0.338549    0.664097
   49      16      1194      1178   96.1326       124    0.436331    0.661901
   50      16      1214      1198   95.8096        80     1.05029    0.661884
   51      16      1237      1221   95.7344        92    0.772366    0.664174
   52      16      1258      1242   95.5084        84    0.641119    0.664483
   53      16      1283      1267   95.5927       100    0.931158    0.665329
   54      16      1309      1293    95.748       104    0.770038    0.665611
   55      16      1335      1319   95.8976       104    0.968497    0.664508
   56      16      1353      1337   95.4707        72     0.87869     0.66501
   57      16      1382      1366   95.8301       116     0.84359    0.663624
   58      16      1406      1390   95.8327        96    0.930628    0.663563
   59      16      1426      1410    95.564        80     1.01355    0.664224
2020-03-08 03:41:36.087460 min lat: 0.0813909 max lat: 1.54837 avg lat: 0.663119
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1454      1438   95.8374       112    0.855286    0.663119
   61      16      1479      1463   95.9052       100    0.660708    0.662736
   62      16      1502      1486   95.8419        92    0.651422    0.663765
   63      16      1525      1509   95.7808        92     0.65479    0.663624
   64      16      1548      1532   95.7214        92    0.108128    0.664383
   65      16      1570      1554   95.6015        88    0.129301    0.664194
   66      16      1596      1580   95.7283       104    0.833698     0.66505
   67      16      1616      1600   95.4931        80    0.484905    0.665961
   68      16      1649      1633   96.0295       132    0.279223     0.66307
   69      16      1676      1660   96.2024       108    0.573273    0.661686
   70      16      1700      1684    96.199        96     0.17875    0.661252
   71      16      1727      1711   96.3648       108    0.677561    0.660304
   72      16      1748      1732   96.1926        84    0.585211    0.661393
   73      16      1777      1761   96.4636       116    0.570952    0.659998
   74      16      1799      1783   96.3491        88    0.424756    0.661915
   75      16      1824      1808   96.3975       100    0.594112    0.661561
   76      16      1845      1829   96.2339        84    0.434104    0.661296
   77      16      1869      1853   96.2304        96    0.188077    0.660654
   78      16      1895      1879   96.3297       104    0.600443    0.661122
   79      16      1923      1907   96.5279       112      0.6466    0.660552
2020-03-08 03:41:56.093084 min lat: 0.07524 max lat: 1.54837 avg lat: 0.659295
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      1946      1930   96.4711        92    0.194369    0.659295
   81      16      1973      1957   96.6131       108    0.477408    0.659159
   82      16      2007      1991    97.093       136    0.700025    0.657076
   83      16      2030      2014   97.0315        92    0.115752      0.6568
   84      16      2054      2038   97.0188        96    0.492626    0.656201
   85      16      2081      2065   97.1476       108    0.260029    0.656981
   86      16      2107      2091   97.2271       104    0.196699    0.655703
   87      16      2133      2117   97.3047       104    0.683626    0.655943
   88      16      2156      2140   97.2443        92    0.505637    0.655697
   89      16      2185      2169   97.4544       116    0.509705     0.65502
   90      16      2205      2189   97.2601        80    0.101186    0.655156
   91      16      2234      2218   97.4657       116    0.581524    0.654454
   92      16      2254      2238   97.2755        80    0.964339    0.653804
   93      16      2282      2266   97.4336       112    0.674037    0.654363
   94      16      2309      2293   97.5458       108    0.663199    0.654833
   95      16      2333      2317   97.5292        96    0.446743    0.652527
   96      16      2361      2345   97.6796       112    0.546457    0.652748
   97      16      2383      2367   97.5795        88    0.969492    0.652821
   98      16      2406      2390   97.5223        92    0.771261    0.653418
   99      16      2430      2414   97.5065        96    0.656395    0.653307
2020-03-08 03:42:16.098625 min lat: 0.07524 max lat: 1.54837 avg lat: 0.65362
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      2457      2441   97.6112       108    0.782992     0.65362
  101      16      2485      2469   97.7534       112    0.623987     0.65313
  102      16      2509      2493    97.736        96    0.504933    0.652843
  103      16      2531      2515   97.6413        88   0.0907685    0.652572
  104      16      2556      2540   97.6639       100    0.571569    0.652897
  105      16      2580      2564   97.6479        96    0.969512    0.653634
  106      16      2599      2583   97.4434        76    0.622518    0.654013
  107      16      2620      2604   97.3175        84    0.721271    0.654591
  108      16      2643      2627   97.2681        92    0.900107    0.655839
  109      16      2663      2647   97.1095        80    0.570751    0.656755
  110      16      2685      2669   97.0264        88    0.389845    0.656463
  111      16      2713      2697   97.1611       112     0.83569    0.657452
  112      16      2735      2719   97.0789        88   0.0963093    0.657552
  113      16      2760      2744   97.1044       100    0.629961     0.65806
  114      16      2786      2770   97.1644       104    0.321872    0.657039
  115      16      2804      2788   96.9453        72    0.476548    0.657546
  116      16      2831      2815   97.0369       108    0.194845     0.65761
  117      16      2852      2836   96.9235        84    0.622427    0.657697
  118      16      2875      2859   96.8813        92     1.15613    0.658838
  119      16      2898      2882   96.8398        92    0.133981    0.658353
2020-03-08 03:42:36.111236 min lat: 0.0735553 max lat: 1.54837 avg lat: 0.658728
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      2925      2909   96.9326       108    0.609364    0.658728
  121      16      2945      2929   96.7926        80    0.381071    0.658807
  122      16      2970      2954   96.8187       100    0.123735    0.658164
  123      16      2996      2980    96.877       104    0.851339    0.658832
  124      16      3029      3013     97.16       132    0.441264    0.657168
  125      16      3051      3035   97.0866        88    0.826064    0.657295
  126      16      3078      3062   97.1731       108    0.499654     0.65676
  127      16      3104      3088   97.2262       104    0.437078    0.657114
  128      16      3125      3109   97.1228        84    0.218222    0.657184
  129      16      3147      3131   97.0518        88    0.686379    0.657279
  130      16      3178      3162   97.2589       124    0.507535    0.656603
  131      16      3204      3188   97.3101       104    0.539586    0.655695
  132      16      3232      3216   97.4209       112    0.873489    0.655031
  133      16      3255      3239   97.3798        92    0.340408    0.655164
  134      16      3289      3273   97.6678       136    0.345793    0.654292
  135      16      3314      3298   97.6821       100    0.529305    0.654292
  136      16      3341      3325   97.7576       108     0.48735    0.653234
  137      16      3370      3354   97.8906       116    0.263476    0.652577
  138      16      3389      3373    97.732        76    0.941394    0.653171
  139      16      3411      3395   97.6618        88    0.214443    0.653669
2020-03-08 03:42:56.120600 min lat: 0.0735553 max lat: 1.54837 avg lat: 0.653734
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      3433      3417   97.5927        88    0.674246    0.653734
  141      16      3454      3438   97.4962        84    0.617448    0.654569
  142      16      3479      3463   97.5137       100    0.218338    0.654403
  143      16      3500      3484   97.4179        84    0.853201     0.65507
  144      16      3525      3509   97.4356       100    0.680571    0.655173
  145      16      3548      3532   97.3981        92    0.575641    0.655856
  146      16      3574      3558   97.4432       104    0.121961    0.655444
  147      16      3599      3583   97.4604       100    0.553883    0.654605
  148      16      3623      3607   97.4502        96    0.517809    0.654811
  149      16      3652      3636   97.5745       116    0.518947    0.654115
  150      16      3672      3656   97.4572        80    0.663762    0.654778
  151      16      3697      3681   97.4738       100     0.75867    0.655086
  152      16      3721      3705   97.4639        96    0.511764    0.655137
  153      16      3743      3727   97.4017        88    0.512141    0.654982
  154      16      3768      3752   97.4184       100    0.705314    0.655127
  155      16      3799      3783   97.5896       124    0.271455    0.654865
  156      16      3826      3810   97.6561       108    0.584037    0.654706
  157      16      3848      3832   97.5945        88    0.332718    0.654546
  158      16      3875      3859   97.6601       108    0.744455    0.654254
  159      16      3897      3881   97.5984        88     0.45706    0.653767
2020-03-08 03:43:16.129325 min lat: 0.0731889 max lat: 1.54837 avg lat: 0.653318
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      3926      3910   97.7132       116    0.120934    0.653318
  161      16      3953      3937   97.7768       108    0.909301    0.653319
  162      16      3975      3959   97.7163        88    0.522766    0.653119
  163      16      4002      3986   97.7792       108    0.413138    0.653256
  164      16      4026      4010   97.7681        96     0.74238    0.653492
  165      16      4049      4033   97.7331        92    0.424301    0.653297
  166      16      4076      4060   97.7948       108    0.558388    0.653234
  167      16      4100      4084    97.784        96    0.161245    0.653062
  168      16      4124      4108   97.7731        96      0.1536    0.652695
  169      16      4152      4136   97.8572       112    0.677584    0.652788
  170      16      4170      4154   97.7049        72    0.565134    0.653094
  171      16      4191      4175   97.6247        84    0.749154    0.653765
  172      16      4217      4201   97.6617       104     0.60786    0.653897
  173      16      4242      4226   97.6743       100    0.404547    0.653995
  174      16      4265      4249   97.6416        92     0.68237    0.654205
  175      16      4288      4272   97.6092        92    0.682304    0.654389
  176      16      4310      4294   97.5546        88      1.3512    0.654532
  177      16      4327      4311   97.3874        68     1.04726    0.655169
  178      16      4352      4336   97.4018       100    0.367301    0.655149
  179      16      4383      4367   97.5502       124    0.838149    0.654715
2020-03-08 03:43:36.136047 min lat: 0.0731889 max lat: 1.81909 avg lat: 0.654916
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      4406      4390   97.5193        92    0.503992    0.654916
  181      16      4434      4418   97.5991       112    0.675084    0.654778
  182      16      4457      4441   97.5683        92    0.389599    0.654577
  183      16      4481      4465   97.5593        96    0.867131    0.654497
  184      16      4505      4489     97.55        96    0.526497     0.65482
  185      16      4526      4510   97.4767        84    0.724663    0.655057
  186      16      4550      4534   97.4687        96    0.763484    0.655434
  187      16      4574      4558   97.4607        96    0.627487    0.655562
  188      16      4596      4580   97.4103        88    0.884156    0.656023
  189      16      4622      4606   97.4451       104    0.426692    0.655285
  190      16      4649      4633   97.5004       108    0.197179    0.654889
  191      16      4674      4658   97.5134       100    0.597208    0.654921
  192      16      4698      4682   97.5054        96    0.269022    0.655068
  193      16      4723      4707   97.5181       100    0.635283    0.654982
  194      16      4744      4728   97.4483        84   0.0821723      0.6548
  195      16      4772      4756   97.5229       112    0.294763    0.654944
  196      16      4795      4779   97.4946        92    0.757824    0.655168
  197      16      4819      4803   97.4868        96    0.710587    0.655305
  198      16      4835      4819   97.3177        64    0.274916      0.6554
  199      16      4861      4845   97.3512       104    0.844698    0.655809
2020-03-08 03:43:56.142296 min lat: 0.0731889 max lat: 1.81909 avg lat: 0.655642
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      4891      4875   97.4643       120    0.686626    0.655642
  201      16      4923      4907   97.6161       128    0.429688    0.655042
  202      16      4941      4925   97.4891        72    0.130057    0.654871
  203      16      4967      4951    97.521       104     0.81226    0.655342
  204      16      4989      4973   97.4743        88    0.561826    0.655461
  205      16      5012      4996   97.4474        92   0.0867183    0.655194
  206      16      5037      5021   97.4596       100    0.650913    0.655479
  207      16      5060      5044   97.4332        92      1.0034    0.655956
  208      16      5083      5067    97.407        92    0.199391    0.655856
  209      16      5110      5094   97.4575       108    0.403252    0.655701
  210      16      5132      5116   97.4124        88    0.882838    0.655858
  211      16      5158      5142   97.4435       104    0.818807     0.65572
  212      16      5187      5171    97.531       116    0.463549    0.655693
  213      16      5207      5191   97.4486        80   0.0813537    0.655316
  214      16      5237      5221   97.5539       120    0.407225    0.655021
  215      16      5260      5244   97.5279        92    0.680947    0.654703
  216      16      5283      5267   97.5022        92    0.394172     0.65523
  217      16      5302      5286   97.4029        76    0.499632    0.655293
  218      16      5329      5313   97.4514       108     0.49312    0.655517
  219      16      5348      5332   97.3534        76    0.665668    0.655652
2020-03-08 03:44:16.147475 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.656167
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      5370      5354   97.3108        88    0.882733    0.656167
  221      16      5397      5381    97.359       108    0.553846    0.656445
  222      16      5423      5407   97.3885       104    0.492728    0.656124
  223      16      5452      5436   97.4718       116    0.604319    0.655787
  224      16      5476      5460   97.4651        96    0.442083    0.655512
  225      16      5502      5486   97.4931       104    0.257033    0.655599
  226      16      5525      5509   97.4687        92    0.145584    0.655854
  227      16      5546      5530   97.4093        84    0.464115    0.655692
  228      16      5574      5558   97.4732       112    0.583735    0.655675
  229      16      5600      5584   97.5016       104    0.657425    0.655223
  230      16      5623      5607   97.4776        92     0.85098    0.655599
  231      16      5644      5628   97.4192        84     1.03594    0.655795
  232      16      5672      5656    97.482       112    0.857936    0.655887
  233      16      5695      5679   97.4584        92    0.520848    0.655773
  234      16      5722      5706   97.5033       108    0.798259    0.655544
  235      16      5748      5732   97.5309       104    0.421147     0.65538
  236      16      5770      5754   97.4904        88     0.89876    0.655434
  237      16      5798      5782   97.5501       112    0.607513    0.655208
  238      16      5828      5812   97.6443       120     0.61895    0.654877
  239      16      5850      5834   97.6039        88    0.678715     0.65488
2020-03-08 03:44:36.157961 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.654761
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      5875      5859   97.6139       100    0.680681    0.654761
  241      16      5898      5882   97.5904        92    0.683184     0.65472
  242      16      5930      5914   97.7159       128    0.181657    0.654245
  243      16      5953      5937   97.6923        92    0.616281    0.654187
  244      16      5979      5963   97.7181       104    0.423063    0.653989
  245      16      6006      5990     97.76       108    0.537171    0.653939
  246      16      6028      6012   97.7203        88    0.398697    0.653941
  247      16      6053      6037   97.7294       100    0.564957    0.653728
  248      16      6080      6064   97.7707       108    0.890527    0.653698
  249      16      6098      6082   97.6671        72     0.60422    0.654047
  250      16      6121      6105   97.6444        92    0.596185    0.654216
  251      16      6143      6127   97.6058        88    0.185879    0.654253
  252      16      6171      6155   97.6629       112    0.576198    0.654299
  253      16      6194      6178   97.6404        92    0.433882    0.654087
  254      16      6226      6210   97.7598       128    0.834579    0.653943
  255      16      6249      6233   97.7371        92    0.702789    0.653851
  256      16      6277      6261   97.7927       112    0.370002    0.653654
  257      16      6300      6284   97.7701        92   0.0794214    0.653368
  258      16      6324      6308   97.7631        96    0.816806    0.653491
  259      16      6350      6334   97.7871       104     0.75631    0.653768
2020-03-08 03:44:56.162892 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.653396
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      6376      6360   97.8109       104    0.521866    0.653396
  261      16      6402      6386   97.8345       104    0.607561    0.653683
  262      16      6420      6404   97.7359        72    0.975247    0.653987
  263      16      6438      6422    97.638        72    0.738197    0.653971
  264      16      6464      6448    97.662       104    0.674055    0.653959
  265      16      6491      6475   97.7004       108    0.232167    0.654134
  266      16      6524      6508   97.8292       132    0.411683    0.653385
  267      16      6550      6534    97.852       104     0.53891    0.653122
  268      16      6576      6560   97.8748       104    0.639489    0.652857
  269      16      6599      6583   97.8529        92    0.452174    0.652917
  270      16      6616      6600   97.7422        68    0.799574    0.653141
  271      16      6645      6629   97.8094       116    0.538319    0.653322
  272      16      6670      6654   97.8173       100     1.16929    0.653417
  273      16      6693      6677   97.7958        92     1.12137    0.653402
  274      16      6717      6701   97.7891        96    0.531808    0.653514
  275      16      6744      6728   97.8261       108    0.602214    0.653414
  276      16      6765      6749   97.7759        84    0.381065    0.653348
  277      16      6791      6775   97.7982       104    0.823354    0.653629
  278      16      6812      6796   97.7486        84    0.165915    0.653343
  279      16      6841      6825   97.8139       116    0.227242    0.653518
2020-03-08 03:45:16.171457 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.653339
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      6870      6854   97.8785       116    0.396636    0.653339
  281      16      6889      6873   97.8005        76    0.439017    0.653331
  282      16      6914      6898   97.8083       100    0.537555    0.653592
  283      16      6940      6924   97.8301       104    0.852184    0.653501
  284      16      6964      6948    97.823        96     1.14132    0.653507
  285      16      6986      6970   97.7884        88    0.916644    0.653348
  286      16      7014      6998   97.8376       112    0.506791    0.653226
  287      16      7040      7024    97.859       104    0.572337    0.653072
  288      16      7060      7044   97.7969        80    0.522784    0.653251
  289      16      7089      7073   97.8598       116    0.353064    0.653152
  290      16      7120      7104   97.9499       124    0.185093    0.652811
  291      16      7152      7136    98.053       128    0.110128    0.652291
  292      16      7173      7157   98.0049        84    0.683066    0.652086
  293      16      7196      7180   97.9843        92    0.836498    0.652206
  294      16      7222      7206   98.0047       104    0.417568    0.652181
  295      16      7251      7235   98.0655       116    0.423006    0.652047
  296      16      7273      7257   98.0315        88    0.555123    0.652055
  297      16      7299      7283   98.0515       104    0.571609    0.652229
  298      16      7318      7302   97.9775        76    0.687783    0.652374
  299      16      7344      7328   97.9967       104    0.654878      0.6524
2020-03-08 03:45:36.181436 min lat: 0.0722055 max lat: 1.81909 avg lat: 0.652362
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      7370      7354   98.0166       104    0.172118    0.652362
  301      16      7390      7374   97.9567        80    0.637129    0.652306
  302      16      7416      7400   97.9767       104    0.174233    0.652427
  303      16      7436      7420   97.9173        80    0.863182    0.652697
  304      16      7457      7441   97.8715        84    0.730474    0.652911
  305      16      7486      7470   97.9308       116     0.36906    0.652781
  306      16      7506      7490   97.8721        80    0.166193    0.652849
  307      16      7530      7514   97.8659        96    0.666696     0.65314
  308      16      7551      7535   97.8209        84     1.08226    0.653418
  309      16      7575      7559   97.8149        96    0.647883    0.653323
  310      16      7599      7583    97.809        96    0.680075     0.65348
  311      16      7623      7607   97.8031        96    0.392244    0.653554
  312      16      7649      7633   97.8229       104    0.441637    0.653621
  313      16      7678      7662   97.8809       116    0.283462    0.653335
  314      16      7697      7681   97.8112        76    0.721894    0.653253
  315      16      7725      7709   97.8561       112    0.345154    0.653423
  316      16      7754      7738   97.9134       116    0.543706    0.652996
  317      16      7773      7757   97.8443        76    0.394007    0.653071
  318      16      7801      7785   97.8888       112    0.873398     0.65298
  319      16      7832      7816   97.9703       124    0.492376    0.652724
2020-03-08 03:45:56.186311 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.652228
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16      7862      7846   98.0391       120    0.306675    0.652228
  321      16      7888      7872   98.0576       104    0.628735    0.652143
  322      16      7913      7897   98.0636       100    0.364634    0.651911
  323      16      7945      7929   98.1562       128    0.401277    0.651503
  324      16      7971      7955    98.174       104     0.58735     0.65123
  325      16      7998      7982   98.2041       108    0.610951    0.651142
  326      16      8024      8008   98.2217       104     0.12831      0.6509
  327      16      8046      8030   98.1904        88     0.13002    0.650769
  328      16      8069      8053   98.1714        92    0.758953    0.650784
  329      16      8089      8073   98.1161        80    0.651354    0.651252
  330      16      8117      8101   98.1581       112    0.292907    0.651184
  331      16      8141      8125   98.1515        96    0.151734    0.651369
  332      16      8167      8151   98.1691       104    0.651185    0.651512
  333      16      8192      8176   98.1745       100    0.642366    0.651203
  334      16      8218      8202   98.1919       104     0.62424    0.651161
  335      16      8234      8218   98.0898        64    0.629394    0.651505
  336      16      8261      8245   98.1193       108    0.641029     0.65164
  337      16      8285      8269   98.1128        96    0.578223    0.651536
  338      16      8313      8297   98.1537       112    0.505778    0.651474
  339      16      8338      8322    98.159       100    0.553791    0.651275
2020-03-08 03:46:16.192894 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651397
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16      8360      8344    98.129        88    0.120949    0.651397
  341      16      8384      8368   98.1225        96    0.154636    0.651536
  342      16      8407      8391   98.1045        92    0.092277    0.651419
  343      16      8434      8418   98.1333       108    0.758751     0.65145
  344      16      8455      8439   98.0921        84    0.361099    0.651556
  345      16      8484      8468   98.1439       116    0.612094    0.651483
  346      16      8507      8491    98.126        92   0.0878817    0.651489
  347      16      8533      8517   98.1428       104     0.11184     0.65146
  348      16      8554      8538   98.1021        84    0.671261    0.651549
  349      16      8578      8562    98.096        96    0.225289    0.651457
  350      16      8606      8590   98.1356       112    0.642619    0.651639
  351      16      8629      8613   98.1181        92    0.500608    0.651663
  352      16      8652      8636   98.1003        92    0.877735    0.651581
  353      16      8679      8663   98.1283       108     0.69056    0.651748
  354      16      8706      8690    98.156       108    0.638846    0.651503
  355      16      8733      8717   98.1834       108    0.499441    0.651573
  356      16      8756      8740    98.166        92     0.52004    0.651275
  357      16      8778      8762   98.1374        88    0.716946    0.651083
  358      16      8814      8798   98.2654       144    0.284235      0.6509
  359      16      8832      8816   98.1922        72    0.173222    0.651092
2020-03-08 03:46:36.201780 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651138
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16      8855      8839   98.1749        92   0.0930602    0.651138
  361      16      8883      8867   98.2131       112    0.621431     0.65103
  362      16      8904      8888   98.1738        84    0.692532    0.650976
  363      16      8931      8915   98.2009       108    0.634838    0.650998
  364      16      8959      8943   98.2387       112    0.642821    0.650817
  365      16      8990      8974   98.3091       124    0.471871    0.650382
  366      16      9016      9000   98.3245       104   0.0863513    0.650132
  367      16      9042      9026   98.3399       104     0.86341    0.650207
  368      16      9066      9050   98.3335        96    0.240845    0.650217
  369      16      9092      9076   98.3487       104    0.155596    0.650023
  370      16      9114      9098   98.3206        88    0.270602    0.649957
  371      16      9144      9128    98.379       120    0.320459    0.649874
  372      16      9166      9150   98.3511        88    0.673788    0.650072
  373      16      9191      9175   98.3554       100   0.0950065    0.650089
  374      16      9213      9197   98.3265        88    0.730065    0.650268
  375      16      9242      9226   98.3735       116    0.708608    0.649841
  376      16      9264      9248   98.3459        88    0.738357    0.649942
  377      16      9297      9281   98.4351       132    0.451988    0.649633
  378      16      9325      9309   98.4709       112    0.751524    0.649429
  379      16      9347      9331   98.4433        88    0.231051     0.64933
2020-03-08 03:46:56.211448 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649382
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16      9374      9358   98.4684       108    0.894548    0.649382
  381      16      9393      9377   98.4094        76    0.258674    0.649351
  382      16      9417      9401    98.403        96    0.633855    0.649682
  383      16      9446      9430   98.4488       116    0.603489    0.649474
  384      16      9472      9456   98.4632       104    0.721873    0.649214
  385      16      9493      9477   98.4255        84    0.134118    0.649484
  386      16      9519      9503   98.4399       104    0.838446    0.649382
  387      16      9546      9530   98.4645       108    0.613837    0.649567
  388      16      9568      9552   98.4374        88    0.108603    0.649265
  389      16      9594      9578   98.4516       104    0.768979    0.649239
  390      16      9624      9608   98.5067       120    0.342783    0.649138
  391      16      9647      9631   98.4901        92   0.0767883    0.649052
  392      16      9677      9661   98.5449       120      0.8313    0.648956
  393      16      9699      9683    98.518        88    0.569558    0.648924
  394      16      9719      9703   98.4709        80     1.16451    0.649148
  395      16      9739      9723    98.424        80     1.00984     0.64933
  396      16      9772      9756   98.5088       132    0.601244    0.649229
  397      16      9795      9779   98.4923        92    0.755582    0.649343
  398      16      9824      9808   98.5363       116    0.381295    0.649113
  399      16      9846      9830   98.5098        88    0.986095    0.648989
2020-03-08 03:47:16.217239 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649237
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16      9871      9855   98.5135       100    0.811227    0.649237
  401      16      9890      9874   98.4573        76    0.880914    0.649308
  402      16      9910      9894   98.4114        80    0.727828    0.649551
  403      16      9936      9920   98.4252       104    0.499338     0.64959
  404      16      9961      9945   98.4291       100     0.45856    0.649678
  405      16      9977      9961    98.344        64    0.700774    0.649913
  406      16     10006      9990   98.3874       116    0.472278    0.650156
  407      16     10027     10011   98.3521        84   0.0919184    0.650152
  408      16     10049     10033   98.3265        88    0.646558    0.650333
  409      16     10080     10064   98.3891       124    0.730495    0.649984
  410      16     10101     10085    98.354        84    0.814081    0.650132
  411      16     10125     10109   98.3482        96    0.913662    0.650053
  412      16     10148     10132   98.3327        92    0.565592    0.650188
  413      16     10171     10155   98.3174        92     0.55227    0.650311
  414      16     10195     10179   98.3117        96    0.547988    0.650383
  415      16     10221     10205   98.3254       104    0.695495    0.650298
  416      16     10244     10228   98.3102        92    0.298014     0.65042
  417      16     10270     10254   98.3237       104     0.70301    0.650549
  418      16     10294     10278   98.3181        96    0.533756    0.650403
  419      16     10316     10300   98.2935        88    0.469423    0.650496
2020-03-08 03:47:36.222336 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650763
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     10335     10319   98.2404        76    0.741445    0.650763
  421      16     10359     10343    98.235        96    0.657943    0.650894
  422      16     10385     10369   98.2486       104     0.56101    0.650909
  423      16     10413     10397    98.281       112    0.240182    0.650805
  424      16     10431     10415   98.2188        72    0.585013    0.650777
  425      16     10448     10432   98.1477        68     1.33352    0.651027
  426      16     10471     10455   98.1331        92    0.550709    0.651319
  427      16     10497     10481   98.1468       104    0.587922    0.651463
  428      16     10521     10505   98.1417        96    0.574542    0.651528
  429      16     10540     10524   98.0901        76    0.500882    0.651556
  430      16     10570     10554   98.1409       120    0.244854    0.651698
  431      16     10600     10584   98.1915       120    0.559218    0.651374
  432      16     10626     10610   98.2049       104    0.406348    0.651151
  433      16     10653     10637   98.2272       108    0.206702     0.65111
  434      16     10676     10660   98.2128        92    0.672195    0.651088
  435      16     10698     10682   98.1892        88    0.782215    0.651299
  436      16     10715     10699   98.1199        68    0.924002    0.651413
  437      16     10744     10728   98.1608       116    0.651134     0.65159
  438      16     10765     10749   98.1283        84    0.504326    0.651649
  439      16     10789     10773   98.1234        96    0.659381    0.651608
2020-03-08 03:47:56.230544 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651727
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     10813     10797   98.1185        96    0.643826    0.651727
  441      16     10834     10818   98.0864        84    0.472631     0.65197
  442      16     10859     10843   98.0906       100    0.481773    0.652183
  443      16     10886     10870   98.1129       108    0.876912    0.651854
  444      16     10906     10890    98.072        80    0.112952    0.651808
  445      16     10930     10914   98.0672        96    0.588135       0.652
  446      16     10949     10933   98.0176        76    0.413525    0.651948
  447      16     10981     10965   98.0846       128    0.178786    0.652009
  448      16     11007     10991   98.0978       104    0.684228     0.65186
  449      16     11031     11015   98.0931        96    0.405251    0.651899
  450      16     11060     11044   98.1328       116    0.574425    0.651817
  451      16     11083     11067   98.1192        92    0.228945    0.651503
  452      16     11108     11092   98.1233       100    0.487363    0.651504
  453      16     11138     11122   98.1715       120    0.573114    0.651313
  454      16     11168     11152   98.2196       120    0.113332    0.651239
  455      16     11190     11174   98.1971        88    0.602596    0.651189
  456      16     11220     11204   98.2448       120    0.494584    0.651004
  457      16     11243     11227   98.2311        92    0.551401    0.650912
  458      16     11265     11249   98.2087        88    0.299751    0.650913
  459      16     11290     11274   98.2126       100    0.758831    0.651052
2020-03-08 03:48:16.236538 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.651116
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     11311     11295   98.1817        84    0.591291    0.651116
  461      16     11335     11319   98.1768        96    0.727311     0.65137
  462      16     11363     11347   98.2067       112    0.652792    0.651309
  463      16     11390     11374   98.2278       108    0.560692     0.65122
  464      16     11416     11400   98.2402       104    0.829137    0.651021
  465      16     11444     11428   98.2698       112    0.163392    0.650791
  466      16     11470     11454    98.282       104    0.454778    0.650629
  467      16     11494     11478   98.2771        96    0.316587    0.650594
  468      16     11521     11505   98.2978       108    0.353619    0.650679
  469      16     11543     11527   98.2758        88    0.569877    0.650651
  470      16     11566     11550   98.2624        92    0.878457      0.6508
  471      16     11591     11575   98.2661       100    0.407657    0.650821
  472      16     11611     11595   98.2273        80     0.77719    0.650922
  473      16     11637     11621   98.2394       104    0.520483    0.650859
  474      16     11666     11650   98.2767       116    0.594113    0.650738
  475      16     11685     11669   98.2297        76    0.479718    0.650856
  476      16     11713     11697   98.2586       112    0.369176    0.650822
  477      16     11743     11727   98.3039       120    0.809787    0.650624
  478      16     11766     11750   98.2907        92    0.663079    0.650689
  479      16     11783     11767   98.2274        68    0.805232    0.650822
2020-03-08 03:48:36.243342 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.65089
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     11808     11792    98.231       100    0.697284     0.65089
  481      16     11837     11821   98.2678       116    0.611617     0.65101
  482      16     11855     11839   98.2129        72     1.20229    0.651155
  483      16     11879     11863   98.2082        96    0.731059     0.65101
  484      16     11911     11895   98.2696       128    0.510453    0.650826
  485      16     11940     11924   98.3062       116    0.659076    0.650693
  486      16     11962     11946   98.2849        88    0.655587    0.650758
  487      16     11986     11970   98.2802        96    0.190026    0.650729
  488      16     12015     11999   98.3165       116    0.691374    0.650607
  489      16     12038     12022   98.3035        92    0.650487    0.650561
  490      16     12065     12049   98.3232       108    0.371408    0.650362
  491      16     12087     12071    98.302        88    0.471904    0.650409
  492      16     12112     12096   98.3054       100    0.713182    0.650466
  493      16     12136     12120   98.3007        96    0.691278     0.65061
  494      16     12162     12146   98.3121       104    0.120674    0.650506
  495      16     12185     12169   98.2993        92    0.740252     0.65063
  496      16     12208     12192   98.2865        92    0.695458    0.650776
  497      16     12228     12212   98.2497        80    0.531881    0.650684
  498      16     12252     12236   98.2451        96    0.675745    0.650922
  499      16     12277     12261   98.2485       100    0.549756    0.650907
2020-03-08 03:48:56.252444 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650717
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     12304     12288    98.268       108     0.38789    0.650717
  501      16     12327     12311   98.2554        92    0.522524    0.650629
  502      16     12355     12339   98.2819       112    0.704965    0.650842
  503      16     12387     12371   98.3409       128    0.265571    0.650511
  504      16     12407     12391   98.3045        80     1.04143    0.650469
  505      16     12435     12419   98.3315       112    0.651006    0.650557
  506      16     12456     12440   98.3031        84     0.84314    0.650636
  507      16     12475     12459   98.2587        76    0.547352    0.650639
  508      16     12504     12488   98.2935       116    0.155038    0.650645
  509      16     12531     12515   98.3125       108    0.475052    0.650498
  510      16     12559     12543   98.3392       112    0.547172    0.650479
  511      16     12586     12570    98.358       108    0.265545     0.65027
  512      16     12610     12594    98.353        96    0.678192    0.650154
  513      16     12635     12619   98.3561       100    0.756847    0.650182
  514      16     12662     12646   98.3747       108     0.68959    0.650072
  515      16     12690     12674   98.4007       112    0.585789    0.650125
  516      16     12709     12693   98.3572        76    0.489568    0.650178
  517      16     12737     12721   98.3836       112    0.608358    0.650071
  518      16     12767     12751   98.4252       120    0.469216    0.649885
  519      16     12788     12772   98.3973        84    0.834859     0.64992
2020-03-08 03:49:16.270759 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.649745
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     12814     12798    98.408       104    0.158556    0.649745
  521      16     12843     12827   98.4417       116    0.681851    0.649672
  522      16     12869     12853   98.4522       104    0.668717    0.649652
  523      16     12893     12877   98.4475        96    0.560833    0.649561
  524      16     12914     12898   98.4198        84    0.802026    0.649686
  525      16     12938     12922   98.4151        96    0.656702     0.64972
  526      16     12962     12946   98.4105        96    0.439403    0.649928
  527      16     12980     12964   98.3603        72    0.525097    0.650017
  528      16     13011     12995   98.4088       124    0.456089    0.650071
  529      16     13040     13024   98.4419       116    0.592353    0.649778
  530      16     13064     13048   98.4372        96    0.078142     0.64971
  531      16     13086     13070   98.4176        88    0.107893    0.649798
  532      16     13112     13096    98.428       104     0.91691    0.649908
  533      16     13135     13119   98.4159        92    0.597152    0.649901
  534      16     13155     13139   98.3814        80    0.563284    0.649952
  535      16     13176     13160   98.3545        84    0.612143    0.650101
  536      16     13199     13183   98.3426        92    0.832742    0.650123
  537      16     13228     13212   98.3754       116    0.823199    0.650273
  538      16     13247     13231   98.3338        76   0.0853203    0.650155
  539      16     13280     13264   98.3962       132    0.315826     0.65008
2020-03-08 03:49:36.277297 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650103
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     13298     13282   98.3473        72    0.953939    0.650103
  541      16     13328     13312   98.3872       120    0.713192    0.650181
  542      16     13356     13340   98.4123       112    0.505564    0.650028
  543      16     13381     13365   98.4152       100    0.552712    0.649885
  544      16     13400     13384    98.374        76    0.492521    0.649989
  545      16     13426     13410   98.3842       104    0.564447     0.65003
  546      16     13451     13435   98.3872       100     1.07467    0.650045
  547      16     13474     13458   98.3755        92    0.790483    0.650031
  548      16     13500     13484   98.3857       104     0.74705    0.650151
  549      16     13522     13506   98.3667        88    0.498086    0.650122
  550      16     13551     13535   98.3988       116    0.625565    0.650102
  551      16     13571     13555   98.3653        80    0.923903     0.65013
  552      16     13595     13579    98.361        96    0.535927    0.650129
  553      16     13625     13609   98.4001       120    0.685549     0.65004
  554      16     13646     13630   98.3741        84    0.931208    0.650087
  555      16     13670     13654   98.3698        96    0.561547    0.650125
  556      16     13691     13675   98.3439        84     1.04318    0.650216
  557      16     13719     13703   98.3684       112    0.677335    0.650262
  558      16     13742     13726   98.3569        92    0.134026    0.650171
  559      16     13766     13750   98.3527        96    0.268108    0.650323
2020-03-08 03:49:56.281318 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650447
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     13787     13771    98.327        84    0.538947    0.650447
  561      16     13817     13801   98.3656       120    0.262342    0.650401
  562      15     13837     13822   98.3398        84    0.470362    0.650362
  563      16     13863     13847   98.3427       100    0.435433    0.650324
  564      16     13887     13871   98.3385        96     1.14796     0.65044
  565      16     13910     13894   98.3273        92    0.746258    0.650412
  566      16     13935     13919   98.3301       100    0.492521    0.650374
  567      16     13960     13944    98.333       100      0.3715    0.650398
  568      16     13985     13969   98.3359       100    0.579969    0.650587
  569      16     14007     13991   98.3177        88    0.555807     0.65055
  570      16     14028     14012   98.2925        84    0.623199    0.650649
  571      16     14048     14032   98.2604        80    0.442698    0.650731
  572      16     14073     14057   98.2634       100     1.16068    0.650952
  573      16     14097     14081   98.2594        96    0.760908    0.650853
  574      16     14125     14109   98.2833       112    0.137254    0.650796
  575      16     14147     14131   98.2654        88    0.781419     0.65081
  576      16     14175     14159   98.2892       112    0.420487     0.65086
  577      16     14203     14187   98.3129       112    0.437793    0.650675
  578      16     14224     14208   98.2881        84    0.411578    0.650664
  579      16     14251     14235   98.3049       108    0.244459    0.650607
2020-03-08 03:50:16.287823 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.65054
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     14279     14263   98.3284       112    0.574155     0.65054
  581      16     14303     14287   98.3244        96    0.398587    0.650546
  582      16     14331     14315   98.3479       112    0.629273    0.650455
  583      16     14351     14335   98.3164        80     1.02209    0.650517
  584      16     14379     14363   98.3398       112    0.465879     0.65041
  585      16     14406     14390   98.3562       108     0.91134    0.650376
  586      16     14432     14416   98.3658       104    0.603578    0.650222
  587      16     14456     14440   98.3618        96    0.433777    0.650184
  588      16     14478     14462   98.3441        88    0.243715    0.650223
  589      16     14507     14491    98.374       116    0.150017    0.650202
  590      16     14528     14512   98.3497        84      1.1645    0.650245
  591      16     14553     14537   98.3523       100     0.68081    0.650382
  592      16     14572     14556   98.3145        76    0.582767    0.650425
  593      16     14600     14584   98.3375       112    0.284335    0.650249
  594      16     14629     14613   98.3672       116    0.521806    0.650191
  595      16     14655     14639   98.3766       104    0.780194    0.650188
  596      16     14678     14662   98.3658        92    0.642921    0.650252
  597      16     14699     14683   98.3417        84    0.496099    0.650398
  598      16     14717     14701   98.2976        72    0.749483    0.650579
  599      16     14743     14727   98.3071       104    0.748699    0.650638
2020-03-08 03:50:36.295938 min lat: 0.0680884 max lat: 1.81909 avg lat: 0.650637
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     14761     14745   98.2629        72     0.55221    0.650637
Total time run:         600.488
Total writes made:      14761
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     98.3266
Stddev Bandwidth:       14.0561
Max bandwidth (MB/sec): 144
Min bandwidth (MB/sec): 64
Average IOPS:           24
Stddev IOPS:            3.51401
Max IOPS:               36
Min IOPS:               16
Average Latency(s):     0.65082
Stddev Latency(s):      0.244896
Max latency(s):         1.81909
Min latency(s):         0.0680884

Total time run:         600.488
Total writes made:      14761
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     98.3266
Stddev Bandwidth:       14.0561
Max bandwidth (MB/sec): 144
Min bandwidth (MB/sec): 64
Average IOPS:           24
Stddev IOPS:            3.51401
Max IOPS:               36
Min IOPS:               16
Average Latency(s):     0.65082
Stddev Latency(s):      0.244896
Max latency(s):         1.81909
Min latency(s):         0.0680884

Sequential Read Test, pool with 8 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        50        34   135.929       136    0.975141    0.284386
    2      16        81        65   129.927       124   0.0218552    0.390906
    3      15       110        95   126.553       120    0.835151    0.445451
    4      16       144       128   127.892       132    0.876989    0.450691
    5      16       182       166   132.628       152    0.777139    0.454517
    6      16       216       200   133.185       136    0.830861     0.44884
    7      16       248       232   132.443       128    0.963055    0.449191
    8      16       286       270   134.883       152    0.829292    0.448969
    9      16       319       303   134.558       132    0.139848    0.451466
   10      16       353       337    134.65       136    0.891417    0.453271
   11      16       386       370   134.407       132     0.17349    0.454992
   12      16       414       398    132.54       112    0.199415    0.461636
   13      16       456       440   135.262       168   0.0210695    0.456778
   14      16       492       476   135.879       144   0.0634224     0.45462
   15      16       525       509   135.612       132    0.147348    0.456454
   16      16       559       543   135.632       136    0.845269    0.456958
   17      16       592       576   135.412       132     0.65893    0.460863
   18      16       625       609   135.216       132    0.828546    0.460951
   19      16       659       643   135.252       136    0.711763    0.462084
2020-03-08 03:56:54.842799 min lat: 0.0203516 max lat: 1.07755 avg lat: 0.46355
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       690       674   134.686       124    0.846478     0.46355
   21      16       723       707   134.551       132    0.255897    0.463483
   22      16       756       740   134.434       132    0.758799     0.46475
   23      15       786       771   133.975       124    0.915678    0.467316
   24      16       818       802   133.559       124    0.899369    0.468738
   25      16       850       834   133.333       128    0.160319    0.468427
   26      16       881       865    132.97       124    0.765729     0.47168
   27      16       914       898    132.93       132    0.797315    0.471295
   28      16       951       935   133.464       148    0.637504    0.471333
   29      16       982       966   133.137       124    0.769804    0.471666
   30      15      1020      1005   133.892       156    0.126077     0.46897
   31      16      1055      1039   133.959       136    0.148952     0.46799
   32      16      1088      1072   133.897       132    0.787927    0.469686
   33      16      1121      1105   133.839       132    0.965245    0.469038
   34      16      1152      1136   133.547       124    0.908163    0.470386
   35      16      1185      1169     133.5       132    0.688508    0.471094
   36      16      1215      1199   133.124       120     1.03773    0.471998
   37      16      1249      1233   133.199       136    0.867507    0.472192
   38      16      1287      1271   133.691       152   0.0462029    0.470585
   39      16      1323      1307   133.952       144     0.66804    0.470431
2020-03-08 03:57:14.855548 min lat: 0.0203516 max lat: 1.16678 avg lat: 0.470094
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1358      1342   134.101       140    0.684519    0.470094
   41      15      1393      1378   134.331       144    0.191476    0.469144
   42      16      1426      1410    134.18       128    0.841362    0.469472
   43      16      1462      1446   134.408       144    0.161133    0.468827
   44      15      1495      1480   134.438       136   0.0507931    0.468935
   45      16      1529      1513   134.383       132     0.72111    0.469962
   46      16      1566      1550   134.677       148    0.901066    0.467784
   47      15      1600      1585   134.781       140    0.213974    0.468174
   48      16      1635      1619   134.805       136   0.0225542    0.467605
   49      16      1669      1653   134.827       136     0.23991    0.467815
   50      16      1705      1689   135.008       144    0.128391    0.467435
   51      16      1736      1720    134.79       124    0.068189    0.468261
   52      16      1768      1752   134.658       128    0.695251    0.468518
   53      16      1803      1787   134.757       140    0.838286    0.468082
   54      16      1840      1824       135       148     0.82736    0.468017
   55      16      1873      1857   134.944       132    0.188068    0.467539
   56      16      1910      1894   135.175       148     1.03513    0.467152
   57      16      1940      1924   134.908       120   0.0673579    0.467887
   58      16      1974      1958   134.924       136     0.12003    0.467981
   59      16      2007      1991   134.874       132    0.862308    0.468366
2020-03-08 03:57:34.874359 min lat: 0.0203516 max lat: 1.16678 avg lat: 0.466844
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2045      2029   135.158       152   0.0689746    0.466844
   61      16      2077      2061   135.038       128    0.804247    0.467989
   62      16      2110      2094   134.989       132    0.189748     0.46756
   63      16      2143      2127   134.936       132   0.0489882    0.468168
   64      16      2178      2162    135.01       140    0.155104    0.467166
   65      16      2213      2197   135.086       140     0.20208    0.468235
   66      16      2251      2235    135.34       152    0.939328    0.466502
   67      16      2283      2267   135.231       128    0.929973    0.466944
   68      16      2315      2299   135.124       128    0.868231     0.46777
   69      16      2350      2334   135.194       140     1.03562    0.467395
   70      16      2380      2364   134.977       120    0.850697    0.468152
   71      16      2418      2402   135.217       152    0.180088    0.467554
   72      16      2452      2436   135.227       136   0.0480928    0.467477
   73      16      2486      2470   135.237       136    0.997825    0.467046
   74      16      2518      2502   135.138       128    0.070269    0.467827
   75      16      2551      2535   135.096       132    0.152167    0.467803
   76      16      2583      2567   135.001       128       0.794    0.468238
   77      15      2615      2600    134.96       132    0.719849    0.469051
   78      16      2647      2631   134.818       124     0.71598    0.469157
   79      16      2681      2665   134.833       136   0.0365208    0.469306
2020-03-08 03:57:54.886943 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.469596
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2712      2696   134.697       124    0.837074    0.469596
   81      16      2743      2727   134.565       124    0.845693     0.47011
   82      16      2777      2761   134.581       136    0.118733    0.470016
   83      16      2806      2790   134.356       116    0.756143    0.470865
   84      15      2842      2827   134.509       148    0.870993    0.470244
   85      16      2873      2857   134.338       120    0.774067    0.470871
   86      16      2910      2894   134.496       148   0.0542694    0.470643
   87      16      2942      2926   134.421       128    0.927292    0.470743
   88      16      2974      2958   134.348       128    0.865166    0.470971
   89      16      3012      2996   134.545       152    0.728277    0.470583
   90      16      3043      3027   134.428       124     0.20032    0.470135
   91      16      3078      3062   134.489       140    0.818342    0.470469
   92      16      3109      3093   134.374       124    0.234331    0.470648
   93      16      3144      3128   134.435       140    0.811083    0.470936
   94      16      3179      3163   134.494       140   0.0224531    0.470709
   95      15      3212      3197   134.509       136    0.960089     0.47034
   96      15      3244      3229    134.44       128    0.175016    0.470492
   97      16      3281      3265   134.539       144     1.06565     0.47031
   98      16      3311      3295    134.39       120    0.639112    0.471379
   99      16      3344      3328   134.366       132    0.928796    0.470734
2020-03-08 03:58:14.898922 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471156
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3377      3361   134.342       132     1.00684    0.471156
  101      16      3406      3390    134.16       116    0.798939    0.471895
  102      16      3442      3426   134.256       144    0.801796     0.47159
  103      16      3477      3461   134.312       140    0.736081    0.471432
  104      16      3509      3493   134.251       128    0.707635    0.471639
  105      16      3541      3525   134.191       128    0.783155     0.47186
  106      16      3579      3563   134.343       152   0.0701487    0.471216
  107      16      3614      3598   134.396       140    0.757796    0.471319
  108      16      3647      3631   134.373       132    0.167515    0.471397
  109      16      3679      3663   134.314       128    0.170124    0.471318
  110      16      3711      3695   134.257       128   0.0715369    0.471835
  111      16      3745      3729   134.272       136   0.0415188    0.471837
  112      16      3776      3760    134.18       124    0.148915    0.472048
  113      16      3811      3795   134.231       140    0.752372    0.471844
  114      16      3846      3830    134.28       140    0.959556    0.471629
  115      16      3880      3864   134.295       136   0.0428712    0.471715
  116      16      3915      3899   134.344       140   0.0452116     0.47172
  117      16      3951      3935   134.423       144    0.176952    0.471427
  118      16      3981      3965     134.3       120    0.683724    0.471923
  119      15      4014      3999   134.312       136   0.0220373    0.471609
2020-03-08 03:58:34.922421 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471021
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4056      4040   134.558       164   0.0383263    0.471021
  121      16      4093      4077   134.668       148    0.877943     0.47034
  122      16      4129      4113   134.744       144    0.334867    0.470029
  123      16      4158      4142    134.59       116    0.779281    0.470851
  124      16      4190      4174   134.537       128     1.13204    0.470701
  125      16      4224      4208   134.549       136    0.696269    0.470985
  126      16      4259      4243   134.592       140    0.692259      0.4708
  127      15      4293      4278   134.632       140     0.71671     0.47083
  128      16      4326      4310    134.58       128    0.678277    0.471015
  129      16      4362      4346   134.652       144    0.136745     0.47043
  130      16      4396      4380   134.662       136    0.796352    0.470561
  131      16      4429      4413   134.641       132    0.734982    0.470756
  132      16      4462      4446   134.621       132    0.414072    0.470607
  133      16      4496      4480   134.631       136    0.220275    0.470567
  134      16      4528      4512   134.581       128    0.651016    0.471096
  135      16      4560      4544   134.532       128    0.321922    0.470934
  136      16      4598      4582    134.66       152   0.0383355    0.470739
  137      16      4635      4619   134.757       148    0.050658    0.470188
  138      16      4671      4655    134.82       144      0.8999    0.470147
  139      16      4708      4692   134.915       148    0.678899    0.469833
2020-03-08 03:58:54.936877 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.469671
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4744      4728   134.979       144    0.216934    0.469671
  141      16      4778      4762   134.985       136    0.100927     0.46967
  142      16      4810      4794   134.936       128    0.222963    0.469965
  143      16      4842      4826   134.887       128    0.786333    0.469959
  144      16      4875      4859   134.867       132    0.735611    0.469992
  145      16      4908      4892   134.847       132   0.0210776    0.469935
  146      16      4942      4926   134.855       136    0.131441    0.469846
  147      16      4972      4956   134.754       120    0.959912    0.470109
  148      16      5008      4992   134.816       144    0.155662    0.470122
  149      16      5042      5026   134.824       136     1.01963    0.470261
  150      16      5079      5063   134.911       148   0.0207624    0.469949
  151      16      5111      5095   134.865       128     1.03945     0.47001
  152      16      5145      5129   134.872       136    0.839832    0.470117
  153      16      5174      5158   134.748       116   0.0561273    0.470465
  154      16      5204      5188   134.652       120   0.0266074    0.470978
  155      16      5236      5220   134.605       128    0.825376    0.470939
  156      15      5265      5250    134.51       120    0.622478    0.471615
  157      15      5298      5283   134.491       132    0.697012    0.471603
  158      16      5331      5315    134.45       128     1.04815    0.471255
  159      16      5363      5347   134.409       128     1.04153    0.471713
2020-03-08 03:59:14.952133 min lat: 0.0203516 max lat: 1.22369 avg lat: 0.471873
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5395      5379   134.369       128    0.930668    0.471873
  161      16      5429      5413   134.379       136    0.900192    0.471789
  162      16      5465      5449   134.438       144   0.0210947    0.471793
  163      16      5495      5479   134.349       120    0.787277    0.472051
  164      16      5526      5510   134.283       124    0.924526    0.472028
  165      16      5559      5543   134.269       132   0.0551954    0.472396
  166      16      5599      5583   134.424       160   0.0781094     0.47185
  167      16      5631      5615   134.385       128    0.870821    0.471827
  168      16      5664      5648   134.371       132    0.982932    0.471739
  169      16      5698      5682    134.38       136    0.722235     0.47203
  170      16      5730      5714   134.343       128    0.175746    0.471645
  171      16      5763      5747   134.329       132    0.133635    0.472029
  172      16      5797      5781   134.338       136    0.783323    0.472232
  173      16      5835      5819    134.44       152    0.614553    0.471872
  174      15      5869      5854   134.472       140   0.0213479    0.471601
  175      16      5908      5892   134.572       152     1.05504     0.47132
  176      16      5942      5926   134.574       136   0.0292772    0.471109
  177      16      5977      5961   134.604       140    0.596964    0.471326
  178      16      6010      5994   134.589       132   0.0807799    0.471159
  179      16      6042      6026   134.552       128    0.894737    0.471273
2020-03-08 03:59:34.970362 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.47137
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      6077      6061   134.581       140    0.819267     0.47137
  181      16      6109      6093   134.544       128    0.807277    0.471377
  182      16      6139      6123   134.464       120    0.794841    0.471707
  183      16      6176      6160   134.538       148   0.0670039     0.47139
  184      16      6211      6195   134.568       140    0.676913    0.471449
  185      16      6245      6229   134.575       136    0.862948    0.471514
  186      16      6284      6268   134.689       156    0.187745    0.470781
  187      16      6315      6299   134.632       124    0.929776    0.470944
  188      16      6350      6334   134.659       140    0.739409    0.471048
  189      16      6385      6369   134.687       140    0.786382    0.470928
  190      16      6418      6402   134.673       132    0.883835    0.470936
  191      16      6450      6434   134.638       128    0.246533     0.47107
  192      16      6486      6470   134.684       144   0.0231771    0.471047
  193      16      6530      6514   134.898       176    0.843133    0.470284
  194      16      6565      6549   134.924       140    0.792298    0.470143
  195      16      6598      6582   134.909       132    0.883757    0.470015
  196      16      6634      6618   134.955       144    0.995747    0.469922
  197      16      6668      6652   134.961       136   0.0534438    0.470077
  198      16      6703      6687   134.986       140    0.778019     0.46993
  199      15      6737      6722    135.01       140   0.0486843    0.469695
2020-03-08 03:59:54.981825 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470245
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6768      6752   134.935       120    0.670818    0.470245
  201      16      6798      6782    134.86       120     0.27175    0.470312
  202      16      6832      6816   134.866       136    0.871731    0.470364
  203      16      6867      6851   134.891       140    0.763679    0.470376
  204      16      6897      6881   134.815       120    0.965714    0.470474
  205      16      6932      6916   134.841       140    0.114796    0.470554
  206      16      6971      6955   134.943       156   0.0479505    0.470198
  207      16      7006      6990   134.967       140    0.155196    0.469929
  208      16      7034      7018   134.857       112     0.71625    0.470463
  209      16      7069      7053   134.881       140    0.145525    0.470375
  210      16      7105      7089   134.924       144   0.0696467    0.470195
  211      16      7142      7126   134.985       148    0.842074     0.47004
  212      16      7175      7159   134.971       132   0.0683049    0.469995
  213      16      7205      7189     134.9       120     0.82763    0.470352
  214      16      7242      7226   134.961       148    0.769618     0.47012
  215      16      7277      7261   134.984       140    0.130375    0.470114
  216      16      7305      7289   134.878       112    0.855446    0.470359
  217      16      7340      7324   134.901       140   0.0218787    0.470229
  218      16      7373      7357   134.888       132    0.301954    0.470317
  219      16      7406      7390   134.874       132    0.174339    0.470329
2020-03-08 04:00:14.994503 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470706
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7437      7421   134.824       124    0.721956    0.470706
  221      16      7472      7456   134.847       140    0.215404    0.470533
  222      16      7506      7490   134.851       136    0.478741    0.470636
  223      16      7542      7526   134.882       144   0.0498284     0.47043
  224      16      7575      7559   134.869       132    0.926292    0.470323
  225      16      7607      7591   134.839       128    0.062399    0.470517
  226      16      7640      7624   134.826       132    0.974291    0.470412
  227      16      7675      7659   134.849       140   0.0410486    0.470404
  228      16      7706      7690   134.801       124    0.249063    0.470585
  229      16      7741      7725   134.824       140    0.241634    0.470435
  230      16      7779      7763   134.898       152    0.934999     0.47046
  231      16      7818      7802   134.989       156   0.0222671    0.470106
  232      16      7854      7838   135.028       144    0.720556    0.469978
  233      16      7887      7871   135.015       132     0.77449    0.470023
  234      16      7920      7904   135.001       132    0.787365        0.47
  235      16      7956      7940   135.039       144    0.954715    0.469977
  236      16      7988      7972   135.008       128     1.09556    0.469822
  237      16      8025      8009   135.063       148    0.187537    0.469881
  238      16      8057      8041   135.033       128    0.208595    0.469911
  239      16      8089      8073   135.004       128    0.820872    0.470072
2020-03-08 04:00:35.019331 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.46997
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      8123      8107   135.008       136    0.153667     0.46997
  241      16      8153      8137   134.946       120    0.879839    0.470223
  242      16      8185      8169   134.917       128    0.434752    0.470248
  243      16      8218      8202   134.905       132    0.652999    0.470429
  244      16      8249      8233    134.86       124    0.657258    0.470621
  245      16      8280      8264   134.815       124    0.726724    0.470887
  246      16      8322      8306    134.95       168    0.918289    0.470133
  247      16      8355      8339   134.938       132    0.672391    0.470415
  248      16      8386      8370   134.893       124    0.793471     0.47043
  249      16      8419      8403   134.882       132    0.941931    0.470427
  250      15      8449      8434   134.838       124    0.776849    0.470756
  251      16      8481      8465   134.795       124     1.01873    0.470612
  252      16      8513      8497   134.768       128    0.961651    0.470841
  253      16      8545      8529   134.741       128    0.122275    0.470873
  254      16      8580      8564   134.762       140   0.0233841    0.470849
  255      16      8611      8595   134.719       124    0.743347    0.471203
  256      16      8642      8626   134.677       124    0.726062     0.47128
  257      16      8671      8655   134.605       116    0.856562    0.471398
  258      16      8703      8687   134.579       128    0.156371    0.471497
  259      16      8739      8723   134.615       144    0.765914    0.471528
2020-03-08 04:00:55.026016 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471454
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8773      8757   134.619       136   0.0213606    0.471454
  261      16      8807      8791   134.625       136    0.162843    0.471297
  262      16      8839      8823   134.599       128   0.0211701    0.471523
  263      16      8871      8855   134.574       128    0.174515    0.471689
  264      16      8905      8889   134.579       136    0.734734    0.471681
  265      16      8936      8920   134.539       124    0.746554    0.471834
  266      16      8972      8956   134.575       144    0.844828      0.4716
  267      15      9007      8992   134.608       144    0.784119    0.471513
  268      16      9046      9030   134.672       152   0.0732528    0.471265
  269      16      9077      9061   134.632       124    0.868041    0.471379
  270      16      9111      9095   134.636       136    0.161968    0.471325
  271      16      9144      9128   134.627       132    0.699678    0.471489
  272      16      9176      9160   134.602       128    0.071531    0.471503
  273      16      9209      9193   134.592       132     0.84155    0.471558
  274      16      9246      9230   134.641       148    0.911372    0.471328
  275      16      9283      9267   134.689       148   0.0226142    0.471188
  276      16      9319      9303   134.722       144   0.0575931    0.471141
  277      16      9354      9338   134.741       140    0.131408     0.47107
  278      16      9392      9376   134.803       152     0.89028    0.470875
  279      16      9425      9409   134.793       132    0.265949    0.470783
2020-03-08 04:01:15.039602 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470947
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9458      9442   134.783       132    0.985732    0.470947
  281      16      9487      9471   134.716       116     0.68781    0.471327
  282      16      9522      9506   134.735       140      1.0122    0.471053
  283      16      9558      9542   134.767       144   0.0465632    0.470978
  284      16      9590      9574   134.743       128   0.0428525    0.470901
  285      16      9623      9607   134.733       132    0.136339    0.471115
  286      16      9657      9641   134.737       136    0.959743    0.471061
  287      16      9689      9673   134.714       128     0.88085     0.47123
  288      16      9719      9703   134.662       120    0.113394    0.471322
  289      16      9759      9743    134.75       160   0.0669207    0.471123
  290      16      9790      9774   134.712       124    0.825954    0.471225
  291      16      9821      9805   134.675       124    0.252587    0.471242
  292      16      9857      9841   134.707       144   0.0298389    0.471182
  293      16      9885      9869   134.629       112   0.0525508    0.471451
  294      15      9916      9901   134.606       128   0.0495012    0.471646
  295      16      9949      9933   134.583       128    0.130198    0.471651
  296      16      9980      9964   134.547       124    0.108019    0.471882
  297      16     10011      9995   134.512       124    0.190732    0.471896
  298      16     10041     10025   134.463       120    0.888149    0.471997
  299      16     10078     10062   134.508       148    0.700575     0.47202
2020-03-08 04:01:35.049816 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471865
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     10115     10099   134.553       148    0.781747    0.471865
  301      16     10149     10133   134.558       136    0.786359    0.471858
  302      16     10183     10167   134.562       136    0.714866    0.471858
  303      16     10217     10201   134.567       136    0.352637    0.471757
  304      16     10252     10236   134.585       140    0.812156    0.471677
  305      16     10285     10269   134.576       132   0.0479317    0.471674
  306      16     10319     10303   134.581       136    0.850186    0.471684
  307      16     10350     10334   134.546       124    0.776645    0.471896
  308      16     10381     10365   134.512       124    0.984267    0.471865
  309      16     10415     10399   134.516       136    0.798314    0.471959
  310      16     10442     10426   134.431       108    0.798814    0.472244
  311      16     10473     10457   134.397       124    0.795969    0.472355
  312      15     10508     10493   134.427       144    0.795033    0.472324
  313      16     10541     10525   134.406       128     1.01292    0.472366
  314      16     10580     10564   134.475       156    0.279062    0.472099
  315      16     10617     10601   134.518       148    0.230418    0.471825
  316      16     10651     10635   134.522       136    0.146848    0.471878
  317      16     10686     10670   134.539       140    0.802798    0.471898
  318      16     10720     10704   134.544       136     1.02918    0.471912
  319      16     10755     10739   134.561       140    0.791742    0.471798
2020-03-08 04:01:55.057587 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.471884
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10789     10773   134.565       136   0.0536146    0.471884
  321      16     10822     10806   134.557       132    0.161004    0.471857
  322      16     10851     10835   134.499       116    0.846112    0.471972
  323      15     10888     10873    134.55       152    0.884879    0.471905
  324      16     10922     10906   134.542       132    0.251293    0.471865
  325      16     10955     10939   134.534       132    0.804962    0.471909
  326      16     10993     10977   134.588       152    0.715206    0.471846
  327      16     11025     11009   134.567       128     0.58962    0.471899
  328      15     11059     11044   134.583       140    0.829034    0.471791
  329      16     11093     11077   134.575       132    0.157537      0.4718
  330      16     11129     11113   134.603       144    0.300787    0.471535
  331      16     11159     11143   134.559       120    0.865221    0.471816
  332      16     11194     11178   134.575       140   0.0391026    0.471751
  333      16     11233     11217   134.639       156    0.812567    0.471432
  334      16     11267     11251   134.643       136    0.700221    0.471611
  335      16     11294     11278   134.563       108   0.0464601    0.471761
  336      16     11331     11315   134.603       148    0.677002    0.471794
  337      16     11369     11353   134.655       152     0.14504    0.471403
  338      16     11399     11383   134.611       120    0.877266    0.471665
  339      16     11437     11421   134.662       152    0.026671    0.471438
2020-03-08 04:02:15.074288 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.47131
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     11475     11459   134.713       152   0.0435939     0.47131
  341      16     11505     11489    134.67       120    0.183064    0.471383
  342      16     11536     11520   134.638       124    0.166195    0.471566
  343      16     11569     11553   134.631       132   0.0225562    0.471598
  344      16     11607     11591   134.681       152   0.0411435    0.471535
  345      16     11643     11627   134.708       144    0.831116    0.471329
  346      16     11674     11658   134.677       124   0.0240121    0.471392
  347      15     11706     11691   134.668       132    0.991573    0.471454
  348      16     11744     11728   134.706       148    0.424977    0.471241
  349      16     11778     11762    134.71       136    0.868019    0.471374
  350      16     11815     11799   134.748       148    0.192133    0.471113
  351      16     11848     11832    134.74       132   0.0205455    0.471171
  352      16     11885     11869   134.777       148     0.73958    0.471088
  353      16     11919     11903    134.78       136    0.855659    0.471055
  354      16     11953     11937   134.784       136     1.01404    0.470989
  355      16     11988     11972   134.798       140     0.14677    0.471023
  356      16     12025     12009   134.835       148     0.16013    0.470892
  357      16     12064     12048   134.894       156   0.0759443    0.470737
  358      16     12099     12083   134.909       140    0.789609    0.470635
  359      16     12133     12117   134.911       136    0.145218    0.470623
2020-03-08 04:02:35.083813 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470506
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     12168     12152   134.926       140    0.939901    0.470506
  361      16     12197     12181   134.873       116    0.691135    0.470821
  362      16     12227     12211   134.832       120   0.0236928    0.470827
  363      15     12263     12248   134.866       148   0.0758222    0.470817
  364      16     12301     12285   134.901       148    0.424673    0.470587
  365      16     12332     12316    134.87       124    0.938377    0.470733
  366      16     12373     12357    134.95       164   0.0962637    0.470436
  367      16     12406     12390   134.942       132     0.86539    0.470478
  368      16     12440     12424   134.944       136     1.13545    0.470388
  369      16     12475     12459   134.958       140    0.845053    0.470473
  370      15     12509     12494   134.965       140    0.888389    0.470394
  371      16     12538     12522   134.903       112     0.96583    0.470503
  372      16     12573     12557   134.916       140    0.994498    0.470652
  373      16     12606     12590   134.908       132    0.334376    0.470671
  374      16     12640     12624    134.91       136    0.230143    0.470627
  375      16     12674     12658   134.913       136    0.718391    0.470628
  376      16     12711     12695   134.948       148     1.00987     0.47044
  377      16     12745     12729    134.95       136   0.0438632    0.470521
  378      16     12776     12760   134.921       124    0.638258     0.47067
  379      16     12815     12799   134.976       156    0.980965    0.470386
2020-03-08 04:02:55.122400 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.470335
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12851     12835       135       144   0.0365161    0.470335
  381      16     12883     12867   134.981       128    0.687467     0.47044
  382      15     12914     12899   134.962       128    0.148723    0.470393
  383      16     12947     12931   134.943       128    0.147092    0.470552
  384      16     12974     12958   134.872       108    0.195706    0.470709
  385      16     13010     12994   134.896       144   0.0379402    0.470712
  386      16     13042     13026   134.878       128     0.85072    0.470721
  387      16     13075     13059    134.87       132    0.780163    0.470841
  388      16     13110     13094   134.883       140    0.143982      0.4708
  389      16     13145     13129   134.896       140    0.792834    0.470797
  390      16     13177     13161   134.879       128       0.668    0.470847
  391      16     13212     13196   134.892       140    0.770741    0.470756
  392      16     13246     13230   134.894       136   0.0461209    0.470636
  393      16     13278     13262   134.877       128    0.689751    0.470764
  394      16     13312     13296   134.879       136    0.592166    0.470869
  395      16     13345     13329   134.872       132    0.768612     0.47079
  396      16     13381     13365   134.895       144    0.886712    0.470734
  397      16     13410     13394   134.847       116    0.316061     0.47087
  398      16     13440     13424   134.809       120    0.954841    0.471008
  399      16     13474     13458   134.812       136    0.889035    0.471037
2020-03-08 04:03:15.138738 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.4713
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     13502     13486   134.755       112    0.694989      0.4713
  401      16     13537     13521   134.767       140    0.685174    0.471272
  402      16     13571     13555    134.77       136    0.814546    0.471202
  403      16     13606     13590   134.783       140    0.178205    0.471167
  404      16     13636     13620   134.745       120    0.910459    0.471202
  405      16     13668     13652   134.728       128    0.807004    0.471354
  406      16     13699     13683   134.702       124    0.163664    0.471398
  407      16     13735     13719   134.724       144     1.08022    0.471238
  408      16     13767     13751   134.707       128    0.132035    0.471331
  409      16     13798     13782   134.681       124    0.807371    0.471481
  410      16     13829     13813   134.655       124   0.0629575    0.471532
  411      16     13864     13848   134.668       140   0.0542624     0.47147
  412      15     13897     13882   134.667       136    0.954487    0.471456
  413      15     13938     13923   134.738       164    0.744029    0.471368
  414      15     13971     13956   134.731       132   0.0819889    0.471289
  415      16     14006     13990   134.734       136    0.750822    0.471307
  416      15     14038     14023   134.727       132    0.160715    0.471283
  417      16     14070     14054   134.701       124      0.1169    0.471388
  418      16     14101     14085   134.675       124   0.0210572    0.471431
  419      16     14138     14122   134.707       148    0.677568    0.471434
2020-03-08 04:03:35.166219 min lat: 0.0203516 max lat: 1.29304 avg lat: 0.4712
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     14174     14158   134.729       144     1.14233      0.4712
  421      16     14215     14199   134.798       164    0.790375    0.471117
  422      16     14250     14234    134.81       140   0.0226378    0.471038
  423      16     14281     14265   134.784       124    0.794616     0.47116
  424      16     14312     14296   134.758       124     1.08469     0.47117
  425      16     14344     14328   134.742       128    0.681334    0.471321
  426      16     14380     14364   134.764       144    0.949349    0.471186
  427      16     14415     14399   134.776       140    0.817058    0.471177
  428      16     14444     14428   134.732       116    0.785967    0.471275
  429      16     14477     14461   134.726       132     0.78142    0.471356
  430      16     14517     14501   134.785       160   0.0237873    0.471104
  431      15     14550     14535   134.787       136     1.05292    0.471088
  432      16     14588     14572   134.817       148     1.08526    0.470971
  433      16     14626     14610   134.857       152    0.771281    0.470927
  434      16     14657     14641   134.832       124      0.7577    0.471031
  435      16     14689     14673   134.816       128    0.779094    0.471056
  436      16     14716     14700   134.755       108    0.131726    0.471154
  437      16     14752     14736   134.776       144    0.293693    0.471108
Total time run:       438.252
Total reads made:     14761
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   134.726
Average IOPS:         33
Stddev IOPS:          2.73859
Max IOPS:             44
Min IOPS:             27
Average Latency(s):   0.471676
Max latency(s):       1.55417
Min latency(s):       0.0203516

Total time run:       438.252
Total reads made:     14761
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   134.726
Average IOPS:         33
Stddev IOPS:          2.73859
Max IOPS:             44
Min IOPS:             27
Average Latency(s):   0.471676
Max latency(s):       1.55417
Min latency(s):       0.0203516

Random Read Test, pool with 8 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      15        52        37   147.942       148    0.158301    0.301189
    2      16        87        71   141.952       136    0.832574    0.362475
    3      16       118       102   135.759       124    0.654045    0.420607
    4      16       152       136   135.704       136    0.204245    0.428888
    5      16       186       170    135.74       136    0.729829    0.437042
    6      16       218       202   134.436       128     1.05547    0.438213
    7      16       255       239   136.361       148   0.0709774      0.4441
    8      16       286       270   134.811       124    0.869281    0.448637
    9      16       316       300   133.164       120    0.685196    0.457519
   10      16       351       335   133.838       140     0.62829    0.459062
   11      16       383       367   133.302       128    0.982444    0.459559
   12      15       416       401   133.524       136    0.128743    0.460705
   13      16       452       436   134.015       140    0.610945    0.462623
   14      16       482       466   133.009       120    0.911242    0.463799
   15      16       515       499   132.904       132     0.31573    0.465893
   16      16       556       540   134.785       164    0.853867    0.459184
   17      16       589       573    134.62       132    0.689177     0.46168
   18      16       627       611   135.583       152   0.0113234    0.458222
   19      16       659       643   135.183       128    0.799105     0.46169
2020-03-08 04:06:56.313003 min lat: 0.0113234 max lat: 1.09869 avg lat: 0.458643
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       694       678   135.422       140    0.310124    0.458643
   21      16       727       711   135.258       132    0.650914    0.463073
   22      16       757       741   134.565       120    0.885841    0.465206
   23      16       792       776   134.799       140    0.786337    0.462671
   24      16       827       811   135.015       140    0.936009    0.462674
   25      16       863       847   135.373       144    0.842838    0.462295
   26      16       896       880   135.238       132     0.19784    0.462491
   27      15       929       914   135.264       136    0.116197    0.462619
   28      15       961       946   134.998       128   0.0664747    0.464232
   29      16       994       978   134.756       128    0.740329     0.46574
   30      16      1028      1012   134.797       136    0.805194    0.465616
   31      16      1061      1045   134.706       132    0.195299    0.465224
   32      16      1093      1077   134.496       128    0.633873    0.467363
   33      16      1128      1112   134.662       140    0.855817    0.466699
   34      15      1158      1143   134.348       124    0.868035    0.467697
   35      16      1192      1176    134.28       132   0.0482176    0.468328
   36      16      1226      1210   134.328       136    0.933862    0.467658
   37      16      1258      1242   134.156       128    0.967674    0.467292
   38      16      1292      1276   134.204       136   0.0855933    0.468221
   39      16      1325      1309   134.144       132    0.130788    0.469733
2020-03-08 04:07:16.320111 min lat: 0.00938755 max lat: 1.19962 avg lat: 0.469549
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1356      1340   133.888       124     1.04387    0.469549
   41      16      1390      1374   133.936       136     1.03238    0.469286
   42      16      1420      1404   133.603       120    0.184339    0.471017
   43      16      1456      1440   133.842       144    0.788066    0.470925
   44      16      1492      1476   134.069       144    0.987314    0.469536
   45      16      1521      1505   133.665       116    0.178718    0.471201
   46      16      1555      1539   133.715       136    0.705637    0.471781
   47      16      1586      1570   133.508       124    0.212288    0.471136
   48      16      1617      1601   133.308       124    0.841423    0.472915
   49      16      1651      1635   133.361       136    0.769515    0.473012
   50      16      1683      1667   133.253       128    0.905305    0.472691
   51      16      1716      1700   133.228       132    0.881412    0.473543
   52      16      1755      1739   133.665       156    0.150636    0.471752
   53      16      1786      1770   133.481       124    0.955843    0.472042
   54      16      1823      1807   133.748       148    0.799171    0.472218
   55      16      1856      1840   133.716       132    0.668243    0.472101
   56      16      1890      1874   133.756       136    0.196989    0.472061
   57      16      1924      1908   133.795       136    0.820321    0.472028
   58      16      1953      1937   133.488       116    0.234328    0.472514
   59      15      1987      1972   133.597       140    0.793436    0.472705
2020-03-08 04:07:36.330639 min lat: 0.00938755 max lat: 1.19962 avg lat: 0.472024
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      2024      2008   133.769       144   0.0448707    0.472024
   61      16      2061      2045   134.002       148     0.72615    0.471522
   62      16      2093      2077   133.903       128    0.046765    0.471707
   63      16      2125      2109   133.808       128    0.707081    0.472379
   64      16      2155      2139   133.592       120    0.724463    0.472926
   65      16      2188      2172   133.567       132    0.892171    0.472853
   66      16      2219      2203   133.422       124    0.160451    0.473304
   67      15      2247      2232   133.158       116    0.836406    0.474656
   68      16      2279      2263   133.023       124    0.136684    0.474207
   69      15      2309      2294    132.89       124    0.160315    0.475248
   70      16      2342      2326   132.818       128   0.0546066    0.475489
   71      16      2373      2357   132.693       124    0.738566    0.476682
   72      16      2407      2391   132.738       136   0.0981024    0.476519
   73      16      2438      2422   132.618       124    0.763401    0.477024
   74      16      2470      2454   132.554       128    0.155362    0.476518
   75      16      2505      2489   132.652       140    0.252369    0.476325
   76      16      2539      2523   132.695       136     0.19797    0.476156
   77      16      2579      2563   133.048       160    0.927667    0.475364
   78      16      2610      2594   132.932       124    0.165136    0.475321
   79      16      2646      2630   133.071       144   0.0831507    0.475251
2020-03-08 04:07:56.343072 min lat: 0.00837324 max lat: 1.241 avg lat: 0.474378
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2681      2665   133.156       140   0.0659497    0.474378
   81      16      2714      2698    133.14       132     1.10988    0.474809
   82      16      2747      2731   133.125       132    0.621805    0.475296
   83      16      2779      2763   133.063       128   0.0752005    0.475567
   84      16      2813      2797   133.097       136   0.0468582      0.4753
   85      15      2848      2833   133.225       144    0.126747    0.475067
   86      15      2883      2868   133.304       140    0.923802    0.474424
   87      16      2917      2901   133.288       132    0.923176    0.474476
   88      16      2948      2932   133.183       124     0.70343    0.475175
   89      16      2980      2964   133.124       128    0.904401    0.475157
   90      16      3013      2997   133.111       132    0.759653    0.475733
   91      16      3041      3025   132.879       112     0.15295     0.47627
   92      16      3073      3057   132.826       128     0.57827    0.476727
   93      16      3107      3091    132.86       136   0.0740786    0.476464
   94      16      3143      3127   132.978       144    0.150103    0.475982
   95      16      3177      3161   133.009       136    0.263318    0.475629
   96      16      3213      3197   133.123       144     0.15989    0.475244
   97      16      3250      3234   133.276       148   0.0543193    0.474866
   98      16      3287      3271   133.426       148    0.858228    0.474385
   99      16      3322      3306   133.493       140   0.0739793    0.474492
2020-03-08 04:08:16.348618 min lat: 0.00836277 max lat: 1.241 avg lat: 0.47448
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3353      3337   133.397       124    0.864282     0.47448
  101      16      3384      3368   133.304       124    0.799985    0.474833
  102      16      3420      3404   133.409       144    0.722673    0.474931
  103      16      3452      3436   133.356       128    0.916916    0.474953
  104      16      3484      3468   133.305       128    0.154439    0.475176
  105      16      3516      3500   133.254       128    0.908337    0.475083
  106      16      3553      3537   133.393       148    0.908646    0.474628
  107      15      3584      3569   133.339       128    0.976593    0.474769
  108      15      3618      3603   133.364       136   0.0672866    0.474941
  109      16      3654      3638   133.424       140    0.946736    0.474684
  110      16      3686      3670   133.375       128    0.161426    0.474637
  111      16      3716      3700   133.254       120    0.689853    0.475435
  112      16      3748      3732   133.207       128    0.859965    0.475279
  113      16      3780      3764   133.161       128    0.710995    0.475825
  114      16      3813      3797   133.151       132    0.829756    0.475785
  115      16      3849      3833   133.245       144    0.062841    0.475646
  116      16      3888      3872   133.441       156    0.135025    0.474524
  117      16      3920      3904   133.394       128    0.151655    0.474847
  118      16      3953      3937   133.382       132    0.860326    0.474784
  119      16      3987      3971   133.404       136     1.11299    0.474391
2020-03-08 04:08:36.354024 min lat: 0.00836277 max lat: 1.241 avg lat: 0.475038
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      4019      4003   133.359       128   0.0520461    0.475038
  121      16      4052      4036   133.347       132    0.901977    0.475312
  122      16      4086      4070   133.369       136    0.204373    0.475127
  123      16      4119      4103   133.357       132    0.618922    0.475332
  124      16      4157      4141   133.507       152    0.184775    0.475108
  125      16      4187      4171   133.399       120    0.126014    0.474958
  126      16      4224      4208   133.515       148    0.837533    0.474626
  127      15      4260      4245   133.629       148     0.94655    0.474256
  128      16      4298      4282    133.74       148    0.783716    0.473807
  129      16      4333      4317   133.788       140    0.786668    0.473581
  130      16      4364      4348   133.713       124    0.860266    0.473926
  131      16      4397      4381     133.7       132    0.794391    0.474063
  132      16      4430      4414   133.686       132    0.897462    0.473995
  133      16      4464      4448   133.704       136    0.321504     0.47396
  134      16      4495      4479   133.631       124    0.816939    0.474315
  135      15      4529      4514   133.677       140    0.112762    0.474173
  136      16      4562      4546   133.635       128    0.773603    0.474196
  137      16      4599      4583    133.74       148    0.175125    0.473752
  138      16      4635      4619   133.814       144    0.870978    0.473602
  139      16      4668      4652     133.8       132    0.960549    0.473509
2020-03-08 04:08:56.361061 min lat: 0.00836277 max lat: 1.241 avg lat: 0.473051
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4703      4687   133.843       140    0.204832    0.473051
  141      16      4730      4714    133.66       108    0.725876    0.474213
  142      16      4766      4750   133.732       144     0.87269    0.473984
  143      15      4798      4783    133.72       132    0.778536    0.474156
  144      16      4829      4813   133.625       120      0.8839    0.474476
  145      16      4862      4846   133.613       132   0.0780186    0.474625
  146      16      4890      4874   133.465       112    0.725218    0.475027
  147      15      4924      4909   133.509       140    0.167976     0.47468
  148      16      4962      4946   133.607       148    0.738299    0.474658
  149      16      4998      4982   133.677       144    0.740786    0.474217
  150      16      5029      5013   133.612       124    0.700061    0.474532
  151      16      5067      5051   133.734       152    0.295676    0.474024
  152      16      5097      5081   133.643       120    0.956228    0.474259
  153      16      5129      5113   133.606       128     0.95389    0.474352
  154      16      5166      5150   133.699       148    0.729123    0.474306
  155      16      5199      5183   133.688       132    0.761308    0.474257
  156      15      5235      5220    133.78       148    0.828421     0.47418
  157      16      5269      5253   133.767       132    0.225284    0.473826
  158      16      5300      5284   133.705       124    0.646216    0.474161
  159      16      5334      5318   133.719       136    0.714883    0.474307
2020-03-08 04:09:16.366578 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.474035
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5367      5351   133.708       132    0.859074    0.474035
  161      16      5401      5385   133.722       136    0.823972    0.474187
  162      16      5437      5421   133.786       144    0.165225    0.473834
  163      16      5469      5453    133.75       128     1.04529    0.473949
  164      16      5504      5488   133.788       140    0.272428    0.473882
  165      16      5536      5520   133.753       128    0.221268    0.473974
  166      16      5573      5557   133.838       148    0.709128    0.473896
  167      16      5608      5592   133.875       140     0.85748    0.473673
  168      16      5635      5619   133.721       108    0.165576    0.474198
  169      15      5668      5653   133.734       136   0.0684903     0.47417
  170      16      5702      5686   133.724       132    0.778847     0.47429
  171      16      5732      5716   133.644       120     0.84184    0.474552
  172      16      5764      5748   133.611       128    0.839272    0.474562
  173      16      5797      5781   133.601       132    0.171637    0.474537
  174      15      5835      5820    133.73       156    0.702629    0.474361
  175      16      5872      5856   133.788       144    0.615781    0.474126
  176      15      5904      5889   133.778       132    0.732444    0.474273
  177      16      5937      5921   133.745       128    0.201052    0.474233
  178      15      5975      5960    133.87       156    0.909111     0.47382
  179      16      6008      5992   133.837       128    0.920218    0.473774
2020-03-08 04:09:36.370434 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473779
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      6042      6026   133.849       136    0.805651    0.473779
  181      16      6072      6056   133.772       120    0.686971    0.474125
  182      16      6109      6093    133.85       148    0.727594    0.473911
  183      16      6138      6122   133.753       116    0.669657     0.47424
  184      16      6172      6156   133.765       136    0.206898    0.473988
  185      16      6209      6193   133.841       148    0.558577     0.47404
  186      16      6244      6228   133.873       140    0.131761    0.473847
  187      16      6279      6263   133.906       140    0.709064    0.473788
  188      16      6309      6293   133.831       120    0.164817    0.473846
  189      16      6341      6325     133.8       128    0.783304    0.474048
  190      16      6376      6360   133.832       140    0.851214    0.473917
  191      16      6407      6391    133.78       124   0.0772549    0.474167
  192      16      6447      6431   133.916       160    0.775545    0.473739
  193      16      6479      6463   133.885       128     0.82808    0.473782
  194      16      6510      6494   133.834       124    0.810658    0.474026
  195      16      6546      6530   133.886       144    0.653968    0.473844
  196      16      6579      6563   133.876       132    0.829998    0.473774
  197      16      6611      6595   133.846       128    0.218761    0.473871
  198      16      6644      6628   133.837       132    0.887982    0.473824
  199      16      6676      6660   133.807       128    0.803168    0.474039
2020-03-08 04:09:56.379317 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473846
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6714      6698   133.898       152    0.878244    0.473846
  201      16      6746      6730   133.869       128    0.768501    0.473831
  202      16      6778      6762   133.839       128    0.261127     0.47385
  203      16      6818      6802   133.968       160    0.825998    0.473531
  204      16      6851      6835   133.958       132    0.819022    0.473366
  205      16      6884      6868   133.949       132   0.0788361    0.473514
  206      16      6918      6902   133.959       136    0.815359    0.473553
  207      16      6946      6930   133.853       112    0.840559    0.473911
  208      16      6977      6961   133.805       124    0.590258    0.474219
  209      16      7012      6996   133.815       140    0.201774    0.474042
  210      16      7044      7028   133.787       128    0.642904    0.474265
  211      16      7081      7065   133.854       148    0.774118    0.473984
  212      16      7117      7101   133.902       144    0.123638    0.473725
  213      16      7157      7141   134.024       160    0.518049    0.473464
  214      16      7191      7175   134.033       136    0.744141    0.473359
  215      16      7227      7211   134.079       144    0.239106    0.473191
  216      16      7264      7248   134.143       148   0.0833049    0.472876
  217      16      7299      7283   134.169       140   0.0610498    0.472635
  218      15      7328      7313   134.104       120    0.842044    0.473032
  219      16      7364      7348   134.131       140    0.771269    0.472945
2020-03-08 04:10:16.416138 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.472968
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7397      7381   134.121       132   0.0725063    0.472968
  221      16      7430      7414   134.111       132    0.720154    0.473083
  222      16      7460      7444   134.047       120    0.730235     0.47333
  223      16      7495      7479   134.074       140   0.0526659     0.47332
  224      16      7527      7511   134.046       128    0.283585    0.473148
  225      16      7560      7544   134.036       132    0.673437    0.473328
  226      16      7597      7581   134.098       148    0.148483    0.473082
  227      16      7629      7613    134.07       128    0.800568    0.473115
  228      16      7666      7650   134.131       148    0.824191    0.473086
  229      15      7698      7683   134.121       132    0.860073    0.473035
  230      16      7731      7715   134.095       128    0.991363    0.473112
  231      16      7765      7749   134.103       136    0.817214    0.473088
  232      16      7798      7782   134.094       132    0.153165    0.473078
  233      16      7828      7812   134.033       120    0.288812    0.473226
  234      16      7862      7846   134.041       136    0.193275    0.473232
  235      16      7895      7879   134.032       132     1.01324    0.473253
  236      16      7927      7911   134.007       128   0.0495286    0.473434
  237      16      7963      7947   134.049       144   0.0794484     0.47327
  238      16      7996      7980    134.04       132    0.700819    0.473372
  239      16      8025      8009   133.964       116    0.908209    0.473622
2020-03-08 04:10:36.424949 min lat: 0.00836277 max lat: 1.28557 avg lat: 0.473488
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      8061      8045   134.006       144     0.68111    0.473488
  241      15      8096      8081   134.047       144    0.762315    0.473437
  242      16      8129      8113   134.022       128    0.832327    0.473471
  243      15      8158      8143   133.964       120    0.045964    0.473642
  244      16      8194      8178   133.988       140   0.0976633    0.473517
  245      16      8225      8209   133.947       124    0.684263    0.473676
  246      16      8262      8246   134.004       148    0.721014    0.473495
  247      16      8299      8283   134.061       148    0.819546    0.473259
  248      16      8332      8316   134.052       132     0.79432    0.473305
  249      15      8366      8351   134.075       140    0.048162    0.473156
  250      16      8404      8388   134.131       148    0.084001    0.473187
  251      16      8434      8418   134.075       120    0.720086    0.473294
  252      16      8469      8453   134.098       140    0.654103    0.473274
  253      16      8503      8487   134.105       136    0.672926    0.473223
  254      16      8539      8523   134.144       144     0.14165    0.473034
  255      16      8568      8552   134.073       116    0.921784    0.473101
  256      16      8600      8584   134.049       128     1.11883    0.473154
  257      16      8628      8612   133.964       112     1.13915     0.47337
  258      16      8660      8644    133.94       128    0.935785    0.473647
  259      16      8692      8676   133.917       128    0.782985    0.473764
2020-03-08 04:10:56.431865 min lat: 0.00821512 max lat: 1.28557 avg lat: 0.473652
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8728      8712   133.956       144    0.886369    0.473652
  261      16      8762      8746   133.964       136    0.699452    0.473737
  262      16      8794      8778   133.941       128    0.689826    0.473825
  263      16      8823      8807   133.873       116    0.172319    0.473919
  264      16      8858      8842   133.896       140   0.0504963     0.47379
  265      16      8890      8874   133.873       128     0.85094    0.473976
  266      16      8921      8905   133.836       124    0.711171    0.474214
  267      15      8951      8936   133.799       124    0.847536    0.474271
  268      16      8983      8967   133.763       124    0.798836    0.474429
  269      16      9016      9000   133.755       132    0.931471    0.474381
  270      16      9048      9032   133.734       128   0.0520949    0.474247
  271      16      9083      9067   133.757       140    0.799474    0.474449
  272      16      9116      9100    133.75       132    0.194583     0.47442
  273      16      9144      9128    133.67       112    0.967392    0.474648
  274      16      9177      9161   133.664       132   0.0933709     0.47475
  275      16      9211      9195   133.672       136   0.0567927    0.474748
  276      16      9244      9228   133.666       132    0.698015    0.474777
  277      16      9278      9262   133.674       136   0.0549317    0.474761
  278      16      9313      9297   133.697       140    0.687856    0.474667
  279      16      9348      9332   133.719       140    0.873569    0.474444
2020-03-08 04:11:16.440225 min lat: 0.00821512 max lat: 1.28557 avg lat: 0.47439
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9378      9362    133.67       120   0.0656221     0.47439
  281      16      9411      9395   133.663       132    0.238052    0.474677
  282      16      9445      9429   133.671       136    0.791446    0.474787
  283      15      9474      9459   133.623       120     0.13692    0.474976
  284      15      9507      9492   133.617       132  0.00851741    0.474967
  285      16      9542      9526   133.625       136    0.639127    0.474988
  286      15      9578      9563   133.675       148    0.885825    0.474767
  287      16      9613      9597   133.683       136    0.792025    0.474791
  288      16      9652      9636    133.76       156   0.0746526    0.474401
  289      16      9685      9669   133.754       132    0.855458    0.474457
  290      16      9720      9704   133.775       140    0.961866    0.474431
  291      16      9752      9736   133.756       128    0.125018    0.474386
  292      16      9782      9766   133.708       120    0.673914    0.474715
  293      16      9815      9799   133.703       132    0.773804    0.474667
  294      16      9848      9832   133.696       132    0.128949    0.474675
  295      15      9886      9871   133.772       156   0.0617269    0.474499
  296      15      9915      9900   133.711       116    0.741222    0.474653
  297      16      9952      9936   133.746       144    0.236945    0.474417
  298      16      9988      9972    133.78       144   0.0498526     0.47441
  299      16     10025     10009   133.827       148    0.182187    0.474246
2020-03-08 04:11:36.448793 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474384
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     10054     10038   133.768       116   0.0567709    0.474384
  301      16     10091     10075   133.815       148    0.318893    0.474225
  302      16     10122     10106   133.782       124     0.82665    0.474335
  303      16     10151     10135   133.724       116    0.575964    0.474601
  304      15     10184     10169   133.731       136    0.529074    0.474756
  305      16     10219     10203   133.738       136    0.813308    0.474582
  306      15     10250     10235   133.719       128    0.255949    0.474599
  307      16     10293     10277   133.831       168    0.125106    0.474227
  308      15     10331     10316   133.902       156     1.02491    0.473992
  309      16     10362     10346   133.857       120    0.950659    0.474122
  310      16     10395     10379    133.85       132   0.0741445    0.474115
  311      16     10428     10412   133.844       132    0.918745    0.474103
  312      16     10460     10444   133.825       128      0.6814    0.474257
  313      16     10497     10481    133.87       148   0.0990792    0.474041
  314      16     10533     10517   133.902       144   0.0709087    0.473945
  315      16     10567     10551   133.908       136    0.768953    0.473895
  316      16     10599     10583    133.89       128      0.9036     0.47405
  317      16     10628     10612   133.833       116   0.0670772    0.474018
  318      16     10662     10646    133.84       136     0.13329    0.474204
  319      15     10696     10681   133.859       140    0.229383    0.474197
2020-03-08 04:11:56.458317 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474212
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10728     10712   133.828       124    0.930734    0.474212
  321      16     10761     10745   133.822       132   0.0505491    0.474265
  322      16     10798     10782   133.866       148    0.903823    0.474141
  323      16     10830     10814   133.848       128    0.807813    0.474186
  324      15     10864     10849   133.867       140    0.831532     0.47415
  325      16     10900     10884   133.886       140    0.296874    0.473992
  326      16     10935     10919   133.905       140    0.063832    0.474013
  327      16     10965     10949   133.862       120     0.79645    0.474126
  328      16     10998     10982   133.856       132    0.678145    0.474212
  329      16     11031     11015    133.85       132    0.626806    0.474218
  330      15     11061     11046    133.82       124   0.0471994    0.474352
  331      16     11090     11074   133.754       112     0.68372    0.474614
  332      16     11125     11109   133.771       140    0.847348    0.474516
  333      16     11158     11142   133.766       132     0.75611    0.474536
  334      15     11195     11180    133.82       152    0.848292     0.47444
  335      16     11227     11211    133.79       124    0.105439    0.474316
  336      16     11259     11243   133.773       128   0.0813823    0.474474
  337      16     11292     11276   133.768       132   0.0459976    0.474436
  338      16     11324     11308    133.75       128    0.893798     0.47453
  339      16     11354     11338    133.71       120     0.79884    0.474674
2020-03-08 04:12:16.472073 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474712
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      15     11385     11370   133.692       128   0.0682948    0.474712
  341      16     11417     11401   133.663       124    0.742561    0.474946
  342      16     11449     11433   133.647       128    0.185589    0.474875
  343      16     11484     11468   133.665       140    0.902721    0.474839
  344      16     11516     11500   133.649       128   0.0688423    0.474803
  345      16     11552     11536   133.679       144    0.836811    0.474812
  346      16     11588     11572   133.708       144    0.137557    0.474792
  347      16     11619     11603    133.68       124     0.87439    0.474779
  348      16     11658     11642   133.744       156    0.191296    0.474667
  349      16     11693     11677   133.762       140    0.702421    0.474598
  350      16     11723     11707   133.723       120    0.196983    0.474567
  351      16     11754     11738   133.695       124    0.167663     0.47476
  352      16     11791     11775   133.736       148    0.109859    0.474685
  353      16     11824     11808   133.731       132    0.750969     0.47472
  354      15     11854     11839   133.703       124     0.14361    0.474778
  355      16     11897     11881   133.799       168    0.688509    0.474499
  356      16     11936     11920   133.861       156    0.677836    0.474238
  357      16     11968     11952   133.844       128   0.0714125    0.474323
  358      16     12007     11991   133.906       156    0.834854    0.474138
  359      15     12034     12019   133.845       112    0.043868    0.474195
2020-03-08 04:12:36.478934 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.474357
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     12066     12050   133.817       124    0.855982    0.474357
  361      15     12100     12085   133.835       140     0.88804    0.474415
  362      15     12139     12124   133.896       156    0.736932    0.474188
  363      16     12173     12157    133.89       132    0.707198    0.474193
  364      16     12212     12196   133.951       156    0.825003    0.473958
  365      15     12249     12234       134       152    0.805106    0.473776
  366      16     12284     12268   134.005       136   0.0502388    0.473744
  367      16     12315     12299   133.978       124    0.245479    0.473661
  368      16     12344     12328   133.929       116    0.867353    0.473996
  369      16     12379     12363   133.945       140    0.990894    0.473867
  370      16     12413     12397    133.95       136     0.15254    0.473861
  371      16     12448     12432   133.967       140    0.734665    0.473869
  372      16     12485     12469   134.004       148    0.640108    0.473788
  373      16     12518     12502   133.999       132    0.912343    0.473676
  374      16     12553     12537   134.015       140     0.71889    0.473748
  375      16     12585     12569   133.998       128     1.03926    0.473665
  376      16     12624     12608   134.057       156    0.802228    0.473576
  377      16     12655     12639    134.03       124    0.141191    0.473628
  378      16     12689     12673   134.035       136    0.904441    0.473585
  379      16     12725     12709   134.062       144     0.16921    0.473534
2020-03-08 04:12:56.485958 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473547
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12758     12742   134.056       132     0.71918    0.473547
  381      15     12794     12779   134.092       148    0.225409     0.47336
  382      16     12830     12814   134.108       140    0.682743    0.473427
  383      15     12861     12846   134.092       128    0.196221    0.473495
  384      15     12897     12882   134.118       144    0.662155    0.473363
  385      16     12931     12915   134.112       132    0.751263    0.473335
  386      16     12967     12951   134.138       144    0.153492    0.473265
  387      16     13006     12990   134.194       156    0.154775    0.473071
  388      16     13042     13026   134.219       144    0.194374    0.472919
  389      16     13077     13061   134.234       140    0.287094    0.472767
  390      16     13117     13101     134.3       160     0.62969    0.472774
  391      16     13153     13137   134.325       144     1.07428    0.472587
  392      16     13185     13169   134.308       128    0.894408    0.472724
  393      16     13216     13200   134.282       124    0.713725    0.472802
  394      16     13247     13231   134.256       124    0.314803    0.472781
  395      15     13279     13264    134.25       132    0.916748    0.472883
  396      16     13315     13299   134.263       140    0.911682    0.472751
  397      16     13346     13330   134.238       124    0.979752    0.472926
  398      16     13382     13366   134.262       144    0.711776    0.472886
  399      16     13416     13400   134.266       136   0.0573474    0.472696
2020-03-08 04:13:16.493917 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472771
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     13450     13434    134.27       136    0.859607    0.472771
  401      16     13483     13467   134.264       132    0.667416    0.472849
  402      15     13516     13501   134.268       136   0.0684735    0.472756
  403      16     13548     13532   134.243       124    0.981625    0.472919
  404      16     13580     13564   134.227       128    0.941701    0.472999
  405      16     13614     13598   134.231       136    0.846183     0.47295
  406      15     13649     13634   134.255       144    0.669232    0.472927
  407      16     13684     13668   134.259       136    0.739767    0.472925
  408      16     13716     13700   134.244       128    0.762265    0.472941
  409      16     13748     13732   134.228       128    0.649967    0.473048
  410      16     13780     13764   134.213       128    0.788554     0.47304
  411      16     13810     13794   134.178       120    0.162429    0.473092
  412      16     13842     13826   134.163       128    0.825973    0.473208
  413      16     13872     13856   134.129       120    0.628105    0.473408
  414      16     13905     13889   134.124       132   0.0464738    0.473404
  415      16     13941     13925   134.147       144    0.139263    0.473245
  416      16     13972     13956   134.123       124    0.717724    0.473413
  417      16     14001     13985   134.079       116    0.990372    0.473427
  418      16     14031     14015   134.046       120    0.076273    0.473565
  419      16     14065     14049    134.05       136    0.822905    0.473641
2020-03-08 04:13:36.502447 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473569
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     14100     14084   134.064       140    0.888226    0.473569
  421      16     14135     14119   134.079       140    0.863827    0.473574
  422      15     14169     14154   134.093       140    0.764471      0.4735
  423      15     14205     14190   134.113       144   0.0485052     0.47338
  424      16     14239     14223   134.108       132    0.880788    0.473372
  425      16     14273     14257   134.112       136    0.851865    0.473417
  426      16     14308     14292   134.126       140    0.778321    0.473365
  427      16     14336     14320   134.074       112    0.699846    0.473582
  428      16     14376     14360   134.135       160    0.662047    0.473371
  429      16     14409     14393   134.128       132     0.80638    0.473395
  430      16     14441     14425   134.114       128    0.661736    0.473451
  431      16     14475     14459   134.118       136    0.821582    0.473375
  432      16     14511     14495   134.141       144    0.623924    0.473341
  433      16     14544     14528   134.136       132    0.133769    0.473283
  434      16     14580     14564   134.159       144   0.0579914     0.47326
  435      16     14612     14596   134.145       128    0.047155      0.4733
  436      15     14643     14628   134.131       128   0.0690828    0.473359
  437      16     14676     14660   134.116       128    0.886116    0.473405
  438      16     14712     14696   134.139       144    0.763484    0.473301
  439      16     14745     14729   134.134       132    0.895585    0.473333
2020-03-08 04:13:56.523028 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473417
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14776     14760    134.11       124     1.01057    0.473417
  441      16     14808     14792   134.096       128    0.247927    0.473425
  442      16     14841     14825   134.091       132    0.742684    0.473471
  443      16     14875     14859   134.095       136    0.760651    0.473463
  444      16     14904     14888   134.055       116    0.847611    0.473571
  445      16     14936     14920   134.041       128    0.180171    0.473594
  446      16     14974     14958   134.081       152   0.0698579    0.473498
  447      16     15010     14994   134.103       144    0.158024    0.473394
  448      16     15044     15028   134.107       136   0.0708973    0.473373
  449      16     15080     15064   134.129       144    0.720252    0.473348
  450      16     15116     15100   134.151       144    0.258729    0.473282
  451      16     15148     15132   134.137       128   0.0758505    0.473335
  452      16     15185     15169   134.168       148    0.196685    0.473168
  453      16     15222     15206   134.198       148    0.127752    0.473132
  454      16     15259     15243   134.229       148   0.0548841    0.472988
  455      16     15294     15278   134.241       140    0.195632    0.472949
  456      16     15326     15310   134.228       128    0.851891    0.472973
  457      16     15361     15345    134.24       140    0.889282    0.472938
  458      16     15396     15380   134.253       140   0.0482713    0.472848
  459      16     15432     15416   134.274       144    0.653691    0.472897
2020-03-08 04:14:16.528328 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472993
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     15462     15446   134.243       120    0.717287    0.472993
  461      16     15495     15479   134.238       132    0.181655    0.472955
  462      16     15528     15512   134.232       132    0.836749    0.472965
  463      16     15559     15543    134.21       124    0.912579    0.473044
  464      16     15597     15581   134.248       152    0.940377    0.472942
  465      15     15629     15614   134.243       132    0.690721    0.473003
  466      16     15666     15650   134.264       144    0.783025    0.472897
  467      16     15700     15684   134.267       136    0.950257    0.472817
  468      16     15729     15713   134.228       116   0.0454483    0.472903
  469      16     15766     15750   134.257       148    0.140922    0.472863
  470      16     15801     15785   134.269       140   0.0680671     0.47289
  471      16     15834     15818   134.264       132     0.82093    0.472864
  472      16     15869     15853   134.276       140    0.928222    0.472811
  473      16     15906     15890   134.305       148    0.678418    0.472805
  474      16     15943     15927   134.334       148    0.800747    0.472668
  475      16     15973     15957   134.304       120    0.908088    0.472756
  476      16     16005     15989    134.29       128    0.619162    0.472867
  477      16     16035     16019    134.26       120    0.130745    0.472862
  478      16     16071     16055   134.281       144   0.0838108    0.472889
  479      16     16106     16090   134.293       140   0.0818253    0.472724
2020-03-08 04:14:36.538910 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472671
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     16142     16126   134.313       144     0.78565    0.472671
  481      16     16175     16159   134.308       132   0.0648891    0.472641
  482      16     16210     16194    134.32       140    0.142354    0.472684
  483      15     16247     16232   134.356       152    0.690424    0.472611
  484      16     16280     16264   134.343       128    0.620573    0.472676
  485      16     16312     16296    134.33       128     1.01156    0.472619
  486      16     16347     16331   134.341       140    0.787834    0.472603
  487      16     16382     16366   134.353       140     0.04787     0.47261
  488      16     16414     16398    134.34       128    0.216779    0.472636
  489      16     16446     16430   134.327       128    0.272641    0.472565
  490      16     16481     16465   134.338       140    0.911449    0.472617
  491      16     16511     16495   134.309       120    0.134488    0.472747
  492      16     16543     16527   134.296       128     1.02804    0.472679
  493      16     16573     16557   134.267       120    0.839804    0.472827
  494      16     16607     16591   134.271       136    0.769509     0.47291
  495      16     16640     16624   134.266       132    0.303948    0.472781
  496      16     16678     16662   134.302       152   0.0835276     0.47275
  497      16     16714     16698   134.321       144    0.786149    0.472696
  498      16     16745     16729     134.3       124    0.707991    0.472805
  499      16     16777     16761   134.288       128     0.22756    0.472735
2020-03-08 04:14:56.543533 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.47277
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16809     16793   134.275       128    0.172409     0.47277
  501      16     16840     16824   134.254       124   0.0822042    0.472909
  502      16     16873     16857    134.25       132    0.795518     0.47294
  503      16     16909     16893   134.269       144   0.0695127    0.472889
  504      16     16944     16928    134.28       140    0.794149    0.472859
  505      16     16981     16965   134.307       148    0.655628    0.472754
  506      16     17010     16994   134.271       116    0.580916    0.472939
  507      16     17048     17032   134.306       152    0.660026    0.472807
  508      16     17084     17068   134.325       144    0.855374    0.472647
  509      16     17116     17100   134.313       128    0.350447     0.47269
  510      16     17146     17130   134.285       120    0.748529    0.472831
  511      16     17183     17167   134.311       148    0.757828    0.472747
  512      16     17216     17200   134.307       132    0.906383    0.472754
  513      16     17247     17231   134.287       124    0.761509    0.472837
  514      16     17280     17264   134.282       132    0.837028     0.47284
  515      15     17312     17297   134.278       132    0.968347    0.472885
  516      15     17344     17329   134.265       128     1.11924     0.47286
  517      16     17376     17360   134.246       124    0.745734    0.472992
  518      16     17413     17397   134.272       148    0.701706    0.472932
  519      16     17442     17426   134.237       116    0.611102    0.473064
2020-03-08 04:15:16.548746 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473031
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     17474     17458   134.225       128    0.840809    0.473031
  521      15     17506     17491    134.22       132    0.222493    0.473045
  522      16     17543     17527   134.239       144    0.901258    0.472949
  523      16     17581     17565   134.273       152     0.81055    0.472926
  524      16     17616     17600   134.284       140   0.0574539    0.472791
  525      16     17653     17637    134.31       148     1.00109    0.472656
  526      16     17688     17672   134.321       140    0.804263    0.472727
  527      16     17720     17704   134.309       128    0.148273    0.472726
  528      16     17754     17738   134.312       136    0.701567    0.472733
  529      16     17785     17769   134.292       124    0.710798     0.47282
  530      16     17816     17800   134.273       124    0.199646    0.472798
  531      16     17845     17829   134.238       116     0.98901    0.472893
  532      16     17883     17867   134.272       152   0.0550675    0.472875
  533      16     17913     17897   134.245       120    0.754817    0.472953
  534      16     17948     17932   134.255       140    0.939429    0.472884
  535      16     17981     17965   134.251       132     1.12066    0.472872
  536      16     18015     17999   134.254       136    0.218078    0.472926
  537      16     18049     18033   134.257       136    0.181749    0.472942
  538      16     18082     18066   134.253       132     0.73172    0.472987
  539      16     18116     18100   134.256       136    0.965622    0.472895
2020-03-08 04:15:36.553794 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.47297
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     18148     18132   134.245       128   0.0514879     0.47297
  541      16     18181     18165   134.241       132    0.175265    0.472992
  542      16     18213     18197   134.229       128   0.0910136    0.473062
  543      16     18246     18230   134.225       132    0.808678     0.47307
  544      16     18276     18260   134.199       120    0.167186    0.473168
  545      16     18314     18298   134.231       152    0.735309    0.473034
  546      16     18349     18333   134.242       140    0.658261     0.47304
  547      16     18385     18369   134.259       144   0.0839557    0.472923
  548      16     18418     18402   134.255       132    0.782334    0.472971
  549      16     18452     18436   134.258       136    0.202949    0.472862
  550      16     18487     18471   134.269       140    0.680696     0.47295
  551      15     18521     18506   134.279       140    0.138018    0.472799
  552      16     18552     18536   134.253       120    0.962779    0.472901
  553      16     18586     18570   134.256       136    0.789027     0.47295
  554      16     18616     18600   134.231       120    0.133948    0.472952
  555      16     18653     18637   134.255       148    0.342443    0.472917
  556      16     18688     18672   134.266       140   0.0801631    0.472948
  557      16     18718     18702    134.24       120     0.70385    0.473009
  558      16     18753     18737    134.25       140    0.156122    0.472998
  559      16     18784     18768   134.232       124   0.0722608     0.47304
2020-03-08 04:15:56.557483 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.473061
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     18816     18800   134.221       128    0.872465    0.473061
  561      15     18852     18837   134.245       148    0.861297    0.472987
  562      16     18886     18870   134.241       132    0.907389    0.473017
  563      16     18919     18903   134.236       132    0.829337     0.47301
  564      16     18952     18936   134.232       132    0.737176    0.473067
  565      16     18984     18968   134.221       128   0.0678889    0.473082
  566      16     19020     19004   134.239       144   0.0815913    0.473049
  567      16     19053     19037   134.235       132    0.822984    0.473047
  568      16     19091     19075   134.266       152    0.683587    0.472959
  569      16     19121     19105   134.241       120     1.03885     0.47299
  570      16     19159     19143   134.272       152   0.0600585    0.472908
  571      16     19190     19174   134.254       124    0.736523    0.472955
  572      16     19224     19208   134.257       136     0.73722    0.472972
  573      16     19257     19241   134.253       132   0.0487991    0.472953
  574      16     19289     19273   134.242       128     0.24744    0.472992
  575      16     19324     19308   134.252       140    0.391084    0.472896
  576      15     19354     19339   134.234       124    0.663745    0.473089
  577      16     19391     19375   134.251       144    0.748631    0.472986
  578      16     19424     19408   134.247       132     1.20114    0.472932
  579      16     19461     19445    134.27       148    0.828344    0.472912
2020-03-08 04:16:16.565398 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472773
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     19498     19482   134.294       148    0.172264    0.472773
  581      16     19527     19511   134.263       116    0.709465    0.472963
  582      15     19562     19547   134.279       144    0.725635    0.472916
  583      16     19599     19583   134.296       144    0.057383    0.472805
  584      16     19630     19614   134.278       124    0.734971    0.472896
  585      16     19664     19648   134.281       136    0.775093    0.472893
  586      16     19694     19678   134.257       120     0.56365    0.473032
  587      16     19728     19712    134.26       136    0.853611    0.472951
  588      16     19762     19746   134.263       136    0.798978    0.472975
  589      15     19798     19783   134.286       148   0.0450345    0.472877
  590      16     19833     19817   134.288       136    0.078982    0.472808
  591      16     19865     19849   134.278       128    0.219248    0.472797
  592      16     19897     19881   134.267       128    0.855589    0.472938
  593      16     19930     19914   134.263       132    0.824729    0.472954
  594      16     19967     19951   134.286       148    0.781349    0.472909
  595      16     20000     19984   134.282       132    0.160079    0.472887
  596      16     20034     20018   134.285       136    0.634084    0.472905
  597      16     20070     20054   134.301       144    0.897115    0.472823
  598      16     20101     20085   134.284       124    0.791925     0.47289
  599      16     20133     20117   134.273       128   0.0607163    0.472916
2020-03-08 04:16:36.574697 min lat: 0.00821512 max lat: 1.35937 avg lat: 0.472884
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     20161     20145   134.236       112    0.778877    0.472884
Total time run:       600.576
Total reads made:     20161
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   134.278
Average IOPS:         33
Stddev IOPS:          2.62918
Max IOPS:             42
Min IOPS:             27
Average Latency(s):   0.473116
Max latency(s):       1.35937
Min latency(s):       0.00821512

Total time run:       600.576
Total reads made:     20161
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   134.278
Average IOPS:         33
Stddev IOPS:          2.62918
Max IOPS:             42
Min IOPS:             27
Average Latency(s):   0.473116
Max latency(s):       1.35937
Min latency(s):       0.00821512

Cleanup, pool with 8 PGs

podman exec ceph-mon-f5-422-01 rados -p testbench cleanup

Performance Baseline, pool with 256 PGs

Since the cluster is brand new and largely unused, I purged everything. Not because I had to, but because I could ™ :-)

I went on to create a single, replica 3, pool testbench with 256 PGs. I also set the application type to rgw. Both of these were mainly done to shut up the (expected) warnings. Now I have 96 PGs per OSD.

Performance Measurements with rados bench, pool with 256 PGs

All rados bench tests were run on f5-422-01.

Sequential Write Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 write --no-cleanup | tee --append $(date +%F)-rados-test-write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects
Object prefix: benchmark_data_f5-422-01.internal.pcfe.net_3074
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        46        30   119.975       120    0.359667    0.380986
    2      16        78        62   123.977       128    0.347748    0.432131
    3      16       112        96   127.978       136    0.749442    0.438789
    4      16       140       124   123.979       112     0.24305     0.48014
    5      16       178       162   129.575       152    0.356074    0.463551
    6      16       201       185   123.308        92    0.136012    0.477297
    7      16       232       216   123.401       124     0.68408     0.49937
    8      16       268       252   125.973       144    0.197888    0.494673
    9      16       304       288   127.974       144    0.469769    0.487006
   10      16       342       326   130.374       152    0.755879    0.479612
   11      16       381       365   132.701       156    0.410596    0.470903
   12      16       418       402   133.969       148    0.518643    0.469246
   13      16       451       435   133.816       132    0.155759    0.469258
   14      16       479       463   132.257       112    0.266196    0.470504
   15      16       519       503   134.105       160    0.341318    0.468726
   16      16       546       530   132.472       108    0.212679    0.473273
   17      16       574       558   131.267       112    0.568618    0.482085
   18      16       601       585   129.973       108    0.234361    0.483411
   19      16       636       620   130.499       140    0.415921    0.483334
2020-03-09 13:47:33.737970 min lat: 0.0717105 max lat: 1.31601 avg lat: 0.484174
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       669       653   130.573       132    0.747809    0.484174
   21      16       699       683   130.068       120    0.429075    0.484329
   22      16       735       719     130.7       144    0.330224    0.483207
   23      16       764       748   130.061       116     0.45127    0.486249
   24      16       799       783   130.473       140    0.608222    0.484097
   25      16       834       818   130.852       140    0.276788    0.484868
   26      16       871       855   131.507       148    0.643607    0.482013
   27      16       903       887   131.373       128    0.569667     0.48285
   28      16       932       916    130.82       116    0.255666    0.483143
   29      16       966       950   130.973       136    0.588522     0.48515
   30      16       998       982   130.872       128     0.65502    0.483678
   31      16      1024      1008   130.004       104    0.762551    0.488108
   32      16      1044      1028   128.439        80    0.375524    0.492252
   33      16      1066      1050   127.214        88    0.356193     0.49718
   34      16      1096      1080   127.001       120    0.337509    0.498001
   35      16      1122      1106   126.343       104    0.186368    0.501584
   36      16      1147      1131   125.611       100     0.51983    0.504995
   37      16      1181      1165   125.891       136    0.565213    0.504746
   38      16      1222      1206   126.893       164    0.120005    0.501804
   39      16      1264      1248   127.946       168    0.218359    0.497675
2020-03-09 13:47:53.750444 min lat: 0.0717105 max lat: 1.46248 avg lat: 0.49587
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1295      1279   127.847       124    0.460262     0.49587
   41      16      1333      1317   128.435       152    0.643408    0.494639
   42      16      1367      1351   128.612       136    0.398122    0.493743
   43      16      1405      1389   129.155       152    0.430469    0.492381
   44      16      1437      1421   129.129       128    0.206729    0.492294
   45      16      1466      1450   128.836       116   0.0939879    0.493724
   46      16      1496      1480   128.642       120    0.532239    0.493498
   47      16      1535      1519   129.224       156     0.30192     0.49243
   48      16      1572      1556   129.614       148    0.303496     0.49201
   49      16      1602      1586   129.416       120    0.790806    0.491831
   50      16      1632      1616   129.228       120    0.719665    0.491551
   51      16      1661      1645   128.968       116     0.49687    0.494285
   52      16      1691      1675   128.794       120    0.643789    0.493206
   53      16      1719      1703   128.477       112    0.174905     0.49502
   54      16      1753      1737   128.615       136    0.343396    0.494993
   55      16      1781      1765   128.313       112    0.218915    0.495612
   56      16      1813      1797   128.307       128    0.651625     0.49658
   57      16      1840      1824   127.949       108   0.0950628    0.496028
   58      16      1878      1862   128.363       152    0.364567    0.496509
   59      16      1914      1898   128.627       144    0.322983    0.494879
2020-03-09 13:48:13.757359 min lat: 0.0716476 max lat: 1.46248 avg lat: 0.495193
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1945      1929    128.55       124    0.277205    0.495193
   61      16      1969      1953   128.015        96     0.15123    0.495979
   62      16      1994      1978   127.563       100    0.863083    0.497656
   63      16      2017      2001   126.998        92    0.740748    0.500637
   64      16      2044      2028   126.701       108    0.132614     0.50229
   65      16      2080      2064   126.966       144    0.199869    0.501354
   66      16      2122      2106   127.587       168    0.505488    0.500336
   67      16      2156      2140   127.712       136    0.513155    0.499443
   68      16      2194      2178   128.069       152    0.251558    0.497586
   69      16      2225      2209    128.01       124     1.05985     0.49788
   70      16      2253      2237   127.781       112    0.528519    0.498064
   71      16      2288      2272   127.953       140    0.273438    0.498547
   72      16      2321      2305   128.009       132    0.919142    0.497856
   73      16      2358      2342   128.282       148    0.260887    0.496907
   74      16      2391      2375   128.331       132    0.440161    0.497008
   75      16      2426      2410   128.487       140    0.195383     0.49716
   76      16      2458      2442    128.48       128    0.269111    0.495936
   77      16      2491      2475   128.525       132    0.180516    0.495634
   78      16      2527      2511   128.723       144    0.272189    0.495631
   79      16      2561      2545   128.814       136    0.725491    0.495048
2020-03-09 13:48:33.762806 min lat: 0.0716476 max lat: 1.46248 avg lat: 0.495577
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2586      2570   128.453       100    0.497292    0.495577
   81      16      2614      2598    128.25       112    0.242421     0.49702
   82      16      2635      2619   127.708        84     0.66775    0.498904
   83      16      2664      2648   127.566       116   0.0780687     0.49973
   84      16      2690      2674   127.285       104    0.254723    0.500546
   85      16      2722      2706   127.294       128    0.393947    0.500498
   86      16      2747      2731   126.976       100    0.229089    0.501614
   87      16      2780      2764   127.034       132    0.087223    0.501226
   88      16      2816      2800   127.226       144    0.645412    0.501348
   89      16      2846      2830   127.145       120     1.23799    0.501021
   90      16      2880      2864   127.243       136    0.621924    0.501547
   91      16      2913      2897   127.295       132    0.125684    0.501073
   92      16      2939      2923    127.04       104    0.859025    0.502364
   93      16      2974      2958   127.179       140    0.571539     0.50204
   94      16      3007      2991    127.23       132    0.188611    0.501303
   95      16      3037      3021   127.153       120    0.192009    0.500719
   96      16      3077      3061   127.493       160    0.484729     0.50058
   97      16      3110      3094   127.531       132    0.734255    0.500618
   98      16      3148      3132    127.78       152    0.433723    0.499386
   99      16      3179      3163   127.742       124    0.431907      0.4993
2020-03-09 13:48:53.777399 min lat: 0.0716476 max lat: 1.78416 avg lat: 0.500599
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3206      3190   127.544       108    0.665649    0.500599
  101      16      3237      3221   127.509       124    0.369357    0.500074
  102      16      3274      3258   127.709       148    0.198156    0.499872
  103      16      3315      3299   128.061       164    0.438459    0.498676
  104      16      3357      3341   128.445       168    0.223767    0.496934
  105      16      3390      3374   128.478       132    0.451224    0.496417
  106      16      3431      3415   128.813       164    0.456859    0.495683
  107      16      3465      3449    128.88       136    0.371715    0.495341
  108      16      3503      3487   129.094       152    0.301211    0.494671
  109      16      3540      3524   129.267       148    0.236523    0.493584
  110      16      3575      3559   129.365       140    0.578098    0.493294
  111      16      3608      3592   129.388       132    0.131863    0.493406
  112      16      3644      3628   129.518       144    0.312151    0.493075
  113      16      3681      3665   129.681       148    0.593661    0.492712
  114      16      3709      3693   129.526       112    0.772584    0.492846
  115      16      3750      3734   129.826       164    0.334616    0.491762
  116      16      3785      3769   129.913       140    0.215087    0.491054
  117      16      3816      3800   129.862       124    0.302777     0.49167
  118      16      3846      3830   129.778       120    0.357825    0.492301
  119      16      3880      3864    129.83       136    0.539661    0.491452
2020-03-09 13:49:13.783504 min lat: 0.0714435 max lat: 1.78416 avg lat: 0.492061
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3911      3895    129.78       124    0.369217    0.492061
  121      16      3950      3934   129.996       156    0.337596    0.491487
  122      16      3985      3969   130.077       140    0.325811    0.491144
  123      16      4015      3999   129.995       120    0.234159    0.490601
  124      16      4050      4034   130.075       140    0.351187     0.49128
  125      16      4075      4059   129.834       100    0.612619    0.491183
  126      16      4109      4093   129.883       136    0.742749    0.491513
  127      16      4142      4126   129.899       132    0.459294    0.491227
  128      16      4179      4163    130.04       148    0.571095    0.491236
  129      16      4219      4203   130.272       160    0.457634    0.490343
  130      16      4250      4234   130.224       124    0.745385    0.490243
  131      16      4275      4259   129.993       100    0.114989    0.490629
  132      16      4313      4297   130.159       152    0.467331    0.490717
  133      16      4344      4328   130.112       124    0.367843    0.490623
  134      16      4380      4364   130.216       144    0.554696    0.490194
  135      16      4422      4406   130.495       168    0.187854     0.48958
  136      16      4452      4436   130.417       120    0.768817    0.489632
  137      16      4491      4475   130.603       156    0.413473    0.489357
  138      16      4535      4519   130.931       176    0.273079    0.488159
  139      16      4576      4560   131.168       164    0.532096    0.487224
2020-03-09 13:49:33.792025 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.487542
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4601      4585   130.946       100    0.343155    0.487542
  141      16      4638      4622   131.065       148     0.49814    0.487485
  142      16      4671      4655   131.071       132    0.629704    0.487162
  143      16      4700      4684   130.965       116    0.212529    0.487468
  144      16      4736      4720   131.056       144    0.460629    0.487877
  145      16      4767      4751   131.007       124    0.436478    0.487351
  146      16      4808      4792   131.231       164    0.482506    0.487213
  147      16      4845      4829   131.345       148    0.554865    0.486456
  148      16      4880      4864   131.403       140     0.83767    0.485906
  149      16      4913      4897   131.407       132     0.23633     0.48595
  150      16      4945      4929   131.384       128    0.513121    0.486439
  151      16      4974      4958   131.281       116    0.565548    0.486642
  152      16      5006      4990   131.259       128    0.171883    0.486551
  153      16      5041      5025   131.316       140    0.435409    0.486609
  154      16      5076      5060   131.372       140    0.158076    0.485858
  155      16      5111      5095   131.427       140    0.304771    0.486107
  156      16      5147      5131   131.508       144    0.374117    0.485598
  157      16      5173      5157   131.332       104   0.0826305     0.48585
  158      16      5206      5190   131.336       132    0.310246    0.486187
  159      16      5245      5229   131.491       156    0.166397    0.485676
2020-03-09 13:49:53.801959 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.485703
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5278      5262   131.494       132    0.743372    0.485703
  161      16      5311      5295   131.497       132    0.518896    0.485829
  162      16      5348      5332   131.599       148    0.559159    0.485547
  163      16      5380      5364   131.576       128    0.272844     0.48551
  164      16      5412      5396   131.554       128    0.984581    0.485553
  165      16      5447      5431   131.605       140    0.686722    0.485379
  166      16      5477      5461   131.535       120    0.300734    0.485402
  167      16      5521      5505   131.801       176      0.4637    0.484717
  168      16      5558      5542   131.896       148    0.381065    0.484266
  169      16      5602      5586   132.156       176    0.746587    0.483521
  170      16      5640      5624   132.272       152    0.370748    0.483163
  171      16      5676      5660    132.34       144      0.2788    0.482865
  172      16      5707      5691   132.292       124    0.408688    0.482885
  173      16      5746      5730   132.428       156    0.584259    0.482697
  174      16      5782      5766   132.495       144   0.0789463    0.482216
  175      16      5816      5800   132.515       136      0.2359    0.482381
  176      16      5849      5833   132.512       132     1.08834    0.482006
  177      16      5882      5866   132.508       132    0.472668    0.482455
  178      16      5921      5905   132.639       156    0.321792    0.481885
  179      16      5960      5944   132.769       156    0.565195    0.481509
2020-03-09 13:50:13.811540 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.481274
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5994      5978   132.787       136    0.470208    0.481274
  181      16      6023      6007   132.694       116    0.390591    0.481245
  182      16      6060      6044   132.778       148      0.1815    0.481144
  183      16      6096      6080   132.839       144    0.335024     0.48077
  184      16      6124      6108   132.726       112    0.534821    0.481351
  185      16      6167      6151   132.938       172    0.335853     0.48095
  186      16      6201      6185   132.954       136    0.386932    0.480916
  187      16      6239      6223   133.056       152    0.972335    0.480284
  188      16      6269      6253   132.987       120    0.376611    0.480494
  189      16      6313      6297   133.214       176   0.0865965    0.479733
  190      16      6339      6323    133.06       104    0.147103    0.479894
  191      16      6377      6361   133.159       152     0.24453    0.480105
  192      16      6404      6388   133.028       108    0.218203     0.48002
  193      16      6444      6428   133.168       160    0.106025    0.479846
  194      16      6477      6461   133.161       132     1.00194    0.479703
  195      16      6514      6498   133.237       148    0.805688    0.479583
  196      16      6550      6534   133.292       144    0.520331    0.479262
  197      16      6588      6572   133.387       152    0.408412    0.478834
  198      16      6624      6608    133.44       144    0.271075    0.478981
  199      16      6664      6648   133.572       160    0.196374    0.478292
2020-03-09 13:50:33.817509 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.478414
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6697      6681   133.564       132     0.65436    0.478414
  201      16      6739      6723   133.735       168    0.274473    0.478079
  202      16      6772      6756   133.727       132    0.753022    0.477889
  203      16      6806      6790   133.738       136    0.538447    0.477985
  204      16      6840      6824   133.749       136    0.373896    0.477825
  205      16      6879      6863   133.857       156    0.320515     0.47767
  206      16      6917      6901   133.945       152     0.33905    0.477112
  207      16      6948      6932   133.897       124    0.745551    0.477341
  208      16      6986      6970   133.984       152    0.217822    0.477273
  209      16      7024      7008    134.07       152    0.580256    0.476888
  210      16      7061      7045   134.136       148    0.438728    0.476647
  211      16      7097      7081   134.182       144    0.610789    0.476413
  212      16      7123      7107   134.039       104    0.309203    0.476553
  213      16      7163      7147   134.161       160    0.168109    0.476489
  214      16      7195      7179   134.132       128    0.389336    0.476631
  215      16      7225      7209   134.066       120    0.608599    0.476835
  216      16      7257      7241   134.036       128    0.376109    0.477007
  217      16      7284      7268   133.916       108    0.674074    0.477033
  218      16      7320      7304   133.962       144    0.518294    0.477249
  219      16      7355      7339   133.989       140    0.255822    0.477096
2020-03-09 13:50:53.827076 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.476958
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7390      7374   134.016       140    0.359536    0.476958
  221      16      7422      7406   133.988       128    0.235458     0.47688
  222      16      7461      7445   134.087       156    0.552057    0.476534
  223      16      7500      7484   134.185       156    0.122898    0.476439
  224      16      7540      7524     134.3       160    0.677597    0.476169
  225      16      7577      7561   134.361       148    0.478742     0.47592
  226      16      7605      7589   134.262       112    0.452826    0.475701
  227      16      7641      7625   134.305       144    0.361525    0.475857
  228      16      7678      7662   134.365       148    0.232513    0.475916
  229      16      7713      7697   134.389       140    0.377874    0.475651
  230      16      7751      7735   134.466       152    0.283504    0.475249
  231      16      7783      7767   134.438       128     0.73329    0.475222
  232      16      7820      7804   134.496       148    0.585296    0.475329
  233      16      7857      7841   134.554       148    0.298701    0.475105
  234      16      7894      7878   134.611       148    0.303633    0.474928
  235      16      7928      7912   134.617       136    0.207075    0.474899
  236      16      7965      7949   134.673       148    0.586429    0.474658
  237      16      7997      7981   134.645       128    0.233565    0.474662
  238      16      8038      8022   134.767       164    0.301697    0.474489
  239      16      8079      8063   134.889       164    0.467994    0.474005
2020-03-09 13:51:13.835219 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.474078
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      8113      8097   134.893       136    0.301468    0.474078
  241      16      8153      8137   134.997       160    0.114605    0.473718
  242      16      8193      8177   135.099       160    0.289684     0.47336
  243      16      8225      8209    135.07       128    0.704905    0.473141
  244      16      8265      8249   135.172       160    0.289248    0.473148
  245      16      8303      8287   135.241       152     0.56666    0.472784
  246      16      8331      8315   135.146       112    0.479352    0.472771
  247      16      8362      8346   135.101       124    0.299596    0.472932
  248      16      8388      8372   134.975       104    0.224243    0.473381
  249      16      8424      8408   135.012       144    0.576815    0.473601
  250      16      8452      8436   134.919       112    0.219891    0.473646
  251      16      8481      8465   134.844       116    0.231252    0.473778
  252      16      8522      8506   134.959       164    0.250522    0.473693
  253      16      8563      8547   135.074       164    0.174336    0.473367
  254      16      8591      8575   134.983       112     0.81775    0.473567
  255      16      8630      8614   135.065       156    0.411743     0.47345
  256      16      8656      8640   134.944       104     0.72775    0.473448
  257      16      8689      8673   134.932       132    0.594755    0.473645
  258      16      8728      8712   135.014       156    0.465425    0.473708
  259      16      8761      8745   135.002       132    0.324839    0.473496
2020-03-09 13:51:33.841271 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.473384
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8798      8782   135.052       148    0.372748    0.473384
  261      16      8830      8814   135.025       128    0.821787     0.47336
  262      16      8856      8840   134.906       104    0.394544    0.473502
  263      16      8890      8874    134.91       136    0.582109    0.473994
  264      16      8928      8912   134.975       152     0.40272    0.473827
  265      16      8967      8951   135.054       156    0.154145    0.473271
  266      16      9000      8984   135.043       132    0.888531    0.473354
  267      16      9036      9020   135.076       144    0.140036    0.473409
  268      16      9069      9053   135.064       132    0.152656    0.473387
  269      16      9100      9084   135.023       124    0.741014    0.473505
  270      16      9137      9121   135.071       148    0.252373     0.47329
  271      16      9180      9164   135.207       172    0.349037    0.472971
  272      16      9203      9187   135.049        92    0.307449     0.47316
  273      16      9241      9225    135.11       152    0.127631    0.473415
  274      16      9279      9263   135.172       152    0.376798     0.47296
  275      16      9305      9289   135.058       104    0.982742    0.473267
  276      16      9340      9324   135.076       140    0.226019    0.473228
  277      16      9378      9362   135.137       152    0.518103    0.473143
  278      16      9395      9379   134.895        68    0.947851     0.47367
  279      16      9432      9416   134.942       148    0.173529    0.473801
2020-03-09 13:51:53.846751 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.47375
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9464      9448   134.917       128    0.218692     0.47375
  281      16      9493      9477    134.85       116    0.175167    0.473961
  282      16      9536      9520    134.98       172    0.325448    0.473871
  283      16      9570      9554   134.984       136     0.30484    0.473574
  284      16      9600      9584   134.931       120    0.766058    0.473857
  285      16      9624      9608   134.794        96    0.565996    0.474319
  286      16      9654      9638   134.742       120    0.164893    0.474432
  287      16      9690      9674   134.774       144    0.156769    0.474394
  288      16      9716      9700   134.667       104     0.18715    0.474512
  289      16      9750      9734   134.672       136    0.652454    0.474872
  290      16      9779      9763   134.607       116   0.0811866    0.474867
  291      16      9812      9796   134.598       132    0.304357    0.475065
  292      16      9850      9834   134.658       152    0.780671    0.474751
  293      16      9888      9872   134.717       152     0.81104    0.474651
  294      16      9925      9909   134.762       148     1.01698    0.474468
  295      16      9956      9940   134.725       124    0.911035    0.474546
  296      16      9992      9976   134.756       144    0.572994    0.474646
  297      16     10032     10016    134.84       160    0.179373    0.474415
  298      16     10060     10044   134.764       112    0.371462    0.474367
  299      16     10099     10083   134.835       156      0.6202    0.474196
2020-03-09 13:52:13.856173 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.474124
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16     10135     10119   134.865       144    0.384481    0.474124
  301      16     10170     10154   134.882       140    0.385599    0.474122
  302      15     10201     10186   134.859       128    0.245145    0.473929
  303      16     10246     10230   134.995       176    0.263429    0.473822
  304      16     10277     10261   134.959       124    0.952323     0.47358
  305      16     10315     10299   135.014       152    0.808054     0.47368
  306      16     10348     10332   135.004       132    0.304334    0.473674
  307      16     10384     10368   135.034       144    0.406567    0.473544
  308      16     10418     10402   135.037       136    0.169952    0.473345
  309      16     10446     10430   134.962       112      1.2108     0.47354
  310      16     10488     10472   135.068       168   0.0938959    0.473156
  311      16     10523     10507   135.084       140    0.364407     0.47338
  312      16     10560     10544   135.126       148    0.503998    0.473255
  313      16     10589     10573   135.064       116    0.544923    0.473554
  314      16     10622     10606   135.055       132     0.59883    0.473577
  315      16     10658     10642   135.083       144    0.932141    0.473393
  316      16     10693     10677   135.098       140    0.379453    0.473252
  317      16     10733     10717   135.177       160    0.100403    0.472957
  318      16     10769     10753   135.204       144    0.487444    0.473017
  319      16     10804     10788   135.219       140    0.165549    0.472857
2020-03-09 13:52:33.859926 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472819
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10839     10823   135.234       140    0.896992    0.472819
  321      16     10871     10855   135.212       128    0.820972    0.472819
  322      16     10908     10892   135.251       148    0.193181    0.472727
  323      16     10948     10932   135.328       160    0.293782    0.472619
  324      16     10976     10960   135.256       112    0.373327    0.472698
  325      16     11016     11000   135.332       160    0.110501    0.472549
  326      16     11049     11033   135.321       132    0.434871    0.472617
  327      16     11088     11072   135.385       156    0.120279    0.472248
  328      16     11123     11107   135.398       140    0.162674    0.472318
  329      16     11153     11137   135.352       120    0.340746    0.472257
  330      16     11186     11170   135.341       132    0.279803     0.47247
  331      16     11224     11208    135.39       152    0.256636     0.47234
  332      16     11246     11230   135.247        88    0.344452    0.472721
  333      16     11278     11262   135.226       128    0.559596     0.47266
  334      16     11309     11293   135.192       124    0.322432     0.47305
  335      16     11352     11336   135.301       172    0.401244    0.472582
  336      16     11385     11369   135.291       132    0.260196    0.472609
  337      16     11416     11400   135.258       124    0.937847    0.472758
  338      16     11450     11434    135.26       136    0.156418    0.472695
  339      16     11481     11465   135.227       124    0.416435    0.472927
2020-03-09 13:52:53.868860 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472805
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     11519     11503   135.276       152    0.735485    0.472805
  341      16     11554     11538   135.289       140    0.713162    0.472787
  342      16     11586     11570   135.268       128    0.395527     0.47277
  343      16     11613     11597   135.188       108     0.16304    0.473001
  344      16     11648     11632   135.202       140    0.498541    0.473054
  345      16     11685     11669   135.239       148    0.431977    0.472929
  346      16     11719     11703   135.241       136    0.365017    0.472964
  347      16     11749     11733   135.197       120    0.208045    0.472912
  348      16     11793     11777   135.314       176    0.288502    0.472714
  349      16     11822     11806   135.259       116    0.615701    0.472621
  350      16     11862     11846   135.329       160    0.433324    0.472623
  351      16     11892     11876   135.285       120    0.460653    0.472535
  352      16     11932     11916   135.355       160    0.187522    0.472529
  353      16     11963     11947   135.323       124    0.252118    0.472454
  354      16     12002     11986   135.381       156    0.579114    0.472448
  355      16     12036     12020   135.383       136    0.411726    0.472376
  356      16     12064     12048   135.317       112    0.261821     0.47244
  357      16     12098     12082   135.319       136    0.696116    0.472521
  358      16     12121     12105   135.198        92     1.37038    0.472704
  359      16     12161     12145   135.267       160    0.428137     0.47286
2020-03-09 13:53:13.875746 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.472768
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     12198     12182   135.302       148    0.322379    0.472768
  361      16     12232     12216   135.304       136    0.931174    0.472525
  362      16     12263     12247   135.273       124    0.231982    0.472463
  363      16     12303     12287   135.341       160     0.42676    0.472412
  364      16     12341     12325   135.386       152     0.46565     0.47234
  365      16     12368     12352   135.311       108    0.382646    0.472508
  366      16     12402     12386   135.313       136    0.455886    0.472637
  367      16     12433     12417   135.282       124   0.0852436    0.472558
  368      16     12473     12457   135.349       160    0.560605    0.472514
  369      16     12506     12490    135.34       132    0.260523    0.472529
  370      16     12549     12533   135.439       172    0.226338    0.472307
  371      16     12582     12566    135.43       132    0.393776    0.471963
  372      16     12617     12601   135.442       140    0.350945     0.47224
  373      16     12655     12639   135.486       152    0.169618    0.471982
  374      16     12695     12679   135.552       160    0.398727    0.471771
  375      16     12721     12705   135.467       104    0.416972    0.471844
  376      16     12762     12746   135.543       164    0.255434    0.471926
  377      16     12798     12782   135.565       144    0.481763    0.471841
  378      16     12830     12814   135.545       128    0.377422    0.471762
  379      16     12872     12856    135.63       168    0.270653    0.471591
2020-03-09 13:53:33.881853 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.471633
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12895     12879   135.516        92      0.6754    0.471633
  381      16     12932     12916   135.548       148    0.365135    0.471846
  382      16     12968     12952   135.569       144    0.476505    0.471797
  383      16     13008     12992   135.633       160    0.514332    0.471609
  384      16     13044     13028   135.654       144    0.252104    0.471462
  385      16     13084     13068   135.717       160    0.181763    0.471242
  386      16     13113     13097   135.666       116    0.188631     0.47131
  387      16     13150     13134   135.698       148    0.500839    0.471274
  388      16     13185     13169   135.709       140    0.144931    0.471286
  389      16     13221     13205    135.73       144    0.606974    0.471286
  390      16     13260     13244   135.782       156    0.414916    0.470931
  391      16     13298     13282   135.823       152    0.403291    0.470906
  392      16     13322     13306    135.72        96     0.32187     0.47124
  393      16     13358     13342   135.741       144     0.53494    0.471113
  394      16     13390     13374   135.722       128    0.343039     0.47122
  395      16     13424     13408   135.722       136    0.366867    0.471182
  396      16     13456     13440   135.702       128    0.307662    0.471252
  397      16     13484     13468   135.643       112    0.136399    0.471323
  398      16     13527     13511   135.734       172    0.412355    0.471182
  399      16     13560     13544   135.724       132    0.444682    0.471163
2020-03-09 13:53:53.895854 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.47118
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     13592     13576   135.705       128    0.209861     0.47118
  401      16     13633     13617   135.776       164    0.285086    0.471092
  402      16     13665     13649   135.756       128    0.118248     0.47104
  403      16     13702     13686   135.786       148    0.717739    0.471018
  404      16     13732     13716   135.747       120    0.510652    0.471057
  405      16     13768     13752   135.768       144    0.154975    0.471053
  406      16     13806     13790   135.807       152    0.456511    0.470924
  407      16     13842     13826   135.828       144    0.230021    0.470898
  408      16     13878     13862   135.847       144    0.390341     0.47073
  409      16     13912     13896   135.848       136    0.223302    0.470697
  410      15     13945     13930   135.848       136    0.332684    0.470824
  411      16     13983     13967   135.877       148    0.492018    0.470817
  412      16     14016     14000   135.868       132    0.561557    0.470724
  413      16     14049     14033   135.859       132    0.269506    0.470821
  414      16     14081     14065   135.839       128    0.353223    0.470723
  415      16     14104     14088   135.734        92     0.83534    0.470963
  416      16     14135     14119   135.705       124    0.540999    0.471235
  417      16     14174     14158   135.753       156    0.760297    0.471222
  418      16     14209     14193   135.764       140    0.265108    0.471054
  419      16     14244     14228   135.774       140    0.322467    0.471034
2020-03-09 13:54:13.902195 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.470908
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     14282     14266   135.812       152    0.312589    0.470908
  421      16     14324     14308   135.889       168    0.477891    0.470726
  422      16     14357     14341   135.879       132    0.893055    0.470598
  423      16     14393     14377   135.898       144    0.284082    0.470633
  424      16     14433     14417   135.955       160    0.158381    0.470417
  425      16     14466     14450   135.946       132     0.41506    0.470367
  426      16     14498     14482   135.927       128    0.263931    0.470394
  427      16     14533     14517   135.936       140    0.201927    0.470579
  428      16     14573     14557   135.992       160    0.322358    0.470369
  429      16     14615     14599   136.067       168    0.394001    0.470152
  430      16     14648     14632   136.057       132     0.91336    0.470056
  431      16     14690     14674   136.131       168    0.461153    0.469947
  432      16     14717     14701   136.066       108    0.277285    0.469965
  433      16     14747     14731   136.029       120    0.659027    0.470118
  434      16     14783     14767   136.047       144    0.109223    0.470067
  435      16     14821     14805   136.084       152    0.280136    0.470037
  436      16     14856     14840   136.093       140    0.436811    0.469963
  437      16     14894     14878   136.129       152    0.223475    0.469821
  438      16     14931     14915   136.156       148    0.511638    0.469668
  439      16     14975     14959   136.246       176    0.157312    0.469545
2020-03-09 13:54:33.911097 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.469359
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     15008     14992   136.236       132    0.238473    0.469359
  441      16     15044     15028   136.254       144    0.431187    0.469515
  442      16     15083     15067   136.298       156     0.19367     0.46924
  443      16     15114     15098    136.27       124    0.533306     0.46928
  444      16     15157     15141   136.351       172    0.520169    0.469149
  445      16     15192     15176   136.359       140    0.285858    0.469029
  446      16     15225     15209   136.349       132    0.413597    0.469004
  447      16     15255     15239   136.312       120    0.297951    0.469232
  448      16     15288     15272   136.303       132    0.378513    0.469229
  449      16     15324     15308    136.32       144    0.127919    0.469246
  450      16     15356     15340   136.301       128    0.765494    0.469174
  451      16     15399     15383    136.38       172    0.118877    0.468965
  452      16     15423     15407   136.291        96     1.17101    0.469233
  453      16     15463     15447   136.342       160    0.270425    0.469159
  454      16     15500     15484   136.368       148    0.722035    0.468955
  455      16     15539     15523   136.411       156    0.376699    0.468883
  456      16     15577     15561   136.445       152    0.266948    0.468724
  457      16     15604     15588   136.383       108    0.362744    0.468817
  458      16     15635     15619   136.356       124    0.400404    0.469087
  459      16     15675     15659   136.407       160    0.219797    0.468832
2020-03-09 13:54:53.919461 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.468942
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     15705     15689   136.371       120    0.517915    0.468942
  461      16     15751     15735   136.474       184    0.314449    0.468747
  462      16     15785     15769   136.473       136    0.254378    0.468726
  463      16     15821     15805   136.489       144    0.554271    0.468577
  464      16     15862     15846   136.548       164    0.486457    0.468442
  465      16     15887     15871   136.469       100    0.871659    0.468612
  466      16     15916     15900   136.424       116    0.593023    0.468817
  467      16     15953     15937   136.449       148    0.201829    0.468741
  468      16     15993     15977   136.498       160    0.324658    0.468577
  469      16     16033     16017   136.548       160    0.112061    0.468452
  470      16     16067     16051   136.547       136    0.311924    0.468319
  471      16     16094     16078   136.486       108     1.21297     0.46868
  472      16     16131     16115   136.511       148    0.105206    0.468558
  473      16     16170     16154   136.552       156    0.245129    0.468456
  474      16     16202     16186   136.534       128     0.40674    0.468355
  475      16     16233     16217   136.507       124     0.18983     0.46859
  476      16     16265     16249   136.489       128     0.32931     0.46855
  477      16     16299     16283   136.488       136    0.319409    0.468624
  478      16     16345     16329   136.588       184    0.352787    0.468407
  479      16     16372     16356   136.528       108     1.16864    0.468421
2020-03-09 13:55:13.932902 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.468281
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     16414     16398   136.593       168    0.713134    0.468281
  481      16     16450     16434   136.609       144    0.380134    0.468194
  482      16     16486     16470   136.624       144    0.506203      0.4682
  483      16     16521     16505   136.631       140    0.202173    0.468043
  484      16     16560     16544   136.671       156    0.094533    0.468021
  485      16     16591     16575   136.645       124    0.771024    0.467972
  486      16     16630     16614   136.684       156    0.238991    0.467953
  487      16     16669     16653   136.722       156    0.314566    0.467874
  488      16     16703     16687    136.72       136      0.5175    0.467886
  489      16     16738     16722   136.727       140    0.358545    0.467782
  490      16     16778     16762   136.775       160    0.301999    0.467673
  491      16     16812     16796   136.773       136    0.438576    0.467625
  492      16     16855     16839   136.844       172    0.315865     0.46749
  493      16     16895     16879   136.891       160    0.155628    0.467266
  494      16     16932     16916   136.914       148    0.349489    0.467335
  495      16     16962     16946    136.88       120    0.530849    0.467282
  496      16     16996     16980   136.878       136    0.951513    0.467268
  497      16     17034     17018   136.908       152    0.438845    0.467129
  498      16     17078     17062   136.987       176    0.113521    0.467034
  499      16     17117     17101   137.025       156    0.929013    0.466866
2020-03-09 13:55:33.943960 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466914
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     17145     17129   136.974       112    0.551434    0.466914
  501      16     17176     17160   136.948       124    0.220509    0.466946
  502      16     17205     17189   136.907       116    0.708215    0.467283
  503      16     17239     17223   136.905       136    0.223902    0.467277
  504      16     17275     17259   136.919       144    0.149271    0.467175
  505      16     17308     17292   136.909       132    0.527162    0.467279
  506      16     17343     17327   136.915       140    0.457346    0.467162
  507      16     17377     17361   136.913       136    0.781563    0.467248
  508      16     17416     17400    136.95       156    0.295068    0.467058
  509      16     17443     17427   136.893       108    0.207526    0.467181
  510      16     17478     17462   136.899       140     0.39138    0.467222
  511      16     17518     17502   136.945       160    0.352586    0.467175
  512      16     17547     17531   136.904       116    0.260512      0.4673
  513      16     17589     17573   136.964       168    0.390446    0.467094
  514      16     17622     17606   136.954       132    0.160975    0.467091
  515      16     17657     17641    136.96       140    0.243762    0.467045
  516      16     17692     17676   136.966       140    0.445149    0.467003
  517      16     17723     17707   136.941       124    0.373801     0.46705
  518      16     17760     17744   136.962       148    0.163027    0.466981
  519      16     17798     17782   136.991       152     0.24347    0.466915
2020-03-09 13:55:53.951850 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466837
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     17837     17821   137.027       156    0.433931    0.466837
  521      16     17864     17848    136.97       108    0.160606    0.466883
  522      16     17906     17890    137.03       168    0.361994    0.466879
  523      16     17942     17926   137.043       144    0.541809    0.466788
  524      16     17974     17958   137.026       128    0.687222    0.466877
  525      16     18006     17990   137.008       128    0.187099    0.466867
  526      16     18033     18017   136.953       108    0.605651    0.466957
  527      16     18061     18045   136.905       112    0.162039    0.467175
  528      16     18093     18077   136.889       128    0.529875    0.467288
  529      16     18128     18112   136.894       140    0.280982    0.467226
  530      16     18166     18150   136.923       152    0.485335     0.46728
  531      16     18201     18185   136.929       140    0.486643    0.467076
  532      16     18243     18227   136.987       168    0.110459    0.466997
  533      16     18267     18251    136.91        96    0.697684     0.46696
  534      16     18309     18293   136.967       168    0.569496    0.467099
  535      16     18345     18329    136.98       144    0.274834    0.466923
  536      16     18378     18362   136.971       132    0.618873     0.46704
  537      16     18411     18395   136.961       132    0.178024    0.466999
  538      16     18452     18436   137.012       164    0.334936    0.466864
  539      16     18483     18467   136.987       124    0.362084    0.466863
2020-03-09 13:56:13.965891 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.466925
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     18516     18500   136.978       132    0.141543    0.466925
  541      16     18551     18535   136.984       140    0.194767    0.466926
  542      16     18583     18567   136.967       128     0.30533    0.466911
  543      16     18617     18601   136.965       136    0.242655     0.46699
  544      16     18651     18635   136.963       136    0.543032    0.466962
  545      16     18681     18665   136.932       120    0.369876    0.467076
  546      16     18714     18698   136.923       132    0.757877    0.467201
  547      16     18742     18726   136.878       112    0.623998    0.467199
  548      16     18778     18762    136.89       144    0.598083    0.467294
  549      16     18822     18806   136.962       176    0.317329    0.467033
  550      16     18861     18845   136.996       156     0.30958    0.466969
  551      16     18900     18884    137.03       156    0.107902     0.46677
  552      16     18929     18913   136.992       116    0.248413    0.466878
  553      16     18972     18956   137.055       172    0.213509    0.466795
  554      16     19008     18992   137.068       144    0.809786     0.46673
  555      16     19042     19026   137.066       136    0.562657    0.466716
  556      16     19074     19058   137.049       128    0.665218     0.46665
  557      16     19114     19098   137.091       160    0.205075    0.466655
  558      16     19151     19135    137.11       148    0.288084    0.466515
  559      16     19181     19165   137.079       120    0.221551    0.466553
2020-03-09 13:56:33.971756 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.46663
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     19220     19204   137.113       156    0.544361     0.46663
  561      16     19263     19247   137.175       172     0.15661    0.466293
  562      16     19297     19281   137.173       136    0.586363    0.466316
  563      16     19332     19316   137.178       140    0.265647    0.466271
  564      16     19370     19354   137.204       152    0.106856    0.466203
  565      16     19404     19388   137.202       136     0.68798    0.466209
  566      16     19448     19432    137.27       176    0.591428     0.46602
  567      16     19478     19462   137.239       120     0.60812    0.466126
  568      16     19517     19501   137.272       156    0.390487    0.466033
  569      16     19556     19540   137.305       156    0.447418    0.465876
  570      16     19594     19578   137.331       152    0.761197    0.465827
  571      16     19635     19619   137.378       164    0.388597    0.465695
  572      16     19655     19639   137.277        80    0.611985    0.465841
  573      16     19697     19681   137.331       168    0.599114    0.465891
  574      16     19721     19705   137.259        96    0.489929    0.465954
  575      16     19763     19747   137.312       168    0.306763    0.465967
  576      16     19799     19783   137.324       144    0.634426    0.465853
  577      16     19840     19824   137.369       164    0.237302    0.465778
  578      16     19876     19860   137.381       144    0.979485    0.465645
  579      16     19910     19894   137.378       136    0.593116     0.46572
2020-03-09 13:56:53.981010 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.465694
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     19945     19929   137.383       140    0.264241    0.465694
  581      16     19982     19966   137.401       148    0.674185    0.465603
  582      16     20019     20003   137.419       148    0.529329    0.465548
  583      16     20051     20035   137.403       128    0.268896    0.465571
  584      16     20093     20077   137.455       168    0.538054    0.465439
  585      16     20127     20111   137.452       136     0.78359    0.465388
  586      16     20169     20153   137.504       168    0.195777    0.465281
  587      16     20200     20184   137.481       124    0.266851    0.465221
  588      16     20238     20222   137.506       152    0.398561    0.465175
  589      16     20271     20255   137.497       132    0.495558    0.465266
  590      16     20309     20293   137.521       152     0.58999    0.465193
  591      16     20340     20324   137.498       124    0.437875    0.465217
  592      16     20385     20369    137.57       180    0.341231    0.465085
  593      16     20413     20397   137.527       112     1.07334    0.465053
  594      16     20453     20437   137.565       160    0.498946     0.46508
  595      16     20487     20471   137.562       136    0.489279    0.465053
  596      16     20523     20507   137.572       144    0.249381    0.464951
  597      16     20550     20534   137.523       108    0.679844    0.465051
  598      16     20586     20570   137.534       144    0.621874    0.465158
  599      16     20621     20605   137.537       140    0.366411      0.4651
2020-03-09 13:57:13.991680 min lat: 0.0712215 max lat: 1.78416 avg lat: 0.465053
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      15     20649     20634   137.501       116    0.477456    0.465053
Total time run:         600.444
Total writes made:      20649
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     137.558
Stddev Bandwidth:       19.6916
Max bandwidth (MB/sec): 184
Min bandwidth (MB/sec): 68
Average IOPS:           34
Stddev IOPS:            4.9229
Max IOPS:               46
Min IOPS:               17
Average Latency(s):     0.465239
Stddev Latency(s):      0.23455
Max latency(s):         1.78416
Min latency(s):         0.0712215

Total time run:         600.444
Total writes made:      20649
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     137.558
Stddev Bandwidth:       19.6916
Max bandwidth (MB/sec): 184
Min bandwidth (MB/sec): 68
Average IOPS:           34
Stddev IOPS:            4.9229
Max IOPS:               46
Min IOPS:               17
Average Latency(s):     0.465239
Stddev Latency(s):      0.23455
Max latency(s):         1.78416
Min latency(s):         0.0712215

Sequential Read Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 seq | tee --append $(date +%F)-rados-test-read-seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        47        31   122.867       124    0.892219    0.299688
    2      15        79        64   127.402       132    0.272987    0.408774
    3      16       109        93   123.607       116    0.445915    0.442824
    4      16       139       123   122.704       120   0.0606516    0.466495
    5      16       169       153    122.16       120    0.890098    0.477531
    6      16       201       185    123.12       128    0.197168    0.478944
    7      15       231       216   123.223       124     0.27612    0.473009
    8      16       266       250   124.815       136     0.16127    0.477635
    9      16       297       281   124.722       124     1.67437    0.474589
   10      16       330       314   125.446       132    0.307985    0.473194
   11      16       362       346   125.657       128    0.416032    0.482695
   12      16       393       377   125.511       124    0.292082    0.482124
   13      16       426       410   126.003       132   0.0735909    0.481627
   14      16       457       441   125.858       124    0.101919    0.483594
   15      16       486       470   125.183       116   0.0970761    0.487313
   16      16       517       501   125.091       124    0.148924     0.49301
   17      16       547       531   124.791       120     1.38568    0.492822
   18      16       578       562   124.746       124    0.266846    0.491729
   19      16       613       597   125.547       140     1.44537    0.491227
2020-03-09 14:14:15.531643 min lat: 0.0209854 max lat: 2.10739 avg lat: 0.491662
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       643       627   125.269       120     1.62711    0.491662
   21      16       672       656   124.828       116   0.0461491    0.495234
   22      16       701       685   124.426       116    0.154277    0.496283
   23      16       733       717   124.576       128    0.229963    0.494833
   24      16       765       749   124.718       128    0.125264    0.497316
   25      16       798       782   125.008       132    0.160512    0.495727
   26      16       832       816    125.43       136    0.331566    0.492769
   27      16       868       852   126.117       144   0.0495642    0.490482
   28      16       901       885   126.326       132   0.0995234    0.495103
   29      16       928       912   125.694       108     1.47158     0.49533
   30      16       961       945   125.903       132    0.103094    0.495363
   31      16       991       975   125.712       120    0.145827    0.496937
   32      16      1022      1006   125.658       124      0.2493     0.49566
   33      16      1052      1036   125.486       120      1.0992     0.49861
   34      16      1082      1066   125.325       120    0.196652    0.497979
   35      16      1120      1104   126.081       152   0.0441845    0.495837
   36      15      1149      1134   125.904       120     1.17875    0.497344
   37      16      1179      1163   125.634       116    0.307539    0.499983
   38      16      1212      1196   125.756       132     1.46409    0.497412
   39      16      1253      1237   126.735       164   0.0898571    0.494788
2020-03-09 14:14:35.553049 min lat: 0.0209854 max lat: 2.10739 avg lat: 0.495789
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1284      1268   126.666       124    0.280382    0.495789
   41      16      1316      1300   126.698       128     1.44724    0.494111
   42      16      1353      1337   127.204       148     1.58673    0.494859
   43      16      1385      1369   127.223       128    0.735345    0.494403
   44      16      1419      1403   127.422       136     0.30188    0.492974
   45      16      1450      1434   127.345       124   0.0577048    0.494533
   46      16      1487      1471   127.793       148     1.29992    0.492074
   47      16      1518      1502   127.712       124     1.28863    0.491668
   48      16      1550      1534   127.718       128    0.632717    0.492283
   49      16      1582      1566   127.723       128    0.338998    0.491939
   50      16      1615      1599   127.808       132     1.09127    0.492657
   51      16      1646      1630   127.733       124    0.275021     0.49258
   52      16      1681      1665   127.967       140     1.54424    0.490845
   53      16      1717      1701   128.269       144    0.206363    0.490046
   54      15      1747      1732   128.186       124    0.232308    0.490402
   55      16      1780      1764   128.182       128     1.38793    0.490511
   56      16      1813      1797    128.25       132    0.314296    0.490262
   57      16      1844      1828   128.175       124    0.360981    0.490292
   58      16      1871      1855   127.827       108    0.414088    0.490588
   59      15      1903      1888   127.893       132    0.376294    0.492249
2020-03-09 14:14:55.560281 min lat: 0.0204813 max lat: 2.10739 avg lat: 0.4939
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1930      1914   127.495       104   0.0389062      0.4939
   61      16      1963      1947   127.568       132     0.39855    0.493329
   62      16      1995      1979   127.574       128    0.172692    0.493467
   63      16      2028      2012   127.644       132   0.0292764    0.492148
   64      15      2062      2047   127.835       140    0.230653    0.491703
   65      16      2092      2076   127.653       116    0.369797    0.492676
   66      16      2127      2111   127.838       140     1.51087     0.49297
   67      16      2158      2142    127.78       124    0.263258    0.492571
   68      15      2187      2172   127.662       120   0.0548647      0.4928
   69      16      2222      2206   127.781       136   0.0629204    0.492871
   70      16      2255      2239   127.837       132   0.0686457    0.492765
   71      16      2290      2274   128.008       140    0.306493    0.491341
   72      16      2320      2304   127.897       120     1.90779    0.492541
   73      16      2350      2334   127.788       120    0.271317    0.493013
   74      16      2381      2365   127.736       124    0.222788    0.493108
   75      16      2415      2399   127.845       136   0.0265713    0.492665
   76      16      2445      2429   127.741       120    0.212303    0.493418
   77      16      2482      2466   128.003       148    0.097387    0.492576
   78      16      2518      2502   128.207       144     1.34222    0.492447
   79      16      2545      2529   127.951       108     1.61514    0.493918
2020-03-09 14:15:15.572748 min lat: 0.0203744 max lat: 2.1132 avg lat: 0.493859
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      16      2577      2561   127.951       128     1.12898    0.493859
   81      16      2612      2596   128.099       140     1.28265    0.492462
   82      16      2643      2627   128.047       124    0.204044    0.492216
   83      16      2681      2665   128.335       152   0.0690425    0.490737
   84      16      2713      2697   128.328       128    0.889824    0.491834
   85      16      2744      2728   128.277       124    0.508321    0.492136
   86      16      2775      2759   128.227       124     1.16227    0.492862
   87      16      2806      2790   128.178       124       1.233    0.492743
   88      16      2841      2825    128.31       140     0.12701    0.491969
   89      16      2875      2859   128.397       136    0.243668    0.491223
   90      16      2906      2890   128.348       124    0.297884    0.491903
   91      16      2935      2919   128.212       116     0.28308    0.492077
   92      16      2969      2953   128.296       136     1.70304    0.491982
   93      16      3003      2987   128.379       136    0.148716    0.491434
   94      16      3038      3022   128.502       140   0.0211009    0.490931
   95      16      3067      3051   128.371       116    0.269733    0.491387
   96      16      3099      3083   128.366       128    0.201617     0.49092
   97      16      3131      3115   128.363       128    0.342916    0.491726
   98      16      3161      3145   128.277       120     1.42997    0.492707
   99      16      3196      3180   128.395       140   0.0253981    0.491718
2020-03-09 14:15:35.580023 min lat: 0.0203744 max lat: 2.15539 avg lat: 0.492147
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3227      3211   128.351       124   0.0467987    0.492147
  101      16      3263      3247   128.506       144    0.164445     0.49198
  102      16      3292      3276   128.383       116     1.48138    0.491628
  103      16      3332      3316    128.69       160   0.0201424    0.490751
  104      16      3361      3345   128.568       116    0.314573    0.491364
  105      16      3395      3379   128.638       136   0.0257904    0.490323
  106      16      3429      3413   128.706       136   0.0459659    0.490438
  107      16      3462      3446   128.736       132     1.79781    0.490499
  108      16      3495      3479   128.765       132   0.0469213    0.490293
  109      16      3530      3514   128.867       140     1.33188    0.490484
  110      16      3563      3547   128.883       132    0.257433    0.490693
  111      16      3592      3576   128.766       116     0.12864    0.490938
  112      16      3626      3610   128.829       136    0.215556    0.490451
  113      16      3660      3644   128.892       136   0.0201329     0.49001
  114      16      3695      3679    128.99       140     1.55332    0.489797
  115      16      3724      3708   128.877       116    0.229602    0.490283
  116      16      3753      3737   128.766       116    0.166641    0.490963
  117      16      3785      3769   128.759       128    0.199312    0.491129
  118      16      3815      3799   128.685       120     0.18168    0.491448
  119      16      3846      3830   128.645       124      1.5559     0.49102
2020-03-09 14:15:55.620068 min lat: 0.0200594 max lat: 2.15539 avg lat: 0.490487
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3883      3867   128.783       148     1.90927    0.490487
  121      16      3913      3897    128.71       120    0.205211    0.490891
  122      16      3947      3931   128.769       136    0.302697    0.490809
  123      16      3978      3962   128.728       124      1.1433    0.491541
  124      16      4010      3994   128.722       128    0.180435    0.491076
  125      16      4046      4030   128.843       144     1.20192    0.491044
  126      16      4075      4059    128.74       116     1.31125    0.491106
  127      16      4107      4091   128.734       128    0.050824    0.491015
  128      16      4138      4122   128.696       124   0.0350849    0.491038
  129      16      4172      4156   128.751       136     1.81892    0.490808
  130      16      4202      4186   128.683       120     1.25662    0.491693
  131      16      4233      4217   128.644       124     0.20783     0.49147
  132      16      4264      4248   128.598       124     1.28707    0.492114
  133      16      4302      4286   128.773       152     0.19773    0.491398
  134      16      4335      4319   128.797       132   0.0550823    0.490833
  135      16      4371      4355    128.91       144     1.50193    0.490568
  136      16      4407      4391    129.02       144   0.0649788     0.49009
  137      16      4440      4424   129.042       132   0.0972908    0.490044
  138      16      4474      4458   129.092       136     1.45655    0.490096
  139      16      4509      4493    129.17       140   0.0410438    0.489488
2020-03-09 14:16:15.644998 min lat: 0.0200542 max lat: 2.29396 avg lat: 0.489154
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4542      4526    129.19       132     0.26454    0.489154
  141      15      4573      4558   129.181       128   0.0215077    0.489651
  142      16      4604      4588   129.117       120    0.186569    0.489814
  143      16      4633      4617   129.025       116     1.56875    0.490359
  144      16      4665      4649   129.018       128   0.0422559    0.490218
  145      15      4697      4682   129.038       132    0.222384    0.490184
  146      16      4724      4708   128.866       104      1.6473    0.490947
  147      16      4759      4743   128.942       140   0.0264929    0.490668
  148      16      4790      4774   128.908       124    0.240668    0.490269
  149      16      4824      4808   128.955       136   0.0527974    0.490019
  150      16      4856      4840   128.949       128    0.313878    0.490551
  151      15      4886      4871   128.913       124     0.42731    0.490692
  152      16      4920      4904   128.929       132    0.163081    0.491003
  153      16      4954      4938   128.975       136     1.49151    0.490549
  154      16      4986      4970   128.969       128    0.267954    0.490443
  155      16      5020      5004   129.014       136     1.67769    0.490629
  156      16      5058      5042   129.161       152   0.0454149    0.489958
  157      16      5091      5075   129.179       132     1.41498    0.490013
  158      16      5125      5109   129.222       136   0.0354356    0.489436
  159      15      5164      5149   129.414       160   0.0780695    0.489058
2020-03-09 14:16:35.658777 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.488735
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5201      5185   129.505       144    0.321195    0.488735
  161      16      5238      5222    129.62       148     0.03819    0.488391
  162      16      5272      5256   129.659       136     0.33829    0.487809
  163      16      5303      5287   129.624       124    0.254569    0.487856
  164      16      5331      5315   129.517       112    0.254206    0.488152
  165      16      5358      5342   129.386       108    0.202445    0.489093
  166      16      5387      5371   129.306       116      1.3374    0.489476
  167      16      5420      5404   129.322       132    0.308261    0.489442
  168      16      5454      5438   129.361       136    0.101547    0.489295
  169      16      5488      5472     129.4       136   0.0542278    0.489301
  170      16      5519      5503   129.369       124     1.62126    0.489662
  171      16      5557      5541   129.501       152   0.0634993    0.488651
  172      16      5593      5577   129.583       144     1.84887    0.488236
  173      16      5625      5609   129.573       128    0.489666    0.488547
  174      16      5654      5638   129.494       116     0.30284    0.488526
  175      16      5682      5666   129.394       112     1.76192    0.489122
  176      15      5713      5698   129.384       128    0.541846    0.488987
  177      16      5749      5733   129.444       140     1.95049    0.488991
  178      16      5776      5760   129.324       108    0.223297    0.489616
  179      16      5808      5792   129.316       128   0.0284227    0.489446
2020-03-09 14:16:55.668785 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.489424
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5841      5825   129.331       132   0.0395301    0.489424
  181      16      5870      5854   129.256       116     1.85892    0.489997
  182      15      5903      5888   129.285       136   0.0311012    0.490041
  183      16      5939      5923   129.343       140   0.0429795    0.489662
  184      16      5966      5950   129.227       108    0.289396    0.489738
  185      16      5996      5980   129.177       120    0.257256     0.49011
  186      16      6027      6011   129.149       124     1.47025    0.490253
  187      15      6059      6044   129.164       132    0.222182    0.489788
  188      16      6091      6075   129.136       124   0.0282552    0.489668
  189      16      6119      6103   129.046       112     0.10614    0.490754
  190      16      6155      6139   129.124       144     1.53179    0.490333
  191      16      6185      6169   129.076       120    0.281941    0.490207
  192      16      6212      6196   128.966       108    0.021823    0.490802
  193      16      6246      6230   129.002       136    0.356081    0.490863
  194      16      6279      6263   129.014       132    0.553909    0.491168
  195      15      6313      6298    129.07       140    0.292374    0.490949
  196      16      6347      6331   129.085       132    0.212525    0.490656
  197      16      6381      6365    129.12       136    0.287295    0.490472
  198      16      6408      6392   129.013       108    0.223018    0.490753
  199      16      6436      6420   128.928       112    0.240123    0.491064
2020-03-09 14:17:15.692483 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491105
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6469      6453   128.943       132    0.118565    0.491105
  201      16      6501      6485   128.938       128     1.22528    0.491256
  202      16      6531      6515   128.893       120    0.337655    0.491108
  203      16      6566      6550   128.948       140     1.63896    0.491307
  204      16      6599      6583   128.963       132    0.076407    0.491135
  205      16      6634      6618   129.016       140   0.0440429    0.490954
  206      16      6668      6652    129.05       136    0.229665    0.490673
  207      16      6702      6686   129.083       136    0.146742    0.490531
  208      16      6735      6719   129.097       132    0.244986    0.490723
  209      16      6763      6747   129.015       112    0.314523    0.490741
  210      16      6795      6779   128.995       128   0.0547352    0.490936
  211      15      6829      6814   129.046       140    0.142437    0.490677
  212      16      6860      6844   129.003       120    0.220339    0.490963
  213      16      6892      6876   128.998       128     1.49846    0.490942
  214      15      6923      6908   128.993       128     1.37875    0.491289
  215      16      6953      6937   128.932       116    0.440148    0.491463
  216      16      6991      6975   129.039       152   0.0512537    0.490873
  217      16      7025      7009    129.07       136    0.425699    0.490622
  218      16      7058      7042   129.083       132    0.320678    0.490707
  219      16      7090      7074   129.078       128    0.332081    0.490646
2020-03-09 14:17:35.726780 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.490969
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7119      7103   129.019       116    0.187146    0.490969
  221      16      7153      7137    129.05       136     1.31869    0.491192
  222      16      7182      7166   128.991       116     1.09072    0.491164
  223      16      7213      7197   128.969       124    0.293644    0.491235
  224      16      7244      7228   128.946       124   0.0474558    0.491304
  225      16      7275      7259   128.924       124     1.72382    0.491409
  226      16      7310      7294    128.97       140    0.293927    0.491241
  227      16      7342      7326   128.966       128    0.475732    0.491443
  228      16      7371      7355   128.909       116    0.234139    0.491569
  229      16      7404      7388   128.922       132    0.387129    0.491578
  230      16      7436      7420   128.918       128     1.29579    0.491705
  231      16      7469      7453   128.931       132   0.0396863     0.49128
  232      16      7500      7484    128.91       124    0.347787    0.491256
  233      16      7532      7516   128.905       128      1.7789    0.491281
  234      16      7569      7553   128.986       148     0.27894    0.491244
  235      16      7598      7582   128.931       116    0.175157     0.49163
  236      16      7628      7612   128.893       120    0.332206    0.491476
  237      16      7662      7646   128.923       136   0.0741658    0.491593
  238      16      7695      7679   128.935       132    0.237976    0.491317
  239      16      7727      7711   128.931       128    0.291509    0.491187
2020-03-09 14:17:55.739590 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491305
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7760      7744   128.944       132    0.177061    0.491305
  241      16      7794      7778   128.973       136   0.0300137    0.491321
  242      16      7826      7810   128.969       128    0.196213     0.49129
  243      16      7854      7838   128.899       112     1.56495    0.491454
  244      16      7887      7871   128.912       132    0.219027     0.49148
  245      16      7919      7903   128.908       128    0.199531     0.49131
  246      16      7951      7935   128.904       128   0.0337205    0.491566
  247      16      7989      7973   128.997       152      1.6093    0.491123
  248      16      8022      8006   129.009       132     0.18837    0.490774
  249      16      8058      8042   129.069       144    0.275771    0.490829
  250      16      8087      8071   129.017       116   0.0723554    0.491164
  251      16      8120      8104   129.028       132   0.0524736    0.491132
  252      16      8156      8140   129.088       144    0.150136    0.491018
  253      16      8185      8169   129.036       116    0.156316    0.490976
  254      16      8220      8204   129.079       140    0.177453    0.491009
  255      16      8253      8237    129.09       132    0.313919    0.490764
  256      16      8287      8271   129.117       136     1.60619    0.490697
  257      16      8317      8301   129.081       120    0.163976    0.490834
  258      16      8349      8333   129.077       128     1.58042    0.490968
  259      16      8382      8366   129.088       132     1.49256    0.490914
2020-03-09 14:18:15.745674 min lat: 0.0198839 max lat: 2.29396 avg lat: 0.491011
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      15      8411      8396   129.053       120     1.49597    0.491011
  261      16      8445      8429   129.063       132    0.311964    0.490963
  262      16      8474      8458   129.013       116     1.77077    0.491087
  263      16      8511      8495   129.085       148   0.0429818    0.490615
  264      16      8546      8530   129.126       140    0.197586    0.490597
  265      16      8581      8565   129.167       140     1.16235    0.490927
  266      16      8611      8595   129.132       120     0.26186    0.490821
  267      15      8644      8629   129.158       136     1.02237    0.491053
  268      16      8672      8656   129.079       108    0.350566    0.491047
  269      16      8703      8687    129.06       124   0.0451803    0.490987
  270      16      8737      8721   129.085       136   0.0226707    0.490842
  271      16      8766      8750   129.037       116    0.062004    0.491211
  272      16      8795      8779   128.989       116     1.57246    0.491425
  273      16      8826      8810    128.97       124    0.228278    0.491391
  274      16      8853      8837   128.893       108   0.0687488     0.49166
  275      16      8888      8872   128.933       140   0.0455489    0.491556
  276      16      8919      8903   128.915       124    0.249098    0.491537
  277      16      8951      8935   128.912       128    0.258679    0.491789
  278      16      8982      8966   128.894       124    0.980672    0.491932
  279      16      9010      8994   128.834       112    0.229116    0.492083
2020-03-09 14:18:35.769350 min lat: 0.0198839 max lat: 2.35194 avg lat: 0.491882
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9041      9025    128.81       124    0.386641    0.491882
  281      16      9077      9061   128.864       144     1.47453    0.491981
  282      16      9113      9097   128.917       144    0.339502    0.491438
  283      16      9145      9129   128.913       128   0.0959521    0.491708
  284      16      9178      9162   128.924       132    0.224604    0.491489
  285      16      9210      9194    128.92       128     1.20296     0.49182
  286      16      9247      9231   128.987       148     0.32857     0.49156
  287      16      9280      9264   128.997       132    0.199186     0.49137
  288      16      9311      9295    128.98       124     1.17839     0.49138
  289      16      9345      9329   129.003       136    0.862595    0.491521
  290      16      9379      9363   129.027       136    0.300099    0.491356
  291      16      9412      9396   129.037       132    0.161756    0.491219
  292      16      9444      9428   129.032       128     0.13603    0.491136
  293      16      9476      9460   129.029       128     1.38082    0.491174
  294      16      9510      9494   129.052       136   0.0663548    0.491154
  295      15      9547      9532   129.129       152   0.0491553    0.491014
  296      16      9581      9565   129.139       132    0.352725     0.49085
  297      16      9610      9594   129.095       116    0.137058    0.490972
  298      16      9644      9628   129.118       136      0.1332    0.490998
  299      16      9674      9658   129.087       120     1.18995    0.491188
2020-03-09 14:18:55.781087 min lat: 0.0198839 max lat: 2.35194 avg lat: 0.491059
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9708      9692    129.11       136   0.0331784    0.491059
  301      16      9743      9727   129.146       140   0.0285308    0.491034
  302      16      9772      9756   129.103       116    0.222227    0.490774
  303      16      9804      9788   129.098       128     1.70105    0.491047
  304      16      9834      9818   129.068       120     1.44688    0.491285
  305      16      9865      9849   129.052       124     0.19288    0.491161
  306      16      9899      9883   129.074       136     1.53346    0.491208
  307      16      9933      9917   129.097       136      1.3147    0.491168
  308      16      9970      9954   129.158       148   0.0487696     0.49082
  309      16     10006      9990   129.206       144    0.201482    0.490437
  310      15     10036     10021   129.188       124    0.206617    0.490465
  311      16     10070     10054   129.197       132   0.0232912    0.490727
  312      16     10102     10086   129.193       128     1.45591    0.490649
  313      16     10134     10118    129.19       128      0.2205    0.490793
  314      16     10162     10146   129.135       112     1.28262    0.491086
  315      16     10192     10176   129.106       120    0.408311    0.491056
  316      16     10226     10210   129.127       136    0.239141     0.49082
  317      16     10257     10241   129.111       124     0.21391    0.490973
  318      16     10286     10270    129.07       116    0.378004    0.491026
  319      16     10318     10302   129.066       128    0.241126    0.491264
2020-03-09 14:19:15.790044 min lat: 0.019786 max lat: 2.35194 avg lat: 0.491518
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      16     10346     10330   129.012       112    0.234121    0.491518
  321      16     10381     10365   129.046       140    0.153162    0.491359
  322      16     10410     10394   129.006       116    0.177621    0.491353
  323      16     10446     10430   129.041       144   0.0287919    0.491359
  324      16     10474     10458   128.988       112     1.60249    0.491507
  325      16     10506     10490   128.985       128     1.59277    0.491495
  326      16     10536     10520   128.957       120    0.206458    0.491606
  327      16     10572     10556   129.003       144     0.17232    0.491646
  328      16     10598     10582   128.927       104    0.217666    0.491672
  329      16     10628     10612   128.899       120   0.0714257    0.491657
  330      16     10663     10647   128.931       140    0.440115    0.491936
  331      16     10694     10678   128.915       124   0.0709322    0.491827
  332      16     10723     10707   128.876       116   0.0598779    0.491949
  333      15     10757     10742   128.909       140    0.236215    0.491691
  334      16     10790     10774   128.906       128    0.143116    0.491915
  335      16     10820     10804    128.88       120     1.48811    0.491884
  336      16     10853     10837   128.889       132    0.568778    0.491976
  337      16     10887     10871    128.91       136   0.0806111     0.49183
  338      16     10920     10904   128.919       132     1.89901    0.491806
  339      16     10950     10934   128.892       120    0.431437    0.491919
2020-03-09 14:19:35.832713 min lat: 0.019786 max lat: 2.35194 avg lat: 0.491975
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      16     10982     10966    128.89       128   0.0308958    0.491975
  341      16     11011     10995   128.851       116      1.6588    0.491907
  342      16     11045     11029   128.872       136    0.359903    0.492043
  343      16     11076     11060   128.858       124    0.142049    0.492059
  344      16     11106     11090   128.831       120    0.325222    0.492177
  345      16     11142     11126   128.875       144    0.401054     0.49203
  346      16     11173     11157   128.861       124   0.0712791    0.492122
  347      16     11201     11185   128.812       112   0.0359587    0.492192
  348      16     11232     11216   128.798       124    0.263129    0.492676
  349      16     11262     11246   128.772       120   0.0495504     0.49273
  350      16     11290     11274   128.724       112    0.436102    0.492964
  351      16     11325     11309   128.756       140    0.833385    0.492704
  352      16     11355     11339   128.731       120     1.01503    0.492859
  353      16     11384     11368   128.695       116   0.0685347    0.492875
  354      16     11414     11398    128.67       120    0.148765    0.492831
  355      16     11448     11432   128.691       136   0.0506394    0.492722
  356      16     11475     11459   128.633       108    0.074135    0.492964
  357      15     11509     11494   128.664       140    0.219638    0.492796
  358      16     11543     11527   128.673       132    0.995272    0.493022
  359      16     11573     11557   128.648       120   0.0206915    0.492867
2020-03-09 14:19:55.846824 min lat: 0.019786 max lat: 2.35194 avg lat: 0.492825
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11608     11592    128.68       140   0.0879656    0.492825
  361      16     11642     11626     128.7       136     1.44311    0.492758
  362      16     11672     11656   128.676       120    0.373551    0.492935
  363      16     11701     11685   128.641       116    0.267157    0.492848
  364      16     11737     11721   128.679       144    0.215786    0.492787
  365      16     11772     11756    128.71       140    0.264887     0.49256
  366      16     11801     11785   128.675       116     1.71657    0.492852
  367      16     11832     11816   128.663       124     1.44494    0.492947
  368      16     11863     11847    128.65       124     1.50337    0.492934
  369      16     11898     11882   128.681       140     1.58296    0.492794
  370      16     11928     11912   128.657       120     1.75785    0.492807
  371      16     11961     11945   128.666       132    0.178991    0.492926
  372      16     11996     11980   128.696       140   0.0613502    0.492963
  373      16     12029     12013   128.705       132    0.211431    0.492761
  374      15     12057     12042    128.67       116    0.186892    0.492973
  375      16     12094     12078   128.711       144    0.174732    0.492628
  376      15     12127     12112    128.73       136     1.28581     0.49271
  377      16     12158     12142   128.706       120     1.47145    0.492728
  378      16     12189     12173   128.694       124   0.0205903     0.49269
  379      16     12219     12203   128.671       120    0.069476    0.492771
2020-03-09 14:20:15.867479 min lat: 0.0197829 max lat: 2.35194 avg lat: 0.492937
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12250     12234   128.658       124   0.0326588    0.492937
  381      16     12285     12269   128.688       140    0.156159    0.492682
  382      16     12315     12299   128.665       120     1.22456    0.492986
  383      16     12344     12328   128.632       116    0.237924    0.493066
  384      16     12379     12363   128.662       140    0.101755    0.492823
  385      16     12413     12397   128.681       136    0.272619    0.492923
  386      16     12445     12429   128.679       128     1.14942    0.492995
  387      16     12476     12460   128.667       124    0.224306     0.49283
  388      16     12511     12495   128.696       140    0.267396    0.492779
  389      16     12549     12533   128.756       152    0.179397    0.492657
  390      16     12575     12559   128.692       104    0.219564    0.492839
  391      16     12605     12589    128.67       120    0.178358    0.492874
  392      16     12645     12629   128.749       160      1.3368    0.492667
  393      16     12677     12661   128.747       128      1.4419      0.4926
  394      16     12709     12693   128.744       128    0.385929    0.492788
  395      16     12741     12725   128.742       128    0.254265    0.492577
  396      16     12776     12760   128.764       140   0.0521672    0.492214
  397      16     12811     12795   128.792       140     2.39436    0.492282
  398      16     12847     12831   128.822       144    0.713194    0.492362
  399      16     12881     12865    128.84       136    0.376951    0.492356
2020-03-09 14:20:35.920429 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492544
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12909     12893   128.798       112     0.25237    0.492544
  401      16     12942     12926   128.806       132     0.31392    0.492272
  402      16     12969     12953   128.754       108    0.299852      0.4925
  403      16     13005     12989   128.792       144    0.313795    0.492555
  404      16     13036     13020    128.78       124    0.315625    0.492574
  405      16     13073     13057   128.827       148    0.416656    0.492385
  406      16     13101     13085   128.786       112    0.222998    0.492428
  407      16     13134     13118   128.794       132    0.244173    0.492491
  408      15     13162     13147   128.762       116   0.0969783    0.492565
  409      16     13201     13185   128.819       152   0.0720224    0.492474
  410      16     13235     13219   128.836       136     1.16168    0.492538
  411      16     13270     13254   128.863       140    0.036247    0.492327
  412      16     13300     13284   128.841       120    0.271222    0.492486
  413      16     13331     13315   128.829       124     1.11787    0.492434
  414      16     13358     13342   128.779       108    0.250923    0.492527
  415      16     13389     13373   128.767       124   0.0231837    0.492637
  416      15     13422     13407   128.784       136   0.0498244    0.492682
  417      16     13457     13441   128.801       136   0.0885117    0.492485
  418      16     13490     13474   128.808       132   0.0763648    0.492563
  419      16     13520     13504   128.787       120    0.212707    0.492394
2020-03-09 14:20:55.931522 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492491
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13556     13540   128.823       144   0.0419348    0.492491
  421      16     13587     13571   128.811       124    0.377949    0.492279
  422      16     13619     13603   128.809       128    0.195854    0.492553
  423      16     13646     13630    128.76       108   0.0835241    0.492499
  424      16     13680     13664   128.777       136     0.21761    0.492531
  425      16     13708     13692   128.737       112    0.244593    0.492776
  426      16     13736     13720   128.698       112    0.287423    0.492787
  427      16     13773     13757   128.743       148   0.0432397    0.492716
  428      16     13802     13786   128.713       116     1.64474    0.492827
  429      16     13835     13819   128.721       132    0.262212    0.492698
  430      16     13869     13853   128.737       136    0.203097    0.492685
  431      15     13897     13882   128.707       116   0.0520222    0.492917
  432      16     13931     13915   128.715       132    0.175716    0.492847
  433      16     13966     13950    128.74       140    0.285042    0.492689
  434      15     13995     13980    128.72       120    0.658221    0.492814
  435      15     14024     14009   128.691       116    0.178883    0.492958
  436      16     14057     14041   128.689       128    0.196827    0.492869
  437      16     14085     14069    128.65       112    0.165573    0.492992
  438      15     14120     14105   128.685       144   0.0217068    0.493053
  439      16     14151     14135   128.665       120     1.40709     0.49308
2020-03-09 14:21:15.946363 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.493203
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14179     14163   128.627       112     1.47629    0.493203
  441      16     14215     14199   128.662       144   0.0853626    0.493081
  442      16     14254     14238   128.724       156    0.221471    0.492772
  443      16     14289     14273   128.749       140    0.230987    0.492739
  444      16     14321     14305   128.747       128    0.297109     0.49275
  445      16     14349     14333    128.71       112     1.09776    0.492916
  446      15     14381     14366   128.716       132     0.12594    0.492769
  447      16     14415     14399   128.723       132    0.146776    0.492748
  448      16     14446     14430   128.712       124    0.352516    0.493012
  449      16     14478     14462    128.71       128    0.249134    0.492915
  450      16     14507     14491   128.682       116   0.0932466    0.492935
  451      16     14539     14523    128.68       128     1.81653     0.49301
  452      16     14577     14561   128.732       152    0.566595    0.492914
  453      16     14611     14595   128.748       136     0.36238    0.492737
  454      16     14643     14627   128.746       128    0.825547    0.492745
  455      15     14674     14659   128.743       128    0.210759    0.492725
  456      16     14710     14694   128.768       140   0.0225277    0.492751
  457      16     14741     14725   128.758       124    0.259295    0.492682
  458      16     14771     14755   128.738       120    0.156868    0.492717
  459      16     14805     14789   128.754       136     1.39997     0.49273
2020-03-09 14:21:35.961346 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492733
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14837     14821   128.752       128   0.0201961    0.492733
  461      16     14868     14852   128.742       124    0.133854    0.492855
  462      15     14904     14889   128.783       148    0.230342    0.492547
  463      16     14933     14917   128.747       112    0.425118    0.492613
  464      16     14967     14951   128.762       136    0.371976    0.492716
  465      16     15004     14988   128.803       148    0.260281    0.492592
  466      16     15030     15014    128.75       104   0.0424625    0.492609
  467      16     15075     15059    128.86       180    0.431316    0.492154
  468      16     15105     15089   128.841       120    0.243816    0.492406
  469      16     15138     15122   128.847       132    0.326468    0.492313
  470      16     15169     15153   128.837       124   0.0220026    0.492382
  471      16     15197     15181   128.797       112   0.0761229    0.492454
  472      15     15231     15216   128.821       140     1.53399    0.492607
  473      16     15265     15249   128.827       132    0.248291    0.492598
  474      16     15298     15282   128.834       132   0.0341461     0.49242
  475      16     15330     15314   128.832       128   0.0204142    0.492312
  476      16     15362     15346    128.83       128    0.356248    0.492415
  477      16     15397     15381   128.854       140    0.191772     0.49233
  478      15     15425     15410   128.826       116    0.211521    0.492388
  479      16     15460     15444   128.841       136    0.244305    0.492445
2020-03-09 14:21:55.987290 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492472
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     15495     15479   128.864       140    0.542079    0.492472
  481      16     15528     15512    128.87       132    0.432979    0.492447
  482      15     15557     15542   128.851       120     1.27866    0.492487
  483      16     15585     15569   128.808       108    0.452379    0.492445
  484      16     15618     15602   128.815       132   0.0219412    0.492504
  485      15     15647     15632   128.788       120   0.0973248    0.492639
  486      16     15678     15662    128.77       120     0.02203    0.492745
  487      16     15717     15701   128.825       156   0.0557803    0.492306
  488      16     15750     15734   128.832       132     1.67333    0.492398
  489      16     15783     15767   128.838       132   0.0436727    0.492468
  490      16     15818     15802    128.86       140    0.179463    0.492364
  491      15     15856     15841   128.915       156    0.466843    0.492076
  492      16     15889     15873   128.913       128    0.155933    0.492155
  493      16     15922     15906   128.919       132    0.268353     0.49211
  494      16     15958     15942    128.95       144    0.297482    0.491754
  495      15     15986     15971   128.923       116   0.0504102    0.491984
  496      16     16022     16006   128.946       140     1.39849    0.492128
  497      16     16054     16038   128.944       128   0.0908147    0.491885
  498      16     16085     16069   128.933       124    0.084014    0.492046
  499      16     16117     16101   128.931       128    0.444321    0.492112
2020-03-09 14:22:16.034211 min lat: 0.0197829 max lat: 2.39436 avg lat: 0.492122
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16148     16132   128.921       124     1.21853    0.492122
  501      16     16176     16160   128.887       112     1.73925    0.492174
  502      16     16207     16191   128.877       124    0.277324    0.492387
  503      16     16236     16220   128.851       116    0.238157    0.492364
  504      16     16269     16253   128.857       132    0.187329    0.492342
  505      16     16301     16285   128.856       128    0.266932    0.492337
  506      16     16333     16317   128.854       128     1.90655     0.49232
  507      16     16361     16345    128.82       112   0.0263439    0.492411
  508      16     16396     16380   128.842       140     0.23099    0.492531
  509      16     16425     16409   128.817       116    0.306136    0.492439
  510      16     16461     16445   128.846       144   0.0483066    0.492355
  511      16     16492     16476   128.837       124    0.340712    0.492339
  512      16     16524     16508   128.835       128    0.446706    0.492331
  513      16     16555     16539   128.826       124    0.306831     0.49248
  514      16     16586     16570   128.816       124    0.398609    0.492589
  515      16     16618     16602   128.815       128     0.23628    0.492496
  516      16     16650     16634   128.813       128   0.0700872    0.492542
  517      16     16684     16668   128.827       136    0.233433    0.492331
  518      16     16715     16699   128.817       124    0.157196    0.492641
  519      16     16746     16730   128.808       124    0.080323      0.4925
2020-03-09 14:22:36.044284 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492391
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16782     16766   128.837       144   0.0908164    0.492391
  521      16     16815     16799   128.843       132    0.476845    0.492301
  522      16     16851     16835   128.872       144   0.0896802    0.492434
  523      16     16884     16868   128.877       132   0.0449508    0.492421
  524      16     16920     16904   128.906       144    0.783382    0.492327
  525      15     16950     16935   128.896       124    0.586525    0.492388
  526      16     16980     16964   128.871       116   0.0621543    0.492412
  527      16     17012     16996    128.87       128    0.408836    0.492329
  528      16     17051     17035    128.92       156     1.67982    0.492279
  529      16     17084     17068   128.926       132     1.22913    0.492165
  530      16     17119     17103   128.947       140    0.101223    0.492206
  531      16     17155     17139   128.975       144     1.34578    0.491957
  532      15     17191     17176    129.01       148      0.1107    0.491861
  533      16     17225     17209   129.015       132     1.25785    0.491807
  534      16     17255     17239   128.998       120   0.0653489    0.491755
  535      15     17282     17267   128.966       112    0.208171    0.491928
  536      16     17315     17299   128.964       128    0.313139    0.492094
  537      16     17345     17329   128.948       120    0.735112    0.492071
  538      16     17380     17364   128.966       140    0.021429    0.491999
  539      16     17414     17398   128.979       136    0.406489    0.492057
2020-03-09 14:22:56.070701 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492151
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     17443     17427   128.955       116     1.00099    0.492151
  541      16     17476     17460   128.961       132    0.971481    0.492127
  542      16     17507     17491   128.952       124     0.68532    0.492186
  543      16     17538     17522   128.942       124    0.220592    0.492132
  544      16     17568     17552   128.926       120     0.14927    0.492168
  545      16     17599     17583   128.917       124    0.231191    0.492159
  546      16     17626     17610   128.878       108     1.94105    0.492308
  547      16     17660     17644   128.891       136   0.0593207    0.492337
  548      16     17688     17672    128.86       112     1.38285    0.492492
  549      16     17716     17700    128.83       112    0.325558    0.492565
  550      16     17748     17732   128.828       128    0.973484    0.492661
  551      16     17784     17768   128.856       144    0.155631    0.492479
  552      16     17816     17800   128.854       128     1.07623     0.49258
  553      16     17852     17836   128.881       144    0.306735    0.492376
  554      16     17884     17868   128.879       128     1.72593    0.492319
  555      16     17916     17900   128.878       128    0.281777    0.492351
  556      16     17943     17927    128.84       108     0.40143     0.49241
  557      16     17975     17959   128.838       128     1.40423    0.492486
  558      16     18007     17991   128.837       128     1.49951    0.492528
  559      16     18038     18022   128.828       124    0.186343    0.492535
2020-03-09 14:23:16.081103 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.492565
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     18071     18055   128.833       132     1.17569    0.492565
  561      16     18103     18087   128.831       128    0.200221    0.492441
  562      16     18135     18119    128.83       128     0.20541    0.492371
  563      15     18168     18153   128.842       136    0.456806     0.49241
  564      16     18204     18188   128.862       140   0.0649121    0.492384
  565      16     18233     18217   128.839       116   0.0274667    0.492404
  566      16     18264     18248   128.831       124    0.315095    0.492614
  567      16     18297     18281   128.836       132    0.491125    0.492562
  568      16     18327     18311   128.821       120    0.197906    0.492617
  569      16     18355     18339   128.791       112   0.0711673    0.492672
  570      16     18384     18368   128.769       116    0.188686    0.492716
  571      16     18418     18402   128.781       136    0.267251    0.492787
  572      16     18451     18435   128.787       132    0.180626    0.492769
  573      16     18477     18461   128.743       104    0.453697    0.492867
  574      16     18507     18491   128.728       120    0.462903    0.492904
  575      16     18539     18523   128.727       128      1.1521    0.492964
  576      16     18567     18551   128.698       112     1.13147    0.493135
  577      16     18595     18579   128.669       112    0.288448     0.49305
  578      16     18625     18609   128.654       120   0.0715439    0.493109
  579      16     18657     18641   128.653       128    0.276054    0.493219
2020-03-09 14:23:36.086298 min lat: 0.0197829 max lat: 2.4451 avg lat: 0.49335
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     18682     18666   128.603       100    0.306074     0.49335
  581      16     18716     18700   128.616       136     1.19553    0.493447
  582      16     18748     18732   128.615       128    0.216555    0.493352
  583      16     18780     18764   128.614       128    0.107101    0.493325
  584      16     18813     18797    128.62       132    0.021941    0.493279
  585      16     18849     18833   128.646       144      1.1934     0.49331
  586      16     18885     18869   128.665       144     1.80438      0.4932
  587      16     18916     18900   128.657       124    0.282091    0.493216
  588      15     18949     18934   128.669       136   0.0688578    0.493132
  589      16     18981     18965   128.661       124     1.17298     0.49327
  590      16     19009     18993   128.633       112    0.104593    0.493249
  591      15     19050     19035   128.699       168    0.191588    0.493082
  592      16     19082     19066   128.691       124     1.17998    0.493128
  593      16     19114     19098   128.689       128   0.0476302    0.493071
  594      16     19143     19127   128.668       116     1.16177    0.493136
  595      16     19174     19158    128.66       124     1.43278    0.493152
  596      16     19213     19197   128.706       156     1.49273    0.493016
  597      16     19244     19228   128.697       124     1.61806    0.492996
  598      16     19276     19260   128.696       128     1.78827    0.492975
  599      16     19308     19292   128.695       128     1.64029    0.493002
Total time run:       600.533
Total reads made:     19324
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.712
Average IOPS:         32
Stddev IOPS:          2.84268
Max IOPS:             45
Min IOPS:             25
Average Latency(s):   0.49342
Max latency(s):       2.4451
Min latency(s):       0.0197829

Total time run:       600.533
Total reads made:     19324
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.712
Average IOPS:         32
Stddev IOPS:          2.84268
Max IOPS:             45
Min IOPS:             25
Average Latency(s):   0.49342
Max latency(s):       2.4451
Min latency(s):       0.0197829

Random Read Test, pool with 256 PGs

podman exec ceph-mon-f5-422-01 rados bench -p testbench 600 rand | tee --append $(date +%F)-rados-test-read-rand
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0      13        13         0         0         0           -           0
    1      15        44        29   115.763       116    0.189163     0.31611
    2      16        77        61   121.858       128   0.0193401    0.430625
    3      16       107        91   121.204       120    0.184531    0.450987
    4      16       136       120   119.886       116    0.191766      0.4677
    5      16       171       155    123.89       140       1.648    0.458556
    6      16       205       189   125.887       136   0.0652734    0.451655
    7      16       236       220   125.607       124     1.36441    0.471769
    8      16       265       249   124.404       116    0.462344    0.481334
    9      16       296       280   124.358       124     1.31412    0.482733
   10      16       330       314   125.519       136   0.0497829    0.481064
   11      16       365       349   126.833       140     1.39185    0.482003
   12      16       398       382   127.255       132    0.226664    0.479182
   13      15       427       412   126.692       120    0.711614    0.486657
   14      15       460       445   127.068       132     0.23561    0.480579
   15      16       490       474   126.298       116    0.130672    0.488069
   16      16       523       507   126.653       132  0.00900812    0.481203
   17      15       550       535   125.791       112    0.276632    0.486413
   18      16       583       567   125.912       128   0.0482817    0.491239
   19      15       616       601   126.442       136     1.51149    0.486239
2020-03-09 14:28:25.379343 min lat: 0.00759327 max lat: 1.92824 avg lat: 0.488058
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16       648       632   126.319       124   0.0240633    0.488058
   21      16       678       662   126.017       120   0.0675982    0.488332
   22      16       710       694   126.107       128     1.53312    0.490658
   23      15       741       726   126.188       128    0.345689    0.489675
   24      16       771       755   125.763       116    0.194888    0.492858
   25      16       801       785   125.532       120       0.377    0.493429
   26      16       834       818    125.78       132      1.7753    0.496889
   27      16       863       847   125.417       116    0.217836    0.498455
   28      16       897       881   125.789       136     1.32996    0.497362
   29      16       928       912   125.727       124     1.41115    0.496599
   30      16       957       941   125.401       116   0.0553987    0.497336
   31      16       990       974   125.613       132   0.0492972    0.498332
   32      16      1019      1003   125.312       116     1.16908    0.499297
   33      16      1050      1034   125.271       124     1.57998    0.497278
   34      16      1084      1068   125.586       136    0.291149    0.497064
   35      16      1113      1097   125.312       116     1.47583    0.499034
   36      16      1145      1129   125.386       128    0.236787    0.498409
   37      16      1177      1161   125.456       128   0.0507549    0.500018
   38      16      1212      1196   125.838       140    0.220674    0.498035
   39      16      1244      1228   125.893       128    0.265011    0.496612
2020-03-09 14:28:45.383869 min lat: 0.00758844 max lat: 1.95098 avg lat: 0.496939
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      1280      1264   126.345       144   0.0737707    0.496939
   41      16      1309      1293   126.089       116     1.16005    0.497671
   42      16      1339      1323   125.944       120    0.318852    0.497165
   43      16      1371      1355   125.991       128    0.234976    0.497265
   44      16      1397      1381   125.491       104    0.452323    0.501171
   45      15      1430      1415   125.724       136  0.00916022    0.501471
   46      16      1462      1446   125.686       124    0.334006    0.500406
   47      16      1493      1477    125.65       124    0.662744    0.500872
   48      16      1526      1510   125.782       132    0.206438    0.499631
   49      16      1554      1538     125.5       112    0.308269     0.49914
   50      15      1587      1572   125.626       136    0.224301    0.499158
   51      16      1614      1598   125.202       104    0.210811    0.500811
   52      16      1642      1626   124.948       112     0.47214    0.501813
   53      16      1675      1659    125.08       132     1.73754    0.503802
   54      16      1704      1688   124.912       116    0.324704    0.503105
   55      16      1737      1721    125.04       132    0.161342    0.502503
   56      16      1773      1757   125.378       144    0.115535     0.50217
   57      16      1803      1787   125.283       120   0.0563292    0.501758
   58      16      1835      1819   125.329       128    0.359336    0.500746
   59      16      1865      1849   125.239       120    0.254863    0.501712
2020-03-09 14:29:05.421934 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.502075
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      1898      1882   125.351       132     0.14297    0.502075
   61      16      1930      1914   125.394       128    0.147922    0.502426
   62      16      1965      1949   125.629       140    0.364127    0.500649
   63      15      1994      1979   125.539       120     1.47404     0.50185
   64      16      2026      2010   125.515       124   0.0561347    0.501803
   65      16      2059      2043   125.614       132    0.307339    0.500391
   66      16      2089      2073   125.529       120    0.172544     0.50154
   67      16      2122      2106   125.625       132     1.40518    0.501262
   68      16      2151      2135   125.484       116     1.64331    0.501343
   69      15      2184      2169   125.635       136    0.261359    0.501214
   70      16      2217      2201   125.669       128    0.203083    0.501397
   71      15      2244      2229   125.476       112   0.0595244    0.502064
   72      16      2277      2261   125.511       128   0.0934983    0.501999
   73      16      2312      2296   125.709       140   0.0643922     0.50122
   74      16      2341      2325   125.577       116    0.216984    0.502041
   75      16      2368      2352   125.343       108     1.46016    0.503179
   76      16      2401      2385    125.43       132   0.0425867    0.503542
   77      16      2438      2422   125.723       148   0.0797482    0.501424
   78      16      2471      2455   125.798       132    0.281652    0.500717
   79      16      2508      2492   126.079       148    0.208958    0.500398
2020-03-09 14:29:25.428699 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.500447
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   80      15      2540      2525   126.152       132   0.0655898    0.500447
   81      16      2570      2554   126.026       116    0.567935    0.499762
   82      16      2607      2591   126.294       148    0.362565    0.499377
   83      16      2642      2626   126.459       140    0.196834    0.499217
   84      16      2679      2663   126.715       148    0.238598    0.497743
   85      16      2711      2695   126.729       128    0.243838    0.497581
   86      16      2747      2731   126.929       144    0.443474     0.49703
   87      16      2780      2764   126.987       132    0.252554    0.497319
   88      16      2808      2792   126.817       112    0.334884    0.497985
   89      16      2837      2821   126.695       116     1.26582    0.498803
   90      16      2868      2852   126.665       124     0.16486    0.498447
   91      16      2901      2885   126.724       132     1.47612    0.497824
   92      16      2933      2917   126.737       128    0.183519    0.497762
   93      16      2966      2950   126.794       132   0.0214082    0.498126
   94      16      2997      2981   126.764       124   0.0473124    0.498186
   95      16      3030      3014   126.818       132    0.269154    0.498132
   96      16      3062      3046    126.83       128   0.0525916    0.497567
   97      16      3096      3080   126.924       136   0.0901356     0.49752
   98      16      3127      3111   126.893       124    0.188486    0.497984
   99      16      3162      3146   127.025       140    0.177758    0.497015
2020-03-09 14:29:45.433613 min lat: 0.00758844 max lat: 2.0412 avg lat: 0.496782
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  100      16      3199      3183   127.235       148   0.0539025    0.496782
  101      15      3230      3215   127.242       128    0.629777    0.497503
  102      16      3266      3250   127.367       140     1.38567     0.49635
  103      16      3300      3284    127.45       136    0.276637    0.495667
  104      16      3332      3316   127.455       128   0.0709163    0.495696
  105      16      3364      3348   127.459       128   0.0724538    0.495799
  106      16      3393      3377   127.351       116    0.383282    0.496338
  107      16      3423      3407   127.282       120    0.768926    0.496838
  108      15      3460      3445   127.511       152    0.381757    0.495984
  109      15      3492      3477   127.515       128    0.071551    0.495776
  110      16      3526      3510   127.556       132      1.3167    0.495523
  111      16      3557      3541   127.523       124   0.0736356     0.49555
  112      15      3592      3577    127.67       144   0.0713146    0.494824
  113      15      3627      3612   127.778       140     0.04953    0.495249
  114      16      3657      3641   127.674       116    0.248503    0.494823
  115      16      3692      3676   127.781       140     1.18351    0.494717
  116      16      3724      3708   127.783       128   0.0240328    0.494033
  117      16      3754      3738   127.716       120   0.0489401    0.493972
  118      16      3789      3773    127.82       140    0.463131    0.494389
  119      16      3819      3803   127.753       120    0.610839    0.495552
2020-03-09 14:30:05.440702 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.495371
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  120      16      3848      3832   127.654       116     1.33663    0.495371
  121      16      3886      3870   127.855       152    0.064901     0.49446
  122      16      3920      3904   127.922       136   0.0451795    0.494092
  123      16      3953      3937   127.954       132    0.291511    0.493635
  124      16      3984      3968   127.922       124    0.657468    0.494118
  125      16      4020      4004    128.05       144   0.0485406    0.493872
  126      16      4054      4038   128.113       136     1.66854    0.493653
  127      16      4081      4065   127.954       108   0.0473508    0.494324
  128      16      4116      4100   128.048       140    0.811657    0.494353
  129      16      4149      4133   128.079       132    0.257035    0.494438
  130      16      4182      4166   128.108       132   0.0948986    0.494335
  131      16      4216      4200   128.168       136     1.08268    0.493817
  132      16      4247      4231   128.136       124   0.0484442    0.494168
  133      15      4279      4264   128.163       132    0.146956    0.493825
  134      16      4311      4295   128.132       124     1.58947    0.493647
  135      16      4345      4329   128.189       136   0.0451388    0.493772
  136      16      4378      4362   128.216       132   0.0936799    0.493416
  137      16      4417      4401   128.419       156    0.169335    0.492621
  138      16      4445      4429     128.3       112     1.59991    0.492925
  139      16      4479      4463   128.355       136    0.277477     0.49307
2020-03-09 14:30:25.450530 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.493065
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  140      16      4511      4495   128.352       128    0.319201    0.493065
  141      16      4545      4529   128.406       136      1.4205     0.49272
  142      16      4581      4565   128.515       144    0.151326    0.492073
  143      16      4613      4597   128.511       128    0.218008    0.492406
  144      16      4644      4628    128.48       124    0.280231    0.492305
  145      16      4674      4658   128.421       120     1.73267     0.49242
  146      16      4703      4687   128.336       116    0.338129    0.493134
  147      15      4735      4720    128.36       132     1.76812    0.493022
  148      16      4766      4750   128.302       120    0.277118    0.493387
  149      16      4800      4784   128.353       136     1.40389    0.493366
  150      16      4835      4819    128.43       140    0.231478     0.49267
  151      16      4865      4849   128.374       120    0.214272    0.493014
  152      16      4898      4882   128.398       132     1.38171    0.493616
  153      16      4929      4913   128.369       124   0.0673812    0.493461
  154      16      4960      4944   128.341       124    0.104399    0.493742
  155      16      4994      4978    128.39       136    0.273172    0.492907
  156      16      5027      5011   128.413       132    0.025074    0.492787
  157      16      5067      5051   128.614       160    0.080997    0.491795
  158      16      5100      5084   128.635       132    0.280364    0.491865
  159      16      5133      5117   128.656       132    0.223638    0.492138
2020-03-09 14:30:45.457259 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.492326
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  160      16      5163      5147   128.602       120     1.53826    0.492326
  161      16      5194      5178   128.573       124    0.222583    0.492617
  162      16      5228      5212   128.618       136     1.46397    0.492283
  163      16      5260      5244   128.609       128     1.23506    0.492249
  164      16      5288      5272   128.507       112    0.522441    0.492651
  165      16      5319      5303   128.479       124    0.397334    0.492819
  166      16      5352      5336     128.5       132   0.0707879    0.493225
  167      16      5383      5367   128.472       124   0.0436325    0.492632
  168      16      5421      5405   128.611       152    0.281206    0.492555
  169      16      5454      5438   128.631       132    0.213111    0.492124
  170      15      5491      5476   128.768       152    0.220927    0.491671
  171      16      5524      5508   128.763       128    0.337313    0.491937
  172      15      5559      5544   128.852       144    0.226524    0.491718
  173      16      5596      5580   128.938       144      1.2731    0.491177
  174      16      5627      5611   128.909       124      0.1746    0.491333
  175      16      5657      5641   128.858       120    0.523888    0.491296
  176      16      5689      5673   128.852       128    0.072858    0.491689
  177      15      5724      5709   128.935       144     1.40637    0.491236
  178      16      5754      5738   128.863       116    0.223111    0.491507
  179      16      5790      5774   128.947       144    0.519273    0.491727
2020-03-09 14:31:05.479185 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.491175
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  180      16      5826      5810    129.03       144    0.316725    0.491175
  181      16      5857      5841       129       124   0.0655163     0.49096
  182      15      5885      5870   128.928       116    0.306004    0.491671
  183      15      5915      5900   128.879       120    0.669872    0.491709
  184      16      5946      5930   128.831       120     1.35695    0.491863
  185      16      5979      5963   128.848       132    0.205182    0.491501
  186      16      6008      5992   128.779       116   0.0579184    0.491649
  187      16      6043      6027   128.839       140    0.536809    0.491505
  188      16      6074      6058   128.813       124    0.108285    0.491614
  189      16      6101      6085   128.703       108     0.31382    0.492289
  190      16      6136      6120   128.761       140     1.40723    0.492137
  191      16      6166      6150   128.715       120    0.179843    0.492069
  192      16      6195      6179   128.649       116   0.0518371    0.492127
  193      16      6226      6210   128.624       124   0.0500438    0.492554
  194      16      6254      6238   128.539       112    0.346221    0.492518
  195      16      6286      6270   128.536       128     1.71537    0.492747
  196      16      6318      6302   128.533       128     1.33163     0.49283
  197      16      6350      6334    128.53       128   0.0451284    0.492875
  198      16      6382      6366   128.527       128    0.330289    0.493035
  199      16      6415      6399   128.545       132     1.13619    0.493026
2020-03-09 14:31:25.487928 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.492909
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  200      16      6447      6431   128.542       128   0.0579771    0.492909
  201      16      6481      6465   128.579       136    0.241814    0.492578
  202      16      6512      6496   128.556       124     1.28887    0.493017
  203      16      6544      6528   128.553       128    0.297991    0.492795
  204      16      6576      6560   128.551       128   0.0553758     0.49279
  205      15      6610      6595   128.606       140     1.30937    0.492809
  206      16      6642      6626   128.584       124    0.196741    0.492283
  207      16      6672      6656   128.542       120   0.0477203    0.492571
  208      16      6703      6687   128.519       124    0.912837    0.493192
  209      16      6735      6719   128.516       128      0.1108    0.492738
  210      16      6769      6753   128.551       136    0.294788    0.492828
  211      16      6802      6786   128.567       132     1.58597    0.492765
  212      16      6835      6819   128.583       132    0.258177    0.492763
  213      16      6865      6849   128.541       120    0.236762    0.493015
  214      16      6897      6881   128.539       128    0.238013    0.493043
  215      16      6928      6912   128.517       124    0.206292    0.493122
  216      16      6958      6942   128.478       120     0.21966    0.493296
  217      16      6990      6974   128.462       128    0.444428     0.49317
  218      16      7023      7007   128.478       132     1.28373    0.493538
  219      16      7059      7043   128.547       144    0.243892    0.493325
2020-03-09 14:31:45.523298 min lat: 0.00758844 max lat: 2.12855 avg lat: 0.49324
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  220      16      7089      7073   128.508       120    0.996789     0.49324
  221      16      7123      7107   128.542       136    0.145334    0.493051
  222      16      7150      7134    128.45       108     1.32096    0.493415
  223      15      7179      7164   128.412       120   0.0839449    0.493748
  224      16      7212      7196   128.409       128    0.181908    0.493767
  225      16      7247      7231    128.46       140    0.228171    0.493494
  226      16      7282      7266   128.511       140   0.0488206    0.492914
  227      16      7315      7299   128.526       132     1.41182    0.493104
  228      16      7348      7332   128.541       132    0.291632    0.492987
  229      15      7386      7371    128.66       156    0.157679    0.492435
  230      16      7421      7405   128.692       136    0.192474    0.492397
  231      16      7448      7432   128.603       108    0.303068    0.492405
  232      15      7480      7465   128.617       132   0.0521112    0.492661
  233      16      7512      7496   128.597       124    0.385918    0.492894
  234      16      7542      7526    128.56       120   0.0511737    0.492914
  235      16      7572      7556   128.524       120    0.542226    0.493334
  236      16      7600      7584   128.453       112    0.201575    0.493548
  237      16      7630      7614   128.417       120   0.0561037     0.49356
  238      16      7670      7654    128.55       160   0.0849755    0.492901
  239      16      7704      7688   128.581       136    0.265047    0.492865
2020-03-09 14:32:05.531148 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.492841
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  240      16      7740      7724   128.645       144   0.0520922    0.492841
  241      15      7769      7754   128.609       120    0.302236    0.492862
  242      16      7803      7787   128.623       132   0.0548091    0.492658
  243      16      7838      7822    128.67       140    0.285622    0.492711
  244      16      7870      7854   128.666       128      1.1918    0.492883
  245      15      7904      7889   128.712       140   0.0848028    0.492829
  246      16      7936      7920   128.693       124   0.0449712    0.492728
  247      15      7968      7953   128.706       132    0.256293    0.492704
  248      16      7999      7983    128.67       120    0.148891    0.492835
  249      16      8030      8014   128.651       124    0.578604    0.492981
  250      16      8066      8050   128.712       144    0.248972    0.492468
  251      16      8102      8086   128.773       144    0.434066    0.492089
  252      16      8130      8114   128.706       112    0.167202    0.492406
  253      16      8164      8148   128.735       136    0.327447     0.49214
  254      16      8195      8179   128.716       124    0.244601    0.492364
  255      16      8234      8218   128.823       156    0.475849    0.492149
  256      16      8266      8250    128.82       128   0.0758576    0.492001
  257      16      8298      8282   128.816       128     1.75357    0.492137
  258      15      8323      8308    128.72       104     1.10742    0.492735
  259      16      8355      8339   128.702       124    0.131107     0.49267
2020-03-09 14:32:25.540825 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.492416
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  260      16      8389      8373   128.729       136  0.00857981    0.492416
  261      16      8420      8404    128.71       124   0.0837172    0.492602
  262      16      8452      8436   128.708       128     1.51838    0.492711
  263      16      8488      8472   128.766       144   0.0882103    0.492343
  264      16      8518      8502   128.732       120     1.74093    0.492447
  265      16      8551      8535   128.745       132    0.504377    0.492406
  266      16      8587      8571   128.802       144   0.0597395    0.492269
  267      16      8622      8606   128.844       140    0.149485    0.492174
  268      16      8651      8635   128.795       116     1.37556    0.492117
  269      15      8683      8668   128.807       132    0.420735    0.492147
  270      16      8714      8698   128.774       120     1.17478    0.492491
  271      16      8746      8730   128.771       128   0.0611217     0.49214
  272      16      8785      8769    128.87       156    0.435239     0.49194
  273      16      8819      8803   128.896       136    0.198297     0.49167
  274      16      8848      8832   128.848       116    0.274941    0.492066
  275      16      8882      8866   128.874       136    0.495659    0.491967
  276      16      8910      8894   128.812       112   0.0707601    0.492035
  277      16      8941      8925   128.794       124    0.499243     0.49212
  278      16      8972      8956   128.775       124    0.595984    0.492319
  279      15      9006      8991   128.815       140    0.142209    0.492165
2020-03-09 14:32:45.556957 min lat: 0.00758844 max lat: 2.15434 avg lat: 0.49211
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  280      16      9042      9026   128.855       140    0.456657     0.49211
  281      16      9074      9058   128.851       128    0.151224     0.49202
  282      15      9103      9088    128.82       120   0.0545886    0.492224
  283      16      9138      9122   128.845       136     1.60324    0.491955
  284      16      9169      9153   128.828       124    0.461395    0.492232
  285      16      9200      9184   128.811       124    0.309175    0.492278
  286      16      9232      9216   128.808       128   0.0684568    0.492433
  287      16      9270      9254   128.889       152     1.33219    0.492114
  288      16      9296      9280   128.802       104    0.232691    0.492141
  289      16      9325      9309   128.757       116    0.462623    0.492671
  290      16      9355      9339   128.727       120    0.223172    0.492626
  291      15      9386      9371   128.724       128    0.304125    0.492512
  292      16      9421      9405   128.749       136     0.51039     0.49245
  293      16      9456      9440   128.788       140     1.36367     0.49226
  294      16      9489      9473   128.798       132    0.291399    0.492226
  295      15      9515      9500   128.728       108    0.178472    0.492375
  296      16      9543      9527   128.657       108       1.855    0.492608
  297      16      9577      9561   128.682       136   0.0490071    0.492966
  298      16      9606      9590   128.639       116    0.160699    0.492809
  299      16      9641      9625   128.677       140    0.113257    0.492842
2020-03-09 14:33:05.565139 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492681
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  300      16      9672      9656   128.661       124     0.19061    0.492681
  301      15      9707      9692   128.712       144    0.290928    0.492389
  302      16      9739      9723   128.696       124    0.202738    0.492598
  303      16      9771      9755   128.694       128    0.745036    0.492747
  304      16      9805      9789   128.718       136    0.446637    0.492782
  305      16      9841      9825   128.767       144    0.250422    0.492416
  306      15      9871      9856   128.751       124   0.0686151    0.492577
  307      16      9904      9888   128.749       128     1.63885    0.492691
  308      16      9930      9914   128.668       104    0.670912    0.492898
  309      15      9965      9950   128.718       144    0.731271     0.49294
  310      16      9997      9981   128.703       124      1.2368    0.492815
  311      16     10029     10013     128.7       128    0.318304    0.492673
  312      16     10061     10045   128.698       128    0.272861    0.492736
  313      16     10096     10080   128.733       140     1.52497    0.492731
  314      16     10124     10108    128.68       112    0.640861    0.492957
  315      16     10155     10139   128.665       124    0.779685     0.49307
  316      16     10185     10169   128.637       120     1.02686    0.492947
  317      16     10217     10201   128.635       128     0.19747    0.492794
  318      16     10247     10231   128.607       120    0.157774    0.492928
  319      16     10275     10259   128.555       112     1.45464    0.493131
2020-03-09 14:33:25.576584 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.493064
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  320      15     10309     10294   128.591       140    0.343314    0.493064
  321      16     10343     10327   128.601       132    0.197438    0.493072
  322      16     10375     10359   128.598       128   0.0525443    0.493091
  323      16     10409     10393   128.621       136    0.242701    0.493092
  324      16     10437     10421    128.57       112     1.41396    0.493195
  325      16     10470     10454    128.58       132    0.225383    0.493195
  326      16     10501     10485   128.566       124    0.248241    0.493161
  327      16     10534     10518   128.577       132   0.0774712    0.493306
  328      15     10566     10551   128.587       132     1.41793    0.493123
  329      15     10593     10578   128.524       108    0.259335    0.493244
  330      16     10628     10612   128.547       136   0.0442344    0.493329
  331      16     10663     10647   128.581       140      0.3253    0.493408
  332      16     10696     10680   128.592       132    0.893511    0.493377
  333      16     10728     10712    128.59       128    0.311234    0.493074
  334      16     10764     10748   128.636       144    0.262498    0.493077
  335      16     10797     10781   128.646       132     1.40241    0.492944
  336      16     10830     10814   128.656       132    0.044292    0.492942
  337      16     10864     10848   128.677       136    0.168146    0.492757
  338      16     10897     10881   128.687       132      1.3778    0.492866
  339      16     10934     10918   128.744       148    0.319363    0.492484
2020-03-09 14:33:45.584674 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492421
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  340      15     10971     10956   128.811       152    0.447902    0.492421
  341      16     11000     10984   128.762       112    0.488588    0.492526
  342      16     11032     11016    128.76       128    0.185412    0.492655
  343      16     11060     11044   128.711       112    0.192753    0.492687
  344      16     11090     11074   128.685       120    0.187793    0.492708
  345      16     11121     11105   128.672       124     1.61635    0.492923
  346      15     11153     11138   128.669       132   0.0609645    0.492882
  347      16     11181     11165   128.609       108     1.56338    0.493049
  348      15     11210     11195   128.584       120    0.241957    0.493386
  349      16     11248     11232    128.64       148    0.195023     0.49309
  350      16     11280     11264   128.638       128     1.49536    0.493015
  351      16     11309     11293   128.602       116  0.00787264    0.493074
  352      16     11339     11323   128.577       120    0.458127    0.493152
  353      16     11372     11356   128.587       132    0.134739    0.493165
  354      16     11405     11389   128.596       132  0.00822177    0.493143
  355      16     11435     11419   128.572       120    0.850725    0.493414
  356      16     11466     11450   128.559       124   0.0163893    0.493191
  357      16     11497     11481   128.546       124  0.00973857    0.493324
  358      15     11533     11518     128.6       148   0.0929121    0.493285
  359      16     11567     11551    128.61       132   0.0446927    0.493259
2020-03-09 14:34:05.623724 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.493115
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  360      16     11601     11585    128.63       136    0.185377    0.493115
  361      16     11633     11617   128.629       128    0.101306    0.493093
  362      16     11665     11649   128.627       128    0.319652    0.492944
  363      15     11698     11683   128.647       136    0.167573    0.493022
  364      16     11728     11712   128.612       116     1.35791     0.49309
  365      16     11758     11742   128.588       120    0.446025    0.493355
  366      16     11789     11773   128.576       124    0.249789    0.493407
  367      16     11819     11803   128.552       120    0.253794    0.493391
  368      15     11849     11834    128.54       124     1.52666    0.493368
  369      16     11885     11869   128.571       140    0.200598    0.493246
  370      15     11912     11897   128.526       112    0.297605    0.493556
  371      16     11946     11930   128.535       132    0.338924    0.493423
  372      15     11981     11966   128.576       144   0.0431001    0.493212
  373      16     12016     12000   128.596       136   0.0616273    0.493273
  374      16     12050     12034   128.615       136   0.0261335    0.493009
  375      16     12079     12063   128.582       116   0.0195697    0.493146
  376      16     12113     12097   128.601       136    0.547183    0.493402
  377      16     12147     12131   128.621       136   0.0578911    0.493176
  378      15     12183     12168   128.672       148     1.88786     0.49296
  379      16     12215     12199    128.66       124    0.194814    0.492892
2020-03-09 14:34:25.630542 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492896
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  380      16     12250     12234    128.69       140    0.290504    0.492896
  381      16     12283     12267   128.698       132    0.555411     0.49277
  382      16     12316     12300   128.707       132    0.167732    0.492893
  383      16     12352     12336   128.747       144     0.63584    0.492801
  384      15     12381     12366   128.724       120     0.74141    0.492988
  385      16     12414     12398   128.722       128    0.744432    0.492995
  386      16     12445     12429    128.71       124    0.233459    0.492936
  387      15     12483     12468    128.78       156    0.348882    0.492793
  388      15     12524     12509    128.87       164   0.0533723    0.492405
  389      16     12557     12541   128.868       128    0.200685    0.492314
  390      16     12589     12573   128.865       128    0.058201    0.492312
  391      15     12622     12607   128.884       136      1.5035    0.492203
  392      16     12657     12641   128.902       136    0.237846    0.491977
  393      16     12688     12672   128.889       124    0.262442    0.492089
  394      16     12716     12700   128.846       112    0.304671    0.492372
  395      15     12745     12730   128.824       120     1.51254    0.492375
  396      16     12780     12764   128.842       136    0.278281    0.492278
  397      16     12813     12797    128.85       132    0.503318    0.492341
  398      16     12846     12830   128.858       132    0.584168    0.492396
  399      16     12875     12859   128.826       116    0.555365    0.492404
2020-03-09 14:34:45.636586 min lat: 0.00758844 max lat: 2.22367 avg lat: 0.492408
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  400      16     12910     12894   128.853       140     1.11721    0.492408
  401      16     12938     12922   128.811       112    0.324473     0.49247
  402      16     12971     12955   128.819       132     1.15171    0.492567
  403      16     13004     12988   128.827       132     0.28324    0.492369
  404      15     13035     13020   128.825       128    0.568556    0.492572
  405      16     13068     13052   128.823       128    0.285947    0.492328
  406      16     13096     13080   128.781       112    0.983314    0.492578
  407      16     13133     13117   128.828       148     1.34305     0.49241
  408      16     13162     13146   128.797       116     1.21148    0.492506
  409      16     13194     13178   128.795       128     1.42986    0.492525
  410      16     13223     13207   128.764       116   0.0734785    0.492599
  411      16     13254     13238   128.752       124    0.233044    0.492378
  412      16     13286     13270    128.75       128    0.410644    0.492725
  413      16     13319     13303   128.758       132    0.453446    0.492756
  414      16     13349     13333   128.737       120      1.5458    0.492846
  415      16     13383     13367   128.754       136   0.0106207    0.492789
  416      16     13415     13399   128.752       128    0.395381    0.492615
  417      16     13447     13431    128.75       128      0.2048     0.49283
  418      16     13480     13464   128.758       132      1.3471    0.492711
  419      16     13507     13491   128.709       108   0.0397955    0.492894
2020-03-09 14:35:05.639344 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.49295
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  420      16     13540     13524   128.716       132   0.0507944     0.49295
  421      16     13572     13556   128.715       128    0.282947    0.492984
  422      16     13606     13590   128.732       136    0.706111    0.493061
  423      16     13636     13620   128.711       120     1.06421    0.492954
  424      16     13668     13652   128.709       128    0.914274    0.493021
  425      16     13704     13688   128.745       144    0.608122    0.492901
  426      16     13735     13719   128.734       124   0.0564004    0.492908
  427      15     13770     13755    128.77       144   0.0526887    0.492775
  428      16     13806     13790   128.796       140   0.0443094    0.492592
  429      16     13835     13819   128.766       116   0.0501112    0.492613
  430      16     13865     13849   128.746       120     0.22598    0.492801
  431      16     13895     13879   128.725       120    0.486399    0.492949
  432      16     13925     13909   128.705       120   0.0852251    0.492927
  433      16     13955     13939   128.685       120   0.0745961    0.492949
  434      16     13989     13973   128.702       136     1.02644    0.493052
  435      16     14025     14009   128.737       144     1.14398    0.492908
  436      16     14055     14039   128.716       120    0.350549    0.492762
  437      16     14088     14072   128.724       132     1.73861    0.492887
  438      16     14119     14103   128.713       124    0.164738    0.492694
  439      16     14154     14138   128.739       140     2.16674    0.492729
2020-03-09 14:35:25.644849 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492933
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  440      16     14183     14167    128.71       116     1.15626    0.492933
  441      16     14216     14200   128.717       132     1.39708    0.492857
  442      16     14249     14233   128.724       132   0.0871006    0.492847
  443      16     14279     14263   128.704       120     1.40747    0.492984
  444      16     14317     14301   128.757       152    0.351914     0.49274
  445      16     14351     14335   128.773       136    0.139878    0.492688
  446      16     14377     14361   128.717       104    0.373503    0.492805
  447      16     14407     14391   128.698       120    0.496401    0.492977
  448      16     14438     14422   128.687       124      1.3126    0.493038
  449      16     14467     14451   128.659       116    0.550495    0.493221
  450      16     14500     14484   128.666       132     0.10734    0.493166
  451      16     14535     14519   128.691       140   0.0491615    0.492978
  452      16     14570     14554   128.716       140    0.050293    0.492956
  453      16     14602     14586   128.714       128   0.0475525    0.492785
  454      16     14631     14615   128.686       116    0.270657    0.492873
  455      16     14663     14647   128.685       128   0.0872759    0.493021
  456      15     14699     14684   128.727       148     1.01948    0.492978
  457      16     14730     14714   128.707       120     1.19433    0.493034
  458      16     14762     14746   128.706       128    0.209505    0.492863
  459      16     14796     14780   128.721       136  0.00978147    0.492855
2020-03-09 14:35:45.652884 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492738
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  460      16     14827     14811   128.711       124    0.209213    0.492738
  461      16     14865     14849   128.762       152   0.0246178    0.492769
  462      15     14894     14879   128.743       120    0.186257    0.492873
  463      16     14929     14913   128.758       136    0.813413     0.49293
  464      16     14961     14945   128.757       128   0.0718973    0.492902
  465      16     14988     14972   128.712       108    0.700856    0.493065
  466      15     15015     15000   128.676       112     1.45437    0.493088
  467      16     15046     15030   128.657       120    0.982401    0.493312
  468      16     15077     15061   128.647       124     1.32528    0.493185
  469      16     15111     15095   128.663       136     0.13288    0.493218
  470      16     15140     15124   128.636       116     1.41088    0.493255
  471      16     15175     15159    128.66       140     0.24409    0.492975
  472      15     15207     15192   128.667       132    0.581044    0.493001
  473      16     15237     15221    128.64       116   0.0664894    0.493291
  474      16     15270     15254   128.647       132   0.0535102    0.493187
  475      16     15299     15283    128.62       116     1.53691    0.493409
  476      16     15330     15314   128.611       124     1.35426    0.493402
  477      16     15361     15345   128.601       124    0.110866    0.493399
  478      16     15393     15377     128.6       128    0.405185    0.493416
  479      16     15424     15408    128.59       124    0.226533    0.493389
2020-03-09 14:36:05.657299 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493266
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  480      16     15459     15443   128.614       140   0.0733489    0.493266
  481      16     15494     15478   128.637       140   0.0616342    0.493254
  482      16     15522     15506   128.603       112     1.21014    0.493424
  483      16     15551     15535   128.576       116     0.13167    0.493514
  484      16     15583     15567   128.575       128     1.16737    0.493633
  485      16     15612     15596   128.549       116   0.0367693    0.493533
  486      16     15645     15629   128.556       132    0.397224    0.493586
  487      16     15677     15661   128.555       128    0.281867    0.493638
  488      16     15707     15691   128.537       120    0.216604     0.49357
  489      16     15735     15719   128.503       112    0.343392    0.493664
  490      16     15769     15753   128.519       136    0.213094    0.493635
  491      16     15798     15782   128.493       116    0.184509     0.49382
  492      16     15828     15812   128.476       120     1.36858    0.493852
  493      16     15862     15846   128.491       136    0.399842    0.493788
  494      16     15896     15880   128.506       136  0.00783099    0.493718
  495      16     15926     15910   128.489       120    0.761711    0.493817
  496      16     15962     15946    128.52       144   0.0600508    0.493766
  497      16     15995     15979   128.527       132    0.162762     0.49367
  498      16     16027     16011   128.526       128    0.293957    0.493739
  499      16     16056     16040   128.501       116   0.0545122    0.493855
2020-03-09 14:36:25.663755 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493762
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  500      16     16090     16074   128.516       136   0.0539079    0.493762
  501      16     16128     16112   128.562       152    0.211005    0.493564
  502      16     16160     16144   128.561       128    0.187399    0.493432
  503      16     16192     16176   128.559       128     1.75977    0.493535
  504      15     16227     16212   128.589       144     1.41888    0.493603
  505      16     16267     16251   128.644       156    0.519815    0.493384
  506      15     16298     16283   128.642       128     1.10476    0.493385
  507      16     16333     16317   128.656       136    0.387107    0.493177
  508      16     16367     16351   128.671       136   0.0528087    0.493226
  509      16     16400     16384   128.677       132   0.0533276    0.493184
  510      16     16436     16420   128.707       144     1.41282    0.492935
  511      16     16471     16455   128.729       140    0.144499    0.492955
  512      16     16508     16492   128.767       148    0.231932    0.492727
  513      16     16545     16529   128.804       148     1.56851    0.492625
  514      15     16583     16568   128.857       156    0.107234    0.492319
  515      16     16618     16602   128.871       136     1.93003    0.492464
  516      16     16649     16633   128.861       124   0.0800015    0.492454
  517      16     16680     16664   128.852       124     0.50851    0.492514
  518      15     16714     16699   128.873       140    0.249528    0.492439
  519      16     16747     16731   128.872       128    0.126583    0.492352
2020-03-09 14:36:45.674097 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492419
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  520      16     16780     16764   128.878       132    0.148523    0.492419
  521      16     16815     16799   128.899       140    0.903131    0.492416
  522      16     16844     16828   128.874       116    0.611226     0.49234
  523      16     16880     16864   128.903       144    0.278245    0.492306
  524      16     16914     16898   128.917       136     1.21513    0.492365
  525      16     16945     16929   128.907       124     1.07566    0.492361
  526      16     16975     16959    128.89       120    0.257593    0.492315
  527      16     17004     16988   128.866       116    0.664407    0.492495
  528      16     17039     17023   128.887       140     1.09811    0.492434
  529      16     17068     17052   128.862       116    0.217041    0.492387
  530      16     17098     17082   128.846       120     1.42617    0.492447
  531      16     17132     17116   128.859       136     1.67557    0.492498
  532      16     17161     17145   128.835       116    0.272223    0.492629
  533      16     17192     17176   128.826       124    0.470807    0.492679
  534      16     17230     17214   128.869       152    0.800189    0.492543
  535      16     17259     17243   128.845       116    0.268247    0.492526
  536      16     17286     17270   128.806       108     1.59423    0.492582
  537      16     17319     17303   128.812       132     1.37513    0.492643
  538      16     17353     17337   128.825       136     1.32731    0.492684
  539      16     17386     17370   128.831       132     0.55438    0.492697
2020-03-09 14:37:05.677000 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492739
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  540      16     17416     17400   128.815       120  0.00819962    0.492739
  541      16     17447     17431   128.806       124   0.0714803    0.492737
  542      16     17484     17468   128.841       148    0.570417    0.492588
  543      16     17512     17496    128.81       112    0.618657    0.492652
  544      16     17547     17531    128.83       140     1.34152    0.492587
  545      16     17577     17561   128.814       120    0.315035    0.492582
  546      16     17607     17591   128.798       120    0.410163     0.49261
  547      16     17639     17623   128.797       128    0.303394    0.492792
  548      16     17669     17653   128.781       120    0.444412     0.49282
  549      16     17706     17690   128.815       148     1.64699    0.492705
  550      16     17734     17718   128.785       112   0.0850279      0.4928
  551      16     17764     17748   128.769       120     1.23936    0.492867
  552      16     17797     17781   128.775       132     1.22765    0.492883
  553      16     17827     17811   128.759       120     1.04583    0.492945
  554      16     17857     17841   128.743       120   0.0581155    0.493009
  555      16     17892     17876   128.763       140    0.529912    0.492957
  556      16     17922     17906   128.747       120    0.270666    0.492965
  557      16     17956     17940    128.76       136    0.516379    0.492999
  558      16     17984     17968    128.73       112    0.496621    0.493072
  559      16     18015     17999   128.721       124   0.0324908    0.493108
2020-03-09 14:37:25.685870 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.493198
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  560      16     18043     18027   128.691       112    0.251608    0.493198
  561      16     18077     18061   128.704       136    0.214448    0.493023
  562      16     18106     18090   128.681       116    0.431902    0.493103
  563      16     18141     18125   128.701       140    0.487035    0.493054
  564      16     18179     18163   128.742       152     1.40696    0.492956
  565      16     18212     18196   128.748       132    0.216091    0.492718
  566      16     18243     18227   128.739       124    0.212926    0.492741
  567      16     18272     18256   128.717       116     1.85285    0.492963
  568      16     18305     18289   128.722       132     0.56152    0.492974
  569      16     18341     18325   128.749       144     1.37575    0.492928
  570      16     18373     18357   128.747       128    0.223864    0.492992
  571      16     18403     18387   128.732       120     1.36241    0.492997
  572      16     18435     18419   128.731       128     1.63608    0.492984
  573      15     18466     18451   128.729       128   0.0785299    0.493063
  574      16     18499     18483   128.722       128     0.01434    0.492989
  575      15     18534     18519   128.749       144   0.0794387    0.493002
  576      16     18565     18549   128.733       120       1.299    0.492965
  577      16     18601     18585    128.76       144     1.43417    0.492892
  578      16     18628     18612   128.723       108     0.17641    0.493021
  579      16     18659     18643   128.714       124   0.0147537    0.493039
2020-03-09 14:37:45.727174 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.492994
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  580      16     18693     18677   128.727       136     1.36927    0.492994
  581      15     18726     18711   128.739       136   0.0761666    0.492962
  582      16     18756     18740   128.717       116   0.0884404     0.49296
  583      16     18787     18771   128.709       124       1.685    0.493085
  584      16     18817     18801   128.694       120      1.6518    0.493044
  585      15     18854     18839   128.734       152   0.0676967     0.49298
  586      16     18883     18867   128.705       112     0.22075    0.492987
  587      16     18915     18899   128.704       128    0.467729    0.493062
  588      16     18955     18939   128.757       160    0.340188    0.492902
  589      16     18990     18974   128.776       140   0.0728635    0.492743
  590      16     19029     19013   128.822       156   0.0705404    0.492615
  591      16     19060     19044   128.813       124    0.175871    0.492696
  592      16     19092     19076   128.812       128    0.218337    0.492612
  593      16     19125     19109   128.817       132   0.0757147    0.492674
  594      16     19156     19140   128.809       124     1.07111    0.492724
  595      16     19195     19179   128.854       156    0.481125    0.492541
  596      16     19225     19209    128.84       120    0.337741     0.49251
  597      16     19256     19240   128.831       124   0.0444039    0.492556
  598      16     19286     19270   128.817       120     1.44237    0.492705
  599      16     19318     19302   128.815       128    0.321825    0.492744
2020-03-09 14:38:05.738163 min lat: 0.00758844 max lat: 2.31737 avg lat: 0.49274
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
  600      16     19337     19321   128.727        76   0.0763224     0.49274
Total time run:       600.554
Total reads made:     19337
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.794
Average IOPS:         32
Stddev IOPS:          2.82388
Max IOPS:             41
Min IOPS:             19
Average Latency(s):   0.493164
Max latency(s):       2.31737
Min latency(s):       0.00758844

Total time run:       600.554
Total reads made:     19337
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   128.794
Average IOPS:         32
Stddev IOPS:          2.82388
Max IOPS:             41
Min IOPS:             19
Average Latency(s):   0.493164
Max latency(s):       2.31737
Min latency(s):       0.00758844

Cleanup, pool with 256 PGs and remove it

[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 rados -p testbench cleanup
Removed 20649 objects
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench testbench --yes-i-really-really-mean-it
Error EPERM: pool deletion is disabled; you must first set the mon_allow_pool_delete config option to true before you can destroy a pool
Error: non zero exit code: 1: OCI runtime error
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=true
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'true' 
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'true' 
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'true' 
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph osd pool delete testbench testbench --yes-i-really-really-mean-it
pool 'testbench' removed
[root@f5-422-01 ~]# podman exec ceph-mon-f5-422-01 ceph tell mon.* injectargs --mon-allow-pool-delete=false
mon.f5-422-01: injectargs:mon_allow_pool_delete = 'false' 
mon.f5-422-02: injectargs:mon_allow_pool_delete = 'false' 
mon.f5-422-03: injectargs:mon_allow_pool_delete = 'false' 

libvirt and Ceph RBD Performance Baseline

Before I start fiddling any tunables, I want to have a baseline also from the RBD consumer side.

Prepare Pool

I created a replica 3 pool with 256 PGs with the Application set to rbd.

Prepare libvirt

Since one can use libvirt with Ceph RBD, I created 2 vols on the libvirt side. Note that I set up libvirt with a pool.

root@hypervisor ~ # virsh vol-list --pool Ceph-HouseNet-libvirt-pool
 Name                 Path
------------------------------------------------------------------------------
 20200309-fio-testing-1TiB libvirt-pool/20200309-fio-testing-1TiB

Prepare Test VM

Attached the 500 GiB RBD (20200309-banana) to an existing VM as a virtio block device, formatted it with XFS, mounted it and ran fio.

[root@centos-stream ~]# lsblk
NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                   11:0    1 1024M  0 rom  
vda                  252:0    0    1G  0 disk
└─vda1               252:1    0 1023M  0 part /boot
vdb                  252:16   0   30G  0 disk
└─vdb1               252:17   0   30G  0 part
  ├─VG_cs-LV_root    253:0    0    8G  0 lvm  /
  ├─VG_cs-LV_swap    253:1    0  3.1G  0 lvm  [SWAP]
  ├─VG_cs-LV_home    253:2    0    1G  0 lvm  /home
  └─VG_cs-LV_var_log 253:3    0    1G  0 lvm  /var/log
vdc                  252:32   0    1T  0 disk 
[root@centos-stream ~]# mkfs.xfs /dev/vdc
meta-data=/dev/vdc               isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@centos-stream ~]# mount /dev/vdc /mnt/
[root@centos-stream ~]# df -h /mnt/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdc        1.0T  7.2G 1017G   1% /mnt
[root@centos-stream ~]# mkdir /mnt/fiotest/

Sequential Read, 4k Blocksize

  • READ: bw=170MiB/s
  • iops: avg=43.6k
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4k
readtest: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
readtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=184MiB/s,w=0KiB/s][r=47.1k,w=0 IOPS][eta 00m:00s]
readtest: (groupid=0, jobs=1): err= 0: pid=3140: Mon Mar  9 18:20:21 2020
   read: IOPS=43.6k, BW=170MiB/s (179MB/s)(160GiB/962069msec)
    clat (nsec): min=1050, max=590644k, avg=21330.22, stdev=839795.46
     lat (nsec): min=1390, max=590645k, avg=21688.15, stdev=839795.50
    clat percentiles (nsec):
     |  1.00th=[    1112],  5.00th=[    1144], 10.00th=[    1160],
     | 20.00th=[    1176], 30.00th=[    1224], 40.00th=[    1240],
     | 50.00th=[    1288], 60.00th=[    1528], 70.00th=[    3728],
     | 80.00th=[    3856], 90.00th=[    4080], 95.00th=[    4320],
     | 99.00th=[    5472], 99.50th=[   10432], 99.90th=[ 3424256],
     | 99.95th=[15007744], 99.99th=[34865152]
   bw (  KiB/s): min= 8192, max=249856, per=99.98%, avg=174358.81, stdev=35849.34, samples=1924
   iops        : min= 2048, max=62464, avg=43589.69, stdev=8962.33, samples=1924
  lat (usec)   : 2=64.16%, 4=22.50%, 10=12.81%, 20=0.37%, 50=0.03%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.03%, 10=0.02%, 20=0.03%, 50=0.03%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=3.36%, sys=16.00%, ctx=51046, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=41943040,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=170MiB/s (179MB/s), 170MiB/s-170MiB/s (179MB/s-179MB/s), io=160GiB (172GB), run=962069-962069msec

Disk stats (read/write):
  vdc: ios=166533/7, merge=0/0, ticks=9340283/505, in_queue=9257463, util=19.25%

Sequential Read, 4m Blocksize

  • READ: bw=172MiB/s
  • iops: avg=43
fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=readtest --rw=read --size=160g --directory=/mnt/fiotest/ --bs=4m
readtest: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][99.9%][r=160MiB/s,w=0KiB/s][r=40,w=0 IOPS][eta 00m:01s] 
readtest: (groupid=0, jobs=1): err= 0: pid=3347: Mon Mar  9 18:41:54 2020
   read: IOPS=43, BW=172MiB/s (181MB/s)(160GiB/949850msec)
    clat (usec): min=407, max=595494, avg=23178.72, stdev=29947.47
     lat (usec): min=408, max=595494, avg=23179.67, stdev=29947.41
    clat percentiles (usec):
     |  1.00th=[   449],  5.00th=[   537], 10.00th=[   685], 20.00th=[  1614],
     | 30.00th=[  2638], 40.00th=[  5800], 50.00th=[ 12780], 60.00th=[ 20579],
     | 70.00th=[ 31589], 80.00th=[ 41681], 90.00th=[ 58983], 95.00th=[ 74974],
     | 99.00th=[114820], 99.50th=[139461], 99.90th=[316670], 99.95th=[383779],
     | 99.99th=[463471]
   bw (  KiB/s): min=24576, max=294912, per=99.98%, avg=176603.21, stdev=34480.37, samples=1899
   iops        : min=    6, max=   72, avg=43.09, stdev= 8.41, samples=1899
  lat (usec)   : 500=3.89%, 750=7.06%, 1000=1.85%
  lat (msec)   : 2=12.08%, 4=11.78%, 10=9.67%, 20=13.01%, 50=26.31%
  lat (msec)   : 100=12.57%, 250=1.61%, 500=0.18%, 750=0.01%
  cpu          : usr=0.05%, sys=7.76%, ctx=41548, majf=0, minf=524
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=40960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=160GiB (172GB), run=949850-949850msec

Disk stats (read/write):
  vdc: ios=166571/0, merge=18/0, ticks=9225180/0, in_queue=9141512, util=19.43%

Sequential Write, 4k Blocksize, 1st run

  • WRITE: bw=113MiB/s
  • iops: avg=28.8k
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
writetest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=111MiB/s][r=0,w=28.4k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=3500: Mon Mar  9 19:07:39 2020
  write: IOPS=28.8k, BW=113MiB/s (118MB/s)(160GiB/1454565msec)
    clat (usec): min=2, max=208014, avg=33.00, stdev=612.11
     lat (usec): min=2, max=208015, avg=33.38, stdev=612.11
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    6], 90.00th=[    6], 95.00th=[    7],
     | 99.00th=[   13], 99.50th=[   19], 99.90th=[ 7898], 99.95th=[ 7963],
     | 99.99th=[11863]
   bw (  KiB/s): min=   16, max=922760, per=99.98%, avg=115323.40, stdev=32338.49, samples=2909
   iops        : min=    4, max=230690, avg=28830.84, stdev=8084.64, samples=2909
  lat (usec)   : 4=76.54%, 10=22.10%, 20=0.89%, 50=0.08%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.34%, 20=0.03%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=2.58%, sys=12.15%, ctx=158764, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=113MiB/s (118MB/s), 113MiB/s-113MiB/s (118MB/s-118MB/s), io=160GiB (172GB), run=1454565-1454565msec

Disk stats (read/write):
  vdc: ios=0/168080, merge=0/7, ticks=0/275742333, in_queue=275658313, util=1.66%

Sequential Write, 4k Blocksize, 2nd run

  • WRITE: bw=140MiB/s
  • iops: avg=35.8k
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4k
writetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=140MiB/s][r=0,w=35.7k IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=3855: Mon Mar  9 19:43:04 2020
  write: IOPS=35.8k, BW=140MiB/s (147MB/s)(160GiB/1170655msec)
    clat (usec): min=2, max=35949, avg=26.17, stdev=512.53
     lat (usec): min=2, max=35949, avg=26.55, stdev=512.53
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    4], 80.00th=[    6], 90.00th=[    6], 95.00th=[    7],
     | 99.00th=[   12], 99.50th=[   17], 99.90th=[11600], 99.95th=[13435],
     | 99.99th=[15270]
   bw (  KiB/s): min=42064, max=944664, per=100.00%, avg=143469.78, stdev=27523.62, samples=2338
   iops        : min=10516, max=236166, avg=35867.42, stdev=6880.91, samples=2338
  lat (usec)   : 4=70.21%, 10=28.56%, 20=0.91%, 50=0.11%, 100=0.01%
  lat (usec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.09%, 20=0.12%, 50=0.01%
  cpu          : usr=3.24%, sys=16.08%, ctx=88932, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,41943040,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=160GiB (172GB), run=1170655-1170655msec

Disk stats (read/write):
  vdc: ios=0/165401, merge=0/2, ticks=0/221847866, in_queue=221764979, util=2.00%

Sequential Write, 4m Blocksize, 1st run

  • WRITE: bw=138MiB/s
  • iops: avg=34
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=152MiB/s][r=0,w=38 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4084: Mon Mar  9 20:06:52 2020
  write: IOPS=34, BW=138MiB/s (144MB/s)(160GiB/1191017msec)
    clat (usec): min=1400, max=57630, avg=28903.19, stdev=6483.51
     lat (usec): min=1522, max=57763, avg=29031.47, stdev=6482.47
    clat percentiles (usec):
     |  1.00th=[11338],  5.00th=[16450], 10.00th=[17695], 20.00th=[25035],
     | 30.00th=[27132], 40.00th=[27657], 50.00th=[30016], 60.00th=[31589],
     | 70.00th=[33424], 80.00th=[33817], 90.00th=[35390], 95.00th=[35914],
     | 99.00th=[39584], 99.50th=[39584], 99.90th=[43779], 99.95th=[46400],
     | 99.99th=[51119]
   bw (  KiB/s): min=114459, max=1294336, per=100.00%, avg=141025.75, stdev=34335.39, samples=2379
   iops        : min=   27, max=  316, avg=34.39, stdev= 8.39, samples=2379
  lat (msec)   : 2=0.19%, 4=0.72%, 10=0.08%, 20=12.65%, 50=86.35%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.36%, sys=7.24%, ctx=100250, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=138MiB/s (144MB/s), 138MiB/s-138MiB/s (144MB/s-144MB/s), io=160GiB (172GB), run=1191017-1191017msec

Disk stats (read/write):
  vdc: ios=0/164711, merge=0/2, ticks=0/225940838, in_queue=225858476, util=1.88%

Sequential Write, 4m Blocksize, 2nd run

  • WRITE: bw=135MiB/s
  • iops: avg=33
fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=writetest --rw=write --size=160g --directory=/mnt/fiotest/ --bs=4m
writetest: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=132MiB/s][r=0,w=33 IOPS][eta 00m:00s]
writetest: (groupid=0, jobs=1): err= 0: pid=4308: Mon Mar  9 20:32:40 2020
  write: IOPS=33, BW=135MiB/s (142MB/s)(160GiB/1213647msec)
    clat (usec): min=1404, max=109269, avg=29455.52, stdev=5986.18
     lat (usec): min=1547, max=109383, avg=29584.63, stdev=5985.02
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[17695], 10.00th=[23987], 20.00th=[26870],
     | 30.00th=[27132], 40.00th=[27132], 50.00th=[29754], 60.00th=[31589],
     | 70.00th=[33424], 80.00th=[35390], 90.00th=[35390], 95.00th=[35914],
     | 99.00th=[39584], 99.50th=[39584], 99.90th=[44827], 99.95th=[68682],
     | 99.99th=[91751]
   bw (  KiB/s): min=49152, max=1449984, per=100.00%, avg=138391.61, stdev=38375.99, samples=2424
   iops        : min=   12, max=  354, avg=33.75, stdev= 9.37, samples=2424
  lat (msec)   : 2=0.47%, 4=0.42%, 10=0.03%, 20=5.69%, 50=93.32%
  lat (msec)   : 100=0.06%, 250=0.01%
  cpu          : usr=0.35%, sys=7.21%, ctx=119939, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,40960,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=160GiB (172GB), run=1213647-1213647msec

Disk stats (read/write):
  vdc: ios=0/164742, merge=0/2, ticks=0/230335859, in_queue=230253272, util=1.88%

Sequential, 80% Read, 20% Write, 4k Blocksize, 1st run

  • READ: bw=133MiB/s
  • iops: avg=34.0k
  • WRITE: bw=33.2MiB/s
  • iops: avg=8503
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/
 --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
read80write20test: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [M(1)][100.0%][r=144MiB/s,w=36.4MiB/s][r=36.9k,w=9325 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5025: Mon Mar  9 21:39:08 2020
   read: IOPS=34.0k, BW=133MiB/s (139MB/s)(128GiB/986447msec)
    clat (nsec): min=1030, max=583009k, avg=26272.54, stdev=1289481.27
     lat (nsec): min=1370, max=583009k, avg=26627.44, stdev=1289481.31
    clat percentiles (nsec):
     |  1.00th=[    1128],  5.00th=[    1160], 10.00th=[    1176],
     | 20.00th=[    1208], 30.00th=[    1240], 40.00th=[    1288],
     | 50.00th=[    1368], 60.00th=[    1576], 70.00th=[    3760],
     | 80.00th=[    3952], 90.00th=[    4192], 95.00th=[    4448],
     | 99.00th=[    5984], 99.50th=[   11072], 99.90th=[ 2441216],
     | 99.95th=[14876672], 99.99th=[49020928]
   bw (  KiB/s): min= 4096, max=245760, per=100.00%, avg=136100.29, stdev=55704.13, samples=1971
   iops        : min= 1024, max=61440, avg=34025.06, stdev=13926.03, samples=1971
  write: IOPS=8503, BW=33.2MiB/s (34.8MB/s)(31.0GiB/986447msec)
    clat (nsec): min=1790, max=12100k, avg=4191.77, stdev=15119.81
     lat (usec): min=2, max=12100, avg= 4.57, stdev=15.13
    clat percentiles (nsec):
     |  1.00th=[ 2008],  5.00th=[ 2320], 10.00th=[ 2448], 20.00th=[ 2544],
     | 30.00th=[ 2608], 40.00th=[ 2768], 50.00th=[ 2992], 60.00th=[ 3696],
     | 70.00th=[ 5472], 80.00th=[ 5920], 90.00th=[ 6560], 95.00th=[ 7584],
     | 99.00th=[12736], 99.50th=[16512], 99.90th=[25728], 99.95th=[30080],
     | 99.99th=[41216]
   bw (  KiB/s): min=  880, max=63016, per=100.00%, avg=34021.30, stdev=13952.04, samples=1971
   iops        : min=  220, max=15754, avg=8505.31, stdev=3488.01, samples=1971
  lat (usec)   : 2=52.62%, 4=26.23%, 10=20.35%, 20=0.59%, 50=0.10%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=3.57%, sys=16.91%, ctx=40056, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=128GiB (137GB), run=986447-986447msec
  WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=31.0GiB (34.4GB), run=986447-986447msec

Disk stats (read/write):
  vdc: ios=132917/33058, merge=0/11, ticks=9237406/9508567, in_queue=18662956, util=15.87%

Sequential, 80% Read, 20% Write, 4k Blocksize, 2nd run

  • READ: bw=131MiB/s
  • iops: avg=33.5k
  • WRITE: bw=32.7MiB/s
  • iops: avg=8372
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4k
read80write20test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=89.0MiB/s,w=22.6MiB/s][r=22.8k,w=5798 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5195: Mon Mar  9 21:58:36 2020
   read: IOPS=33.5k, BW=131MiB/s (137MB/s)(128GiB/1001856msec)
    clat (nsec): min=1040, max=504078k, avg=26745.04, stdev=1301910.00
     lat (nsec): min=1380, max=504078k, avg=27099.54, stdev=1301913.08
    clat percentiles (nsec):
     |  1.00th=[    1112],  5.00th=[    1144], 10.00th=[    1176],
     | 20.00th=[    1208], 30.00th=[    1240], 40.00th=[    1272],
     | 50.00th=[    1336], 60.00th=[    1432], 70.00th=[    1720],
     | 80.00th=[    3888], 90.00th=[    4192], 95.00th=[    4448],
     | 99.00th=[    5984], 99.50th=[   10176], 99.90th=[ 2637824],
     | 99.95th=[15269888], 99.99th=[51118080]
   bw (  KiB/s): min= 4096, max=245760, per=100.00%, avg=134179.48, stdev=55187.04, samples=2000
   iops        : min= 1024, max=61440, avg=33544.87, stdev=13796.76, samples=2000
  write: IOPS=8372, BW=32.7MiB/s (34.3MB/s)(31.0GiB/1001856msec)
    clat (nsec): min=1790, max=12108k, avg=3939.87, stdev=8882.46
     lat (usec): min=2, max=12108, avg= 4.33, stdev= 9.56
    clat percentiles (nsec):
     |  1.00th=[ 1976],  5.00th=[ 2320], 10.00th=[ 2416], 20.00th=[ 2512],
     | 30.00th=[ 2576], 40.00th=[ 2672], 50.00th=[ 2832], 60.00th=[ 3216],
     | 70.00th=[ 4128], 80.00th=[ 5792], 90.00th=[ 6432], 95.00th=[ 7520],
     | 99.00th=[12608], 99.50th=[16512], 99.90th=[24960], 99.95th=[29312],
     | 99.99th=[39680]
   bw (  KiB/s): min=  992, max=63360, per=100.00%, avg=33540.82, stdev=13817.76, samples=2000
   iops        : min=  248, max=15840, avg=8385.20, stdev=3454.44, samples=2000
  lat (usec)   : 2=58.41%, 4=23.03%, 10=17.80%, 20=0.56%, 50=0.09%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.02%, 20=0.02%, 50=0.02%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  cpu          : usr=3.53%, sys=15.99%, ctx=39306, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=33555281,8387759,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=128GiB (137GB), run=1001856-1001856msec
  WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=31.0GiB (34.4GB), run=1001856-1001856msec

Disk stats (read/write):
  vdc: ios=132838/33023, merge=0/12, ticks=9344607/9568626, in_queue=18830124, util=15.68%

Sequential, 80% Read, 20% Write, 4m Blocksize, 1st run

  • READ: bw=129MiB/s
  • iops: avg=32
  • WRITE: bw=32.7MiB/s
  • iops: avg=8
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=176MiB/s,w=56.1MiB/s][r=44,w=14 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5355: Mon Mar  9 22:16:49 2020
   read: IOPS=32, BW=129MiB/s (135MB/s)(128GiB/1012922msec)
    clat (usec): min=405, max=646248, avg=30349.54, stdev=50502.76
     lat (usec): min=405, max=646249, avg=30350.45, stdev=50502.71
    clat percentiles (usec):
     |  1.00th=[   449],  5.00th=[   562], 10.00th=[   734], 20.00th=[  1565],
     | 30.00th=[  2311], 40.00th=[  4621], 50.00th=[ 12518], 60.00th=[ 22152],
     | 70.00th=[ 34341], 80.00th=[ 47973], 90.00th=[ 73925], 95.00th=[106431],
     | 99.00th=[270533], 99.50th=[350225], 99.90th=[450888], 99.95th=[484443],
     | 99.99th=[557843]
   bw (  KiB/s): min= 8175, max=245760, per=100.00%, avg=132405.29, stdev=56654.42, samples=2021
   iops        : min=    1, max=   60, avg=32.30, stdev=13.83, samples=2021
  write: IOPS=8, BW=32.7MiB/s (34.3MB/s)(32.3GiB/1012922msec)
    clat (usec): min=838, max=13117, avg=2202.18, stdev=882.87
     lat (usec): min=883, max=13231, avg=2292.22, stdev=886.68
    clat percentiles (usec):
     |  1.00th=[ 1106],  5.00th=[ 1303], 10.00th=[ 1352], 20.00th=[ 1434],
     | 30.00th=[ 1500], 40.00th=[ 1647], 50.00th=[ 1893], 60.00th=[ 2245],
     | 70.00th=[ 2606], 80.00th=[ 2999], 90.00th=[ 3556], 95.00th=[ 3884],
     | 99.00th=[ 4490], 99.50th=[ 4686], 99.90th=[ 5145], 99.95th=[ 5604],
     | 99.99th=[13173]
   bw (  KiB/s): min= 8078, max=139264, per=100.00%, avg=36216.64, stdev=22038.23, samples=1870
   iops        : min=    1, max=   34, avg= 8.82, stdev= 5.38, samples=1870
  lat (usec)   : 500=2.16%, 750=5.95%, 1000=1.86%
  lat (msec)   : 2=22.01%, 4=18.53%, 10=7.24%, 20=8.78%, 50=18.39%
  lat (msec)   : 100=10.62%, 250=3.49%, 500=0.94%, 750=0.03%
  cpu          : usr=0.12%, sys=7.75%, ctx=32560, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=128GiB (137GB), run=1012922-1012922msec
  WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=32.3GiB (34.7GB), run=1012922-1012922msec

Disk stats (read/write):
  vdc: ios=132738/33417, merge=15/0, ticks=9532728/11600214, in_queue=21049821, util=15.38%

Sequential, 80% Read, 20% Write, 4m Blocksize, 2nd run

  • READ: bw=128MiB/s
  • iops: avg=32
  • WRITE: bw=32.5MiB/s
  • iops: avg=8
fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
[root@centos-stream ~]# fio --name=read80write20test --rw=readwrite --rwmixread=80 --size=160g --directory=/mnt/fiotest/ --bs=4m
read80write20test: (g=0): rw=rw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [M(1)][100.0%][r=144MiB/s,w=44.0MiB/s][r=36,w=11 IOPS][eta 00m:00s]
read80write20test: (groupid=0, jobs=1): err= 0: pid=5539: Mon Mar  9 22:36:54 2020
   read: IOPS=32, BW=128MiB/s (135MB/s)(128GiB/1017899msec)
    clat (usec): min=415, max=722288, avg=30504.65, stdev=51268.90
     lat (usec): min=416, max=722289, avg=30505.49, stdev=51268.86
    clat percentiles (usec):
     |  1.00th=[   453],  5.00th=[   553], 10.00th=[   758], 20.00th=[  1582],
     | 30.00th=[  2311], 40.00th=[  4424], 50.00th=[ 12649], 60.00th=[ 22414],
     | 70.00th=[ 34341], 80.00th=[ 47973], 90.00th=[ 74974], 95.00th=[106431],
     | 99.00th=[270533], 99.50th=[350225], 99.90th=[480248], 99.95th=[517997],
     | 99.99th=[616563]
   bw (  KiB/s): min= 8192, max=245760, per=100.00%, avg=131905.37, stdev=56989.64, samples=2029
   iops        : min=    2, max=   60, avg=32.18, stdev=13.92, samples=2029
  write: IOPS=8, BW=32.5MiB/s (34.1MB/s)(32.3GiB/1017899msec)
    clat (usec): min=827, max=20250, avg=2193.92, stdev=953.01
     lat (usec): min=869, max=20452, avg=2282.39, stdev=956.96
    clat percentiles (usec):
     |  1.00th=[  898],  5.00th=[ 1090], 10.00th=[ 1336], 20.00th=[ 1418],
     | 30.00th=[ 1483], 40.00th=[ 1647], 50.00th=[ 1893], 60.00th=[ 2245],
     | 70.00th=[ 2606], 80.00th=[ 2999], 90.00th=[ 3556], 95.00th=[ 3949],
     | 99.00th=[ 4555], 99.50th=[ 4686], 99.90th=[ 6128], 99.95th=[10290],
     | 99.99th=[20317]
   bw (  KiB/s): min= 8143, max=155648, per=100.00%, avg=36108.79, stdev=22228.68, samples=1876
   iops        : min=    1, max=   38, avg= 8.78, stdev= 5.43, samples=1876
  lat (usec)   : 500=2.29%, 750=5.60%, 1000=2.45%
  lat (msec)   : 2=21.33%, 4=18.91%, 10=7.06%, 20=8.78%, 50=18.43%
  lat (msec)   : 100=10.68%, 250=3.52%, 500=0.91%, 750=0.06%
  cpu          : usr=0.10%, sys=7.70%, ctx=32782, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=32688,8272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=128MiB/s (135MB/s), 128MiB/s-128MiB/s (135MB/s-135MB/s), io=128GiB (137GB), run=1017899-1017899msec
  WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=32.3GiB (34.7GB), run=1017899-1017899msec

Disk stats (read/write):
  vdc: ios=132648/33381, merge=15/0, ticks=9582613/11738047, in_queue=21237573, util=15.26%

sequential write, 2300 bytes, fsync

Normally this test is not part of my baseline but a colleague inquired, so pasting here.

  • WRITE: bw=232KiB/s
  • iops: avg=103
fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
[root@centos-stream ~]# fio --rw=write --ioengine=sync --fdatasync=1 --size=22m --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 22MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=242KiB/s][r=0,w=108 IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=24595: Wed Mar 11 09:56:35 2020
  write: IOPS=103, BW=232KiB/s (237kB/s)(21.0MiB/97165msec)
    clat (usec): min=6, max=175, avg=15.50, stdev= 6.16
     lat (usec): min=7, max=191, avg=16.18, stdev= 6.22
    clat percentiles (nsec):
     |  1.00th=[ 8256],  5.00th=[ 8896], 10.00th=[ 9152], 20.00th=[ 9408],
     | 30.00th=[ 9792], 40.00th=[10304], 50.00th=[17536], 60.00th=[19072],
     | 70.00th=[19840], 80.00th=[20352], 90.00th=[21376], 95.00th=[22656],
     | 99.00th=[31360], 99.50th=[33536], 99.90th=[43264], 99.95th=[47360],
     | 99.99th=[93696]
   bw (  KiB/s): min=   53, max=  260, per=100.00%, avg=231.48, stdev=21.29, samples=194
   iops        : min=   24, max=  116, avg=103.21, stdev= 9.46, samples=194
  lat (usec)   : 10=35.28%, 20=38.59%, 50=26.08%, 100=0.04%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (msec): min=3, max=204, avg= 9.67, stdev= 6.68
    sync percentiles (msec):
     |  1.00th=[    5],  5.00th=[    5], 10.00th=[    6], 20.00th=[    6],
     | 30.00th=[    6], 40.00th=[    7], 50.00th=[   11], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   12], 90.00th=[   13], 95.00th=[   16],
     | 99.00th=[   36], 99.50th=[   40], 99.90th=[   66], 99.95th=[   94],
     | 99.99th=[  203]
  cpu          : usr=0.06%, sys=0.45%, ctx=35706, majf=0, minf=12
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=232KiB/s (237kB/s), 232KiB/s-232KiB/s (237kB/s-237kB/s), io=21.0MiB (23.1MB), run=97165-97165msec

Disk stats (read/write):
  vdc: ios=0/25676, merge=0/8, ticks=0/97623, in_queue=89570, util=13.10%

sequential write, 2300 bytes, NO fsync

Since it seems sily to me to call fsync after 2300 bytes, same test without that option to have a comparison.

  • WRITE: bw=108MiB/s
  • iops: avg=49.4k
fio --rw=write --ioengine=sync  --size=160g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
[root@centos-stream ~]# fio --rw=write --ioengine=sync  --size=160g --bs=2300 --name=etcdtest --directory=/mnt/fiotest/
etcdtest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.7
Starting 1 process
etcdtest: Laying out IO file (1 file / 163840MiB)
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=113MiB/s][r=0,w=51.4k IOPS][eta 00m:00s]
etcdtest: (groupid=0, jobs=1): err= 0: pid=24660: Wed Mar 11 10:24:51 2020
  write: IOPS=49.4k, BW=108MiB/s (114MB/s)(160GiB/1512486msec)
    clat (nsec): min=1560, max=207919k, avg=18588.48, stdev=418983.46
     lat (nsec): min=1910, max=207919k, avg=18957.71, stdev=418983.68
    clat percentiles (nsec):
     |  1.00th=[    1624],  5.00th=[    1640], 10.00th=[    1656],
     | 20.00th=[    1672], 30.00th=[    1704], 40.00th=[    1768],
     | 50.00th=[    2800], 60.00th=[    2928], 70.00th=[    3216],
     | 80.00th=[    5088], 90.00th=[    5920], 95.00th=[    6496],
     | 99.00th=[   10176], 99.50th=[   14784], 99.90th=[ 6979584],
     | 99.95th=[ 7766016], 99.99th=[11862016]
   bw (  KiB/s): min=  247, max=405042, per=99.97%, avg=110893.31, stdev=28246.21, samples=3024
   iops        : min=  110, max=180332, avg=49371.82, stdev=12575.70, samples=3024
  lat (usec)   : 2=42.13%, 4=35.25%, 10=21.60%, 20=0.75%, 50=0.07%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.15%, 20=0.04%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  cpu          : usr=4.16%, sys=19.94%, ctx=143227, majf=1, minf=19
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,74695083,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=108MiB/s (114MB/s), 108MiB/s-108MiB/s (114MB/s-114MB/s), io=160GiB (172GB), run=1512486-1512486msec

Disk stats (read/write):
  vdc: ios=0/169867, merge=0/8, ticks=0/286486813, in_queue=286401832, util=1.66%