[2024-11-20 01:59:20.084374] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-11-20 01:59:20.084603] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3490850 ] Set seed to 1732064360080888 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-20 01:59:20.184388] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-20 01:59:20.223979] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-20 01:59:20.224026] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3490850' to capture a snapshot of events at runtime. [2024-11-20 01:59:20.224039] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3490850 for offline analysis/debug. [2024-11-20 01:59:20.224082] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Wed Nov 20 01:59:24 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/20-01:59:31 ... thread 0: (1000000,1000000) ops and (134245.2,134245.2) ops/second in (7.449058,7.449058) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 9/4 1016.83 MB 63.9 0.3 0.0 0.3 1.1 0.8 0.0 1.3 29.8 123.4 8.75 3.41 10 0.875 265K 251 0.0 0.0 L1 3/1 204.62 MB 7.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 9/9 1.13 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 27/8 2.03 GB 1.2 2.2 1.5 0.6 2.1 1.5 0.0 1.4 106.4 105.1 20.71 10.08 6 3.452 1308K 0 0.0 0.0 L4 199/23 12.44 GB 1.2 3.6 2.6 0.9 3.5 2.6 0.0 1.3 105.2 104.3 34.75 16.90 25 1.390 1766K 443 0.0 0.0 L5 1064/0 88.03 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1311/45 104.83 GB 0.0 6.0 4.1 1.8 6.7 4.9 0.0 8.4 95.3 107.1 64.22 30.39 41 1.566 3340K 694 0.0 0.0 Int 0/0 0.00 KB 0.0 6.0 4.1 1.8 6.7 4.9 0.0 8.4 95.3 107.1 64.22 30.39 41 1.566 3340K 694 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.0 4.1 1.8 5.9 4.1 0.0 0.0 106.7 105.7 57.34 28.03 32 1.792 3340K 694 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 119.0 6.88 2.36 9 0.765 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.6 total, 7.5 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.72 GB write, 900.99 MB/s write, 5.98 GB read, 801.45 MB/s read, 64.2 seconds Interval compaction: 6.72 GB write, 923.47 MB/s write, 5.98 GB read, 821.45 MB/s read, 64.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 24, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 24, interval: 24 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 147802 Average: 10.6326 StdDev: 142.64 Min: 1 Median: 1.8045 Max: 8898 Percentiles: P50: 1.80 P75: 2.86 P99: 9.97 P99.9: 2314.98 P99.99: 5630.47 ------------------------------------------------------ [ 0, 1 ] 10209 6.907% 6.907% # ( 1, 2 ] 79173 53.567% 60.474% ########### ( 2, 3 ] 24905 16.850% 77.324% ### ( 3, 4 ] 14376 9.727% 87.051% ## ( 4, 6 ] 14661 9.919% 96.970% ## ( 6, 10 ] 3022 2.045% 99.015% ( 10, 15 ] 607 0.411% 99.426% ( 15, 22 ] 57 0.039% 99.464% ( 22, 34 ] 4 0.003% 99.467% ( 34, 51 ] 3 0.002% 99.469% ( 51, 76 ] 8 0.005% 99.474% ( 76, 110 ] 1 0.001% 99.475% ( 110, 170 ] 4 0.003% 99.478% ( 170, 250 ] 20 0.014% 99.491% ( 250, 380 ] 31 0.021% 99.512% ( 380, 580 ] 93 0.063% 99.575% ( 580, 870 ] 179 0.121% 99.696% ( 870, 1300 ] 164 0.111% 99.807% ( 1300, 1900 ] 106 0.072% 99.879% ( 1900, 2900 ] 80 0.054% 99.933% ( 2900, 4400 ] 70 0.047% 99.980% ( 4400, 6600 ] 29 0.020% 100.000% ( 6600, 9900 ] 4 0.003% 100.003% ** Level 1 read latency histogram (micros): Count: 8411 Average: 54.9855 StdDev: 394.00 Min: 1 Median: 2.0646 Max: 8760 Percentiles: P50: 2.06 P75: 3.23 P99: 1781.74 P99.9: 5816.43 P99.99: 8512.18 ------------------------------------------------------ [ 0, 1 ] 221 2.628% 2.628% # ( 1, 2 ] 3852 45.797% 48.425% ######### ( 2, 3 ] 2051 24.385% 72.809% ##### ( 3, 4 ] 797 9.476% 82.285% ## ( 4, 6 ] 918 10.914% 93.199% ## ( 6, 10 ] 228 2.711% 95.910% # ( 10, 15 ] 72 0.856% 96.766% ( 15, 22 ] 11 0.131% 96.897% ( 22, 34 ] 1 0.012% 96.909% ( 51, 76 ] 1 0.012% 96.921% ( 110, 170 ] 4 0.048% 96.968% ( 170, 250 ] 4 0.048% 97.016% ( 250, 380 ] 20 0.238% 97.254% ( 380, 580 ] 21 0.250% 97.503% ( 580, 870 ] 52 0.618% 98.122% ( 870, 1300 ] 49 0.583% 98.704% ( 1300, 1900 ] 31 0.369% 99.073% ( 1900, 2900 ] 30 0.357% 99.429% ( 2900, 4400 ] 28 0.333% 99.762% ( 4400, 6600 ] 18 0.214% 99.976% ( 6600, 9900 ] 2 0.024% 100.000% ** Level 2 read latency histogram (micros): Count: 522224 Average: 14.5276 StdDev: 172.29 Min: 1 Median: 1.8871 Max: 7859 Percentiles: P50: 1.89 P75: 2.98 P99: 13.20 P99.9: 1880.42 P99.99: 2766.88 ------------------------------------------------------ [ 0, 1 ] 27091 5.188% 5.188% # ( 1, 2 ] 263815 50.518% 55.705% ########## ( 2, 3 ] 102500 19.628% 75.333% #### ( 3, 4 ] 52689 10.089% 85.422% ## ( 4, 6 ] 54434 10.423% 95.846% ## ( 6, 10 ] 14162 2.712% 98.558% # ( 10, 15 ] 3615 0.692% 99.250% ( 15, 22 ] 495 0.095% 99.345% ( 22, 34 ] 73 0.014% 99.359% ( 34, 51 ] 44 0.008% 99.367% ( 51, 76 ] 45 0.009% 99.376% ( 76, 110 ] 27 0.005% 99.381% ( 110, 170 ] 51 0.010% 99.390% ( 170, 250 ] 87 0.017% 99.407% ( 250, 380 ] 208 0.040% 99.447% ( 380, 580 ] 405 0.078% 99.525% ( 580, 870 ] 650 0.124% 99.649% ( 870, 1300 ] 741 0.142% 99.791% ( 1300, 1900 ] 589 0.113% 99.904% ( 1900, 2900 ] 520 0.100% 100.003% ( 2900, 4400 ] 447 0.086% 100.089% ( 4400, 6600 ] 132 0.025% 100.114% ( 6600, 9900 ] 17 0.003% 100.117% ** Level 3 read latency histogram (micros): Count: 740452 Average: 11.8561 StdDev: 157.07 Min: 1 Median: 1.9029 Max: 9262 Percentiles: P50: 1.90 P75: 3.00 P99: 9.96 P99.9: 1060.05 P99.99: 1543.83 ------------------------------------------------------ [ 0, 1 ] 27443 3.706% 3.706% # ( 1, 2 ] 379647 51.272% 54.979% ########## ( 2, 3 ] 148933 20.114% 75.092% #### ( 3, 4 ] 77912 10.522% 85.615% ## ( 4, 6 ] 81646 11.027% 96.641% ## ( 6, 10 ] 17655 2.384% 99.025% ( 10, 15 ] 4104 0.554% 99.580% ( 15, 22 ] 480 0.065% 99.645% ( 22, 34 ] 68 0.009% 99.654% ( 34, 51 ] 38 0.005% 99.659% ( 51, 76 ] 63 0.009% 99.667% ( 76, 110 ] 22 0.003% 99.670% ( 110, 170 ] 42 0.006% 99.676% ( 170, 250 ] 80 0.011% 99.687% ( 250, 380 ] 164 0.022% 99.709% ( 380, 580 ] 403 0.054% 99.763% ( 580, 870 ] 675 0.091% 99.855% ( 870, 1300 ] 766 0.103% 99.958% ( 1300, 1900 ] 588 0.079% 100.037% ( 1900, 2900 ] 517 0.070% 100.107% ( 2900, 4400 ] 513 0.069% 100.177% ( 4400, 6600 ] 167 0.023% 100.199% ( 6600, 9900 ] 27 0.004% 100.203% ** Level 4 read latency histogram (micros): Count: 330045 Average: 28.8500 StdDev: 262.25 Min: 1 Median: 1.8887 Max: 13429 Percentiles: P50: 1.89 P75: 3.02 P99: 829.90 P99.9: 3505.55 P99.99: 4271.12 ------------------------------------------------------ [ 0, 1 ] 15101 4.575% 4.575% # ( 1, 2 ] 168698 51.114% 55.689% ########## ( 2, 3 ] 63058 19.106% 74.795% #### ( 3, 4 ] 34688 10.510% 85.305% ## ( 4, 6 ] 33197 10.058% 95.363% ## ( 6, 10 ] 7960 2.412% 97.775% ( 10, 15 ] 1847 0.560% 98.335% ( 15, 22 ] 227 0.069% 98.404% ( 22, 34 ] 24 0.007% 98.411% ( 34, 51 ] 25 0.008% 98.418% ( 51, 76 ] 26 0.008% 98.426% ( 76, 110 ] 13 0.004% 98.430% ( 110, 170 ] 56 0.017% 98.447% ( 170, 250 ] 138 0.042% 98.489% ( 250, 380 ] 267 0.081% 98.570% ( 380, 580 ] 564 0.171% 98.741% ( 580, 870 ] 994 0.301% 99.042% ( 870, 1300 ] 1092 0.331% 99.373% ( 1300, 1900 ] 871 0.264% 99.637% ( 1900, 2900 ] 635 0.192% 99.829% ( 2900, 4400 ] 582 0.176% 100.005% ( 4400, 6600 ] 221 0.067% 100.072% ( 6600, 9900 ] 34 0.010% 100.083% ( 9900, 14000 ] 1 0.000% 100.083% ** Level 5 read latency histogram (micros): Count: 11978 Average: 1784.4621 StdDev: 1503.50 Min: 84 Median: 1223.7736 Max: 10342 Percentiles: P50: 1223.77 P75: 2537.71 P99: 6589.34 P99.9: 9584.98 P99.99: 9894.32 ------------------------------------------------------ ( 76, 110 ] 18 0.150% 0.150% ( 110, 170 ] 90 0.751% 0.902% ( 170, 250 ] 193 1.611% 2.513% ( 250, 380 ] 517 4.316% 6.829% # ( 380, 580 ] 1223 10.210% 17.040% ## ( 580, 870 ] 2073 17.307% 34.346% ### ( 870, 1300 ] 2279 19.027% 53.373% #### ( 1300, 1900 ] 1662 13.875% 67.248% ### ( 1900, 2900 ] 1457 12.164% 79.412% ## ( 2900, 4400 ] 1571 13.116% 92.528% ### ( 4400, 6600 ] 780 6.512% 99.040% # ( 6600, 9900 ] 115 0.960% 100.000% ( 9900, 14000 ] 2 0.017% 100.017% ** DB Stats ** Uptime(secs): 7.6 total, 7.5 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 128.86 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.596 H:M:S, 47.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 132.07 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.596 H:M:S, 48.3 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/20-01:59:37 ... thread 0: (1000000,2000000) ops and (182809.8,154808.0) ops/second in (5.470167,12.919225) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/10 1.98 GB 31.5 1.2 0.0 1.2 3.0 1.8 0.0 1.7 61.7 155.2 19.53 10.29 27 0.724 1224K 1429 0.0 0.0 L1 2/2 75.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.67 MB 0.7 1.3 0.1 1.1 1.3 0.1 0.0 10.0 126.8 126.7 10.16 5.87 1 10.155 654K 0 0.0 0.0 L3 15/2 1.14 GB 0.9 2.4 1.6 0.8 2.3 1.6 1.1 1.5 107.1 106.0 22.69 11.26 7 3.241 1408K 0 0.0 0.0 L4 177/11 10.16 GB 1.0 7.4 4.5 2.9 7.3 4.4 0.2 1.6 112.0 110.6 67.49 35.98 44 1.534 3695K 19K 0.0 0.0 L5 1117/0 92.33 GB 0.9 0.0 0.0 0.0 0.0 0.0 4.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1326/25 105.75 GB 0.0 12.2 6.2 6.0 13.9 7.9 5.6 7.8 104.1 118.3 119.86 63.40 79 1.517 6982K 20K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.2 6.2 6.0 13.9 7.9 5.6 7.8 104.1 118.3 119.86 63.40 79 1.517 6982K 20K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.2 6.2 6.0 12.1 6.1 0.0 0.0 116.0 114.8 107.62 58.05 56 1.922 6982K 20K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 149.2 12.24 5.35 23 0.532 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.1 total, 12.9 interval Flush(GB): cumulative 1.784, interval 1.784 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 13.85 GB write, 1082.07 MB/s write, 12.19 GB read, 952.13 MB/s read, 119.9 seconds Interval compaction: 13.85 GB write, 1097.64 MB/s write, 12.19 GB read, 965.83 MB/s read, 119.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 36, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 36, interval: 36 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 314213 Average: 13.4593 StdDev: 171.82 Min: 1 Median: 1.6803 Max: 34284 Percentiles: P50: 1.68 P75: 2.48 P99: 222.52 P99.9: 1965.38 P99.99: 4152.39 ------------------------------------------------------ [ 0, 1 ] 34908 11.110% 11.110% ## ( 1, 2 ] 179624 57.166% 68.276% ########### ( 2, 3 ] 44290 14.096% 82.372% ### ( 3, 4 ] 23776 7.567% 89.938% ## ( 4, 6 ] 21825 6.946% 96.884% # ( 6, 10 ] 4197 1.336% 98.220% ( 10, 15 ] 910 0.290% 98.510% ( 15, 22 ] 92 0.029% 98.539% ( 22, 34 ] 5 0.002% 98.540% ( 34, 51 ] 4 0.001% 98.542% ( 51, 76 ] 13 0.004% 98.546% ( 76, 110 ] 155 0.049% 98.595% ( 110, 170 ] 811 0.258% 98.853% ( 170, 250 ] 702 0.223% 99.077% ( 250, 380 ] 583 0.186% 99.262% ( 380, 580 ] 622 0.198% 99.460% ( 580, 870 ] 629 0.200% 99.660% ( 870, 1300 ] 482 0.153% 99.814% ( 1300, 1900 ] 260 0.083% 99.897% ( 1900, 2900 ] 165 0.053% 99.949% ( 2900, 4400 ] 154 0.049% 99.998% ( 4400, 6600 ] 44 0.014% 100.012% ( 6600, 9900 ] 7 0.002% 100.014% ( 9900, 14000 ] 1 0.000% 100.015% ( 22000, 33000 ] 2 0.001% 100.015% ( 33000, 50000 ] 1 0.000% 100.016% ** Level 1 read latency histogram (micros): Count: 33356 Average: 24.7422 StdDev: 234.79 Min: 1 Median: 1.7454 Max: 8760 Percentiles: P50: 1.75 P75: 2.55 P99: 644.13 P99.9: 4109.63 P99.99: 6477.57 ------------------------------------------------------ [ 0, 1 ] 1658 4.971% 4.971% # ( 1, 2 ] 20150 60.409% 65.380% ############ ( 2, 3 ] 5881 17.631% 83.011% #### ( 3, 4 ] 2117 6.347% 89.357% # ( 4, 6 ] 1975 5.921% 95.278% # ( 6, 10 ] 700 2.099% 97.377% ( 10, 15 ] 128 0.384% 97.761% ( 15, 22 ] 12 0.036% 97.796% ( 22, 34 ] 1 0.003% 97.799% ( 51, 76 ] 1 0.003% 97.802% ( 76, 110 ] 26 0.078% 97.880% ( 110, 170 ] 100 0.300% 98.180% ( 170, 250 ] 84 0.252% 98.432% ( 250, 380 ] 77 0.231% 98.663% ( 380, 580 ] 89 0.267% 98.930% ( 580, 870 ] 106 0.318% 99.248% ( 870, 1300 ] 85 0.255% 99.502% ( 1300, 1900 ] 54 0.162% 99.664% ( 1900, 2900 ] 48 0.144% 99.808% ( 2900, 4400 ] 38 0.114% 99.922% ( 4400, 6600 ] 24 0.072% 99.994% ( 6600, 9900 ] 2 0.006% 100.000% ** Level 2 read latency histogram (micros): Count: 802678 Average: 12.3744 StdDev: 162.47 Min: 1 Median: 1.8583 Max: 34851 Percentiles: P50: 1.86 P75: 2.92 P99: 12.08 P99.9: 1683.30 P99.99: 2688.94 ------------------------------------------------------ [ 0, 1 ] 40988 5.106% 5.106% # ( 1, 2 ] 419826 52.303% 57.410% ########## ( 2, 3 ] 153616 19.138% 76.548% #### ( 3, 4 ] 77555 9.662% 86.210% ## ( 4, 6 ] 79987 9.965% 96.175% ## ( 6, 10 ] 20640 2.571% 98.746% # ( 10, 15 ] 4891 0.609% 99.355% ( 15, 22 ] 654 0.081% 99.437% ( 22, 34 ] 89 0.011% 99.448% ( 34, 51 ] 59 0.007% 99.455% ( 51, 76 ] 73 0.009% 99.464% ( 76, 110 ] 63 0.008% 99.472% ( 110, 170 ] 158 0.020% 99.492% ( 170, 250 ] 179 0.022% 99.514% ( 250, 380 ] 316 0.039% 99.553% ( 380, 580 ] 561 0.070% 99.623% ( 580, 870 ] 870 0.108% 99.732% ( 870, 1300 ] 914 0.114% 99.846% ( 1300, 1900 ] 683 0.085% 99.931% ( 1900, 2900 ] 603 0.075% 100.006% ( 2900, 4400 ] 569 0.071% 100.077% ( 4400, 6600 ] 164 0.020% 100.097% ( 6600, 9900 ] 26 0.003% 100.100% ( 9900, 14000 ] 1 0.000% 100.101% ( 14000, 22000 ] 1 0.000% 100.101% ( 33000, 50000 ] 1 0.000% 100.101% ** Level 3 read latency histogram (micros): Count: 1015329 Average: 11.4373 StdDev: 160.97 Min: 1 Median: 1.8943 Max: 31080 Percentiles: P50: 1.89 P75: 2.98 P99: 10.02 P99.9: 1054.20 P99.99: 1630.84 ------------------------------------------------------ [ 0, 1 ] 37766 3.720% 3.720% # ( 1, 2 ] 525464 51.753% 55.473% ########## ( 2, 3 ] 202175 19.912% 75.385% #### ( 3, 4 ] 104717 10.314% 85.699% ## ( 4, 6 ] 110262 10.860% 96.558% ## ( 6, 10 ] 24770 2.440% 98.998% ( 10, 15 ] 5716 0.563% 99.561% ( 15, 22 ] 653 0.064% 99.625% ( 22, 34 ] 80 0.008% 99.633% ( 34, 51 ] 68 0.007% 99.640% ( 51, 76 ] 106 0.010% 99.650% ( 76, 110 ] 54 0.005% 99.655% ( 110, 170 ] 140 0.014% 99.669% ( 170, 250 ] 183 0.018% 99.687% ( 250, 380 ] 286 0.028% 99.715% ( 380, 580 ] 586 0.058% 99.773% ( 580, 870 ] 888 0.087% 99.861% ( 870, 1300 ] 933 0.092% 99.953% ( 1300, 1900 ] 690 0.068% 100.020% ( 1900, 2900 ] 671 0.066% 100.087% ( 2900, 4400 ] 657 0.065% 100.151% ( 4400, 6600 ] 219 0.022% 100.173% ( 6600, 9900 ] 45 0.004% 100.177% ( 14000, 22000 ] 2 0.000% 100.177% ( 22000, 33000 ] 2 0.000% 100.178% ** Level 4 read latency histogram (micros): Count: 790837 Average: 21.2672 StdDev: 293.83 Min: 0 Median: 1.8637 Max: 43970 Percentiles: P50: 1.86 P75: 2.94 P99: 329.12 P99.9: 2250.02 P99.99: 3198.35 ------------------------------------------------------ [ 0, 1 ] 34518 4.365% 4.365% # ( 1, 2 ] 417869 52.839% 57.204% ########### ( 2, 3 ] 150035 18.972% 76.175% #### ( 3, 4 ] 77657 9.820% 85.995% ## ( 4, 6 ] 76700 9.699% 95.693% ## ( 6, 10 ] 18646 2.358% 98.051% ( 10, 15 ] 4071 0.515% 98.566% ( 15, 22 ] 499 0.063% 98.629% ( 22, 34 ] 59 0.007% 98.637% ( 34, 51 ] 63 0.008% 98.644% ( 51, 76 ] 87 0.011% 98.655% ( 76, 110 ] 236 0.030% 98.685% ( 110, 170 ] 995 0.126% 98.811% ( 170, 250 ] 928 0.117% 98.928% ( 250, 380 ] 931 0.118% 99.046% ( 380, 580 ] 1568 0.198% 99.244% ( 580, 870 ] 1960 0.248% 99.492% ( 870, 1300 ] 1738 0.220% 99.712% ( 1300, 1900 ] 1190 0.150% 99.863% ( 1900, 2900 ] 849 0.107% 99.970% ( 2900, 4400 ] 804 0.102% 100.072% ( 4400, 6600 ] 294 0.037% 100.109% ( 6600, 9900 ] 44 0.006% 100.114% ( 9900, 14000 ] 6 0.001% 100.115% ( 14000, 22000 ] 18 0.002% 100.117% ( 22000, 33000 ] 31 0.004% 100.121% ( 33000, 50000 ] 5 0.001% 100.122% ** Level 5 read latency histogram (micros): Count: 38296 Average: 979.1229 StdDev: 1523.88 Min: 77 Median: 508.0975 Max: 39983 Percentiles: P50: 508.10 P75: 1121.98 P99: 6173.61 P99.9: 20979.56 P99.99: 31312.48 ------------------------------------------------------ ( 76, 110 ] 1371 3.580% 3.580% # ( 110, 170 ] 5934 15.495% 19.075% ### ( 170, 250 ] 4608 12.033% 31.108% ## ( 250, 380 ] 4030 10.523% 41.631% ## ( 380, 580 ] 5004 13.067% 54.698% ### ( 580, 870 ] 5230 13.657% 68.354% ### ( 870, 1300 ] 4343 11.341% 79.695% ## ( 1300, 1900 ] 2560 6.685% 86.380% # ( 1900, 2900 ] 1988 5.191% 91.571% # ( 2900, 4400 ] 2084 5.442% 97.013% # ( 4400, 6600 ] 944 2.465% 99.478% ( 6600, 9900 ] 134 0.350% 99.828% ( 9900, 14000 ] 12 0.031% 99.859% ( 14000, 22000 ] 18 0.047% 99.906% ( 22000, 33000 ] 38 0.099% 100.005% ( 33000, 50000 ] 1 0.003% 100.008% ** DB Stats ** Uptime(secs): 13.1 total, 5.5 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 150.17 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.284 H:M:S, 40.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 179.92 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.688 H:M:S, 30.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/11/20-01:59:41 ... thread 0: (1000000,3000000) ops and (240449.2,175663.5) ops/second in (4.158882,17.078107) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/10 2.96 GB 132.2 2.2 0.0 2.2 5.0 2.8 0.0 1.8 82.2 189.6 26.94 17.16 48 0.561 2248K 2371 0.0 0.0 L1 2/2 75.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.67 MB 0.7 1.3 0.1 1.1 1.3 0.1 0.0 10.0 126.8 126.7 10.16 5.87 1 10.155 654K 0 0.0 0.0 L3 13/0 914.80 MB 0.9 2.4 1.6 0.8 2.3 1.6 1.1 1.5 107.1 106.0 22.69 11.26 7 3.241 1408K 0 0.0 0.0 L4 172/0 9.73 GB 1.0 8.1 4.7 3.4 8.0 4.6 0.2 1.7 116.5 115.1 71.37 39.34 46 1.551 4465K 26K 0.0 0.0 L5 1125/0 93.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1331/12 106.72 GB 0.0 13.9 6.4 7.5 16.6 9.1 6.3 5.9 108.6 129.7 131.16 73.62 102 1.286 8775K 29K 0.0 0.0 Int 0/0 0.00 KB 0.0 13.9 6.4 7.5 16.6 9.1 6.3 5.9 108.6 129.7 131.16 73.62 102 1.286 8775K 29K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 13.9 6.4 7.5 13.8 6.3 0.0 0.0 123.4 122.3 115.44 65.25 62 1.862 8775K 29K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 184.4 15.72 8.37 40 0.393 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 17.3 total, 17.1 interval Flush(GB): cumulative 2.830, interval 2.830 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 16.61 GB write, 985.23 MB/s write, 13.91 GB read, 825.07 MB/s read, 131.2 seconds Interval compaction: 16.61 GB write, 995.96 MB/s write, 13.91 GB read, 834.05 MB/s read, 131.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 4, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 4, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 36, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 4, total-stops: 36, interval: 40 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 599540 Average: 16.1014 StdDev: 134.20 Min: 1 Median: 1.5828 Max: 34284 Percentiles: P50: 1.58 P75: 2.15 P99: 314.78 P99.9: 1213.90 P99.99: 2665.62 ------------------------------------------------------ [ 0, 1 ] 104399 17.413% 17.413% ### ( 1, 2 ] 335221 55.913% 73.326% ########### ( 2, 3 ] 65621 10.945% 84.271% ## ( 3, 4 ] 32497 5.420% 89.692% # ( 4, 6 ] 27332 4.559% 94.251% # ( 6, 10 ] 5203 0.868% 95.118% ( 10, 15 ] 1118 0.186% 95.305% ( 15, 22 ] 120 0.020% 95.325% ( 22, 34 ] 13 0.002% 95.327% ( 34, 51 ] 7 0.001% 95.328% ( 51, 76 ] 13 0.002% 95.330% ( 76, 110 ] 2916 0.486% 95.817% ( 110, 170 ] 11823 1.972% 97.789% ( 170, 250 ] 5830 0.972% 98.761% ( 250, 380 ] 2873 0.479% 99.240% ( 380, 580 ] 2308 0.385% 99.625% ( 580, 870 ] 1161 0.194% 99.819% ( 870, 1300 ] 607 0.101% 99.920% ( 1300, 1900 ] 281 0.047% 99.967% ( 1900, 2900 ] 179 0.030% 99.997% ( 2900, 4400 ] 161 0.027% 100.024% ( 4400, 6600 ] 44 0.007% 100.031% ( 6600, 9900 ] 7 0.001% 100.032% ( 9900, 14000 ] 1 0.000% 100.033% ( 22000, 33000 ] 2 0.000% 100.033% ( 33000, 50000 ] 1 0.000% 100.033% ** Level 1 read latency histogram (micros): Count: 35003 Average: 33.0873 StdDev: 235.76 Min: 1 Median: 1.7850 Max: 8760 Percentiles: P50: 1.79 P75: 2.75 P99: 700.77 P99.9: 4053.73 P99.99: 6462.47 ------------------------------------------------------ [ 0, 1 ] 1663 4.751% 4.751% # ( 1, 2 ] 20176 57.641% 62.392% ############ ( 2, 3 ] 5906 16.873% 79.265% ### ( 3, 4 ] 2129 6.082% 85.347% # ( 4, 6 ] 1980 5.657% 91.004% # ( 6, 10 ] 702 2.006% 93.009% ( 10, 15 ] 128 0.366% 93.375% ( 15, 22 ] 12 0.034% 93.409% ( 22, 34 ] 1 0.003% 93.412% ( 51, 76 ] 1 0.003% 93.415% ( 76, 110 ] 210 0.600% 94.015% ( 110, 170 ] 867 2.477% 96.492% ( 170, 250 ] 398 1.137% 97.629% ( 250, 380 ] 214 0.611% 98.240% ( 380, 580 ] 206 0.589% 98.829% ( 580, 870 ] 144 0.411% 99.240% ( 870, 1300 ] 95 0.271% 99.511% ( 1300, 1900 ] 56 0.160% 99.671% ( 1900, 2900 ] 50 0.143% 99.814% ( 2900, 4400 ] 39 0.111% 99.926% ( 4400, 6600 ] 24 0.069% 99.994% ( 6600, 9900 ] 2 0.006% 100.000% ** Level 2 read latency histogram (micros): Count: 855603 Average: 11.8505 StdDev: 157.47 Min: 1 Median: 1.8414 Max: 34851 Percentiles: P50: 1.84 P75: 2.88 P99: 12.00 P99.9: 1632.80 P99.99: 2671.87 ------------------------------------------------------ [ 0, 1 ] 45631 5.333% 5.333% # ( 1, 2 ] 454231 53.089% 58.422% ########### ( 2, 3 ] 161179 18.838% 77.260% #### ( 3, 4 ] 80361 9.392% 86.653% ## ( 4, 6 ] 82005 9.584% 96.237% ## ( 6, 10 ] 21621 2.527% 98.764% # ( 10, 15 ] 5039 0.589% 99.353% ( 15, 22 ] 668 0.078% 99.431% ( 22, 34 ] 91 0.011% 99.442% ( 34, 51 ] 59 0.007% 99.449% ( 51, 76 ] 73 0.009% 99.457% ( 76, 110 ] 95 0.011% 99.468% ( 110, 170 ] 318 0.037% 99.505% ( 170, 250 ] 252 0.029% 99.535% ( 250, 380 ] 354 0.041% 99.576% ( 380, 580 ] 593 0.069% 99.646% ( 580, 870 ] 882 0.103% 99.749% ( 870, 1300 ] 916 0.107% 99.856% ( 1300, 1900 ] 684 0.080% 99.936% ( 1900, 2900 ] 603 0.070% 100.006% ( 2900, 4400 ] 569 0.067% 100.073% ( 4400, 6600 ] 164 0.019% 100.092% ( 6600, 9900 ] 26 0.003% 100.095% ( 9900, 14000 ] 1 0.000% 100.095% ( 14000, 22000 ] 1 0.000% 100.095% ( 33000, 50000 ] 1 0.000% 100.095% ** Level 3 read latency histogram (micros): Count: 1015972 Average: 11.5380 StdDev: 161.02 Min: 1 Median: 1.8949 Max: 31080 Percentiles: P50: 1.89 P75: 2.98 P99: 10.48 P99.9: 1054.36 P99.99: 1631.65 ------------------------------------------------------ [ 0, 1 ] 37766 3.717% 3.717% # ( 1, 2 ] 525470 51.721% 55.438% ########## ( 2, 3 ] 202218 19.904% 75.342% #### ( 3, 4 ] 104742 10.310% 85.652% ## ( 4, 6 ] 110293 10.856% 96.507% ## ( 6, 10 ] 24778 2.439% 98.946% ( 10, 15 ] 5717 0.563% 99.509% ( 15, 22 ] 653 0.064% 99.573% ( 22, 34 ] 80 0.008% 99.581% ( 34, 51 ] 68 0.007% 99.588% ( 51, 76 ] 106 0.010% 99.598% ( 76, 110 ] 124 0.012% 99.611% ( 110, 170 ] 376 0.037% 99.648% ( 170, 250 ] 295 0.029% 99.677% ( 250, 380 ] 345 0.034% 99.711% ( 380, 580 ] 623 0.061% 99.772% ( 580, 870 ] 902 0.089% 99.861% ( 870, 1300 ] 933 0.092% 99.952% ( 1300, 1900 ] 690 0.068% 100.020% ( 1900, 2900 ] 672 0.066% 100.087% ( 2900, 4400 ] 657 0.065% 100.151% ( 4400, 6600 ] 219 0.022% 100.173% ( 6600, 9900 ] 45 0.004% 100.177% ( 14000, 22000 ] 2 0.000% 100.177% ( 22000, 33000 ] 2 0.000% 100.178% ** Level 4 read latency histogram (micros): Count: 875765 Average: 21.3249 StdDev: 280.22 Min: 0 Median: 1.8504 Max: 43970 Percentiles: P50: 1.85 P75: 2.92 P99: 349.72 P99.9: 2118.59 P99.99: 3128.12 ------------------------------------------------------ [ 0, 1 ] 39988 4.566% 4.566% # ( 1, 2 ] 467915 53.429% 57.995% ########### ( 2, 3 ] 162355 18.539% 76.534% #### ( 3, 4 ] 81764 9.336% 85.870% ## ( 4, 6 ] 80234 9.162% 95.032% ## ( 6, 10 ] 19929 2.276% 97.307% ( 10, 15 ] 4261 0.487% 97.794% ( 15, 22 ] 525 0.060% 97.854% ( 22, 34 ] 61 0.007% 97.861% ( 34, 51 ] 65 0.007% 97.868% ( 51, 76 ] 87 0.010% 97.878% ( 76, 110 ] 1187 0.136% 98.014% ( 110, 170 ] 4690 0.536% 98.549% ( 170, 250 ] 2663 0.304% 98.853% ( 250, 380 ] 1673 0.191% 99.044% ( 380, 580 ] 2160 0.247% 99.291% ( 580, 870 ] 2176 0.248% 99.540% ( 870, 1300 ] 1777 0.203% 99.743% ( 1300, 1900 ] 1193 0.136% 99.879% ( 1900, 2900 ] 852 0.097% 99.976% ( 2900, 4400 ] 805 0.092% 100.068% ( 4400, 6600 ] 294 0.034% 100.102% ( 6600, 9900 ] 44 0.005% 100.107% ( 9900, 14000 ] 6 0.001% 100.107% ( 14000, 22000 ] 18 0.002% 100.109% ( 22000, 33000 ] 31 0.004% 100.113% ( 33000, 50000 ] 5 0.001% 100.113% ** Level 5 read latency histogram (micros): Count: 88126 Average: 542.7466 StdDev: 1081.19 Min: 75 Median: 208.0122 Max: 39983 Percentiles: P50: 208.01 P75: 507.67 P99: 5007.66 P99.9: 9305.85 P99.99: 29291.09 ------------------------------------------------------ ( 51, 76 ] 2 0.002% 0.002% ( 76, 110 ] 7713 8.752% 8.755% ## ( 110, 170 ] 29196 33.130% 41.884% ####### ( 170, 250 ] 15052 17.080% 58.964% ### ( 250, 380 ] 8601 9.760% 68.724% ## ( 380, 580 ] 8664 9.831% 78.556% ## ( 580, 870 ] 6454 7.324% 85.879% # ( 870, 1300 ] 4592 5.211% 91.090% # ( 1300, 1900 ] 2609 2.961% 94.051% # ( 1900, 2900 ] 2014 2.285% 96.336% ( 2900, 4400 ] 2087 2.368% 98.704% ( 4400, 6600 ] 944 1.071% 99.775% ( 6600, 9900 ] 134 0.152% 99.927% ( 9900, 14000 ] 12 0.014% 99.941% ( 14000, 22000 ] 18 0.020% 99.961% ( 22000, 33000 ] 38 0.043% 100.005% ( 33000, 50000 ] 1 0.001% 100.006% ** DB Stats ** Uptime(secs): 17.3 total, 4.2 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 171.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.915 H:M:S, 34.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 236.65 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.631 H:M:S, 15.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/11/20-01:59:49 ... thread 0: (1000000,4000000) ops and (118226.0,156638.7) ops/second in (8.458373,25.536480) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/14 3.93 GB 206.8 4.3 0.0 4.3 8.1 3.7 0.0 2.1 115.3 215.3 38.31 27.90 67 0.572 4485K 12K 0.0 0.0 L1 2/2 75.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.67 MB 0.7 1.3 0.1 1.1 1.3 0.1 0.0 10.0 126.8 126.7 10.16 5.87 1 10.155 654K 0 0.0 0.0 L3 13/0 914.80 MB 0.9 2.4 1.6 0.8 2.3 1.6 1.1 1.5 107.1 106.0 22.69 11.26 7 3.241 1408K 0 0.0 0.0 L4 172/0 9.73 GB 1.0 8.1 4.7 3.4 8.0 4.6 0.2 1.7 116.5 115.1 71.37 39.34 46 1.551 4465K 26K 0.0 0.0 L5 1125/0 93.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/16 107.70 GB 0.0 16.1 6.4 9.6 19.7 10.1 6.3 5.2 115.4 141.4 142.52 84.37 121 1.178 11M 39K 0.0 0.0 Int 0/0 0.00 KB 0.0 16.1 6.4 9.6 19.7 10.1 6.3 5.2 115.4 141.4 142.52 84.37 121 1.178 11M 39K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.1 6.4 9.6 15.9 6.3 0.0 0.0 132.8 131.7 123.83 73.42 66 1.876 11M 39K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 205.6 18.69 10.94 55 0.340 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 25.7 total, 25.5 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 19.68 GB write, 783.30 MB/s write, 16.06 GB read, 639.44 MB/s read, 142.5 seconds Interval compaction: 19.68 GB write, 789.00 MB/s write, 16.06 GB read, 644.10 MB/s read, 142.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 11, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 11, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 36, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 11, total-stops: 36, interval: 47 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1270556 Average: 16.9208 StdDev: 103.26 Min: 0 Median: 1.4778 Max: 34284 Percentiles: P50: 1.48 P75: 1.94 P99: 269.79 P99.9: 787.31 P99.99: 1264.76 ------------------------------------------------------ [ 0, 1 ] 308932 24.315% 24.315% ##### ( 1, 2 ] 683074 53.762% 78.077% ########### ( 2, 3 ] 100448 7.906% 85.982% ## ( 3, 4 ] 46069 3.626% 89.608% # ( 4, 6 ] 34609 2.724% 92.332% # ( 6, 10 ] 7182 0.565% 92.897% ( 10, 15 ] 1442 0.113% 93.011% ( 15, 22 ] 157 0.012% 93.023% ( 22, 34 ] 17 0.001% 93.025% ( 34, 51 ] 10 0.001% 93.025% ( 51, 76 ] 22 0.002% 93.027% ( 76, 110 ] 12603 0.992% 94.019% ( 110, 170 ] 45541 3.584% 97.603% # ( 170, 250 ] 16792 1.322% 98.925% ( 250, 380 ] 6255 0.492% 99.417% ( 380, 580 ] 4840 0.381% 99.798% ( 580, 870 ] 1808 0.142% 99.941% ( 870, 1300 ] 684 0.054% 99.994% ( 1300, 1900 ] 298 0.023% 100.018% ( 1900, 2900 ] 217 0.017% 100.035% ( 2900, 4400 ] 180 0.014% 100.049% ( 4400, 6600 ] 45 0.004% 100.053% ( 6600, 9900 ] 7 0.001% 100.053% ( 9900, 14000 ] 1 0.000% 100.053% ( 22000, 33000 ] 2 0.000% 100.053% ( 33000, 50000 ] 1 0.000% 100.054% ** Level 1 read latency histogram (micros): Count: 38711 Average: 46.2489 StdDev: 230.37 Min: 1 Median: 1.8769 Max: 8760 Percentiles: P50: 1.88 P75: 3.61 P99: 686.32 P99.9: 3911.12 P99.99: 6428.48 ------------------------------------------------------ [ 0, 1 ] 1663 4.296% 4.296% # ( 1, 2 ] 20176 52.120% 56.415% ########## ( 2, 3 ] 5906 15.257% 71.672% ### ( 3, 4 ] 2129 5.500% 77.172% # ( 4, 6 ] 1980 5.115% 82.287% # ( 6, 10 ] 702 1.813% 84.100% ( 10, 15 ] 128 0.331% 84.431% ( 15, 22 ] 12 0.031% 84.462% ( 22, 34 ] 1 0.003% 84.464% ( 51, 76 ] 2 0.005% 84.470% ( 76, 110 ] 936 2.418% 86.887% ( 110, 170 ] 2828 7.305% 94.193% # ( 170, 250 ] 1008 2.604% 96.797% # ( 250, 380 ] 437 1.129% 97.926% ( 380, 580 ] 351 0.907% 98.832% ( 580, 870 ] 177 0.457% 99.290% ( 870, 1300 ] 102 0.263% 99.553% ( 1300, 1900 ] 57 0.147% 99.700% ( 1900, 2900 ] 51 0.132% 99.832% ( 2900, 4400 ] 39 0.101% 99.933% ( 4400, 6600 ] 24 0.062% 99.995% ( 6600, 9900 ] 2 0.005% 100.000% ** Level 2 read latency histogram (micros): Count: 856357 Average: 11.9904 StdDev: 157.50 Min: 1 Median: 1.8422 Max: 34851 Percentiles: P50: 1.84 P75: 2.88 P99: 12.74 P99.9: 1632.14 P99.99: 2671.75 ------------------------------------------------------ [ 0, 1 ] 45631 5.329% 5.329% # ( 1, 2 ] 454231 53.042% 58.371% ########### ( 2, 3 ] 161179 18.821% 77.192% #### ( 3, 4 ] 80361 9.384% 86.576% ## ( 4, 6 ] 82005 9.576% 96.152% ## ( 6, 10 ] 21621 2.525% 98.677% # ( 10, 15 ] 5039 0.588% 99.265% ( 15, 22 ] 668 0.078% 99.343% ( 22, 34 ] 91 0.011% 99.354% ( 34, 51 ] 59 0.007% 99.361% ( 51, 76 ] 73 0.009% 99.370% ( 76, 110 ] 231 0.027% 99.397% ( 110, 170 ] 717 0.084% 99.480% ( 170, 250 ] 394 0.046% 99.526% ( 250, 380 ] 394 0.046% 99.572% ( 380, 580 ] 621 0.073% 99.645% ( 580, 870 ] 890 0.104% 99.749% ( 870, 1300 ] 917 0.107% 99.856% ( 1300, 1900 ] 684 0.080% 99.936% ( 1900, 2900 ] 603 0.070% 100.006% ( 2900, 4400 ] 569 0.066% 100.073% ( 4400, 6600 ] 164 0.019% 100.092% ( 6600, 9900 ] 26 0.003% 100.095% ( 9900, 14000 ] 1 0.000% 100.095% ( 14000, 22000 ] 1 0.000% 100.095% ( 33000, 50000 ] 1 0.000% 100.095% ** Level 3 read latency histogram (micros): Count: 1017499 Average: 11.7728 StdDev: 161.10 Min: 1 Median: 1.8963 Max: 31080 Percentiles: P50: 1.90 P75: 2.99 P99: 11.73 P99.9: 1055.37 P99.99: 1633.26 ------------------------------------------------------ [ 0, 1 ] 37766 3.712% 3.712% # ( 1, 2 ] 525479 51.644% 55.356% ########## ( 2, 3 ] 202251 19.877% 75.233% #### ( 3, 4 ] 104756 10.295% 85.529% ## ( 4, 6 ] 110311 10.841% 96.370% ## ( 6, 10 ] 24780 2.435% 98.805% ( 10, 15 ] 5717 0.562% 99.367% ( 15, 22 ] 653 0.064% 99.431% ( 22, 34 ] 80 0.008% 99.439% ( 34, 51 ] 68 0.007% 99.446% ( 51, 76 ] 106 0.010% 99.456% ( 76, 110 ] 373 0.037% 99.493% ( 110, 170 ] 1168 0.115% 99.608% ( 170, 250 ] 536 0.053% 99.660% ( 250, 380 ] 424 0.042% 99.702% ( 380, 580 ] 700 0.069% 99.771% ( 580, 870 ] 910 0.089% 99.860% ( 870, 1300 ] 936 0.092% 99.952% ( 1300, 1900 ] 690 0.068% 100.020% ( 1900, 2900 ] 674 0.066% 100.086% ( 2900, 4400 ] 657 0.065% 100.151% ( 4400, 6600 ] 219 0.022% 100.172% ( 6600, 9900 ] 45 0.004% 100.177% ( 14000, 22000 ] 2 0.000% 100.177% ( 22000, 33000 ] 2 0.000% 100.177% ** Level 4 read latency histogram (micros): Count: 896926 Average: 24.5235 StdDev: 278.20 Min: 0 Median: 1.8725 Max: 43970 Percentiles: P50: 1.87 P75: 3.02 P99: 409.07 P99.9: 2096.58 P99.99: 3124.18 ------------------------------------------------------ [ 0, 1 ] 39989 4.458% 4.458% # ( 1, 2 ] 468152 52.195% 56.654% ########## ( 2, 3 ] 163048 18.179% 74.832% #### ( 3, 4 ] 82127 9.156% 83.989% ## ( 4, 6 ] 80857 9.015% 93.004% ## ( 6, 10 ] 19994 2.229% 95.233% ( 10, 15 ] 4277 0.477% 95.710% ( 15, 22 ] 526 0.059% 95.768% ( 22, 34 ] 61 0.007% 95.775% ( 34, 51 ] 65 0.007% 95.782% ( 51, 76 ] 88 0.010% 95.792% ( 76, 110 ] 4287 0.478% 96.270% ( 110, 170 ] 15321 1.708% 97.978% ( 170, 250 ] 6016 0.671% 98.649% ( 250, 380 ] 2718 0.303% 98.952% ( 380, 580 ] 2963 0.330% 99.282% ( 580, 870 ] 2370 0.264% 99.547% ( 870, 1300 ] 1804 0.201% 99.748% ( 1300, 1900 ] 1198 0.134% 99.881% ( 1900, 2900 ] 855 0.095% 99.977% ( 2900, 4400 ] 805 0.090% 100.066% ( 4400, 6600 ] 294 0.033% 100.099% ( 6600, 9900 ] 44 0.005% 100.104% ( 9900, 14000 ] 6 0.001% 100.105% ( 14000, 22000 ] 18 0.002% 100.107% ( 22000, 33000 ] 31 0.003% 100.110% ( 33000, 50000 ] 5 0.001% 100.111% ** Level 5 read latency histogram (micros): Count: 205922 Average: 331.4207 StdDev: 735.47 Min: 74 Median: 158.2580 Max: 39983 Percentiles: P50: 158.26 P75: 244.34 P99: 3738.67 P99.9: 6567.55 P99.99: 24144.36 ------------------------------------------------------ ( 51, 76 ] 13 0.006% 0.006% ( 76, 110 ] 26829 13.029% 13.035% ### ( 110, 170 ] 94640 45.959% 58.994% ######### ( 170, 250 ] 35467 17.224% 76.218% ### ( 250, 380 ] 15122 7.344% 83.561% # ( 380, 580 ] 13520 6.566% 90.127% # ( 580, 870 ] 7670 3.725% 93.852% # ( 870, 1300 ] 4749 2.306% 96.158% ( 1300, 1900 ] 2632 1.278% 97.436% ( 1900, 2900 ] 2050 0.996% 98.431% ( 2900, 4400 ] 2094 1.017% 99.448% ( 4400, 6600 ] 944 0.458% 99.907% ( 6600, 9900 ] 134 0.065% 99.972% ( 9900, 14000 ] 12 0.006% 99.978% ( 14000, 22000 ] 18 0.009% 99.986% ( 22000, 33000 ] 38 0.018% 100.005% ( 33000, 50000 ] 1 0.000% 100.005% ** DB Stats ** Uptime(secs): 25.7 total, 8.5 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 153.03 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:10.768 H:M:S, 41.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 116.36 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.853 H:M:S, 57.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/20-02:00:04 ... thread 0: (1000000,5000000) ops and (71018.6,126207.5) ops/second in (14.080819,39.617299) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/10 4.97 GB 337.9 7.1 0.0 7.1 11.9 4.8 0.0 2.5 138.5 232.0 52.34 41.37 90 0.582 7362K 24K 0.0 0.0 L1 2/2 75.85 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.67 MB 0.7 1.3 0.1 1.1 1.3 0.1 0.0 10.0 126.8 126.7 10.16 5.87 1 10.155 654K 0 0.0 0.0 L3 13/0 914.80 MB 0.9 2.4 1.6 0.8 2.3 1.6 1.1 1.5 107.1 106.0 22.69 11.26 7 3.241 1408K 0 0.0 0.0 L4 172/0 9.73 GB 1.0 8.1 4.7 3.4 8.0 4.6 0.2 1.7 116.5 115.1 71.37 39.34 46 1.551 4465K 26K 0.0 0.0 L5 1125/0 93.00 GB 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1328/12 108.73 GB 0.0 18.8 6.4 12.4 23.5 11.1 6.3 4.9 123.2 153.6 156.55 97.83 144 1.087 13M 51K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.8 6.4 12.4 23.5 11.1 6.3 4.9 123.2 153.6 156.55 97.83 144 1.087 13M 51K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.8 6.4 12.4 18.7 6.3 0.0 0.0 143.5 142.3 134.41 83.88 72 1.867 13M 51K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.8 4.8 0.0 0.0 0.0 221.9 22.14 13.95 72 0.308 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 39.8 total, 39.6 interval Flush(GB): cumulative 4.799, interval 4.799 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 23.48 GB write, 603.99 MB/s write, 18.83 GB read, 484.42 MB/s read, 156.6 seconds Interval compaction: 23.48 GB write, 606.83 MB/s write, 18.83 GB read, 486.70 MB/s read, 156.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 24, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 24, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 36, pending-compaction-bytes-delays: 4, pending-compaction-bytes-stops: 0, total-delays: 28, total-stops: 36, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 2075126 Average: 19.2545 StdDev: 91.06 Min: 0 Median: 1.4485 Max: 34284 Percentiles: P50: 1.45 P75: 1.94 P99: 251.96 P99.9: 686.39 P99.99: 1022.20 ------------------------------------------------------ [ 0, 1 ] 560108 26.992% 26.992% ##### ( 1, 2 ] 1064551 51.301% 78.292% ########## ( 2, 3 ] 137856 6.643% 84.935% # ( 3, 4 ] 61500 2.964% 87.899% # ( 4, 6 ] 44692 2.154% 90.053% ( 6, 10 ] 9183 0.443% 90.495% ( 10, 15 ] 1995 0.096% 90.591% ( 15, 22 ] 232 0.011% 90.603% ( 22, 34 ] 28 0.001% 90.604% ( 34, 51 ] 12 0.001% 90.604% ( 51, 76 ] 40 0.002% 90.606% ( 76, 110 ] 32131 1.548% 92.155% ( 110, 170 ] 109340 5.269% 97.424% # ( 170, 250 ] 32554 1.569% 98.993% ( 250, 380 ] 10188 0.491% 99.484% ( 380, 580 ] 7717 0.372% 99.855% ( 580, 870 ] 2521 0.121% 99.977% ( 870, 1300 ] 767 0.037% 100.014% ( 1300, 1900 ] 321 0.015% 100.029% ( 1900, 2900 ] 253 0.012% 100.042% ( 2900, 4400 ] 207 0.010% 100.052% ( 4400, 6600 ] 46 0.002% 100.054% ( 6600, 9900 ] 7 0.000% 100.054% ( 9900, 14000 ] 1 0.000% 100.054% ( 22000, 33000 ] 2 0.000% 100.054% ( 33000, 50000 ] 1 0.000% 100.054% ** Level 1 read latency histogram (micros): Count: 45213 Average: 61.9729 StdDev: 219.47 Min: 1 Median: 2.1185 Max: 8760 Percentiles: P50: 2.12 P75: 92.69 P99: 637.82 P99.9: 3661.04 P99.99: 6368.88 ------------------------------------------------------ [ 0, 1 ] 1663 3.678% 3.678% # ( 1, 2 ] 20238 44.761% 48.440% ######### ( 2, 3 ] 5954 13.169% 61.608% ### ( 3, 4 ] 2139 4.731% 66.339% # ( 4, 6 ] 1986 4.393% 70.732% # ( 6, 10 ] 704 1.557% 72.289% ( 10, 15 ] 128 0.283% 72.572% ( 15, 22 ] 12 0.027% 72.599% ( 22, 34 ] 1 0.002% 72.601% ( 51, 76 ] 4 0.009% 72.610% ( 76, 110 ] 2202 4.870% 77.480% # ( 110, 170 ] 6551 14.489% 91.969% ### ( 170, 250 ] 1925 4.258% 96.227% # ( 250, 380 ] 667 1.475% 97.702% ( 380, 580 ] 544 1.203% 98.905% ( 580, 870 ] 215 0.476% 99.381% ( 870, 1300 ] 104 0.230% 99.611% ( 1300, 1900 ] 59 0.130% 99.741% ( 1900, 2900 ] 52 0.115% 99.856% ( 2900, 4400 ] 39 0.086% 99.942% ( 4400, 6600 ] 24 0.053% 99.996% ( 6600, 9900 ] 2 0.004% 100.000% ** Level 2 read latency histogram (micros): Count: 857636 Average: 12.2127 StdDev: 157.52 Min: 1 Median: 1.8436 Max: 34851 Percentiles: P50: 1.84 P75: 2.89 P99: 14.00 P99.9: 1631.02 P99.99: 2671.54 ------------------------------------------------------ [ 0, 1 ] 45631 5.321% 5.321% # ( 1, 2 ] 454231 52.963% 58.284% ########### ( 2, 3 ] 161179 18.793% 77.077% #### ( 3, 4 ] 80361 9.370% 86.447% ## ( 4, 6 ] 82005 9.562% 96.009% ## ( 6, 10 ] 21621 2.521% 98.530% # ( 10, 15 ] 5039 0.588% 99.117% ( 15, 22 ] 668 0.078% 99.195% ( 22, 34 ] 91 0.011% 99.206% ( 34, 51 ] 59 0.007% 99.213% ( 51, 76 ] 73 0.009% 99.221% ( 76, 110 ] 457 0.053% 99.275% ( 110, 170 ] 1473 0.172% 99.446% ( 170, 250 ] 601 0.070% 99.516% ( 250, 380 ] 439 0.051% 99.568% ( 380, 580 ] 653 0.076% 99.644% ( 580, 870 ] 902 0.105% 99.749% ( 870, 1300 ] 918 0.107% 99.856% ( 1300, 1900 ] 684 0.080% 99.936% ( 1900, 2900 ] 603 0.070% 100.006% ( 2900, 4400 ] 569 0.066% 100.072% ( 4400, 6600 ] 164 0.019% 100.092% ( 6600, 9900 ] 26 0.003% 100.095% ( 9900, 14000 ] 1 0.000% 100.095% ( 14000, 22000 ] 1 0.000% 100.095% ( 33000, 50000 ] 1 0.000% 100.095% ** Level 3 read latency histogram (micros): Count: 1020176 Average: 12.1692 StdDev: 161.17 Min: 1 Median: 1.8988 Max: 31080 Percentiles: P50: 1.90 P75: 3.00 P99: 14.05 P99.9: 1056.50 P99.99: 1635.53 ------------------------------------------------------ [ 0, 1 ] 37766 3.702% 3.702% # ( 1, 2 ] 525479 51.509% 55.211% ########## ( 2, 3 ] 202251 19.825% 75.036% #### ( 3, 4 ] 104756 10.268% 85.304% ## ( 4, 6 ] 110311 10.813% 96.117% ## ( 6, 10 ] 24780 2.429% 98.546% ( 10, 15 ] 5717 0.560% 99.106% ( 15, 22 ] 653 0.064% 99.170% ( 22, 34 ] 80 0.008% 99.178% ( 34, 51 ] 68 0.007% 99.185% ( 51, 76 ] 107 0.010% 99.195% ( 76, 110 ] 891 0.087% 99.283% ( 110, 170 ] 2740 0.269% 99.551% ( 170, 250 ] 904 0.089% 99.640% ( 250, 380 ] 536 0.053% 99.693% ( 380, 580 ] 781 0.077% 99.769% ( 580, 870 ] 929 0.091% 99.860% ( 870, 1300 ] 938 0.092% 99.952% ( 1300, 1900 ] 692 0.068% 100.020% ( 1900, 2900 ] 676 0.066% 100.086% ( 2900, 4400 ] 657 0.064% 100.151% ( 4400, 6600 ] 219 0.021% 100.172% ( 6600, 9900 ] 45 0.004% 100.176% ( 14000, 22000 ] 2 0.000% 100.177% ( 22000, 33000 ] 2 0.000% 100.177% ** Level 4 read latency histogram (micros): Count: 934141 Average: 29.8407 StdDev: 274.50 Min: 0 Median: 1.9122 Max: 43970 Percentiles: P50: 1.91 P75: 3.36 P99: 445.52 P99.9: 2064.19 P99.99: 3120.42 ------------------------------------------------------ [ 0, 1 ] 39989 4.281% 4.281% # ( 1, 2 ] 468169 50.118% 54.398% ########## ( 2, 3 ] 163117 17.462% 71.860% ### ( 3, 4 ] 82160 8.795% 80.655% ## ( 4, 6 ] 80928 8.663% 89.319% ## ( 6, 10 ] 20011 2.142% 91.461% ( 10, 15 ] 4283 0.458% 91.919% ( 15, 22 ] 527 0.056% 91.976% ( 22, 34 ] 61 0.007% 91.982% ( 34, 51 ] 65 0.007% 91.989% ( 51, 76 ] 91 0.010% 91.999% ( 76, 110 ] 11264 1.206% 93.205% ( 110, 170 ] 37509 4.015% 97.220% # ( 170, 250 ] 11276 1.207% 98.427% ( 250, 380 ] 4065 0.435% 98.862% ( 380, 580 ] 3921 0.420% 99.282% ( 580, 870 ] 2593 0.278% 99.560% ( 870, 1300 ] 1825 0.195% 99.755% ( 1300, 1900 ] 1211 0.130% 99.885% ( 1900, 2900 ] 864 0.092% 99.977% ( 2900, 4400 ] 807 0.086% 100.064% ( 4400, 6600 ] 294 0.031% 100.095% ( 6600, 9900 ] 44 0.005% 100.100% ( 9900, 14000 ] 6 0.001% 100.101% ( 14000, 22000 ] 18 0.002% 100.102% ( 22000, 33000 ] 31 0.003% 100.106% ( 33000, 50000 ] 5 0.001% 100.106% ** Level 5 read latency histogram (micros): Count: 414733 Average: 244.6224 StdDev: 529.76 Min: 74 Median: 148.4742 Max: 39983 Percentiles: P50: 148.47 P75: 200.77 P99: 2468.61 P99.9: 6076.26 P99.99: 15122.98 ------------------------------------------------------ ( 51, 76 ] 35 0.008% 0.008% ( 76, 110 ] 66510 16.037% 16.045% ### ( 110, 170 ] 219610 52.952% 68.997% ########### ( 170, 250 ] 64734 15.609% 84.606% ### ( 250, 380 ] 22779 5.492% 90.098% # ( 380, 580 ] 19237 4.638% 94.737% # ( 580, 870 ] 8930 2.153% 96.890% ( 870, 1300 ] 4890 1.179% 98.069% ( 1300, 1900 ] 2671 0.644% 98.713% ( 1900, 2900 ] 2094 0.505% 99.218% ( 2900, 4400 ] 2110 0.509% 99.727% ( 4400, 6600 ] 944 0.228% 99.954% ( 6600, 9900 ] 134 0.032% 99.987% ( 9900, 14000 ] 12 0.003% 99.990% ( 14000, 22000 ] 18 0.004% 99.994% ( 22000, 33000 ] 38 0.009% 100.003% ( 33000, 50000 ] 1 0.000% 100.003% ** DB Stats ** Uptime(secs): 39.8 total, 14.1 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 123.62 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:20.863 H:M:S, 52.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 69.89 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:10.095 H:M:S, 71.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 251.203 micros/op 15912 ops/sec 60.077 seconds 955996 operations; 15.4 MB/s (237999 of 237999 found) Microseconds per read: Count: 955996 Average: 251.2052 StdDev: 716.10 Min: 2 Median: 179.0176 Max: 72457 Percentiles: P50: 179.02 P75: 238.83 P99: 1198.63 P99.9: 12576.15 P99.99: 29161.73 ------------------------------------------------------ ( 1, 2 ] 1 0.000% 0.000% ( 2, 3 ] 22 0.002% 0.002% ( 3, 4 ] 45 0.005% 0.007% ( 4, 6 ] 77 0.008% 0.015% ( 6, 10 ] 517 0.054% 0.069% ( 10, 15 ] 3514 0.368% 0.437% ( 15, 22 ] 3114 0.326% 0.763% ( 22, 34 ] 9118 0.954% 1.716% ( 34, 51 ] 2510 0.263% 1.979% ( 51, 76 ] 408 0.043% 2.022% ( 76, 110 ] 17843 1.866% 3.888% ( 110, 170 ] 404799 42.343% 46.231% ######## ( 170, 250 ] 319642 33.435% 79.667% ####### ( 250, 380 ] 116175 12.152% 91.819% ## ( 380, 580 ] 46127 4.825% 96.644% # ( 580, 870 ] 17360 1.816% 98.460% ( 870, 1300 ] 6757 0.707% 99.167% ( 1300, 1900 ] 2685 0.281% 99.447% ( 1900, 2900 ] 1762 0.184% 99.632% ( 2900, 4400 ] 1480 0.155% 99.787% ( 4400, 6600 ] 593 0.062% 99.849% ( 6600, 9900 ] 226 0.024% 99.872% ( 9900, 14000 ] 406 0.042% 99.915% ( 14000, 22000 ] 597 0.062% 99.977% ( 22000, 33000 ] 188 0.020% 99.997% ( 33000, 50000 ] 25 0.003% 99.999% ( 50000, 75000 ] 5 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 17107258 rocksdb.bloom.filter.full.positive COUNT : 1122698 rocksdb.bloom.filter.full.true.positive COUNT : 955622 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 374 rocksdb.memtable.miss COUNT : 955622 rocksdb.l0.hit COUNT : 42631 rocksdb.l1.hit COUNT : 743 rocksdb.l2andup.hit COUNT : 912248 rocksdb.compaction.key.drop.new COUNT : 164485 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5081309 rocksdb.number.keys.read COUNT : 955996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5243910888 rocksdb.bytes.read COUNT : 955996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1588 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 40571442 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 21307127701 rocksdb.compact.write.bytes COUNT : 21134472006 rocksdb.flush.write.bytes COUNT : 5218699903 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 42 rocksdb.number.superversion_acquires COUNT : 1003 rocksdb.number.superversion_releases COUNT : 953 rocksdb.number.superversion_cleanups COUNT : 245 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 158 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5176468480 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1874944 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 25463270552 rocksdb.non.last.level.read.count COUNT : 6010703 rocksdb.block.checksum.compute.count COUNT : 6293231 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 177.409017 P95 : 510.746582 P99 : 1197.740226 P100 : 72456.000000 COUNT : 955996 SUM : 239239554 rocksdb.db.write.micros P50 : 1.540720 P95 : 4.511753 P99 : 9.788123 P100 : 944226.000000 COUNT : 5081309 SUM : 53417089 rocksdb.compaction.times.micros P50 : 1588888.888889 P95 : 5070000.000000 P99 : 10155150.000000 P100 : 10155150.000000 COUNT : 72 SUM : 134410634 rocksdb.compaction.times.cpu_micros P50 : 1007333.333333 P95 : 2700000.000000 P99 : 5866805.000000 P100 : 5866805.000000 COUNT : 72 SUM : 83875655 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 777.051282 P95 : 102416.666667 P99 : 332550.000000 P100 : 334925.000000 COUNT : 73 SUM : 1280313 rocksdb.compaction.outfile.sync.micros P50 : 28769.230769 P95 : 229200.000000 P99 : 444600.000000 P100 : 815753.000000 COUNT : 232 SUM : 14963957 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 190.000000 P95 : 4231.250000 P99 : 7689.000000 P100 : 34906.000000 COUNT : 167 SUM : 199463 rocksdb.table.open.io.micros P50 : 9394.285714 P95 : 19859.915612 P99 : 22082.500000 P100 : 27032.000000 COUNT : 1588 SUM : 14688516 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.780183 P95 : 90.149885 P99 : 210.402390 P100 : 43978.000000 COUNT : 5159207 SUM : 92850208 rocksdb.read.block.get.micros P50 : 141.471331 P95 : 366.609851 P99 : 1122.634001 P100 : 39988.000000 COUNT : 1132416 SUM : 204173792 rocksdb.write.raw.block.micros P50 : 0.529179 P95 : 1.149679 P99 : 3.570492 P100 : 55959.000000 COUNT : 6382629 SUM : 19676624 rocksdb.numfiles.in.singlecompaction P50 : 3.197674 P95 : 6.250000 P99 : 10.250000 P100 : 12.000000 COUNT : 95 SUM : 347 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.576315 P95 : 2010.497487 P99 : 2723.144724 P100 : 944053.000000 COUNT : 177271 SUM : 40588918 rocksdb.sst.read.micros P50 : 1.914528 P95 : 168.590715 P99 : 479.329651 P100 : 43970.000000 COUNT : 6010730 SUM : 279734345 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 955996 SUM : 955996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5081309 SUM : 5243910888 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 220344.827586 P95 : 993166.666667 P99 : 1158633.333333 P100 : 1176531.000000 COUNT : 73 SUM : 22387628 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 260 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0