[2024-11-17 15:50:59.922494] Starting SPDK v25.01-pre git sha1 83e8405e4 / DPDK 24.03.0 initialization... [2024-11-17 15:50:59.922594] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1712502 ] Set seed to 1731855059920067 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-17 15:51:00.037410] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-17 15:51:00.150170] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-17 15:51:00.150217] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1712502' to capture a snapshot of events at runtime. [2024-11-17 15:51:00.150229] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-17 15:51:00.150240] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-17 15:51:00.150248] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1712502 for offline analysis/debug. [2024-11-17 15:51:00.151568] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sun Nov 17 15:51:08 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/17-15:51:16 ... thread 0: (1000000,1000000) ops and (130573.4,130573.4) ops/second in (7.658530,7.658530) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 9/7 1.37 GB 44.0 0.4 0.0 0.4 1.2 0.8 0.0 1.5 38.2 120.9 9.90 3.90 10 0.990 383K 504 0.0 0.0 L1 5/5 514.15 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 128.78 MB 1.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 44/9 2.85 GB 2.2 2.7 1.7 1.0 2.7 1.7 0.1 1.6 99.3 98.5 28.32 13.39 14 2.023 1520K 0 0.0 0.0 L4 189/16 11.82 GB 1.1 2.7 2.1 0.6 2.7 2.1 0.0 1.3 99.4 99.0 28.04 13.22 19 1.476 1006K 500 0.0 0.0 L5 1084/0 89.14 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1332/37 105.80 GB 0.0 5.8 3.8 2.0 6.6 4.6 0.1 8.3 90.2 102.1 66.25 30.51 43 1.541 2910K 1004 0.0 0.0 Int 0/0 0.00 KB 0.0 5.8 3.8 2.0 6.6 4.6 0.1 8.3 90.2 102.1 66.25 30.51 43 1.541 2910K 1004 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.8 3.8 2.0 5.8 3.8 0.0 0.0 101.0 100.4 59.19 28.17 34 1.741 2910K 1004 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 115.9 7.06 2.35 9 0.785 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.9 total, 7.7 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.60 GB write, 859.97 MB/s write, 5.84 GB read, 760.11 MB/s read, 66.2 seconds Interval compaction: 6.60 GB write, 882.72 MB/s write, 5.84 GB read, 780.22 MB/s read, 66.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 31, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 31, interval: 31 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 248261 Average: 14.0673 StdDev: 179.14 Min: 1 Median: 1.9728 Max: 10055 Percentiles: P50: 1.97 P75: 2.92 P99: 11.02 P99.9: 2806.11 P99.99: 4363.80 ------------------------------------------------------ [ 0, 1 ] 5903 2.378% 2.378% ( 1, 2 ] 121535 48.955% 51.332% ########## ( 2, 3 ] 63710 25.663% 76.995% ##### ( 3, 4 ] 22873 9.213% 86.208% ## ( 4, 6 ] 25340 10.207% 96.415% ## ( 6, 10 ] 6168 2.484% 98.900% ( 10, 15 ] 1219 0.491% 99.391% ( 15, 22 ] 155 0.062% 99.453% ( 22, 34 ] 7 0.003% 99.456% ( 34, 51 ] 4 0.002% 99.457% ( 51, 76 ] 7 0.003% 99.460% ( 110, 170 ] 6 0.002% 99.463% ( 170, 250 ] 15 0.006% 99.469% ( 250, 380 ] 30 0.012% 99.481% ( 380, 580 ] 71 0.029% 99.509% ( 580, 870 ] 150 0.060% 99.570% ( 870, 1300 ] 242 0.097% 99.667% ( 1300, 1900 ] 305 0.123% 99.790% ( 1900, 2900 ] 301 0.121% 99.911% ( 2900, 4400 ] 200 0.081% 99.992% ( 4400, 6600 ] 67 0.027% 100.019% ( 6600, 9900 ] 13 0.005% 100.024% ( 9900, 14000 ] 1 0.000% 100.025% ** Level 1 read latency histogram (micros): Count: 74285 Average: 17.8869 StdDev: 215.34 Min: 1 Median: 2.1458 Max: 10917 Percentiles: P50: 2.15 P75: 3.09 P99: 14.42 P99.9: 3778.58 P99.99: 7248.24 ------------------------------------------------------ [ 0, 1 ] 960 1.292% 1.292% ( 1, 2 ] 33128 44.596% 45.888% ######### ( 2, 3 ] 20953 28.206% 74.094% ###### ( 3, 4 ] 7278 9.797% 83.892% ## ( 4, 6 ] 8475 11.409% 95.301% ## ( 6, 10 ] 2250 3.029% 98.329% # ( 10, 15 ] 564 0.759% 99.089% ( 15, 22 ] 128 0.172% 99.261% ( 22, 34 ] 8 0.011% 99.272% ( 34, 51 ] 2 0.003% 99.274% ( 76, 110 ] 2 0.003% 99.277% ( 110, 170 ] 3 0.004% 99.281% ( 170, 250 ] 6 0.008% 99.289% ( 250, 380 ] 13 0.018% 99.307% ( 380, 580 ] 38 0.051% 99.358% ( 580, 870 ] 63 0.085% 99.443% ( 870, 1300 ] 88 0.118% 99.561% ( 1300, 1900 ] 91 0.123% 99.684% ( 1900, 2900 ] 108 0.145% 99.829% ( 2900, 4400 ] 90 0.121% 99.950% ( 4400, 6600 ] 28 0.038% 99.988% ( 6600, 9900 ] 8 0.011% 99.999% ( 9900, 14000 ] 1 0.001% 100.000% ** Level 2 read latency histogram (micros): Count: 318180 Average: 18.8716 StdDev: 226.17 Min: 1 Median: 2.0987 Max: 12601 Percentiles: P50: 2.10 P75: 3.06 P99: 13.56 P99.9: 2579.82 P99.99: 3324.93 ------------------------------------------------------ [ 0, 1 ] 5512 1.732% 1.732% ( 1, 2 ] 145091 45.600% 47.333% ######### ( 2, 3 ] 86021 27.035% 74.368% ##### ( 3, 4 ] 31420 9.875% 84.243% ## ( 4, 6 ] 35592 11.186% 95.429% ## ( 6, 10 ] 9527 2.994% 98.423% # ( 10, 15 ] 2576 0.810% 99.233% ( 15, 22 ] 494 0.155% 99.388% ( 22, 34 ] 41 0.013% 99.401% ( 34, 51 ] 18 0.006% 99.407% ( 51, 76 ] 8 0.003% 99.409% ( 76, 110 ] 7 0.002% 99.411% ( 110, 170 ] 12 0.004% 99.415% ( 170, 250 ] 26 0.008% 99.423% ( 250, 380 ] 50 0.016% 99.439% ( 380, 580 ] 136 0.043% 99.482% ( 580, 870 ] 198 0.062% 99.544% ( 870, 1300 ] 314 0.099% 99.643% ( 1300, 1900 ] 449 0.141% 99.784% ( 1900, 2900 ] 544 0.171% 99.955% ( 2900, 4400 ] 396 0.124% 100.079% ( 4400, 6600 ] 143 0.045% 100.124% ( 6600, 9900 ] 33 0.010% 100.135% ( 9900, 14000 ] 9 0.003% 100.137% ** Level 3 read latency histogram (micros): Count: 856179 Average: 16.5015 StdDev: 194.99 Min: 1 Median: 2.1851 Max: 12291 Percentiles: P50: 2.19 P75: 3.02 P99: 10.92 P99.9: 1426.50 P99.99: 1789.12 ------------------------------------------------------ [ 0, 1 ] 8152 0.952% 0.952% ( 1, 2 ] 371648 43.408% 44.360% ######### ( 2, 3 ] 260870 30.469% 74.829% ###### ( 3, 4 ] 87020 10.164% 84.993% ## ( 4, 6 ] 95836 11.193% 96.186% ## ( 6, 10 ] 23094 2.697% 98.884% # ( 10, 15 ] 5426 0.634% 99.517% ( 15, 22 ] 796 0.093% 99.610% ( 22, 34 ] 64 0.007% 99.618% ( 34, 51 ] 28 0.003% 99.621% ( 51, 76 ] 26 0.003% 99.624% ( 76, 110 ] 18 0.002% 99.626% ( 110, 170 ] 30 0.004% 99.630% ( 170, 250 ] 65 0.008% 99.637% ( 250, 380 ] 123 0.014% 99.652% ( 380, 580 ] 270 0.032% 99.683% ( 580, 870 ] 562 0.066% 99.749% ( 870, 1300 ] 1028 0.120% 99.869% ( 1300, 1900 ] 1275 0.149% 100.018% ( 1900, 2900 ] 1292 0.151% 100.169% ( 2900, 4400 ] 861 0.101% 100.269% ( 4400, 6600 ] 279 0.033% 100.302% ( 6600, 9900 ] 39 0.005% 100.306% ( 9900, 14000 ] 2 0.000% 100.307% ** Level 4 read latency histogram (micros): Count: 258724 Average: 37.2974 StdDev: 312.98 Min: 1 Median: 2.1296 Max: 11168 Percentiles: P50: 2.13 P75: 2.99 P99: 1191.29 P99.9: 3396.50 P99.99: 3982.54 ------------------------------------------------------ [ 0, 1 ] 2845 1.100% 1.100% ( 1, 2 ] 116765 45.131% 46.231% ######### ( 2, 3 ] 75226 29.076% 75.307% ###### ( 3, 4 ] 26047 10.067% 85.374% ## ( 4, 6 ] 25892 10.008% 95.382% ## ( 6, 10 ] 6213 2.401% 97.783% ( 10, 15 ] 1329 0.514% 98.297% ( 15, 22 ] 196 0.076% 98.372% ( 22, 34 ] 12 0.005% 98.377% ( 34, 51 ] 11 0.004% 98.381% ( 51, 76 ] 19 0.007% 98.389% ( 76, 110 ] 9 0.003% 98.392% ( 110, 170 ] 15 0.006% 98.398% ( 170, 250 ] 61 0.024% 98.421% ( 250, 380 ] 111 0.043% 98.464% ( 380, 580 ] 249 0.096% 98.561% ( 580, 870 ] 544 0.210% 98.771% ( 870, 1300 ] 796 0.308% 99.079% ( 1300, 1900 ] 982 0.380% 99.458% ( 1900, 2900 ] 948 0.366% 99.825% ( 2900, 4400 ] 596 0.230% 100.055% ( 4400, 6600 ] 246 0.095% 100.150% ( 6600, 9900 ] 36 0.014% 100.164% ( 9900, 14000 ] 1 0.000% 100.164% ** Level 5 read latency histogram (micros): Count: 9119 Average: 2367.2830 StdDev: 1707.34 Min: 100 Median: 1855.4627 Max: 13754 Percentiles: P50: 1855.46 P75: 3434.48 P99: 8558.36 P99.9: 9835.88 P99.99: 12693.53 ------------------------------------------------------ ( 76, 110 ] 4 0.044% 0.044% ( 110, 170 ] 14 0.154% 0.197% ( 170, 250 ] 57 0.625% 0.822% ( 250, 380 ] 186 2.040% 2.862% ( 380, 580 ] 457 5.012% 7.874% # ( 580, 870 ] 971 10.648% 18.522% ## ( 870, 1300 ] 1430 15.682% 34.203% ### ( 1300, 1900 ] 1556 17.063% 51.267% ### ( 1900, 2900 ] 1561 17.118% 68.385% ### ( 2900, 4400 ] 1693 18.566% 86.950% #### ( 4400, 6600 ] 973 10.670% 97.620% ## ( 6600, 9900 ] 212 2.325% 99.945% ( 9900, 14000 ] 6 0.066% 100.011% ** DB Stats ** Uptime(secs): 7.9 total, 7.7 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 125.16 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.887 H:M:S, 49.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 128.46 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.887 H:M:S, 50.7 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/17-15:51:22 ... thread 0: (1000000,2000000) ops and (155070.3,141771.4) ops/second in (6.448687,14.107217) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 13/10 1.84 GB 44.1 1.0 0.0 1.0 2.9 1.8 0.0 1.6 51.4 142.1 20.82 9.78 25 0.833 1087K 1379 0.0 0.0 L1 2/2 173.89 MB 0.0 1.1 0.6 0.5 1.1 0.6 0.0 1.9 112.8 112.4 9.71 5.23 1 9.708 1111K 4015 0.0 0.0 L2 2/1 186.46 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 26/8 1.74 GB 1.0 4.2 2.4 1.7 4.1 2.4 0.2 1.7 111.8 111.0 38.29 20.26 21 1.823 2903K 4649 0.0 0.0 L4 169/6 9.75 GB 1.0 6.6 4.0 2.6 6.5 4.0 0.0 1.6 109.2 107.8 61.96 32.14 45 1.377 2769K 3308 0.0 0.0 L5 1132/0 93.08 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1344/27 106.76 GB 0.0 12.9 7.0 5.9 14.6 8.8 5.1 7.9 101.0 114.5 130.78 67.40 92 1.422 7871K 13K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.9 7.0 5.9 14.6 8.8 5.1 7.9 101.0 114.5 130.78 67.40 92 1.422 7871K 13K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.9 7.0 5.9 12.8 6.9 0.0 0.0 112.6 111.5 117.36 62.04 70 1.677 7871K 13K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 140.8 13.42 5.37 22 0.610 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.3 total, 14.1 interval Flush(GB): cumulative 1.845, interval 1.845 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 14.63 GB write, 1046.74 MB/s write, 12.90 GB read, 923.15 MB/s read, 130.8 seconds Interval compaction: 14.63 GB write, 1061.77 MB/s write, 12.90 GB read, 936.41 MB/s read, 130.8 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 57, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 57, interval: 57 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 605014 Average: 11.8703 StdDev: 150.44 Min: 1 Median: 1.9424 Max: 25095 Percentiles: P50: 1.94 P75: 2.86 P99: 11.67 P99.9: 2070.13 P99.99: 3767.32 ------------------------------------------------------ [ 0, 1 ] 16259 2.687% 2.687% # ( 1, 2 ] 303737 50.203% 52.891% ########## ( 2, 3 ] 155089 25.634% 78.525% ##### ( 3, 4 ] 53029 8.765% 87.290% ## ( 4, 6 ] 57044 9.429% 96.718% ## ( 6, 10 ] 13030 2.154% 98.872% ( 10, 15 ] 2329 0.385% 99.257% ( 15, 22 ] 358 0.059% 99.316% ( 22, 34 ] 17 0.003% 99.319% ( 34, 51 ] 21 0.003% 99.322% ( 51, 76 ] 15 0.002% 99.325% ( 76, 110 ] 15 0.002% 99.327% ( 110, 170 ] 167 0.028% 99.355% ( 170, 250 ] 348 0.058% 99.412% ( 250, 380 ] 410 0.068% 99.480% ( 380, 580 ] 513 0.085% 99.565% ( 580, 870 ] 689 0.114% 99.679% ( 870, 1300 ] 720 0.119% 99.798% ( 1300, 1900 ] 546 0.090% 99.888% ( 1900, 2900 ] 429 0.071% 99.959% ( 2900, 4400 ] 326 0.054% 100.013% ( 4400, 6600 ] 107 0.018% 100.030% ( 6600, 9900 ] 15 0.002% 100.033% ( 9900, 14000 ] 2 0.000% 100.033% ( 22000, 33000 ] 1 0.000% 100.033% ** Level 1 read latency histogram (micros): Count: 313501 Average: 11.9271 StdDev: 173.22 Min: 1 Median: 2.1204 Max: 45629 Percentiles: P50: 2.12 P75: 2.95 P99: 11.84 P99.9: 1991.87 P99.99: 3464.73 ------------------------------------------------------ [ 0, 1 ] 3291 1.050% 1.050% ( 1, 2 ] 142118 45.333% 46.382% ######### ( 2, 3 ] 94240 30.061% 76.443% ###### ( 3, 4 ] 29280 9.340% 85.783% ## ( 4, 6 ] 32393 10.333% 96.115% ## ( 6, 10 ] 8422 2.686% 98.802% # ( 10, 15 ] 1695 0.541% 99.342% ( 15, 22 ] 285 0.091% 99.433% ( 22, 34 ] 19 0.006% 99.439% ( 34, 51 ] 20 0.006% 99.446% ( 51, 76 ] 39 0.012% 99.458% ( 76, 110 ] 8 0.003% 99.461% ( 110, 170 ] 44 0.014% 99.475% ( 170, 250 ] 75 0.024% 99.499% ( 250, 380 ] 114 0.036% 99.535% ( 380, 580 ] 238 0.076% 99.611% ( 580, 870 ] 299 0.095% 99.706% ( 870, 1300 ] 326 0.104% 99.810% ( 1300, 1900 ] 261 0.083% 99.893% ( 1900, 2900 ] 234 0.075% 99.968% ( 2900, 4400 ] 185 0.059% 100.027% ( 4400, 6600 ] 49 0.016% 100.043% ( 6600, 9900 ] 15 0.005% 100.048% ( 9900, 14000 ] 2 0.001% 100.048% ( 33000, 50000 ] 1 0.000% 100.048% ** Level 2 read latency histogram (micros): Count: 350944 Average: 18.1240 StdDev: 219.01 Min: 1 Median: 2.1040 Max: 12601 Percentiles: P50: 2.10 P75: 3.05 P99: 13.52 P99.9: 2560.28 P99.99: 3361.19 ------------------------------------------------------ [ 0, 1 ] 5833 1.662% 1.662% ( 1, 2 ] 159667 45.496% 47.159% ######### ( 2, 3 ] 95883 27.321% 74.480% ##### ( 3, 4 ] 34681 9.882% 84.362% ## ( 4, 6 ] 38932 11.094% 95.456% ## ( 6, 10 ] 10512 2.995% 98.451% # ( 10, 15 ] 2737 0.780% 99.231% ( 15, 22 ] 510 0.145% 99.376% ( 22, 34 ] 45 0.013% 99.389% ( 34, 51 ] 36 0.010% 99.399% ( 51, 76 ] 20 0.006% 99.405% ( 76, 110 ] 10 0.003% 99.408% ( 110, 170 ] 19 0.005% 99.413% ( 170, 250 ] 29 0.008% 99.422% ( 250, 380 ] 62 0.018% 99.439% ( 380, 580 ] 170 0.048% 99.488% ( 580, 870 ] 242 0.069% 99.557% ( 870, 1300 ] 359 0.102% 99.659% ( 1300, 1900 ] 473 0.135% 99.794% ( 1900, 2900 ] 565 0.161% 99.955% ( 2900, 4400 ] 403 0.115% 100.070% ( 4400, 6600 ] 151 0.043% 100.113% ( 6600, 9900 ] 33 0.009% 100.122% ( 9900, 14000 ] 9 0.003% 100.125% ** Level 3 read latency histogram (micros): Count: 1513223 Average: 13.2381 StdDev: 168.75 Min: 1 Median: 2.1545 Max: 25058 Percentiles: P50: 2.15 P75: 2.98 P99: 9.96 P99.9: 1088.11 P99.99: 1503.90 ------------------------------------------------------ [ 0, 1 ] 14190 0.938% 0.938% ( 1, 2 ] 671987 44.408% 45.345% ######### ( 2, 3 ] 455742 30.117% 75.463% ###### ( 3, 4 ] 150689 9.958% 85.421% ## ( 4, 6 ] 165534 10.939% 96.360% ## ( 6, 10 ] 40312 2.664% 99.024% # ( 10, 15 ] 8262 0.546% 99.570% ( 15, 22 ] 1255 0.083% 99.653% ( 22, 34 ] 112 0.007% 99.660% ( 34, 51 ] 88 0.006% 99.666% ( 51, 76 ] 96 0.006% 99.672% ( 76, 110 ] 44 0.003% 99.675% ( 110, 170 ] 94 0.006% 99.682% ( 170, 250 ] 249 0.016% 99.698% ( 250, 380 ] 378 0.025% 99.723% ( 380, 580 ] 685 0.045% 99.768% ( 580, 870 ] 1166 0.077% 99.845% ( 870, 1300 ] 1630 0.108% 99.953% ( 1300, 1900 ] 1644 0.109% 100.062% ( 1900, 2900 ] 1569 0.104% 100.165% ( 2900, 4400 ] 1087 0.072% 100.237% ( 4400, 6600 ] 335 0.022% 100.259% ( 6600, 9900 ] 53 0.004% 100.263% ( 9900, 14000 ] 4 0.000% 100.263% ( 14000, 22000 ] 2 0.000% 100.263% ( 22000, 33000 ] 1 0.000% 100.263% ** Level 4 read latency histogram (micros): Count: 686030 Average: 24.6261 StdDev: 246.55 Min: 1 Median: 2.1492 Max: 44393 Percentiles: P50: 2.15 P75: 3.03 P99: 445.21 P99.9: 2358.21 P99.99: 2831.34 ------------------------------------------------------ [ 0, 1 ] 7040 1.026% 1.026% ( 1, 2 ] 306292 44.647% 45.673% ######### ( 2, 3 ] 199000 29.007% 74.681% ###### ( 3, 4 ] 69095 10.072% 84.752% ## ( 4, 6 ] 73554 10.722% 95.474% ## ( 6, 10 ] 18293 2.667% 98.141% # ( 10, 15 ] 3490 0.509% 98.649% ( 15, 22 ] 462 0.067% 98.717% ( 22, 34 ] 35 0.005% 98.722% ( 34, 51 ] 50 0.007% 98.729% ( 51, 76 ] 63 0.009% 98.738% ( 76, 110 ] 38 0.006% 98.744% ( 110, 170 ] 240 0.035% 98.779% ( 170, 250 ] 494 0.072% 98.851% ( 250, 380 ] 666 0.097% 98.948% ( 380, 580 ] 1097 0.160% 99.108% ( 580, 870 ] 1554 0.227% 99.334% ( 870, 1300 ] 1756 0.256% 99.590% ( 1300, 1900 ] 1527 0.223% 99.813% ( 1900, 2900 ] 1305 0.190% 100.003% ( 2900, 4400 ] 910 0.133% 100.136% ( 4400, 6600 ] 340 0.050% 100.185% ( 6600, 9900 ] 55 0.008% 100.193% ( 9900, 14000 ] 4 0.001% 100.194% ( 33000, 50000 ] 2 0.000% 100.194% ** Level 5 read latency histogram (micros): Count: 30012 Average: 1361.5945 StdDev: 1450.26 Min: 85 Median: 852.8338 Max: 44478 Percentiles: P50: 852.83 P75: 1736.84 P99: 6501.63 P99.9: 9660.09 P99.99: 21995.20 ------------------------------------------------------ ( 76, 110 ] 133 0.443% 0.443% ( 110, 170 ] 1318 4.392% 4.835% # ( 170, 250 ] 2229 7.427% 12.262% # ( 250, 380 ] 2689 8.960% 21.222% ## ( 380, 580 ] 3853 12.838% 34.060% ### ( 580, 870 ] 5085 16.943% 51.003% ### ( 870, 1300 ] 4803 16.004% 67.007% ### ( 1300, 1900 ] 3295 10.979% 77.985% ## ( 1900, 2900 ] 2611 8.700% 86.685% ## ( 2900, 4400 ] 2561 8.533% 95.219% ## ( 4400, 6600 ] 1188 3.958% 99.177% # ( 6600, 9900 ] 234 0.780% 99.957% ( 9900, 14000 ] 8 0.027% 99.983% ( 14000, 22000 ] 2 0.007% 99.990% ( 22000, 33000 ] 2 0.007% 99.997% ( 33000, 50000 ] 2 0.007% 100.003% ** DB Stats ** Uptime(secs): 14.3 total, 6.4 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 137.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.492 H:M:S, 45.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 152.62 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.605 H:M:S, 40.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 6 num-running-flushes: 1 2024/11/17-15:51:26 ... thread 0: (1000000,3000000) ops and (231949.7,162879.7) ops/second in (4.311280,18.418497) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/10 2.77 GB 144.7 2.2 0.0 2.2 4.9 2.8 0.0 1.8 74.7 170.6 29.52 16.86 43 0.687 2238K 2579 0.0 0.0 L1 2/2 173.89 MB 0.0 1.1 0.6 0.5 1.1 0.6 0.0 1.9 112.8 112.4 9.71 5.23 1 9.708 1111K 4015 0.0 0.0 L2 1/0 57.70 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 926.19 MB 0.9 4.6 2.6 2.1 4.6 2.5 0.2 1.8 115.2 114.4 41.25 22.44 22 1.875 3121K 4649 0.0 0.0 L4 169/0 9.70 GB 1.0 8.8 5.0 3.8 8.7 4.9 0.0 1.7 115.7 114.3 77.52 42.49 53 1.463 4542K 4804 0.0 0.0 L5 1144/0 94.06 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1349/12 107.66 GB 0.0 16.6 8.1 8.5 19.2 10.8 6.1 7.0 107.7 124.7 158.00 87.01 119 1.328 11M 16K 0.0 0.0 Int 0/0 0.00 KB 0.0 16.6 8.1 8.5 19.2 10.8 6.1 7.0 107.7 124.7 158.00 87.01 119 1.328 11M 16K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.6 8.1 8.5 16.5 8.0 0.0 0.0 121.1 120.0 140.56 79.02 83 1.693 11M 16K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 162.6 17.44 8.00 36 0.484 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.6 total, 18.4 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 19.24 GB write, 1058.13 MB/s write, 16.62 GB read, 913.97 MB/s read, 158.0 seconds Interval compaction: 19.24 GB write, 1069.77 MB/s write, 16.62 GB read, 924.02 MB/s read, 158.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 63, interval: 63 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1010823 Average: 10.7217 StdDev: 128.56 Min: 1 Median: 1.8767 Max: 25095 Percentiles: P50: 1.88 P75: 2.77 P99: 17.29 P99.9: 1469.82 P99.99: 2718.02 ------------------------------------------------------ [ 0, 1 ] 42638 4.218% 4.218% # ( 1, 2 ] 527872 52.222% 56.440% ########## ( 2, 3 ] 243292 24.069% 80.509% ##### ( 3, 4 ] 80546 7.968% 88.477% ## ( 4, 6 ] 84422 8.352% 96.829% ## ( 6, 10 ] 18626 1.843% 98.672% ( 10, 15 ] 3166 0.313% 98.985% ( 15, 22 ] 469 0.046% 99.031% ( 22, 34 ] 25 0.002% 99.034% ( 34, 51 ] 29 0.003% 99.037% ( 51, 76 ] 40 0.004% 99.041% ( 76, 110 ] 150 0.015% 99.055% ( 110, 170 ] 1551 0.153% 99.209% ( 170, 250 ] 1841 0.182% 99.391% ( 250, 380 ] 1364 0.135% 99.526% ( 380, 580 ] 1355 0.134% 99.660% ( 580, 870 ] 1271 0.126% 99.786% ( 870, 1300 ] 964 0.095% 99.881% ( 1300, 1900 ] 679 0.067% 99.948% ( 1900, 2900 ] 517 0.051% 99.999% ( 2900, 4400 ] 397 0.039% 100.039% ( 4400, 6600 ] 123 0.012% 100.051% ( 6600, 9900 ] 16 0.002% 100.052% ( 9900, 14000 ] 2 0.000% 100.053% ( 22000, 33000 ] 1 0.000% 100.053% ** Level 1 read latency histogram (micros): Count: 346580 Average: 11.5489 StdDev: 166.96 Min: 1 Median: 2.1142 Max: 45629 Percentiles: P50: 2.11 P75: 2.95 P99: 12.28 P99.9: 1910.08 P99.99: 3483.84 ------------------------------------------------------ [ 0, 1 ] 3902 1.126% 1.126% ( 1, 2 ] 157505 45.445% 46.571% ######### ( 2, 3 ] 104096 30.035% 76.607% ###### ( 3, 4 ] 32082 9.257% 85.863% ## ( 4, 6 ] 35411 10.217% 96.081% ## ( 6, 10 ] 9290 2.680% 98.761% # ( 10, 15 ] 1817 0.524% 99.285% ( 15, 22 ] 314 0.091% 99.376% ( 22, 34 ] 24 0.007% 99.383% ( 34, 51 ] 23 0.007% 99.389% ( 51, 76 ] 40 0.012% 99.401% ( 76, 110 ] 15 0.004% 99.405% ( 110, 170 ] 139 0.040% 99.445% ( 170, 250 ] 197 0.057% 99.502% ( 250, 380 ] 164 0.047% 99.550% ( 380, 580 ] 278 0.080% 99.630% ( 580, 870 ] 331 0.096% 99.725% ( 870, 1300 ] 336 0.097% 99.822% ( 1300, 1900 ] 267 0.077% 99.899% ( 1900, 2900 ] 240 0.069% 99.969% ( 2900, 4400 ] 191 0.055% 100.024% ( 4400, 6600 ] 52 0.015% 100.039% ( 6600, 9900 ] 15 0.004% 100.043% ( 9900, 14000 ] 2 0.001% 100.044% ( 33000, 50000 ] 1 0.000% 100.044% ** Level 2 read latency histogram (micros): Count: 350999 Average: 18.1950 StdDev: 219.21 Min: 1 Median: 2.1043 Max: 12601 Percentiles: P50: 2.10 P75: 3.05 P99: 13.62 P99.9: 2561.95 P99.99: 3363.74 ------------------------------------------------------ [ 0, 1 ] 5833 1.662% 1.662% ( 1, 2 ] 159667 45.489% 47.151% ######### ( 2, 3 ] 95883 27.317% 74.468% ##### ( 3, 4 ] 34681 9.881% 84.349% ## ( 4, 6 ] 38932 11.092% 95.441% ## ( 6, 10 ] 10512 2.995% 98.436% # ( 10, 15 ] 2737 0.780% 99.215% ( 15, 22 ] 510 0.145% 99.361% ( 22, 34 ] 45 0.013% 99.374% ( 34, 51 ] 36 0.010% 99.384% ( 51, 76 ] 20 0.006% 99.389% ( 76, 110 ] 10 0.003% 99.392% ( 110, 170 ] 36 0.010% 99.403% ( 170, 250 ] 40 0.011% 99.414% ( 250, 380 ] 69 0.020% 99.434% ( 380, 580 ] 175 0.050% 99.483% ( 580, 870 ] 250 0.071% 99.555% ( 870, 1300 ] 362 0.103% 99.658% ( 1300, 1900 ] 476 0.136% 99.793% ( 1900, 2900 ] 565 0.161% 99.954% ( 2900, 4400 ] 404 0.115% 100.070% ( 4400, 6600 ] 151 0.043% 100.113% ( 6600, 9900 ] 33 0.009% 100.122% ( 9900, 14000 ] 9 0.003% 100.125% ** Level 3 read latency histogram (micros): Count: 1754226 Average: 12.2790 StdDev: 160.09 Min: 1 Median: 2.1398 Max: 25058 Percentiles: P50: 2.14 P75: 2.97 P99: 9.87 P99.9: 1021.32 P99.99: 1457.62 ------------------------------------------------------ [ 0, 1 ] 16478 0.939% 0.939% ( 1, 2 ] 786920 44.859% 45.798% ######### ( 2, 3 ] 527134 30.049% 75.847% ###### ( 3, 4 ] 172520 9.835% 85.682% ## ( 4, 6 ] 189325 10.793% 96.474% ## ( 6, 10 ] 45744 2.608% 99.082% # ( 10, 15 ] 8987 0.512% 99.594% ( 15, 22 ] 1340 0.076% 99.671% ( 22, 34 ] 118 0.007% 99.677% ( 34, 51 ] 100 0.006% 99.683% ( 51, 76 ] 110 0.006% 99.689% ( 76, 110 ] 54 0.003% 99.692% ( 110, 170 ] 169 0.010% 99.702% ( 170, 250 ] 336 0.019% 99.721% ( 250, 380 ] 471 0.027% 99.748% ( 380, 580 ] 769 0.044% 99.792% ( 580, 870 ] 1282 0.073% 99.865% ( 870, 1300 ] 1747 0.100% 99.965% ( 1300, 1900 ] 1700 0.097% 100.061% ( 1900, 2900 ] 1631 0.093% 100.154% ( 2900, 4400 ] 1129 0.064% 100.219% ( 4400, 6600 ] 354 0.020% 100.239% ( 6600, 9900 ] 54 0.003% 100.242% ( 9900, 14000 ] 5 0.000% 100.242% ( 14000, 22000 ] 2 0.000% 100.242% ( 22000, 33000 ] 1 0.000% 100.243% ** Level 4 read latency histogram (micros): Count: 984176 Average: 22.7753 StdDev: 229.41 Min: 1 Median: 2.1572 Max: 44393 Percentiles: P50: 2.16 P75: 3.03 P99: 360.84 P99.9: 2072.78 P99.99: 2633.74 ------------------------------------------------------ [ 0, 1 ] 9120 0.927% 0.927% ( 1, 2 ] 437683 44.472% 45.399% ######### ( 2, 3 ] 287984 29.261% 74.660% ###### ( 3, 4 ] 98204 9.978% 84.638% ## ( 4, 6 ] 105069 10.676% 95.314% ## ( 6, 10 ] 26464 2.689% 98.003% # ( 10, 15 ] 4842 0.492% 98.495% ( 15, 22 ] 603 0.061% 98.556% ( 22, 34 ] 54 0.005% 98.562% ( 34, 51 ] 73 0.007% 98.569% ( 51, 76 ] 104 0.011% 98.580% ( 76, 110 ] 153 0.016% 98.595% ( 110, 170 ] 1156 0.117% 98.713% ( 170, 250 ] 1554 0.158% 98.871% ( 250, 380 ] 1491 0.151% 99.022% ( 380, 580 ] 1972 0.200% 99.223% ( 580, 870 ] 2279 0.232% 99.454% ( 870, 1300 ] 2318 0.236% 99.690% ( 1300, 1900 ] 1796 0.182% 99.872% ( 1900, 2900 ] 1579 0.160% 100.033% ( 2900, 4400 ] 1172 0.119% 100.152% ( 4400, 6600 ] 403 0.041% 100.193% ( 6600, 9900 ] 74 0.008% 100.200% ( 9900, 14000 ] 7 0.001% 100.201% ( 33000, 50000 ] 2 0.000% 100.201% ** Level 5 read latency histogram (micros): Count: 54538 Average: 966.7599 StdDev: 1239.24 Min: 84 Median: 528.1009 Max: 44478 Percentiles: P50: 528.10 P75: 1142.53 P99: 6111.80 P99.9: 9387.20 P99.99: 15456.53 ------------------------------------------------------ ( 76, 110 ] 747 1.370% 1.370% ( 110, 170 ] 6727 12.335% 13.704% ## ( 170, 250 ] 7798 14.298% 28.002% ### ( 250, 380 ] 6538 11.988% 39.990% ## ( 380, 580 ] 7372 13.517% 53.508% ### ( 580, 870 ] 7721 14.157% 67.665% ### ( 870, 1300 ] 6312 11.574% 79.238% ## ( 1300, 1900 ] 3881 7.116% 86.354% # ( 1900, 2900 ] 3014 5.526% 91.881% # ( 2900, 4400 ] 2896 5.310% 97.191% # ( 4400, 6600 ] 1268 2.325% 99.516% ( 6600, 9900 ] 248 0.455% 99.971% ( 9900, 14000 ] 10 0.018% 99.989% ( 14000, 22000 ] 3 0.006% 99.994% ( 22000, 33000 ] 2 0.004% 99.998% ( 33000, 50000 ] 2 0.004% 100.002% ** DB Stats ** Uptime(secs): 18.6 total, 4.3 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 158.54 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.166 H:M:S, 38.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 228.28 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.674 H:M:S, 15.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/17-15:51:30 ... thread 0: (1000000,4000000) ops and (295662.4,183480.1) ops/second in (3.382236,21.800733) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 16/8 2.64 GB 220.3 2.9 0.0 2.9 6.6 3.7 0.0 1.8 82.8 190.1 35.77 22.52 62 0.577 3006K 3315 0.0 0.0 L1 13/0 1.27 GB 129.9 2.3 1.7 0.7 2.3 1.7 0.0 1.4 127.4 126.8 18.84 11.74 2 9.422 2438K 11K 0.0 0.0 L2 1/0 57.70 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 926.19 MB 0.9 4.6 2.6 2.1 4.6 2.5 0.2 1.8 115.2 114.4 41.25 22.44 22 1.875 3121K 4649 0.0 0.0 L4 169/0 9.70 GB 1.0 8.8 5.0 3.8 8.7 4.9 0.0 1.7 115.7 114.3 77.52 42.49 53 1.463 4542K 4804 0.0 0.0 L5 1144/0 94.06 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1361/8 108.63 GB 0.0 18.6 9.2 9.4 22.2 12.8 6.1 5.9 110.1 131.3 173.38 99.19 139 1.247 13M 23K 0.0 0.0 Int 0/0 0.00 KB 0.0 18.6 9.2 9.4 22.2 12.8 6.1 5.9 110.1 131.3 173.38 99.19 139 1.247 13M 23K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 18.6 9.2 9.4 18.5 9.1 0.0 0.0 124.9 123.9 152.74 88.50 87 1.756 13M 23K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 186.1 20.64 10.69 52 0.397 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 22.0 total, 21.8 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 22.23 GB write, 1034.66 MB/s write, 18.64 GB read, 867.22 MB/s read, 173.4 seconds Interval compaction: 22.23 GB write, 1044.28 MB/s write, 18.64 GB read, 875.28 MB/s read, 173.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 63, interval: 63 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1273445 Average: 10.0291 StdDev: 116.17 Min: 1 Median: 1.7768 Max: 25095 Percentiles: P50: 1.78 P75: 2.61 P99: 149.45 P99.9: 1238.32 P99.99: 2510.45 ------------------------------------------------------ [ 0, 1 ] 101514 7.972% 7.972% ## ( 1, 2 ] 688959 54.102% 62.074% ########### ( 2, 3 ] 268761 21.105% 83.179% #### ( 3, 4 ] 86236 6.772% 89.950% # ( 4, 6 ] 88771 6.971% 96.921% # ( 6, 10 ] 19522 1.533% 98.454% ( 10, 15 ] 3427 0.269% 98.724% ( 15, 22 ] 506 0.040% 98.763% ( 22, 34 ] 28 0.002% 98.765% ( 34, 51 ] 31 0.002% 98.768% ( 51, 76 ] 49 0.004% 98.772% ( 76, 110 ] 414 0.033% 98.804% ( 110, 170 ] 3792 0.298% 99.102% ( 170, 250 ] 3788 0.297% 99.400% ( 250, 380 ] 2148 0.169% 99.568% ( 380, 580 ] 1914 0.150% 99.718% ( 580, 870 ] 1456 0.114% 99.833% ( 870, 1300 ] 1000 0.079% 99.911% ( 1300, 1900 ] 684 0.054% 99.965% ( 1900, 2900 ] 522 0.041% 100.006% ( 2900, 4400 ] 397 0.031% 100.037% ( 4400, 6600 ] 123 0.010% 100.047% ( 6600, 9900 ] 16 0.001% 100.048% ( 9900, 14000 ] 2 0.000% 100.048% ( 22000, 33000 ] 1 0.000% 100.048% ** Level 1 read latency histogram (micros): Count: 362467 Average: 11.7001 StdDev: 163.89 Min: 1 Median: 2.1529 Max: 45629 Percentiles: P50: 2.15 P75: 2.99 P99: 14.28 P99.9: 1874.61 P99.99: 3471.36 ------------------------------------------------------ [ 0, 1 ] 4013 1.107% 1.107% ( 1, 2 ] 160739 44.346% 45.453% ######### ( 2, 3 ] 107809 29.743% 75.196% ###### ( 3, 4 ] 34833 9.610% 84.806% ## ( 4, 6 ] 39494 10.896% 95.702% ## ( 6, 10 ] 10223 2.820% 98.522% # ( 10, 15 ] 2023 0.558% 99.080% ( 15, 22 ] 430 0.119% 99.199% ( 22, 34 ] 41 0.011% 99.210% ( 34, 51 ] 23 0.006% 99.217% ( 51, 76 ] 40 0.011% 99.228% ( 76, 110 ] 66 0.018% 99.246% ( 110, 170 ] 457 0.126% 99.372% ( 170, 250 ] 381 0.105% 99.477% ( 250, 380 ] 225 0.062% 99.539% ( 380, 580 ] 344 0.095% 99.634% ( 580, 870 ] 358 0.099% 99.733% ( 870, 1300 ] 346 0.095% 99.828% ( 1300, 1900 ] 271 0.075% 99.903% ( 1900, 2900 ] 242 0.067% 99.970% ( 2900, 4400 ] 191 0.053% 100.023% ( 4400, 6600 ] 52 0.014% 100.037% ( 6600, 9900 ] 15 0.004% 100.041% ( 9900, 14000 ] 2 0.001% 100.042% ( 33000, 50000 ] 1 0.000% 100.042% ** Level 2 read latency histogram (micros): Count: 351081 Average: 18.2385 StdDev: 219.21 Min: 1 Median: 2.1047 Max: 12601 Percentiles: P50: 2.10 P75: 3.06 P99: 13.77 P99.9: 2561.80 P99.99: 3363.71 ------------------------------------------------------ [ 0, 1 ] 5833 1.661% 1.661% ( 1, 2 ] 159667 45.479% 47.140% ######### ( 2, 3 ] 95883 27.311% 74.451% ##### ( 3, 4 ] 34681 9.878% 84.329% ## ( 4, 6 ] 38932 11.089% 95.418% ## ( 6, 10 ] 10512 2.994% 98.413% # ( 10, 15 ] 2737 0.780% 99.192% ( 15, 22 ] 510 0.145% 99.337% ( 22, 34 ] 45 0.013% 99.350% ( 34, 51 ] 36 0.010% 99.361% ( 51, 76 ] 20 0.006% 99.366% ( 76, 110 ] 15 0.004% 99.371% ( 110, 170 ] 77 0.022% 99.392% ( 170, 250 ] 60 0.017% 99.410% ( 250, 380 ] 76 0.022% 99.431% ( 380, 580 ] 183 0.052% 99.483% ( 580, 870 ] 251 0.071% 99.555% ( 870, 1300 ] 362 0.103% 99.658% ( 1300, 1900 ] 476 0.136% 99.793% ( 1900, 2900 ] 565 0.161% 99.954% ( 2900, 4400 ] 404 0.115% 100.069% ( 4400, 6600 ] 151 0.043% 100.113% ( 6600, 9900 ] 33 0.009% 100.122% ( 9900, 14000 ] 9 0.003% 100.124% ** Level 3 read latency histogram (micros): Count: 1754696 Average: 12.3308 StdDev: 160.11 Min: 1 Median: 2.1403 Max: 25058 Percentiles: P50: 2.14 P75: 2.97 P99: 9.92 P99.9: 1021.20 P99.99: 1457.60 ------------------------------------------------------ [ 0, 1 ] 16478 0.939% 0.939% ( 1, 2 ] 786920 44.847% 45.786% ######### ( 2, 3 ] 527134 30.041% 75.827% ###### ( 3, 4 ] 172520 9.832% 85.659% ## ( 4, 6 ] 189325 10.790% 96.448% ## ( 6, 10 ] 45744 2.607% 99.055% # ( 10, 15 ] 8987 0.512% 99.568% ( 15, 22 ] 1340 0.076% 99.644% ( 22, 34 ] 118 0.007% 99.651% ( 34, 51 ] 100 0.006% 99.656% ( 51, 76 ] 110 0.006% 99.663% ( 76, 110 ] 82 0.005% 99.667% ( 110, 170 ] 391 0.022% 99.690% ( 170, 250 ] 465 0.027% 99.716% ( 250, 380 ] 520 0.030% 99.746% ( 380, 580 ] 798 0.045% 99.791% ( 580, 870 ] 1295 0.074% 99.865% ( 870, 1300 ] 1747 0.100% 99.965% ( 1300, 1900 ] 1700 0.097% 100.061% ( 1900, 2900 ] 1631 0.093% 100.154% ( 2900, 4400 ] 1129 0.064% 100.219% ( 4400, 6600 ] 354 0.020% 100.239% ( 6600, 9900 ] 54 0.003% 100.242% ( 9900, 14000 ] 5 0.000% 100.242% ( 14000, 22000 ] 2 0.000% 100.242% ( 22000, 33000 ] 1 0.000% 100.242% ** Level 4 read latency histogram (micros): Count: 991355 Average: 24.0706 StdDev: 229.49 Min: 1 Median: 2.1694 Max: 44393 Percentiles: P50: 2.17 P75: 3.09 P99: 416.48 P99.9: 2070.97 P99.99: 2634.60 ------------------------------------------------------ [ 0, 1 ] 9120 0.920% 0.920% ( 1, 2 ] 437740 44.156% 45.076% ######### ( 2, 3 ] 288198 29.071% 74.147% ###### ( 3, 4 ] 98310 9.917% 84.064% ## ( 4, 6 ] 105243 10.616% 94.680% ## ( 6, 10 ] 26486 2.672% 97.351% # ( 10, 15 ] 4853 0.490% 97.841% ( 15, 22 ] 603 0.061% 97.902% ( 22, 34 ] 54 0.005% 97.907% ( 34, 51 ] 73 0.007% 97.914% ( 51, 76 ] 104 0.010% 97.925% ( 76, 110 ] 591 0.060% 97.985% ( 110, 170 ] 4140 0.418% 98.402% ( 170, 250 ] 3311 0.334% 98.736% ( 250, 380 ] 2164 0.218% 98.954% ( 380, 580 ] 2475 0.250% 99.204% ( 580, 870 ] 2466 0.249% 99.453% ( 870, 1300 ] 2365 0.239% 99.691% ( 1300, 1900 ] 1797 0.181% 99.873% ( 1900, 2900 ] 1583 0.160% 100.032% ( 2900, 4400 ] 1173 0.118% 100.151% ( 4400, 6600 ] 403 0.041% 100.191% ( 6600, 9900 ] 74 0.007% 100.199% ( 9900, 14000 ] 7 0.001% 100.200% ( 33000, 50000 ] 2 0.000% 100.200% ** Level 5 read latency histogram (micros): Count: 94306 Average: 651.1105 StdDev: 1017.07 Min: 80 Median: 258.0244 Max: 44478 Percentiles: P50: 258.02 P75: 682.84 P99: 5421.82 P99.9: 8858.03 P99.99: 12593.45 ------------------------------------------------------ ( 76, 110 ] 3490 3.701% 3.701% # ( 110, 170 ] 24485 25.963% 29.664% ##### ( 170, 250 ] 18520 19.638% 49.302% #### ( 250, 380 ] 10660 11.304% 60.606% ## ( 380, 580 ] 10450 11.081% 71.687% ## ( 580, 870 ] 8811 9.343% 81.030% ## ( 870, 1300 ] 6517 6.910% 87.940% # ( 1300, 1900 ] 3913 4.149% 92.090% # ( 1900, 2900 ] 3030 3.213% 95.303% # ( 2900, 4400 ] 2898 3.073% 98.376% # ( 4400, 6600 ] 1268 1.345% 99.720% ( 6600, 9900 ] 248 0.263% 99.983% ( 9900, 14000 ] 10 0.011% 99.994% ( 14000, 22000 ] 3 0.003% 99.997% ( 22000, 33000 ] 2 0.002% 99.999% ( 33000, 50000 ] 2 0.002% 100.001% ** DB Stats ** Uptime(secs): 22.0 total, 3.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 178.90 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.166 H:M:S, 32.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 290.99 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/17-15:51:33 ... thread 0: (1000000,5000000) ops and (288272.7,197865.7) ops/second in (3.468938,25.269671) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/4 3.63 GB 346.1 4.1 0.0 4.1 8.9 4.7 0.0 1.9 96.6 207.4 43.71 29.93 83 0.527 4285K 4476 0.0 0.0 L1 13/0 1.27 GB 129.9 2.3 1.7 0.7 2.3 1.7 0.0 1.4 127.4 126.8 18.84 11.74 2 9.422 2438K 11K 0.0 0.0 L2 1/0 57.70 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 926.19 MB 0.9 4.6 2.6 2.1 4.6 2.5 0.2 1.8 115.2 114.4 41.25 22.44 22 1.875 3121K 4649 0.0 0.0 L4 169/0 9.70 GB 1.0 8.8 5.0 3.8 8.7 4.9 0.0 1.7 115.7 114.3 77.52 42.49 53 1.463 4542K 4804 0.0 0.0 L5 1144/0 94.06 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1362/4 109.61 GB 0.0 19.9 9.2 10.6 24.4 13.8 6.1 5.2 112.2 138.1 181.32 106.60 160 1.133 14M 25K 0.0 0.0 Int 0/0 0.00 KB 0.0 19.9 9.2 10.6 24.4 13.8 6.1 5.2 112.2 138.1 181.32 106.60 160 1.133 14M 25K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 19.9 9.2 10.6 19.7 9.1 0.0 0.0 129.1 128.1 157.56 93.23 92 1.713 14M 25K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 204.1 23.76 13.37 68 0.349 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 25.5 total, 25.3 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 24.45 GB write, 982.75 MB/s write, 19.87 GB read, 798.59 MB/s read, 181.3 seconds Interval compaction: 24.45 GB write, 990.63 MB/s write, 19.87 GB read, 804.99 MB/s read, 181.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 63, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 63, interval: 64 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1540305 Average: 9.1349 StdDev: 106.47 Min: 1 Median: 1.7090 Max: 25095 Percentiles: P50: 1.71 P75: 2.47 P99: 144.49 P99.9: 1090.37 P99.99: 2290.00 ------------------------------------------------------ [ 0, 1 ] 165641 10.754% 10.754% ## ( 1, 2 ] 852570 55.351% 66.105% ########### ( 2, 3 ] 291708 18.938% 85.043% #### ( 3, 4 ] 92419 6.000% 91.043% # ( 4, 6 ] 93967 6.101% 97.143% # ( 6, 10 ] 20517 1.332% 98.475% ( 10, 15 ] 3843 0.249% 98.725% ( 15, 22 ] 561 0.036% 98.761% ( 22, 34 ] 33 0.002% 98.763% ( 34, 51 ] 34 0.002% 98.766% ( 51, 76 ] 50 0.003% 98.769% ( 76, 110 ] 598 0.039% 98.808% ( 110, 170 ] 5151 0.334% 99.142% ( 170, 250 ] 4889 0.317% 99.460% ( 250, 380 ] 2531 0.164% 99.624% ( 380, 580 ] 2174 0.141% 99.765% ( 580, 870 ] 1558 0.101% 99.866% ( 870, 1300 ] 1016 0.066% 99.932% ( 1300, 1900 ] 687 0.045% 99.977% ( 1900, 2900 ] 523 0.034% 100.011% ( 2900, 4400 ] 399 0.026% 100.037% ( 4400, 6600 ] 123 0.008% 100.045% ( 6600, 9900 ] 16 0.001% 100.046% ( 9900, 14000 ] 2 0.000% 100.046% ( 22000, 33000 ] 1 0.000% 100.046% ** Level 1 read latency histogram (micros): Count: 363873 Average: 12.2880 StdDev: 164.12 Min: 1 Median: 2.1592 Max: 45629 Percentiles: P50: 2.16 P75: 3.01 P99: 117.79 P99.9: 1873.71 P99.99: 3475.10 ------------------------------------------------------ [ 0, 1 ] 4013 1.103% 1.103% ( 1, 2 ] 160748 44.177% 45.280% ######### ( 2, 3 ] 107868 29.644% 74.924% ###### ( 3, 4 ] 34899 9.591% 84.515% ## ( 4, 6 ] 39624 10.890% 95.405% ## ( 6, 10 ] 10240 2.814% 98.219% # ( 10, 15 ] 2024 0.556% 98.775% ( 15, 22 ] 430 0.118% 98.893% ( 22, 34 ] 41 0.011% 98.905% ( 34, 51 ] 23 0.006% 98.911% ( 51, 76 ] 40 0.011% 98.922% ( 76, 110 ] 154 0.042% 98.964% ( 110, 170 ] 1004 0.276% 99.240% ( 170, 250 ] 682 0.187% 99.428% ( 250, 380 ] 313 0.086% 99.514% ( 380, 580 ] 413 0.114% 99.627% ( 580, 870 ] 387 0.106% 99.733% ( 870, 1300 ] 347 0.095% 99.829% ( 1300, 1900 ] 271 0.074% 99.903% ( 1900, 2900 ] 242 0.067% 99.970% ( 2900, 4400 ] 192 0.053% 100.023% ( 4400, 6600 ] 52 0.014% 100.037% ( 6600, 9900 ] 15 0.004% 100.041% ( 9900, 14000 ] 2 0.001% 100.041% ( 33000, 50000 ] 1 0.000% 100.042% ** Level 2 read latency histogram (micros): Count: 351166 Average: 18.2856 StdDev: 219.21 Min: 1 Median: 2.1052 Max: 12601 Percentiles: P50: 2.11 P75: 3.06 P99: 13.92 P99.9: 2561.65 P99.99: 3363.68 ------------------------------------------------------ [ 0, 1 ] 5833 1.661% 1.661% ( 1, 2 ] 159667 45.468% 47.129% ######### ( 2, 3 ] 95883 27.304% 74.433% ##### ( 3, 4 ] 34681 9.876% 84.309% ## ( 4, 6 ] 38932 11.086% 95.395% ## ( 6, 10 ] 10512 2.993% 98.389% # ( 10, 15 ] 2737 0.779% 99.168% ( 15, 22 ] 510 0.145% 99.313% ( 22, 34 ] 45 0.013% 99.326% ( 34, 51 ] 36 0.010% 99.336% ( 51, 76 ] 20 0.006% 99.342% ( 76, 110 ] 21 0.006% 99.348% ( 110, 170 ] 115 0.033% 99.381% ( 170, 250 ] 80 0.023% 99.404% ( 250, 380 ] 87 0.025% 99.428% ( 380, 580 ] 192 0.055% 99.483% ( 580, 870 ] 252 0.072% 99.555% ( 870, 1300 ] 362 0.103% 99.658% ( 1300, 1900 ] 476 0.136% 99.794% ( 1900, 2900 ] 565 0.161% 99.954% ( 2900, 4400 ] 404 0.115% 100.069% ( 4400, 6600 ] 151 0.043% 100.112% ( 6600, 9900 ] 33 0.009% 100.122% ( 9900, 14000 ] 9 0.003% 100.124% ** Level 3 read latency histogram (micros): Count: 1755209 Average: 12.3894 StdDev: 160.15 Min: 1 Median: 2.1408 Max: 25058 Percentiles: P50: 2.14 P75: 2.97 P99: 9.96 P99.9: 1021.80 P99.99: 1457.93 ------------------------------------------------------ [ 0, 1 ] 16478 0.939% 0.939% ( 1, 2 ] 786920 44.833% 45.772% ######### ( 2, 3 ] 527134 30.033% 75.805% ###### ( 3, 4 ] 172520 9.829% 85.634% ## ( 4, 6 ] 189325 10.786% 96.420% ## ( 6, 10 ] 45744 2.606% 99.026% # ( 10, 15 ] 8987 0.512% 99.538% ( 15, 22 ] 1340 0.076% 99.615% ( 22, 34 ] 118 0.007% 99.622% ( 34, 51 ] 100 0.006% 99.627% ( 51, 76 ] 110 0.006% 99.633% ( 76, 110 ] 123 0.007% 99.640% ( 110, 170 ] 645 0.037% 99.677% ( 170, 250 ] 583 0.033% 99.710% ( 250, 380 ] 566 0.032% 99.743% ( 380, 580 ] 834 0.048% 99.790% ( 580, 870 ] 1309 0.075% 99.865% ( 870, 1300 ] 1750 0.100% 99.965% ( 1300, 1900 ] 1700 0.097% 100.061% ( 1900, 2900 ] 1632 0.093% 100.154% ( 2900, 4400 ] 1129 0.064% 100.219% ( 4400, 6600 ] 354 0.020% 100.239% ( 6600, 9900 ] 54 0.003% 100.242% ( 9900, 14000 ] 5 0.000% 100.242% ( 14000, 22000 ] 2 0.000% 100.242% ( 22000, 33000 ] 1 0.000% 100.242% ** Level 4 read latency histogram (micros): Count: 998982 Average: 25.4251 StdDev: 229.49 Min: 1 Median: 2.1825 Max: 44393 Percentiles: P50: 2.18 P75: 3.14 P99: 453.95 P99.9: 2069.82 P99.99: 2635.28 ------------------------------------------------------ [ 0, 1 ] 9120 0.913% 0.913% ( 1, 2 ] 437751 43.820% 44.733% ######### ( 2, 3 ] 288255 28.855% 73.588% ###### ( 3, 4 ] 98372 9.847% 83.435% ## ( 4, 6 ] 105345 10.545% 93.980% ## ( 6, 10 ] 26498 2.653% 96.632% # ( 10, 15 ] 4859 0.486% 97.119% ( 15, 22 ] 603 0.060% 97.179% ( 22, 34 ] 54 0.005% 97.185% ( 34, 51 ] 73 0.007% 97.192% ( 51, 76 ] 104 0.010% 97.202% ( 76, 110 ] 1189 0.119% 97.321% ( 110, 170 ] 7611 0.762% 98.083% ( 170, 250 ] 5250 0.526% 98.609% ( 250, 380 ] 2793 0.280% 98.888% ( 380, 580 ] 3016 0.302% 99.190% ( 580, 870 ] 2637 0.264% 99.454% ( 870, 1300 ] 2386 0.239% 99.693% ( 1300, 1900 ] 1797 0.180% 99.873% ( 1900, 2900 ] 1590 0.159% 100.032% ( 2900, 4400 ] 1173 0.117% 100.150% ( 4400, 6600 ] 403 0.040% 100.190% ( 6600, 9900 ] 74 0.007% 100.197% ( 9900, 14000 ] 7 0.001% 100.198% ( 33000, 50000 ] 2 0.000% 100.198% ** Level 5 read latency histogram (micros): Count: 136688 Average: 512.8222 StdDev: 873.05 Min: 80 Median: 214.9018 Max: 44478 Percentiles: P50: 214.90 P75: 483.85 P99: 4686.49 P99.9: 8294.07 P99.99: 10855.79 ------------------------------------------------------ ( 76, 110 ] 6634 4.853% 4.853% # ( 110, 170 ] 44965 32.896% 37.749% ####### ( 170, 250 ] 29834 21.826% 59.576% #### ( 250, 380 ] 14183 10.376% 69.952% ## ( 380, 580 ] 13289 9.722% 79.674% ## ( 580, 870 ] 9730 7.118% 86.793% # ( 870, 1300 ] 6642 4.859% 91.652% # ( 1300, 1900 ] 3934 2.878% 94.530% # ( 1900, 2900 ] 3041 2.225% 96.755% ( 2900, 4400 ] 2904 2.125% 98.879% ( 4400, 6600 ] 1268 0.928% 99.807% ( 6600, 9900 ] 248 0.181% 99.988% ( 9900, 14000 ] 10 0.007% 99.996% ( 14000, 22000 ] 3 0.002% 99.998% ( 22000, 33000 ] 2 0.001% 99.999% ( 33000, 50000 ] 2 0.001% 100.001% ** DB Stats ** Uptime(secs): 25.5 total, 3.5 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 193.17 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.266 H:M:S, 28.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 283.71 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.100 H:M:S, 2.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 277.988 micros/op 14371 ops/sec 60.189 seconds 864996 operations; 13.9 MB/s (221999 of 221999 found) Microseconds per read: Count: 864996 Average: 277.9901 StdDev: 905.71 Min: 2 Median: 195.3852 Max: 75398 Percentiles: P50: 195.39 P75: 239.90 P99: 1690.23 P99.9: 16647.36 P99.99: 33125.10 ------------------------------------------------------ ( 1, 2 ] 2 0.000% 0.000% ( 2, 3 ] 37 0.004% 0.005% ( 3, 4 ] 57 0.007% 0.011% ( 4, 6 ] 85 0.010% 0.021% ( 6, 10 ] 294 0.034% 0.055% ( 10, 15 ] 4122 0.477% 0.531% ( 15, 22 ] 4807 0.556% 1.087% ( 22, 34 ] 7974 0.922% 2.009% ( 34, 51 ] 2662 0.308% 2.317% ( 51, 76 ] 332 0.038% 2.355% ( 76, 110 ] 3742 0.433% 2.788% ( 110, 170 ] 285053 32.954% 35.742% ####### ( 170, 250 ] 388670 44.933% 80.675% ######### ( 250, 380 ] 89242 10.317% 90.992% ## ( 380, 580 ] 41036 4.744% 95.736% # ( 580, 870 ] 16782 1.940% 97.676% ( 870, 1300 ] 8614 0.996% 98.672% ( 1300, 1900 ] 4359 0.504% 99.176% ( 1900, 2900 ] 2916 0.337% 99.513% ( 2900, 4400 ] 2107 0.244% 99.757% ( 4400, 6600 ] 688 0.080% 99.836% ( 6600, 9900 ] 152 0.018% 99.854% ( 9900, 14000 ] 170 0.020% 99.874% ( 14000, 22000 ] 689 0.080% 99.953% ( 22000, 33000 ] 317 0.037% 99.990% ( 33000, 50000 ] 68 0.008% 99.998% ( 50000, 75000 ] 18 0.002% 100.000% ( 75000, 110000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 15943828 rocksdb.bloom.filter.full.positive COUNT : 1020887 rocksdb.bloom.filter.full.true.positive COUNT : 864632 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 364 rocksdb.memtable.miss COUNT : 864632 rocksdb.l0.hit COUNT : 31064 rocksdb.l1.hit COUNT : 10244 rocksdb.l2andup.hit COUNT : 823324 rocksdb.compaction.key.drop.new COUNT : 201822 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5208358 rocksdb.number.keys.read COUNT : 864996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5375025456 rocksdb.bytes.read COUNT : 864996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1670 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 40313027 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 23449154871 rocksdb.compact.write.bytes COUNT : 23244414076 rocksdb.flush.write.bytes COUNT : 5350610686 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 26 rocksdb.number.superversion_acquires COUNT : 1045 rocksdb.number.superversion_releases COUNT : 1018 rocksdb.number.superversion_cleanups COUNT : 265 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 307 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5307529216 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2121728 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 27523139915 rocksdb.non.last.level.read.count COUNT : 6509048 rocksdb.block.checksum.compute.count COUNT : 6713321 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 194.386863 P95 : 548.198654 P99 : 1688.483819 P100 : 75394.000000 COUNT : 864996 SUM : 239741423 rocksdb.db.write.micros P50 : 1.550340 P95 : 4.614683 P99 : 9.719648 P100 : 886738.000000 COUNT : 5208358 SUM : 53596506 rocksdb.compaction.times.micros P50 : 1462500.000000 P95 : 3372500.000000 P99 : 8265500.000000 P100 : 9708488.000000 COUNT : 93 SUM : 158573796 rocksdb.compaction.times.cpu_micros P50 : 877586.206897 P95 : 2237500.000000 P99 : 6506873.000000 P100 : 6506873.000000 COUNT : 93 SUM : 94206825 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 931.428571 P95 : 36400.000000 P99 : 192400.000000 P100 : 211249.000000 COUNT : 72 SUM : 617863 rocksdb.compaction.outfile.sync.micros P50 : 13717.241379 P95 : 159714.285714 P99 : 317600.000000 P100 : 924630.000000 COUNT : 296 SUM : 12757286 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 673.214286 P95 : 4293.421053 P99 : 6814.500000 P100 : 7918.000000 COUNT : 187 SUM : 237169 rocksdb.table.open.io.micros P50 : 11882.066277 P95 : 21499.036609 P99 : 29860.416667 P100 : 35718.000000 COUNT : 1670 SUM : 18968166 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 4.390126 P95 : 9.957900 P99 : 194.499854 P100 : 38028.000000 COUNT : 5679865 SUM : 92936214 rocksdb.read.block.get.micros P50 : 150.276695 P95 : 422.308661 P99 : 1623.687028 P100 : 45637.000000 COUNT : 1031794 SUM : 205526387 rocksdb.write.raw.block.micros P50 : 0.534974 P95 : 1.341434 P99 : 3.641740 P100 : 50448.000000 COUNT : 6926159 SUM : 23237178 rocksdb.numfiles.in.singlecompaction P50 : 2.803030 P95 : 6.200000 P99 : 9.880000 P100 : 29.000000 COUNT : 115 SUM : 409 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.580828 P95 : 2105.064267 P99 : 2742.575835 P100 : 886551.000000 COUNT : 154995 SUM : 40331044 rocksdb.sst.read.micros P50 : 2.327205 P95 : 182.556657 P99 : 508.158128 P100 : 45629.000000 COUNT : 6509102 SUM : 282054786 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 864996 SUM : 864996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5208358 SUM : 5375025456 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 228775.510204 P95 : 955200.000000 P99 : 1121641.000000 P100 : 1121641.000000 COUNT : 72 SUM : 24648871 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 310 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0