[2024-11-07 00:24:22.918393] Starting SPDK v25.01-pre git sha1 d1c46ed8e / DPDK 24.11.0-rc1 initialization... [2024-11-07 00:24:22.918665] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid4159016 ] Set seed to 1730935462911606 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-07 00:24:23.090675] pci_dpdk.c: 37:dpdk_pci_init: *NOTICE*: In-development DPDK 24.11.0-rc1 is used. There is no support for it in SPDK. Enabled only for validation. [2024-11-07 00:24:23.139423] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-07 00:24:23.162779] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-07 00:24:23.162817] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 4159016' to capture a snapshot of events at runtime. [2024-11-07 00:24:23.162829] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-07 00:24:23.162837] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-07 00:24:23.162847] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid4159016 for offline analysis/debug. [2024-11-07 00:24:23.163405] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Thu Nov 7 00:24:27 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/07-00:24:35 ... thread 0: (1000000,1000000) ops and (126409.0,126409.0) ops/second in (7.910829,7.910829) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 12/10 1.53 GB 50.3 0.4 0.0 0.4 1.2 0.8 0.0 1.5 38.4 121.5 9.85 3.86 10 0.985 383K 534 0.0 0.0 L1 3/3 368.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 3/1 317.08 MB 1.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 59/9 3.39 GB 2.9 4.7 3.2 1.5 4.7 3.2 0.0 1.5 101.3 100.7 47.51 22.05 21 2.262 1678K 1451 0.0 0.0 L4 171/11 10.92 GB 1.1 1.8 1.2 0.5 1.7 1.2 0.1 1.4 129.1 127.3 14.05 8.60 14 1.004 1167K 0 0.0 0.0 L5 1096/0 90.00 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1344/34 106.51 GB 0.0 6.8 4.5 2.4 7.6 5.2 0.1 9.5 98.1 108.8 71.41 34.50 45 1.587 3229K 1985 0.0 0.0 Int 0/0 0.00 KB 0.0 6.8 4.5 2.4 7.6 5.2 0.1 9.5 98.1 108.8 71.41 34.50 45 1.587 3229K 1985 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.8 4.5 2.4 6.8 4.4 0.0 0.0 108.9 108.1 64.30 32.15 36 1.786 3229K 1985 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 115.1 7.11 2.36 9 0.790 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 8.1 total, 7.9 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.59 GB write, 960.94 MB/s write, 6.84 GB read, 866.19 MB/s read, 71.4 seconds Interval compaction: 7.59 GB write, 981.98 MB/s write, 6.84 GB read, 885.15 MB/s read, 71.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 29, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 29, interval: 29 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 235705 Average: 20.7344 StdDev: 268.94 Min: 1 Median: 1.8389 Max: 55914 Percentiles: P50: 1.84 P75: 2.82 P99: 568.05 P99.9: 3225.77 P99.99: 5122.90 ------------------------------------------------------ [ 0, 1 ] 10362 4.396% 4.396% # ( 1, 2 ] 128139 54.364% 58.760% ########### ( 2, 3 ] 46845 19.874% 78.635% #### ( 3, 4 ] 21062 8.936% 87.570% ## ( 4, 6 ] 20790 8.820% 96.391% ## ( 6, 10 ] 4497 1.908% 98.299% ( 10, 15 ] 828 0.351% 98.650% ( 15, 22 ] 87 0.037% 98.687% ( 22, 34 ] 18 0.008% 98.695% ( 34, 51 ] 6 0.003% 98.697% ( 51, 76 ] 3 0.001% 98.698% ( 76, 110 ] 9 0.004% 98.702% ( 110, 170 ] 25 0.011% 98.713% ( 170, 250 ] 74 0.031% 98.744% ( 250, 380 ] 193 0.082% 98.826% ( 380, 580 ] 436 0.185% 99.011% ( 580, 870 ] 655 0.278% 99.289% ( 870, 1300 ] 640 0.272% 99.560% ( 1300, 1900 ] 439 0.186% 99.747% ( 1900, 2900 ] 307 0.130% 99.877% ( 2900, 4400 ] 250 0.106% 99.983% ( 4400, 6600 ] 50 0.021% 100.004% ( 6600, 9900 ] 8 0.003% 100.008% ( 14000, 22000 ] 1 0.000% 100.008% ( 22000, 33000 ] 1 0.000% 100.008% ( 33000, 50000 ] 2 0.001% 100.009% ( 50000, 75000 ] 1 0.000% 100.010% ** Level 1 read latency histogram (micros): Count: 93705 Average: 9.6327 StdDev: 116.02 Min: 1 Median: 1.9735 Max: 6754 Percentiles: P50: 1.97 P75: 2.96 P99: 10.98 P99.9: 1810.79 P99.99: 4379.42 ------------------------------------------------------ [ 0, 1 ] 1507 1.608% 1.608% ( 1, 2 ] 46580 49.709% 51.317% ########## ( 2, 3 ] 23077 24.627% 75.945% ##### ( 3, 4 ] 9062 9.671% 85.615% ## ( 4, 6 ] 9821 10.481% 96.096% ## ( 6, 10 ] 2633 2.810% 98.906% # ( 10, 15 ] 447 0.477% 99.383% ( 15, 22 ] 56 0.060% 99.443% ( 22, 34 ] 10 0.011% 99.454% ( 34, 51 ] 4 0.004% 99.458% ( 51, 76 ] 5 0.005% 99.463% ( 76, 110 ] 1 0.001% 99.464% ( 110, 170 ] 8 0.009% 99.473% ( 170, 250 ] 11 0.012% 99.485% ( 250, 380 ] 30 0.032% 99.517% ( 380, 580 ] 74 0.079% 99.596% ( 580, 870 ] 96 0.102% 99.698% ( 870, 1300 ] 128 0.137% 99.835% ( 1300, 1900 ] 72 0.077% 99.911% ( 1900, 2900 ] 47 0.050% 99.962% ( 2900, 4400 ] 27 0.029% 99.990% ( 4400, 6600 ] 7 0.007% 99.998% ( 6600, 9900 ] 2 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 724523 Average: 12.2512 StdDev: 142.50 Min: 1 Median: 1.9309 Max: 10236 Percentiles: P50: 1.93 P75: 2.94 P99: 11.15 P99.9: 1194.60 P99.99: 1587.49 ------------------------------------------------------ [ 0, 1 ] 17336 2.393% 2.393% ( 1, 2 ] 370537 51.142% 53.535% ########## ( 2, 3 ] 166010 22.913% 76.448% ##### ( 3, 4 ] 71774 9.906% 86.354% ## ( 4, 6 ] 71272 9.837% 96.191% ## ( 6, 10 ] 19501 2.692% 98.883% # ( 10, 15 ] 3674 0.507% 99.390% ( 15, 22 ] 576 0.080% 99.470% ( 22, 34 ] 79 0.011% 99.480% ( 34, 51 ] 42 0.006% 99.486% ( 51, 76 ] 39 0.005% 99.492% ( 76, 110 ] 26 0.004% 99.495% ( 110, 170 ] 93 0.013% 99.508% ( 170, 250 ] 165 0.023% 99.531% ( 250, 380 ] 307 0.042% 99.573% ( 380, 580 ] 627 0.087% 99.660% ( 580, 870 ] 932 0.129% 99.788% ( 870, 1300 ] 1071 0.148% 99.936% ( 1300, 1900 ] 813 0.112% 100.048% ( 1900, 2900 ] 597 0.082% 100.131% ( 2900, 4400 ] 363 0.050% 100.181% ( 4400, 6600 ] 106 0.015% 100.196% ( 6600, 9900 ] 4 0.001% 100.196% ( 9900, 14000 ] 1 0.000% 100.196% ** Level 3 read latency histogram (micros): Count: 705733 Average: 10.3819 StdDev: 151.15 Min: 1 Median: 1.9181 Max: 30987 Percentiles: P50: 1.92 P75: 2.90 P99: 9.56 P99.9: 996.71 P99.99: 1384.42 ------------------------------------------------------ [ 0, 1 ] 13165 1.865% 1.865% ( 1, 2 ] 370015 52.430% 54.295% ########## ( 2, 3 ] 161898 22.940% 77.236% ##### ( 3, 4 ] 70879 10.043% 87.279% ## ( 4, 6 ] 69360 9.828% 97.107% ## ( 6, 10 ] 15002 2.126% 99.233% ( 10, 15 ] 2474 0.351% 99.583% ( 15, 22 ] 267 0.038% 99.621% ( 22, 34 ] 36 0.005% 99.626% ( 34, 51 ] 26 0.004% 99.630% ( 51, 76 ] 40 0.006% 99.636% ( 76, 110 ] 9 0.001% 99.637% ( 110, 170 ] 59 0.008% 99.645% ( 170, 250 ] 137 0.019% 99.665% ( 250, 380 ] 206 0.029% 99.694% ( 380, 580 ] 481 0.068% 99.762% ( 580, 870 ] 741 0.105% 99.867% ( 870, 1300 ] 795 0.113% 99.980% ( 1300, 1900 ] 529 0.075% 100.055% ( 1900, 2900 ] 416 0.059% 100.114% ( 2900, 4400 ] 326 0.046% 100.160% ( 4400, 6600 ] 76 0.011% 100.171% ( 6600, 9900 ] 7 0.001% 100.172% ( 9900, 14000 ] 2 0.000% 100.172% ( 14000, 22000 ] 4 0.001% 100.172% ( 22000, 33000 ] 4 0.001% 100.173% ** Level 4 read latency histogram (micros): Count: 176395 Average: 35.0398 StdDev: 382.42 Min: 1 Median: 1.9074 Max: 51083 Percentiles: P50: 1.91 P75: 2.95 P99: 990.03 P99.9: 3274.33 P99.99: 4011.58 ------------------------------------------------------ [ 0, 1 ] 4050 2.296% 2.296% ( 1, 2 ] 92735 52.572% 54.868% ########### ( 2, 3 ] 37308 21.150% 76.019% #### ( 3, 4 ] 17003 9.639% 85.658% ## ( 4, 6 ] 16766 9.505% 95.163% ## ( 6, 10 ] 4284 2.429% 97.591% ( 10, 15 ] 629 0.357% 97.948% ( 15, 22 ] 71 0.040% 97.988% ( 22, 34 ] 4 0.002% 97.990% ( 34, 51 ] 9 0.005% 97.995% ( 51, 76 ] 11 0.006% 98.002% ( 76, 110 ] 6 0.003% 98.005% ( 110, 170 ] 37 0.021% 98.026% ( 170, 250 ] 87 0.049% 98.075% ( 250, 380 ] 191 0.108% 98.184% ( 380, 580 ] 481 0.273% 98.456% ( 580, 870 ] 743 0.421% 98.878% ( 870, 1300 ] 774 0.439% 99.316% ( 1300, 1900 ] 537 0.304% 99.621% ( 1900, 2900 ] 412 0.234% 99.854% ( 2900, 4400 ] 323 0.183% 100.037% ( 4400, 6600 ] 105 0.060% 100.097% ( 6600, 9900 ] 14 0.008% 100.105% ( 9900, 14000 ] 5 0.003% 100.108% ( 14000, 22000 ] 6 0.003% 100.111% ( 22000, 33000 ] 4 0.002% 100.113% ( 33000, 50000 ] 3 0.002% 100.115% ( 50000, 75000 ] 1 0.001% 100.116% ** Level 5 read latency histogram (micros): Count: 14232 Average: 1603.6896 StdDev: 1730.49 Min: 84 Median: 1112.2642 Max: 57211 Percentiles: P50: 1112.26 P75: 2148.80 P99: 6384.72 P99.9: 9889.06 P99.99: 48972.23 ------------------------------------------------------ ( 76, 110 ] 14 0.098% 0.098% ( 110, 170 ] 98 0.689% 0.787% ( 170, 250 ] 272 1.911% 2.698% ( 250, 380 ] 784 5.509% 8.207% # ( 380, 580 ] 1711 12.022% 20.229% ## ( 580, 870 ] 2642 18.564% 38.793% #### ( 870, 1300 ] 2831 19.892% 58.685% #### ( 1300, 1900 ] 1907 13.399% 72.084% ### ( 1900, 2900 ] 1668 11.720% 83.804% ## ( 2900, 4400 ] 1625 11.418% 95.222% ## ( 4400, 6600 ] 596 4.188% 99.410% # ( 6600, 9900 ] 70 0.492% 99.902% ( 9900, 14000 ] 3 0.021% 99.923% ( 14000, 22000 ] 1 0.007% 99.930% ( 22000, 33000 ] 2 0.014% 99.944% ( 33000, 50000 ] 7 0.049% 99.993% ( 50000, 75000 ] 1 0.007% 100.000% ** DB Stats ** Uptime(secs): 8.1 total, 7.9 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 121.72 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:4.096 H:M:S, 50.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 124.37 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.096 H:M:S, 51.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/07-00:24:41 ... thread 0: (1000000,2000000) ops and (162593.7,142236.1) ops/second in (6.150298,14.061127) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 10/4 1.78 GB 157.3 1.4 0.0 1.4 3.2 1.8 0.0 1.8 65.0 147.0 22.27 10.95 26 0.857 1470K 2027 0.0 0.0 L1 18/0 1.09 GB 111.6 1.1 0.7 0.4 1.1 0.7 0.0 1.5 81.6 81.2 13.75 5.29 1 13.745 1138K 4875 0.0 0.0 L3 19/0 966.26 MB 1.0 5.1 3.5 1.6 5.1 3.5 0.0 1.5 102.9 102.4 51.19 24.19 24 2.133 2142K 2736 0.0 0.0 L4 168/0 9.63 GB 1.0 6.7 3.7 3.0 6.6 3.6 0.4 1.8 124.5 122.1 55.35 31.96 53 1.044 3431K 0 0.0 0.0 L5 1144/0 93.94 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1359/4 107.38 GB 0.0 14.4 8.0 6.4 16.0 9.6 4.3 9.0 103.3 115.0 142.56 72.38 104 1.371 8183K 9638 0.0 0.0 Int 0/0 0.00 KB 0.0 14.4 8.0 6.4 16.0 9.6 4.3 9.0 103.3 115.0 142.56 72.38 104 1.371 8183K 9638 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 14.4 8.0 6.4 14.2 7.8 0.0 0.0 114.0 112.7 129.23 67.25 82 1.576 8183K 9638 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 137.0 13.33 5.13 22 0.606 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.2 total, 14.1 interval Flush(GB): cumulative 1.784, interval 1.784 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 16.01 GB write, 1151.25 MB/s write, 14.38 GB read, 1034.68 MB/s read, 142.6 seconds Interval compaction: 16.01 GB write, 1165.43 MB/s write, 14.38 GB read, 1047.43 MB/s read, 142.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 60, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 60, interval: 60 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 599955 Average: 20.1384 StdDev: 249.62 Min: 1 Median: 1.7764 Max: 55914 Percentiles: P50: 1.78 P75: 2.66 P99: 561.57 P99.9: 2785.13 P99.99: 4240.32 ------------------------------------------------------ [ 0, 1 ] 33747 5.625% 5.625% # ( 1, 2 ] 342908 57.156% 62.781% ########### ( 2, 3 ] 111495 18.584% 81.364% #### ( 3, 4 ] 46681 7.781% 89.145% ## ( 4, 6 ] 44649 7.442% 96.587% # ( 6, 10 ] 10098 1.683% 98.270% ( 10, 15 ] 1464 0.244% 98.514% ( 15, 22 ] 159 0.027% 98.541% ( 22, 34 ] 28 0.005% 98.546% ( 34, 51 ] 13 0.002% 98.548% ( 51, 76 ] 10 0.002% 98.549% ( 76, 110 ] 58 0.010% 98.559% ( 110, 170 ] 230 0.038% 98.597% ( 170, 250 ] 445 0.074% 98.672% ( 250, 380 ] 754 0.126% 98.797% ( 380, 580 ] 1341 0.224% 99.021% ( 580, 870 ] 1888 0.315% 99.335% ( 870, 1300 ] 1785 0.298% 99.633% ( 1300, 1900 ] 1064 0.177% 99.810% ( 1900, 2900 ] 609 0.102% 99.912% ( 2900, 4400 ] 526 0.088% 99.999% ( 4400, 6600 ] 128 0.021% 100.021% ( 6600, 9900 ] 21 0.004% 100.024% ( 9900, 14000 ] 1 0.000% 100.025% ( 14000, 22000 ] 6 0.001% 100.026% ( 22000, 33000 ] 4 0.001% 100.026% ( 33000, 50000 ] 3 0.001% 100.027% ( 50000, 75000 ] 2 0.000% 100.027% ** Level 1 read latency histogram (micros): Count: 93922 Average: 10.9852 StdDev: 124.70 Min: 1 Median: 1.9753 Max: 6754 Percentiles: P50: 1.98 P75: 2.97 P99: 12.84 P99.9: 1939.21 P99.99: 4382.70 ------------------------------------------------------ [ 0, 1 ] 1519 1.617% 1.617% ( 1, 2 ] 46592 49.607% 51.224% ########## ( 2, 3 ] 23091 24.585% 75.810% ##### ( 3, 4 ] 9064 9.651% 85.460% ## ( 4, 6 ] 9828 10.464% 95.924% ## ( 6, 10 ] 2634 2.804% 98.729% # ( 10, 15 ] 448 0.477% 99.206% ( 15, 22 ] 57 0.061% 99.266% ( 22, 34 ] 10 0.011% 99.277% ( 34, 51 ] 4 0.004% 99.281% ( 51, 76 ] 5 0.005% 99.287% ( 76, 110 ] 3 0.003% 99.290% ( 110, 170 ] 32 0.034% 99.324% ( 170, 250 ] 25 0.027% 99.351% ( 250, 380 ] 55 0.059% 99.409% ( 380, 580 ] 97 0.103% 99.512% ( 580, 870 ] 132 0.141% 99.653% ( 870, 1300 ] 146 0.155% 99.808% ( 1300, 1900 ] 84 0.089% 99.898% ( 1900, 2900 ] 53 0.056% 99.954% ( 2900, 4400 ] 34 0.036% 99.990% ( 4400, 6600 ] 7 0.007% 99.998% ( 6600, 9900 ] 2 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 784584 Average: 12.2311 StdDev: 144.93 Min: 1 Median: 1.9310 Max: 10236 Percentiles: P50: 1.93 P75: 2.93 P99: 11.09 P99.9: 1226.60 P99.99: 1661.09 ------------------------------------------------------ [ 0, 1 ] 18142 2.312% 2.312% ( 1, 2 ] 401899 51.224% 53.537% ########## ( 2, 3 ] 180594 23.018% 76.555% ##### ( 3, 4 ] 77071 9.823% 86.378% ## ( 4, 6 ] 77064 9.822% 96.200% ## ( 6, 10 ] 21118 2.692% 98.892% # ( 10, 15 ] 3895 0.496% 99.388% ( 15, 22 ] 606 0.077% 99.465% ( 22, 34 ] 81 0.010% 99.476% ( 34, 51 ] 55 0.007% 99.483% ( 51, 76 ] 48 0.006% 99.489% ( 76, 110 ] 29 0.004% 99.492% ( 110, 170 ] 96 0.012% 99.505% ( 170, 250 ] 170 0.022% 99.526% ( 250, 380 ] 328 0.042% 99.568% ( 380, 580 ] 670 0.085% 99.654% ( 580, 870 ] 988 0.126% 99.780% ( 870, 1300 ] 1140 0.145% 99.925% ( 1300, 1900 ] 850 0.108% 100.033% ( 1900, 2900 ] 628 0.080% 100.113% ( 2900, 4400 ] 406 0.052% 100.165% ( 4400, 6600 ] 114 0.015% 100.179% ( 6600, 9900 ] 12 0.002% 100.181% ( 9900, 14000 ] 2 0.000% 100.181% ** Level 3 read latency histogram (micros): Count: 1324011 Average: 10.3113 StdDev: 151.68 Min: 1 Median: 1.8961 Max: 30987 Percentiles: P50: 1.90 P75: 2.87 P99: 9.56 P99.9: 1008.53 P99.99: 1392.55 ------------------------------------------------------ [ 0, 1 ] 27456 2.074% 2.074% ( 1, 2 ] 708156 53.486% 55.559% ########### ( 2, 3 ] 295325 22.305% 77.865% #### ( 3, 4 ] 126610 9.563% 87.427% ## ( 4, 6 ] 126179 9.530% 96.957% ## ( 6, 10 ] 30351 2.292% 99.250% ( 10, 15 ] 4226 0.319% 99.569% ( 15, 22 ] 496 0.037% 99.606% ( 22, 34 ] 51 0.004% 99.610% ( 34, 51 ] 60 0.005% 99.615% ( 51, 76 ] 93 0.007% 99.622% ( 76, 110 ] 28 0.002% 99.624% ( 110, 170 ] 90 0.007% 99.631% ( 170, 250 ] 225 0.017% 99.648% ( 250, 380 ] 442 0.033% 99.681% ( 380, 580 ] 940 0.071% 99.752% ( 580, 870 ] 1459 0.110% 99.862% ( 870, 1300 ] 1552 0.117% 99.979% ( 1300, 1900 ] 905 0.068% 100.048% ( 1900, 2900 ] 681 0.051% 100.099% ( 2900, 4400 ] 560 0.042% 100.142% ( 4400, 6600 ] 167 0.013% 100.154% ( 6600, 9900 ] 33 0.002% 100.157% ( 9900, 14000 ] 8 0.001% 100.157% ( 14000, 22000 ] 14 0.001% 100.158% ( 22000, 33000 ] 4 0.000% 100.159% ** Level 4 read latency histogram (micros): Count: 784820 Average: 18.4267 StdDev: 240.65 Min: 0 Median: 1.8824 Max: 51083 Percentiles: P50: 1.88 P75: 2.88 P99: 193.33 P99.9: 1751.90 P99.99: 2515.57 ------------------------------------------------------ [ 0, 1 ] 18407 2.345% 2.345% ( 1, 2 ] 423860 54.007% 56.353% ########### ( 2, 3 ] 166304 21.190% 77.543% #### ( 3, 4 ] 72726 9.267% 86.809% ## ( 4, 6 ] 72542 9.243% 96.052% ## ( 6, 10 ] 19831 2.527% 98.579% # ( 10, 15 ] 2394 0.305% 98.884% ( 15, 22 ] 330 0.042% 98.926% ( 22, 34 ] 23 0.003% 98.929% ( 34, 51 ] 39 0.005% 98.934% ( 51, 76 ] 66 0.008% 98.943% ( 76, 110 ] 57 0.007% 98.950% ( 110, 170 ] 256 0.033% 98.983% ( 170, 250 ] 469 0.060% 99.042% ( 250, 380 ] 668 0.085% 99.127% ( 380, 580 ] 1397 0.178% 99.305% ( 580, 870 ] 1944 0.248% 99.553% ( 870, 1300 ] 1909 0.243% 99.796% ( 1300, 1900 ] 1081 0.138% 99.934% ( 1900, 2900 ] 714 0.091% 100.025% ( 2900, 4400 ] 648 0.083% 100.108% ( 4400, 6600 ] 224 0.029% 100.136% ( 6600, 9900 ] 46 0.006% 100.142% ( 9900, 14000 ] 12 0.002% 100.144% ( 14000, 22000 ] 13 0.002% 100.145% ( 22000, 33000 ] 4 0.001% 100.146% ( 33000, 50000 ] 4 0.001% 100.146% ( 50000, 75000 ] 1 0.000% 100.146% ** Level 5 read latency histogram (micros): Count: 33693 Average: 1209.3552 StdDev: 1454.61 Min: 76 Median: 810.1529 Max: 57211 Percentiles: P50: 810.15 P75: 1439.65 P99: 6048.83 P99.9: 9845.23 P99.99: 45972.19 ------------------------------------------------------ ( 51, 76 ] 1 0.003% 0.003% ( 76, 110 ] 200 0.594% 0.597% ( 110, 170 ] 1206 3.579% 4.176% # ( 170, 250 ] 1962 5.823% 9.999% # ( 250, 380 ] 3088 9.165% 19.164% ## ( 380, 580 ] 5019 14.896% 34.060% ### ( 580, 870 ] 6767 20.084% 54.145% #### ( 870, 1300 ] 6214 18.443% 72.588% #### ( 1300, 1900 ] 3492 10.364% 82.952% ## ( 1900, 2900 ] 2466 7.319% 90.271% # ( 2900, 4400 ] 2334 6.927% 97.198% # ( 4400, 6600 ] 810 2.404% 99.602% ( 6600, 9900 ] 102 0.303% 99.905% ( 9900, 14000 ] 8 0.024% 99.929% ( 14000, 22000 ] 7 0.021% 99.950% ( 22000, 33000 ] 6 0.018% 99.967% ( 33000, 50000 ] 10 0.030% 99.997% ( 50000, 75000 ] 1 0.003% 100.000% ** DB Stats ** Uptime(secs): 14.2 total, 6.2 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 138.26 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.342 H:M:S, 44.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 160.02 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.246 H:M:S, 36.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/07-00:24:44 ... thread 0: (1000000,3000000) ops and (297325.1,172171.9) ops/second in (3.363322,17.424449) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/4 2.77 GB 258.0 2.4 0.0 2.4 5.2 2.8 0.0 1.9 83.8 180.4 29.31 17.48 46 0.637 2494K 3002 0.0 0.0 L1 18/0 1.09 GB 111.6 1.1 0.7 0.4 1.1 0.7 0.0 1.5 81.6 81.2 13.75 5.29 1 13.745 1138K 4875 0.0 0.0 L3 19/0 966.26 MB 1.0 5.1 3.5 1.6 5.1 3.5 0.0 1.5 102.9 102.4 51.19 24.19 24 2.133 2142K 2736 0.0 0.0 L4 168/0 9.63 GB 1.0 6.7 3.7 3.0 6.6 3.6 0.4 1.8 124.5 122.1 55.35 31.96 53 1.044 3431K 0 0.0 0.0 L5 1144/0 93.94 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1363/4 108.37 GB 0.0 15.4 8.0 7.4 18.0 10.6 4.3 6.5 105.2 123.0 149.60 78.91 124 1.206 9206K 10K 0.0 0.0 Int 0/0 0.00 KB 0.0 15.4 8.0 7.4 18.0 10.6 4.3 6.5 105.2 123.0 149.60 78.91 124 1.206 9206K 10K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 15.4 8.0 7.4 15.2 7.8 0.0 0.0 118.3 117.0 133.09 71.04 86 1.548 9206K 10K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.8 2.8 0.0 0.0 0.0 171.7 16.51 7.87 38 0.434 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 17.6 total, 17.4 interval Flush(GB): cumulative 2.768, interval 2.768 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 17.97 GB write, 1045.75 MB/s write, 15.37 GB read, 894.24 MB/s read, 149.6 seconds Interval compaction: 17.97 GB write, 1056.15 MB/s write, 15.37 GB read, 903.13 MB/s read, 149.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 60, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 60, interval: 60 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 870488 Average: 15.3503 StdDev: 208.13 Min: 0 Median: 1.6393 Max: 55914 Percentiles: P50: 1.64 P75: 2.28 P99: 246.51 P99.9: 2230.93 P99.99: 3954.99 ------------------------------------------------------ [ 0, 1 ] 113423 13.030% 13.030% ### ( 1, 2 ] 503434 57.834% 70.863% ############ ( 2, 3 ] 128948 14.813% 85.677% ### ( 3, 4 ] 52307 6.009% 91.686% # ( 4, 6 ] 47420 5.448% 97.133% # ( 6, 10 ] 10743 1.234% 98.367% ( 10, 15 ] 1681 0.193% 98.560% ( 15, 22 ] 188 0.022% 98.582% ( 22, 34 ] 32 0.004% 98.586% ( 34, 51 ] 14 0.002% 98.587% ( 51, 76 ] 11 0.001% 98.588% ( 76, 110 ] 501 0.058% 98.646% ( 110, 170 ] 1856 0.213% 98.859% ( 170, 250 ] 1282 0.147% 99.007% ( 250, 380 ] 1111 0.128% 99.134% ( 380, 580 ] 1617 0.186% 99.320% ( 580, 870 ] 1973 0.227% 99.547% ( 870, 1300 ] 1805 0.207% 99.754% ( 1300, 1900 ] 1068 0.123% 99.877% ( 1900, 2900 ] 618 0.071% 99.948% ( 2900, 4400 ] 526 0.060% 100.008% ( 4400, 6600 ] 128 0.015% 100.023% ( 6600, 9900 ] 21 0.002% 100.025% ( 9900, 14000 ] 1 0.000% 100.025% ( 14000, 22000 ] 6 0.001% 100.026% ( 22000, 33000 ] 4 0.000% 100.026% ( 33000, 50000 ] 3 0.000% 100.027% ( 50000, 75000 ] 2 0.000% 100.027% ** Level 1 read latency histogram (micros): Count: 95011 Average: 12.2358 StdDev: 125.65 Min: 1 Median: 1.9859 Max: 6754 Percentiles: P50: 1.99 P75: 2.99 P99: 153.27 P99.9: 1936.83 P99.99: 4377.89 ------------------------------------------------------ [ 0, 1 ] 1523 1.603% 1.603% ( 1, 2 ] 46640 49.089% 50.692% ########## ( 2, 3 ] 23263 24.485% 75.177% ##### ( 3, 4 ] 9159 9.640% 84.816% ## ( 4, 6 ] 9923 10.444% 95.261% ## ( 6, 10 ] 2645 2.784% 98.044% # ( 10, 15 ] 460 0.484% 98.529% ( 15, 22 ] 57 0.060% 98.589% ( 22, 34 ] 10 0.011% 98.599% ( 34, 51 ] 4 0.004% 98.603% ( 51, 76 ] 6 0.006% 98.610% ( 76, 110 ] 130 0.137% 98.746% ( 110, 170 ] 334 0.352% 99.098% ( 170, 250 ] 130 0.137% 99.235% ( 250, 380 ] 106 0.112% 99.346% ( 380, 580 ] 146 0.154% 99.500% ( 580, 870 ] 144 0.152% 99.652% ( 870, 1300 ] 149 0.157% 99.808% ( 1300, 1900 ] 85 0.089% 99.898% ( 1900, 2900 ] 54 0.057% 99.955% ( 2900, 4400 ] 34 0.036% 99.991% ( 4400, 6600 ] 7 0.007% 99.998% ( 6600, 9900 ] 2 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 784584 Average: 12.2311 StdDev: 144.93 Min: 1 Median: 1.9310 Max: 10236 Percentiles: P50: 1.93 P75: 2.93 P99: 11.09 P99.9: 1226.60 P99.99: 1661.09 ------------------------------------------------------ [ 0, 1 ] 18142 2.312% 2.312% ( 1, 2 ] 401899 51.224% 53.537% ########## ( 2, 3 ] 180594 23.018% 76.555% ##### ( 3, 4 ] 77071 9.823% 86.378% ## ( 4, 6 ] 77064 9.822% 96.200% ## ( 6, 10 ] 21118 2.692% 98.892% # ( 10, 15 ] 3895 0.496% 99.388% ( 15, 22 ] 606 0.077% 99.465% ( 22, 34 ] 81 0.010% 99.476% ( 34, 51 ] 55 0.007% 99.483% ( 51, 76 ] 48 0.006% 99.489% ( 76, 110 ] 29 0.004% 99.492% ( 110, 170 ] 96 0.012% 99.505% ( 170, 250 ] 170 0.022% 99.526% ( 250, 380 ] 328 0.042% 99.568% ( 380, 580 ] 670 0.085% 99.654% ( 580, 870 ] 988 0.126% 99.780% ( 870, 1300 ] 1140 0.145% 99.925% ( 1300, 1900 ] 850 0.108% 100.033% ( 1900, 2900 ] 628 0.080% 100.113% ( 2900, 4400 ] 406 0.052% 100.165% ( 4400, 6600 ] 114 0.015% 100.179% ( 6600, 9900 ] 12 0.002% 100.181% ( 9900, 14000 ] 2 0.000% 100.181% ** Level 3 read latency histogram (micros): Count: 1324887 Average: 10.4191 StdDev: 151.73 Min: 1 Median: 1.8967 Max: 30987 Percentiles: P50: 1.90 P75: 2.87 P99: 9.67 P99.9: 1008.94 P99.99: 1393.05 ------------------------------------------------------ [ 0, 1 ] 27456 2.072% 2.072% ( 1, 2 ] 708160 53.451% 55.523% ########### ( 2, 3 ] 295353 22.293% 77.816% #### ( 3, 4 ] 126637 9.558% 87.374% ## ( 4, 6 ] 126206 9.526% 96.900% ## ( 6, 10 ] 30353 2.291% 99.191% ( 10, 15 ] 4228 0.319% 99.510% ( 15, 22 ] 496 0.037% 99.547% ( 22, 34 ] 51 0.004% 99.551% ( 34, 51 ] 60 0.005% 99.556% ( 51, 76 ] 94 0.007% 99.563% ( 76, 110 ] 199 0.015% 99.578% ( 110, 170 ] 450 0.034% 99.612% ( 170, 250 ] 348 0.026% 99.638% ( 250, 380 ] 493 0.037% 99.675% ( 380, 580 ] 994 0.075% 99.750% ( 580, 870 ] 1482 0.112% 99.862% ( 870, 1300 ] 1554 0.117% 99.979% ( 1300, 1900 ] 906 0.068% 100.048% ( 1900, 2900 ] 681 0.051% 100.099% ( 2900, 4400 ] 560 0.042% 100.141% ( 4400, 6600 ] 167 0.013% 100.154% ( 6600, 9900 ] 33 0.002% 100.157% ( 9900, 14000 ] 8 0.001% 100.157% ( 14000, 22000 ] 14 0.001% 100.158% ( 22000, 33000 ] 4 0.000% 100.159% ** Level 4 read latency histogram (micros): Count: 792192 Average: 19.8780 StdDev: 240.41 Min: 0 Median: 1.8908 Max: 51083 Percentiles: P50: 1.89 P75: 2.91 P99: 334.43 P99.9: 1747.82 P99.99: 2513.14 ------------------------------------------------------ [ 0, 1 ] 18407 2.324% 2.324% ( 1, 2 ] 423972 53.519% 55.842% ########### ( 2, 3 ] 166610 21.032% 76.874% #### ( 3, 4 ] 72934 9.207% 86.081% ## ( 4, 6 ] 72717 9.179% 95.260% ## ( 6, 10 ] 19851 2.506% 97.766% # ( 10, 15 ] 2404 0.303% 98.069% ( 15, 22 ] 330 0.042% 98.111% ( 22, 34 ] 23 0.003% 98.114% ( 34, 51 ] 39 0.005% 98.119% ( 51, 76 ] 66 0.008% 98.127% ( 76, 110 ] 1270 0.160% 98.287% ( 110, 170 ] 3313 0.418% 98.705% ( 170, 250 ] 1582 0.200% 98.905% ( 250, 380 ] 1158 0.146% 99.051% ( 380, 580 ] 1887 0.238% 99.289% ( 580, 870 ] 2093 0.264% 99.554% ( 870, 1300 ] 1934 0.244% 99.798% ( 1300, 1900 ] 1085 0.137% 99.935% ( 1900, 2900 ] 714 0.090% 100.025% ( 2900, 4400 ] 648 0.082% 100.107% ( 4400, 6600 ] 224 0.028% 100.135% ( 6600, 9900 ] 46 0.006% 100.141% ( 9900, 14000 ] 12 0.002% 100.142% ( 14000, 22000 ] 13 0.002% 100.144% ( 22000, 33000 ] 4 0.001% 100.144% ( 33000, 50000 ] 4 0.001% 100.145% ( 50000, 75000 ] 1 0.000% 100.145% ** Level 5 read latency histogram (micros): Count: 78191 Average: 633.2334 StdDev: 1084.42 Min: 72 Median: 242.4836 Max: 57211 Percentiles: P50: 242.48 P75: 726.31 P99: 4829.35 P99.9: 8276.14 P99.99: 31498.13 ------------------------------------------------------ ( 51, 76 ] 11 0.014% 0.014% ( 76, 110 ] 8288 10.600% 10.614% ## ( 110, 170 ] 22075 28.232% 38.846% ###### ( 170, 250 ] 9627 12.312% 51.158% ## ( 250, 380 ] 6582 8.418% 59.576% ## ( 380, 580 ] 8110 10.372% 69.948% ## ( 580, 870 ] 7831 10.015% 79.963% ## ( 870, 1300 ] 6383 8.163% 88.127% ## ( 1300, 1900 ] 3522 4.504% 92.631% # ( 1900, 2900 ] 2483 3.176% 95.806% # ( 2900, 4400 ] 2340 2.993% 98.799% # ( 4400, 6600 ] 810 1.036% 99.835% ( 6600, 9900 ] 102 0.130% 99.965% ( 9900, 14000 ] 8 0.010% 99.976% ( 14000, 22000 ] 7 0.009% 99.985% ( 22000, 33000 ] 6 0.008% 99.992% ( 33000, 50000 ] 10 0.013% 100.005% ( 50000, 75000 ] 1 0.001% 100.006% ** DB Stats ** Uptime(secs): 17.6 total, 3.4 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 167.76 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.342 H:M:S, 36.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 292.66 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/11/07-00:24:48 ... thread 0: (1000000,4000000) ops and (296935.7,192380.0) ops/second in (3.367733,20.792182) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/4 3.75 GB 358.7 3.4 0.0 3.4 7.1 3.7 0.0 1.9 95.9 202.1 36.14 23.83 66 0.548 3518K 3994 0.0 0.0 L1 18/0 1.09 GB 111.6 1.1 0.7 0.4 1.1 0.7 0.0 1.5 81.6 81.2 13.75 5.29 1 13.745 1138K 4875 0.0 0.0 L3 19/0 966.26 MB 1.0 5.1 3.5 1.6 5.1 3.5 0.0 1.5 102.9 102.4 51.19 24.19 24 2.133 2142K 2736 0.0 0.0 L4 168/0 9.63 GB 1.0 6.7 3.7 3.0 6.6 3.6 0.4 1.8 124.5 122.1 55.35 31.96 53 1.044 3431K 0 0.0 0.0 L5 1144/0 93.94 GB 1.0 0.0 0.0 0.0 0.0 0.0 3.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1367/4 109.35 GB 0.0 16.4 8.0 8.4 19.9 11.6 4.3 5.3 107.1 130.5 156.43 85.27 144 1.086 10M 11K 0.0 0.0 Int 0/0 0.00 KB 0.0 16.4 8.0 8.4 19.9 11.6 4.3 5.3 107.1 130.5 156.43 85.27 144 1.086 10M 11K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.4 8.0 8.4 16.2 7.8 0.0 0.0 122.3 121.1 136.89 74.76 90 1.521 10M 11K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 196.6 19.54 10.50 54 0.362 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 21.0 total, 20.8 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 19.94 GB write, 973.89 MB/s write, 16.35 GB read, 798.68 MB/s read, 156.4 seconds Interval compaction: 19.94 GB write, 982.00 MB/s write, 16.35 GB read, 805.34 MB/s read, 156.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 60, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 60, interval: 60 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1129082 Average: 12.9585 StdDev: 183.44 Min: 0 Median: 1.5712 Max: 55914 Percentiles: P50: 1.57 P75: 2.02 P99: 177.09 P99.9: 1830.49 P99.99: 3665.92 ------------------------------------------------------ [ 0, 1 ] 191456 16.957% 16.957% ### ( 1, 2 ] 653164 57.849% 74.806% ############ ( 2, 3 ] 145578 12.893% 87.699% ### ( 3, 4 ] 58390 5.171% 92.871% # ( 4, 6 ] 50746 4.494% 97.365% # ( 6, 10 ] 11477 1.016% 98.382% ( 10, 15 ] 1941 0.172% 98.554% ( 15, 22 ] 230 0.020% 98.574% ( 22, 34 ] 36 0.003% 98.577% ( 34, 51 ] 15 0.001% 98.579% ( 51, 76 ] 12 0.001% 98.580% ( 76, 110 ] 999 0.088% 98.668% ( 110, 170 ] 3552 0.315% 98.983% ( 170, 250 ] 2212 0.196% 99.179% ( 250, 380 ] 1437 0.127% 99.306% ( 380, 580 ] 1891 0.167% 99.473% ( 580, 870 ] 2048 0.181% 99.655% ( 870, 1300 ] 1823 0.161% 99.816% ( 1300, 1900 ] 1071 0.095% 99.911% ( 1900, 2900 ] 623 0.055% 99.966% ( 2900, 4400 ] 527 0.047% 100.013% ( 4400, 6600 ] 128 0.011% 100.024% ( 6600, 9900 ] 21 0.002% 100.026% ( 9900, 14000 ] 1 0.000% 100.026% ( 14000, 22000 ] 6 0.001% 100.027% ( 22000, 33000 ] 4 0.000% 100.027% ( 33000, 50000 ] 3 0.000% 100.027% ( 50000, 75000 ] 2 0.000% 100.028% ** Level 1 read latency histogram (micros): Count: 96106 Average: 13.6308 StdDev: 126.51 Min: 1 Median: 1.9968 Max: 6754 Percentiles: P50: 2.00 P75: 3.06 P99: 215.00 P99.9: 1916.56 P99.99: 4373.06 ------------------------------------------------------ [ 0, 1 ] 1525 1.587% 1.587% ( 1, 2 ] 46677 48.568% 50.155% ########## ( 2, 3 ] 23354 24.300% 74.455% ##### ( 3, 4 ] 9242 9.616% 84.072% ## ( 4, 6 ] 10014 10.420% 94.491% ## ( 6, 10 ] 2661 2.769% 97.260% # ( 10, 15 ] 464 0.483% 97.743% ( 15, 22 ] 58 0.060% 97.803% ( 22, 34 ] 10 0.010% 97.814% ( 34, 51 ] 4 0.004% 97.818% ( 51, 76 ] 8 0.008% 97.826% ( 76, 110 ] 272 0.283% 98.109% ( 110, 170 ] 717 0.746% 98.855% ( 170, 250 ] 247 0.257% 99.112% ( 250, 380 ] 158 0.164% 99.277% ( 380, 580 ] 201 0.209% 99.486% ( 580, 870 ] 161 0.168% 99.654% ( 870, 1300 ] 150 0.156% 99.810% ( 1300, 1900 ] 86 0.089% 99.899% ( 1900, 2900 ] 54 0.056% 99.955% ( 2900, 4400 ] 34 0.035% 99.991% ( 4400, 6600 ] 7 0.007% 99.998% ( 6600, 9900 ] 2 0.002% 100.000% ** Level 2 read latency histogram (micros): Count: 784584 Average: 12.2311 StdDev: 144.93 Min: 1 Median: 1.9310 Max: 10236 Percentiles: P50: 1.93 P75: 2.93 P99: 11.09 P99.9: 1226.60 P99.99: 1661.09 ------------------------------------------------------ [ 0, 1 ] 18142 2.312% 2.312% ( 1, 2 ] 401899 51.224% 53.537% ########## ( 2, 3 ] 180594 23.018% 76.555% ##### ( 3, 4 ] 77071 9.823% 86.378% ## ( 4, 6 ] 77064 9.822% 96.200% ## ( 6, 10 ] 21118 2.692% 98.892% # ( 10, 15 ] 3895 0.496% 99.388% ( 15, 22 ] 606 0.077% 99.465% ( 22, 34 ] 81 0.010% 99.476% ( 34, 51 ] 55 0.007% 99.483% ( 51, 76 ] 48 0.006% 99.489% ( 76, 110 ] 29 0.004% 99.492% ( 110, 170 ] 96 0.012% 99.505% ( 170, 250 ] 170 0.022% 99.526% ( 250, 380 ] 328 0.042% 99.568% ( 380, 580 ] 670 0.085% 99.654% ( 580, 870 ] 988 0.126% 99.780% ( 870, 1300 ] 1140 0.145% 99.925% ( 1300, 1900 ] 850 0.108% 100.033% ( 1900, 2900 ] 628 0.080% 100.113% ( 2900, 4400 ] 406 0.052% 100.165% ( 4400, 6600 ] 114 0.015% 100.179% ( 6600, 9900 ] 12 0.002% 100.181% ( 9900, 14000 ] 2 0.000% 100.181% ** Level 3 read latency histogram (micros): Count: 1325774 Average: 10.5298 StdDev: 151.79 Min: 1 Median: 1.8973 Max: 30987 Percentiles: P50: 1.90 P75: 2.88 P99: 9.78 P99.9: 1009.62 P99.99: 1394.22 ------------------------------------------------------ [ 0, 1 ] 27456 2.071% 2.071% ( 1, 2 ] 708164 53.415% 55.486% ########### ( 2, 3 ] 295365 22.279% 77.765% #### ( 3, 4 ] 126649 9.553% 87.318% ## ( 4, 6 ] 126214 9.520% 96.838% ## ( 6, 10 ] 30355 2.290% 99.127% ( 10, 15 ] 4228 0.319% 99.446% ( 15, 22 ] 496 0.037% 99.484% ( 22, 34 ] 51 0.004% 99.487% ( 34, 51 ] 60 0.005% 99.492% ( 51, 76 ] 94 0.007% 99.499% ( 76, 110 ] 390 0.029% 99.528% ( 110, 170 ] 862 0.065% 99.593% ( 170, 250 ] 470 0.035% 99.629% ( 250, 380 ] 553 0.042% 99.671% ( 380, 580 ] 1042 0.079% 99.749% ( 580, 870 ] 1494 0.113% 99.862% ( 870, 1300 ] 1556 0.117% 99.979% ( 1300, 1900 ] 907 0.068% 100.048% ( 1900, 2900 ] 682 0.051% 100.099% ( 2900, 4400 ] 560 0.042% 100.141% ( 4400, 6600 ] 167 0.013% 100.154% ( 6600, 9900 ] 33 0.002% 100.156% ( 9900, 14000 ] 8 0.001% 100.157% ( 14000, 22000 ] 14 0.001% 100.158% ( 22000, 33000 ] 4 0.000% 100.158% ** Level 4 read latency histogram (micros): Count: 799548 Average: 21.3320 StdDev: 240.14 Min: 0 Median: 1.8995 Max: 51083 Percentiles: P50: 1.90 P75: 2.94 P99: 387.55 P99.9: 1746.39 P99.99: 2514.27 ------------------------------------------------------ [ 0, 1 ] 18407 2.302% 2.302% ( 1, 2 ] 423991 53.029% 55.331% ########### ( 2, 3 ] 166730 20.853% 76.184% #### ( 3, 4 ] 73033 9.134% 85.318% ## ( 4, 6 ] 72823 9.108% 94.426% ## ( 6, 10 ] 19865 2.485% 96.911% ( 10, 15 ] 2408 0.301% 97.212% ( 15, 22 ] 330 0.041% 97.253% ( 22, 34 ] 23 0.003% 97.256% ( 34, 51 ] 39 0.005% 97.261% ( 51, 76 ] 67 0.008% 97.269% ( 76, 110 ] 2567 0.321% 97.591% ( 110, 170 ] 6783 0.848% 98.439% ( 170, 250 ] 2724 0.341% 98.780% ( 250, 380 ] 1675 0.209% 98.989% ( 380, 580 ] 2317 0.290% 99.279% ( 580, 870 ] 2202 0.275% 99.554% ( 870, 1300 ] 1955 0.245% 99.799% ( 1300, 1900 ] 1088 0.136% 99.935% ( 1900, 2900 ] 718 0.090% 100.025% ( 2900, 4400 ] 648 0.081% 100.106% ( 4400, 6600 ] 224 0.028% 100.134% ( 6600, 9900 ] 46 0.006% 100.139% ( 9900, 14000 ] 12 0.002% 100.141% ( 14000, 22000 ] 13 0.002% 100.143% ( 22000, 33000 ] 4 0.001% 100.143% ( 33000, 50000 ] 4 0.001% 100.144% ( 50000, 75000 ] 1 0.000% 100.144% ** Level 5 read latency histogram (micros): Count: 124250 Average: 467.3066 StdDev: 890.60 Min: 72 Median: 171.1193 Max: 57211 Percentiles: P50: 171.12 P75: 458.48 P99: 4204.25 P99.9: 6688.97 P99.99: 19228.57 ------------------------------------------------------ ( 51, 76 ] 15 0.012% 0.012% ( 76, 110 ] 17053 13.725% 13.737% ### ( 110, 170 ] 44815 36.068% 49.805% ####### ( 170, 250 ] 17296 13.920% 63.726% ### ( 250, 380 ] 9731 7.832% 71.557% ## ( 380, 580 ] 10901 8.773% 80.331% ## ( 580, 870 ] 8617 6.935% 87.266% # ( 870, 1300 ] 6502 5.233% 92.499% # ( 1300, 1900 ] 3546 2.854% 95.353% # ( 1900, 2900 ] 2496 2.009% 97.362% ( 2900, 4400 ] 2341 1.884% 99.246% ( 4400, 6600 ] 810 0.652% 99.898% ( 6600, 9900 ] 102 0.082% 99.980% ( 9900, 14000 ] 8 0.006% 99.986% ( 14000, 22000 ] 7 0.006% 99.992% ( 22000, 33000 ] 6 0.005% 99.997% ( 33000, 50000 ] 10 0.008% 100.005% ( 50000, 75000 ] 1 0.001% 100.006% ** DB Stats ** Uptime(secs): 21.0 total, 3.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 187.76 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.342 H:M:S, 30.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 292.24 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 243.567 micros/op 16414 ops/sec 60.128 seconds 986996 operations; 15.9 MB/s (249999 of 249999 found) Microseconds per read: Count: 986996 Average: 243.5696 StdDev: 682.15 Min: 2 Median: 169.9720 Max: 67331 Percentiles: P50: 169.97 P75: 229.87 P99: 1412.31 P99.9: 10984.86 P99.99: 27748.08 ------------------------------------------------------ ( 1, 2 ] 3 0.000% 0.000% ( 2, 3 ] 30 0.003% 0.003% ( 3, 4 ] 48 0.005% 0.008% ( 4, 6 ] 76 0.008% 0.016% ( 6, 10 ] 617 0.063% 0.078% ( 10, 15 ] 4217 0.427% 0.506% ( 15, 22 ] 3748 0.380% 0.885% ( 22, 34 ] 10821 1.096% 1.982% ( 34, 51 ] 2864 0.290% 2.272% ( 51, 76 ] 380 0.039% 2.310% ( 76, 110 ] 22844 2.314% 4.625% ( 110, 170 ] 448059 45.396% 50.021% ######### ( 170, 250 ] 329430 33.377% 83.398% ####### ( 250, 380 ] 96190 9.746% 93.144% ## ( 380, 580 ] 34907 3.537% 96.681% # ( 580, 870 ] 14373 1.456% 98.137% ( 870, 1300 ] 7772 0.787% 98.924% ( 1300, 1900 ] 3991 0.404% 99.329% ( 1900, 2900 ] 2601 0.264% 99.592% ( 2900, 4400 ] 1921 0.195% 99.787% ( 4400, 6600 ] 671 0.068% 99.855% ( 6600, 9900 ] 310 0.031% 99.886% ( 9900, 14000 ] 514 0.052% 99.938% ( 14000, 22000 ] 457 0.046% 99.985% ( 22000, 33000 ] 102 0.010% 99.995% ( 33000, 50000 ] 43 0.004% 99.999% ( 50000, 75000 ] 7 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 20192308 rocksdb.bloom.filter.full.positive COUNT : 1183561 rocksdb.bloom.filter.full.true.positive COUNT : 986565 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 431 rocksdb.memtable.miss COUNT : 986565 rocksdb.l0.hit COUNT : 42371 rocksdb.l1.hit COUNT : 10441 rocksdb.l2andup.hit COUNT : 933753 rocksdb.compaction.key.drop.new COUNT : 224192 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 4809680 rocksdb.number.keys.read COUNT : 986996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 4963589760 rocksdb.bytes.read COUNT : 986996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1628 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 41955811 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 20723898582 rocksdb.compact.write.bytes COUNT : 20498920141 rocksdb.flush.write.bytes COUNT : 4954447133 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 31 rocksdb.number.superversion_acquires COUNT : 1019 rocksdb.number.superversion_releases COUNT : 996 rocksdb.number.superversion_cleanups COUNT : 255 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 251 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 4914376704 rocksdb.memtable.garbage.bytes.at.flush COUNT : 1795072 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 25539445889 rocksdb.non.last.level.read.count COUNT : 6029324 rocksdb.block.checksum.compute.count COUNT : 6218116 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 169.075922 P95 : 483.680065 P99 : 1411.217457 P100 : 67328.000000 COUNT : 986996 SUM : 239546691 rocksdb.db.write.micros P50 : 1.542909 P95 : 4.500536 P99 : 9.719948 P100 : 891275.000000 COUNT : 4809680 SUM : 54094814 rocksdb.compaction.times.micros P50 : 1133478.260870 P95 : 3264000.000000 P99 : 10052000.000000 P100 : 13745109.000000 COUNT : 94 SUM : 140719968 rocksdb.compaction.times.cpu_micros P50 : 767727.272727 P95 : 1664545.454545 P99 : 4432000.000000 P100 : 5290739.000000 COUNT : 94 SUM : 78520885 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 810.294118 P95 : 319333.333333 P99 : 386595.000000 P100 : 386595.000000 COUNT : 68 SUM : 2418339 rocksdb.compaction.outfile.sync.micros P50 : 62500.000000 P95 : 269500.000000 P99 : 448400.000000 P100 : 1067261.000000 COUNT : 264 SUM : 23821025 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 380.000000 P95 : 4100.000000 P99 : 7260.000000 P100 : 20163.000000 COUNT : 180 SUM : 207534 rocksdb.table.open.io.micros P50 : 8545.841035 P95 : 17850.322581 P99 : 21211.354839 P100 : 22789.000000 COUNT : 1628 SUM : 13790121 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.939844 P95 : 12.120376 P99 : 167.981036 P100 : 55919.000000 COUNT : 5020397 SUM : 82192463 rocksdb.read.block.get.micros P50 : 135.314923 P95 : 343.265644 P99 : 1259.430553 P100 : 57215.000000 COUNT : 1196127 SUM : 203417414 rocksdb.write.raw.block.micros P50 : 0.527338 P95 : 1.056728 P99 : 3.714891 P100 : 80482.000000 COUNT : 6165052 SUM : 18351407 rocksdb.numfiles.in.singlecompaction P50 : 2.057692 P95 : 3.941071 P99 : 9.480000 P100 : 35.000000 COUNT : 113 SUM : 348 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.585762 P95 : 2271.807972 P99 : 2775.374937 P100 : 891096.000000 COUNT : 149081 SUM : 41970829 rocksdb.sst.read.micros P50 : 2.146644 P95 : 163.884207 P99 : 505.240116 P100 : 57211.000000 COUNT : 6029435 SUM : 269666528 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 986996 SUM : 986996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 4809680 SUM : 4963589760 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 225000.000000 P95 : 1007333.333333 P99 : 1117010.000000 P100 : 1117010.000000 COUNT : 68 SUM : 22354248 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 265 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0