[2024-11-16 20:48:23.704748] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-11-16 20:48:23.704898] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1992371 ] Set seed to 1731786503701252 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-16 20:48:23.813444] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-16 20:48:24.016791] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-16 20:48:24.016836] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1992371' to capture a snapshot of events at runtime. [2024-11-16 20:48:24.016847] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1992371 for offline analysis/debug. [2024-11-16 20:48:24.016875] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sat Nov 16 20:48:32 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/16-20:48:39 ... thread 0: (1000000,1000000) ops and (136884.2,136884.2) ops/second in (7.305444,7.305444) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 14/12 2.12 GB 50.3 0.4 0.0 0.4 1.2 0.8 0.0 1.5 43.4 137.3 8.71 3.93 10 0.871 383K 530 0.0 0.0 L1 10/10 1.16 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.96 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 29/5 1.24 GB 1.0 1.6 0.9 0.7 1.5 0.9 0.0 1.7 96.4 95.5 16.49 7.63 13 1.269 141K 0 0.0 0.0 L4 171/11 10.18 GB 1.0 4.1 2.4 1.6 4.0 2.4 0.0 1.7 103.9 102.7 40.03 19.36 29 1.380 1330K 0 0.0 0.0 L5 1120/0 91.91 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1345/38 106.68 GB 0.0 6.0 3.3 2.7 6.7 4.0 1.6 8.4 93.9 105.5 65.23 30.92 52 1.254 1855K 530 0.0 0.0 Int 0/0 0.00 KB 0.0 6.0 3.3 2.7 6.7 4.0 1.6 8.4 93.9 105.5 65.23 30.92 52 1.254 1855K 530 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 6.0 3.3 2.7 5.9 3.2 0.0 0.0 104.3 103.2 58.75 28.51 43 1.366 1855K 530 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 126.3 6.49 2.41 9 0.721 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 7.5 total, 7.3 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.72 GB write, 918.57 MB/s write, 5.98 GB read, 817.68 MB/s read, 65.2 seconds Interval compaction: 6.72 GB write, 941.58 MB/s write, 5.98 GB read, 838.17 MB/s read, 65.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 35, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 35, interval: 35 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 319551 Average: 15.0967 StdDev: 189.68 Min: 1 Median: 1.8653 Max: 14176 Percentiles: P50: 1.87 P75: 2.87 P99: 11.74 P99.9: 2902.40 P99.99: 4621.23 ------------------------------------------------------ [ 0, 1 ] 14713 4.604% 4.604% # ( 1, 2 ] 167653 52.465% 57.069% ########## ( 2, 3 ] 66214 20.721% 77.790% #### ( 3, 4 ] 27554 8.623% 86.413% ## ( 4, 6 ] 32452 10.155% 96.569% ## ( 6, 10 ] 7275 2.277% 98.845% ( 10, 15 ] 1428 0.447% 99.292% ( 15, 22 ] 170 0.053% 99.345% ( 22, 34 ] 11 0.003% 99.349% ( 34, 51 ] 7 0.002% 99.351% ( 51, 76 ] 7 0.002% 99.353% ( 76, 110 ] 2 0.001% 99.354% ( 110, 170 ] 5 0.002% 99.355% ( 170, 250 ] 20 0.006% 99.362% ( 250, 380 ] 76 0.024% 99.385% ( 380, 580 ] 156 0.049% 99.434% ( 580, 870 ] 275 0.086% 99.520% ( 870, 1300 ] 418 0.131% 99.651% ( 1300, 1900 ] 430 0.135% 99.786% ( 1900, 2900 ] 367 0.115% 99.900% ( 2900, 4400 ] 280 0.088% 99.988% ( 4400, 6600 ] 80 0.025% 100.013% ( 6600, 9900 ] 17 0.005% 100.018% ( 9900, 14000 ] 6 0.002% 100.020% ( 14000, 22000 ] 1 0.000% 100.021% ** Level 1 read latency histogram (micros): Count: 9302 Average: 53.6172 StdDev: 511.40 Min: 1 Median: 2.0450 Max: 12527 Percentiles: P50: 2.04 P75: 3.19 P99: 1471.00 P99.9: 8885.96 P99.99: 12527.00 ------------------------------------------------------ [ 0, 1 ] 160 1.720% 1.720% ( 1, 2 ] 4389 47.183% 48.903% ######### ( 2, 3 ] 2269 24.393% 73.296% ##### ( 3, 4 ] 843 9.063% 82.359% ## ( 4, 6 ] 1131 12.159% 94.517% ## ( 6, 10 ] 257 2.763% 97.280% # ( 10, 15 ] 64 0.688% 97.968% ( 15, 22 ] 3 0.032% 98.000% ( 34, 51 ] 1 0.011% 98.011% ( 51, 76 ] 2 0.022% 98.033% ( 250, 380 ] 6 0.065% 98.097% ( 380, 580 ] 18 0.194% 98.291% ( 580, 870 ] 33 0.355% 98.645% ( 870, 1300 ] 25 0.269% 98.914% ( 1300, 1900 ] 28 0.301% 99.215% ( 1900, 2900 ] 20 0.215% 99.430% ( 2900, 4400 ] 19 0.204% 99.634% ( 4400, 6600 ] 15 0.161% 99.796% ( 6600, 9900 ] 14 0.151% 99.946% ( 9900, 14000 ] 5 0.054% 100.000% ** Level 2 read latency histogram (micros): Count: 71201 Average: 15.5865 StdDev: 190.86 Min: 1 Median: 1.9535 Max: 8072 Percentiles: P50: 1.95 P75: 2.97 P99: 13.32 P99.9: 2784.08 P99.99: 4247.00 ------------------------------------------------------ [ 0, 1 ] 2449 3.440% 3.440% # ( 1, 2 ] 34768 48.831% 52.270% ########## ( 2, 3 ] 16691 23.442% 75.712% ##### ( 3, 4 ] 6369 8.945% 84.658% ## ( 4, 6 ] 7783 10.931% 95.589% ## ( 6, 10 ] 2100 2.949% 98.538% # ( 10, 15 ] 496 0.697% 99.235% ( 15, 22 ] 92 0.129% 99.364% ( 22, 34 ] 10 0.014% 99.378% ( 34, 51 ] 3 0.004% 99.382% ( 110, 170 ] 1 0.001% 99.383% ( 170, 250 ] 5 0.007% 99.390% ( 250, 380 ] 16 0.022% 99.413% ( 380, 580 ] 29 0.041% 99.454% ( 580, 870 ] 58 0.081% 99.535% ( 870, 1300 ] 74 0.104% 99.639% ( 1300, 1900 ] 108 0.152% 99.791% ( 1900, 2900 ] 88 0.124% 99.914% ( 2900, 4400 ] 60 0.084% 99.999% ( 4400, 6600 ] 25 0.035% 100.034% ( 6600, 9900 ] 5 0.007% 100.041% ** Level 3 read latency histogram (micros): Count: 737649 Average: 17.4407 StdDev: 206.20 Min: 1 Median: 1.9821 Max: 21736 Percentiles: P50: 1.98 P75: 3.03 P99: 12.36 P99.9: 1965.47 P99.99: 2513.23 ------------------------------------------------------ [ 0, 1 ] 19168 2.599% 2.599% # ( 1, 2 ] 356033 48.266% 50.864% ########## ( 2, 3 ] 175999 23.859% 74.724% ##### ( 3, 4 ] 71133 9.643% 84.367% ## ( 4, 6 ] 85314 11.566% 95.933% ## ( 6, 10 ] 20181 2.736% 98.669% # ( 10, 15 ] 5185 0.703% 99.372% ( 15, 22 ] 680 0.092% 99.464% ( 22, 34 ] 56 0.008% 99.471% ( 34, 51 ] 39 0.005% 99.477% ( 51, 76 ] 37 0.005% 99.482% ( 76, 110 ] 16 0.002% 99.484% ( 110, 170 ] 35 0.005% 99.489% ( 170, 250 ] 60 0.008% 99.497% ( 250, 380 ] 104 0.014% 99.511% ( 380, 580 ] 274 0.037% 99.548% ( 580, 870 ] 533 0.072% 99.620% ( 870, 1300 ] 883 0.120% 99.740% ( 1300, 1900 ] 1103 0.150% 99.889% ( 1900, 2900 ] 1212 0.164% 100.054% ( 2900, 4400 ] 832 0.113% 100.166% ( 4400, 6600 ] 247 0.033% 100.200% ( 6600, 9900 ] 38 0.005% 100.205% ( 9900, 14000 ] 1 0.000% 100.205% ( 14000, 22000 ] 4 0.001% 100.206% ** Level 4 read latency histogram (micros): Count: 488656 Average: 23.4950 StdDev: 247.68 Min: 1 Median: 2.0164 Max: 23225 Percentiles: P50: 2.02 P75: 3.11 P99: 19.67 P99.9: 2682.70 P99.99: 3392.54 ------------------------------------------------------ [ 0, 1 ] 11455 2.344% 2.344% ( 1, 2 ] 230932 47.259% 49.603% ######### ( 2, 3 ] 118579 24.266% 73.869% ##### ( 3, 4 ] 48760 9.978% 83.848% ## ( 4, 6 ] 56512 11.565% 95.412% ## ( 6, 10 ] 13970 2.859% 98.271% # ( 10, 15 ] 3310 0.677% 98.949% ( 15, 22 ] 378 0.077% 99.026% ( 22, 34 ] 37 0.008% 99.033% ( 34, 51 ] 37 0.008% 99.041% ( 51, 76 ] 29 0.006% 99.047% ( 76, 110 ] 12 0.002% 99.049% ( 110, 170 ] 35 0.007% 99.057% ( 170, 250 ] 108 0.022% 99.079% ( 250, 380 ] 158 0.032% 99.111% ( 380, 580 ] 368 0.075% 99.186% ( 580, 870 ] 660 0.135% 99.321% ( 870, 1300 ] 1023 0.209% 99.531% ( 1300, 1900 ] 1061 0.217% 99.748% ( 1900, 2900 ] 951 0.195% 99.942% ( 2900, 4400 ] 710 0.145% 100.088% ( 4400, 6600 ] 247 0.051% 100.138% ( 6600, 9900 ] 40 0.008% 100.147% ( 9900, 14000 ] 9 0.002% 100.148% ( 14000, 22000 ] 3 0.001% 100.149% ( 22000, 33000 ] 1 0.000% 100.149% ** Level 5 read latency histogram (micros): Count: 9022 Average: 2261.9905 StdDev: 1740.54 Min: 117 Median: 1745.1613 Max: 28204 Percentiles: P50: 1745.16 P75: 3244.54 P99: 8589.97 P99.9: 11755.44 P99.99: 28037.90 ------------------------------------------------------ ( 110, 170 ] 30 0.333% 0.333% ( 170, 250 ] 68 0.754% 1.086% ( 250, 380 ] 228 2.527% 3.613% # ( 380, 580 ] 531 5.886% 9.499% # ( 580, 870 ] 1041 11.538% 21.037% ## ( 870, 1300 ] 1486 16.471% 37.508% ### ( 1300, 1900 ] 1519 16.837% 54.345% ### ( 1900, 2900 ] 1527 16.925% 71.270% ### ( 2900, 4400 ] 1465 16.238% 87.508% ### ( 4400, 6600 ] 921 10.208% 97.717% ## ( 6600, 9900 ] 192 2.128% 99.845% ( 9900, 14000 ] 11 0.122% 99.967% ( 14000, 22000 ] 1 0.011% 99.978% ( 22000, 33000 ] 2 0.022% 100.000% ** DB Stats ** Uptime(secs): 7.5 total, 7.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 131.39 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.446 H:M:S, 46.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 134.67 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:3.446 H:M:S, 47.1 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 7 num-running-flushes: 1 2024/11/16-20:48:43 ... thread 0: (1000000,2000000) ops and (252735.3,177585.9) ops/second in (3.956709,11.262153) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/12 3.17 GB 163.6 1.5 0.0 1.5 3.4 1.8 0.0 1.8 89.6 197.0 17.57 11.46 30 0.586 1598K 1956 0.0 0.0 L1 10/10 1.16 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 69.96 MB 0.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 24/0 987.50 MB 1.0 1.6 0.9 0.7 1.5 0.9 0.0 1.7 96.4 95.5 16.49 7.63 13 1.269 141K 0 0.0 0.0 L4 167/0 9.76 GB 1.0 4.8 2.7 2.1 4.8 2.6 0.0 1.8 107.4 106.1 45.94 23.04 34 1.351 1631K 0 0.0 0.0 L5 1128/0 92.59 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1348/22 107.71 GB 0.0 7.9 3.6 4.3 9.7 5.4 2.3 5.2 101.2 123.9 80.00 42.14 77 1.039 3372K 1956 0.0 0.0 Int 0/0 0.00 KB 0.0 7.9 3.6 4.3 9.7 5.4 2.3 5.2 101.2 123.9 80.00 42.14 77 1.039 3372K 1956 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 7.9 3.6 4.3 7.8 3.5 0.0 0.0 116.3 115.2 69.65 36.80 52 1.339 3372K 1956 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 182.4 10.36 5.34 25 0.414 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 11.4 total, 11.3 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 9.68 GB write, 865.74 MB/s write, 7.91 GB read, 707.43 MB/s read, 80.0 seconds Interval compaction: 9.68 GB write, 879.81 MB/s write, 7.91 GB read, 718.93 MB/s read, 80.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 2, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 2, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 39, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 2, total-stops: 39, interval: 41 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 658284 Average: 14.3309 StdDev: 141.66 Min: 1 Median: 1.6821 Max: 14176 Percentiles: P50: 1.68 P75: 2.47 P99: 235.31 P99.9: 1674.03 P99.99: 2921.28 ------------------------------------------------------ [ 0, 1 ] 78122 11.868% 11.868% ## ( 1, 2 ] 367989 55.901% 67.769% ########### ( 2, 3 ] 101849 15.472% 83.241% ### ( 3, 4 ] 39431 5.990% 89.231% # ( 4, 6 ] 41065 6.238% 95.469% # ( 6, 10 ] 9589 1.457% 96.925% ( 10, 15 ] 2040 0.310% 97.235% ( 15, 22 ] 278 0.042% 97.278% ( 22, 34 ] 23 0.003% 97.281% ( 34, 51 ] 13 0.002% 97.283% ( 51, 76 ] 9 0.001% 97.284% ( 76, 110 ] 528 0.080% 97.365% ( 110, 170 ] 6440 0.978% 98.343% ( 170, 250 ] 5298 0.805% 99.148% ( 250, 380 ] 1842 0.280% 99.428% ( 380, 580 ] 1426 0.217% 99.644% ( 580, 870 ] 774 0.118% 99.762% ( 870, 1300 ] 608 0.092% 99.854% ( 1300, 1900 ] 484 0.074% 99.928% ( 1900, 2900 ] 406 0.062% 99.989% ( 2900, 4400 ] 294 0.045% 100.034% ( 4400, 6600 ] 81 0.012% 100.046% ( 6600, 9900 ] 17 0.003% 100.049% ( 9900, 14000 ] 6 0.001% 100.050% ( 14000, 22000 ] 1 0.000% 100.050% ** Level 1 read latency histogram (micros): Count: 18482 Average: 37.6508 StdDev: 370.87 Min: 1 Median: 2.1029 Max: 12527 Percentiles: P50: 2.10 P75: 3.35 P99: 640.90 P99.9: 6722.10 P99.99: 12484.48 ------------------------------------------------------ [ 0, 1 ] 318 1.721% 1.721% ( 1, 2 ] 8457 45.758% 47.479% ######### ( 2, 3 ] 4530 24.510% 71.989% ##### ( 3, 4 ] 1597 8.641% 80.630% ## ( 4, 6 ] 2144 11.600% 92.230% ## ( 6, 10 ] 480 2.597% 94.827% # ( 10, 15 ] 117 0.633% 95.460% ( 15, 22 ] 11 0.060% 95.520% ( 22, 34 ] 1 0.005% 95.525% ( 34, 51 ] 1 0.005% 95.531% ( 51, 76 ] 3 0.016% 95.547% ( 76, 110 ] 19 0.103% 95.650% ( 110, 170 ] 239 1.293% 96.943% ( 170, 250 ] 202 1.093% 98.036% ( 250, 380 ] 89 0.482% 98.517% ( 380, 580 ] 77 0.417% 98.934% ( 580, 870 ] 58 0.314% 99.248% ( 870, 1300 ] 29 0.157% 99.405% ( 1300, 1900 ] 33 0.179% 99.583% ( 1900, 2900 ] 22 0.119% 99.702% ( 2900, 4400 ] 20 0.108% 99.811% ( 4400, 6600 ] 16 0.087% 99.897% ( 6600, 9900 ] 14 0.076% 99.973% ( 9900, 14000 ] 5 0.027% 100.000% ** Level 2 read latency histogram (micros): Count: 71272 Average: 15.8130 StdDev: 190.97 Min: 1 Median: 1.9545 Max: 8072 Percentiles: P50: 1.95 P75: 2.97 P99: 14.02 P99.9: 2783.27 P99.99: 4246.82 ------------------------------------------------------ [ 0, 1 ] 2449 3.436% 3.436% # ( 1, 2 ] 34768 48.782% 52.218% ########## ( 2, 3 ] 16691 23.419% 75.637% ##### ( 3, 4 ] 6369 8.936% 84.573% ## ( 4, 6 ] 7783 10.920% 95.493% ## ( 6, 10 ] 2100 2.946% 98.440% # ( 10, 15 ] 496 0.696% 99.136% ( 15, 22 ] 92 0.129% 99.265% ( 22, 34 ] 10 0.014% 99.279% ( 34, 51 ] 3 0.004% 99.283% ( 76, 110 ] 6 0.008% 99.291% ( 110, 170 ] 30 0.042% 99.334% ( 170, 250 ] 23 0.032% 99.366% ( 250, 380 ] 22 0.031% 99.397% ( 380, 580 ] 36 0.051% 99.447% ( 580, 870 ] 62 0.087% 99.534% ( 870, 1300 ] 75 0.105% 99.639% ( 1300, 1900 ] 108 0.152% 99.791% ( 1900, 2900 ] 88 0.123% 99.914% ( 2900, 4400 ] 60 0.084% 99.999% ( 4400, 6600 ] 25 0.035% 100.034% ( 6600, 9900 ] 5 0.007% 100.041% ** Level 3 read latency histogram (micros): Count: 778406 Average: 16.8783 StdDev: 201.02 Min: 1 Median: 1.9763 Max: 21736 Percentiles: P50: 1.98 P75: 3.01 P99: 12.65 P99.9: 1919.43 P99.99: 2496.51 ------------------------------------------------------ [ 0, 1 ] 20406 2.622% 2.622% # ( 1, 2 ] 377751 48.529% 51.150% ########## ( 2, 3 ] 185113 23.781% 74.931% ##### ( 3, 4 ] 74370 9.554% 84.485% ## ( 4, 6 ] 88957 11.428% 95.914% ## ( 6, 10 ] 21178 2.721% 98.634% # ( 10, 15 ] 5381 0.691% 99.326% ( 15, 22 ] 705 0.091% 99.416% ( 22, 34 ] 60 0.008% 99.424% ( 34, 51 ] 39 0.005% 99.429% ( 51, 76 ] 39 0.005% 99.434% ( 76, 110 ] 37 0.005% 99.439% ( 110, 170 ] 267 0.034% 99.473% ( 170, 250 ] 250 0.032% 99.505% ( 250, 380 ] 179 0.023% 99.528% ( 380, 580 ] 319 0.041% 99.569% ( 580, 870 ] 553 0.071% 99.640% ( 870, 1300 ] 890 0.114% 99.754% ( 1300, 1900 ] 1110 0.143% 99.897% ( 1900, 2900 ] 1214 0.156% 100.053% ( 2900, 4400 ] 833 0.107% 100.160% ( 4400, 6600 ] 247 0.032% 100.192% ( 6600, 9900 ] 38 0.005% 100.197% ( 9900, 14000 ] 1 0.000% 100.197% ( 14000, 22000 ] 4 0.001% 100.197% ** Level 4 read latency histogram (micros): Count: 557850 Average: 23.6666 StdDev: 236.46 Min: 1 Median: 2.0046 Max: 23225 Percentiles: P50: 2.00 P75: 3.12 P99: 304.84 P99.9: 2583.08 P99.99: 3291.39 ------------------------------------------------------ [ 0, 1 ] 13449 2.411% 2.411% ( 1, 2 ] 264864 47.479% 49.890% ######### ( 2, 3 ] 133387 23.911% 73.801% ##### ( 3, 4 ] 54111 9.700% 83.501% ## ( 4, 6 ] 62524 11.208% 94.709% ## ( 6, 10 ] 15581 2.793% 97.502% # ( 10, 15 ] 3640 0.653% 98.155% ( 15, 22 ] 424 0.076% 98.231% ( 22, 34 ] 45 0.008% 98.239% ( 34, 51 ] 40 0.007% 98.246% ( 51, 76 ] 34 0.006% 98.252% ( 76, 110 ] 178 0.032% 98.284% ( 110, 170 ] 1999 0.358% 98.642% ( 170, 250 ] 1704 0.305% 98.948% ( 250, 380 ] 691 0.124% 99.072% ( 380, 580 ] 810 0.145% 99.217% ( 580, 870 ] 899 0.161% 99.378% ( 870, 1300 ] 1112 0.199% 99.577% ( 1300, 1900 ] 1128 0.202% 99.780% ( 1900, 2900 ] 984 0.176% 99.956% ( 2900, 4400 ] 729 0.131% 100.087% ( 4400, 6600 ] 255 0.046% 100.132% ( 6600, 9900 ] 40 0.007% 100.139% ( 9900, 14000 ] 9 0.002% 100.141% ( 14000, 22000 ] 3 0.001% 100.142% ( 22000, 33000 ] 1 0.000% 100.142% ** Level 5 read latency histogram (micros): Count: 48774 Average: 629.3353 StdDev: 1111.77 Min: 82 Median: 216.0508 Max: 28204 Percentiles: P50: 216.05 P75: 491.28 P99: 5946.17 P99.9: 9302.32 P99.99: 13300.24 ------------------------------------------------------ ( 76, 110 ] 1492 3.059% 3.059% # ( 110, 170 ] 15604 31.992% 35.051% ###### ( 170, 250 ] 12666 25.969% 61.020% ##### ( 250, 380 ] 4649 9.532% 70.552% ## ( 380, 580 ] 3899 7.994% 78.546% ## ( 580, 870 ] 2472 5.068% 83.614% # ( 870, 1300 ] 1946 3.990% 87.604% # ( 1300, 1900 ] 1699 3.483% 91.087% # ( 1900, 2900 ] 1643 3.369% 94.456% # ( 2900, 4400 ] 1550 3.178% 97.634% # ( 4400, 6600 ] 948 1.944% 99.578% ( 6600, 9900 ] 192 0.394% 99.971% ( 9900, 14000 ] 11 0.023% 99.994% ( 14000, 22000 ] 1 0.002% 99.996% ( 22000, 33000 ] 2 0.004% 100.000% ** DB Stats ** Uptime(secs): 11.4 total, 4.0 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 171.95 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:3.928 H:M:S, 34.3 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 248.74 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.483 H:M:S, 12.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/16-20:49:17 ... thread 0: (1000000,3000000) ops and (29892.9,67091.7) ops/second in (33.452743,44.714896) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 11/6 2.70 GB 125.8 2.5 0.0 2.5 5.2 2.7 0.0 1.9 102.7 212.8 25.15 17.68 47 0.535 2622K 2896 0.0 0.0 L1 6/0 589.56 MB 59.0 2.5 1.3 1.2 2.5 1.3 0.0 1.9 60.2 59.6 42.34 17.50 1 42.343 2589K 24K 0.0 0.0 L2 5/4 385.27 MB 1.3 0.0 0.0 0.0 0.0 0.0 1.9 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 41/12 2.37 GB 1.5 2.6 1.5 1.1 2.6 1.5 1.0 1.7 101.8 101.0 26.23 13.11 23 1.140 804K 2518 0.0 0.0 L4 170/6 9.93 GB 1.0 4.9 2.7 2.1 4.8 2.7 0.1 1.8 107.4 106.1 46.33 23.25 35 1.324 1677K 0 0.0 0.0 L5 1128/0 92.59 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.3 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1361/28 108.54 GB 0.0 12.5 5.6 6.9 15.1 8.2 5.3 5.6 91.3 110.3 140.05 71.54 106 1.321 7693K 29K 0.0 0.0 Int 0/0 0.00 KB 0.0 12.5 5.6 6.9 15.1 8.2 5.3 5.6 91.3 110.3 140.05 71.54 106 1.321 7693K 29K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 12.5 5.6 6.9 12.4 5.5 0.0 0.0 101.5 100.6 125.95 63.74 68 1.852 7693K 29K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.7 2.7 0.0 0.0 0.0 196.6 14.10 7.80 38 0.371 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 44.9 total, 44.7 interval Flush(GB): cumulative 2.707, interval 2.707 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 15.08 GB write, 343.98 MB/s write, 12.48 GB read, 284.66 MB/s read, 140.1 seconds Interval compaction: 15.08 GB write, 345.38 MB/s write, 12.48 GB read, 285.83 MB/s read, 140.1 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 45, pending-compaction-bytes-delays: 7, pending-compaction-bytes-stops: 0, total-delays: 16, total-stops: 45, interval: 61 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1388886 Average: 31.6957 StdDev: 138.83 Min: 0 Median: 1.7652 Max: 34411 Percentiles: P50: 1.77 P75: 3.37 P99: 274.71 P99.9: 1131.75 P99.99: 2521.37 ------------------------------------------------------ [ 0, 1 ] 189648 13.655% 13.655% ### ( 1, 2 ] 659721 47.500% 61.155% ########## ( 2, 3 ] 165894 11.944% 73.099% ## ( 3, 4 ] 70624 5.085% 78.184% # ( 4, 6 ] 72319 5.207% 83.391% # ( 6, 10 ] 14326 1.031% 84.422% ( 10, 15 ] 3445 0.248% 84.671% ( 15, 22 ] 371 0.027% 84.697% ( 22, 34 ] 33 0.002% 84.700% ( 34, 51 ] 18 0.001% 84.701% ( 51, 76 ] 23 0.002% 84.703% ( 76, 110 ] 13270 0.955% 85.658% ( 110, 170 ] 126769 9.127% 94.785% ## ( 170, 250 ] 57135 4.114% 98.899% # ( 250, 380 ] 7372 0.531% 99.430% ( 380, 580 ] 4304 0.310% 99.740% ( 580, 870 ] 1701 0.122% 99.862% ( 870, 1300 ] 861 0.062% 99.924% ( 1300, 1900 ] 590 0.042% 99.967% ( 1900, 2900 ] 520 0.037% 100.004% ( 2900, 4400 ] 393 0.028% 100.032% ( 4400, 6600 ] 103 0.007% 100.040% ( 6600, 9900 ] 19 0.001% 100.041% ( 9900, 14000 ] 8 0.001% 100.042% ( 14000, 22000 ] 9 0.001% 100.042% ( 22000, 33000 ] 2 0.000% 100.043% ( 33000, 50000 ] 1 0.000% 100.043% ** Level 1 read latency histogram (micros): Count: 535387 Average: 11.2480 StdDev: 185.20 Min: 0 Median: 2.4136 Max: 43795 Percentiles: P50: 2.41 P75: 3.46 P99: 155.95 P99.9: 903.85 P99.99: 1887.03 ------------------------------------------------------ [ 0, 1 ] 16864 3.150% 3.150% # ( 1, 2 ] 191065 35.687% 38.837% ####### ( 2, 3 ] 144490 26.988% 65.825% ##### ( 3, 4 ] 107508 20.080% 85.906% #### ( 4, 6 ] 47585 8.888% 94.793% ## ( 6, 10 ] 12189 2.277% 97.070% ( 10, 15 ] 4318 0.807% 97.877% ( 15, 22 ] 1037 0.194% 98.070% ( 22, 34 ] 58 0.011% 98.081% ( 34, 51 ] 44 0.008% 98.089% ( 51, 76 ] 44 0.008% 98.098% ( 76, 110 ] 596 0.111% 98.209% ( 110, 170 ] 5531 1.033% 99.242% ( 170, 250 ] 2449 0.457% 99.699% ( 250, 380 ] 366 0.068% 99.768% ( 380, 580 ] 368 0.069% 99.837% ( 580, 870 ] 317 0.059% 99.896% ( 870, 1300 ] 300 0.056% 99.952% ( 1300, 1900 ] 210 0.039% 99.991% ( 1900, 2900 ] 135 0.025% 100.016% ( 2900, 4400 ] 141 0.026% 100.043% ( 4400, 6600 ] 63 0.012% 100.054% ( 6600, 9900 ] 25 0.005% 100.059% ( 9900, 14000 ] 10 0.002% 100.061% ( 14000, 22000 ] 11 0.002% 100.063% ( 22000, 33000 ] 7 0.001% 100.064% ( 33000, 50000 ] 1 0.000% 100.064% ** Level 2 read latency histogram (micros): Count: 90052 Average: 16.4499 StdDev: 186.22 Min: 1 Median: 1.9665 Max: 13448 Percentiles: P50: 1.97 P75: 3.03 P99: 168.95 P99.9: 2705.89 P99.99: 4353.73 ------------------------------------------------------ [ 0, 1 ] 2835 3.148% 3.148% # ( 1, 2 ] 43652 48.474% 51.622% ########## ( 2, 3 ] 20779 23.074% 74.697% ##### ( 3, 4 ] 8003 8.887% 83.584% ## ( 4, 6 ] 9867 10.957% 94.541% ## ( 6, 10 ] 2578 2.863% 97.404% # ( 10, 15 ] 593 0.659% 98.062% ( 15, 22 ] 103 0.114% 98.177% ( 22, 34 ] 12 0.013% 98.190% ( 34, 51 ] 9 0.010% 98.200% ( 51, 76 ] 11 0.012% 98.212% ( 76, 110 ] 65 0.072% 98.284% ( 110, 170 ] 656 0.728% 99.013% ( 170, 250 ] 295 0.328% 99.340% ( 250, 380 ] 61 0.068% 99.408% ( 380, 580 ] 63 0.070% 99.478% ( 580, 870 ] 93 0.103% 99.581% ( 870, 1300 ] 93 0.103% 99.685% ( 1300, 1900 ] 119 0.132% 99.817% ( 1900, 2900 ] 93 0.103% 99.920% ( 2900, 4400 ] 65 0.072% 99.992% ( 4400, 6600 ] 29 0.032% 100.024% ( 6600, 9900 ] 6 0.007% 100.031% ( 9900, 14000 ] 1 0.001% 100.032% ** Level 3 read latency histogram (micros): Count: 1014228 Average: 17.2624 StdDev: 205.86 Min: 1 Median: 1.9960 Max: 28973 Percentiles: P50: 2.00 P75: 3.10 P99: 139.30 P99.9: 1822.84 P99.99: 2458.64 ------------------------------------------------------ [ 0, 1 ] 24179 2.384% 2.384% ( 1, 2 ] 484855 47.805% 50.189% ########## ( 2, 3 ] 242106 23.871% 74.060% ##### ( 3, 4 ] 97058 9.570% 83.630% ## ( 4, 6 ] 117322 11.568% 95.198% ## ( 6, 10 ] 27409 2.702% 97.900% # ( 10, 15 ] 6899 0.680% 98.580% ( 15, 22 ] 880 0.087% 98.667% ( 22, 34 ] 81 0.008% 98.675% ( 34, 51 ] 67 0.007% 98.682% ( 51, 76 ] 71 0.007% 98.689% ( 76, 110 ] 647 0.064% 98.752% ( 110, 170 ] 5148 0.508% 99.260% ( 170, 250 ] 2215 0.218% 99.478% ( 250, 380 ] 482 0.048% 99.526% ( 380, 580 ] 652 0.064% 99.590% ( 580, 870 ] 860 0.085% 99.675% ( 870, 1300 ] 1172 0.116% 99.790% ( 1300, 1900 ] 1277 0.126% 99.916% ( 1900, 2900 ] 1340 0.132% 100.049% ( 2900, 4400 ] 976 0.096% 100.145% ( 4400, 6600 ] 305 0.030% 100.175% ( 6600, 9900 ] 58 0.006% 100.181% ( 9900, 14000 ] 5 0.000% 100.181% ( 14000, 22000 ] 9 0.001% 100.182% ( 22000, 33000 ] 5 0.000% 100.182% ** Level 4 read latency histogram (micros): Count: 634031 Average: 39.7316 StdDev: 244.56 Min: 1 Median: 2.2661 Max: 31250 Percentiles: P50: 2.27 P75: 4.10 P99: 457.10 P99.9: 2607.66 P99.99: 3410.72 ------------------------------------------------------ [ 0, 1 ] 13538 2.135% 2.135% ( 1, 2 ] 267337 42.165% 44.300% ######## ( 2, 3 ] 135795 21.418% 65.718% #### ( 3, 4 ] 55577 8.766% 74.483% ## ( 4, 6 ] 64212 10.128% 84.611% ## ( 6, 10 ] 15783 2.489% 87.100% ( 10, 15 ] 3714 0.586% 87.686% ( 15, 22 ] 428 0.068% 87.753% ( 22, 34 ] 46 0.007% 87.761% ( 34, 51 ] 40 0.006% 87.767% ( 51, 76 ] 34 0.005% 87.772% ( 76, 110 ] 5354 0.844% 88.617% ( 110, 170 ] 44444 7.010% 95.627% # ( 170, 250 ] 18230 2.875% 98.502% # ( 250, 380 ] 2469 0.389% 98.891% ( 380, 580 ] 1789 0.282% 99.173% ( 580, 870 ] 1320 0.208% 99.382% ( 870, 1300 ] 1304 0.206% 99.587% ( 1300, 1900 ] 1247 0.197% 99.784% ( 1900, 2900 ] 1040 0.164% 99.948% ( 2900, 4400 ] 783 0.123% 100.071% ( 4400, 6600 ] 276 0.044% 100.115% ( 6600, 9900 ] 44 0.007% 100.122% ( 9900, 14000 ] 10 0.002% 100.123% ( 14000, 22000 ] 7 0.001% 100.125% ( 22000, 33000 ] 3 0.000% 100.125% ** Level 5 read latency histogram (micros): Count: 541289 Average: 214.2305 StdDev: 436.26 Min: 78 Median: 152.5242 Max: 40849 Percentiles: P50: 152.52 P75: 194.49 P99: 1859.46 P99.9: 6024.34 P99.99: 9819.78 ------------------------------------------------------ ( 76, 110 ] 39456 7.289% 7.289% # ( 110, 170 ] 326198 60.263% 67.552% ############ ( 170, 250 ] 131666 24.325% 91.877% ##### ( 250, 380 ] 17125 3.164% 95.041% # ( 380, 580 ] 11030 2.038% 97.078% ( 580, 870 ] 5149 0.951% 98.030% ( 870, 1300 ] 3101 0.573% 98.603% ( 1300, 1900 ] 2307 0.426% 99.029% ( 1900, 2900 ] 2040 0.377% 99.406% ( 2900, 4400 ] 1882 0.348% 99.753% ( 4400, 6600 ] 1075 0.199% 99.952% ( 6600, 9900 ] 211 0.039% 99.991% ( 9900, 14000 ] 26 0.005% 99.996% ( 14000, 22000 ] 9 0.002% 99.997% ( 22000, 33000 ] 9 0.002% 99.999% ( 33000, 50000 ] 7 0.001% 100.000% ** DB Stats ** Uptime(secs): 44.9 total, 33.5 interval Cumulative writes: 0 writes, 3000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 2.88 GB, 65.76 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:32.954 H:M:S, 73.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 29.42 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:29.026 H:M:S, 86.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 10 num-running-flushes: 1 2024/11/16-20:49:23 ... thread 0: (1000000,4000000) ops and (155943.6,78235.8) ops/second in (6.412576,51.127472) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 15/13 3.74 GB 12.6 4.7 0.0 4.7 8.4 3.7 0.0 2.2 110.5 198.9 43.32 29.77 65 0.666 4859K 13K 0.0 0.0 L1 6/6 589.56 MB 0.0 2.5 1.3 1.2 2.5 1.3 0.0 1.9 60.2 59.6 42.34 17.50 1 42.343 2589K 24K 0.0 0.0 L3 24/1 1.09 GB 1.0 3.6 1.9 1.7 3.6 1.9 1.0 1.9 108.0 107.1 34.39 18.12 28 1.228 1560K 5493 0.0 0.0 L4 174/3 9.72 GB 1.0 9.2 4.4 4.8 9.0 4.2 0.1 2.1 115.1 113.4 81.50 44.05 56 1.455 4478K 4745 0.0 0.0 L5 1149/0 94.37 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.1 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1368/23 109.49 GB 0.0 20.0 7.6 12.4 23.5 11.1 5.2 6.3 101.4 119.4 201.55 109.44 150 1.344 13M 47K 0.0 0.0 Int 0/0 0.00 KB 0.0 20.0 7.6 12.4 23.5 11.1 5.2 6.3 101.4 119.4 201.55 109.44 150 1.344 13M 47K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 20.0 7.6 12.4 19.7 7.4 0.0 0.0 112.7 111.5 181.33 98.47 97 1.869 13M 47K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 3.8 3.8 0.0 0.0 0.0 190.0 20.22 10.98 53 0.382 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 51.3 total, 51.1 interval Flush(GB): cumulative 3.753, interval 3.753 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 23.50 GB write, 468.97 MB/s write, 19.95 GB read, 398.21 MB/s read, 201.6 seconds Interval compaction: 23.50 GB write, 470.65 MB/s write, 19.95 GB read, 399.64 MB/s read, 201.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 55, pending-compaction-bytes-delays: 20, pending-compaction-bytes-stops: 0, total-delays: 29, total-stops: 55, interval: 84 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1741583 Average: 28.5525 StdDev: 141.04 Min: 0 Median: 1.7823 Max: 34411 Percentiles: P50: 1.78 P75: 3.18 P99: 283.92 P99.9: 1291.07 P99.99: 2816.80 ------------------------------------------------------ [ 0, 1 ] 209833 12.048% 12.048% ## ( 1, 2 ] 844936 48.515% 60.564% ########## ( 2, 3 ] 233807 13.425% 73.989% ### ( 3, 4 ] 99415 5.708% 79.697% # ( 4, 6 ] 105772 6.073% 85.770% # ( 6, 10 ] 21525 1.236% 87.006% ( 10, 15 ] 4895 0.281% 87.287% ( 15, 22 ] 519 0.030% 87.317% ( 22, 34 ] 41 0.002% 87.320% ( 34, 51 ] 28 0.002% 87.321% ( 51, 76 ] 45 0.003% 87.324% ( 76, 110 ] 13429 0.771% 88.095% ( 110, 170 ] 128510 7.379% 95.474% # ( 170, 250 ] 59164 3.397% 98.871% # ( 250, 380 ] 8615 0.495% 99.366% ( 380, 580 ] 5342 0.307% 99.672% ( 580, 870 ] 2524 0.145% 99.817% ( 870, 1300 ] 1472 0.085% 99.902% ( 1300, 1900 ] 907 0.052% 99.954% ( 1900, 2900 ] 687 0.039% 99.993% ( 2900, 4400 ] 585 0.034% 100.027% ( 4400, 6600 ] 165 0.009% 100.036% ( 6600, 9900 ] 27 0.002% 100.038% ( 9900, 14000 ] 8 0.000% 100.038% ( 14000, 22000 ] 9 0.001% 100.039% ( 22000, 33000 ] 2 0.000% 100.039% ( 33000, 50000 ] 1 0.000% 100.039% ** Level 1 read latency histogram (micros): Count: 671107 Average: 11.2417 StdDev: 180.42 Min: 0 Median: 2.2999 Max: 43795 Percentiles: P50: 2.30 P75: 3.38 P99: 150.78 P99.9: 1093.40 P99.99: 2599.94 ------------------------------------------------------ [ 0, 1 ] 21600 3.219% 3.219% # ( 1, 2 ] 261187 38.919% 42.137% ######## ( 2, 3 ] 175945 26.217% 68.355% ##### ( 3, 4 ] 118628 17.676% 86.031% #### ( 4, 6 ] 60361 8.994% 95.025% ## ( 6, 10 ] 15736 2.345% 97.370% ( 10, 15 ] 5173 0.771% 98.141% ( 15, 22 ] 1143 0.170% 98.311% ( 22, 34 ] 65 0.010% 98.321% ( 34, 51 ] 53 0.008% 98.329% ( 51, 76 ] 57 0.008% 98.337% ( 76, 110 ] 612 0.091% 98.428% ( 110, 170 ] 5645 0.841% 99.270% ( 170, 250 ] 2584 0.385% 99.655% ( 250, 380 ] 477 0.071% 99.726% ( 380, 580 ] 491 0.073% 99.799% ( 580, 870 ] 470 0.070% 99.869% ( 870, 1300 ] 404 0.060% 99.929% ( 1300, 1900 ] 279 0.042% 99.971% ( 1900, 2900 ] 187 0.028% 99.999% ( 2900, 4400 ] 226 0.034% 100.032% ( 4400, 6600 ] 85 0.013% 100.045% ( 6600, 9900 ] 39 0.006% 100.051% ( 9900, 14000 ] 10 0.001% 100.052% ( 14000, 22000 ] 12 0.002% 100.054% ( 22000, 33000 ] 7 0.001% 100.055% ( 33000, 50000 ] 1 0.000% 100.055% ** Level 2 read latency histogram (micros): Count: 102189 Average: 15.6489 StdDev: 177.61 Min: 1 Median: 1.9651 Max: 13448 Percentiles: P50: 1.97 P75: 3.02 P99: 165.82 P99.9: 2592.75 P99.99: 4326.84 ------------------------------------------------------ [ 0, 1 ] 3105 3.038% 3.038% # ( 1, 2 ] 49727 48.662% 51.700% ########## ( 2, 3 ] 23642 23.136% 74.836% ##### ( 3, 4 ] 9094 8.899% 83.735% ## ( 4, 6 ] 11161 10.922% 94.657% ## ( 6, 10 ] 2925 2.862% 97.519% # ( 10, 15 ] 684 0.669% 98.189% ( 15, 22 ] 115 0.113% 98.301% ( 22, 34 ] 15 0.015% 98.316% ( 34, 51 ] 9 0.009% 98.325% ( 51, 76 ] 12 0.012% 98.336% ( 76, 110 ] 65 0.064% 98.400% ( 110, 170 ] 659 0.645% 99.045% ( 170, 250 ] 299 0.293% 99.338% ( 250, 380 ] 69 0.068% 99.405% ( 380, 580 ] 79 0.077% 99.482% ( 580, 870 ] 108 0.106% 99.588% ( 870, 1300 ] 119 0.116% 99.704% ( 1300, 1900 ] 134 0.131% 99.836% ( 1900, 2900 ] 95 0.093% 99.929% ( 2900, 4400 ] 66 0.065% 99.993% ( 4400, 6600 ] 29 0.028% 100.022% ( 6600, 9900 ] 6 0.006% 100.027% ( 9900, 14000 ] 1 0.001% 100.028% ** Level 3 read latency histogram (micros): Count: 1397552 Average: 14.3230 StdDev: 182.31 Min: 1 Median: 1.9932 Max: 28973 Percentiles: P50: 1.99 P75: 3.08 P99: 90.41 P99.9: 1602.60 P99.99: 2308.65 ------------------------------------------------------ [ 0, 1 ] 33882 2.424% 2.424% ( 1, 2 ] 669423 47.900% 50.324% ########## ( 2, 3 ] 334733 23.951% 74.275% ##### ( 3, 4 ] 134619 9.632% 83.908% ## ( 4, 6 ] 163551 11.703% 95.611% ## ( 6, 10 ] 37052 2.651% 98.262% # ( 10, 15 ] 8614 0.616% 98.878% ( 15, 22 ] 1115 0.080% 98.958% ( 22, 34 ] 107 0.008% 98.966% ( 34, 51 ] 83 0.006% 98.972% ( 51, 76 ] 114 0.008% 98.980% ( 76, 110 ] 669 0.048% 99.028% ( 110, 170 ] 5236 0.375% 99.402% ( 170, 250 ] 2358 0.169% 99.571% ( 250, 380 ] 616 0.044% 99.615% ( 380, 580 ] 832 0.060% 99.675% ( 580, 870 ] 1113 0.080% 99.754% ( 870, 1300 ] 1346 0.096% 99.851% ( 1300, 1900 ] 1371 0.098% 99.949% ( 1900, 2900 ] 1415 0.101% 100.050% ( 2900, 4400 ] 1064 0.076% 100.126% ( 4400, 6600 ] 328 0.023% 100.149% ( 6600, 9900 ] 61 0.004% 100.154% ( 9900, 14000 ] 5 0.000% 100.154% ( 14000, 22000 ] 11 0.001% 100.155% ( 22000, 33000 ] 5 0.000% 100.155% ** Level 4 read latency histogram (micros): Count: 1330386 Average: 27.6129 StdDev: 215.12 Min: 1 Median: 2.1649 Max: 31250 Percentiles: P50: 2.16 P75: 3.59 P99: 307.86 P99.9: 1928.88 P99.99: 2703.86 ------------------------------------------------------ [ 0, 1 ] 26013 1.955% 1.955% ( 1, 2 ] 588200 44.213% 46.168% ######### ( 2, 3 ] 309208 23.242% 69.410% ##### ( 3, 4 ] 126451 9.505% 78.915% ## ( 4, 6 ] 149422 11.231% 90.146% ## ( 6, 10 ] 36334 2.731% 92.877% # ( 10, 15 ] 8918 0.670% 93.548% ( 15, 22 ] 1000 0.075% 93.623% ( 22, 34 ] 88 0.007% 93.630% ( 34, 51 ] 95 0.007% 93.637% ( 51, 76 ] 148 0.011% 93.648% ( 76, 110 ] 5422 0.408% 94.055% ( 110, 170 ] 45158 3.394% 97.450% # ( 170, 250 ] 19200 1.443% 98.893% ( 250, 380 ] 3202 0.241% 99.134% ( 380, 580 ] 2852 0.214% 99.348% ( 580, 870 ] 2688 0.202% 99.550% ( 870, 1300 ] 2597 0.195% 99.745% ( 1300, 1900 ] 2015 0.151% 99.897% ( 1900, 2900 ] 1545 0.116% 100.013% ( 2900, 4400 ] 1427 0.107% 100.120% ( 4400, 6600 ] 480 0.036% 100.156% ( 6600, 9900 ] 83 0.006% 100.162% ( 9900, 14000 ] 11 0.001% 100.163% ( 14000, 22000 ] 9 0.001% 100.164% ( 22000, 33000 ] 3 0.000% 100.164% ** Level 5 read latency histogram (micros): Count: 569002 Average: 235.5163 StdDev: 476.83 Min: 78 Median: 154.3472 Max: 40849 Percentiles: P50: 154.35 P75: 202.53 P99: 2364.33 P99.9: 6174.56 P99.99: 9836.34 ------------------------------------------------------ ( 76, 110 ] 39837 7.001% 7.001% # ( 110, 170 ] 331021 58.176% 65.177% ############ ( 170, 250 ] 137445 24.155% 89.332% ##### ( 250, 380 ] 21001 3.691% 93.023% # ( 380, 580 ] 14796 2.600% 95.624% # ( 580, 870 ] 8625 1.516% 97.139% ( 870, 1300 ] 5768 1.014% 98.153% ( 1300, 1900 ] 3582 0.630% 98.783% ( 1900, 2900 ] 2664 0.468% 99.251% ( 2900, 4400 ] 2597 0.456% 99.707% ( 4400, 6600 ] 1360 0.239% 99.946% ( 6600, 9900 ] 254 0.045% 99.991% ( 9900, 14000 ] 27 0.005% 99.996% ( 14000, 22000 ] 11 0.002% 99.998% ( 22000, 33000 ] 9 0.002% 99.999% ( 33000, 50000 ] 7 0.001% 100.000% ** DB Stats ** Uptime(secs): 51.3 total, 6.4 interval Cumulative writes: 0 writes, 4000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 3.84 GB, 76.72 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:35.635 H:M:S, 69.4 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 153.48 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.681 H:M:S, 41.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 3 num-running-flushes: 1 2024/11/16-20:49:27 ... thread 0: (1000000,5000000) ops and (285477.1,91524.2) ops/second in (3.502908,54.630380) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/13 4.72 GB 113.3 5.7 0.0 5.7 10.4 4.7 0.0 2.2 114.8 210.7 50.46 36.33 85 0.594 5883K 14K 0.0 0.0 L1 6/6 589.56 MB 0.0 2.5 1.3 1.2 2.5 1.3 0.0 1.9 60.2 59.6 42.34 17.50 1 42.343 2589K 24K 0.0 0.0 L3 23/0 983.56 MB 1.0 3.6 1.9 1.7 3.6 1.9 1.0 1.9 108.0 107.1 34.39 18.12 28 1.228 1560K 5493 0.0 0.0 L4 171/0 9.54 GB 1.0 9.4 4.5 4.9 9.2 4.4 0.1 2.0 116.3 114.5 82.70 45.12 57 1.451 4714K 5481 0.0 0.0 L5 1153/0 94.67 GB 1.0 0.0 0.0 0.0 0.0 0.0 4.4 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1372/19 110.47 GB 0.0 21.2 7.7 13.4 25.7 12.2 5.5 5.4 103.3 125.4 209.89 117.07 171 1.227 14M 49K 0.0 0.0 Int 0/0 0.00 KB 0.0 21.2 7.7 13.4 25.7 12.2 5.5 5.4 103.3 125.4 209.89 117.07 171 1.227 14M 49K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 21.2 7.7 13.4 21.0 7.5 0.0 0.0 116.3 115.1 186.42 103.35 102 1.828 14M 49K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 4.7 4.7 0.0 0.0 0.0 206.7 23.47 13.72 69 0.340 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 54.8 total, 54.6 interval Flush(GB): cumulative 4.737, interval 4.737 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 25.69 GB write, 479.99 MB/s write, 21.17 GB read, 395.40 MB/s read, 209.9 seconds Interval compaction: 25.69 GB write, 481.59 MB/s write, 21.17 GB read, 396.72 MB/s read, 209.9 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 10, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 10, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 55, pending-compaction-bytes-delays: 20, pending-compaction-bytes-stops: 0, total-delays: 30, total-stops: 55, interval: 85 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 2029350 Average: 26.8308 StdDev: 133.33 Min: 0 Median: 1.7331 Max: 34411 Percentiles: P50: 1.73 P75: 2.92 P99: 292.26 P99.9: 1176.62 P99.99: 2539.30 ------------------------------------------------------ [ 0, 1 ] 272335 13.420% 13.420% ### ( 1, 2 ] 1012558 49.896% 63.315% ########## ( 2, 3 ] 257952 12.711% 76.027% ### ( 3, 4 ] 107440 5.294% 81.321% # ( 4, 6 ] 111812 5.510% 86.831% # ( 6, 10 ] 22615 1.114% 87.945% ( 10, 15 ] 5357 0.264% 88.209% ( 15, 22 ] 554 0.027% 88.236% ( 22, 34 ] 48 0.002% 88.239% ( 34, 51 ] 29 0.001% 88.240% ( 51, 76 ] 50 0.002% 88.243% ( 76, 110 ] 14048 0.692% 88.935% ( 110, 170 ] 135434 6.674% 95.609% # ( 170, 250 ] 65313 3.218% 98.827% # ( 250, 380 ] 10805 0.532% 99.359% ( 380, 580 ] 6840 0.337% 99.696% ( 580, 870 ] 3010 0.148% 99.845% ( 870, 1300 ] 1573 0.078% 99.922% ( 1300, 1900 ] 925 0.046% 99.968% ( 1900, 2900 ] 704 0.035% 100.003% ( 2900, 4400 ] 591 0.029% 100.032% ( 4400, 6600 ] 165 0.008% 100.040% ( 6600, 9900 ] 27 0.001% 100.041% ( 9900, 14000 ] 8 0.000% 100.042% ( 14000, 22000 ] 9 0.000% 100.042% ( 22000, 33000 ] 2 0.000% 100.042% ( 33000, 50000 ] 1 0.000% 100.042% ** Level 1 read latency histogram (micros): Count: 683259 Average: 11.3182 StdDev: 179.01 Min: 0 Median: 2.2876 Max: 43795 Percentiles: P50: 2.29 P75: 3.37 P99: 154.36 P99.9: 1083.15 P99.99: 2593.44 ------------------------------------------------------ [ 0, 1 ] 22181 3.246% 3.246% # ( 1, 2 ] 268268 39.263% 42.509% ######## ( 2, 3 ] 177983 26.049% 68.558% ##### ( 3, 4 ] 119316 17.463% 86.021% ### ( 4, 6 ] 61016 8.930% 94.951% ## ( 6, 10 ] 16077 2.353% 97.304% ( 10, 15 ] 5223 0.764% 98.069% ( 15, 22 ] 1161 0.170% 98.239% ( 22, 34 ] 67 0.010% 98.249% ( 34, 51 ] 54 0.008% 98.256% ( 51, 76 ] 57 0.008% 98.265% ( 76, 110 ] 635 0.093% 98.358% ( 110, 170 ] 5935 0.869% 99.226% ( 170, 250 ] 2806 0.411% 99.637% ( 250, 380 ] 557 0.082% 99.719% ( 380, 580 ] 546 0.080% 99.798% ( 580, 870 ] 491 0.072% 99.870% ( 870, 1300 ] 409 0.060% 99.930% ( 1300, 1900 ] 279 0.041% 99.971% ( 1900, 2900 ] 187 0.027% 99.998% ( 2900, 4400 ] 226 0.033% 100.031% ( 4400, 6600 ] 85 0.012% 100.044% ( 6600, 9900 ] 39 0.006% 100.050% ( 9900, 14000 ] 10 0.001% 100.051% ( 14000, 22000 ] 12 0.002% 100.053% ( 22000, 33000 ] 7 0.001% 100.054% ( 33000, 50000 ] 1 0.000% 100.054% ** Level 2 read latency histogram (micros): Count: 102189 Average: 15.6489 StdDev: 177.61 Min: 1 Median: 1.9651 Max: 13448 Percentiles: P50: 1.97 P75: 3.02 P99: 165.82 P99.9: 2592.75 P99.99: 4326.84 ------------------------------------------------------ [ 0, 1 ] 3105 3.038% 3.038% # ( 1, 2 ] 49727 48.662% 51.700% ########## ( 2, 3 ] 23642 23.136% 74.836% ##### ( 3, 4 ] 9094 8.899% 83.735% ## ( 4, 6 ] 11161 10.922% 94.657% ## ( 6, 10 ] 2925 2.862% 97.519% # ( 10, 15 ] 684 0.669% 98.189% ( 15, 22 ] 115 0.113% 98.301% ( 22, 34 ] 15 0.015% 98.316% ( 34, 51 ] 9 0.009% 98.325% ( 51, 76 ] 12 0.012% 98.336% ( 76, 110 ] 65 0.064% 98.400% ( 110, 170 ] 659 0.645% 99.045% ( 170, 250 ] 299 0.293% 99.338% ( 250, 380 ] 69 0.068% 99.405% ( 380, 580 ] 79 0.077% 99.482% ( 580, 870 ] 108 0.106% 99.588% ( 870, 1300 ] 119 0.116% 99.704% ( 1300, 1900 ] 134 0.131% 99.836% ( 1900, 2900 ] 95 0.093% 99.929% ( 2900, 4400 ] 66 0.065% 99.993% ( 4400, 6600 ] 29 0.028% 100.022% ( 6600, 9900 ] 6 0.006% 100.027% ( 9900, 14000 ] 1 0.001% 100.028% ** Level 3 read latency histogram (micros): Count: 1397972 Average: 14.3603 StdDev: 182.32 Min: 1 Median: 1.9935 Max: 28973 Percentiles: P50: 1.99 P75: 3.08 P99: 102.40 P99.9: 1602.42 P99.99: 2308.62 ------------------------------------------------------ [ 0, 1 ] 33882 2.424% 2.424% ( 1, 2 ] 669430 47.886% 50.309% ########## ( 2, 3 ] 334784 23.948% 74.257% ##### ( 3, 4 ] 134658 9.632% 83.890% ## ( 4, 6 ] 163613 11.704% 95.593% ## ( 6, 10 ] 37062 2.651% 98.244% # ( 10, 15 ] 8617 0.616% 98.861% ( 15, 22 ] 1115 0.080% 98.941% ( 22, 34 ] 107 0.008% 98.948% ( 34, 51 ] 83 0.006% 98.954% ( 51, 76 ] 114 0.008% 98.962% ( 76, 110 ] 679 0.049% 99.011% ( 110, 170 ] 5319 0.380% 99.391% ( 170, 250 ] 2457 0.176% 99.567% ( 250, 380 ] 641 0.046% 99.613% ( 380, 580 ] 851 0.061% 99.674% ( 580, 870 ] 1123 0.080% 99.754% ( 870, 1300 ] 1348 0.096% 99.851% ( 1300, 1900 ] 1371 0.098% 99.949% ( 1900, 2900 ] 1415 0.101% 100.050% ( 2900, 4400 ] 1064 0.076% 100.126% ( 4400, 6600 ] 328 0.023% 100.149% ( 6600, 9900 ] 61 0.004% 100.154% ( 9900, 14000 ] 5 0.000% 100.154% ( 14000, 22000 ] 11 0.001% 100.155% ( 22000, 33000 ] 5 0.000% 100.155% ** Level 4 read latency histogram (micros): Count: 1352937 Average: 28.1926 StdDev: 214.01 Min: 1 Median: 2.1630 Max: 31250 Percentiles: P50: 2.16 P75: 3.60 P99: 337.37 P99.9: 1916.17 P99.99: 2701.75 ------------------------------------------------------ [ 0, 1 ] 26662 1.971% 1.971% ( 1, 2 ] 598846 44.263% 46.233% ######### ( 2, 3 ] 312602 23.105% 69.339% ##### ( 3, 4 ] 127175 9.400% 78.739% ## ( 4, 6 ] 150287 11.108% 89.847% ## ( 6, 10 ] 36703 2.713% 92.560% # ( 10, 15 ] 8964 0.663% 93.222% ( 15, 22 ] 1007 0.074% 93.297% ( 22, 34 ] 88 0.007% 93.303% ( 34, 51 ] 96 0.007% 93.310% ( 51, 76 ] 148 0.011% 93.321% ( 76, 110 ] 5589 0.413% 93.734% ( 110, 170 ] 47428 3.506% 97.240% # ( 170, 250 ] 21178 1.565% 98.805% ( 250, 380 ] 3920 0.290% 99.095% ( 380, 580 ] 3361 0.248% 99.343% ( 580, 870 ] 2859 0.211% 99.555% ( 870, 1300 ] 2626 0.194% 99.749% ( 1300, 1900 ] 2020 0.149% 99.898% ( 1900, 2900 ] 1550 0.115% 100.013% ( 2900, 4400 ] 1427 0.105% 100.118% ( 4400, 6600 ] 480 0.035% 100.154% ( 6600, 9900 ] 83 0.006% 100.160% ( 9900, 14000 ] 11 0.001% 100.161% ( 14000, 22000 ] 9 0.001% 100.161% ( 22000, 33000 ] 3 0.000% 100.162% ** Level 5 read latency histogram (micros): Count: 608304 Average: 235.0790 StdDev: 462.95 Min: 78 Median: 155.5164 Max: 40849 Percentiles: P50: 155.52 P75: 206.30 P99: 2227.12 P99.9: 6110.98 P99.99: 9785.27 ------------------------------------------------------ ( 76, 110 ] 41178 6.769% 6.769% # ( 110, 170 ] 346654 56.987% 63.756% ########### ( 170, 250 ] 150716 24.776% 88.533% ##### ( 250, 380 ] 25508 4.193% 92.726% # ( 380, 580 ] 17974 2.955% 95.681% # ( 580, 870 ] 9718 1.598% 97.278% ( 870, 1300 ] 5979 0.983% 98.261% ( 1300, 1900 ] 3614 0.594% 98.855% ( 1900, 2900 ] 2690 0.442% 99.298% ( 2900, 4400 ] 2607 0.429% 99.726% ( 4400, 6600 ] 1360 0.224% 99.950% ( 6600, 9900 ] 254 0.042% 99.991% ( 9900, 14000 ] 27 0.004% 99.996% ( 14000, 22000 ] 11 0.002% 99.998% ( 22000, 33000 ] 9 0.001% 99.999% ( 33000, 50000 ] 7 0.001% 100.000% ** DB Stats ** Uptime(secs): 54.8 total, 3.5 interval Cumulative writes: 0 writes, 5000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 4.81 GB, 89.77 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:35.726 H:M:S, 65.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 280.96 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.091 H:M:S, 2.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 292.337 micros/op 13660 ops/sec 60.175 seconds 821996 operations; 13.2 MB/s (209999 of 209999 found) Microseconds per read: Count: 821996 Average: 292.3394 StdDev: 936.79 Min: 2 Median: 204.1172 Max: 89751 Percentiles: P50: 204.12 P75: 248.03 P99: 1670.48 P99.9: 16847.31 P99.99: 35563.18 ------------------------------------------------------ ( 1, 2 ] 6 0.001% 0.001% ( 2, 3 ] 31 0.004% 0.005% ( 3, 4 ] 55 0.007% 0.011% ( 4, 6 ] 75 0.009% 0.020% ( 6, 10 ] 263 0.032% 0.052% ( 10, 15 ] 3075 0.374% 0.426% ( 15, 22 ] 4167 0.507% 0.933% ( 22, 34 ] 7245 0.881% 1.815% ( 34, 51 ] 3782 0.460% 2.275% ( 51, 76 ] 559 0.068% 2.343% ( 76, 110 ] 2340 0.285% 2.628% ( 110, 170 ] 229759 27.951% 30.579% ###### ( 170, 250 ] 374335 45.540% 76.119% ######### ( 250, 380 ] 107474 13.075% 89.193% ### ( 380, 580 ] 51070 6.213% 95.406% # ( 580, 870 ] 18972 2.308% 97.714% ( 870, 1300 ] 8163 0.993% 98.707% ( 1300, 1900 ] 3895 0.474% 99.181% ( 1900, 2900 ] 2348 0.286% 99.467% ( 2900, 4400 ] 1960 0.238% 99.705% ( 4400, 6600 ] 881 0.107% 99.813% ( 6600, 9900 ] 259 0.032% 99.844% ( 9900, 14000 ] 213 0.026% 99.870% ( 14000, 22000 ] 694 0.084% 99.954% ( 22000, 33000 ] 283 0.034% 99.989% ( 33000, 50000 ] 65 0.008% 99.997% ( 50000, 75000 ] 25 0.003% 100.000% ( 75000, 110000 ] 2 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 15505648 rocksdb.bloom.filter.full.positive COUNT : 972281 rocksdb.bloom.filter.full.true.positive COUNT : 821624 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 372 rocksdb.memtable.miss COUNT : 821624 rocksdb.l0.hit COUNT : 32189 rocksdb.l1.hit COUNT : 8626 rocksdb.l2andup.hit COUNT : 780809 rocksdb.compaction.key.drop.new COUNT : 224132 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 5388523 rocksdb.number.keys.read COUNT : 821996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 5560955736 rocksdb.bytes.read COUNT : 821996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1703 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 39627336 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 24476960487 rocksdb.compact.write.bytes COUNT : 24237170453 rocksdb.flush.write.bytes COUNT : 5548864751 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 29 rocksdb.number.superversion_acquires COUNT : 1127 rocksdb.number.superversion_releases COUNT : 1093 rocksdb.number.superversion_cleanups COUNT : 284 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 290 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 5504114688 rocksdb.memtable.garbage.bytes.at.flush COUNT : 2137088 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 28168981658 rocksdb.non.last.level.read.count COUNT : 6662186 rocksdb.block.checksum.compute.count COUNT : 6911295 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 203.209053 P95 : 566.171465 P99 : 1669.331961 P100 : 89748.000000 COUNT : 821996 SUM : 239551234 rocksdb.db.write.micros P50 : 1.534122 P95 : 4.410919 P99 : 9.321533 P100 : 796472.000000 COUNT : 5388523 SUM : 53165525 rocksdb.compaction.times.micros P50 : 1303409.090909 P95 : 2712500.000000 P99 : 9635000.000000 P100 : 42342773.000000 COUNT : 105 SUM : 189358638 rocksdb.compaction.times.cpu_micros P50 : 777833.333333 P95 : 1672500.000000 P99 : 6390000.000000 P100 : 17501368.000000 COUNT : 105 SUM : 106226161 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 891.500000 P95 : 78500.000000 P99 : 186949.000000 P100 : 186949.000000 COUNT : 76 SUM : 794551 rocksdb.compaction.outfile.sync.micros P50 : 21172.413793 P95 : 153142.857143 P99 : 383800.000000 P100 : 871478.000000 COUNT : 298 SUM : 13528110 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 468.888889 P95 : 4840.000000 P99 : 9240.000000 P100 : 31971.000000 COUNT : 200 SUM : 278134 rocksdb.table.open.io.micros P50 : 11194.134078 P95 : 21525.471698 P99 : 30753.148148 P100 : 41661.000000 COUNT : 1703 SUM : 18359915 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.930598 P95 : 10.780333 P99 : 220.041394 P100 : 43800.000000 COUNT : 5927609 SUM : 100709806 rocksdb.read.block.get.micros P50 : 155.072431 P95 : 460.580152 P99 : 1611.621115 P100 : 40855.000000 COUNT : 982248 SUM : 205504629 rocksdb.write.raw.block.micros P50 : 0.533252 P95 : 1.293443 P99 : 3.886411 P100 : 58919.000000 COUNT : 7216381 SUM : 23166033 rocksdb.numfiles.in.singlecompaction P50 : 2.637931 P95 : 4.575000 P99 : 13.150000 P100 : 18.000000 COUNT : 137 SUM : 447 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.567862 P95 : 1920.119438 P99 : 2711.464961 P100 : 796289.000000 COUNT : 187173 SUM : 39644496 rocksdb.sst.read.micros P50 : 2.054012 P95 : 194.448477 P99 : 511.301713 P100 : 43795.000000 COUNT : 6662628 SUM : 288924437 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 821996 SUM : 821996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 5388523 SUM : 5560955736 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 227358.490566 P95 : 1015428.571429 P99 : 1032066.000000 P100 : 1032066.000000 COUNT : 76 SUM : 24912182 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 333 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0