[2024-11-20 17:39:29.748190] Starting SPDK v24.09.1-pre git sha1 b18e1bd62 / DPDK 22.11.4 initialization... [2024-11-20 17:39:29.748490] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3174725 ] Set seed to 1732120769741003 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-20 17:39:29.861144] app.c: 917:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-20 17:39:29.892592] app.c: 610:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-20 17:39:29.892638] app.c: 611:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3174725' to capture a snapshot of events at runtime. [2024-11-20 17:39:29.892650] app.c: 616:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-20 17:39:29.892661] app.c: 617:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-20 17:39:29.892670] app.c: 618:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3174725 for offline analysis/debug. [2024-11-20 17:39:29.892762] reactor.c: 990:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Wed Nov 20 17:39:31 2024 CPU: 144 * Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz CPUCache: 55296 KB 2024/11/20-17:39:36 ... thread 0: (1000000,1000000) ops and (184836.1,184836.1) ops/second in (5.410199,5.410199) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/18 3.24 GB 6.3 0.7 0.0 0.7 1.6 0.9 0.0 1.9 140.1 303.3 5.40 4.85 17 0.318 767K 759 0.0 0.0 L1 15/15 1.39 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 945.58 MB 0.9 1.0 0.5 0.5 1.0 0.5 0.0 2.0 213.9 212.6 4.96 4.49 4 1.241 169K 0 0.0 0.0 L4 168/0 9.47 GB 1.0 0.8 0.3 0.5 0.8 0.3 0.2 2.4 217.2 211.7 3.89 3.40 6 0.648 343K 0 0.0 0.0 L5 1244/0 94.24 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1464/33 109.27 GB 0.0 2.6 0.8 1.8 3.4 1.7 0.8 4.0 186.8 246.7 14.25 12.73 27 0.528 1280K 759 0.0 0.0 Int 0/0 0.00 KB 0.0 2.6 0.8 1.8 3.4 1.7 0.8 4.0 186.8 246.7 14.25 12.73 27 0.528 1280K 759 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.6 0.8 1.8 2.6 0.8 0.0 0.0 234.5 232.1 11.35 10.31 13 0.873 1280K 759 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 304.1 2.90 2.43 14 0.207 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.6 total, 5.4 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 3.43 GB write, 628.88 MB/s write, 2.60 GB read, 476.14 MB/s read, 14.2 seconds Interval compaction: 3.43 GB write, 649.54 MB/s write, 2.60 GB read, 491.78 MB/s read, 14.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 9, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 9, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 9, total-stops: 0, interval: 9 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 283294 Average: 23.1718 StdDev: 97.56 Min: 0 Median: 1.4275 Max: 4352 Percentiles: P50: 1.43 P75: 1.98 P99: 425.26 P99.9: 807.70 P99.99: 1047.25 ------------------------------------------------------ [ 0, 1 ] 86415 30.504% 30.504% ###### ( 1, 2 ] 129186 45.601% 76.105% ######### ( 2, 3 ] 24715 8.724% 84.829% ## ( 3, 4 ] 11720 4.137% 88.966% # ( 4, 6 ] 7395 2.610% 91.577% # ( 6, 10 ] 2478 0.875% 92.451% ( 10, 15 ] 1045 0.369% 92.820% ( 15, 22 ] 184 0.065% 92.885% ( 22, 34 ] 98 0.035% 92.920% ( 34, 51 ] 60 0.021% 92.941% ( 51, 76 ] 16 0.006% 92.947% ( 76, 110 ] 505 0.178% 93.125% ( 110, 170 ] 3904 1.378% 94.503% ( 170, 250 ] 4504 1.590% 96.093% ( 250, 380 ] 7682 2.712% 98.804% # ( 380, 580 ] 2453 0.866% 99.670% ( 580, 870 ] 830 0.293% 99.963% ( 870, 1300 ] 186 0.066% 100.029% ( 1300, 1900 ] 25 0.009% 100.038% ( 1900, 2900 ] 33 0.012% 100.049% ( 2900, 4400 ] 37 0.013% 100.062% ** Level 1 read latency histogram (micros): Count: 819 Average: 324.3175 StdDev: 319.72 Min: 90 Median: 266.8914 Max: 3540 Percentiles: P50: 266.89 P75: 354.45 P99: 2158.57 P99.9: 3540.00 P99.99: 3540.00 ------------------------------------------------------ ( 76, 110 ] 13 1.587% 1.587% ( 110, 170 ] 157 19.170% 20.757% #### ( 170, 250 ] 200 24.420% 45.177% ##### ( 250, 380 ] 304 37.118% 82.295% ####### ( 380, 580 ] 92 11.233% 93.529% ## ( 580, 870 ] 24 2.930% 96.459% # ( 870, 1300 ] 16 1.954% 98.413% ( 1300, 1900 ] 3 0.366% 98.779% ( 1900, 2900 ] 7 0.855% 99.634% ( 2900, 4400 ] 3 0.366% 100.000% ** Level 2 read latency histogram (micros): Count: 32737 Average: 3.2411 StdDev: 22.11 Min: 1 Median: 1.4751 Max: 976 Percentiles: P50: 1.48 P75: 1.92 P99: 7.57 P99.9: 423.19 P99.99: 796.74 ------------------------------------------------------ [ 0, 1 ] 7565 23.108% 23.108% ##### ( 1, 2 ] 18530 56.603% 79.711% ########### ( 2, 3 ] 4082 12.469% 92.180% ## ( 3, 4 ] 1133 3.461% 95.641% # ( 4, 6 ] 993 3.033% 98.674% # ( 6, 10 ] 271 0.828% 99.502% ( 10, 15 ] 35 0.107% 99.609% ( 15, 22 ] 14 0.043% 99.652% ( 22, 34 ] 10 0.031% 99.682% ( 34, 51 ] 2 0.006% 99.688% ( 51, 76 ] 2 0.006% 99.695% ( 110, 170 ] 8 0.024% 99.719% ( 170, 250 ] 25 0.076% 99.795% ( 250, 380 ] 28 0.086% 99.881% ( 380, 580 ] 29 0.089% 99.969% ( 580, 870 ] 9 0.027% 99.997% ( 870, 1300 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 203085 Average: 3.4265 StdDev: 24.64 Min: 0 Median: 1.5135 Max: 2003 Percentiles: P50: 1.51 P75: 1.98 P99: 7.05 P99.9: 377.09 P99.99: 828.47 ------------------------------------------------------ [ 0, 1 ] 45382 22.346% 22.346% #### ( 1, 2 ] 109367 53.853% 76.199% ########### ( 2, 3 ] 25327 12.471% 88.670% ## ( 3, 4 ] 12378 6.095% 94.765% # ( 4, 6 ] 8238 4.056% 98.822% # ( 6, 10 ] 1375 0.677% 99.499% ( 10, 15 ] 195 0.096% 99.595% ( 15, 22 ] 39 0.019% 99.614% ( 22, 34 ] 52 0.026% 99.640% ( 34, 51 ] 42 0.021% 99.660% ( 51, 76 ] 19 0.009% 99.670% ( 76, 110 ] 18 0.009% 99.678% ( 110, 170 ] 92 0.045% 99.724% ( 170, 250 ] 136 0.067% 99.791% ( 250, 380 ] 227 0.112% 99.903% ( 380, 580 ] 122 0.060% 99.963% ( 580, 870 ] 65 0.032% 99.995% ( 870, 1300 ] 19 0.009% 100.004% ( 1300, 1900 ] 4 0.002% 100.006% ( 1900, 2900 ] 1 0.000% 100.006% ** Level 4 read latency histogram (micros): Count: 133576 Average: 15.8461 StdDev: 80.47 Min: 1 Median: 1.7131 Max: 4421 Percentiles: P50: 1.71 P75: 2.77 P99: 368.85 P99.9: 802.52 P99.99: 1041.34 ------------------------------------------------------ [ 0, 1 ] 20540 15.377% 15.377% ### ( 1, 2 ] 64855 48.553% 63.930% ########## ( 2, 3 ] 19153 14.339% 78.269% ### ( 3, 4 ] 12170 9.111% 87.379% ## ( 4, 6 ] 9048 6.774% 94.153% # ( 6, 10 ] 1944 1.455% 95.608% ( 10, 15 ] 214 0.160% 95.769% ( 15, 22 ] 35 0.026% 95.795% ( 22, 34 ] 41 0.031% 95.826% ( 34, 51 ] 47 0.035% 95.861% ( 51, 76 ] 14 0.010% 95.871% ( 76, 110 ] 100 0.075% 95.946% ( 110, 170 ] 865 0.648% 96.594% ( 170, 250 ] 1362 1.020% 97.613% ( 250, 380 ] 2026 1.517% 99.130% ( 380, 580 ] 766 0.573% 99.704% ( 580, 870 ] 342 0.256% 99.960% ( 870, 1300 ] 102 0.076% 100.036% ( 1300, 1900 ] 13 0.010% 100.046% ( 1900, 2900 ] 6 0.004% 100.050% ( 2900, 4400 ] 8 0.006% 100.056% ( 4400, 6600 ] 1 0.001% 100.057% ** Level 5 read latency histogram (micros): Count: 41341 Average: 314.2317 StdDev: 229.54 Min: 84 Median: 274.5487 Max: 5401 Percentiles: P50: 274.55 P75: 363.78 P99: 1206.32 P99.9: 2839.05 P99.99: 4244.97 ------------------------------------------------------ ( 76, 110 ] 891 2.155% 2.155% ( 110, 170 ] 7403 17.907% 20.062% #### ( 170, 250 ] 9533 23.059% 43.122% ##### ( 250, 380 ] 15058 36.424% 79.546% ####### ( 380, 580 ] 5164 12.491% 92.037% ## ( 580, 870 ] 2256 5.457% 97.494% # ( 870, 1300 ] 796 1.925% 99.419% ( 1300, 1900 ] 178 0.431% 99.850% ( 1900, 2900 ] 22 0.053% 99.903% ( 2900, 4400 ] 40 0.097% 100.000% ( 4400, 6600 ] 4 0.010% 100.010% ** DB Stats ** Uptime(secs): 5.6 total, 5.4 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 176.04 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.673 H:M:S, 29.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 181.79 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.673 H:M:S, 30.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/20-17:39:50 ... thread 0: (1000000,2000000) ops and (72906.5,104567.5) ops/second in (13.716194,19.126393) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/14 4.22 GB 188.3 3.3 0.0 3.3 5.2 1.8 0.0 2.8 198.1 307.8 17.17 16.19 39 0.440 3452K 6585 0.0 0.0 L1 15/15 1.39 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 945.58 MB 0.9 1.0 0.5 0.5 1.0 0.5 0.0 2.0 213.9 212.6 4.96 4.49 4 1.241 169K 0 0.0 0.0 L4 168/0 9.47 GB 1.0 0.8 0.3 0.5 0.8 0.3 0.2 2.4 217.2 211.7 3.89 3.40 6 0.648 343K 0 0.0 0.0 L5 1244/0 94.24 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1462/29 110.25 GB 0.0 5.2 0.8 4.3 7.0 2.6 0.8 3.8 203.9 275.3 26.02 24.08 49 0.531 3965K 6585 0.0 0.0 Int 0/0 0.00 KB 0.0 5.2 0.8 4.3 7.0 2.6 0.8 3.8 203.9 275.3 26.02 24.08 49 0.531 3965K 6585 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.2 0.8 4.3 5.1 0.8 0.0 0.0 261.3 259.6 20.31 19.15 19 1.069 3965K 6585 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 330.9 5.71 4.93 30 0.190 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 19.3 total, 19.1 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.99 GB write, 370.98 MB/s write, 5.18 GB read, 274.84 MB/s read, 26.0 seconds Interval compaction: 6.99 GB write, 374.43 MB/s write, 5.18 GB read, 277.40 MB/s read, 26.0 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 22, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 22, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 22, total-stops: 0, interval: 22 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1010359 Average: 24.9120 StdDev: 82.13 Min: 0 Median: 1.3745 Max: 4352 Percentiles: P50: 1.37 P75: 1.86 P99: 364.11 P99.9: 568.45 P99.99: 757.56 ------------------------------------------------------ [ 0, 1 ] 311878 30.868% 30.868% ###### ( 1, 2 ] 516207 51.091% 81.959% ########## ( 2, 3 ] 46244 4.577% 86.536% # ( 3, 4 ] 20022 1.982% 88.518% ( 4, 6 ] 11874 1.175% 89.693% ( 6, 10 ] 3853 0.381% 90.075% ( 10, 15 ] 4191 0.415% 90.490% ( 15, 22 ] 310 0.031% 90.520% ( 22, 34 ] 164 0.016% 90.536% ( 34, 51 ] 69 0.007% 90.543% ( 51, 76 ] 18 0.002% 90.545% ( 76, 110 ] 3820 0.378% 90.923% ( 110, 170 ] 26715 2.644% 93.567% # ( 170, 250 ] 20440 2.023% 95.590% ( 250, 380 ] 39248 3.885% 99.475% # ( 380, 580 ] 4559 0.451% 99.926% ( 580, 870 ] 1055 0.104% 100.030% ( 870, 1300 ] 197 0.019% 100.050% ( 1300, 1900 ] 29 0.003% 100.053% ( 1900, 2900 ] 35 0.003% 100.056% ( 2900, 4400 ] 37 0.004% 100.060% ** Level 1 read latency histogram (micros): Count: 3878 Average: 251.6764 StdDev: 170.32 Min: 84 Median: 241.2456 Max: 3540 Percentiles: P50: 241.25 P75: 321.67 P99: 786.58 P99.9: 2774.57 P99.99: 3540.00 ------------------------------------------------------ ( 76, 110 ] 148 3.816% 3.816% # ( 110, 170 ] 1026 26.457% 30.273% ##### ( 170, 250 ] 859 22.151% 52.424% #### ( 250, 380 ] 1588 40.949% 93.373% ######## ( 380, 580 ] 194 5.003% 98.375% # ( 580, 870 ] 34 0.877% 99.252% ( 870, 1300 ] 16 0.413% 99.665% ( 1300, 1900 ] 3 0.077% 99.742% ( 1900, 2900 ] 7 0.181% 99.923% ( 2900, 4400 ] 3 0.077% 100.000% ** Level 2 read latency histogram (micros): Count: 32737 Average: 3.2411 StdDev: 22.11 Min: 1 Median: 1.4751 Max: 976 Percentiles: P50: 1.48 P75: 1.92 P99: 7.57 P99.9: 423.19 P99.99: 796.74 ------------------------------------------------------ [ 0, 1 ] 7565 23.108% 23.108% ##### ( 1, 2 ] 18530 56.603% 79.711% ########### ( 2, 3 ] 4082 12.469% 92.180% ## ( 3, 4 ] 1133 3.461% 95.641% # ( 4, 6 ] 993 3.033% 98.674% # ( 6, 10 ] 271 0.828% 99.502% ( 10, 15 ] 35 0.107% 99.609% ( 15, 22 ] 14 0.043% 99.652% ( 22, 34 ] 10 0.031% 99.682% ( 34, 51 ] 2 0.006% 99.688% ( 51, 76 ] 2 0.006% 99.695% ( 110, 170 ] 8 0.024% 99.719% ( 170, 250 ] 25 0.076% 99.795% ( 250, 380 ] 28 0.086% 99.881% ( 380, 580 ] 29 0.089% 99.969% ( 580, 870 ] 9 0.027% 99.997% ( 870, 1300 ] 1 0.003% 100.000% ** Level 3 read latency histogram (micros): Count: 205329 Average: 5.1145 StdDev: 32.13 Min: 0 Median: 1.5234 Max: 2003 Percentiles: P50: 1.52 P75: 1.99 P99: 114.09 P99.9: 427.51 P99.99: 828.74 ------------------------------------------------------ [ 0, 1 ] 45394 22.108% 22.108% #### ( 1, 2 ] 109414 53.287% 75.395% ########### ( 2, 3 ] 25368 12.355% 87.750% ## ( 3, 4 ] 12507 6.091% 93.841% # ( 4, 6 ] 8426 4.104% 97.945% # ( 6, 10 ] 1410 0.687% 98.631% ( 10, 15 ] 453 0.221% 98.852% ( 15, 22 ] 47 0.023% 98.875% ( 22, 34 ] 62 0.030% 98.905% ( 34, 51 ] 42 0.020% 98.926% ( 51, 76 ] 19 0.009% 98.935% ( 76, 110 ] 97 0.047% 98.982% ( 110, 170 ] 538 0.262% 99.244% ( 170, 250 ] 433 0.211% 99.455% ( 250, 380 ] 874 0.426% 99.881% ( 380, 580 ] 167 0.081% 99.962% ( 580, 870 ] 67 0.033% 99.995% ( 870, 1300 ] 19 0.009% 100.004% ( 1300, 1900 ] 4 0.002% 100.006% ( 1900, 2900 ] 1 0.000% 100.006% ** Level 4 read latency histogram (micros): Count: 149920 Average: 37.1286 StdDev: 103.03 Min: 1 Median: 1.8377 Max: 4421 Percentiles: P50: 1.84 P75: 3.62 P99: 400.60 P99.9: 796.60 P99.99: 1037.02 ------------------------------------------------------ [ 0, 1 ] 20557 13.712% 13.712% ### ( 1, 2 ] 64940 43.316% 57.028% ######### ( 2, 3 ] 19220 12.820% 69.849% ### ( 3, 4 ] 12395 8.268% 78.116% ## ( 4, 6 ] 9466 6.314% 84.430% # ( 6, 10 ] 1996 1.331% 85.762% ( 10, 15 ] 692 0.462% 86.223% ( 15, 22 ] 56 0.037% 86.261% ( 22, 34 ] 50 0.033% 86.294% ( 34, 51 ] 49 0.033% 86.327% ( 51, 76 ] 14 0.009% 86.336% ( 76, 110 ] 733 0.489% 86.825% ( 110, 170 ] 5368 3.581% 90.406% # ( 170, 250 ] 4452 2.970% 93.375% # ( 250, 380 ] 8310 5.543% 98.918% # ( 380, 580 ] 1192 0.795% 99.713% ( 580, 870 ] 375 0.250% 99.963% ( 870, 1300 ] 103 0.069% 100.032% ( 1300, 1900 ] 13 0.009% 100.041% ( 1900, 2900 ] 6 0.004% 100.045% ( 2900, 4400 ] 8 0.005% 100.050% ( 4400, 6600 ] 1 0.001% 100.051% ** Level 5 read latency histogram (micros): Count: 177295 Average: 250.1432 StdDev: 139.19 Min: 82 Median: 242.2479 Max: 5401 Percentiles: P50: 242.25 P75: 323.21 P99: 791.38 P99.9: 1431.62 P99.99: 2730.48 ------------------------------------------------------ ( 76, 110 ] 6845 3.861% 3.861% # ( 110, 170 ] 47324 26.692% 30.553% ##### ( 170, 250 ] 38178 21.534% 52.087% #### ( 250, 380 ] 72134 40.686% 92.772% ######## ( 380, 580 ] 9092 5.128% 97.901% # ( 580, 870 ] 2674 1.508% 99.409% ( 870, 1300 ] 831 0.469% 99.878% ( 1300, 1900 ] 181 0.102% 99.980% ( 1900, 2900 ] 22 0.012% 99.992% ( 2900, 4400 ] 40 0.023% 100.015% ( 4400, 6600 ] 4 0.002% 100.017% ** DB Stats ** Uptime(secs): 19.3 total, 13.7 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 101.95 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:10.525 H:M:S, 54.5 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 71.75 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:8.852 H:M:S, 64.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 332.619 micros/op 12015 ops/sec 60.254 seconds 723996 operations; 11.6 MB/s (179999 of 179999 found) Microseconds per read: Count: 723996 Average: 332.6218 StdDev: 319.91 Min: 2 Median: 295.7336 Max: 36391 Percentiles: P50: 295.73 P75: 444.60 P99: 942.17 P99.9: 4158.07 P99.99: 10344.23 ------------------------------------------------------ ( 1, 2 ] 5 0.001% 0.001% ( 2, 3 ] 5 0.001% 0.001% ( 3, 4 ] 1 0.000% 0.002% ( 4, 6 ] 23 0.003% 0.005% ( 6, 10 ] 639 0.088% 0.093% ( 10, 15 ] 847 0.117% 0.210% ( 15, 22 ] 1281 0.177% 0.387% ( 22, 34 ] 5335 0.737% 1.124% ( 34, 51 ] 2748 0.380% 1.503% ( 51, 76 ] 5419 0.748% 2.252% ( 76, 110 ] 4767 0.658% 2.910% ( 110, 170 ] 70024 9.672% 12.582% ## ( 170, 250 ] 206161 28.475% 41.058% ###### ( 250, 380 ] 184035 25.419% 66.477% ##### ( 380, 580 ] 191052 26.389% 92.865% ##### ( 580, 870 ] 43374 5.991% 98.856% # ( 870, 1300 ] 6197 0.856% 99.712% ( 1300, 1900 ] 519 0.072% 99.784% ( 1900, 2900 ] 216 0.030% 99.814% ( 2900, 4400 ] 744 0.103% 99.917% ( 4400, 6600 ] 406 0.056% 99.973% ( 6600, 9900 ] 123 0.017% 99.990% ( 9900, 14000 ] 24 0.003% 99.993% ( 14000, 22000 ] 25 0.003% 99.996% ( 22000, 33000 ] 24 0.003% 100.000% ( 33000, 50000 ] 2 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 14693304 rocksdb.bloom.filter.full.positive COUNT : 866859 rocksdb.bloom.filter.full.true.positive COUNT : 723741 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 255 rocksdb.memtable.miss COUNT : 723741 rocksdb.l0.hit COUNT : 31041 rocksdb.l1.hit COUNT : 9850 rocksdb.l2andup.hit COUNT : 682850 rocksdb.compaction.key.drop.new COUNT : 51783 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2267961 rocksdb.number.keys.read COUNT : 723996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2340535752 rocksdb.bytes.read COUNT : 723996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1534 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 48203087 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 7115568828 rocksdb.compact.write.bytes COUNT : 7064047858 rocksdb.flush.write.bytes COUNT : 2312172606 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 27 rocksdb.number.superversion_acquires COUNT : 385 rocksdb.number.superversion_releases COUNT : 368 rocksdb.number.superversion_cleanups COUNT : 93 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2293391360 rocksdb.memtable.garbage.bytes.at.flush COUNT : 775168 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 10774958101 rocksdb.non.last.level.read.count COUNT : 2449657 rocksdb.block.checksum.compute.count COUNT : 2601149 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 291.245762 P95 : 680.384927 P99 : 933.921753 P100 : 36390.000000 COUNT : 723996 SUM : 238870713 rocksdb.db.write.micros P50 : 1.722872 P95 : 9.782476 P99 : 1074.111808 P100 : 5371.000000 COUNT : 2267961 SUM : 56205196 rocksdb.compaction.times.micros P50 : 973333.333333 P95 : 2400000.000000 P99 : 2800000.000000 P100 : 2802943.000000 COUNT : 20 SUM : 21215790 rocksdb.compaction.times.cpu_micros P50 : 945000.000000 P95 : 2400000.000000 P99 : 2767030.000000 P100 : 2767030.000000 COUNT : 20 SUM : 20026011 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 572.857143 P95 : 2025.000000 P99 : 2173.000000 P100 : 2173.000000 COUNT : 35 SUM : 24706 rocksdb.compaction.outfile.sync.micros P50 : 771.136364 P95 : 1625.000000 P99 : 1803.000000 P100 : 1803.000000 COUNT : 55 SUM : 46704 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 145.625000 P95 : 681.500000 P99 : 900.000000 P100 : 900.000000 COUNT : 59 SUM : 12356 rocksdb.table.open.io.micros P50 : 2748.714070 P95 : 4373.294509 P99 : 8325.428571 P100 : 13018.000000 COUNT : 1534 SUM : 4363685 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 2.865960 P95 : 259.427411 P99 : 364.159549 P100 : 19874.000000 COUNT : 1723750 SUM : 54259954 rocksdb.read.block.get.micros P50 : 221.329534 P95 : 373.636424 P99 : 544.443021 P100 : 23857.000000 COUNT : 875911 SUM : 195606720 rocksdb.write.raw.block.micros P50 : 0.556263 P95 : 1.828153 P99 : 5.918133 P100 : 7797.000000 COUNT : 2270895 SUM : 4403938 rocksdb.numfiles.in.singlecompaction P50 : 3.285714 P95 : 5.600000 P99 : 29.000000 P100 : 29.000000 COUNT : 24 SUM : 111 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.625174 P95 : 1226.427344 P99 : 2585.684404 P100 : 5357.000000 COUNT : 258253 SUM : 48268831 rocksdb.sst.read.micros P50 : 2.527234 P95 : 350.278895 P99 : 393.728420 P100 : 5401.000000 COUNT : 2449776 SUM : 236304364 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 723996 SUM : 723996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2267961 SUM : 2340535752 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 200400.000000 P95 : 266250.000000 P99 : 271583.000000 P100 : 271583.000000 COUNT : 35 SUM : 6688709 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 71 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0