[2024-11-19 21:57:49.035101] Starting SPDK v24.01.1-pre git sha1 c13c99a5e / DPDK 23.11.0 initialization... [2024-11-19 21:57:49.035247] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid2210889 ] Set seed to 1732049869031608 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-19 21:57:49.140983] app.c: 798:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-19 21:57:49.349771] app.c: 488:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-19 21:57:49.349816] app.c: 489:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 2210889' to capture a snapshot of events at runtime. [2024-11-19 21:57:49.349826] app.c: 494:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid2210889 for offline analysis/debug. [2024-11-19 21:57:49.349869] reactor.c: 937:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Tue Nov 19 21:57:57 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/19-21:58:02 ... thread 0: (1000000,1000000) ops and (205781.0,205781.0) ops/second in (4.859534,4.859534) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/16 3.38 GB 62.9 0.6 0.0 0.6 1.5 0.9 0.0 1.7 100.6 241.3 6.26 4.72 14 0.447 639K 750 0.0 0.0 L1 5/5 621.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 57.02 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 22/0 918.21 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 165/0 9.66 GB 1.0 1.5 0.8 0.7 1.5 0.8 0.0 1.9 119.3 118.6 13.00 7.31 7 1.857 0 0 0.0 0.0 L5 1116/0 91.78 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1327/21 106.38 GB 0.0 2.1 0.8 1.3 3.0 1.7 2.0 3.5 113.2 158.5 19.26 12.03 21 0.917 639K 750 0.0 0.0 Int 0/0 0.00 KB 0.0 2.1 0.8 1.3 3.0 1.7 2.0 3.5 113.2 158.5 19.26 12.03 21 0.917 639K 750 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.1 0.8 1.3 2.1 0.8 0.0 0.0 141.5 140.8 15.42 9.67 9 1.713 639K 750 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 229.6 3.84 2.36 12 0.320 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.1 total, 4.9 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 2.98 GB write, 602.10 MB/s write, 2.13 GB read, 430.12 MB/s read, 19.3 seconds Interval compaction: 2.98 GB write, 628.02 MB/s write, 2.13 GB read, 448.64 MB/s read, 19.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 2, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 2, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 6, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 2, total-stops: 6, interval: 8 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 332048 Average: 13.3488 StdDev: 153.57 Min: 1 Median: 1.5238 Max: 18803 Percentiles: P50: 1.52 P75: 1.92 P99: 229.27 P99.9: 1702.95 P99.99: 2648.22 ------------------------------------------------------ [ 0, 1 ] 55749 16.789% 16.789% ### ( 1, 2 ] 210544 63.408% 80.197% ############# ( 2, 3 ] 35228 10.609% 90.806% ## ( 3, 4 ] 11091 3.340% 94.147% # ( 4, 6 ] 8380 2.524% 96.670% # ( 6, 10 ] 3181 0.958% 97.628% ( 10, 15 ] 568 0.171% 97.799% ( 15, 22 ] 76 0.023% 97.822% ( 22, 34 ] 8 0.002% 97.825% ( 34, 51 ] 4 0.001% 97.826% ( 76, 110 ] 219 0.066% 97.892% ( 110, 170 ] 2309 0.695% 98.587% ( 170, 250 ] 1850 0.557% 99.144% ( 250, 380 ] 730 0.220% 99.364% ( 380, 580 ] 680 0.205% 99.569% ( 580, 870 ] 496 0.149% 99.718% ( 870, 1300 ] 382 0.115% 99.833% ( 1300, 1900 ] 329 0.099% 99.933% ( 1900, 2900 ] 255 0.077% 100.009% ( 2900, 4400 ] 74 0.022% 100.032% ( 4400, 6600 ] 16 0.005% 100.036% ( 6600, 9900 ] 7 0.002% 100.039% ( 9900, 14000 ] 7 0.002% 100.041% ( 14000, 22000 ] 7 0.002% 100.043% ** Level 1 read latency histogram (micros): Count: 106 Average: 1224.3019 StdDev: 2723.44 Min: 98 Median: 232.6087 Max: 14395 Percentiles: P50: 232.61 P75: 700.83 P99: 13938.50 P99.9: 14395.00 P99.99: 14395.00 ------------------------------------------------------ ( 76, 110 ] 4 3.774% 3.774% # ( 110, 170 ] 31 29.245% 33.019% ###### ( 170, 250 ] 23 21.698% 54.717% #### ( 250, 380 ] 9 8.491% 63.208% ## ( 380, 580 ] 10 9.434% 72.642% ## ( 580, 870 ] 6 5.660% 78.302% # ( 870, 1300 ] 4 3.774% 82.075% # ( 1300, 1900 ] 4 3.774% 85.849% # ( 1900, 2900 ] 3 2.830% 88.679% # ( 2900, 4400 ] 5 4.717% 93.396% # ( 4400, 6600 ] 1 0.943% 94.340% ( 6600, 9900 ] 1 0.943% 95.283% ( 9900, 14000 ] 4 3.774% 99.057% # ( 14000, 22000 ] 1 0.943% 100.000% ** Level 2 read latency histogram (micros): Count: 18 Average: 924.2778 StdDev: 1638.32 Min: 126 Median: 380.0000 Max: 7260 Percentiles: P50: 380.00 P75: 652.50 P99: 7260.00 P99.9: 7260.00 P99.99: 7260.00 ------------------------------------------------------ ( 110, 170 ] 2 11.111% 11.111% ## ( 170, 250 ] 4 22.222% 33.333% #### ( 250, 380 ] 3 16.667% 50.000% ### ( 380, 580 ] 4 22.222% 72.222% #### ( 580, 870 ] 2 11.111% 83.333% ## ( 1900, 2900 ] 2 11.111% 94.444% ## ( 6600, 9900 ] 1 5.556% 100.000% # ** Level 3 read latency histogram (micros): Count: 318 Average: 753.3899 StdDev: 956.23 Min: 90 Median: 271.6667 Max: 7007 Percentiles: P50: 271.67 P75: 1047.61 P99: 4239.09 P99.9: 7007.00 P99.99: 7007.00 ------------------------------------------------------ ( 76, 110 ] 4 1.258% 1.258% ( 110, 170 ] 77 24.214% 25.472% ##### ( 170, 250 ] 74 23.270% 48.742% ##### ( 250, 380 ] 24 7.547% 56.289% ## ( 380, 580 ] 33 10.377% 66.667% ## ( 580, 870 ] 17 5.346% 72.013% # ( 870, 1300 ] 23 7.233% 79.245% # ( 1300, 1900 ] 29 9.119% 88.365% ## ( 1900, 2900 ] 24 7.547% 95.912% ## ( 2900, 4400 ] 11 3.459% 99.371% # ( 4400, 6600 ] 1 0.314% 99.686% ( 6600, 9900 ] 1 0.314% 100.000% ** Level 4 read latency histogram (micros): Count: 104818 Average: 35.0649 StdDev: 256.57 Min: 1 Median: 1.8313 Max: 8688 Percentiles: P50: 1.83 P75: 2.85 P99: 1147.37 P99.9: 3041.95 P99.99: 3869.46 ------------------------------------------------------ [ 0, 1 ] 6314 6.024% 6.024% # ( 1, 2 ] 55448 52.899% 58.923% ########### ( 2, 3 ] 19785 18.876% 77.799% #### ( 3, 4 ] 8665 8.267% 86.065% ## ( 4, 6 ] 8107 7.734% 93.800% ## ( 6, 10 ] 2388 2.278% 96.078% ( 10, 15 ] 749 0.715% 96.793% ( 15, 22 ] 113 0.108% 96.900% ( 22, 34 ] 17 0.016% 96.917% ( 34, 51 ] 4 0.004% 96.920% ( 76, 110 ] 43 0.041% 96.961% ( 110, 170 ] 463 0.442% 97.403% ( 170, 250 ] 444 0.424% 97.827% ( 250, 380 ] 274 0.261% 98.088% ( 380, 580 ] 318 0.303% 98.391% ( 580, 870 ] 354 0.338% 98.729% ( 870, 1300 ] 440 0.420% 99.149% ( 1300, 1900 ] 435 0.415% 99.564% ( 1900, 2900 ] 336 0.321% 99.885% ( 2900, 4400 ] 171 0.163% 100.048% ( 4400, 6600 ] 35 0.033% 100.081% ( 6600, 9900 ] 3 0.003% 100.084% ** Level 5 read latency histogram (micros): Count: 17763 Average: 756.2691 StdDev: 1111.80 Min: 83 Median: 249.7959 Max: 13269 Percentiles: P50: 249.80 P75: 788.58 P99: 5759.93 P99.9: 8056.07 P99.99: 9814.61 ------------------------------------------------------ ( 76, 110 ] 476 2.680% 2.680% # ( 110, 170 ] 4692 26.414% 29.094% ##### ( 170, 250 ] 3723 20.959% 50.053% #### ( 250, 380 ] 1737 9.779% 59.832% ## ( 380, 580 ] 1665 9.373% 69.206% ## ( 580, 870 ] 1431 8.056% 77.262% ## ( 870, 1300 ] 1053 5.928% 83.190% # ( 1300, 1900 ] 916 5.157% 88.347% # ( 1900, 2900 ] 866 4.875% 93.222% # ( 2900, 4400 ] 789 4.442% 97.664% # ( 4400, 6600 ] 384 2.162% 99.825% ( 6600, 9900 ] 30 0.169% 99.994% ( 9900, 14000 ] 1 0.006% 100.000% ** DB Stats ** Uptime(secs): 5.1 total, 4.9 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 194.10 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.312 H:M:S, 25.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 202.42 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.312 H:M:S, 27.0 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/19-21:58:11 ... thread 0: (1000000,2000000) ops and (121159.5,152519.1) ops/second in (8.253580,13.113114) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/12 4.36 GB 188.6 2.4 0.0 2.4 4.2 1.8 0.0 2.3 151.5 267.8 16.21 14.07 35 0.463 2493K 4222 0.0 0.0 L1 5/5 621.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 57.02 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 22/0 918.21 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 165/0 9.66 GB 1.0 1.5 0.8 0.7 1.5 0.8 0.0 1.9 119.3 118.6 13.00 7.31 7 1.857 0 0 0.0 0.0 L5 1116/0 91.78 GB 0.9 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1328/17 107.36 GB 0.0 3.9 0.8 3.1 5.7 2.6 2.0 3.1 137.2 201.4 29.21 21.38 42 0.695 2493K 4222 0.0 0.0 Int 0/0 0.00 KB 0.0 3.9 0.8 3.1 5.7 2.6 2.0 3.1 137.2 201.4 29.21 21.38 42 0.695 2493K 4222 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.9 0.8 3.1 3.9 0.8 0.0 0.0 180.2 179.6 22.24 16.37 14 1.588 2493K 4222 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 271.0 6.98 5.01 28 0.249 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.3 total, 13.1 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.75 GB write, 441.58 MB/s write, 3.91 GB read, 300.74 MB/s read, 29.2 seconds Interval compaction: 5.75 GB write, 448.63 MB/s write, 3.91 GB read, 305.54 MB/s read, 29.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 14, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 14, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 6, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 14, total-stops: 6, interval: 20 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 820468 Average: 19.8927 StdDev: 114.04 Min: 1 Median: 1.4773 Max: 18803 Percentiles: P50: 1.48 P75: 1.92 P99: 280.68 P99.9: 860.85 P99.99: 1733.92 ------------------------------------------------------ [ 0, 1 ] 187300 22.828% 22.828% ##### ( 1, 2 ] 467078 56.928% 79.757% ########### ( 2, 3 ] 60694 7.397% 87.154% # ( 3, 4 ] 20531 2.502% 89.657% # ( 4, 6 ] 13916 1.696% 91.353% ( 6, 10 ] 4419 0.539% 91.891% ( 10, 15 ] 1036 0.126% 92.017% ( 15, 22 ] 115 0.014% 92.031% ( 22, 34 ] 14 0.002% 92.033% ( 34, 51 ] 6 0.001% 92.034% ( 51, 76 ] 6 0.001% 92.035% ( 76, 110 ] 2925 0.357% 92.391% ( 110, 170 ] 32615 3.975% 96.366% # ( 170, 250 ] 20536 2.503% 98.869% # ( 250, 380 ] 4544 0.554% 99.423% ( 380, 580 ] 2854 0.348% 99.771% ( 580, 870 ] 1093 0.133% 99.904% ( 870, 1300 ] 453 0.055% 99.959% ( 1300, 1900 ] 347 0.042% 100.002% ( 1900, 2900 ] 277 0.034% 100.035% ( 2900, 4400 ] 100 0.012% 100.048% ( 4400, 6600 ] 16 0.002% 100.050% ( 6600, 9900 ] 8 0.001% 100.051% ( 9900, 14000 ] 7 0.001% 100.051% ( 14000, 22000 ] 7 0.001% 100.052% ** Level 1 read latency histogram (micros): Count: 869 Average: 315.9275 StdDev: 1014.32 Min: 85 Median: 166.8160 Max: 14395 Percentiles: P50: 166.82 P75: 227.70 P99: 3893.00 P99.9: 14395.00 P99.99: 14395.00 ------------------------------------------------------ ( 76, 110 ] 33 3.797% 3.797% # ( 110, 170 ] 424 48.792% 52.589% ########## ( 170, 250 ] 270 31.070% 83.659% ###### ( 250, 380 ] 60 6.904% 90.564% # ( 380, 580 ] 44 5.063% 95.627% # ( 580, 870 ] 14 1.611% 97.238% ( 870, 1300 ] 4 0.460% 97.699% ( 1300, 1900 ] 5 0.575% 98.274% ( 1900, 2900 ] 3 0.345% 98.619% ( 2900, 4400 ] 5 0.575% 99.194% ( 4400, 6600 ] 1 0.115% 99.310% ( 6600, 9900 ] 1 0.115% 99.425% ( 9900, 14000 ] 4 0.460% 99.885% ( 14000, 22000 ] 1 0.115% 100.000% ** Level 2 read latency histogram (micros): Count: 154 Average: 289.7922 StdDev: 613.90 Min: 92 Median: 186.3265 Max: 7260 Percentiles: P50: 186.33 P75: 249.18 P99: 2630.00 P99.9: 7260.00 P99.99: 7260.00 ------------------------------------------------------ ( 76, 110 ] 5 3.247% 3.247% # ( 110, 170 ] 62 40.260% 43.506% ######## ( 170, 250 ] 49 31.818% 75.325% ###### ( 250, 380 ] 18 11.688% 87.013% ## ( 380, 580 ] 14 9.091% 96.104% ## ( 580, 870 ] 3 1.948% 98.052% ( 1900, 2900 ] 2 1.299% 99.351% ( 6600, 9900 ] 1 0.649% 100.000% ** Level 3 read latency histogram (micros): Count: 2115 Average: 271.8241 StdDev: 431.41 Min: 85 Median: 166.7575 Max: 7007 Percentiles: P50: 166.76 P75: 229.34 P99: 2560.42 P99.9: 4384.32 P99.99: 7007.00 ------------------------------------------------------ ( 76, 110 ] 86 4.066% 4.066% # ( 110, 170 ] 1027 48.558% 52.624% ########## ( 170, 250 ] 638 30.165% 82.790% ###### ( 250, 380 ] 133 6.288% 89.078% # ( 380, 580 ] 106 5.012% 94.090% # ( 580, 870 ] 34 1.608% 95.697% ( 870, 1300 ] 25 1.182% 96.879% ( 1300, 1900 ] 29 1.371% 98.251% ( 1900, 2900 ] 24 1.135% 99.385% ( 2900, 4400 ] 11 0.520% 99.905% ( 4400, 6600 ] 1 0.047% 99.953% ( 6600, 9900 ] 1 0.047% 100.000% ** Level 4 read latency histogram (micros): Count: 118574 Average: 50.9140 StdDev: 248.62 Min: 1 Median: 1.9538 Max: 8688 Percentiles: P50: 1.95 P75: 3.78 P99: 1032.08 P99.9: 2921.03 P99.99: 3846.32 ------------------------------------------------------ [ 0, 1 ] 6314 5.325% 5.325% # ( 1, 2 ] 55538 46.838% 52.163% ######### ( 2, 3 ] 20158 17.000% 69.164% ### ( 3, 4 ] 8897 7.503% 76.667% ## ( 4, 6 ] 8531 7.195% 83.862% # ( 6, 10 ] 2429 2.049% 85.910% ( 10, 15 ] 767 0.647% 86.557% ( 15, 22 ] 117 0.099% 86.656% ( 22, 34 ] 17 0.014% 86.670% ( 34, 51 ] 4 0.003% 86.673% ( 76, 110 ] 677 0.571% 87.244% ( 110, 170 ] 7034 5.932% 93.176% # ( 170, 250 ] 4447 3.750% 96.927% # ( 250, 380 ] 1027 0.866% 97.793% ( 380, 580 ] 798 0.673% 98.466% ( 580, 870 ] 461 0.389% 98.855% ( 870, 1300 ] 457 0.385% 99.240% ( 1300, 1900 ] 438 0.369% 99.610% ( 1900, 2900 ] 342 0.288% 99.898% ( 2900, 4400 ] 173 0.146% 100.044% ( 4400, 6600 ] 35 0.030% 100.073% ( 6600, 9900 ] 3 0.003% 100.076% ** Level 5 read latency histogram (micros): Count: 129592 Average: 265.6976 StdDev: 467.93 Min: 81 Median: 165.7156 Max: 13269 Percentiles: P50: 165.72 P75: 227.09 P99: 2815.92 P99.9: 6035.15 P99.99: 8584.49 ------------------------------------------------------ ( 76, 110 ] 6017 4.643% 4.643% # ( 110, 170 ] 63299 48.845% 53.488% ########## ( 170, 250 ] 39062 30.142% 83.630% ###### ( 250, 380 ] 8444 6.516% 90.146% # ( 380, 580 ] 5932 4.577% 94.723% # ( 580, 870 ] 2555 1.972% 96.695% ( 870, 1300 ] 1200 0.926% 97.621% ( 1300, 1900 ] 960 0.741% 98.362% ( 1900, 2900 ] 903 0.697% 99.059% ( 2900, 4400 ] 805 0.621% 99.680% ( 4400, 6600 ] 384 0.296% 99.976% ( 6600, 9900 ] 30 0.023% 99.999% ( 9900, 14000 ] 1 0.001% 100.000% ** DB Stats ** Uptime(secs): 13.3 total, 8.3 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 147.73 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:5.987 H:M:S, 44.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 119.25 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:4.675 H:M:S, 56.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 229.366 micros/op 17428 ops/sec 60.130 seconds 1047996 operations; 16.9 MB/s (268999 of 268999 found) Microseconds per read: Count: 1047996 Average: 229.3678 StdDev: 492.29 Min: 2 Median: 199.1627 Max: 58889 Percentiles: P50: 199.16 P75: 238.62 P99: 659.94 P99.9: 6336.50 P99.99: 22284.76 ------------------------------------------------------ ( 1, 2 ] 11 0.001% 0.001% ( 2, 3 ] 37 0.004% 0.005% ( 3, 4 ] 46 0.004% 0.009% ( 4, 6 ] 82 0.008% 0.017% ( 6, 10 ] 480 0.046% 0.063% ( 10, 15 ] 4533 0.433% 0.495% ( 15, 22 ] 2504 0.239% 0.734% ( 22, 34 ] 10388 0.991% 1.725% ( 34, 51 ] 1804 0.172% 1.897% ( 51, 76 ] 168 0.016% 1.913% ( 76, 110 ] 2559 0.244% 2.158% ( 110, 170 ] 307741 29.365% 31.522% ###### ( 170, 250 ] 531212 50.688% 82.211% ########## ( 250, 380 ] 122203 11.661% 93.871% ## ( 380, 580 ] 51223 4.888% 98.759% # ( 580, 870 ] 9160 0.874% 99.633% ( 870, 1300 ] 1667 0.159% 99.792% ( 1300, 1900 ] 393 0.038% 99.830% ( 1900, 2900 ] 336 0.032% 99.862% ( 2900, 4400 ] 232 0.022% 99.884% ( 4400, 6600 ] 192 0.018% 99.902% ( 6600, 9900 ] 355 0.034% 99.936% ( 9900, 14000 ] 287 0.027% 99.963% ( 14000, 22000 ] 276 0.026% 99.990% ( 22000, 33000 ] 85 0.008% 99.998% ( 33000, 50000 ] 21 0.002% 100.000% ( 50000, 75000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 20866093 rocksdb.bloom.filter.full.positive COUNT : 1248058 rocksdb.bloom.filter.full.true.positive COUNT : 1047674 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 322 rocksdb.memtable.miss COUNT : 1047674 rocksdb.l0.hit COUNT : 47389 rocksdb.l1.hit COUNT : 6366 rocksdb.l2andup.hit COUNT : 993919 rocksdb.compaction.key.drop.new COUNT : 33170 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2208446 rocksdb.number.keys.read COUNT : 1047996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2279116272 rocksdb.bytes.read COUNT : 1047996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1389 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 51457844 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 6190779658 rocksdb.compact.write.bytes COUNT : 6156545421 rocksdb.flush.write.bytes COUNT : 2246066132 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 52 rocksdb.number.superversion_acquires COUNT : 354 rocksdb.number.superversion_releases COUNT : 332 rocksdb.number.superversion_cleanups COUNT : 86 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2227865600 rocksdb.memtable.garbage.bytes.at.flush COUNT : 785408 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 10835201585 rocksdb.non.last.level.read.count COUNT : 2471306 rocksdb.block.checksum.compute.count COUNT : 2761343 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 198.078383 P95 : 423.279105 P99 : 657.456085 P100 : 58887.000000 COUNT : 1047996 SUM : 239330196 rocksdb.db.write.micros P50 : 1.542868 P95 : 5.330265 P99 : 1138.954265 P100 : 420584.000000 COUNT : 2208446 SUM : 57104412 rocksdb.compaction.times.micros P50 : 1375000.000000 P95 : 2987836.000000 P99 : 2987836.000000 P100 : 2987836.000000 COUNT : 15 SUM : 23139305 rocksdb.compaction.times.cpu_micros P50 : 1060909.090909 P95 : 2957415.000000 P99 : 2957415.000000 P100 : 2957415.000000 COUNT : 15 SUM : 17259503 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 923.750000 P95 : 5940.000000 P99 : 7965.000000 P100 : 7965.000000 COUNT : 32 SUM : 53663 rocksdb.compaction.outfile.sync.micros P50 : 2685.714286 P95 : 9603.000000 P99 : 10229.000000 P100 : 10229.000000 COUNT : 49 SUM : 170114 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 231.538462 P95 : 2000.000000 P99 : 2720.000000 P100 : 2765.000000 COUNT : 54 SUM : 33284 rocksdb.table.open.io.micros P50 : 8119.981238 P95 : 16533.333333 P99 : 21642.298851 P100 : 43028.000000 COUNT : 1389 SUM : 11785242 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.751038 P95 : 194.803230 P99 : 248.920227 P100 : 14304.000000 COUNT : 1499582 SUM : 60564672 rocksdb.read.block.get.micros P50 : 151.440100 P95 : 245.156711 P99 : 497.862652 P100 : 28999.000000 COUNT : 1260340 SUM : 204699595 rocksdb.write.raw.block.micros P50 : 0.549015 P95 : 1.597659 P99 : 3.615416 P100 : 8426.000000 COUNT : 2035339 SUM : 5413631 rocksdb.numfiles.in.singlecompaction P50 : 3.529412 P95 : 5.900000 P99 : 17.000000 P100 : 17.000000 COUNT : 22 SUM : 101 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.598114 P95 : 2319.815891 P99 : 2784.156977 P100 : 420425.000000 COUNT : 191680 SUM : 51474910 rocksdb.sst.read.micros P50 : 119.498036 P95 : 235.218140 P99 : 411.773460 P100 : 18803.000000 COUNT : 2471338 SUM : 253925364 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1047996 SUM : 1047996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2208446 SUM : 2279116272 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 221200.000000 P95 : 456000.000000 P99 : 636202.000000 P100 : 636202.000000 COUNT : 32 SUM : 7845183 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 60 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0