[2024-11-15 12:09:40.405680] Starting SPDK v25.01-pre git sha1 1c3ed84fd / DPDK 24.03.0 initialization... [2024-11-15 12:09:40.405823] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3508867 ] Set seed to 1731668980402176 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-15 12:09:40.522702] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-15 12:09:40.627645] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-15 12:09:40.627693] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3508867' to capture a snapshot of events at runtime. [2024-11-15 12:09:40.627704] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-15 12:09:40.627714] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-15 12:09:40.627722] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3508867 for offline analysis/debug. [2024-11-15 12:09:40.629085] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Fri Nov 15 12:09:46 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/11/15-12:09:50 ... thread 0: (1000000,1000000) ops and (234495.5,234495.5) ops/second in (4.264475,4.264475) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/12 3.03 GB 87.7 0.7 0.0 0.7 1.6 0.9 0.0 1.9 133.7 289.0 5.65 5.08 17 0.332 766K 3713 0.0 0.0 L1 14/14 1.68 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 84.71 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 13/0 951.93 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 118/0 9.53 GB 1.0 0.5 0.4 0.1 0.5 0.4 0.0 1.3 164.4 161.2 3.05 2.41 3 1.016 0 0 0.0 0.0 L5 124/0 10.15 GB 0.1 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 287/26 25.40 GB 0.0 1.2 0.4 0.8 2.1 1.2 0.6 2.4 144.5 244.2 8.70 7.49 20 0.435 766K 3713 0.0 0.0 Int 0/0 0.00 KB 0.0 1.2 0.4 0.8 2.1 1.2 0.6 2.4 144.5 244.2 8.70 7.49 20 0.435 766K 3713 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 1.2 0.4 0.8 1.2 0.4 0.0 0.0 213.2 210.9 5.89 5.21 6 0.982 766K 3713 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 314.1 2.80 2.29 14 0.200 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 4.4 total, 4.3 interval Flush(GB): cumulative 0.860, interval 0.860 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 2.07 GB write, 484.76 MB/s write, 1.23 GB read, 286.81 MB/s read, 8.7 seconds Interval compaction: 2.07 GB write, 497.53 MB/s write, 1.23 GB read, 294.37 MB/s read, 8.7 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 3, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 3, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 3, total-stops: 0, interval: 3 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 223118 Average: 32.5452 StdDev: 162.42 Min: 0 Median: 1.4927 Max: 15509 Percentiles: P50: 1.49 P75: 1.95 P99: 466.19 P99.9: 1032.51 P99.99: 1522.55 ------------------------------------------------------ [ 0, 1 ] 51886 23.255% 23.255% ##### ( 1, 2 ] 121110 54.281% 77.536% ########### ( 2, 3 ] 14086 6.313% 83.849% # ( 3, 4 ] 3893 1.745% 85.594% ( 4, 6 ] 3096 1.388% 86.981% ( 6, 10 ] 908 0.407% 87.388% ( 10, 15 ] 202 0.091% 87.479% ( 15, 22 ] 30 0.013% 87.492% ( 22, 34 ] 3 0.001% 87.494% ( 34, 51 ] 2 0.001% 87.495% ( 51, 76 ] 3 0.001% 87.496% ( 76, 110 ] 1110 0.497% 87.993% ( 110, 170 ] 11976 5.368% 93.361% # ( 170, 250 ] 9088 4.073% 97.434% # ( 250, 380 ] 2669 1.196% 98.630% ( 380, 580 ] 1914 0.858% 99.488% ( 580, 870 ] 821 0.368% 99.856% ( 870, 1300 ] 259 0.116% 99.972% ( 1300, 1900 ] 107 0.048% 100.020% ( 1900, 2900 ] 67 0.030% 100.050% ( 2900, 4400 ] 28 0.013% 100.063% ( 4400, 6600 ] 9 0.004% 100.067% ( 6600, 9900 ] 8 0.004% 100.070% ( 9900, 14000 ] 13 0.006% 100.076% ( 14000, 22000 ] 2 0.001% 100.077% ** Level 1 read latency histogram (micros): Count: 4263 Average: 264.3237 StdDev: 598.09 Min: 71 Median: 176.5879 Max: 14578 Percentiles: P50: 176.59 P75: 239.56 P99: 1444.43 P99.9: 11770.28 P99.99: 14578.00 ------------------------------------------------------ ( 51, 76 ] 1 0.023% 0.023% ( 76, 110 ] 160 3.753% 3.777% # ( 110, 170 ] 1859 43.608% 47.384% ######### ( 170, 250 ] 1354 31.762% 79.146% ###### ( 250, 380 ] 424 9.946% 89.092% ## ( 380, 580 ] 280 6.568% 95.660% # ( 580, 870 ] 108 2.533% 98.194% # ( 870, 1300 ] 31 0.727% 98.921% ( 1300, 1900 ] 14 0.328% 99.249% ( 1900, 2900 ] 8 0.188% 99.437% ( 2900, 4400 ] 9 0.211% 99.648% ( 4400, 6600 ] 5 0.117% 99.765% ( 6600, 9900 ] 3 0.070% 99.836% ( 9900, 14000 ] 6 0.141% 99.977% ( 14000, 22000 ] 1 0.023% 100.000% ** Level 2 read latency histogram (micros): Count: 276 Average: 258.8986 StdDev: 322.74 Min: 91 Median: 184.8454 Max: 3274 Percentiles: P50: 184.85 P75: 241.75 P99: 2313.33 P99.9: 3274.00 P99.99: 3274.00 ------------------------------------------------------ ( 76, 110 ] 9 3.261% 3.261% # ( 110, 170 ] 111 40.217% 43.478% ######## ( 170, 250 ] 97 35.145% 78.623% ####### ( 250, 380 ] 25 9.058% 87.681% ## ( 380, 580 ] 20 7.246% 94.928% # ( 580, 870 ] 7 2.536% 97.464% # ( 870, 1300 ] 2 0.725% 98.188% ( 1300, 1900 ] 1 0.362% 98.551% ( 1900, 2900 ] 3 1.087% 99.638% ( 2900, 4400 ] 1 0.362% 100.000% ** Level 3 read latency histogram (micros): Count: 2411 Average: 246.9871 StdDev: 286.03 Min: 85 Median: 179.9369 Max: 4782 Percentiles: P50: 179.94 P75: 240.74 P99: 1348.55 P99.9: 4135.44 P99.99: 4782.00 ------------------------------------------------------ ( 76, 110 ] 99 4.106% 4.106% # ( 110, 170 ] 1008 41.808% 45.915% ######## ( 170, 250 ] 793 32.891% 78.805% ####### ( 250, 380 ] 222 9.208% 88.013% ## ( 380, 580 ] 169 7.010% 95.023% # ( 580, 870 ] 65 2.696% 97.719% # ( 870, 1300 ] 30 1.244% 98.963% ( 1300, 1900 ] 11 0.456% 99.419% ( 1900, 2900 ] 5 0.207% 99.627% ( 2900, 4400 ] 8 0.332% 99.959% ( 4400, 6600 ] 1 0.041% 100.000% ** Level 4 read latency histogram (micros): Count: 30496 Average: 169.4110 StdDev: 317.16 Min: 0 Median: 136.2633 Max: 7404 Percentiles: P50: 136.26 P75: 207.11 P99: 1305.83 P99.9: 4327.17 P99.99: 6281.82 ------------------------------------------------------ [ 0, 1 ] 541 1.774% 1.774% ( 1, 2 ] 5859 19.212% 20.986% #### ( 2, 3 ] 2254 7.391% 28.377% # ( 3, 4 ] 767 2.515% 30.893% # ( 4, 6 ] 1051 3.446% 34.339% # ( 6, 10 ] 258 0.846% 35.185% ( 10, 15 ] 65 0.213% 35.398% ( 15, 22 ] 7 0.023% 35.421% ( 22, 34 ] 5 0.016% 35.437% ( 34, 51 ] 2 0.007% 35.444% ( 76, 110 ] 749 2.456% 37.900% ( 110, 170 ] 8430 27.643% 65.543% ###### ( 170, 250 ] 6217 20.386% 85.929% #### ( 250, 380 ] 1851 6.070% 91.999% # ( 380, 580 ] 1368 4.486% 96.485% # ( 580, 870 ] 573 1.879% 98.364% ( 870, 1300 ] 194 0.636% 99.000% ( 1300, 1900 ] 107 0.351% 99.351% ( 1900, 2900 ] 100 0.328% 99.679% ( 2900, 4400 ] 72 0.236% 99.915% ( 4400, 6600 ] 28 0.092% 100.007% ( 6600, 9900 ] 1 0.003% 100.010% ** Level 5 read latency histogram (micros): Count: 19024 Average: 256.1130 StdDev: 352.44 Min: 82 Median: 177.2380 Max: 6171 Percentiles: P50: 177.24 P75: 239.83 P99: 1908.54 P99.9: 4507.36 P99.99: 6171.00 ------------------------------------------------------ ( 76, 110 ] 693 3.643% 3.643% # ( 110, 170 ] 8269 43.466% 47.109% ######### ( 170, 250 ] 6079 31.954% 79.063% ###### ( 250, 380 ] 1761 9.257% 88.320% ## ( 380, 580 ] 1226 6.444% 94.765% # ( 580, 870 ] 533 2.802% 97.566% # ( 870, 1300 ] 180 0.946% 98.512% ( 1300, 1900 ] 92 0.484% 98.996% ( 1900, 2900 ] 89 0.468% 99.464% ( 2900, 4400 ] 82 0.431% 99.895% ( 4400, 6600 ] 20 0.105% 100.000% ** DB Stats ** Uptime(secs): 4.4 total, 4.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 224.66 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.830 H:M:S, 19.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 230.56 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.830 H:M:S, 19.5 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/11/15-12:10:00 ... thread 0: (1000000,2000000) ops and (104630.0,144697.3) ops/second in (9.557484,13.821959) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/16 3.97 GB 159.3 2.8 0.0 2.8 4.6 1.8 0.0 2.5 171.7 283.9 16.45 15.23 37 0.445 2868K 43K 0.0 0.0 L1 14/14 1.68 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 84.71 MB 0.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 13/0 951.93 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 118/0 9.53 GB 1.0 0.5 0.4 0.1 0.5 0.4 0.0 1.3 164.4 161.2 3.05 2.41 3 1.016 0 0 0.0 0.0 L5 124/0 10.15 GB 0.1 0.0 0.0 0.0 0.0 0.0 0.6 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 289/30 26.34 GB 0.0 3.2 0.4 2.9 5.0 2.2 0.6 2.7 170.6 264.7 19.50 17.64 40 0.487 2868K 43K 0.0 0.0 Int 0/0 0.00 KB 0.0 3.2 0.4 2.9 5.0 2.2 0.6 2.7 170.6 264.7 19.50 17.64 40 0.487 2868K 43K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.2 0.4 2.9 3.2 0.3 0.0 0.0 245.3 241.4 13.56 12.73 10 1.356 2868K 43K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 317.8 5.94 4.91 30 0.198 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 13.9 total, 13.8 interval Flush(GB): cumulative 1.843, interval 1.843 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.04 GB write, 370.26 MB/s write, 3.25 GB read, 238.64 MB/s read, 19.5 seconds Interval compaction: 5.04 GB write, 373.27 MB/s write, 3.25 GB read, 240.58 MB/s read, 19.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 12, total-stops: 0, interval: 12 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 868290 Average: 28.1344 StdDev: 105.99 Min: 0 Median: 1.4674 Max: 15509 Percentiles: P50: 1.47 P75: 1.94 P99: 352.75 P99.9: 725.38 P99.99: 858.76 ------------------------------------------------------ [ 0, 1 ] 219517 25.282% 25.282% ##### ( 1, 2 ] 459174 52.883% 78.164% ########### ( 2, 3 ] 49564 5.708% 83.872% # ( 3, 4 ] 14458 1.665% 85.537% ( 4, 6 ] 9928 1.143% 86.681% ( 6, 10 ] 2488 0.287% 86.967% ( 10, 15 ] 677 0.078% 87.045% ( 15, 22 ] 96 0.011% 87.056% ( 22, 34 ] 5 0.001% 87.057% ( 34, 51 ] 7 0.001% 87.058% ( 51, 76 ] 8 0.001% 87.059% ( 76, 110 ] 5029 0.579% 87.638% ( 110, 170 ] 55513 6.393% 94.031% # ( 170, 250 ] 36746 4.232% 98.263% # ( 250, 380 ] 8094 0.932% 99.195% ( 380, 580 ] 5266 0.606% 99.802% ( 580, 870 ] 1699 0.196% 99.998% ( 870, 1300 ] 348 0.040% 100.038% ( 1300, 1900 ] 125 0.014% 100.052% ( 1900, 2900 ] 107 0.012% 100.064% ( 2900, 4400 ] 52 0.006% 100.070% ( 4400, 6600 ] 9 0.001% 100.071% ( 6600, 9900 ] 8 0.001% 100.072% ( 9900, 14000 ] 13 0.001% 100.074% ( 14000, 22000 ] 2 0.000% 100.074% ** Level 1 read latency histogram (micros): Count: 16764 Average: 207.1758 StdDev: 318.10 Min: 71 Median: 164.3404 Max: 14578 Percentiles: P50: 164.34 P75: 220.91 P99: 795.42 P99.9: 4106.00 P99.99: 13537.79 ------------------------------------------------------ ( 51, 76 ] 1 0.006% 0.006% ( 76, 110 ] 815 4.862% 4.868% # ( 110, 170 ] 8354 49.833% 54.701% ########## ( 170, 250 ] 5348 31.902% 86.602% ###### ( 250, 380 ] 1162 6.932% 93.534% # ( 380, 580 ] 721 4.301% 97.835% # ( 580, 870 ] 263 1.569% 99.403% ( 870, 1300 ] 43 0.257% 99.660% ( 1300, 1900 ] 20 0.119% 99.779% ( 1900, 2900 ] 13 0.078% 99.857% ( 2900, 4400 ] 9 0.054% 99.911% ( 4400, 6600 ] 5 0.030% 99.940% ( 6600, 9900 ] 3 0.018% 99.958% ( 9900, 14000 ] 6 0.036% 99.994% ( 14000, 22000 ] 1 0.006% 100.000% ** Level 2 read latency histogram (micros): Count: 1070 Average: 209.9271 StdDev: 187.67 Min: 91 Median: 168.7059 Max: 3274 Percentiles: P50: 168.71 P75: 226.84 P99: 817.80 P99.9: 2876.67 P99.99: 3274.00 ------------------------------------------------------ ( 76, 110 ] 36 3.364% 3.364% # ( 110, 170 ] 510 47.664% 51.028% ########## ( 170, 250 ] 361 33.738% 84.766% ####### ( 250, 380 ] 84 7.850% 92.617% ## ( 380, 580 ] 56 5.234% 97.850% # ( 580, 870 ] 15 1.402% 99.252% ( 870, 1300 ] 3 0.280% 99.533% ( 1300, 1900 ] 1 0.093% 99.626% ( 1900, 2900 ] 3 0.280% 99.907% ( 2900, 4400 ] 1 0.093% 100.000% ** Level 3 read latency histogram (micros): Count: 9363 Average: 202.5306 StdDev: 169.54 Min: 84 Median: 165.4743 Max: 4782 Percentiles: P50: 165.47 P75: 222.28 P99: 803.93 P99.9: 2839.50 P99.99: 4540.14 ------------------------------------------------------ ( 76, 110 ] 422 4.507% 4.507% # ( 110, 170 ] 4607 49.204% 53.711% ########## ( 170, 250 ] 3050 32.575% 86.286% ####### ( 250, 380 ] 645 6.889% 93.175% # ( 380, 580 ] 428 4.571% 97.746% # ( 580, 870 ] 152 1.623% 99.370% ( 870, 1300 ] 32 0.342% 99.712% ( 1300, 1900 ] 12 0.128% 99.840% ( 1900, 2900 ] 6 0.064% 99.904% ( 2900, 4400 ] 8 0.085% 99.989% ( 4400, 6600 ] 1 0.011% 100.000% ** Level 4 read latency histogram (micros): Count: 89929 Average: 178.8228 StdDev: 205.18 Min: 0 Median: 155.3272 Max: 7404 Percentiles: P50: 155.33 P75: 212.06 P99: 794.95 P99.9: 3177.72 P99.99: 5841.92 ------------------------------------------------------ [ 0, 1 ] 541 0.602% 0.602% ( 1, 2 ] 5973 6.642% 7.243% # ( 2, 3 ] 2683 2.983% 10.227% # ( 3, 4 ] 989 1.100% 11.327% ( 4, 6 ] 1404 1.561% 12.888% ( 6, 10 ] 310 0.345% 13.233% ( 10, 15 ] 77 0.086% 13.318% ( 15, 22 ] 11 0.012% 13.331% ( 22, 34 ] 5 0.006% 13.336% ( 34, 51 ] 2 0.002% 13.338% ( 76, 110 ] 3704 4.119% 17.457% # ( 110, 170 ] 38739 43.077% 60.534% ######### ( 170, 250 ] 24742 27.513% 88.047% ###### ( 250, 380 ] 5389 5.993% 94.040% # ( 380, 580 ] 3615 4.020% 98.060% # ( 580, 870 ] 1141 1.269% 99.328% ( 870, 1300 ] 264 0.294% 99.622% ( 1300, 1900 ] 120 0.133% 99.755% ( 1900, 2900 ] 116 0.129% 99.884% ( 2900, 4400 ] 76 0.085% 99.969% ( 4400, 6600 ] 29 0.032% 100.001% ( 6600, 9900 ] 1 0.001% 100.002% ** Level 5 read latency histogram (micros): Count: 73767 Average: 204.8560 StdDev: 201.84 Min: 80 Median: 164.6169 Max: 6171 Percentiles: P50: 164.62 P75: 221.48 P99: 818.91 P99.9: 3439.88 P99.99: 5788.56 ------------------------------------------------------ ( 76, 110 ] 3539 4.798% 4.798% # ( 110, 170 ] 36631 49.658% 54.455% ########## ( 170, 250 ] 23553 31.929% 86.384% ###### ( 250, 380 ] 5056 6.854% 93.238% # ( 380, 580 ] 3368 4.566% 97.804% # ( 580, 870 ] 1071 1.452% 99.256% ( 870, 1300 ] 243 0.329% 99.585% ( 1300, 1900 ] 102 0.138% 99.723% ( 1900, 2900 ] 100 0.136% 99.859% ( 2900, 4400 ] 84 0.114% 99.973% ( 4400, 6600 ] 20 0.027% 100.000% ** DB Stats ** Uptime(secs): 13.9 total, 9.6 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 141.22 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.679 H:M:S, 47.9 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 102.98 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.849 H:M:S, 61.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 20000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 19378.7 MB (estimated) FileSize: 19378.7 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 256.196 micros/op 15574 ops/sec 20.225 seconds 314996 operations; 15.1 MB/s (78999 of 78999 found) Microseconds per read: Count: 314996 Average: 256.2018 StdDev: 408.55 Min: 1 Median: 211.3298 Max: 34647 Percentiles: P50: 211.33 P75: 279.89 P99: 864.42 P99.9: 3795.61 P99.99: 19729.82 ------------------------------------------------------ [ 0, 1 ] 1 0.000% 0.000% ( 1, 2 ] 15 0.005% 0.005% ( 2, 3 ] 68 0.022% 0.027% ( 3, 4 ] 93 0.030% 0.056% ( 4, 6 ] 166 0.053% 0.109% ( 6, 10 ] 451 0.143% 0.252% ( 10, 15 ] 2986 0.948% 1.200% ( 15, 22 ] 3407 1.082% 2.282% ( 22, 34 ] 5281 1.677% 3.958% ( 34, 51 ] 1680 0.533% 4.491% ( 51, 76 ] 173 0.055% 4.546% ( 76, 110 ] 661 0.210% 4.756% ( 110, 170 ] 70364 22.338% 27.094% #### ( 170, 250 ] 139661 44.337% 71.432% ######### ( 250, 380 ] 48878 15.517% 86.949% ### ( 380, 580 ] 28540 9.060% 96.009% ## ( 580, 870 ] 9606 3.050% 99.059% # ( 870, 1300 ] 2163 0.687% 99.745% ( 1300, 1900 ] 296 0.094% 99.839% ( 1900, 2900 ] 151 0.048% 99.887% ( 2900, 4400 ] 67 0.021% 99.909% ( 4400, 6600 ] 24 0.008% 99.916% ( 6600, 9900 ] 113 0.036% 99.952% ( 9900, 14000 ] 93 0.030% 99.982% ( 14000, 22000 ] 37 0.012% 99.993% ( 22000, 33000 ] 20 0.006% 100.000% ( 33000, 50000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 6020213 rocksdb.bloom.filter.full.positive COUNT : 373164 rocksdb.bloom.filter.full.true.positive COUNT : 314193 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 803 rocksdb.memtable.miss COUNT : 314193 rocksdb.l0.hit COUNT : 55265 rocksdb.l1.hit COUNT : 22967 rocksdb.l2andup.hit COUNT : 235961 rocksdb.compaction.key.drop.new COUNT : 89624 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2630761 rocksdb.number.keys.read COUNT : 314996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2714945352 rocksdb.bytes.read COUNT : 314996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 339 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 10749605 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 5429912308 rocksdb.compact.write.bytes COUNT : 5334608067 rocksdb.flush.write.bytes COUNT : 2704792614 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 22 rocksdb.number.superversion_acquires COUNT : 401 rocksdb.number.superversion_releases COUNT : 375 rocksdb.number.superversion_cleanups COUNT : 97 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2686484480 rocksdb.memtable.garbage.bytes.at.flush COUNT : 4235264 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 6665466394 rocksdb.non.last.level.read.count COUNT : 1577063 rocksdb.block.checksum.compute.count COUNT : 1694052 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 210.389029 P95 : 556.868031 P99 : 863.939880 P100 : 34627.000000 COUNT : 314996 SUM : 80358007 rocksdb.db.write.micros P50 : 1.440907 P95 : 3.968211 P99 : 8.665511 P100 : 2483.000000 COUNT : 2630761 SUM : 16900436 rocksdb.compaction.times.micros P50 : 1044166.666667 P95 : 4788372.000000 P99 : 4788372.000000 P100 : 4788372.000000 COUNT : 13 SUM : 16388234 rocksdb.compaction.times.cpu_micros P50 : 992222.222222 P95 : 4714091.000000 P99 : 4714091.000000 P100 : 4714091.000000 COUNT : 13 SUM : 15500193 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 781.388889 P95 : 1895.714286 P99 : 5134.000000 P100 : 5134.000000 COUNT : 41 SUM : 42490 rocksdb.compaction.outfile.sync.micros P50 : 1085.000000 P95 : 1870.000000 P99 : 3382.000000 P100 : 3382.000000 COUNT : 21 SUM : 24517 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 182.173913 P95 : 395.000000 P99 : 1397.000000 P100 : 1397.000000 COUNT : 57 SUM : 12145 rocksdb.table.open.io.micros P50 : 7042.156863 P95 : 13325.208333 P99 : 44237.000000 P100 : 44943.000000 COUNT : 339 SUM : 2572842 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.453962 P95 : 137.818738 P99 : 248.600535 P100 : 10232.000000 COUNT : 1316687 SUM : 23224072 rocksdb.read.block.get.micros P50 : 163.089821 P95 : 416.612767 P99 : 738.730549 P100 : 15694.000000 COUNT : 377287 SUM : 70693746 rocksdb.write.raw.block.micros P50 : 0.525100 P95 : 0.997690 P99 : 3.050535 P100 : 5172.000000 COUNT : 1949362 SUM : 4189965 rocksdb.numfiles.in.singlecompaction P50 : 3.500000 P95 : 23.800000 P99 : 26.000000 P100 : 26.000000 COUNT : 17 SUM : 87 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.571770 P95 : 1210.509831 P99 : 2471.000000 P100 : 2471.000000 COUNT : 65375 SUM : 10756374 rocksdb.sst.read.micros P50 : 1.666483 P95 : 237.239133 P99 : 500.944622 P100 : 15509.000000 COUNT : 1577488 SUM : 88907741 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 314996 SUM : 314996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2630761 SUM : 2714945352 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 211000.000000 P95 : 247900.000000 P99 : 276252.000000 P100 : 276252.000000 COUNT : 41 SUM : 8098413 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 48 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0