[2024-11-19 09:11:01.653378] Starting SPDK v25.01-pre git sha1 a7ec5bc8e / DPDK 24.03.0 initialization... [2024-11-19 09:11:01.653542] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid3962188 ] Set seed to 1732003861649856 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-11-19 09:11:01.803626] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-11-19 09:11:01.908458] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-11-19 09:11:01.908517] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 3962188' to capture a snapshot of events at runtime. [2024-11-19 09:11:01.908530] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-11-19 09:11:01.908543] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-11-19 09:11:01.908553] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid3962188 for offline analysis/debug. [2024-11-19 09:11:01.909861] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Tue Nov 19 09:11:04 2024 CPU: 72 * Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz CPUCache: 25344 KB 2024/11/19-09:11:09 ... thread 0: (1000000,1000000) ops and (229603.5,229603.5) ops/second in (4.355334,4.355334) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/15 3.03 GB 62.6 0.6 0.0 0.6 1.4 0.9 0.0 1.6 99.0 252.4 5.72 4.72 15 0.381 574K 3076 0.0 0.0 L1 3/3 215.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 994.77 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 132/0 9.52 GB 1.0 2.3 0.6 1.7 2.1 0.4 0.0 3.4 156.8 144.0 14.82 10.14 8 1.853 455K 0 0.0 0.0 L5 140/0 10.71 GB 0.1 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 311/18 24.45 GB 0.0 2.8 0.6 2.2 3.5 1.3 0.5 4.1 140.7 174.2 20.54 14.86 23 0.893 1030K 3076 0.0 0.0 Int 0/0 0.00 KB 0.0 2.8 0.6 2.2 3.5 1.3 0.5 4.1 140.7 174.2 20.54 14.86 23 0.893 1030K 3076 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.8 0.6 2.2 2.6 0.4 0.0 0.0 169.3 158.1 17.07 12.32 10 1.707 1030K 3076 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 253.4 3.48 2.54 13 0.267 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 4.4 total, 4.4 interval Flush(GB): cumulative 0.860, interval 0.860 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 3.49 GB write, 806.32 MB/s write, 2.82 GB read, 651.15 MB/s read, 20.5 seconds Interval compaction: 3.49 GB write, 821.32 MB/s write, 2.82 GB read, 663.26 MB/s read, 20.5 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 1, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 1, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 6, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 1, total-stops: 6, interval: 7 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 200328 Average: 36.9829 StdDev: 227.83 Min: 1 Median: 1.6384 Max: 33274 Percentiles: P50: 1.64 P75: 2.38 P99: 500.95 P99.9: 1980.84 P99.99: 3607.61 ------------------------------------------------------ [ 0, 1 ] 28170 14.062% 14.062% ### ( 1, 2 ] 112778 56.297% 70.359% ########### ( 2, 3 ] 24475 12.217% 82.576% ## ( 3, 4 ] 7462 3.725% 86.301% # ( 4, 6 ] 4778 2.385% 88.686% ( 6, 10 ] 1186 0.592% 89.278% ( 10, 15 ] 137 0.068% 89.346% ( 15, 22 ] 17 0.008% 89.355% ( 22, 34 ] 7 0.003% 89.358% ( 34, 51 ] 6 0.003% 89.361% ( 51, 76 ] 1 0.000% 89.362% ( 76, 110 ] 130 0.065% 89.427% ( 110, 170 ] 3753 1.873% 91.300% ( 170, 250 ] 10115 5.049% 96.349% # ( 250, 380 ] 4173 2.083% 98.433% ( 380, 580 ] 1883 0.940% 99.373% ( 580, 870 ] 660 0.329% 99.702% ( 870, 1300 ] 263 0.131% 99.833% ( 1300, 1900 ] 125 0.062% 99.896% ( 1900, 2900 ] 132 0.066% 99.962% ( 2900, 4400 ] 125 0.062% 100.024% ( 4400, 6600 ] 40 0.020% 100.044% ( 6600, 9900 ] 19 0.009% 100.053% ( 9900, 14000 ] 1 0.000% 100.054% ( 14000, 22000 ] 8 0.004% 100.058% ( 33000, 50000 ] 1 0.000% 100.058% ** Level 1 read latency histogram (micros): Count: 429 Average: 537.6387 StdDev: 1646.24 Min: 104 Median: 230.0000 Max: 17309 Percentiles: P50: 230.00 P75: 325.21 P99: 8481.00 P99.9: 17309.00 P99.99: 17309.00 ------------------------------------------------------ ( 76, 110 ] 3 0.699% 0.699% ( 110, 170 ] 72 16.783% 17.483% ### ( 170, 250 ] 186 43.357% 60.839% ######### ( 250, 380 ] 105 24.476% 85.315% ##### ( 380, 580 ] 31 7.226% 92.541% # ( 580, 870 ] 10 2.331% 94.872% ( 870, 1300 ] 1 0.233% 95.105% ( 1300, 1900 ] 3 0.699% 95.804% ( 1900, 2900 ] 3 0.699% 96.503% ( 2900, 4400 ] 3 0.699% 97.203% ( 4400, 6600 ] 6 1.399% 98.601% ( 6600, 9900 ] 3 0.699% 99.301% ( 14000, 22000 ] 3 0.699% 100.000% ** Level 3 read latency histogram (micros): Count: 51371 Average: 23.4217 StdDev: 192.63 Min: 1 Median: 1.9198 Max: 17246 Percentiles: P50: 1.92 P75: 2.98 P99: 422.98 P99.9: 2552.58 P99.99: 5029.84 ------------------------------------------------------ [ 0, 1 ] 2189 4.261% 4.261% # ( 1, 2 ] 25546 49.728% 53.990% ########## ( 2, 3 ] 10978 21.370% 75.360% #### ( 3, 4 ] 4536 8.830% 84.190% ## ( 4, 6 ] 4068 7.919% 92.108% ## ( 6, 10 ] 1270 2.472% 94.581% ( 10, 15 ] 289 0.563% 95.143% ( 15, 22 ] 43 0.084% 95.227% ( 22, 34 ] 11 0.021% 95.248% ( 34, 51 ] 9 0.018% 95.266% ( 51, 76 ] 8 0.016% 95.281% ( 76, 110 ] 15 0.029% 95.311% ( 110, 170 ] 376 0.732% 96.043% ( 170, 250 ] 1014 1.974% 98.016% ( 250, 380 ] 452 0.880% 98.896% ( 380, 580 ] 248 0.483% 99.379% ( 580, 870 ] 109 0.212% 99.591% ( 870, 1300 ] 63 0.123% 99.714% ( 1300, 1900 ] 63 0.123% 99.836% ( 1900, 2900 ] 50 0.097% 99.934% ( 2900, 4400 ] 26 0.051% 99.984% ( 4400, 6600 ] 10 0.019% 100.004% ( 6600, 9900 ] 1 0.002% 100.006% ( 14000, 22000 ] 2 0.004% 100.010% ** Level 4 read latency histogram (micros): Count: 323617 Average: 23.1891 StdDev: 180.09 Min: 1 Median: 1.9074 Max: 24334 Percentiles: P50: 1.91 P75: 2.90 P99: 358.72 P99.9: 1102.80 P99.99: 1767.23 ------------------------------------------------------ [ 0, 1 ] 10737 3.318% 3.318% # ( 1, 2 ] 166485 51.445% 54.763% ########## ( 2, 3 ] 72584 22.429% 77.192% #### ( 3, 4 ] 25932 8.013% 85.205% ## ( 4, 6 ] 22173 6.852% 92.057% # ( 6, 10 ] 7221 2.231% 94.288% ( 10, 15 ] 1222 0.378% 94.666% ( 15, 22 ] 211 0.065% 94.731% ( 22, 34 ] 42 0.013% 94.744% ( 34, 51 ] 34 0.011% 94.754% ( 51, 76 ] 31 0.010% 94.764% ( 76, 110 ] 115 0.036% 94.799% ( 110, 170 ] 2899 0.896% 95.695% ( 170, 250 ] 7853 2.427% 98.122% ( 250, 380 ] 3398 1.050% 99.172% ( 380, 580 ] 1577 0.487% 99.659% ( 580, 870 ] 624 0.193% 99.852% ( 870, 1300 ] 287 0.089% 99.941% ( 1300, 1900 ] 205 0.063% 100.004% ( 1900, 2900 ] 239 0.074% 100.078% ( 2900, 4400 ] 205 0.063% 100.141% ( 4400, 6600 ] 56 0.017% 100.159% ( 6600, 9900 ] 9 0.003% 100.161% ( 9900, 14000 ] 3 0.001% 100.162% ( 14000, 22000 ] 6 0.002% 100.164% ( 22000, 33000 ] 1 0.000% 100.164% ** Level 5 read latency histogram (micros): Count: 17076 Average: 332.6269 StdDev: 502.86 Min: 92 Median: 224.0573 Max: 15284 Percentiles: P50: 224.06 P75: 317.41 P99: 2894.97 P99.9: 6106.47 P99.99: 15169.60 ------------------------------------------------------ ( 76, 110 ] 129 0.755% 0.755% ( 110, 170 ] 3008 17.615% 18.371% #### ( 170, 250 ] 7993 46.808% 65.179% ######### ( 250, 380 ] 3234 18.939% 84.118% #### ( 380, 580 ] 1477 8.650% 92.768% ## ( 580, 870 ] 523 3.063% 95.830% # ( 870, 1300 ] 224 1.312% 97.142% ( 1300, 1900 ] 167 0.978% 98.120% ( 1900, 2900 ] 151 0.884% 99.004% ( 2900, 4400 ] 125 0.732% 99.736% ( 4400, 6600 ] 36 0.211% 99.947% ( 6600, 9900 ] 7 0.041% 99.988% ( 14000, 22000 ] 2 0.012% 100.000% ** DB Stats ** Uptime(secs): 4.4 total, 4.4 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 221.73 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.343 H:M:S, 7.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 225.81 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.343 H:M:S, 7.9 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/11/19-09:11:19 ... thread 0: (1000000,2000000) ops and (97665.4,137039.1) ops/second in (10.239036,14.594370) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 17/15 3.98 GB 159.3 2.8 0.0 2.8 4.6 1.8 0.0 2.5 164.0 268.7 17.61 16.09 36 0.489 2932K 43K 0.0 0.0 L1 3/3 215.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 18/0 994.77 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 132/0 9.52 GB 1.0 2.3 0.6 1.7 2.1 0.4 0.0 3.4 156.8 144.0 14.82 10.14 8 1.853 455K 0 0.0 0.0 L5 140/0 10.71 GB 0.1 0.0 0.0 0.0 0.0 0.0 0.5 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 310/18 25.40 GB 0.0 5.1 0.6 4.5 6.7 2.2 0.5 3.6 160.7 211.7 32.43 26.23 44 0.737 3387K 43K 0.0 0.0 Int 0/0 0.00 KB 0.0 5.1 0.6 4.5 6.7 2.2 0.5 3.6 160.7 211.7 32.43 26.23 44 0.737 3387K 43K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.1 0.6 4.5 4.9 0.4 0.0 0.0 201.9 192.9 25.82 20.90 15 1.721 3387K 43K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 285.2 6.62 5.33 29 0.228 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.7 total, 14.6 interval Flush(GB): cumulative 1.843, interval 1.843 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 6.71 GB write, 467.87 MB/s write, 5.09 GB read, 355.08 MB/s read, 32.4 seconds Interval compaction: 6.71 GB write, 470.47 MB/s write, 5.09 GB read, 357.05 MB/s read, 32.4 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 12, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 12, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 6, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 12, total-stops: 6, interval: 18 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 868654 Average: 29.5837 StdDev: 124.40 Min: 0 Median: 1.4970 Max: 33274 Percentiles: P50: 1.50 P75: 1.98 P99: 312.38 P99.9: 614.94 P99.99: 979.24 ------------------------------------------------------ [ 0, 1 ] 211142 24.307% 24.307% ##### ( 1, 2 ] 449023 51.692% 75.999% ########## ( 2, 3 ] 59797 6.884% 82.882% # ( 3, 4 ] 19524 2.248% 85.130% ( 4, 6 ] 12288 1.415% 86.545% ( 6, 10 ] 2940 0.338% 86.883% ( 10, 15 ] 434 0.050% 86.933% ( 15, 22 ] 83 0.010% 86.943% ( 22, 34 ] 35 0.004% 86.947% ( 34, 51 ] 10 0.001% 86.948% ( 51, 76 ] 3 0.000% 86.948% ( 76, 110 ] 2078 0.239% 87.187% ( 110, 170 ] 38015 4.376% 91.564% # ( 170, 250 ] 60397 6.953% 98.517% # ( 250, 380 ] 8752 1.008% 99.524% ( 380, 580 ] 3168 0.365% 99.889% ( 580, 870 ] 808 0.093% 99.982% ( 870, 1300 ] 280 0.032% 100.014% ( 1300, 1900 ] 130 0.015% 100.029% ( 1900, 2900 ] 132 0.015% 100.044% ( 2900, 4400 ] 125 0.014% 100.059% ( 4400, 6600 ] 41 0.005% 100.063% ( 6600, 9900 ] 19 0.002% 100.066% ( 9900, 14000 ] 1 0.000% 100.066% ( 14000, 22000 ] 8 0.001% 100.067% ( 33000, 50000 ] 1 0.000% 100.067% ** Level 1 read latency histogram (micros): Count: 2093 Average: 258.5896 StdDev: 759.98 Min: 94 Median: 194.8264 Max: 17309 Percentiles: P50: 194.83 P75: 232.10 P99: 1314.00 P99.9: 16418.67 P99.99: 17309.00 ------------------------------------------------------ ( 76, 110 ] 32 1.529% 1.529% ( 110, 170 ] 666 31.820% 33.349% ###### ( 170, 250 ] 1123 53.655% 87.004% ########### ( 250, 380 ] 186 8.887% 95.891% ## ( 380, 580 ] 53 2.532% 98.423% # ( 580, 870 ] 11 0.526% 98.949% ( 870, 1300 ] 1 0.048% 98.997% ( 1300, 1900 ] 3 0.143% 99.140% ( 1900, 2900 ] 3 0.143% 99.283% ( 2900, 4400 ] 3 0.143% 99.427% ( 4400, 6600 ] 6 0.287% 99.713% ( 6600, 9900 ] 3 0.143% 99.857% ( 14000, 22000 ] 3 0.143% 100.000% ** Level 3 read latency histogram (micros): Count: 59094 Average: 44.8153 StdDev: 188.98 Min: 1 Median: 2.1651 Max: 17246 Percentiles: P50: 2.17 P75: 4.53 P99: 437.02 P99.9: 2398.12 P99.99: 4859.93 ------------------------------------------------------ [ 0, 1 ] 2189 3.704% 3.704% # ( 1, 2 ] 25546 43.229% 46.934% ######### ( 2, 3 ] 10978 18.577% 65.511% #### ( 3, 4 ] 4536 7.676% 73.187% ## ( 4, 6 ] 4068 6.884% 80.071% # ( 6, 10 ] 1270 2.149% 82.220% ( 10, 15 ] 289 0.489% 82.709% ( 15, 22 ] 43 0.073% 82.782% ( 22, 34 ] 11 0.019% 82.800% ( 34, 51 ] 9 0.015% 82.816% ( 51, 76 ] 8 0.014% 82.829% ( 76, 110 ] 165 0.279% 83.108% ( 110, 170 ] 3181 5.383% 88.491% # ( 170, 250 ] 5261 8.903% 97.394% ## ( 250, 380 ] 847 1.433% 98.827% ( 380, 580 ] 358 0.606% 99.433% ( 580, 870 ] 124 0.210% 99.643% ( 870, 1300 ] 64 0.108% 99.751% ( 1300, 1900 ] 63 0.107% 99.858% ( 1900, 2900 ] 50 0.085% 99.942% ( 2900, 4400 ] 26 0.044% 99.986% ( 4400, 6600 ] 10 0.017% 100.003% ( 6600, 9900 ] 1 0.002% 100.005% ( 14000, 22000 ] 2 0.003% 100.008% ** Level 4 read latency histogram (micros): Count: 390897 Average: 50.4709 StdDev: 176.37 Min: 1 Median: 2.2463 Max: 24334 Percentiles: P50: 2.25 P75: 5.43 P99: 375.61 P99.9: 1010.12 P99.99: 1739.54 ------------------------------------------------------ [ 0, 1 ] 10737 2.747% 2.747% # ( 1, 2 ] 166684 42.641% 45.388% ######### ( 2, 3 ] 73195 18.725% 64.113% #### ( 3, 4 ] 26337 6.738% 70.851% # ( 4, 6 ] 22732 5.815% 76.666% # ( 6, 10 ] 7280 1.862% 78.528% ( 10, 15 ] 1237 0.316% 78.845% ( 15, 22 ] 216 0.055% 78.900% ( 22, 34 ] 44 0.011% 78.911% ( 34, 51 ] 34 0.009% 78.920% ( 51, 76 ] 31 0.008% 78.928% ( 76, 110 ] 1386 0.355% 79.283% ( 110, 170 ] 26958 6.896% 86.179% # ( 170, 250 ] 43744 11.191% 97.370% ## ( 250, 380 ] 6596 1.687% 99.057% ( 380, 580 ] 2443 0.625% 99.682% ( 580, 870 ] 755 0.193% 99.875% ( 870, 1300 ] 298 0.076% 99.951% ( 1300, 1900 ] 206 0.053% 100.004% ( 1900, 2900 ] 239 0.061% 100.065% ( 2900, 4400 ] 205 0.052% 100.118% ( 4400, 6600 ] 56 0.014% 100.132% ( 6600, 9900 ] 9 0.002% 100.134% ( 9900, 14000 ] 3 0.001% 100.135% ( 14000, 22000 ] 6 0.002% 100.137% ( 22000, 33000 ] 1 0.000% 100.137% ** Level 5 read latency histogram (micros): Count: 82863 Average: 216.7337 StdDev: 240.87 Min: 90 Median: 193.4049 Max: 15284 Percentiles: P50: 193.40 P75: 230.99 P99: 822.06 P99.9: 3945.64 P99.99: 6936.46 ------------------------------------------------------ ( 76, 110 ] 1357 1.638% 1.638% ( 110, 170 ] 27174 32.794% 34.432% ####### ( 170, 250 ] 44095 53.214% 87.646% ########### ( 250, 380 ] 6527 7.877% 95.523% ## ( 380, 580 ] 2348 2.834% 98.356% # ( 580, 870 ] 639 0.771% 99.127% ( 870, 1300 ] 235 0.284% 99.411% ( 1300, 1900 ] 167 0.202% 99.613% ( 1900, 2900 ] 151 0.182% 99.795% ( 2900, 4400 ] 125 0.151% 99.946% ( 4400, 6600 ] 36 0.043% 99.989% ( 6600, 9900 ] 7 0.008% 99.998% ( 14000, 22000 ] 2 0.002% 100.000% ** DB Stats ** Uptime(secs): 14.7 total, 10.2 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 134.11 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:6.632 H:M:S, 45.2 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 96.12 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:6.289 H:M:S, 61.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 20000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 19378.7 MB (estimated) FileSize: 19378.7 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 268.721 micros/op 14857 ops/sec 20.124 seconds 298996 operations; 14.4 MB/s (75999 of 75999 found) Microseconds per read: Count: 298996 Average: 268.7291 StdDev: 550.17 Min: 2 Median: 222.5732 Max: 64020 Percentiles: P50: 222.57 P75: 295.22 P99: 823.42 P99.9: 7139.55 P99.99: 25488.70 ------------------------------------------------------ ( 1, 2 ] 4 0.001% 0.001% ( 2, 3 ] 34 0.011% 0.013% ( 3, 4 ] 106 0.035% 0.048% ( 4, 6 ] 190 0.064% 0.112% ( 6, 10 ] 421 0.141% 0.253% ( 10, 15 ] 2176 0.728% 0.980% ( 15, 22 ] 3389 1.133% 2.114% ( 22, 34 ] 5617 1.879% 3.992% ( 34, 51 ] 2116 0.708% 4.700% ( 51, 76 ] 425 0.142% 4.842% ( 76, 110 ] 171 0.057% 4.899% ( 110, 170 ] 29537 9.879% 14.778% ## ( 170, 250 ] 160252 53.597% 68.375% ########### ( 250, 380 ] 56947 19.046% 87.421% #### ( 380, 580 ] 29008 9.702% 97.123% ## ( 580, 870 ] 6687 2.236% 99.359% ( 870, 1300 ] 998 0.334% 99.693% ( 1300, 1900 ] 194 0.065% 99.758% ( 1900, 2900 ] 172 0.058% 99.815% ( 2900, 4400 ] 165 0.055% 99.871% ( 4400, 6600 ] 71 0.024% 99.894% ( 6600, 9900 ] 104 0.035% 99.929% ( 9900, 14000 ] 81 0.027% 99.956% ( 14000, 22000 ] 90 0.030% 99.986% ( 22000, 33000 ] 35 0.012% 99.998% ( 33000, 50000 ] 4 0.001% 99.999% ( 50000, 75000 ] 2 0.001% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 5380347 rocksdb.bloom.filter.full.positive COUNT : 350155 rocksdb.bloom.filter.full.true.positive COUNT : 298088 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 908 rocksdb.memtable.miss COUNT : 298088 rocksdb.l0.hit COUNT : 53492 rocksdb.l1.hit COUNT : 2649 rocksdb.l2andup.hit COUNT : 241947 rocksdb.compaction.key.drop.new COUNT : 256571 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2866400 rocksdb.number.keys.read COUNT : 298996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2958124800 rocksdb.bytes.read COUNT : 298996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 390 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 8998842 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 6992202516 rocksdb.compact.write.bytes COUNT : 6722307193 rocksdb.flush.write.bytes COUNT : 2836506049 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 36 rocksdb.number.superversion_acquires COUNT : 441 rocksdb.number.superversion_releases COUNT : 418 rocksdb.number.superversion_cleanups COUNT : 107 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2817548288 rocksdb.memtable.garbage.bytes.at.flush COUNT : 4669440 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 7642325114 rocksdb.non.last.level.read.count COUNT : 1813981 rocksdb.block.checksum.compute.count COUNT : 2047119 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 221.953073 P95 : 535.564645 P99 : 822.798552 P100 : 64019.000000 COUNT : 298996 SUM : 80081856 rocksdb.db.write.micros P50 : 1.517164 P95 : 5.074689 P99 : 8.809171 P100 : 249460.000000 COUNT : 2866400 SUM : 16434465 rocksdb.compaction.times.micros P50 : 1153636.363636 P95 : 4410000.000000 P99 : 4846304.000000 P100 : 4846304.000000 COUNT : 19 SUM : 29707279 rocksdb.compaction.times.cpu_micros P50 : 1044166.666667 P95 : 4410000.000000 P99 : 4766005.000000 P100 : 4766005.000000 COUNT : 19 SUM : 24741965 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 787.142857 P95 : 1805.714286 P99 : 3626.000000 P100 : 3626.000000 COUNT : 42 SUM : 39117 rocksdb.compaction.outfile.sync.micros P50 : 1023.571429 P95 : 5060.000000 P99 : 10629.000000 P100 : 10629.000000 COUNT : 48 SUM : 77142 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 154.000000 P95 : 371.333333 P99 : 4860.000000 P100 : 4860.000000 COUNT : 64 SUM : 19552 rocksdb.table.open.io.micros P50 : 6574.117647 P95 : 19875.000000 P99 : 38718.000000 P100 : 38718.000000 COUNT : 390 SUM : 2871694 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.685053 P95 : 115.224041 P99 : 243.294548 P100 : 18863.000000 COUNT : 1692944 SUM : 28822162 rocksdb.read.block.get.micros P50 : 191.895162 P95 : 350.420004 P99 : 575.052061 P100 : 33279.000000 COUNT : 354029 SUM : 69400952 rocksdb.write.raw.block.micros P50 : 0.525804 P95 : 0.999027 P99 : 3.983329 P100 : 22207.000000 COUNT : 2331755 SUM : 5820500 rocksdb.numfiles.in.singlecompaction P50 : 3.423077 P95 : 9.700000 P99 : 13.850000 P100 : 14.000000 COUNT : 23 SUM : 105 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.563646 P95 : 1187.375738 P99 : 2544.717668 P100 : 249145.000000 COUNT : 58715 SUM : 9003703 rocksdb.sst.read.micros P50 : 1.737854 P95 : 233.854541 P99 : 369.705907 P100 : 33274.000000 COUNT : 1814401 SUM : 90150646 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 298996 SUM : 298996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2866400 SUM : 2958124800 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 213000.000000 P95 : 433833.333333 P99 : 534926.000000 P100 : 534926.000000 COUNT : 43 SUM : 9421985 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 79 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0