[2024-10-14 06:13:11.566256] Starting SPDK v25.01-pre git sha1 bbce7a874 / DPDK 23.11.0 initialization... [2024-10-14 06:13:11.566527] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid2320139 ] Set seed to 1728879191559557 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-10-14 06:13:11.675429] app.c: 919:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-10-14 06:13:11.696975] app.c: 612:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-10-14 06:13:11.697020] app.c: 613:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 2320139' to capture a snapshot of events at runtime. [2024-10-14 06:13:11.697031] app.c: 618:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-10-14 06:13:11.697041] app.c: 619:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-10-14 06:13:11.697049] app.c: 620:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid2320139 for offline analysis/debug. [2024-10-14 06:13:11.697575] reactor.c:1005:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Mon Oct 14 06:13:15 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/10/14-06:13:20 ... thread 0: (1000000,1000000) ops and (230602.9,230602.9) ops/second in (4.336458,4.336458) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 13/11 2.77 GB 56.6 0.6 0.0 0.6 1.4 0.8 0.0 1.7 93.9 229.5 6.04 4.48 14 0.431 575K 652 0.0 0.0 L1 20/20 1.96 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 961.55 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 167/0 9.68 GB 1.0 2.3 1.0 1.3 2.2 0.9 0.0 2.3 120.8 118.6 19.25 10.79 11 1.750 0 0 0.0 0.0 L5 1125/0 92.53 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1340/31 107.88 GB 0.0 2.8 1.0 1.8 3.6 1.7 1.7 4.5 114.4 145.1 25.28 15.27 25 1.011 575K 652 0.0 0.0 Int 0/0 0.00 KB 0.0 2.8 1.0 1.8 3.6 1.7 1.7 4.5 114.4 145.1 25.28 15.27 25 1.011 575K 652 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.8 1.0 1.8 2.8 0.9 0.0 0.0 134.4 132.4 21.52 13.01 13 1.655 575K 652 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.8 0.8 0.0 0.0 0.0 217.7 3.76 2.26 12 0.314 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 4.5 total, 4.3 interval Flush(GB): cumulative 0.800, interval 0.800 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 3.58 GB write, 809.38 MB/s write, 2.82 GB read, 638.24 MB/s read, 25.3 seconds Interval compaction: 3.58 GB write, 845.65 MB/s write, 2.82 GB read, 666.84 MB/s read, 25.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 9, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 0, total-stops: 9, interval: 9 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 333789 Average: 6.8532 StdDev: 97.54 Min: 1 Median: 1.7108 Max: 9902 Percentiles: P50: 1.71 P75: 2.43 P99: 9.07 P99.9: 884.36 P99.99: 1938.94 ------------------------------------------------------ [ 0, 1 ] 26089 7.816% 7.816% ## ( 1, 2 ] 198100 59.349% 67.165% ############ ( 2, 3 ] 60950 18.260% 85.425% #### ( 3, 4 ] 21835 6.542% 91.966% # ( 4, 6 ] 19799 5.932% 97.898% # ( 6, 10 ] 4794 1.436% 99.334% ( 10, 15 ] 638 0.191% 99.525% ( 15, 22 ] 111 0.033% 99.559% ( 22, 34 ] 19 0.006% 99.564% ( 34, 51 ] 8 0.002% 99.567% ( 51, 76 ] 6 0.002% 99.569% ( 76, 110 ] 56 0.017% 99.585% ( 110, 170 ] 204 0.061% 99.646% ( 170, 250 ] 186 0.056% 99.702% ( 250, 380 ] 195 0.058% 99.761% ( 380, 580 ] 227 0.068% 99.829% ( 580, 870 ] 233 0.070% 99.898% ( 870, 1300 ] 186 0.056% 99.954% ( 1300, 1900 ] 117 0.035% 99.989% ( 1900, 2900 ] 93 0.028% 100.017% ( 2900, 4400 ] 85 0.025% 100.043% ( 4400, 6600 ] 18 0.005% 100.048% ( 6600, 9900 ] 3 0.001% 100.049% ( 9900, 14000 ] 1 0.000% 100.049% ** Level 1 read latency histogram (micros): Count: 231 Average: 1322.5714 StdDev: 1675.30 Min: 88 Median: 576.6667 Max: 9334 Percentiles: P50: 576.67 P75: 1757.50 P99: 7994.25 P99.9: 9334.00 P99.99: 9334.00 ------------------------------------------------------ ( 76, 110 ] 7 3.030% 3.030% # ( 110, 170 ] 36 15.584% 18.615% ### ( 170, 250 ] 19 8.225% 26.840% ## ( 250, 380 ] 24 10.390% 37.229% ## ( 380, 580 ] 30 12.987% 50.216% ### ( 580, 870 ] 22 9.524% 59.740% ## ( 870, 1300 ] 20 8.658% 68.398% ## ( 1300, 1900 ] 20 8.658% 77.056% ## ( 1900, 2900 ] 20 8.658% 85.714% ## ( 2900, 4400 ] 16 6.926% 92.641% # ( 4400, 6600 ] 13 5.628% 98.268% # ( 6600, 9900 ] 4 1.732% 100.000% ** Level 3 read latency histogram (micros): Count: 65553 Average: 7.8239 StdDev: 86.09 Min: 1 Median: 1.8678 Max: 4006 Percentiles: P50: 1.87 P75: 2.81 P99: 10.60 P99.9: 1202.87 P99.99: 2378.64 ------------------------------------------------------ [ 0, 1 ] 2276 3.472% 3.472% # ( 1, 2 ] 35148 53.618% 57.090% ########### ( 2, 3 ] 14543 22.185% 79.275% #### ( 3, 4 ] 5627 8.584% 87.859% ## ( 4, 6 ] 5523 8.425% 96.284% ## ( 6, 10 ] 1753 2.674% 98.958% # ( 10, 15 ] 230 0.351% 99.309% ( 15, 22 ] 60 0.092% 99.400% ( 22, 34 ] 8 0.012% 99.413% ( 34, 51 ] 2 0.003% 99.416% ( 51, 76 ] 3 0.005% 99.420% ( 76, 110 ] 4 0.006% 99.426% ( 110, 170 ] 30 0.046% 99.472% ( 170, 250 ] 38 0.058% 99.530% ( 250, 380 ] 53 0.081% 99.611% ( 380, 580 ] 67 0.102% 99.713% ( 580, 870 ] 76 0.116% 99.829% ( 870, 1300 ] 60 0.092% 99.921% ( 1300, 1900 ] 33 0.050% 99.971% ( 1900, 2900 ] 26 0.040% 100.011% ( 2900, 4400 ] 9 0.014% 100.024% ** Level 4 read latency histogram (micros): Count: 211384 Average: 18.5328 StdDev: 187.05 Min: 1 Median: 1.8712 Max: 8457 Percentiles: P50: 1.87 P75: 2.83 P99: 375.58 P99.9: 2402.63 P99.99: 3477.04 ------------------------------------------------------ [ 0, 1 ] 7427 3.514% 3.514% # ( 1, 2 ] 112790 53.358% 56.871% ########### ( 2, 3 ] 46289 21.898% 78.769% #### ( 3, 4 ] 18119 8.572% 87.341% ## ( 4, 6 ] 17334 8.200% 95.541% ## ( 6, 10 ] 5430 2.569% 98.110% # ( 10, 15 ] 768 0.363% 98.473% ( 15, 22 ] 127 0.060% 98.533% ( 22, 34 ] 37 0.018% 98.551% ( 34, 51 ] 24 0.011% 98.562% ( 51, 76 ] 13 0.006% 98.568% ( 76, 110 ] 55 0.026% 98.595% ( 110, 170 ] 222 0.105% 98.700% ( 170, 250 ] 270 0.128% 98.827% ( 250, 380 ] 378 0.179% 99.006% ( 380, 580 ] 520 0.246% 99.252% ( 580, 870 ] 531 0.251% 99.503% ( 870, 1300 ] 433 0.205% 99.708% ( 1300, 1900 ] 288 0.136% 99.844% ( 1900, 2900 ] 234 0.111% 99.955% ( 2900, 4400 ] 192 0.091% 100.046% ( 4400, 6600 ] 57 0.027% 100.073% ( 6600, 9900 ] 10 0.005% 100.078% ** Level 5 read latency histogram (micros): Count: 11301 Average: 1053.9681 StdDev: 1282.81 Min: 77 Median: 522.9587 Max: 11376 Percentiles: P50: 522.96 P75: 1288.85 P99: 6146.77 P99.9: 9220.13 P99.99: 9891.41 ------------------------------------------------------ ( 76, 110 ] 338 2.991% 2.991% # ( 110, 170 ] 1456 12.884% 15.875% ### ( 170, 250 ] 1346 11.910% 27.785% ## ( 250, 380 ] 1404 12.424% 40.209% ## ( 380, 580 ] 1548 13.698% 53.907% ### ( 580, 870 ] 1398 12.371% 66.277% ## ( 870, 1300 ] 1012 8.955% 75.232% ## ( 1300, 1900 ] 761 6.734% 81.966% # ( 1900, 2900 ] 849 7.513% 89.479% ## ( 2900, 4400 ] 837 7.406% 96.885% # ( 4400, 6600 ] 301 2.663% 99.549% # ( 6600, 9900 ] 50 0.442% 99.991% ( 9900, 14000 ] 1 0.009% 100.000% ** DB Stats ** Uptime(secs): 4.5 total, 4.3 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 217.14 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.548 H:M:S, 12.1 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 226.82 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:0.548 H:M:S, 12.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/10/14-06:13:48 ... thread 0: (1000000,2000000) ops and (35375.1,61340.4) ops/second in (28.268467,32.604925) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 11/7 3.87 GB 194.1 3.3 0.0 3.3 5.2 1.9 0.0 2.7 167.2 264.4 19.96 17.80 38 0.525 3388K 11K 0.0 0.0 L1 20/20 1.96 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 15/0 961.55 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 167/0 9.68 GB 1.0 2.3 1.0 1.3 2.2 0.9 0.0 2.3 120.8 118.6 19.25 10.79 11 1.750 0 0 0.0 0.0 L5 1125/0 92.53 GB 0.9 0.0 0.0 0.0 0.0 0.0 1.7 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1338/27 108.98 GB 0.0 5.5 1.0 4.5 7.4 2.8 1.7 3.9 144.4 192.9 39.21 28.59 49 0.800 3388K 11K 0.0 0.0 Int 0/0 0.00 KB 0.0 5.5 1.0 4.5 7.4 2.8 1.7 3.9 144.4 192.9 39.21 28.59 49 0.800 3388K 11K 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 5.5 1.0 4.5 5.5 0.9 0.0 0.0 177.9 176.2 31.83 23.18 19 1.675 3388K 11K 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 264.5 7.38 5.41 30 0.246 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 32.8 total, 32.6 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 7.38 GB write, 230.54 MB/s write, 5.53 GB read, 172.65 MB/s read, 39.2 seconds Interval compaction: 7.38 GB write, 231.92 MB/s write, 5.53 GB read, 173.68 MB/s read, 39.2 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 0, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 0, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 9, pending-compaction-bytes-delays: 10, pending-compaction-bytes-stops: 0, total-delays: 10, total-stops: 9, interval: 19 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1311826 Average: 25.1192 StdDev: 81.24 Min: 0 Median: 1.5968 Max: 9902 Percentiles: P50: 1.60 P75: 2.60 P99: 237.14 P99.9: 569.90 P99.99: 839.02 ------------------------------------------------------ [ 0, 1 ] 274667 20.938% 20.938% #### ( 1, 2 ] 638848 48.699% 69.637% ########## ( 2, 3 ] 117281 8.940% 78.577% ## ( 3, 4 ] 42535 3.242% 81.820% # ( 4, 6 ] 33092 2.523% 84.342% # ( 6, 10 ] 7631 0.582% 84.924% ( 10, 15 ] 1180 0.090% 85.014% ( 15, 22 ] 194 0.015% 85.029% ( 22, 34 ] 31 0.002% 85.031% ( 34, 51 ] 8 0.001% 85.032% ( 51, 76 ] 85 0.006% 85.038% ( 76, 110 ] 45158 3.442% 88.480% # ( 110, 170 ] 119273 9.092% 97.573% ## ( 170, 250 ] 22311 1.701% 99.273% ( 250, 380 ] 4913 0.375% 99.648% ( 380, 580 ] 3483 0.266% 99.913% ( 580, 870 ] 1125 0.086% 99.999% ( 870, 1300 ] 310 0.024% 100.023% ( 1300, 1900 ] 143 0.011% 100.034% ( 1900, 2900 ] 172 0.013% 100.047% ( 2900, 4400 ] 122 0.009% 100.056% ( 4400, 6600 ] 19 0.001% 100.058% ( 6600, 9900 ] 3 0.000% 100.058% ( 9900, 14000 ] 1 0.000% 100.058% ** Level 1 read latency histogram (micros): Count: 148973 Average: 18.4902 StdDev: 96.46 Min: 1 Median: 2.2263 Max: 9334 Percentiles: P50: 2.23 P75: 3.77 P99: 213.89 P99.9: 692.24 P99.99: 4222.13 ------------------------------------------------------ [ 0, 1 ] 10215 6.857% 6.857% # ( 1, 2 ] 56131 37.679% 44.536% ######## ( 2, 3 ] 35966 24.143% 68.678% ##### ( 3, 4 ] 12282 8.244% 76.923% ## ( 4, 6 ] 17021 11.426% 88.348% ## ( 6, 10 ] 2505 1.682% 90.030% ( 10, 15 ] 486 0.326% 90.356% ( 15, 22 ] 278 0.187% 90.543% ( 22, 34 ] 8 0.005% 90.548% ( 51, 76 ] 7 0.005% 90.553% ( 76, 110 ] 3039 2.040% 92.593% ( 110, 170 ] 8685 5.830% 98.423% # ( 170, 250 ] 1568 1.053% 99.475% ( 250, 380 ] 349 0.234% 99.709% ( 380, 580 ] 255 0.171% 99.881% ( 580, 870 ] 75 0.050% 99.931% ( 870, 1300 ] 30 0.020% 99.951% ( 1300, 1900 ] 22 0.015% 99.966% ( 1900, 2900 ] 22 0.015% 99.981% ( 2900, 4400 ] 16 0.011% 99.991% ( 4400, 6600 ] 13 0.009% 100.000% ( 6600, 9900 ] 4 0.003% 100.003% ** Level 3 read latency histogram (micros): Count: 72771 Average: 21.9585 StdDev: 96.03 Min: 1 Median: 1.9705 Max: 4006 Percentiles: P50: 1.97 P75: 3.46 P99: 244.07 P99.9: 1177.70 P99.99: 2408.26 ------------------------------------------------------ [ 0, 1 ] 2276 3.128% 3.128% # ( 1, 2 ] 35148 48.299% 51.427% ########## ( 2, 3 ] 14543 19.985% 71.412% #### ( 3, 4 ] 5627 7.732% 79.144% ## ( 4, 6 ] 5523 7.590% 86.734% ## ( 6, 10 ] 1753 2.409% 89.143% ( 10, 15 ] 230 0.316% 89.459% ( 15, 22 ] 60 0.082% 89.541% ( 22, 34 ] 8 0.011% 89.552% ( 34, 51 ] 2 0.003% 89.555% ( 51, 76 ] 5 0.007% 89.562% ( 76, 110 ] 1535 2.109% 91.671% ( 110, 170 ] 4512 6.200% 97.871% # ( 170, 250 ] 887 1.219% 99.090% ( 250, 380 ] 230 0.316% 99.406% ( 380, 580 ] 205 0.282% 99.688% ( 580, 870 ] 107 0.147% 99.835% ( 870, 1300 ] 66 0.091% 99.926% ( 1300, 1900 ] 33 0.045% 99.971% ( 1900, 2900 ] 27 0.037% 100.008% ( 2900, 4400 ] 10 0.014% 100.022% ** Level 4 read latency histogram (micros): Count: 276912 Average: 47.9236 StdDev: 176.48 Min: 1 Median: 2.3833 Max: 8457 Percentiles: P50: 2.38 P75: 8.45 P99: 465.09 P99.9: 2164.59 P99.99: 3430.90 ------------------------------------------------------ [ 0, 1 ] 7430 2.683% 2.683% # ( 1, 2 ] 112981 40.800% 43.483% ######## ( 2, 3 ] 47077 17.001% 60.484% ### ( 3, 4 ] 18711 6.757% 67.241% # ( 4, 6 ] 18128 6.546% 73.788% # ( 6, 10 ] 5482 1.980% 75.767% ( 10, 15 ] 788 0.285% 76.052% ( 15, 22 ] 127 0.046% 76.098% ( 22, 34 ] 37 0.013% 76.111% ( 34, 51 ] 24 0.009% 76.120% ( 51, 76 ] 26 0.009% 76.129% ( 76, 110 ] 14054 5.075% 81.204% # ( 110, 170 ] 39497 14.263% 95.468% ### ( 170, 250 ] 7266 2.624% 98.092% # ( 250, 380 ] 1858 0.671% 98.763% ( 380, 580 ] 1544 0.558% 99.320% ( 580, 870 ] 760 0.274% 99.595% ( 870, 1300 ] 475 0.172% 99.766% ( 1300, 1900 ] 305 0.110% 99.876% ( 1900, 2900 ] 246 0.089% 99.965% ( 2900, 4400 ] 193 0.070% 100.035% ( 4400, 6600 ] 57 0.021% 100.056% ( 6600, 9900 ] 10 0.004% 100.059% ** Level 5 read latency histogram (micros): Count: 496467 Average: 168.9294 StdDev: 248.93 Min: 72 Median: 137.7198 Max: 11376 Percentiles: P50: 137.72 P75: 162.34 P99: 811.69 P99.9: 4130.44 P99.99: 6544.11 ------------------------------------------------------ ( 51, 76 ] 78 0.016% 0.016% ( 76, 110 ] 108423 21.839% 21.855% #### ( 110, 170 ] 302454 60.921% 82.776% ############ ( 170, 250 ] 55578 11.195% 93.971% ## ( 250, 380 ] 12872 2.593% 96.563% # ( 380, 580 ] 9317 1.877% 98.440% ( 580, 870 ] 3480 0.701% 99.141% ( 870, 1300 ] 1310 0.264% 99.405% ( 1300, 1900 ] 834 0.168% 99.573% ( 1900, 2900 ] 924 0.186% 99.759% ( 2900, 4400 ] 854 0.172% 99.931% ( 4400, 6600 ] 301 0.061% 99.992% ( 6600, 9900 ] 50 0.010% 100.002% ( 9900, 14000 ] 1 0.000% 100.002% ** DB Stats ** Uptime(secs): 32.8 total, 28.3 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 60.01 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:24.806 H:M:S, 75.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 34.82 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:24.258 H:M:S, 85.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 200.728 micros/op 19916 ops/sec 60.150 seconds 1197996 operations; 19.3 MB/s (296999 of 296999 found) Microseconds per read: Count: 1197996 Average: 200.7292 StdDev: 471.62 Min: 2 Median: 164.8790 Max: 70049 Percentiles: P50: 164.88 P75: 220.62 P99: 578.19 P99.9: 4653.53 P99.99: 21617.71 ------------------------------------------------------ ( 1, 2 ] 9 0.001% 0.001% ( 2, 3 ] 35 0.003% 0.004% ( 3, 4 ] 49 0.004% 0.008% ( 4, 6 ] 85 0.007% 0.015% ( 6, 10 ] 317 0.026% 0.041% ( 10, 15 ] 2985 0.249% 0.290% ( 15, 22 ] 2044 0.171% 0.461% ( 22, 34 ] 8157 0.681% 1.142% ( 34, 51 ] 909 0.076% 1.218% ( 51, 76 ] 98 0.008% 1.226% ( 76, 110 ] 27498 2.295% 3.521% ( 110, 170 ] 608771 50.816% 54.337% ########## ( 170, 250 ] 391213 32.656% 86.993% ####### ( 250, 380 ] 111937 9.344% 96.336% ## ( 380, 580 ] 32200 2.688% 99.024% # ( 580, 870 ] 7947 0.663% 99.688% ( 870, 1300 ] 1570 0.131% 99.819% ( 1300, 1900 ] 375 0.031% 99.850% ( 1900, 2900 ] 299 0.025% 99.875% ( 2900, 4400 ] 272 0.023% 99.898% ( 4400, 6600 ] 243 0.020% 99.918% ( 6600, 9900 ] 313 0.026% 99.944% ( 9900, 14000 ] 335 0.028% 99.972% ( 14000, 22000 ] 226 0.019% 99.991% ( 22000, 33000 ] 69 0.006% 99.997% ( 33000, 50000 ] 36 0.003% 100.000% ( 50000, 75000 ] 4 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 15182193 rocksdb.bloom.filter.full.positive COUNT : 1344590 rocksdb.bloom.filter.full.true.positive COUNT : 1197596 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 400 rocksdb.memtable.miss COUNT : 1197596 rocksdb.l0.hit COUNT : 46972 rocksdb.l1.hit COUNT : 23568 rocksdb.l2andup.hit COUNT : 1127056 rocksdb.compaction.key.drop.new COUNT : 102570 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2075990 rocksdb.number.keys.read COUNT : 1197996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2142421680 rocksdb.bytes.read COUNT : 1197996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1416 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 51899319 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 9591253729 rocksdb.compact.write.bytes COUNT : 9496604468 rocksdb.flush.write.bytes COUNT : 2113919119 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 63 rocksdb.number.superversion_acquires COUNT : 353 rocksdb.number.superversion_releases COUNT : 334 rocksdb.number.superversion_cleanups COUNT : 85 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2096794624 rocksdb.memtable.garbage.bytes.at.flush COUNT : 750592 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 14467364840 rocksdb.non.last.level.read.count COUNT : 3351880 rocksdb.block.checksum.compute.count COUNT : 3681777 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 163.944095 P95 : 360.840163 P99 : 577.797276 P100 : 70047.000000 COUNT : 1197996 SUM : 239321614 rocksdb.db.write.micros P50 : 1.637354 P95 : 5.399740 P99 : 1187.454636 P100 : 483163.000000 COUNT : 2075990 SUM : 57444400 rocksdb.compaction.times.micros P50 : 1506250.000000 P95 : 4410000.000000 P99 : 5486103.000000 P100 : 5486103.000000 COUNT : 19 SUM : 31827945 rocksdb.compaction.times.cpu_micros P50 : 981428.571429 P95 : 4410000.000000 P99 : 5446340.000000 P100 : 5446340.000000 COUNT : 19 SUM : 23175529 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 858.846154 P95 : 61250.000000 P99 : 99150.000000 P100 : 102559.000000 COUNT : 31 SUM : 190195 rocksdb.compaction.outfile.sync.micros P50 : 1787.500000 P95 : 150714.285714 P99 : 175729.000000 P100 : 175729.000000 COUNT : 85 SUM : 2247570 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 153.846154 P95 : 2650.000000 P99 : 4862.000000 P100 : 4862.000000 COUNT : 55 SUM : 29322 rocksdb.table.open.io.micros P50 : 7360.368664 P95 : 17624.778761 P99 : 21634.690265 P100 : 27730.000000 COUNT : 1416 SUM : 11356965 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.959441 P95 : 143.462423 P99 : 225.770409 P100 : 9476.000000 COUNT : 2325981 SUM : 52807622 rocksdb.read.block.get.micros P50 : 138.519315 P95 : 241.358849 P99 : 521.699177 P100 : 12097.000000 COUNT : 1354320 SUM : 203990257 rocksdb.write.raw.block.micros P50 : 0.572571 P95 : 1.771181 P99 : 3.649720 P100 : 13861.000000 COUNT : 2812534 SUM : 6909288 rocksdb.numfiles.in.singlecompaction P50 : 3.423077 P95 : 9.500000 P99 : 27.000000 P100 : 27.000000 COUNT : 25 SUM : 124 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.584046 P95 : 2240.755526 P99 : 2768.246440 P100 : 482984.000000 COUNT : 221322 SUM : 51913848 rocksdb.sst.read.micros P50 : 4.898305 P95 : 195.554111 P99 : 375.134227 P100 : 11376.000000 COUNT : 3351896 SUM : 242466364 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1197996 SUM : 1197996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2075990 SUM : 2142421680 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 221666.666667 P95 : 465500.000000 P99 : 753114.000000 P100 : 753114.000000 COUNT : 31 SUM : 7612009 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 92 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0