[2024-07-15 20:19:55.195387] Starting SPDK v24.09-pre git sha1 a95bbf233 / DPDK 22.11.4 initialization... [2024-07-15 20:19:55.195691] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1261994 ] Set seed to 1721067595188994 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-07-15 20:19:55.292717] app.c: 909:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-07-15 20:19:55.334556] app.c: 603:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-07-15 20:19:55.334608] app.c: 604:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1261994' to capture a snapshot of events at runtime. [2024-07-15 20:19:55.334618] app.c: 609:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-07-15 20:19:55.334647] app.c: 610:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-07-15 20:19:55.334655] app.c: 611:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1261994 for offline analysis/debug. [2024-07-15 20:19:55.334741] reactor.c: 941:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Mon Jul 15 20:19:59 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/07/15-20:20:04 ... thread 0: (1000000,1000000) ops and (199536.8,199536.8) ops/second in (5.011606,5.011606) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/13 3.26 GB 88.1 0.7 0.0 0.7 1.6 0.9 0.0 1.9 133.8 289.8 5.65 5.14 17 0.332 767K 760 0.0 0.0 L1 3/3 243.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 63.03 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 14/0 972.12 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 162/0 9.49 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1116/0 91.78 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1314/16 105.79 GB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 133.8 289.8 5.65 5.14 17 0.332 767K 760 0.0 0.0 Int 0/0 0.00 KB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 133.8 289.8 5.65 5.14 17 0.332 767K 760 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.7 0.0 0.7 0.7 -0.0 0.0 0.0 273.0 272.7 2.77 2.71 3 0.923 767K 760 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 306.3 2.88 2.43 14 0.206 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.2 total, 5.0 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 1.60 GB write, 316.68 MB/s write, 0.74 GB read, 146.23 MB/s read, 5.6 seconds Interval compaction: 1.60 GB write, 326.58 MB/s write, 0.74 GB read, 150.81 MB/s read, 5.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 4, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 4, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 4, total-stops: 0, interval: 4 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 292956 Average: 16.4441 StdDev: 83.99 Min: 0 Median: 1.4563 Max: 7008 Percentiles: P50: 1.46 P75: 1.91 P99: 255.04 P99.9: 716.91 P99.99: 1227.09 ------------------------------------------------------ [ 0, 1 ] 73597 25.122% 25.122% ##### ( 1, 2 ] 159722 54.521% 79.643% ########### ( 2, 3 ] 23985 8.187% 87.830% ## ( 3, 4 ] 6966 2.378% 90.208% ( 4, 6 ] 4936 1.685% 91.893% ( 6, 10 ] 2112 0.721% 92.614% ( 10, 15 ] 324 0.111% 92.725% ( 15, 22 ] 84 0.029% 92.753% ( 22, 34 ] 39 0.013% 92.766% ( 34, 51 ] 10 0.003% 92.770% ( 51, 76 ] 21 0.007% 92.777% ( 76, 110 ] 2841 0.970% 93.747% ( 110, 170 ] 11156 3.808% 97.555% # ( 170, 250 ] 4177 1.426% 98.981% ( 250, 380 ] 1455 0.497% 99.477% ( 380, 580 ] 1078 0.368% 99.845% ( 580, 870 ] 339 0.116% 99.961% ( 870, 1300 ] 102 0.035% 99.996% ( 1300, 1900 ] 56 0.019% 100.015% ( 1900, 2900 ] 35 0.012% 100.027% ( 2900, 4400 ] 17 0.006% 100.033% ( 4400, 6600 ] 17 0.006% 100.039% ( 6600, 9900 ] 1 0.000% 100.039% ** Level 1 read latency histogram (micros): Count: 16523 Average: 18.0292 StdDev: 114.08 Min: 1 Median: 1.7313 Max: 5785 Percentiles: P50: 1.73 P75: 2.86 P99: 243.99 P99.9: 1183.47 P99.99: 5691.23 ------------------------------------------------------ [ 0, 1 ] 2435 14.737% 14.737% ### ( 1, 2 ] 7967 48.218% 62.955% ########## ( 2, 3 ] 2315 14.011% 76.965% ### ( 3, 4 ] 1240 7.505% 84.470% ## ( 4, 6 ] 785 4.751% 89.221% # ( 6, 10 ] 476 2.881% 92.102% # ( 10, 15 ] 50 0.303% 92.405% ( 15, 22 ] 30 0.182% 92.586% ( 22, 34 ] 10 0.061% 92.647% ( 34, 51 ] 5 0.030% 92.677% ( 76, 110 ] 155 0.938% 93.615% ( 110, 170 ] 690 4.176% 97.791% # ( 170, 250 ] 216 1.307% 99.098% ( 250, 380 ] 66 0.399% 99.498% ( 380, 580 ] 47 0.284% 99.782% ( 580, 870 ] 10 0.061% 99.843% ( 870, 1300 ] 13 0.079% 99.921% ( 1300, 1900 ] 5 0.030% 99.952% ( 1900, 2900 ] 2 0.012% 99.964% ( 2900, 4400 ] 2 0.012% 99.976% ( 4400, 6600 ] 4 0.024% 100.000% ** Level 2 read latency histogram (micros): Count: 104 Average: 205.2404 StdDev: 197.89 Min: 91 Median: 146.0000 Max: 1419 Percentiles: P50: 146.00 P75: 184.55 P99: 1282.80 P99.9: 1419.00 P99.99: 1419.00 ------------------------------------------------------ ( 76, 110 ] 16 15.385% 15.385% ### ( 110, 170 ] 60 57.692% 73.077% ############ ( 170, 250 ] 11 10.577% 83.654% ## ( 250, 380 ] 6 5.769% 89.423% # ( 380, 580 ] 5 4.808% 94.231% # ( 580, 870 ] 4 3.846% 98.077% # ( 870, 1300 ] 1 0.962% 99.038% ( 1300, 1900 ] 1 0.962% 100.000% ** Level 3 read latency histogram (micros): Count: 755 Average: 221.0821 StdDev: 248.28 Min: 82 Median: 152.3760 Max: 2179 Percentiles: P50: 152.38 P75: 213.57 P99: 1690.00 P99.9: 2179.00 P99.99: 2179.00 ------------------------------------------------------ ( 76, 110 ] 107 14.172% 14.172% ### ( 110, 170 ] 383 50.728% 64.901% ########## ( 170, 250 ] 140 18.543% 83.444% #### ( 250, 380 ] 53 7.020% 90.464% # ( 380, 580 ] 31 4.106% 94.570% # ( 580, 870 ] 16 2.119% 96.689% ( 870, 1300 ] 9 1.192% 97.881% ( 1300, 1900 ] 13 1.722% 99.603% ( 1900, 2900 ] 3 0.397% 100.000% ** Level 4 read latency histogram (micros): Count: 5823 Average: 246.3488 StdDev: 337.75 Min: 74 Median: 153.1783 Max: 4460 Percentiles: P50: 153.18 P75: 223.95 P99: 1920.26 P99.9: 4098.18 P99.99: 4460.00 ------------------------------------------------------ ( 51, 76 ] 5 0.086% 0.086% ( 76, 110 ] 880 15.112% 15.198% ### ( 110, 170 ] 2816 48.360% 63.558% ########## ( 170, 250 ] 988 16.967% 80.526% ### ( 250, 380 ] 426 7.316% 87.841% # ( 380, 580 ] 329 5.650% 93.491% # ( 580, 870 ] 153 2.628% 96.119% # ( 870, 1300 ] 100 1.717% 97.836% ( 1300, 1900 ] 67 1.151% 98.987% ( 1900, 2900 ] 38 0.653% 99.639% ( 2900, 4400 ] 19 0.326% 99.966% ( 4400, 6600 ] 2 0.034% 100.000% ** Level 5 read latency histogram (micros): Count: 49248 Average: 265.1913 StdDev: 428.06 Min: 75 Median: 152.8852 Max: 6748 Percentiles: P50: 152.89 P75: 220.71 P99: 2670.88 P99.9: 4274.21 P99.99: 5929.43 ------------------------------------------------------ ( 51, 76 ] 6 0.012% 0.012% ( 76, 110 ] 7275 14.772% 14.784% ### ( 110, 170 ] 24265 49.271% 64.055% ########## ( 170, 250 ] 8504 17.268% 81.323% ### ( 250, 380 ] 3422 6.949% 88.272% # ( 380, 580 ] 2520 5.117% 93.389% # ( 580, 870 ] 1165 2.366% 95.754% ( 870, 1300 ] 547 1.111% 96.865% ( 1300, 1900 ] 526 1.068% 97.933% ( 1900, 2900 ] 683 1.387% 99.320% ( 2900, 4400 ] 313 0.636% 99.955% ( 4400, 6600 ] 26 0.053% 100.008% ( 6600, 9900 ] 1 0.002% 100.010% ** DB Stats ** Uptime(secs): 5.2 total, 5.0 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 190.36 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:1.382 H:M:S, 26.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 196.28 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:1.382 H:M:S, 27.6 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 1 2024/07/15-20:20:14 ... thread 0: (1000000,2000000) ops and (104017.2,136748.3) ops/second in (9.613799,14.625405) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/17 4.24 GB 106.5 2.8 0.0 2.8 4.6 1.8 0.0 2.5 173.9 289.5 16.30 15.23 38 0.429 2877K 6308 0.0 0.0 L1 3/3 243.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L2 1/0 63.03 MB 0.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 14/0 972.12 MB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 162/0 9.49 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1116/0 91.78 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1314/20 106.76 GB 0.0 2.8 0.0 2.8 4.6 1.8 0.0 2.5 173.9 289.5 16.30 15.23 38 0.429 2877K 6308 0.0 0.0 Int 0/0 0.00 KB 0.0 2.8 0.0 2.8 4.6 1.8 0.0 2.5 173.9 289.5 16.30 15.23 38 0.429 2877K 6308 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 2.8 0.0 2.8 2.8 -0.0 0.0 0.0 274.6 274.0 10.32 10.13 8 1.290 2877K 6308 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 0.0 316.2 5.98 5.09 30 0.199 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 14.8 total, 14.6 interval Flush(GB): cumulative 1.846, interval 1.846 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 4.61 GB write, 319.11 MB/s write, 2.77 GB read, 191.70 MB/s read, 16.3 seconds Interval compaction: 4.61 GB write, 322.53 MB/s write, 2.77 GB read, 193.75 MB/s read, 16.3 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 15, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 15, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 15, total-stops: 0, interval: 15 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 919791 Average: 19.5738 StdDev: 72.81 Min: 0 Median: 1.4169 Max: 7008 Percentiles: P50: 1.42 P75: 1.90 P99: 249.83 P99.9: 576.98 P99.99: 810.16 ------------------------------------------------------ [ 0, 1 ] 260928 28.368% 28.368% ###### ( 1, 2 ] 477285 51.891% 80.259% ########## ( 2, 3 ] 56744 6.169% 86.428% # ( 3, 4 ] 16500 1.794% 88.222% ( 4, 6 ] 10564 1.149% 89.370% ( 6, 10 ] 3513 0.382% 89.752% ( 10, 15 ] 638 0.069% 89.822% ( 15, 22 ] 134 0.015% 89.836% ( 22, 34 ] 44 0.005% 89.841% ( 34, 51 ] 13 0.001% 89.842% ( 51, 76 ] 45 0.005% 89.847% ( 76, 110 ] 15566 1.692% 91.540% ( 110, 170 ] 52479 5.706% 97.245% # ( 170, 250 ] 16177 1.759% 99.004% ( 250, 380 ] 4670 0.508% 99.512% ( 380, 580 ] 3629 0.395% 99.906% ( 580, 870 ] 974 0.106% 100.012% ( 870, 1300 ] 169 0.018% 100.031% ( 1300, 1900 ] 68 0.007% 100.038% ( 1900, 2900 ] 81 0.009% 100.047% ( 2900, 4400 ] 31 0.003% 100.050% ( 4400, 6600 ] 17 0.002% 100.052% ( 6600, 9900 ] 1 0.000% 100.052% ** Level 1 read latency histogram (micros): Count: 25691 Average: 41.3531 StdDev: 120.14 Min: 1 Median: 2.4958 Max: 5785 Percentiles: P50: 2.50 P75: 5.95 P99: 419.90 P99.9: 1066.43 P99.99: 5186.99 ------------------------------------------------------ [ 0, 1 ] 2456 9.560% 9.560% ## ( 1, 2 ] 8781 34.179% 43.739% ####### ( 2, 3 ] 3244 12.627% 56.366% ### ( 3, 4 ] 3278 12.759% 69.125% ### ( 4, 6 ] 1549 6.029% 75.155% # ( 6, 10 ] 683 2.659% 77.813% # ( 10, 15 ] 91 0.354% 78.167% ( 15, 22 ] 52 0.202% 78.370% ( 22, 34 ] 15 0.058% 78.428% ( 34, 51 ] 6 0.023% 78.452% ( 76, 110 ] 791 3.079% 81.530% # ( 110, 170 ] 3204 12.471% 94.002% ## ( 170, 250 ] 969 3.772% 97.774% # ( 250, 380 ] 271 1.055% 98.828% ( 380, 580 ] 221 0.860% 99.689% ( 580, 870 ] 47 0.183% 99.872% ( 870, 1300 ] 16 0.062% 99.934% ( 1300, 1900 ] 5 0.019% 99.953% ( 1900, 2900 ] 4 0.016% 99.969% ( 2900, 4400 ] 4 0.016% 99.984% ( 4400, 6600 ] 4 0.016% 100.000% ** Level 2 read latency histogram (micros): Count: 380 Average: 179.0368 StdDev: 139.95 Min: 79 Median: 143.5455 Max: 1419 Percentiles: P50: 143.55 P75: 169.45 P99: 812.00 P99.9: 1419.00 P99.99: 1419.00 ------------------------------------------------------ ( 76, 110 ] 67 17.632% 17.632% #### ( 110, 170 ] 220 57.895% 75.526% ############ ( 170, 250 ] 51 13.421% 88.947% ### ( 250, 380 ] 15 3.947% 92.895% # ( 380, 580 ] 16 4.211% 97.105% # ( 580, 870 ] 9 2.368% 99.474% ( 870, 1300 ] 1 0.263% 99.737% ( 1300, 1900 ] 1 0.263% 100.000% ** Level 3 read latency histogram (micros): Count: 3051 Average: 181.0410 StdDev: 153.36 Min: 78 Median: 146.2931 Max: 2179 Percentiles: P50: 146.29 P75: 182.09 P99: 842.49 P99.9: 2137.25 P99.99: 2179.00 ------------------------------------------------------ ( 76, 110 ] 473 15.503% 15.503% ### ( 110, 170 ] 1740 57.030% 72.534% ########### ( 170, 250 ] 498 16.323% 88.856% ### ( 250, 380 ] 154 5.048% 93.904% # ( 380, 580 ] 122 3.999% 97.902% # ( 580, 870 ] 37 1.213% 99.115% ( 870, 1300 ] 10 0.328% 99.443% ( 1300, 1900 ] 13 0.426% 99.869% ( 1900, 2900 ] 4 0.131% 100.000% ** Level 4 read latency histogram (micros): Count: 22886 Average: 186.8954 StdDev: 197.19 Min: 74 Median: 145.6550 Max: 4460 Percentiles: P50: 145.66 P75: 184.24 P99: 961.05 P99.9: 2879.40 P99.99: 4378.35 ------------------------------------------------------ ( 51, 76 ] 6 0.026% 0.026% ( 76, 110 ] 4016 17.548% 17.574% #### ( 110, 170 ] 12488 54.566% 72.140% ########### ( 170, 250 ] 3678 16.071% 88.211% ### ( 250, 380 ] 1188 5.191% 93.402% # ( 380, 580 ] 972 4.247% 97.649% # ( 580, 870 ] 285 1.245% 98.895% ( 870, 1300 ] 114 0.498% 99.393% ( 1300, 1900 ] 74 0.323% 99.716% ( 1900, 2900 ] 43 0.188% 99.904% ( 2900, 4400 ] 20 0.087% 99.991% ( 4400, 6600 ] 2 0.009% 100.000% ** Level 5 read latency histogram (micros): Count: 192502 Average: 190.9571 StdDev: 237.39 Min: 74 Median: 145.6260 Max: 6748 Percentiles: P50: 145.63 P75: 182.57 P99: 1090.21 P99.9: 3552.68 P99.99: 4375.08 ------------------------------------------------------ ( 51, 76 ] 18 0.009% 0.009% ( 76, 110 ] 33042 17.164% 17.174% ### ( 110, 170 ] 106424 55.285% 72.458% ########### ( 170, 250 ] 31137 16.175% 88.633% ### ( 250, 380 ] 9840 5.112% 93.745% # ( 380, 580 ] 7436 3.863% 97.608% # ( 580, 870 ] 2343 1.217% 98.825% ( 870, 1300 ] 658 0.342% 99.167% ( 1300, 1900 ] 553 0.287% 99.454% ( 1900, 2900 ] 721 0.375% 99.829% ( 2900, 4400 ] 316 0.164% 99.993% ( 4400, 6600 ] 26 0.014% 100.006% ( 6600, 9900 ] 1 0.001% 100.007% ** DB Stats ** Uptime(secs): 14.8 total, 9.6 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 133.14 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:7.193 H:M:S, 48.7 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 102.37 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:5.811 H:M:S, 60.4 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 207.306 micros/op 19274 ops/sec 60.182 seconds 1159996 operations; 18.7 MB/s (300999 of 300999 found) Microseconds per read: Count: 1159996 Average: 207.3074 StdDev: 275.70 Min: 2 Median: 172.5207 Max: 38834 Percentiles: P50: 172.52 P75: 232.51 P99: 676.92 P99.9: 4226.75 P99.99: 10419.74 ------------------------------------------------------ ( 1, 2 ] 11 0.001% 0.001% ( 2, 3 ] 38 0.003% 0.004% ( 3, 4 ] 58 0.005% 0.009% ( 4, 6 ] 99 0.009% 0.018% ( 6, 10 ] 1204 0.104% 0.122% ( 10, 15 ] 4788 0.413% 0.534% ( 15, 22 ] 2090 0.180% 0.714% ( 22, 34 ] 11071 0.954% 1.669% ( 34, 51 ] 796 0.069% 1.738% ( 51, 76 ] 100 0.009% 1.746% ( 76, 110 ] 26241 2.262% 4.008% ( 110, 170 ] 521316 44.941% 48.949% ######### ( 170, 250 ] 386751 33.341% 82.290% ####### ( 250, 380 ] 144483 12.455% 94.746% ## ( 380, 580 ] 45599 3.931% 98.677% # ( 580, 870 ] 11224 0.968% 99.644% ( 870, 1300 ] 2041 0.176% 99.820% ( 1300, 1900 ] 328 0.028% 99.848% ( 1900, 2900 ] 330 0.028% 99.877% ( 2900, 4400 ] 303 0.026% 99.903% ( 4400, 6600 ] 634 0.055% 99.958% ( 6600, 9900 ] 366 0.032% 99.989% ( 9900, 14000 ] 71 0.006% 99.995% ( 14000, 22000 ] 39 0.003% 99.999% ( 22000, 33000 ] 12 0.001% 100.000% ( 33000, 50000 ] 3 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 22093658 rocksdb.bloom.filter.full.positive COUNT : 1375320 rocksdb.bloom.filter.full.true.positive COUNT : 1159584 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 412 rocksdb.memtable.miss COUNT : 1159584 rocksdb.l0.hit COUNT : 55450 rocksdb.l1.hit COUNT : 2820 rocksdb.l2andup.hit COUNT : 1101314 rocksdb.compaction.key.drop.new COUNT : 33603 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2855198 rocksdb.number.keys.read COUNT : 1159996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2946564336 rocksdb.bytes.read COUNT : 1159996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1366 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 48657661 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 7279630913 rocksdb.compact.write.bytes COUNT : 7253295867 rocksdb.flush.write.bytes COUNT : 2906746125 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 4 rocksdb.number.superversion_acquires COUNT : 424 rocksdb.number.superversion_releases COUNT : 400 rocksdb.number.superversion_cleanups COUNT : 101 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2883093504 rocksdb.memtable.garbage.bytes.at.flush COUNT : 933888 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 13664464847 rocksdb.non.last.level.read.count COUNT : 3156823 rocksdb.block.checksum.compute.count COUNT : 3154095 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 170.909184 P95 : 390.774476 P99 : 675.316089 P100 : 38832.000000 COUNT : 1159996 SUM : 239512453 rocksdb.db.write.micros P50 : 1.509967 P95 : 5.281283 P99 : 947.275570 P100 : 2561.000000 COUNT : 2855198 SUM : 55887865 rocksdb.compaction.times.micros P50 : 1081000.000000 P95 : 3843519.000000 P99 : 3843519.000000 P100 : 3843519.000000 COUNT : 13 SUM : 18908115 rocksdb.compaction.times.cpu_micros P50 : 1081000.000000 P95 : 3810243.000000 P99 : 3810243.000000 P100 : 3810243.000000 COUNT : 13 SUM : 18607598 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 730.800000 P95 : 1570.000000 P99 : 1616.000000 P100 : 1616.000000 COUNT : 44 SUM : 35670 rocksdb.compaction.outfile.sync.micros P50 : 1002.307692 P95 : 2466.666667 P99 : 2626.000000 P100 : 2626.000000 COUNT : 26 SUM : 30662 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 159.285714 P95 : 390.000000 P99 : 1050.600000 P100 : 1071.000000 COUNT : 58 SUM : 11312 rocksdb.table.open.io.micros P50 : 4590.690209 P95 : 8221.612903 P99 : 11906.266667 P100 : 18486.000000 COUNT : 1366 SUM : 6350146 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.504116 P95 : 161.125815 P99 : 241.518596 P100 : 11107.000000 COUNT : 1765003 SUM : 54655286 rocksdb.read.block.get.micros P50 : 138.657591 P95 : 243.489370 P99 : 522.075743 P100 : 15317.000000 COUNT : 1387735 SUM : 204967316 rocksdb.write.raw.block.micros P50 : 0.544536 P95 : 1.501763 P99 : 3.227070 P100 : 2227.000000 COUNT : 2460535 SUM : 5001216 rocksdb.numfiles.in.singlecompaction P50 : 4.000000 P95 : 16.000000 P99 : 16.000000 P100 : 16.000000 COUNT : 14 SUM : 69 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.574333 P95 : 2037.722870 P99 : 2556.000000 P100 : 2556.000000 COUNT : 245680 SUM : 48679866 rocksdb.sst.read.micros P50 : 82.549402 P95 : 209.211935 P99 : 397.191987 P100 : 7008.000000 COUNT : 3156842 SUM : 249301856 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1159996 SUM : 1159996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2855198 SUM : 2946564336 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 211904.761905 P95 : 249619.047619 P99 : 322402.000000 P100 : 322402.000000 COUNT : 44 SUM : 8855852 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 55 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0