Invalid opts->opts_size 0 too small, please set opts_size correctly [2024-08-10 23:04:14.492785] Starting SPDK v24.09-pre git sha1 227b8322c / DPDK 22.11.4 initialization... [2024-08-10 23:04:14.493054] [ DPDK EAL parameters: rocksdb --no-shconf -c 0x1 --huge-unlink --no-telemetry --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=lib.power:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid1871594 ] Set seed to 1723323854486127 because --seed was 0 EAL: No free 2048 kB hugepages reported on node 1 [2024-08-10 23:04:14.592004] app.c: 910:spdk_app_start: *NOTICE*: Total cores available: 1 [2024-08-10 23:04:14.633444] app.c: 604:app_setup_trace: *NOTICE*: Tracepoint Group Mask 0x80 specified. [2024-08-10 23:04:14.633492] app.c: 605:app_setup_trace: *NOTICE*: Use 'spdk_trace -s rocksdb -p 1871594' to capture a snapshot of events at runtime. [2024-08-10 23:04:14.633503] app.c: 610:app_setup_trace: *NOTICE*: 'spdk_trace' without parameters will also work if this is the only [2024-08-10 23:04:14.633517] app.c: 611:app_setup_trace: *NOTICE*: SPDK application currently running. [2024-08-10 23:04:14.633525] app.c: 612:app_setup_trace: *NOTICE*: Or copy /dev/shm/rocksdb_trace.pid1871594 for offline analysis/debug. [2024-08-10 23:04:14.633651] reactor.c: 941:reactor_run: *NOTICE*: Reactor started on core 0 RocksDB: version 8.1.1 Date: Sat Aug 10 23:04:18 2024 CPU: 96 * Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz CPUCache: 36608 KB 2024/08/10-23:04:24 ... thread 0: (1000000,1000000) ops and (179497.5,179497.5) ops/second in (5.571108,5.571108) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 19/18 3.07 GB 6.3 0.7 0.0 0.7 1.6 0.9 0.0 1.9 135.0 292.3 5.60 5.14 17 0.329 767K 714 0.0 0.0 L1 18/18 1.59 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 14/0 945.07 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 160/0 9.60 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1104/0 90.80 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1315/36 105.98 GB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 135.0 292.3 5.60 5.14 17 0.329 767K 714 0.0 0.0 Int 0/0 0.00 KB 0.0 0.7 0.0 0.7 1.6 0.9 0.0 1.9 135.0 292.3 5.60 5.14 17 0.329 767K 714 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 0.7 0.0 0.7 0.7 -0.0 0.0 0.0 274.2 273.9 2.76 2.71 3 0.919 767K 714 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.9 0.9 0.0 0.0 0.0 310.2 2.84 2.43 14 0.203 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 5.8 total, 5.6 interval Flush(GB): cumulative 0.861, interval 0.861 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 1.60 GB write, 284.34 MB/s write, 0.74 GB read, 131.30 MB/s read, 5.6 seconds Interval compaction: 1.60 GB write, 293.80 MB/s write, 0.74 GB read, 135.66 MB/s read, 5.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 7, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 7, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 7, total-stops: 0, interval: 7 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 307479 Average: 18.6026 StdDev: 86.89 Min: 0 Median: 1.4109 Max: 8151 Percentiles: P50: 1.41 P75: 1.89 P99: 293.11 P99.9: 723.67 P99.99: 1119.79 ------------------------------------------------------ [ 0, 1 ] 87559 28.476% 28.476% ###### ( 1, 2 ] 161052 52.378% 80.855% ########## ( 2, 3 ] 20112 6.541% 87.396% # ( 3, 4 ] 6381 2.075% 89.471% ( 4, 6 ] 4002 1.302% 90.772% ( 6, 10 ] 1756 0.571% 91.343% ( 10, 15 ] 317 0.103% 91.447% ( 15, 22 ] 68 0.022% 91.469% ( 22, 34 ] 22 0.007% 91.476% ( 34, 51 ] 12 0.004% 91.480% ( 51, 76 ] 12 0.004% 91.484% ( 76, 110 ] 3794 1.234% 92.718% ( 110, 170 ] 13734 4.467% 97.184% # ( 170, 250 ] 4967 1.615% 98.800% ( 250, 380 ] 1858 0.604% 99.404% ( 380, 580 ] 1312 0.427% 99.831% ( 580, 870 ] 431 0.140% 99.971% ( 870, 1300 ] 102 0.033% 100.004% ( 1300, 1900 ] 51 0.017% 100.020% ( 1900, 2900 ] 39 0.013% 100.033% ( 2900, 4400 ] 18 0.006% 100.039% ( 4400, 6600 ] 13 0.004% 100.043% ( 6600, 9900 ] 4 0.001% 100.045% ** Level 1 read latency histogram (micros): Count: 5826 Average: 65.5874 StdDev: 199.32 Min: 1 Median: 5.0165 Max: 6685 Percentiles: P50: 5.02 P75: 101.75 P99: 689.83 P99.9: 2952.20 P99.99: 6685.00 ------------------------------------------------------ [ 0, 1 ] 20 0.343% 0.343% ( 1, 2 ] 351 6.025% 6.368% # ( 2, 3 ] 718 12.324% 18.692% ## ( 3, 4 ] 1176 20.185% 38.877% #### ( 4, 6 ] 1275 21.885% 60.762% #### ( 6, 10 ] 460 7.896% 68.658% ## ( 10, 15 ] 80 1.373% 70.031% ( 15, 22 ] 49 0.841% 70.872% ( 22, 34 ] 2 0.034% 70.906% ( 34, 51 ] 3 0.051% 70.958% ( 51, 76 ] 3 0.051% 71.009% ( 76, 110 ] 307 5.269% 76.279% # ( 110, 170 ] 861 14.779% 91.057% ### ( 170, 250 ] 291 4.995% 96.052% # ( 250, 380 ] 83 1.425% 97.477% ( 380, 580 ] 77 1.322% 98.798% ( 580, 870 ] 31 0.532% 99.331% ( 870, 1300 ] 15 0.257% 99.588% ( 1300, 1900 ] 13 0.223% 99.811% ( 1900, 2900 ] 5 0.086% 99.897% ( 2900, 4400 ] 5 0.086% 99.983% ( 6600, 9900 ] 1 0.017% 100.000% ** Level 3 read latency histogram (micros): Count: 777 Average: 215.6139 StdDev: 259.95 Min: 78 Median: 148.1481 Max: 2566 Percentiles: P50: 148.15 P75: 200.41 P99: 1784.50 P99.9: 2566.00 P99.99: 2566.00 ------------------------------------------------------ ( 76, 110 ] 131 16.860% 16.860% ### ( 110, 170 ] 405 52.124% 68.983% ########## ( 170, 250 ] 123 15.830% 84.813% ### ( 250, 380 ] 49 6.306% 91.120% # ( 380, 580 ] 26 3.346% 94.466% # ( 580, 870 ] 25 3.218% 97.683% # ( 870, 1300 ] 7 0.901% 98.584% ( 1300, 1900 ] 4 0.515% 99.099% ( 1900, 2900 ] 7 0.901% 100.000% ** Level 4 read latency histogram (micros): Count: 6983 Average: 235.2877 StdDev: 341.70 Min: 59 Median: 150.1072 Max: 4660 Percentiles: P50: 150.11 P75: 212.01 P99: 2095.96 P99.9: 4115.50 P99.99: 4660.00 ------------------------------------------------------ ( 51, 76 ] 3 0.043% 0.043% ( 76, 110 ] 1181 16.913% 16.955% ### ( 110, 170 ] 3452 49.434% 66.390% ########## ( 170, 250 ] 1145 16.397% 82.787% ### ( 250, 380 ] 453 6.487% 89.274% # ( 380, 580 ] 378 5.413% 94.687% # ( 580, 870 ] 143 2.048% 96.735% ( 870, 1300 ] 88 1.260% 97.995% ( 1300, 1900 ] 59 0.845% 98.840% ( 1900, 2900 ] 57 0.816% 99.656% ( 2900, 4400 ] 21 0.301% 99.957% ( 4400, 6600 ] 3 0.043% 100.000% ** Level 5 read latency histogram (micros): Count: 58510 Average: 244.9754 StdDev: 386.13 Min: 73 Median: 150.2699 Max: 6097 Percentiles: P50: 150.27 P75: 211.39 P99: 2461.87 P99.9: 4221.37 P99.99: 5743.27 ------------------------------------------------------ ( 51, 76 ] 12 0.021% 0.021% ( 76, 110 ] 9651 16.495% 16.515% ### ( 110, 170 ] 29191 49.891% 66.406% ########## ( 170, 250 ] 9719 16.611% 83.017% ### ( 250, 380 ] 3832 6.549% 89.566% # ( 380, 580 ] 2887 4.934% 94.500% # ( 580, 870 ] 1166 1.993% 96.493% ( 870, 1300 ] 566 0.967% 97.460% ( 1300, 1900 ] 534 0.913% 98.373% ( 1900, 2900 ] 653 1.116% 99.489% ( 2900, 4400 ] 273 0.467% 99.956% ( 4400, 6600 ] 33 0.056% 100.012% ** DB Stats ** Uptime(secs): 5.8 total, 5.6 interval Cumulative writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.96 GB, 170.91 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:2.018 H:M:S, 35.0 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 176.57 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:2.018 H:M:S, 36.2 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 2 num-running-flushes: 1 2024/08/10-23:04:36 ... thread 0: (1000000,2000000) ops and (79791.6,110474.3) ops/second in (12.532643,18.103751) seconds ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ L0 18/14 4.11 GB 194.6 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.2 288.1 18.56 17.49 40 0.464 3452K 6537 0.0 0.0 L1 18/18 1.59 GB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L3 14/0 945.07 MB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L4 160/0 9.60 GB 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 L5 1104/0 90.80 GB 0.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 Sum 1314/32 107.02 GB 0.0 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.2 288.1 18.56 17.49 40 0.464 3452K 6537 0.0 0.0 Int 0/0 0.00 KB 0.0 3.3 0.0 3.3 5.2 1.9 0.0 2.7 183.2 288.1 18.56 17.49 40 0.464 3452K 6537 0.0 0.0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 3.3 0.0 3.3 3.3 -0.0 0.0 0.0 274.5 273.9 12.39 12.20 9 1.377 3452K 6537 0.0 0.0 High 0/0 0.00 KB 0.0 0.0 0.0 0.0 1.9 1.9 0.0 0.0 0.0 316.6 6.17 5.29 31 0.199 0 0 0.0 0.0 Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 Uptime(secs): 18.3 total, 18.1 interval Flush(GB): cumulative 1.907, interval 1.907 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 5.22 GB write, 292.33 MB/s write, 3.32 GB read, 185.92 MB/s read, 18.6 seconds Interval compaction: 5.22 GB write, 295.33 MB/s write, 3.32 GB read, 187.82 MB/s read, 18.6 seconds Write Stall (count): cf-l0-file-count-limit-delays-with-ongoing-compaction: 25, cf-l0-file-count-limit-stops-with-ongoing-compaction: 0, l0-file-count-limit-delays: 25, l0-file-count-limit-stops: 0, memtable-limit-delays: 0, memtable-limit-stops: 0, pending-compaction-bytes-delays: 0, pending-compaction-bytes-stops: 0, total-delays: 25, total-stops: 0, interval: 25 total count ** File Read Latency Histogram By Level [default] ** ** Level 0 read latency histogram (micros): Count: 1051139 Average: 21.9005 StdDev: 74.10 Min: 0 Median: 1.4133 Max: 8151 Percentiles: P50: 1.41 P75: 1.91 P99: 267.58 P99.9: 574.98 P99.99: 793.01 ------------------------------------------------------ [ 0, 1 ] 307555 29.259% 29.259% ###### ( 1, 2 ] 527513 50.185% 79.444% ########## ( 2, 3 ] 57267 5.448% 84.892% # ( 3, 4 ] 18912 1.799% 86.691% ( 4, 6 ] 10784 1.026% 87.717% ( 6, 10 ] 3485 0.332% 88.049% ( 10, 15 ] 658 0.063% 88.111% ( 15, 22 ] 116 0.011% 88.123% ( 22, 34 ] 28 0.003% 88.125% ( 34, 51 ] 12 0.001% 88.126% ( 51, 76 ] 42 0.004% 88.130% ( 76, 110 ] 23363 2.223% 90.353% ( 110, 170 ] 69890 6.649% 97.002% # ( 170, 250 ] 20192 1.921% 98.923% ( 250, 380 ] 5995 0.570% 99.493% ( 380, 580 ] 4386 0.417% 99.910% ( 580, 870 ] 1138 0.108% 100.019% ( 870, 1300 ] 201 0.019% 100.038% ( 1300, 1900 ] 61 0.006% 100.044% ( 1900, 2900 ] 76 0.007% 100.051% ( 2900, 4400 ] 39 0.004% 100.055% ( 4400, 6600 ] 13 0.001% 100.056% ( 6600, 9900 ] 4 0.000% 100.056% ** Level 1 read latency histogram (micros): Count: 46123 Average: 29.8287 StdDev: 94.30 Min: 1 Median: 3.4251 Max: 6685 Percentiles: P50: 3.43 P75: 5.65 P99: 328.63 P99.9: 888.86 P99.99: 3316.31 ------------------------------------------------------ [ 0, 1 ] 695 1.507% 1.507% ( 1, 2 ] 9556 20.719% 22.225% #### ( 2, 3 ] 9640 20.901% 43.126% #### ( 3, 4 ] 7458 16.170% 59.296% ### ( 4, 6 ] 8803 19.086% 78.382% #### ( 6, 10 ] 2236 4.848% 83.230% # ( 10, 15 ] 458 0.993% 84.223% ( 15, 22 ] 257 0.557% 84.780% ( 22, 34 ] 25 0.054% 84.834% ( 34, 51 ] 5 0.011% 84.845% ( 51, 76 ] 5 0.011% 84.856% ( 76, 110 ] 1282 2.780% 87.635% # ( 110, 170 ] 3938 8.538% 96.173% ## ( 170, 250 ] 1132 2.454% 98.628% ( 250, 380 ] 284 0.616% 99.243% ( 380, 580 ] 229 0.496% 99.740% ( 580, 870 ] 73 0.158% 99.898% ( 870, 1300 ] 20 0.043% 99.941% ( 1300, 1900 ] 14 0.030% 99.972% ( 1900, 2900 ] 7 0.015% 99.987% ( 2900, 4400 ] 5 0.011% 99.998% ( 6600, 9900 ] 1 0.002% 100.000% ** Level 3 read latency histogram (micros): Count: 3218 Average: 175.9310 StdDev: 157.74 Min: 76 Median: 144.5474 Max: 2968 Percentiles: P50: 144.55 P75: 174.08 P99: 787.99 P99.9: 2583.14 P99.99: 2968.00 ------------------------------------------------------ ( 51, 76 ] 1 0.031% 0.031% ( 76, 110 ] 552 17.154% 17.185% ### ( 110, 170 ] 1834 56.992% 74.177% ########### ( 170, 250 ] 519 16.128% 90.305% ### ( 250, 380 ] 162 5.034% 95.339% # ( 380, 580 ] 92 2.859% 98.198% # ( 580, 870 ] 36 1.119% 99.316% ( 870, 1300 ] 9 0.280% 99.596% ( 1300, 1900 ] 5 0.155% 99.751% ( 1900, 2900 ] 7 0.218% 99.969% ( 2900, 4400 ] 1 0.031% 100.000% ** Level 4 read latency histogram (micros): Count: 28495 Average: 180.2818 StdDev: 189.96 Min: 59 Median: 144.4981 Max: 4660 Percentiles: P50: 144.50 P75: 175.71 P99: 837.72 P99.9: 2838.68 P99.99: 4510.37 ------------------------------------------------------ ( 51, 76 ] 3 0.011% 0.011% ( 76, 110 ] 5045 17.705% 17.715% #### ( 110, 170 ] 16000 56.150% 73.866% ########### ( 170, 250 ] 4526 15.883% 89.749% ### ( 250, 380 ] 1346 4.724% 94.473% # ( 380, 580 ] 1019 3.576% 98.049% # ( 580, 870 ] 305 1.070% 99.119% ( 870, 1300 ] 106 0.372% 99.491% ( 1300, 1900 ] 63 0.221% 99.712% ( 1900, 2900 ] 57 0.200% 99.912% ( 2900, 4400 ] 22 0.077% 99.989% ( 4400, 6600 ] 3 0.011% 100.000% ** Level 5 read latency histogram (micros): Count: 242469 Average: 182.9233 StdDev: 210.78 Min: 73 Median: 144.4885 Max: 6097 Percentiles: P50: 144.49 P75: 175.04 P99: 855.10 P99.9: 3212.44 P99.99: 4377.33 ------------------------------------------------------ ( 51, 76 ] 27 0.011% 0.011% ( 76, 110 ] 42518 17.535% 17.547% #### ( 110, 170 ] 136897 56.460% 74.006% ########### ( 170, 250 ] 38238 15.770% 89.776% ### ( 250, 380 ] 11431 4.714% 94.491% # ( 380, 580 ] 8558 3.530% 98.020% # ( 580, 870 ] 2504 1.033% 99.053% ( 870, 1300 ] 731 0.301% 99.355% ( 1300, 1900 ] 579 0.239% 99.593% ( 1900, 2900 ] 685 0.283% 99.876% ( 2900, 4400 ] 281 0.116% 99.992% ( 4400, 6600 ] 33 0.014% 100.005% ** DB Stats ** Uptime(secs): 18.3 total, 12.5 interval Cumulative writes: 0 writes, 2000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 1.92 GB, 107.62 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:10.895 H:M:S, 59.6 percent Interval writes: 0 writes, 1000K keys, 0 commit groups, 0.0 writes per commit group, ingest: 984.19 MB, 78.53 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Interval stall: 00:00:8.878 H:M:S, 70.8 percent Write Stall (count): write-buffer-manager-limit-stops: 0, num-running-compactions: 1 num-running-flushes: 0 using bdev Nvme0n1 Initializing RocksDB Options from the specified file Initializing RocksDB Options from command-line flags Integrated BlobDB: blob cache disabled Keys: 16 bytes each (+ 0 bytes user-defined timestamp) Values: 1000 bytes each (1000 bytes after compression) Entries: 100000000 Prefix: 0 bytes Keys per prefix: 0 RawSize: 96893.3 MB (estimated) FileSize: 96893.3 MB (estimated) Write rate: 0 bytes/second Read rate: 0 ops/second Compression: NoCompression Compression sampling rate: 0 Memtablerep: SkipListFactory Perf Level: 1 ------------------------------------------------ DB path: [/mnt/rocksdb] readwhilewriting : 207.158 micros/op 19293 ops/sec 60.175 seconds 1160996 operations; 18.7 MB/s (295999 of 295999 found) Microseconds per read: Count: 1160996 Average: 207.1596 StdDev: 261.24 Min: 2 Median: 174.1388 Max: 33494 Percentiles: P50: 174.14 P75: 233.24 P99: 643.63 P99.9: 3966.42 P99.99: 9759.85 ------------------------------------------------------ ( 1, 2 ] 7 0.001% 0.001% ( 2, 3 ] 35 0.003% 0.004% ( 3, 4 ] 63 0.005% 0.009% ( 4, 6 ] 95 0.008% 0.017% ( 6, 10 ] 640 0.055% 0.072% ( 10, 15 ] 5154 0.444% 0.516% ( 15, 22 ] 2066 0.178% 0.694% ( 22, 34 ] 10018 0.863% 1.557% ( 34, 51 ] 693 0.060% 1.617% ( 51, 76 ] 46 0.004% 1.621% ( 76, 110 ] 22124 1.906% 3.526% ( 110, 170 ] 519230 44.723% 48.249% ######### ( 170, 250 ] 392903 33.842% 82.091% ####### ( 250, 380 ] 149802 12.903% 94.994% ### ( 380, 580 ] 44305 3.816% 98.810% # ( 580, 870 ] 10050 0.866% 99.676% ( 870, 1300 ] 1737 0.150% 99.825% ( 1300, 1900 ] 288 0.025% 99.850% ( 1900, 2900 ] 306 0.026% 99.876% ( 2900, 4400 ] 384 0.033% 99.910% ( 4400, 6600 ] 616 0.053% 99.963% ( 6600, 9900 ] 332 0.029% 99.991% ( 9900, 14000 ] 57 0.005% 99.996% ( 14000, 22000 ] 33 0.003% 99.999% ( 22000, 33000 ] 11 0.001% 100.000% ( 33000, 50000 ] 1 0.000% 100.000% STATISTICS: rocksdb.block.cache.miss COUNT : 0 rocksdb.block.cache.hit COUNT : 0 rocksdb.block.cache.add COUNT : 0 rocksdb.block.cache.add.failures COUNT : 0 rocksdb.block.cache.index.miss COUNT : 0 rocksdb.block.cache.index.hit COUNT : 0 rocksdb.block.cache.index.add COUNT : 0 rocksdb.block.cache.index.bytes.insert COUNT : 0 rocksdb.block.cache.filter.miss COUNT : 0 rocksdb.block.cache.filter.hit COUNT : 0 rocksdb.block.cache.filter.add COUNT : 0 rocksdb.block.cache.filter.bytes.insert COUNT : 0 rocksdb.block.cache.data.miss COUNT : 0 rocksdb.block.cache.data.hit COUNT : 0 rocksdb.block.cache.data.add COUNT : 0 rocksdb.block.cache.data.bytes.insert COUNT : 0 rocksdb.block.cache.bytes.read COUNT : 0 rocksdb.block.cache.bytes.write COUNT : 0 rocksdb.bloom.filter.useful COUNT : 23369144 rocksdb.bloom.filter.full.positive COUNT : 1388697 rocksdb.bloom.filter.full.true.positive COUNT : 1160602 rocksdb.persistent.cache.hit COUNT : 0 rocksdb.persistent.cache.miss COUNT : 0 rocksdb.sim.block.cache.hit COUNT : 0 rocksdb.sim.block.cache.miss COUNT : 0 rocksdb.memtable.hit COUNT : 394 rocksdb.memtable.miss COUNT : 1160602 rocksdb.l0.hit COUNT : 48963 rocksdb.l1.hit COUNT : 18602 rocksdb.l2andup.hit COUNT : 1093037 rocksdb.compaction.key.drop.new COUNT : 40875 rocksdb.compaction.key.drop.obsolete COUNT : 0 rocksdb.compaction.key.drop.range_del COUNT : 0 rocksdb.compaction.key.drop.user COUNT : 0 rocksdb.compaction.range_del.drop.obsolete COUNT : 0 rocksdb.compaction.optimized.del.drop.obsolete COUNT : 0 rocksdb.compaction.cancelled COUNT : 0 rocksdb.number.keys.written COUNT : 2267522 rocksdb.number.keys.read COUNT : 1160996 rocksdb.number.keys.updated COUNT : 0 rocksdb.bytes.written COUNT : 2340082704 rocksdb.bytes.read COUNT : 1160996000 rocksdb.number.db.seek COUNT : 0 rocksdb.number.db.next COUNT : 0 rocksdb.number.db.prev COUNT : 0 rocksdb.number.db.seek.found COUNT : 0 rocksdb.number.db.next.found COUNT : 0 rocksdb.number.db.prev.found COUNT : 0 rocksdb.db.iter.bytes.read COUNT : 0 rocksdb.no.file.opens COUNT : 1355 rocksdb.no.file.errors COUNT : 0 rocksdb.stall.micros COUNT : 51463145 rocksdb.db.mutex.wait.micros COUNT : 0 rocksdb.number.multiget.get COUNT : 0 rocksdb.number.multiget.keys.read COUNT : 0 rocksdb.number.multiget.bytes.read COUNT : 0 rocksdb.number.merge.failures COUNT : 0 rocksdb.bloom.filter.prefix.checked COUNT : 0 rocksdb.bloom.filter.prefix.useful COUNT : 0 rocksdb.number.reseeks.iteration COUNT : 0 rocksdb.getupdatessince.calls COUNT : 0 rocksdb.wal.synced COUNT : 0 rocksdb.wal.bytes COUNT : 0 rocksdb.write.self COUNT : 0 rocksdb.write.other COUNT : 0 rocksdb.write.wal COUNT : 0 rocksdb.compact.read.bytes COUNT : 6193181918 rocksdb.compact.write.bytes COUNT : 6161842891 rocksdb.flush.write.bytes COUNT : 2312194440 rocksdb.compact.read.marked.bytes COUNT : 0 rocksdb.compact.read.periodic.bytes COUNT : 0 rocksdb.compact.read.ttl.bytes COUNT : 0 rocksdb.compact.write.marked.bytes COUNT : 0 rocksdb.compact.write.periodic.bytes COUNT : 0 rocksdb.compact.write.ttl.bytes COUNT : 0 rocksdb.number.direct.load.table.properties COUNT : 21 rocksdb.number.superversion_acquires COUNT : 335 rocksdb.number.superversion_releases COUNT : 313 rocksdb.number.superversion_cleanups COUNT : 80 rocksdb.number.block.compressed COUNT : 0 rocksdb.number.block.decompressed COUNT : 0 rocksdb.number.block.not_compressed COUNT : 0 rocksdb.merge.operation.time.nanos COUNT : 0 rocksdb.filter.operation.time.nanos COUNT : 0 rocksdb.row.cache.hit COUNT : 0 rocksdb.row.cache.miss COUNT : 0 rocksdb.read.amp.estimate.useful.bytes COUNT : 0 rocksdb.read.amp.total.read.bytes COUNT : 0 rocksdb.number.rate_limiter.drains COUNT : 0 rocksdb.number.iter.skip COUNT : 0 rocksdb.blobdb.num.put COUNT : 0 rocksdb.blobdb.num.write COUNT : 0 rocksdb.blobdb.num.get COUNT : 0 rocksdb.blobdb.num.multiget COUNT : 0 rocksdb.blobdb.num.seek COUNT : 0 rocksdb.blobdb.num.next COUNT : 0 rocksdb.blobdb.num.prev COUNT : 0 rocksdb.blobdb.num.keys.written COUNT : 0 rocksdb.blobdb.num.keys.read COUNT : 0 rocksdb.blobdb.bytes.written COUNT : 0 rocksdb.blobdb.bytes.read COUNT : 0 rocksdb.blobdb.write.inlined COUNT : 0 rocksdb.blobdb.write.inlined.ttl COUNT : 0 rocksdb.blobdb.write.blob COUNT : 0 rocksdb.blobdb.write.blob.ttl COUNT : 0 rocksdb.blobdb.blob.file.bytes.written COUNT : 0 rocksdb.blobdb.blob.file.bytes.read COUNT : 0 rocksdb.blobdb.blob.file.synced COUNT : 0 rocksdb.blobdb.blob.index.expired.count COUNT : 0 rocksdb.blobdb.blob.index.expired.size COUNT : 0 rocksdb.blobdb.blob.index.evicted.count COUNT : 0 rocksdb.blobdb.blob.index.evicted.size COUNT : 0 rocksdb.blobdb.gc.num.files COUNT : 0 rocksdb.blobdb.gc.num.new.files COUNT : 0 rocksdb.blobdb.gc.failures COUNT : 0 rocksdb.blobdb.gc.num.keys.relocated COUNT : 0 rocksdb.blobdb.gc.bytes.relocated COUNT : 0 rocksdb.blobdb.fifo.num.files.evicted COUNT : 0 rocksdb.blobdb.fifo.num.keys.evicted COUNT : 0 rocksdb.blobdb.fifo.bytes.evicted COUNT : 0 rocksdb.txn.overhead.mutex.prepare COUNT : 0 rocksdb.txn.overhead.mutex.old.commit.map COUNT : 0 rocksdb.txn.overhead.duplicate.key COUNT : 0 rocksdb.txn.overhead.mutex.snapshot COUNT : 0 rocksdb.txn.get.tryagain COUNT : 0 rocksdb.number.multiget.keys.found COUNT : 0 rocksdb.num.iterator.created COUNT : 0 rocksdb.num.iterator.deleted COUNT : 0 rocksdb.block.cache.compression.dict.miss COUNT : 0 rocksdb.block.cache.compression.dict.hit COUNT : 0 rocksdb.block.cache.compression.dict.add COUNT : 0 rocksdb.block.cache.compression.dict.bytes.insert COUNT : 0 rocksdb.block.cache.add.redundant COUNT : 0 rocksdb.block.cache.index.add.redundant COUNT : 0 rocksdb.block.cache.filter.add.redundant COUNT : 0 rocksdb.block.cache.data.add.redundant COUNT : 0 rocksdb.block.cache.compression.dict.add.redundant COUNT : 0 rocksdb.files.marked.trash COUNT : 0 rocksdb.files.deleted.immediately COUNT : 0 rocksdb.error.handler.bg.errro.count COUNT : 0 rocksdb.error.handler.bg.io.errro.count COUNT : 0 rocksdb.error.handler.bg.retryable.io.errro.count COUNT : 0 rocksdb.error.handler.autoresume.count COUNT : 0 rocksdb.error.handler.autoresume.retry.total.count COUNT : 0 rocksdb.error.handler.autoresume.success.count COUNT : 0 rocksdb.memtable.payload.bytes.at.flush COUNT : 2293371904 rocksdb.memtable.garbage.bytes.at.flush COUNT : 735232 rocksdb.secondary.cache.hits COUNT : 0 rocksdb.verify_checksum.read.bytes COUNT : 0 rocksdb.backup.read.bytes COUNT : 0 rocksdb.backup.write.bytes COUNT : 0 rocksdb.remote.compact.read.bytes COUNT : 0 rocksdb.remote.compact.write.bytes COUNT : 0 rocksdb.hot.file.read.bytes COUNT : 0 rocksdb.warm.file.read.bytes COUNT : 0 rocksdb.cold.file.read.bytes COUNT : 0 rocksdb.hot.file.read.count COUNT : 0 rocksdb.warm.file.read.count COUNT : 0 rocksdb.cold.file.read.count COUNT : 0 rocksdb.last.level.read.bytes COUNT : 0 rocksdb.last.level.read.count COUNT : 0 rocksdb.non.last.level.read.bytes COUNT : 12484223118 rocksdb.non.last.level.read.count COUNT : 2872578 rocksdb.block.checksum.compute.count COUNT : 2905134 rocksdb.multiget.coroutine.count COUNT : 0 rocksdb.blobdb.cache.miss COUNT : 0 rocksdb.blobdb.cache.hit COUNT : 0 rocksdb.blobdb.cache.add COUNT : 0 rocksdb.blobdb.cache.add.failures COUNT : 0 rocksdb.blobdb.cache.bytes.read COUNT : 0 rocksdb.blobdb.cache.bytes.write COUNT : 0 rocksdb.read.async.micros COUNT : 0 rocksdb.async.read.error.count COUNT : 0 rocksdb.secondary.cache.filter.hits COUNT : 0 rocksdb.secondary.cache.index.hits COUNT : 0 rocksdb.secondary.cache.data.hits COUNT : 0 rocksdb.table.open.prefetch.tail.miss COUNT : 0 rocksdb.table.open.prefetch.tail.hit COUNT : 0 rocksdb.db.get.micros P50 : 172.499054 P95 : 379.570835 P99 : 642.025794 P100 : 33493.000000 COUNT : 1160996 SUM : 239505299 rocksdb.db.write.micros P50 : 1.444107 P95 : 5.040338 P99 : 1149.612229 P100 : 2685.000000 COUNT : 2267522 SUM : 57034303 rocksdb.compaction.times.micros P50 : 1072500.000000 P95 : 2968577.000000 P99 : 2968577.000000 P100 : 2968577.000000 COUNT : 10 SUM : 13326466 rocksdb.compaction.times.cpu_micros P50 : 1072500.000000 P95 : 2938563.000000 P99 : 2938563.000000 P100 : 2938563.000000 COUNT : 10 SUM : 13120643 rocksdb.subcompaction.setup.times.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.sync.micros P50 : 725.000000 P95 : 1635.000000 P99 : 1635.000000 P100 : 1635.000000 COUNT : 35 SUM : 28023 rocksdb.compaction.outfile.sync.micros P50 : 851.875000 P95 : 2150.000000 P99 : 2647.000000 P100 : 2647.000000 COUNT : 30 SUM : 30696 rocksdb.wal.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.manifest.file.sync.micros P50 : 152.500000 P95 : 372.200000 P99 : 410.000000 P100 : 410.000000 COUNT : 46 SUM : 8253 rocksdb.table.open.io.micros P50 : 4332.471264 P95 : 8037.371134 P99 : 9881.288660 P100 : 16530.000000 COUNT : 1355 SUM : 6123587 rocksdb.db.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.block.compaction.micros P50 : 3.733778 P95 : 161.720237 P99 : 241.850163 P100 : 10334.000000 COUNT : 1502582 SUM : 51871440 rocksdb.read.block.get.micros P50 : 137.819782 P95 : 240.113379 P99 : 498.944462 P100 : 10912.000000 COUNT : 1401223 SUM : 203611420 rocksdb.write.raw.block.micros P50 : 0.551484 P95 : 1.618463 P99 : 3.708496 P100 : 3051.000000 COUNT : 2052354 SUM : 4330061 rocksdb.numfiles.in.singlecompaction P50 : 4.000000 P95 : 27.400000 P99 : 32.000000 P100 : 32.000000 COUNT : 11 SUM : 72 rocksdb.db.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.write.stall P50 : 0.627425 P95 : 2511.418199 P99 : 2682.000000 P100 : 2682.000000 COUNT : 146643 SUM : 51478870 rocksdb.sst.read.micros P50 : 94.802197 P95 : 209.541815 P99 : 382.009622 P100 : 8151.000000 COUNT : 2872650 SUM : 245215803 rocksdb.num.subcompactions.scheduled P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.per.read P50 : 1000.000000 P95 : 1000.000000 P99 : 1000.000000 P100 : 1000.000000 COUNT : 1160996 SUM : 1160996000 rocksdb.bytes.per.write P50 : 1032.000000 P95 : 1032.000000 P99 : 1032.000000 P100 : 1032.000000 COUNT : 2267522 SUM : 2340082704 rocksdb.bytes.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.compressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.bytes.decompressed P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.compression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.decompression.times.nanos P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.read.num.merge_operands P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.key.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.value.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.get.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.multiget.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.seek.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.next.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.prev.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.write.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.read.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.blob.file.sync.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.compression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.blobdb.decompression.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.db.flush.micros P50 : 211176.470588 P95 : 248235.294118 P99 : 332980.000000 P100 : 332980.000000 COUNT : 35 SUM : 6964806 rocksdb.sst.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.index.and.filter.blocks.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.sst.read.per.level P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.error.handler.autoresume.retry.count P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.poll.wait.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.prefetched.bytes.discarded P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 48 SUM : 0 rocksdb.multiget.io.batch.size P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.num.level.read.per.multiget P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.async.prefetch.abort.micros P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0 rocksdb.table.open.prefetch.tail.read.bytes P50 : 0.000000 P95 : 0.000000 P99 : 0.000000 P100 : 0.000000 COUNT : 0 SUM : 0