[00:21:06] *** Joins: klateck (klateck@nat/intel/x-eiglnrmfxclkgvci) [00:21:10] *** Joins: mszwed (~mszwed@134.134.139.76) [01:03:33] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-lldrfkzddidcszjx) [02:29:58] *** Joins: michalwy (~michalwy@192.55.54.42) [02:42:20] *** Joins: pniedzwx (pniedzwx@nat/intel/x-eepwdudlqwhsyzis) [03:02:32] *** Joins: travis-ci (~travis-ci@ec2-184-72-93-125.compute-1.amazonaws.com) [03:02:33] (spdk/master) event: Move thread lib init/fini into reactor.c (Ben Walker) [03:02:33] Diff URL: https://github.com/spdk/spdk/compare/06cc80459fca...514358e5d372 [03:02:33] *** Parts: travis-ci (~travis-ci@ec2-184-72-93-125.compute-1.amazonaws.com) () [03:03:35] *** Joins: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) [06:56:13] *** Joins: sethhowe (sethhowe@nat/intel/x-locqfgtxqujmbixi) [09:31:15] *** Quits: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) () [09:48:12] *** Joins: travis-ci (~travis-ci@ec2-54-92-171-81.compute-1.amazonaws.com) [09:48:13] (spdk/master) test, nvmf: remove last usages of max_delay_us (Jim Harris) [09:48:13] Diff URL: https://github.com/spdk/spdk/compare/514358e5d372...20fba4d8867d [09:48:13] *** Parts: travis-ci (~travis-ci@ec2-54-92-171-81.compute-1.amazonaws.com) () [09:52:52] *** bwalker_ is now known as bwalker [10:10:14] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-lldrfkzddidcszjx) (Quit: Connection closed for inactivity) [10:12:50] jimharris: it would be much cleaner if we allocated a TAILQ pointer in the thread, but we don't currently make the details of the thread structure public [11:04:28] *** Joins: travis-ci (~travis-ci@ec2-54-242-186-138.compute-1.amazonaws.com) [11:04:29] (spdk/master) setup.sh: use vfio kernel driver in case of vfio no-IOMMU mode (Nikos Dragazis) [11:04:30] Diff URL: https://github.com/spdk/spdk/compare/20fba4d8867d...5e5acd3dbfb8 [11:04:30] *** Parts: travis-ci (~travis-ci@ec2-54-242-186-138.compute-1.amazonaws.com) () [11:19:29] *** Joins: travis-ci (~travis-ci@ec2-174-129-114-137.compute-1.amazonaws.com) [11:19:29] (spdk/master) bdev/aio: Reap completions from userspace if supported (Ben Walker) [11:19:30] Diff URL: https://github.com/spdk/spdk/compare/5e5acd3dbfb8...dadb9485854a [11:19:30] *** Parts: travis-ci (~travis-ci@ec2-174-129-114-137.compute-1.amazonaws.com) () [11:26:27] *** Joins: travis-ci (~travis-ci@ec2-54-224-202-7.compute-1.amazonaws.com) [11:26:28] (spdk/master) nvmf/tcp: Implement correct behavior of timeout for C2Htermreq case (Ziye Yang) [11:26:28] Diff URL: https://github.com/spdk/spdk/compare/dadb9485854a...2d0ce5b48b92 [11:26:28] *** Parts: travis-ci (~travis-ci@ec2-54-224-202-7.compute-1.amazonaws.com) () [11:28:19] *** Joins: travis-ci (~travis-ci@ec2-54-224-202-7.compute-1.amazonaws.com) [11:28:20] (spdk/master) nvme/perf: fix the execption case for NVMe qpair polling. (Ziye Yang) [11:28:20] Diff URL: https://github.com/spdk/spdk/compare/2d0ce5b48b92...a1c2b2d0a5b1 [11:28:20] *** Parts: travis-ci (~travis-ci@ec2-54-224-202-7.compute-1.amazonaws.com) () [11:40:55] *** Joins: travis-ci (~travis-ci@ec2-54-204-75-202.compute-1.amazonaws.com) [11:40:56] (spdk/master) bdev/crypto: fix unit test ASAN issue (paul luse) [11:40:57] Diff URL: https://github.com/spdk/spdk/compare/a1c2b2d0a5b1...467debe38240 [11:40:57] *** Parts: travis-ci (~travis-ci@ec2-54-204-75-202.compute-1.amazonaws.com) () [11:43:11] *** Joins: peluse (peluse@nat/intel/x-vodyuajkodygizfx) [11:43:11] *** ChanServ sets mode: +o peluse [11:55:03] *** Joins: travis-ci (~travis-ci@ec2-54-92-222-250.compute-1.amazonaws.com) [11:55:04] (spdk/master) test/unit/ftl: Add module unit tests for relocate (Wojciech Malikowski) [11:55:04] Diff URL: https://github.com/spdk/spdk/compare/467debe38240...64c30c0fe162 [11:55:04] *** Parts: travis-ci (~travis-ci@ec2-54-92-222-250.compute-1.amazonaws.com) () [13:18:47] https://review.gerrithub.io/c/spdk/spdk/+/444458 [14:14:32] jimharris: Will you be on tonight's SPDK community call? [14:26:24] *** Joins: travis-ci (~travis-ci@ec2-54-210-196-46.compute-1.amazonaws.com) [14:26:25] (spdk/master) scripts: Fix missing line break in gdb_macros.py (Ben Walker) [14:26:25] Diff URL: https://github.com/spdk/spdk/compare/64c30c0fe162...e7121a262f42 [14:26:25] *** Parts: travis-ci (~travis-ci@ec2-54-210-196-46.compute-1.amazonaws.com) () [14:37:07] peluse: I reverted those 2 crypto patches from our DPDK submodule, then run jenkins through it and everything passed [14:37:22] here https://review.gerrithub.io/c/spdk/dpdk/+/444379 and here https://review.gerrithub.io/c/spdk/spdk/+/444381 [14:38:20] could it be some crypto tests are disabled on Jenkins, or those patches are really not needed? [14:46:00] I think our tests have mostly stopped using secondary processes actually [14:46:07] that's probably what changed [15:27:19] lhodev: yes [16:03:44] *** Joins: travis-ci (~travis-ci@ec2-3-82-202-179.compute-1.amazonaws.com) [16:03:45] (spdk/master) scripts: Use range instead of xrange (shahar salzman) [16:03:45] Diff URL: https://github.com/spdk/spdk/compare/e7121a262f42...77a55a2fc4d6 [16:03:45] *** Parts: travis-ci (~travis-ci@ec2-3-82-202-179.compute-1.amazonaws.com) () [16:29:05] *** Joins: nate12112_spdk (6881c05f@gateway/web/freenode/ip.104.129.192.95) [16:29:44] When running fio with spdk as the io engine... [16:29:49] I see the following: [16:29:52] file1: (g=0): rw=read, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=spdk, iodepth=32 fio-3.3 Starting 1 thread Starting SPDK v19.04-pre / DPDK 18.11.0 initialization... [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 0 --base-virtaddr=0x200000000000 --file-prefix=spdk_pid20715 ] EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO s [16:30:05] it seems to just hang [16:30:10] never finishes [16:31:33] any ideas? [16:32:19] does the examples/nvme/perf/perf tool work? [16:32:48] running fio as root i presume [16:34:00] the example in example_config.fio works [16:34:07] and yes running as root [16:35:45] the example in spdk/examples/nvme/fio_plugin/example_config.fio works just fine [16:36:18] after adding filename entries for your controllers? [16:36:46] can you describe what's different between the fio config that's hanging, and what's in example_config.fio? [16:37:36] the config file that hangs [16:37:39] [global] name=fio-seq-reads filename=fio-seq-reads group_reporting=1 thread=1 norandommap=1 randrepeat=0 do_verify=0 rw=read bs=128K ba=4k direct=1 numjobs=1 log_avg_msec=1000 [file1] size=20G ioengine=/home/jenkins/Desktop/Git/spdk/examples/nvme/fio_plugin/fio_plugin iodepth=32 rwmixread=100 filename=trtype=PCIe traddr=0000.02.00.0 ns=1 [16:38:19] The example config that works: [16:38:20] [global] ioengine=/home/jenkins/Desktop/Git/spdk/examples/nvme/fio_plugin/fio_plugin thread=1 group_reporting=1 direct=1 verify=0 time_based=1 ramp_time=0 runtime=2 iodepth=128 rw=randrw bs=4k [test] numjobs=1 [16:38:40] the one that works is timed? [16:38:43] the example config doesn't have any ssds specified though [16:39:03] i don't remember what fio does in that case [16:39:07] I did it on command line [16:39:10] ah [16:39:11] ok [16:39:33] can you try a time-based test with your modified workload (seq 128kb reads)? [16:40:12] --filename="trtype=PCIe traddr=0000.02.0.0 ns=1" [16:40:18] ok [16:40:22] one sec [16:44:23] seems to be going futher...waiting for to see what it does [16:44:39] Starting 1 thread Starting SPDK v19.04-pre / DPDK 18.11.0 initialization... [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 0 --base-virtaddr=0x200000000000 --file-prefix=spdk_pid22955 ] EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... fio_plugin.c: 422:spdk_fio_setup: *ERROR*: Failed to find namespace 'ns=X' file1: Laying out IO file (1 fi [16:44:52] file1: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][98.4%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:02s] [16:45:20] what does the 98.4% mean? [16:45:26] 98.7% now [16:47:39] could you try your fio config file again, but take out the filename from the config file and pass it on the command line like you did with the example_config.fio? [16:48:17] I did that the first time...and decided to put it in the config file after it hung [16:48:29] 99.4% [16:48:41] ok - so it hangs the same way whether it's on the command line or in the config file? [16:49:24] the "Failed to find namespace" error message is the part I don't understand [16:49:50] same way regardless of where I put the --filename=trtype=PCIe traddr=0000.02.00.0 ns=1 [16:50:25] according to lhodev I can ignore that message [16:51:08] he apparently pushed a fix to it but I still the error msg [16:52:21] obs: 1 (f=1): [R(1)][99.6%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:02s] [16:52:37] what does the 99.6% represent? [16:52:57] usually the % of time towards completion [16:52:57] how far along is the .? [16:53:21] ok...I set it to 20G size [16:54:22] how long did it take to write 20G? [16:54:26] or read 20G [16:54:38] still going...? [16:54:48] 99.7% [16:55:26] hold on - i'm getting my system set up to try to duplicate what you're seeing [16:55:52] ok [16:55:58] thnx [17:02:36] im at 99.8%.....what do you see on your system? [17:06:46] i'm not seeing the same issue - i've tried to duplicate what i can see from the config you posted [17:06:57] could you file an issue in github for this? http://github.com/spdk/spdk/issues [17:07:21] ok.. [17:07:23] my guess is there's a configuration issue here but if you can post the full fio config file there it will make it easier to try to reproduce [17:12:08] so i'm not able to reproduce that "Failed to find namespace" error - but if I remove the "ns=1" from my config file, I get the "Laying out IO file" stuff and it hangs [17:12:56] it *looks* right from your post earlier, but if you could paste it directly into the github issue we can make sure [17:20:05] nate12112_spdk - i'm leaving the office now - once you've posted an issue on github, i'll take a further look (probably tomorrow) [17:21:49] ok thank you [17:33:37] Issue filed [17:33:38] using fio_plugin with spdk hangs when doing fio-seq-read (128K) #665 [18:33:32] *** Joins: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) [18:33:33] (spdk/master) nvmf: add get/set features with reservation notification mask support (Changpeng Liu) [18:33:34] Diff URL: https://github.com/spdk/spdk/compare/77a55a2fc4d6...d5b89466ccf9 [18:33:34] *** Parts: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) () [20:34:47] FYI: community meeting in less than 30 min from now.... https://spdk.io/community/ [20:53:08] *** Joins: travis-ci (~travis-ci@ec2-54-242-134-5.compute-1.amazonaws.com) [20:53:09] (spdk/master) nvme: add a wait for completion timeout API (Changpeng Liu) [20:53:09] Diff URL: https://github.com/spdk/spdk/compare/d5b89466ccf9...7d4d22a8465f [20:53:09] *** Parts: travis-ci (~travis-ci@ec2-54-242-134-5.compute-1.amazonaws.com) () [22:18:52] *** Joins: travis-ci (~travis-ci@ec2-3-82-5-232.compute-1.amazonaws.com) [22:18:53] (spdk/master) test/qos: allow more variance due to iSCSI overhead (Jim Harris) [22:18:53] Diff URL: https://github.com/spdk/spdk/compare/c217d7287d4c...5ab609b3bb8a [22:18:53] *** Parts: travis-ci (~travis-ci@ec2-3-82-5-232.compute-1.amazonaws.com) () [22:20:47] *** Joins: travis-ci (~travis-ci@ec2-54-242-134-5.compute-1.amazonaws.com) [22:20:48] (spdk/master) build: fix duplicated clean target in shared_lib/Makefile (Jim Harris) [22:20:48] Diff URL: https://github.com/spdk/spdk/compare/5ab609b3bb8a...14e443bdfbf9 [22:20:48] *** Parts: travis-ci (~travis-ci@ec2-54-242-134-5.compute-1.amazonaws.com) ()