[00:27:08] jimharris: since klateck is going on holiday next week, is there any chance we could merge env/config.mk changes this week? [00:27:14] https://review.gerrithub.io/c/spdk/spdk/+/426364 [00:27:44] we'd like to re-enable automatic jenkins tests vs upstream DPDK [00:28:51] they currently fail on nvme-cli compilation, which doesn't properly read WITH_DPDK_DIR variable [00:30:25] nvme-cli fixes for that were already merged, but we cannot update the nvme-cli on jenkins machines right now, because corresponding env/config.mk changes were merged in nvme-cli as well [00:54:49] *** Joins: tkulasek (~tkulasek@134.134.139.76) [01:03:12] *** Joins: klateck (~klateck@134.134.139.72) [01:24:18] *** Joins: mszwed (mszwed@nat/intel/x-fongsiczjdjgcjsa) [02:00:32] *** Joins: pniedzwx_ (86bfdc4a@gateway/web/freenode/ip.134.191.220.74) [02:10:39] *** Parts: pniedzwx_ (86bfdc4a@gateway/web/freenode/ip.134.191.220.74) () [02:18:02] *** Joins: pniedzwx__ (86bfdc4a@gateway/web/freenode/ip.134.191.220.74) [04:02:55] *** Quits: KipIngram (~kipingram@185.149.90.58) (Quit: WeeChat) [04:23:52] *** Joins: KipIngram (~kipingram@185.149.90.58) [05:14:20] chandler TP got really unstable recently [05:15:50] i see a multitude of different nvmf failures [05:37:25] KipIngram: you can specify a core mask to pick which cores drive I/O to the nvme devices, but it is not as granular as fio (i.e. you can pick which core for each device) [05:38:22] darsto: yes - i could merge the config.mk changes now although i was hoping to get a +1 from pawel first (since the latest versions were pushed by me) [05:41:10] darsto: i just pushed the config.mk patches to spdk master [05:47:29] darsto: what nvme-cli patches do we need to get in now? [05:58:11] *** Joins: travis-ci (~travis-ci@ec2-54-81-239-164.compute-1.amazonaws.com) [05:58:12] (spdk/master) lib/nvmf: add the nvmf qpair to the available poll group (GangCao) [05:58:12] Diff URL: https://github.com/spdk/spdk/compare/85bc2bbe7dce...98e119f7a98f [05:58:12] *** Parts: travis-ci (~travis-ci@ec2-54-81-239-164.compute-1.amazonaws.com) () [06:09:53] jimharris: great. all nvme-cli patches were already merged yesterday [06:10:52] i guess we can begin updating nvme-cli on test machines, can we? [06:56:43] *** Joins: travis-ci (~travis-ci@ec2-54-166-66-206.compute-1.amazonaws.com) [06:56:44] (spdk/master) dpdkbuild: add EXTRA_DPDK_CFLAGS (Jim Harris) [06:56:44] Diff URL: https://github.com/spdk/spdk/compare/98e119f7a98f...e66d09029bf8 [06:56:44] *** Parts: travis-ci (~travis-ci@ec2-54-166-66-206.compute-1.amazonaws.com) () [07:49:48] *** Quits: guerby (~guerby@april/board/guerby) (Ping timeout: 252 seconds) [07:59:09] *** Joins: guerby (~guerby@april/board/guerby) [08:18:23] *** Quits: pniedzwx__ (86bfdc4a@gateway/web/freenode/ip.134.191.220.74) (Quit: Page closed) [08:31:34] Does perf use the time stamp counter for its timing? [08:57:18] darsto: yeah - we're at the spdk developer meetup today and tomorrow, but maybe seth will be able to do some updates while he's here [10:57:33] *** Quits: tkulasek (~tkulasek@134.134.139.76) (Ping timeout: 245 seconds) [14:16:16] Afternoon guys. Ok, so I've only just started testing this, and I'm only a few minutes in. But still considerably beyond where I've typically seen things fall apart previously. I found a web link where someone was getting the same error messages in the log that I get when I run this test *without* spdk (with os drivers). They solved their problem by adding "nvme_core.io_timeout=255" to the [14:16:18] GRUB_CMDLINE_LINUX line in /etc/default/grub. [14:16:33] I tried it, and I'm five minutes into a test now, using spdk, whereas previously it's run less than a minute. [14:16:53] It seems like that would affect the OS driver only, though - do you guys "inherit" things like timeout values from the os driver? [14:17:29] Ok - never mind. LITERALLY seconds after I typed that it failed. [14:17:51] That's just how things go for me. That's the longest it's run all day. :-) [14:18:52] KipIngram: the parameter you specified would only be interpreted, if loaded, the kernel's nvme driver. [14:19:35] Ok, that's what it felt like to me too, and it wasn't "fixed" after all, so no mystery there. [14:19:55] Didn't fix the os driver based test either. [15:00:03] KipIngram: what's the failure you're seeing? [15:00:25] (responses may be delayed - in the SPDK developer meetup in Sunnyvale today) [15:29:36] *** Joins: darsto_ (~darsto@89-78-174-111.dynamic.chello.pl) [15:30:35] *** Quits: darsto (~darsto@89-78-174-111.dynamic.chello.pl) (Ping timeout: 250 seconds) [15:30:35] *** darsto_ is now known as darsto [15:38:02] When I run the test with os drivers, I get a lot of these in the log: [15:38:04] nvme nvme1: I/O 205 QID 1 timeout, aborting [15:38:19] paired with this later: [15:38:21] nvme nvme1: Abort status: 0x0 [15:38:28] A group of the first, followed by a group of the second. [15:38:55] When I run with spdk, I don't know where to find any logging; the fio real-time output line just "becomes corrupt," and if I let it time on down to zero the test doesn't stop. [15:39:03] So I never get any log files. [15:39:19] No clue what the performance actually is during that time - I could check on my storage box to see, though. [15:48:49] Those are in /var/log/messages [18:43:09] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 252 seconds) [21:54:03] *** Joins: travis-ci (~travis-ci@ec2-54-198-79-67.compute-1.amazonaws.com) [21:54:04] (spdk/master) Doc/Bdev/QoS: add the user information for the QoS on bdev section (GangCao) [21:54:04] Diff URL: https://github.com/spdk/spdk/compare/e66d09029bf8...136160ad463c [21:54:04] *** Parts: travis-ci (~travis-ci@ec2-54-198-79-67.compute-1.amazonaws.com) () [21:54:41] *** Joins: travis-ci (~travis-ci@ec2-54-221-118-45.compute-1.amazonaws.com) [21:54:42] (spdk/master) nvme: do not retry AER if ASYNC_LIMIT_EXCEEDED received (Jim Harris) [21:54:42] Diff URL: https://github.com/spdk/spdk/compare/136160ad463c...073f2dd8f28d [21:54:42] *** Parts: travis-ci (~travis-ci@ec2-54-221-118-45.compute-1.amazonaws.com) ()