[09:38:42] *** Joins: bwalker (~bwalker@134.134.139.83)
[09:38:43] *** ChanServ sets mode: +o bwalker
[09:38:43] *** Server sets mode: +cnrt 
[09:38:43] *** Server sets mode: +cnrt 
[10:35:43] *** Joins: peluse (~peluse@134.134.139.83)
[10:35:44] *** ChanServ sets mode: +o peluse
[10:36:43] <peluse> I've seen a few patches unrelated to FreeBSD fail on FreeBSD, has anyone looked into that yet?
[10:38:10] <peluse> ex: FATAL: java.io.IOException: Unexpected termination of the channel
[10:39:15] <bwalker> that's just the same regular issue where the SSH connection drops
[10:39:24] <bwalker> due to excessive output or bad networking or something
[10:39:34] <jimharris> peluse: what's the last one you've seen that on?
[10:39:45] <jimharris> i've seen it with the reest test - even after we've disabled all of the output
[10:40:06] <peluse> I saw it on your nvme test patch, the one to do reset nightly and just now on my compression enable queing compression ops patch
[10:40:20] <peluse> intenral link on the last one: https://10.102.17.104:8080/job/autotest-per-patch/31017/
[10:44:15] <jimharris> hmmm -something's not working right
[10:44:29] <jimharris> see here: https://10.102.17.104:8080/job/autotest-per-patch/31011/
[10:44:39] <jimharris> the nvme test failed too, but it's still printing all of the output
[10:45:18] <jimharris> you know, i saw some test last week, where the failure seemed to indicate that it wasn't testing the actual patch, like it was testing against old code or something on the test system
[10:46:23] <peluse> hmmm...
[10:46:48] <peluse> you want me to look into the nvme one or are you digging in?
[10:47:07] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-gcjbzxvucrwodmfs) (Quit: Connection closed for inactivity)
[10:49:06] <jimharris> i'm looking into it
[10:58:50] <jimharris> sethhowe: i may need your help on this
[11:10:36] <sethhowe> jimharris: catching up on the conversation now. . .
[11:14:26] <sethhowe> I've had suspicions about patches testing old code in the past too.
[11:26:16] <sethhowe> jimharris: It's bizarre though. You can see for sure that the commit is correct by looking at the beginning of the log. One of the first things we do is run copy_or_clone.sh which copies down the source, unzips it and does a git log.
[12:35:58] <jimharris> i'm questioning qemu nvme reset emulation
[12:36:06] <jimharris> all of these failures are on the emulated nvme ssds
[12:36:16] <jimharris> nvme_autotest and freebsd_autotest
[12:36:18] <bwalker> hmm good observation
[12:36:21] <jimharris> nvme_phy_autotest never fails
[12:36:28] <bwalker> there is a very solid chance that qemu doesn't work right
[12:37:00] <bwalker> for reset, which is way out of the normal operation mode
[12:37:13] <jimharris> especially how this test is doing it
[12:37:18] <jimharris> submit 64 I/O then immediately reset
[12:37:27] <bwalker> yeah it's intentionally designed to break things
[12:37:54] <jimharris> i think we enable it per-patch but only for physical nvme ssds - but how do i detect that easily?
[12:38:37] <jimharris> oh - i wonder if qemu is crashing?
[12:38:56] <jimharris> that could explain the java exceptions
[12:39:23] <bwalker> interesting thought - we probably don't have much way to tell right now
[12:39:32] <bwalker> we'd need to leave a log on the physical machine somewhere
[12:40:40] <jimharris> i think i'll modify the reset test to check the device/vendor ID
[12:40:53] <jimharris> and just exit 0 if it's a QEMU SSD
[12:41:02] <jimharris> with a log message
[12:41:02] <bwalker> we should really get some infrastructure in place to detect QEMU crashes
[12:41:13] <bwalker> I wonder how often that occurs now that you've mentioned it
[12:43:02] <jimharris> i love being able to run all of these nvmf tests in isolation
[12:43:08] <bwalker> I know it's great
[12:43:32] <bwalker> as soon as we got that in place, we fixed a bunch of bugs and now we don't get anywhere near the number of bug reports
[12:43:46] <jimharris> i'm moving a lot more boilerplate stuff into the common.sh
[12:44:07] <jimharris> so far 46 insertions, 194 deletions
[12:44:14] <jimharris> to get it ready to run both rdma and tcp
[12:48:12] <jimharris> i don't think we have any systems testing SPDK_TEST_NVMF + SPDK_TEST_NVME_CLI
[13:22:07] <klateck> jimharris, bwalker - yes, you're right - the VMs are crashing from time to time and I can see that in the dmesg of the host servers. I've been trying to catch a coredump.
[13:22:29] <klateck> For some reason I have difficulties enabling them on Ubuntu 18 - tht's what host servers use
[13:27:32] <klateck> Here's whats visible in dmesg
[13:27:57] <klateck> Or maybe I'll send you a mail, it's too long to pase here
[16:37:15] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97)
[20:17:17] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds)
[22:34:00] <spdk-jenkins-bot> Project autotest-nightly build #501: STILL FAILING in 33 min. See https://dqtibwqq6s6ux.cloudfront.net for results.
[22:49:05] <spdk-jenkins-bot> Project autotest-nightly-failing build #357: STILL FAILING in 41 min. See https://dqtibwqq6s6ux.cloudfront.net for results.
[23:47:46] *** Joins: gila (~gila@94-212-217-121.cable.dynamic.v4.ziggo.nl)