[08:07:22] peluse: There were a couple of jobs that appeared "stuck" in the Jenkins queue. They appeared as "Incomplete", and their elapsed run time climbed absurdly. Ultimately, one of them got their status changed to "Not Run" after it had elapsed over 14 hours, or so it such appeared on the Jenkins CI GUI. Another one went over 15 hours. [08:08:28] For example: https://ci.spdk.io/spdk-jenkins/public_build/autotest-per-patch_17543.html [08:15:02] lhodev, OK thanks. I'll look into with some of the guys first thing Mon morn. We are *trying* to get the switch done by the end of this week. Appreciate the info [08:15:39] we had one problem last week, I'm not sure what day it was, that resulted in a halt of CI for quite some time, a network issue basically (sort of) :) [08:17:06] I share your eagerness for resolution. I'm yearning to get 18.10.1 out the door. [08:29:42] rock on! [08:41:30] peluse: wrt to the FIO/lvol failure to which you alluded, I'm trying to eyeball that in https://ci.spdk.io/spdk/builds/review/66fff3d99a35f383f3a7973997672a896f0a1e53.1544826616/fedora-03/build.log [08:42:42] OK... [08:42:43] There's so much output in the log and occasionally the appearance of the word "error" isn't a real error, per se. As such, I'm struggling to identify a search string that indicates the actual point of failure. [08:43:03] one sec, let me put it into something with line numbers and see if I can be of any help... [08:46:40] look for "Backtrace start" at line 27616 [08:47:07] then look below that at to see the fio command that failed on line 27625 [08:48:54] and 27408 looks like the fio error being reported (might be an earlier one too, I'm fighting a robot vacuum under my fight right now so am a bit distracted :) [08:49:20] fight=feet [08:50:14] At least you can blame a robot vacuum... I'm seeing the lines you're pointing to. These "failures" are unrelated to the actual patch which was just the SPDK's package spec file. [08:52:06] ahhh, yeah well that's unfortunate and another thing we need to get better at - tracking and root causing failures like these [08:52:14] On line 27398, I see: "fio: io_u error on file /dev/nvme0n5: Input/output error: write offset=10223616, buflen=262144" [08:52:18] that are unrelated to the submitters patch [08:52:25] Was that the actual failure? [08:52:30] I believe so, yes [08:52:45] fio was doing a verify job so it found that at that offset the wrong data was there [08:53:25] Well, at least I know that that I misconstrued the output at the bottom of the leg; i.e. it was NOT a failure during cleanup attempting to remove the uio_pci_generic module ;-) [08:53:45] leg? log [08:53:53] And, again, I don't even have a robot..... [08:54:05] I also am caffeine-deprived thus far this morning. [08:54:16] LOL [08:54:24] It is a Sunday morning. Maybe I should be having a mimosa. [08:55:05] hmmm, not a bad idea.... [08:57:55] Have you seen this type of fio verify error previously from our test runs? [09:00:05] I just realized the time, and I need to be somewhere in 1/2 hour. Catch you later. [12:00:41] lhodev, sorry was out on a trail run - gotta love AZ :) Can't say as I've seen that in the test pool with any frequency for sure [21:29:40] *** Joins: travis-ci (~travis-ci@ec2-54-205-113-181.compute-1.amazonaws.com) [21:29:41] (spdk/master) nvmf/tcp: Use generic transport options structure (Ziye Yang) [21:29:42] Diff URL: https://github.com/spdk/spdk/compare/c14dd64467fd...04d09f920731 [21:29:42] *** Parts: travis-ci (~travis-ci@ec2-54-205-113-181.compute-1.amazonaws.com) ()