[01:14:35] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-ckdpqnzvgfonersl) [01:22:09] Project autotest-nightly build #405: STILL FAILING in 20 min. See https://ci.spdk.io/spdk-jenkins for results. [01:30:03] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [01:34:18] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 250 seconds) [02:03:35] *** Joins: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) [05:31:37] *** Joins: felipef (~felipef@62.254.189.133) [06:29:29] *** Quits: felipef (~felipef@62.254.189.133) (Remote host closed the connection) [07:43:46] FYI everyone... the 3 posts above are meant for entire SPDK community. There are no "assigned owners" to investigate these, please feel free to dig into any failure as they're great opportunities for learning and improving SPDK software quality [08:15:10] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Quit: Textual IRC Client: www.textualapp.com) [08:22:14] klateck: shouldn't we point to https://ci.spdk.io/spdk-jenkins/public_build/nightly_status.html ? [08:22:29] actually meh, that url is too long [08:22:43] we should shorten it :) [08:24:05] How about using goo.gl or tinyurl? Lol :) [08:24:13] hehe [08:24:33] klateck, does the STILL imply two nights in a row? [08:25:38] I don't see that described anywhere, but yeah I think so. Probably more than 2 fails in a row [08:26:07] *** Joins: lhodev (~lhodev@inet-hqmc06-o.oracle.com) [08:26:14] the last nightly run gives me 404 on the failed subjobs [08:26:30] "STILL FAILING" feels like guilt-shaming. "You still haven't fixed that, shame on you!" [08:26:30] cool, one more question as I look into these. Why are there 2 of the same job, 404 and 405? [08:26:31] and the run before that (#404 !) shows no subjobs at all [08:27:07] I Jenkins UI or on ci.spdk.io? [08:27:31] maybe 404 was aborted or something, looking in Jenkins now [08:27:41] I had to retrigger build #404 this morning. I amde a config mistake [08:27:48] klateck, I mean in the IRC posts above [08:28:04] You can see here that it only run for 1 minute, so it finished very quickly [08:28:55] darsto, where are the subjob logs missing? Jenkins UI or ci.spdk.io? [08:29:08] ci.spdk.io [08:29:12] https://ci.spdk.io/spdk-jenkins/public_build/autotest-nightly_405.html [08:29:20] looking in Jenkins it appears that 404 was 'aborted': 07:01:02 mv: cannot move 'spdkde976cf33180050fb9a6b2bcd73c17c8c3976b80.zip' to '/var/lib/jenkins/ci_repos': No such file or directory [08:29:30] (see console output for job) [08:29:31] you can see the failed subjobs have run for less than 30 seconds [08:30:00] peluse, yeah, 404 ran with configuration error [08:30:35] darsto looking into #405 logs now [08:31:18] I jsut looked at 405. Looks like a java exception error, no logs in Jenkins https://10.102.17.104:8080/job/NVMe_tests/job/nvme_autotest/18975/console [08:32:07] Yup. Seems I still have some error in autotest-nightly config. Artifact copying to ci.spdk.io looks fine. [08:33:18] OK, cool. Also, its not clear to me what triggered 405? I don't see anything on first glance that would do anything but run reset_job in a failure. How did the autotest job start 2 hours later after 404 failure? [08:33:40] Manually. I started it;) [08:33:48] ahhh :) [08:34:39] I wonder if we can do a post-processing script that can tell the difference between fully executed job (independent of test results) and an aborted job and retrigger if aborted.... that would be cool [08:35:19] Giving it one more shot. We've been working on test VM setups today; we tried not to interfere with the tests but it could be that we unintentionally broke the build [08:35:50] If it fails with any more strange java errors I'll dig in more into that tomorrow [08:45:09] Just got one "Unexpected termination of channel". This one is expected though. Just caught pniedzwx working manually on a VM system without taking it offline in Jenkins ;) [08:55:33] Hey @jimharris, I was configuring "vs DPDK" jobs today and have a question. When testing "SPDK master vs DPDK release", which DPDK release should we use? [08:55:39] Should this be actual last stable release from http://dpdk.org/git/dpdk-stable (that'd be 18.05.1) or latest from mainline repo (19.02)? [09:00:10] klateck: There's an email thread on that (subject: Add Jenkins jobs for periodic....). We clarified that it should be "Latest Major" instead of "Latest Stable", in which case Latest Major would be 19.02. [09:01:36] Project autotest-nightly build #406: STILL FAILING in 27 min. See https://ci.spdk.io/spdk-jenkins for results. [09:02:27] Thanks lhodev! Must have missed that [09:03:23] pniedzwx, shame on you :) [09:08:55] klateck: No prob. Thanks for tackling this for us! [09:32:15] klateck: 19.02 [09:46:46] *** Joins: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) [09:46:47] (spdk/master) ioat/perf: allow queue depths > 256 (Jim Harris) [09:46:47] Diff URL: https://github.com/spdk/spdk/compare/de976cf33180...5ec6b08eb03b [09:46:47] *** Parts: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) () [10:07:17] *** Joins: travis-ci (~travis-ci@ec2-54-159-63-236.compute-1.amazonaws.com) [10:07:17] (spdk/master) bdev/iscsi: remove unused master_ch (Jim Harris) [10:07:17] Diff URL: https://github.com/spdk/spdk/compare/5ec6b08eb03b...c7598147fffe [10:07:17] *** Parts: travis-ci (~travis-ci@ec2-54-159-63-236.compute-1.amazonaws.com) () [10:46:11] jimharris, do you think we need to add libpmem to pkgdep.sh w/the upcoming compression stuff? [10:47:18] i'm not sure it's broadly available yet [10:47:57] i think on ubuntu 18.10, i had to connect to a special package server [10:48:22] i'm fine with adding it with a "|| true" to avoid script failures [10:49:30] cool, yeah I just installed a fresh 18.04 and it was available with apt-get. I'm still having the failure I mentioned the other day building either with cryto or compress on 18.04, even a brand new fresh install. I'll dig in a little more, something fishy... [10:49:35] probably my mistake :) [12:20:59] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-ckdpqnzvgfonersl) (Quit: Connection closed for inactivity) [13:16:56] *** Quits: gila (~gila@5ED4D979.cm-7-5d.dynamic.ziggo.nl) (Quit: My Mac Pro has gone to sleep. ZZZzzz…) [13:23:22] *** Joins: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) [13:23:23] (spdk/master) scsi: Inline spdk_bdev_scsi_read/write into spdk_bdev_scsi_read_write (Shuhei Matsumoto) [13:23:23] Diff URL: https://github.com/spdk/spdk/compare/c7598147fffe...e7dc23696b9d [13:23:23] *** Parts: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) () [13:24:48] *** Joins: travis-ci (~travis-ci@ec2-34-230-20-236.compute-1.amazonaws.com) [13:24:49] (spdk/master) nvme/tcp: Implement nvme_tcp_qpair_fail function. (Ziye Yang) [13:24:50] Diff URL: https://github.com/spdk/spdk/compare/e7dc23696b9d...7bf5e1dee3cf [13:24:50] *** Parts: travis-ci (~travis-ci@ec2-34-230-20-236.compute-1.amazonaws.com) () [13:26:31] *** Joins: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) [13:26:32] (spdk/master) bdev/nvme: Enable the timeout function if timeout value is provided (Ziye Yang) [13:26:32] Diff URL: https://github.com/spdk/spdk/compare/7bf5e1dee3cf...73c5108684f2 [13:26:32] *** Parts: travis-ci (~travis-ci@ec2-54-235-60-214.compute-1.amazonaws.com) () [13:28:46] *** Joins: travis-ci (~travis-ci@ec2-54-198-113-213.compute-1.amazonaws.com) [13:28:47] (spdk/master) bdev/crypto: fix error path memory leak in driver init (paul luse) [13:28:47] Diff URL: https://github.com/spdk/spdk/compare/73c5108684f2...ba82b412cb94 [13:28:47] *** Parts: travis-ci (~travis-ci@ec2-54-198-113-213.compute-1.amazonaws.com) () [13:33:13] *** Joins: travis-ci (~travis-ci@ec2-54-197-99-91.compute-1.amazonaws.com) [13:33:14] (spdk/master) nvme/tcp: call nvme_ctrlr_add_process in construct function. (Ziye Yang) [13:33:14] Diff URL: https://github.com/spdk/spdk/compare/ba82b412cb94...14e1d0c74756 [13:33:14] *** Parts: travis-ci (~travis-ci@ec2-54-197-99-91.compute-1.amazonaws.com) () [13:52:49] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [13:57:27] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 268 seconds) [14:04:23] *** Joins: travis-ci (~travis-ci@ec2-3-94-191-56.compute-1.amazonaws.com) [14:04:24] (spdk/master) spdkcli: Skip refreshing node if spdkcli is run noninteractive (Pawel Kaminski) [14:04:24] Diff URL: https://github.com/spdk/spdk/compare/14e1d0c74756...6a35d0fd1384 [14:04:24] *** Parts: travis-ci (~travis-ci@ec2-3-94-191-56.compute-1.amazonaws.com) () [14:05:41] *** Joins: travis-ci (~travis-ci@ec2-34-230-20-236.compute-1.amazonaws.com) [14:05:42] (spdk/master) doc/lvol: clarify decouple only removes single dependency (Tomasz Zawadzki) [14:05:42] Diff URL: https://github.com/spdk/spdk/compare/6a35d0fd1384...9178235324be [14:05:42] *** Parts: travis-ci (~travis-ci@ec2-34-230-20-236.compute-1.amazonaws.com) () [14:54:06] *** Joins: travis-ci (~travis-ci@ec2-3-94-191-56.compute-1.amazonaws.com) [14:54:07] (spdk/master) nvme/tcp: remove the unnecessary active_r2t_reqs (Ziye Yang) [14:54:07] Diff URL: https://github.com/spdk/spdk/compare/a818023095d5...3a486ab6be5e [14:54:07] *** Parts: travis-ci (~travis-ci@ec2-3-94-191-56.compute-1.amazonaws.com) () [15:41:04] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [17:58:27] guys, I am very confused why I do not see significant difference when switching from nvme-cli to spdk [17:58:37] I used an enterprise drive [17:58:50] so the drive itself is NOT the limitation [17:59:41] I use the ./scripts/setup.sh to switch TO SPDK [18:00:10] and then use ./scripts/setup.sh reset to switch to NVMe-CLI [18:00:29] I can share some performance FIO numbers if needed [18:01:43] I am wondering if somehow my setup is incorrect [18:01:45] ? [18:01:57] if so...I would like to get some pointers on it [18:08:52] any ideas? [18:13:58] file1: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=32 fio-3.3 Starting 1 thread Starting SPDK v19.04-pre / DPDK 18.11.0 initialization... [ DPDK EAL parameters: fio --no-shconf -c 0x1 -m 0 --base-virtaddr=0x200000000000 --file-prefix=spdk_pid14169 ] EAL: Probing VFIO support... [18:14:22] the above tells me that I am indeed using SPDK [18:36:59] *** Joins: zhouhui (~wzh@114.255.44.140) [18:42:06] *** Quits: zhouhui (~wzh@114.255.44.140) (Quit: WeeChat 1.9.1) [18:47:22] *** Joins: zhouhui (~wzh@114.255.44.139) [18:49:29] *** Quits: zhouhui (~wzh@114.255.44.139) (Client Quit) [20:03:57] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [20:42:33] *** Joins: travis-ci (~travis-ci@ec2-54-173-246-246.compute-1.amazonaws.com) [20:42:34] (spdk/master) nvmf/tcp: remove the tqpair->group = NULL statement (Ziye Yang) [20:42:34] Diff URL: https://github.com/spdk/spdk/compare/3a486ab6be5e...a1c5442d166f [20:42:34] *** Parts: travis-ci (~travis-ci@ec2-54-173-246-246.compute-1.amazonaws.com) () [23:24:24] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [23:28:50] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 255 seconds) [23:42:10] *** Quits: lhodev (~lhodev@inet-hqmc06-o.oracle.com) (Remote host closed the connection) [23:42:47] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [23:47:04] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 244 seconds)