[04:21:11] *** Joins: felipef (~felipef@62.254.189.133) [04:35:38] *** Quits: felipef (~felipef@62.254.189.133) (Remote host closed the connection) [07:38:58] *** Joins: travis-ci (~travis-ci@ec2-54-196-126-137.compute-1.amazonaws.com) [07:38:59] (spdk/master) nvme: Modify the memory barrier mode to improve performance (h00448672) [07:38:59] Diff URL: https://github.com/spdk/spdk/compare/9e2eb8cb51a1...dc61452d6ede [07:38:59] *** Parts: travis-ci (~travis-ci@ec2-54-196-126-137.compute-1.amazonaws.com) () [09:35:17] *** Joins: travis-ci (~travis-ci@ec2-34-229-140-143.compute-1.amazonaws.com) [09:35:17] (spdk/master) nvmf: trivial optimization to make the code more consistent (Changpeng Liu) [09:35:17] Diff URL: https://github.com/spdk/spdk/compare/dc61452d6ede...68bb3995aa37 [09:35:17] *** Parts: travis-ci (~travis-ci@ec2-34-229-140-143.compute-1.amazonaws.com) () [09:56:47] *** Joins: travis-ci (~travis-ci@ec2-52-205-255-77.compute-1.amazonaws.com) [09:56:47] (spdk/master) NVMF: Add model number as parameter to construct_nvmf_subsystem (-d option). (Gregory Shapiro) [09:56:47] Diff URL: https://github.com/spdk/spdk/compare/68bb3995aa37...14032a984cfe [09:56:47] *** Parts: travis-ci (~travis-ci@ec2-52-205-255-77.compute-1.amazonaws.com) () [10:11:53] *** Joins: travis-ci (~travis-ci@ec2-54-83-64-201.compute-1.amazonaws.com) [10:11:54] (spdk/master) bdev: temporarily allow bdev descriptors to be closed from any thread (Darek Stojaczyk) [10:11:54] Diff URL: https://github.com/spdk/spdk/compare/14032a984cfe...283abcb9a244 [10:11:54] *** Parts: travis-ci (~travis-ci@ec2-54-83-64-201.compute-1.amazonaws.com) () [10:13:37] *** Joins: travis-ci (~travis-ci@ec2-3-83-66-245.compute-1.amazonaws.com) [10:13:37] (spdk/master) env_dpdk: always print on stderr (Jörg Thalheim) [10:13:37] Diff URL: https://github.com/Mic92/spdk/compare/0c3069545dec...1859fe8e2cf5 [10:13:37] *** Parts: travis-ci (~travis-ci@ec2-3-83-66-245.compute-1.amazonaws.com) () [10:47:26] jrlusby: that datain code is used for all read operations - even reads that are smaller than max segment length [10:47:45] it makes sure we don't have too many outstanding reads open at once for any one connection [11:04:07] *** Joins: travis-ci (~travis-ci@ec2-3-87-174-24.compute-1.amazonaws.com) [11:04:07] (spdk/master) nvmf: add tcp trace points for data read from socket (Jim Harris) [11:04:07] Diff URL: https://github.com/spdk/spdk/compare/283abcb9a244...b92c3d412dc8 [11:04:07] *** Parts: travis-ci (~travis-ci@ec2-3-87-174-24.compute-1.amazonaws.com) () [11:09:31] jimharris: I was thinking about how to solve the issue with clearing locked_operation_in_progress in _spdk_bs_snapshot_unfreeze_cpl()... One way to do that is to check ctx->bserrno against -EBUSY, but that may be a bit risky, because someone may use this return code for something else in future. Second option would be to add that clearing everywhere we use _spdk_bs_clone_snapshot_origblob_cleanup except where [11:09:34] it is actually in use by other operation. 3rd option would be to add field to ctx structure saying something like do_not_clear_in_use_flag and add that to cases where locked_operation_in_progress has been set by other operation, and check that flag later in _spdk_bs_snapshot_unfreeze_cpl. Any thougths on that? [11:11:10] i think it needs to be done in the main code path - not the cleanup function - so something more like your option #2 [11:24:05] ok, thanks [11:43:08] *** Joins: travis-ci (~travis-ci@ec2-54-83-64-201.compute-1.amazonaws.com) [11:43:08] (spdk/master) bdevperf: Use new zcopy API for reads (Ben Walker) [11:43:08] Diff URL: https://github.com/spdk/spdk/compare/b92c3d412dc8...14bf912df9de [11:43:08] *** Parts: travis-ci (~travis-ci@ec2-54-83-64-201.compute-1.amazonaws.com) () [12:13:24] jimharris, let me know when you have a sec, on that crypto test failure I have narrowed it down to the patch where it started failing and the only diff between it and the prev commit is a dpdk submodule change but I can't for the life of me find the commit that made that change.... [13:01:45] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [13:05:52] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 245 seconds) [13:24:45] are we using nfs from the ci vms? [13:24:54] https://dqtibwqq6s6ux.cloudfront.net/results/autotest-per-patch/builds/29089/archive/unittest_autotests/dmesg.log [13:26:35] https://github.com/spdk/spdk/issues/765 [13:27:28] *** Joins: travis-ci (~travis-ci@ec2-54-166-36-203.compute-1.amazonaws.com) [13:27:28] (spdk/master) nbd: Remove unused spdk_event forward declaration (Ben Walker) [13:27:28] Diff URL: https://github.com/spdk/spdk/compare/14bf912df9de...598795ba9740 [13:27:28] *** Parts: travis-ci (~travis-ci@ec2-54-166-36-203.compute-1.amazonaws.com) () [13:36:34] peluse: could you fix up https://review.gerrithub.io/c/spdk/spdk/+/451064? this is pretty low hanging fruit we can get in for 19.04 [13:36:49] bwalker posted another comment on it this morning [13:36:55] jimharris, ya [13:40:52] jimharris, I just added a comment if you can take a quick look before I push it [13:43:14] it's trivial to print the string error. SPDK_ERRLOG("My error code is %d and the string is %s\n", rc, spdk_strerror(-rc)); [13:45:14] OK, will fix it in just this one place for now and update the rest later to be consistent [13:46:16] sounds good to me [13:46:24] don't worry about the other ones unless you have spare time [13:46:33] jimharris, found the patch with bisect. now need to figure out what's behind the actual dpdk change [13:46:36] but please remember to change the other ones at some point [13:50:55] Yeah, I will. [13:51:13] trying to make my patches smaller for you guys :) [13:51:23] its up there BTW [13:52:08] damn it, I only did the first one. one sec [13:53:56] OK, *now* its done [13:55:35] +2 [13:55:42] me too [13:57:54] sethhowe: working on this rocksdb patch with the commit id - i think we'll do this in three phases [13:58:05] phase 1 - commit the patch that just adds a file with the rocksdb commit id [13:58:21] phase 2 - you or Karol do the Jenkins updates to put that rev of rocksdb somewhere on the test system [13:58:40] phase 3 - commit a patch that runs the rocksdb tests based on the directory used in phase 2 [13:59:58] right now the scripts use /usr/src/rocksdb - i think you should pick a different directory for phase 2, just so that we don't screw up the existing tests while we work this out [14:04:11] jimharris, on the aesni thing I see the change, they introduced that new file and use it conditionally in the makefile but I haven't seen what the changes are between it and the original. Hope its just a matter of applying to changes to this new compat file [14:05:15] ohhh, its based on the version of the ipsec lib [14:05:42] which I have a patch out to update that fails that I haven't investigated yet, guess I'll start there [14:06:42] but we basically need the same patch for both versions? [14:06:49] how much of those two files are the same? [14:13:45] I haven't compared them yet but just ran the test with .52 ipseclib and it works. For < .52, which is what we were, it was using the new compat file w/o the patch. For .52 its using the old file that does have the patch [14:22:10] jimharris, yeah the compat file is new but based on the old version but they didn't have our patch in there yet to the compat file doesn't have the patch. Not a ton of changes to support or use new features of .52 but on first glance I don't think we care about them. I'll look closer though [14:22:50] and of course we applied the patch to the file of the same name so its there when ipsec is .5 or greater [14:58:06] jimharris, bwalker - I just updated the crypto issue with details, I do believe this is critical for 19.04 though. It is not an ASAN error as I think most were assuming. Crypto is broken with the new DPDK unless we make one of two changes I describe in the issue [14:58:07] https://github.com/spdk/spdk/issues/710 [14:59:33] leaning toward upgrading ipsec [15:06:07] *** Joins: pniedzwx_ (~pniedzwx_@89-64-58-51.dynamic.chello.pl) [15:07:01] i'm fine with upgrading ipsec [15:07:47] i'm not following though how our automated tests didn't catch this? [15:10:05] haven't investigated that yet. [15:10:27] *** Quits: pniedzwx_ (~pniedzwx_@89-64-58-51.dynamic.chello.pl) (Ping timeout: 245 seconds) [15:10:57] so upgrading ipsec exposes a mem leak. there were a few changes darek had to make to the crypto driver to suport 19.02, I'll go look at those and see what might have been introduced there maybe [15:11:33] ...BTW I believe the github issue does mention that the tests are failing in CI just somehow not getting picked up or at least it was brought up in bug scrub this morning [15:12:37] FYI I confirmed the leak was not there prior to 19.02 [15:15:23] blockdev tests in CI seem to show crypto running [15:15:36] is it possible the test pool is only testing QAT? [15:15:48] vbdev_crypto.c:1511:vbdev_crypto_claim: *NOTICE*: QAT in use: Required alignment set to 9 [15:23:22] https://review.gerrithub.io/c/spdk/spdk/+/451784 [15:23:31] i'll bet this fails the test pool :) [15:24:16] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [15:33:00] jimharris, that's likely it, I'm working on the meleak now so haven't looked yet [15:33:23] mszwed: i think i have an easier way for you to handle these locked flag cleanups [15:33:32] if so, we need to update the tests to run both QAT and aesni not one or the other, I believe crypto_gen.sh picks one or the other based on presence of QAT [15:33:47] see my patch link above :) [15:34:16] i changed it to create two crypto devs on two malloc bdevs [15:34:27] jimharris: I'm listening :) [15:34:28] one will always be aesni, the other will be either aesni or qat [15:34:38] set a flag in the ctx as well [15:34:45] LOL [15:35:06] meaning when you set the locked flag on the blob, also set a flag on the ctx [15:35:32] then when you clean up the context, you'll know whether or not you need to clear the blob's locked flag [15:36:00] jimharris: sounds good. Clearing stays in the function as previously? [15:36:17] *in the smae [15:36:24] **same [15:36:44] i think so - kind of like you had it before [15:36:51] ok [15:37:27] i'm still reviewing the patch, but after seeing how many places you have to set the flag back to false, we will surely miss one of them now or at some point in the future [15:38:38] in _spdk_bs_delete_open_cpl(), you should also wait to set the locked flag until much later - then you avoid several cases where you have to clear it in error cases [15:38:41] i'll make a note on gerrit [15:39:06] yeah, exactly... that is why I wanted to do this in clearing function, but missed the fact, that I can clear there flag set by other operation :/ [15:42:40] no - I mean in _spdk_bs_delete_open_cpl() - in these error cases you do not call the cleanup function, so you must either clear the flag explicitly, or wait to set the flag until you've checked for these error cases [15:43:06] jimharris: this function changes a bit in next patch, but thic check can be moved there as well [15:43:50] I was refering to previous topic with my previous sentence - not to _spdk_bs_delete_open_cpl comment - i'm writing to slow ;) [15:46:35] Hi, in the CHANGELOG, I found NVMe-oF target and nvmf section in SPDK 19.04. nvmf section should be merged into NVMe-oF target ? [15:48:24] oh yes - you are right Shuhei [15:48:41] OK I'll do that, thank you. [15:55:07] jimharris, OK, found the mem leak. one thing was missed when updating crypto to 19.02, an uninit call that causes the aesni_driver to free a new mem struct introduced at the same time [16:43:08] *** Joins: travis-ci (~travis-ci@ec2-3-82-205-200.compute-1.amazonaws.com) [16:43:09] (spdk/master) blobfs: track num outstanding requests per channel (Jim Harris) [16:43:09] Diff URL: https://github.com/spdk/spdk/compare/ae693db98605...b282c927ea15 [16:43:09] *** Parts: travis-ci (~travis-ci@ec2-3-82-205-200.compute-1.amazonaws.com) () [16:48:48] *** Joins: travis-ci (~travis-ci@ec2-3-81-231-125.compute-1.amazonaws.com) [16:48:48] (spdk/master) test/rocksdb: add rocksdb_commit_id file (Jim Harris) [16:48:48] Diff URL: https://github.com/spdk/spdk/compare/b282c927ea15...7a7cf7ba9474 [16:48:48] *** Parts: travis-ci (~travis-ci@ec2-3-81-231-125.compute-1.amazonaws.com) () [17:40:24] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [17:42:19] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [17:44:59] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 255 seconds) [21:43:20] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [21:48:01] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 258 seconds) [22:23:05] Project autotest-nightly build #474: STILL FAILING in 23 min. See https://dqtibwqq6s6ux.cloudfront.net for results. [22:35:03] Project autotest-nightly-failing build #338: STILL FAILING in 35 min. See https://dqtibwqq6s6ux.cloudfront.net for results. [23:19:18] *** Joins: pniedzwx_ (~pniedzwx_@host-185-93-94-213.ip-point.pl) [23:23:01] *** Quits: pniedzwx_ (~pniedzwx_@host-185-93-94-213.ip-point.pl) (Remote host closed the connection) [23:23:26] *** Joins: pniedzwx_ (~pniedzwx_@host-185-93-94-213.ip-point.pl) [23:28:03] *** Quits: pniedzwx_ (~pniedzwx_@host-185-93-94-213.ip-point.pl) (Ping timeout: 258 seconds)