[00:14:08] *** Quits: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) (Read error: Connection reset by peer) [00:22:40] *** Joins: tkulasek (tkulasek@nat/intel/x-snvwxucicrevhpvg) [00:33:43] *** Quits: guerby (~guerby@april/board/guerby) (Read error: Connection reset by peer) [00:33:59] *** Joins: guerby (~guerby@april/board/guerby) [00:36:32] *** Joins: guerby_ (~guerby@ip165.tetaneutral.net) [00:36:53] *** Quits: guerby (~guerby@april/board/guerby) (Read error: Connection reset by peer) [00:37:31] *** guerby_ is now known as guerby [00:37:49] *** Quits: guerby (~guerby@ip165.tetaneutral.net) (Changing host) [00:37:49] *** Joins: guerby (~guerby@april/board/guerby) [00:39:24] *** Quits: guerby (~guerby@april/board/guerby) (Excess Flood) [00:39:41] *** Joins: guerby (~guerby@ip165.tetaneutral.net) [00:39:41] *** Quits: guerby (~guerby@ip165.tetaneutral.net) (Changing host) [00:39:41] *** Joins: guerby (~guerby@april/board/guerby) [01:51:39] *** Quits: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) (Quit: Leaving.) [01:52:21] *** Joins: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) [02:00:52] *** Quits: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) (Quit: Leaving.) [02:01:58] *** Joins: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) [03:50:36] *** Quits: dlw (~Thunderbi@114.255.44.143) (Ping timeout: 268 seconds) [05:07:31] *** Joins: lyan (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) [05:07:55] *** lyan is now known as Guest36165 [05:14:55] *** Quits: Guest36165 (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) (Remote host closed the connection) [05:37:17] *** Joins: lyan_ (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) [06:01:27] *** Joins: philipp-sk (~Philipp@ktnron0916w-lp130-04-70-51-156-4.dsl.bell.ca) [06:10:35] *** Quits: lyan_ (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) (Remote host closed the connection) [06:19:01] *** Joins: lyan (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) [06:19:14] *** Quits: philipp-sk (~Philipp@ktnron0916w-lp130-04-70-51-156-4.dsl.bell.ca) (Ping timeout: 276 seconds) [06:19:25] *** lyan is now known as Guest41094 [07:12:47] *** Quits: Guest41094 (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) (Remote host closed the connection) [07:25:54] *** Joins: lyan (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) [07:26:18] *** lyan is now known as Guest75326 [07:30:13] *** Joins: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) [08:15:23] *** Joins: bwalker (bwalker@nat/intel/x-ntyfgkymnslhbuzm) [08:15:23] *** Server sets mode: +cnt [08:15:23] *** Server sets mode: +cnt [08:15:28] *** Joins: jstern (~jstern@134.134.139.72) [08:16:24] *** Joins: cunyinch (cunyinch@nat/intel/x-cayotlahxnnorirj) [08:19:00] *** Joins: kjakimia (kjakimia@nat/intel/x-drzkotgwydwxkhsl) [08:21:31] *** Joins: pbshah1 (pbshah1@nat/intel/x-hstceafgagwtyntr) [08:22:27] *** Joins: pniedzwx (pniedzwx@nat/intel/x-jkruvajuiexiiszd) [08:26:00] *** Joins: ziyeyang (ziyeyang@nat/intel/x-uugsztrurtctwujc) [08:31:34] *** Joins: ppelplin (~ppelplin@134.134.139.72) [08:36:38] *** Joins: klateck (klateck@nat/intel/x-lsfhksizbbfrrjme) [08:38:25] darsto: Which limit(s) do you think specifically I should be examining? A man on getrlimit suggests that perhaps I should focus on the values of RLIMIT_MEMLOCK and RLIMIT_AS. [08:40:17] *** Joins: qdai2 (qdai2@nat/intel/x-aqmodeepgpgwtbfn) [08:41:12] *** Joins: vermavis (vermavis@nat/intel/x-ujoievgnpzwzabbv) [08:41:43] *** Joins: changpe1 (changpe1@nat/intel/x-vtmcwgoquabnatel) [08:49:51] *** Joins: peluse (~peluse@134.134.139.72) [08:50:09] *** ChanServ sets mode: +o peluse [08:50:15] *** Joins: pwodkowx (~pwodkowx@134.134.139.72) [08:50:51] *** Joins: pzedlews (pzedlews@nat/intel/x-vhnzlligbwfinjgl) [08:52:47] *** Joins: lgalkax (lgalkax@nat/intel/x-vqsndwlbstkpqqap) [08:54:18] *** Joins: tsg (~tsg@134.134.139.72) [08:55:19] *** Joins: jimharris (~jimharris@134.134.139.72) [08:55:20] *** ChanServ sets mode: +o jimharris [08:55:48] *** Joins: pawelkax (pawelkax@nat/intel/x-eoslezkehaxutdgd) [08:57:54] *** Joins: gangcao (gangcao@nat/intel/x-ihnhrrpurolokmpb) [09:34:28] *** Joins: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) [09:34:29] (spdk/master) ut/ioat: drop legacy mocks (Dariusz Stojaczyk) [09:34:30] Diff URL: https://github.com/spdk/spdk/compare/71098fd4899d...893f26e84b72 [09:34:30] *** Parts: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) () [09:43:39] *** Joins: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) [09:43:40] (spdk/master) bdev/rbd: change the poller to timer poller (Ziye Yang) [09:43:40] Diff URL: https://github.com/spdk/spdk/compare/893f26e84b72...abc3bc4f8e07 [09:43:40] *** Parts: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) () [10:11:24] *** Quits: Guest75326 (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) (Remote host closed the connection) [10:24:15] *** Joins: lyan (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) [10:24:39] *** lyan is now known as Guest53095 [10:27:57] Back to debugging the mmap() failure during init of hugepages for nvmf_tgt. An inspection of the rlimits revealed that the soft and hard size for memlock was 65536. So, I altered that to unlimited:unlimited (for my shell) and verified the change both with prlimit and "cat /proc/$$/limits". Still, I'm seeing the same failure. [10:28:21] When I launch nvmf_tgt, as a child process, it should inherit the rlimits from my parent shell, right? [10:29:14] you're not using sudo or anything, right? [10:30:42] bwalker: correct. I am running directly in a bash shell as root. No sudo. [10:31:03] I was able to do a "prlimit -p " against the running nvmf_tgt, and verified that it DID inherit the rlimits appropriately. [10:31:52] The AS limit is unlimited as are DATA, MEMLOCK and RSS. That would seem sufficient, right? Is there something else? [10:36:50] Attempting to divine their actual build, per their config.h, it looks like they are pointing to DPDK based on 17.11 for their SPDK 18.04. At least, that's how it appears. Dunno why they'd point to a DPDK 17.11. Could that be the issue? [10:37:15] I think 17.11 is the version of DPDK that SPDK 18.04 released with [10:38:18] we did an 18.04.1 I think that bumps it to DPDK 18.05 - either that or we're still in the process of doing one [10:38:47] but generally speaking, no - 17.11 is fine [10:39:43] bwalker: Yeah? On that note, what's the best way to determine that? That is, if I checkout to the 18.04 tag of the SPDK and then do a "git submodule update", how do I determine the DPDK version? When I examined the DPDK hash and compared it to that of the hashes associated with the tags via the git reflog, the topmost hash of DPDK does *not* match any of the hashes shown for each of the DPDK tags. [10:41:15] we just have a couple patches on top of their release each time [10:41:31] if you do a git log [10:41:39] and walk by like ~6-7 patches, you'll see the tag [10:42:06] I think the 18.04.1 bumps DPDK to 18.02 actually - not 18.05 [10:44:27] the submodule is a fork of DPDK (github.com/spdk/dpdk), so the hashes won't match upstream DPDK [10:47:52] *** Joins: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) [10:47:53] (spdk/master) iscsi, initiator: solve the use after issue. (Ziye Yang) [10:47:53] Diff URL: https://github.com/spdk/spdk/compare/2d7cfda2d413...75dce8a25c8c [10:47:53] *** Parts: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) () [10:48:22] drv: Thx Daniel for the fork/hash comment. [10:48:24] *** Joins: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) [10:48:25] (spdk/master) doc/jsonrpc: Add RPCs for logical volume store (Shuhei Matsumoto) [10:48:25] Diff URL: https://github.com/spdk/spdk/compare/75dce8a25c8c...de5e709cfc6c [10:48:25] *** Parts: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) () [10:49:57] lhodev: maybe there's an inaccessible hugetlbfs mount on your system [10:50:08] is it always the same page that fails to mmap? [10:50:10] bwalker: It *appears* to me that SPDK v18.04 (not, v18.04.1) is relying on DPDK 18.02 if I'm trying to follow your recommendation of following the git log in the DPDK, but maybe I overlooked something. [10:50:50] darsto: when I do a "mount | grep huge", I only see one hugetlbfs. [10:52:01] darsto: I'm re-running with strace to see if it's the same page....back to you shortly. [10:53:28] darsto: Previously, I'd see the mmap failure on page 7553. On this most recent run, it died at page 7566. So, it wasn't the identical page number, but "in the neighborhood". [10:54:54] was this mount mounted automatically by linux? [10:56:09] *** Quits: tkulasek (tkulasek@nat/intel/x-snvwxucicrevhpvg) (Ping timeout: 256 seconds) [10:57:52] darsto: I re-ran, and it died this time at exactly the same page number (7566). I'm attempting to see how they mount the hugetlbfs. [10:58:15] what's the actual error msg? [10:58:33] it seems to me like DPDK should handle that MAP_FAILED just fine [10:58:41] no matter what caused it [10:59:19] open("/dev/hugepages/spdk0map_7566", O_RDWR|O_CREAT, 0600) = 6 [10:59:19] mmap(0x1ff6600000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 6, 0) = -1 ENOMEM (Cannot allocate memory) [10:59:19] close(6) = 0 [10:59:20] write(1, "EAL: Failed to remap 2 MB pages\n", 32EAL: Failed to remap 2 MB pages [10:59:21] ) = 32 [10:59:21] sendto(4, "<27>Jul 3 10:54:42 nvmf[101355]"..., 66, MSG_NOSIGNAL, NULL, 0) = 66 [10:59:23] rt_sigaction(SIGBUS, {SIG_DFL, [], SA_RESTORER, 0x7f3d64db4100}, NULL, 8) = 0 [10:59:23] munmap(0x7f3d5bee0000, 135794688) = 0 [10:59:25] write(2, "EAL: FATAL: Cannot init memory\n\n", 32EAL: FATAL: Cannot init memory [10:59:26] write(1, "EAL: Cannot init memory\n\n", 25EAL: Cannot init memory [11:02:20] darsto: it appears the system is mounting the hugetlbfs automatically at boot. [11:09:32] bwalker: could you weigh in on https://review.gerrithub.io/#/c/spdk/spdk/+/414935/? [11:10:11] it's piotr's patch to support native backing device sector size instead of always 4Kb [11:10:26] Should I see if they can re-build their SPDK such that it points to DPDK 18.02 instead of 17.11 and see if that resolves it (at least experimentally) ? [11:10:43] which will enable lvol boot with vhost (unless nvme namespace is formatted to 4kb sectors) [11:20:10] ok will do [11:20:18] lhodev: I wouldn't expect upgrading DPDK to do anything in this area [11:20:22] until DPDK 18.05, that is [11:20:24] lhodev: so it's the remapping that fails. This means these hugepages have been already mapped by DPDK [11:21:09] could it be you're running out of available descriptors? [11:24:56] jimharris: I'm not sure we should make the "io_unit_size" even configurable [11:25:14] I guess the user really does need to know it though [11:25:59] I think it should maybe just be a reported value - "io_unit_size" [11:26:08] and it's always the backing device's block size [11:26:35] I like to keep the number of configurable things as small as possible, and I don't htink being able to configure this one is useful [11:26:46] lhodev, darsto: https://www.systutorials.com/241561/maximum-number-of-mmaped-ranges-and-how-to-set-it-on-linux/ [11:26:52] i'd expect it to fail on open(), but maybe somehow... [11:27:03] exactly, i'm reading through this as well [11:27:17] mmap man page says ENOMEM can be returned if maximum number of mappings is exceeded - but that's not part of the limits [11:27:24] mmap manual says it returns ENOMEM when "The process's maximum number of mappings would have been exceeded. " [11:28:37] bwalker: i'm fine with not making it configurable, but we will still need to maintain backward compatibility and save the sector size [11:28:42] (or io unit size) [11:28:55] jimharris: out of curiosity - how do you know? [11:28:59] for blobstore's that already exist on 512b disks? [11:29:04] exactly [11:29:12] yeah [11:29:39] darsto: how do I know what? [11:29:40] darsto: descriptors? I have the same soft:hard limits on "open files" on a working-system vs. the failing-system, so it wouldn't appear that's the issue. [11:30:01] no - it seems mappings are different [11:30:12] what does vm.max_map_count return? [11:30:32] you could try sysctl -w vm.max_map_count to a bigger value and see if that helps [11:30:39] jimharris: that "but that's not part of the limits" part [11:30:46] exactly [11:30:56] sorry [11:31:30] darsto: jimharris: drv: I just learned that this group has been downloading the DPDK as a tarball directly from the DPDK site, instead of employing the SPDK's fork of the DPDK. I just told them why they may have been "fortunate" that things have worked for them in the past doing that, I'm urging them to use SPDK's fork of the DPDK. [11:31:40] I mean that /proc/$pid/limits and get/setrlimit don't control this mapping count it seems [11:32:21] jimharris: A cat of /proc/sys/vm/max_map_count is returning 65530 [11:32:58] jimharris: The max_map_count value on this failing system is the same as on a successful system. [11:33:37] maybe the memory layout, number of dimms, maybe something else is different between the two systems? [11:33:39] ah ok [11:33:57] should be easy to rule it out though - just do: [11:34:10] sysctl -w vm.max_map_count=1000000 [11:34:13] and see if that helps [11:35:15] jimharris: trying now.... [11:35:58] bwalker: i'm on board with not configuring the io_unit_size - we could always add this later, or lvol could decide to present logical volumes as 4K LBA [11:36:24] lhodev: with our DPDK 18.02 fork there's a way to allocate exactly as many hugepages as requested [11:36:38] jimharris: You the man! That (vm.max_map_count) *did* make a difference. It appears to be running now. [11:36:39] if everything else fails, then it's an option [11:37:07] excellent! [11:37:08] nevermind :) [11:37:43] * jimharris makes a note to add this somewhere in our documentation [11:42:00] jimharris: I need to understand this max_map_count, and how/why it results in different behavior on two different systems. [11:42:28] jimharris: they both of large amounts of memory, but on the "successful" system, it does have a bit more physical memory than the "failing" one. [12:43:11] *** Joins: pdardeau (uid308189@gateway/web/irccloud.com/x-gktzlodtherprncy) [12:43:57] *** pdardeau is now known as pauldardeau [12:44:07] *** Quits: pauldardeau (uid308189@gateway/web/irccloud.com/x-gktzlodtherprncy) (Client Quit) [12:44:28] *** Joins: pauldardeau (uid308189@gateway/web/irccloud.com/x-muufcpyfexzizsxu) [12:44:42] *** Quits: pauldardeau (uid308189@gateway/web/irccloud.com/x-muufcpyfexzizsxu) (Client Quit) [12:46:16] *** Joins: pdardeau (uid308189@gateway/web/irccloud.com/x-llyxjwqnhygqamip) [12:46:25] *** Quits: pdardeau (uid308189@gateway/web/irccloud.com/x-llyxjwqnhygqamip) (Client Quit) [13:25:07] *** Joins: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) [13:25:07] *** ChanServ sets mode: +o bwalker_ [13:27:35] *** Quits: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) (Quit: Leaving.) [13:46:18] bwalker: any idea why this patch doesn't have a Signed-off-by? https://review.gerrithub.io/#/c/spdk/spdk/+/386166/ [13:54:21] hmm [13:54:39] could have accidentally deleted it during a rebase [13:55:31] I'll fix it, once it's clear that everyone is on board with this idea [13:56:23] shall i go ahead and submit the first two patches then? [13:56:34] yeah those two aren't controversial I don't think [13:57:46] I've been working through all the threading stuff for the last several days [13:58:01] trying to see all of the details on a strategy to get where we want to go [13:58:38] I made a bunch of comments on Madhu's outstanding review - it is very tricky to get over the "hump" here to where the poller/event live in the thread instead of the reactor [13:58:48] without having the patch blow up to an enormous size [13:59:04] or crossing all sorts of architectural boundaries [13:59:24] sethhowe: looks like fedora-04 is down? [13:59:33] *** cunyinch was kicked by jimharris (cunyinch) [13:59:38] but I think I made some headway here: https://review.gerrithub.io/#/c/spdk/spdk/+/417784/ [13:59:39] *** jstern was kicked by jimharris (jstern) [13:59:46] *** kjakimia was kicked by jimharris (kjakimia) [14:00:07] *** qdai2 was kicked by jimharris (qdai2) [14:00:10] jimharris: they'll just rejoin unless banned. I need to remove them from znc [14:00:17] *** Joins: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) [14:00:18] (spdk/master) bdev: Make spdk_bdev_io_set_buf public (Ben Walker) [14:00:19] Diff URL: https://github.com/spdk/spdk/compare/61469cf4704f...ed8df49d2f05 [14:00:20] *** Parts: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) () [14:01:41] how do they rejoin? znc automatically rejoins them after some period of time? [14:01:51] *** lgalkax was kicked by jimharris (lgalkax) [14:01:53] yeah [14:02:13] but I can just delete their znc accounts [14:11:20] jimharris: it's back up. It didn't come back up after a pdu reboot. I will check the relevant bios settings when we get a lull in builds. [14:19:10] *** Quits: Guest53095 (~lyan@2605:a000:160e:2124:4a4d:7eff:fef2:eea3) (Quit: Leaving) [14:22:36] thanks sethhowe [14:23:31] if a bdev has an lvolstore on it, and we run mkfs on that bdev, shouldn't it just blow away (i.e. overwrite) the lvolstore? [14:23:53] currently it does not - i'm thinking about just removing the lvol bdev module from mkfs [14:25:32] yes, that seems like the behavior we want [14:32:20] yeah that's one of those tools where you just want it to do what you say [14:32:24] even if it is destructive [14:34:01] if we wanted to get fancy, we could make it at least detect if the device has an existing blobstore of some kind on it and warn (and have a -f to override?) [14:34:08] but I think for starters just overwriting it is fine [15:07:49] *** Joins: travis-ci (~travis-ci@ec2-54-161-95-239.compute-1.amazonaws.com) [15:07:50] (spdk/master) scripts/rpc: change virtio print_dict to print_array (Karol Latecki) [15:07:50] Diff URL: https://github.com/spdk/spdk/compare/ed8df49d2f05...81261049396b [15:07:50] *** Parts: travis-ci (~travis-ci@ec2-54-161-95-239.compute-1.amazonaws.com) () [15:22:48] *** Joins: travis-ci (~travis-ci@ec2-54-161-95-239.compute-1.amazonaws.com) [15:22:49] (spdk/master) ut/vhost: remove backend-specific tests (Dariusz Stojaczyk) [15:22:49] Diff URL: https://github.com/spdk/spdk/compare/81261049396b...889ee6194a71 [15:22:49] *** Parts: travis-ci (~travis-ci@ec2-54-161-95-239.compute-1.amazonaws.com) () [15:54:19] *** Joins: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) [15:54:20] (spdk/master) blob_ut: fix bs_dev_common for dev blocklen != SPDK_BS_PAGE SIZE (Piotr Pelplinski) [15:54:21] Diff URL: https://github.com/spdk/spdk/compare/889ee6194a71...32f35c169e86 [15:54:21] *** Parts: travis-ci (~travis-ci@ec2-54-161-216-78.compute-1.amazonaws.com) () [17:12:57] *** Quits: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) (Ping timeout: 240 seconds) [18:59:55] *** Joins: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) [19:01:45] *** Joins: dlw (~Thunderbi@114.255.44.143) [19:39:41] *** Joins: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) [19:39:42] (spdk/master) ioat: fix "XFCERCAP" typo in comment (Daniel Verkamp) [19:39:42] Diff URL: https://github.com/spdk/spdk/compare/32f35c169e86...5aa5f3dc2218 [19:39:42] *** Parts: travis-ci (~travis-ci@ec2-54-234-101-19.compute-1.amazonaws.com) () [19:49:42] *** Quits: bwalker_ (~bwalker@ip70-190-226-244.ph.ph.cox.net) (Quit: Leaving) [21:48:33] *** Quits: philipp-sk (~Philipp@ktnron0916w-lp130-02-76-66-162-159.dsl.bell.ca) (Ping timeout: 248 seconds) [22:57:29] *** Joins: pohly (~pohly@p5484976F.dip0.t-ipconnect.de) [23:45:17] *** Joins: mszwed (mszwed@nat/intel/x-cxlprfyamjuzqofp)