[00:43:47] *** Joins: mszwed (mszwed@nat/intel/x-qlolckrojppqzmkp) [00:53:31] *** Joins: tkulasek (tkulasek@nat/intel/x-pdjxkwrkmiqepjdz) [01:10:19] *** Joins: pwodkowx (~pwodkowx@134.134.139.72) [01:44:04] *** Quits: tkulasek (tkulasek@nat/intel/x-pdjxkwrkmiqepjdz) (Remote host closed the connection) [04:06:53] *** Joins: tkulasek (tkulasek@nat/intel/x-yawzkvxnzisxcmou) [07:51:55] hi waelhalbawi - that behavior does not seem correct to me - can you provide the command line you used with scripts/rpc.py to delete the passthru bdev? [07:53:14] bwalker: jimharris: Hey guys, after one has submitted a patch to gerrit which has undergone preliminary review -- e.g. only CI +1's and one maintainer +2 -- so not yet ready to merge, is there a way to somehow "mark" a new, subsequent patch series that depends on the former being processed first? Or, do I need to wait until the first one is fully reviewed and merged? [07:54:40] hi lhodev - the usual way to do this is just put the following patches in the same local branch on your system, on top of the first shared_lib patch [07:55:05] then when you push them to Gerrit, we will see that the following patches depend on the first one [07:55:59] jimharris: Thx Jim. I had placed the new series on their own dev branch, but I guess I can just rebase the latter onto the former to achieve that, huh? [07:56:09] yep [07:56:36] after you've pushed them, bring the patch up in Gerrit and you'll see the patch series in the upper right corner under "Related Changes" [07:56:41] *** Joins: travis-ci (~travis-ci@ec2-54-211-248-34.compute-1.amazonaws.com) [07:56:42] (spdk/master) examples/sock/hello_sock: introduce sample application for libsock (Tomasz Kulasek) [07:56:42] Diff URL: https://github.com/spdk/spdk/compare/6ff8f60ebbe0...a2ab3cccc2c4 [07:56:42] *** Parts: travis-ci (~travis-ci@ec2-54-211-248-34.compute-1.amazonaws.com) () [07:56:49] here's an example: https://review.gerrithub.io/#/c/spdk/spdk/+/421408/ [08:04:21] jimharris: Got it. Done, and thanks. [08:31:29] bwalker, darsto, tkulasek: https://review.gerrithub.io/#/c/spdk/spdk/+/422310/ - fixes merge error with the recently merged hello_sock app [08:53:22] jimharris: bwalker: Ugh, looks like there's a different version of gcc/binutils on FreeBSD that doesn't support an option that runs on the Linux-based systems. What to do? https://ci.spdk.io/spdk-jenkins/public_build/autotest-per-patch_9310.html [08:55:55] +1 [08:59:41] can we specify these extra ld flags only when shared lib has been enabled? [09:00:03] and then i would say have the configure script not allow shared library builds (for now) on FreeBSD [09:00:27] we can add FreeBSD shared library support to the backlog [09:01:33] our dpdk forks fail to build with T=x86_64-native-linuxapp-gcc [09:02:53] jimharris: on a call now. Back to you soon. [09:04:19] https://review.gerrithub.io/c/spdk/dpdk/+/422245 (Always build librte_kvargs). Here I thought it's just rte_kvargs that we're missing, but there are more libs like that [09:05:28] some libs depend on crypto stuff we explicitly disabled [09:07:17] *** Joins: peluse (~peluse@134.134.139.72) [09:07:18] *** ChanServ sets mode: +o peluse [09:07:28] dpdk guides suggest to sed `config_base` in place to change compile flags [09:08:16] yuck [09:08:17] so how about we drop our extra spdk build target and ship with already modified dpdk default config_base? [09:08:39] i really wanted to avoid having to build all of the ethernet drivers that we don't use [09:10:28] are we sure we can just remove our extra spdk build target? peluse will need to weigh in since there are crypto config options we modify now in the dpdkbuild [09:12:17] T=x86-... is broken for over a month now [09:12:32] i don't recall anyone complaining about that [09:47:32] *** Joins: waelhalbawi (a022586e@gateway/web/freenode/ip.160.34.88.110) [09:53:07] Hi jimharris. The command is scripts/rpc.py delete_passthru_bdev pt where 'pt' was the name of my vbdev. [09:53:16] The complete sequence of steps is as follows: [09:53:32] scripts/rpc.py construct_aio_bdev /tmp/my_aio aio 512 [09:53:47] scripts/rpc.py construct_passthru_bdev -b aio -p pt [09:54:01] scripts/rpc.py remove_passthru_bdev pt [09:54:16] scripts/rpc.py delete_aio_bdev aio [09:55:02] Now try again creating the aio bdev using the same name: scripts/rpc.py delete_aio_bdev aio. You will see the passthru vbdev appearing. The code flow (at least to me) does imply this behavior. [09:56:57] *** Quits: tkulasek (tkulasek@nat/intel/x-yawzkvxnzisxcmou) (Ping timeout: 240 seconds) [09:58:20] yeah - this is a bug, we need to not only unregister the bdev, but also remove the associated entry in the g_bdev_names TAILQ in vbdev_passthru.c [09:58:42] could you submit a bug in the SPDK github repository? https://github.com/spdk/spdk/issues [09:59:32] just provide the details you've shown here - just through code inspection I can confirm this issue still exists in SPDK 18.07 and latest master [10:00:09] Yes sure. I am happy to submit a patch too. [10:23:49] jimharris: done with call. Sorry 'bout that. I'll modify the patch for adding the special ld flags only on shared, and then look into excluding the build of shared on FreeBSD. [10:25:33] sounds good [10:25:47] i'm not familiar with that push/pop-stack ld stuff - will need to read up on it [10:33:31] FreeBSD has a really old version of binutils in base - the last one that was GPLv2 [10:33:40] from 2010 [12:34:00] jimharris: Hey Jim, I re-did the patch series, and I'll be darned if ubuntu14.04 (Chandler test pool) failed due to the same issue about ld on there lacking support for push/pop-state. Ugh. Now what? [12:35:07] Looks like the jenkins pool delivered a pass across its systems. [12:36:04] fun :-) [12:38:25] dumb question - what happens if you don't use this -Bstatic? [12:39:55] jimharris: not dumb at all. If we don't specify it, then it attempts to link with the shared version. gcc by default, upon seeing both a shared and static lib of the same name in the same dir, will go for the shared one unless specified. [12:41:16] what happens if it uses the shared lib instead of the static one? [12:43:21] Per earlier exchange with Ben, we were just going to slowly wade into the shared support, so he suggested we build all the SPDK executables with the static. Now, that said, I *have* successfully built (by hand) and test run a couple executables that were linked with the shared ones, but I haven't done so with all of them. [12:43:46] got it [12:47:20] i'm not sure how the --whole-archive stuff works with shared libraries [12:47:30] I'm not sure if with the existing linker flags whether [12:47:32] we need it for static libraries [12:47:46] or the constructor functions don't get linked in [12:48:05] Funny you should mention that. I read an article about --whole-archive with shared libs. There was an odd corner case that it was demonstrated to address. [12:48:27] That was the scenario: a constructor. [12:49:36] i'd like bwalker to weigh in although he's mostly out today with sick kiddos [12:50:07] but i'm curious how this all works if you remove the --push-stack and -Bstatic stuff completely [12:50:22] the default configure will *not* build the shared libraries, correct? [12:50:52] I'll give that a try (i.e. without the push/static). Unsure if the linker flags otherwise will link everything properly at the end or not. [12:51:05] Yes, the default will *not* build the shared libs. [12:52:00] the almighty test pool oracle will tell us :) [12:52:27] I'll give the "everything shared" in a little bit. It's almost 3pm (TX time) and I've yet to have lunch. Ready to gnaw off my arm. [12:52:41] i normally don't condone using the test pool to just try something out, but in cases like this it is more than warranted [12:53:04] Cool [12:57:56] One thought before I dash off to lunch: if all builds/links properly for shared, then the libs will need to be installed to "standard lib" locations or LD_LIBRARY_PATH set accordingly. [12:59:09] On Linux systems, following the install, the standard procedure is to run "ldconfig" (which, according to my read, is not suggested to do automatically within Makefiles). Dunno about FreeBSD. [13:52:59] jimharris: I tried an all-shared build without the push/pop/Bstatic just in my own sandbox, and final linking failed (unable to resolve symbols). Blech. The library containing the code for those was the very next one on the ld line. Unsure at this point why it didn't resolve. [13:58:00] jimharris: And, both of the libs -- one with the references to the syms in the very next lib -- were in the same whole-archive "block", too. Hmmmm.... [14:02:40] can you share which symbols failed to resolve? [14:03:50] virtio_dev_queue_get_thread, virtio_dev_read_dev_config, virtio_pci_dev_init, virtio_pci_dev_enumerate, and about 12 or so more. [14:05:59] The references are in spdk_bdev_virtio. The definitions are in the very next library in the ld line, spdk_virtio. [14:07:01] could you try commenting out those modules from mk/spdk.modules.mk? [14:07:21] just want to see if there are any other gremlins hiding behind virtio [14:07:40] Right now, I'm trying to comment out the build of examples/bdev, and see how much further it progresses. [14:08:53] That didn't take long. Crapped out with same complaint linking examples/blob/hello_world/hello_blob [14:09:23] yeah - almost every application and example will link in all of the bdev modules [14:09:53] i'd try hacking mk/spdk.modules.mk and see if that helps - then if it's just that one module I'm sure we can fix that separately [14:31:31] A couple things linked, but then when going into app, started getting undefined refs to SPDK_LOG_REACTOR, g_subsystems_deps, g_subsystems, g_spdk_iscsi_opts, SPDK_LOG_LOG (from refs in libspdk_app_rpc.so) [14:33:24] jimharris: for grins, I tried hand-building, adding a start/end-group around the offending libs, but still failed to link. [14:38:18] could you push a patch that I could pull to get the list of undefined refs? [14:38:51] or just send me the build log [14:40:12] g_spdk_iscsi_opts and g_subsystems_deps are both declared as extern structs [14:40:34] g_subsystems as well [14:40:44] that doesn't fly with shared libraries [14:40:48] but I think we can fix those [14:47:13] I've tried commenting out various directories in Makefiles. nvmf_tgt didn't build nor iscsi_tgt or spdk_tgt. vhost just crapped out. Ugh. This is painful. [14:52:00] could you send me the full list of undefined refs? i think a lot of this can (and should) be fixed [14:53:11] git grep "extern struct" - those all need to be reworked [14:54:19] the log ones (i.e. SPDK_LOG_REACTOR) are going to need some work [14:54:33] not clear to me yet how to fix those [14:56:59] I'll send you the difficulties via email. Meanwhile, I'd like to understand what's the issue with shared libs and extern structs? And, as it would follow, what's the "magic" to resolve that? Wrap it with some kind of accessor function? [14:57:37] a lot of them could be resolved by accessors I think [14:58:02] for the various LOG flags - we will need to stop sharing LOG flags across library boundaries [14:59:45] accessor functions for log flags won't work without introducing performance penalties [15:00:01] Keeps getting better.... [15:50:51] jimharris: sent you mail on those undefined refs [16:15:56] the virtio ones I haven't wrapped my head around yet [16:28:04] lhodev: were you able to build the test directory? [16:29:24] I just stopped after the first failure in test with hopes of resolving the earlier link failures in app first before delving further. [16:33:49] i'm curious - were you building with -j1? [16:34:13] i pulled your three patches, removed the .mk changes, and am able to build all of the example apps [16:34:23] even the nvmf_tgt linked [16:34:24] I only ran with "make V=1" [16:35:18] Really? How interesting. All those built, huh? Tried to exec any of them? [16:35:29] ha [16:35:33] proof by compilation [16:35:38] ;-) [16:35:56] My gcc version is 5.2.1 and ld version is 2.25-10 [16:37:16] were you able to run any of them? [16:37:31] i get a loading shared libraries error - it's looking for libspdk_nvme.so.1 [16:37:42] but that doesn't exist - we have libspdk_nvme.so.1.0 [16:38:06] Did you "make install" and then run ldconfig -n $directory_where_your_libs_are_installed ? [16:38:14] nope [16:38:20] let me do that now [16:40:33] nvme identify app seems to run ok [16:40:51] How nice for you [16:42:22] I wonder if my link failures are due to the versions of my gcc/binutils. [16:42:49] gcc 7.3.0, ld 2.30 [16:44:08] Both of yours apparently more recent than my archaic ones. [16:44:40] I'm gonna see if we have software collections that contain more recent gcc/binutils, and if so, build with them and see what happens. [16:45:56] i wonder if there's an option to export the structures in shared libraries? [16:46:00] with the older ld [16:48:02] Interestingly, I hammered out some code writing a couple simple libraries (including with an extern struct) and I was able to build, link and run that without any troubles at all. [16:48:29] i am certainly finding some dependency issues related to library name ordering [16:48:59] for example, to get iscsi_tgt to link, I needed to move event closer to the front of SPDK_LIB_LIST [16:49:11] if I may pop in for a minute... :) [16:49:21] LOL [16:49:27] Indeed ! [16:49:33] i knew you couldn't stay away! [16:49:36] the symbols to export from the shared libraries are defined in shared_lib/spdk.map, currently set to anything named spdk_* [16:49:49] so if you have any other symbols that need to be exported, they would need to be added there [16:50:31] drv: if you don't specify such a map (which I didn't in my little experimental code), then do all symbols get exported (that aren't static)? [16:51:48] and the main reason i didn't get to individual shared libs is that to do it properly, they need to have dependencies on each other (e.g. libspdk_nvme.so should depend on libspdk_env_dpdk.so and so on) [16:51:51] so that the link order shouldn't matter [16:51:56] lhodev: right [16:52:18] unless you set symbol visibility to be something other than 'default' on the command line, everything is exported [16:53:48] also, depending on how you want it to work, it might be a good idea to add the build directory to rpath if you are going to link the binaries against the shared libs (rather than playing tricks with LD_LIBRARY_PATH) [16:53:58] ok, back to work :) [16:54:29] drv: Well..... ideally, yes (re: build with dependencies). Is it essential? Kind of depends. Alas, you wrote "properly" ;-) [16:55:09] well, I think it would be best to link against the dependencies within the SPDK build system (otherwise the end user needs to know all of the dependencies instead) [16:55:27] I didn't mess with rpath at all. I had this distant memory that using rpath wasn't ideal. And, one wouldn't have to monkey around with LD_LIBRARY_PATH at all if the libs are installed in "standard locations". [16:56:38] yes, after a 'make install' into the standard location (/usr/lib or whatever), no LD_LIBRARY_PATH stuff should be necessary [16:56:56] only needed if you want to run the binaries from the SPDK source directory before a make install [16:57:04] right [17:02:07] On the "properly" angle each spdk*so would have the dependencies set, would you expect that an SPDK app writer would arbitrarily choose just one of them they *know* they need from their sources and just expect the run-time linker to resolve all the rest? [17:48:05] lhodev: i think have the library dependencies figured out for now - at least enough that we shouldn't need to spell out the dependencies this first go-round [17:54:48] i'll get some of these patches cleaned up tomorrow and pushed out for review [19:01:42] *** Quits: waelhalbawi (a022586e@gateway/web/freenode/ip.160.34.88.110) (Ping timeout: 252 seconds) [21:30:45] *** Joins: waelhalbawi (94571708@gateway/web/freenode/ip.148.87.23.8) [22:52:09] *** Quits: waelhalbawi (94571708@gateway/web/freenode/ip.148.87.23.8) (Ping timeout: 252 seconds)