[00:20:00] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 246 seconds) [00:24:52] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [00:29:20] *** Quits: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) (Ping timeout: 244 seconds) [00:30:54] *** Joins: lhodev (~lhodev@66-90-218-190.dyn.grandenetworks.net) [03:35:21] bwalker: please see lhodev's comments in https://review.gerrithub.io/c/spdk/spdk/+/447851 ([TEST] SPDK 18.10.1 against DPDK 18.11 + one patch) [04:31:00] *** Joins: travis-ci (~travis-ci@ec2-3-95-160-2.compute-1.amazonaws.com) [04:31:01] (spdk/master) test/vhost: Reduce time needed to wait for vm boot in vhost tests. (Pawel Kaminski) [04:31:01] Diff URL: https://github.com/spdk/spdk/compare/092360ce49fc...8228a208a67d [04:31:01] *** Parts: travis-ci (~travis-ci@ec2-3-95-160-2.compute-1.amazonaws.com) () [05:06:19] *** Joins: travis-ci (~travis-ci@ec2-54-162-255-218.compute-1.amazonaws.com) [05:06:20] (spdk/master) env_dpdk: Run DPDK in legacy memory mode through spdk_env_opts (Shuhei Matsumoto) [05:06:21] Diff URL: https://github.com/spdk/spdk/compare/8228a208a67d...3c4199d6c631 [05:06:21] *** Parts: travis-ci (~travis-ci@ec2-54-162-255-218.compute-1.amazonaws.com) () [05:06:53] *** Joins: travis-ci (~travis-ci@ec2-3-88-3-71.compute-1.amazonaws.com) [05:06:54] (spdk/master) bdev/ftl: treat null UUID as no UUID (Konrad Sztyber) [05:06:55] Diff URL: https://github.com/spdk/spdk/compare/3c4199d6c631...02b0230296c3 [05:06:55] *** Parts: travis-ci (~travis-ci@ec2-3-88-3-71.compute-1.amazonaws.com) () [08:16:29] *** Joins: felipef (~felipef@62.254.189.133) [09:58:34] lhodev: Does ./configure --without-igb-uio-driver not work in disabling the build of the DPDK kernel modules? [09:58:48] that's the SPDK configure script [09:59:01] The more pressing problem is the KNI module. My compile fails. [09:59:02] oh I guess you're doing vanilla DPDK [09:59:35] so the way the DPDK build system works is you can use one of the default config files [09:59:37] Not vanilla DPDK, per se. I tried building with the unique branch you created in our fork of the DPDK. [09:59:39] or you can make your own [10:00:05] SPDK makes its own, but if you build against the vanilla DPDK (which is more or less what my branch is) [10:00:15] it's on by default I think [10:00:20] I'll put in a quick patch that turns it off [10:02:24] we modified the default config in our DPDK fork [10:02:37] oh - I'll find that patch [10:03:08] I see it [10:03:55] in both config/common_base and config/common_linuxapp [10:04:24] lhodev - is is DPDK build that's failing for you? [10:04:30] is it* [10:04:32] darsto: yes [10:04:56] so KNI module and anything related to it require the kernel headers [10:05:01] Right [10:05:26] I then added a 'DPDK_OPTS += CONFIG_RTE_KNI_KMOD=n' to the SPDK's dpdkbuild/Makefile [10:05:37] if you don't have them, then DPDK will just fail to compile by default with a very ambiguous error message [10:05:40] Unsure if that was entirely necessary, but it certainly enabled me to get the build done. [10:09:20] okay - so what are the next steps here? [10:09:36] I'm updating the DPDK branch he's working from now [10:09:40] As for SPDK-originated dpdk commit 754c3db, it appeared this is necessary so that we can make it all the way through the tests. Passing all the tests is essential for me to communicate to folks in my company to give them confidence to sign off on the validation. [10:11:35] hmm, dpdk-18.11.1-oracle contains just 18.11 + our custom patches, not 18.11.1 [10:11:52] yes DPDK 18.11.1 isn't out yet [10:11:57] or wasn't, when I made that [10:12:00] Correct. That's because -- well, last time I checked -- upstream DPDK still had not yet tagged 18.11.1. [10:12:18] so we're going for DPDK 18.11 plus the minimum number of patches just to get this out the door [10:12:21] i see [10:14:57] https://review.gerrithub.io/c/spdk/spdk/+/447851 [10:14:58] ok updated [10:15:03] we'll see how this looks [10:15:12] it's 18.10.2 against a vanilla DPDK 18.11 + 3 patches [10:22:42] bwalker: In the series you just uploaded, I see that config/common_linuxapp still has: 'CONFIG_RTE_KNI_KMOD=y'; i.e. not commented out. I'm concerned that may still try to build and thus break, right? [10:26:47] bwalker: yup, it fails to build for me. [10:27:11] == Build drivers [10:27:20] make: *** /lib/modules/4.14.14-6.el7uek.lghdebug.x86_64/build: No such file or directory. Stop. [10:27:29] make[7]: *** [rte_kni.ko] Error 2 [10:35:17] ok one sec [10:39:44] try now [10:40:14] Will do [10:40:16] Thx [10:50:57] bwalker: Still dies. Your patch commented out LIBRTE_.*KNI. We need to comment out CONFIG_RTE_KNI_KMOD [10:52:37] crap [10:54:05] that's fixed now [10:54:21] Fetching and retrying..... [10:58:45] Aaargh. Compile failed, but in a different way. [11:00:31] dpdk/drivers/crypto/scheduler/scheduler_pmd.c:11:10: fatal error: rte_reorder.h: No such file or directory [11:00:31] #include [11:00:31] ^~~~~~~~~~~~~~~ [11:00:32] compilation terminated. [11:01:38] I probably need to comment out crypto too [11:02:24] we have these all commented out in the spdk fork, but they were done in separate patches [11:02:27] I need to pull each of them over [11:03:03] Dang. I'm sorry that this peel-the-onion has been such a slog. [11:09:35] bwalker: you want me to take over? [11:09:52] I can disable all those option and test the build locally [11:18:24] please do [11:27:10] darsto: Thanks for the help. It's appx 1:30pm in my timezone, and so I'm going to run out for lunch. Back in a little while. [11:28:52] *** Quits: felipef (~felipef@62.254.189.133) (Remote host closed the connection) [11:29:54] kk. i need to finish up one small thing first [11:56:19] done [12:20:59] nice work - https://review.gerrithub.io/c/spdk/spdk/+/447851 [12:21:17] darsto: bwalker: Built for me, *and* I note per Gerrit that it appears to have passed all the tests! *happy dance* [13:11:16] The link I had for the SPDK's IRC log (https://ci.spdk.io/irclog/) is complaining about an invalid certificate. Is there a new link? [13:14:09] not today, but there will be shortly [13:14:28] I told the DNS company to delete ci.spdk.io on Friday [13:14:32] so of course they deleted it today [13:14:50] it doesn't matter much - we're all converted over except the irc log links [13:16:01] Where will the IRC logs live? AWS? [13:16:07] yeah [13:16:09] S3 bucket [13:17:07] My DNS dropped on me a couple times in the past hour or so hence I was gonna snoop at the IRC logs to see if I had missed anything. [13:17:39] DNS? I meant VPN. *sheesh* Life in the world of acronyms/abbreviations.... [13:22:09] *** bwalker sets mode: -o bwalker [13:25:12] *** peluse sets mode: +o bwalker [13:25:21] *** bwalker changes topic to 'Storage Performance Development Kit - https://spdk.io/. This channel is logged at https://ci.spdk.io/irclog/' [13:25:28] *** bwalker changes topic to 'Storage Performance Development Kit - https://spdk.io/. This channel is logged at https://dqtibwqq6s6ux.cloudfront.net/irclog/index.html' [13:25:35] there we go [13:40:40] Cool. Thanks! [13:45:39] bwalker: I assume darsto is done for the night. Can we move forward to merge 447851 and tag 18.10.2 ? [13:46:57] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [13:47:35] I honestly wasn't sure if 447851 should be merged [13:48:52] I wasn't going to merge 447851 - that was just a patch to confirm the scenario you need [13:48:53] adding support for a newer DPDK version is one one thing, and updating the submodule another [13:49:11] right now the tip of 18.10.2 points at the spdk fork of DPDK 18.11.x [13:49:30] that final patch is just to force the tests to run to confirm the way you want to package it also works [13:49:50] which is SPDK 18.10.2 against vanilla DPDK 18.11 + a couple patches [13:50:16] we can just tag 18.10.2 now that we've confirmed it [13:50:42] I don't know if we need to do some internal release process [13:50:51] usually we have to trigger some extra scans and stuff [13:51:02] so it might need to be tomorrow [13:51:10] they won't find any problems of course [13:51:16] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 250 seconds) [13:51:19] But doesn't SPDK 18.10.2 contains some 30 patches to facilitate its ability to work with a DPDK 18.11.x ? [13:52:53] it does [13:53:18] we're using the exact same SPDK code for both scenarios [13:53:33] https://review.gerrithub.io/c/spdk/spdk/+/448379 proves it works against the SPDK fork of DPDK 18.11.x [13:53:52] and https://review.gerrithub.io/c/spdk/spdk/+/448379 proves it works against the exact code you'd like to package for Oracle as DPDK 18.11 [13:54:01] Oh, I misunderstood. I thought you were stating that the only diff between SPDK 18.10.1 and 18.10.2 currently was an updated ptr to the special fork branch of dpdk. [13:54:14] ah yeah [13:54:36] the scans won't find anything because I know what the scans are looking for - not because we didn't make changes [13:55:54] also all the changes we made are just cherry-picks of other already released patches [13:56:04] I had been so focused on that last commit for updating the dpdk hash that I neglected to take note that all of the (29?) SPDK-commits for 18.10.2 had been merged [13:56:15] yep [13:56:24] we're basically waiting on internal tool process stuff and it's out [13:56:40] Tomek can probably tag during EU business hours tomorrow [13:58:31] Ok, then I'll hang tight for that tag and then do a pull to sync up our copy which will enable me to kick off our package build. [14:03:43] wait, we don't have tomek for this entire week [14:04:06] who approved that?? :) [14:04:13] (chuckle) [14:04:45] I assume someone else can do the tag business, huh? [14:04:51] oh yeah he's on vacation [14:05:08] either darsto or I can - I just have to figure out how he runs the scans [14:05:16] not sure if you know darsto [14:05:18] I have no idea [14:05:33] not even where to start [14:12:04] I think I can do it [14:19:26] looks like EU is going to abolish daylight savings time? [14:20:08] all of the Intel SPDK folks would never have to worry about time changes ;) [14:24:19] hopefully the whole world sees the "light" soon enough [14:24:29] Ouch [14:24:49] I want to retain daylight-savings all the time. Dispense with "standard" time. [14:26:05] Didn't California recently pass that? [14:41:17] okay so forgive any ignorance on my part, I'm just hopping into figuring out this issue with iscsi_tgtd and im not super familiar with spdk in general, but here goes my question [14:42:24] we have some tests that do rude things to clusters like kick drives from the kernel and iscsi_tgtd seems to be hanging sometimes in these tests [14:42:42] pstack of hung iscsi https://www.irccloud.com/pastebin/GfDmwxwR/ [14:44:39] I could use some help getting to the bottom of why its hanging and how to prevent it and minimize downtime [14:45:31] is this the latest version of spdk? [14:45:39] we initially noticed this because our rpc.py calls would all start hanging once iscsi_tgtd got into a bad state, and we put together some hacky babysitting in another service to check for these commands hanging and kill iscsi_tgtd when that happens which mostly recovers it [14:45:43] we're on 19.01 i believe [14:45:47] I think it's tripping a problem in DPDK's hotplug handling [14:46:13] "kick drives from the kernel" -> can you explain exactly what this means? [14:47:42] i can do my best but these are parts of the system that I've not messed with much personally, let me dig into one of the tests and try to figure out exactly how its working [14:48:26] and in general im not 100% clear on whats causing the hang, it having to do with fiddling with drives is just our best guess based on what tests seem to run into this problem [14:51:52] knowing what the test is doing would help in this case - entering this callstack that you've posted is related to a LOGICAL_UNIT_RESET on one of the iSCSI LUNs - would be good to know more about what's causing that LUN reset [14:52:08] something along the lines of injecting errors into a scsi sense buffer [14:52:26] for example - is it an explicit reset sent by one of your tests? [14:52:55] interesting - which bdev modules are you using as the LUN backing storage? [14:53:34] let me see if i can figure that out [15:02:37] aio [15:02:46] ok [15:03:11] how are errors injected into the sense buffer? [15:04:15] and any idea how it is "kicking drives from the kernel"? is it literally deleting the kernel block device while the iscsi_tgtd is using it? [15:04:36] that's a valid test case that SPDK should handle - just trying to understand if that's what's being done in your test [15:04:50] understood [15:05:11] doing my best to explain but I've kinda black boxed all that stuff mentally up until now [15:06:04] https://www.irccloud.com/pastebin/OP6MpCsq/ [15:06:17] aaand [15:06:23] https://www.irccloud.com/pastebin/Qyachrrd/ [15:08:30] i dont know if this explains it well at all [15:10:26] I dont think this is the one that kicks it from the kernel [15:10:28] lets not get confused here [15:10:38] let me checkout the test that specifically mentions kicking drives from the kernel [15:11:41] sorry - in a meeting now, will take a look afterwards (30 minutes or so) [15:11:45] the one that removes from the kernel is way easier to understand [15:11:51] https://www.irccloud.com/pastebin/b1a3VL1u/ [15:26:00] so yea I definitely misrepresented the first test, that one is replicating io errors, which is causing our system to decide that a drive has gone bad and evac it, which is probably somewhere else kicking the drive out or something, most likely by directly telling iscsi that it should remove that drive, which is what is then causing the hang [15:34:41] well that last one is certainly pretty clear [15:35:04] how reproducible is this hang, when the backing kernel block device is deleted like this? [15:39:43] its not too difficult to reproduce with the tests but I've not yet figured out a good way to manually reproduce it quickly [15:40:43] im gonna try futzing with the above device delete but I still need to get my head around how the iscsi_tgtd management code was written [15:40:54] our code not the spdk stuff* [15:42:09] bwalker: Per our previous conversation about spdk_nvme_ctrlr_get_num_ns(), I am updating the doxygen comments in nvme.h. Related, I believe the comments for spdk_nvme_ctrlr_get_ns() are incorrect in that it states "There will never be any gaps in the numbering." Right? Presuming so, I'll fix that too. [15:42:26] i think i understand your test case enough to help guide someone from our team on repro - would you mind filing an issue on github for this? https://github.com/spdk/spdk/issues - don't worry about filling out all of the template information, a quick synopsis of what you've described here will be sufficient [15:42:57] we'll discuss at our next spdk bug scrub meeting [15:43:15] once it's in github, it will get broader attention [15:45:20] will do [15:45:45] well, it depends on what you mean by "gaps" [15:46:00] let me go read the comment real quick [15:47:03] so I think there aren't any gaps in the sense that you can call that function with any ns_id from 1 to the total number of namespaces and it's going to return to you a valid struct spdk_nvme_ns * pointer [15:47:18] that pointer may just be pointing at a namespace that is "inactive" [15:47:26] Remember how we came up with that example if one had MaxNamespaces set to say 10, but someone instantiated a new namespace with id 4. Hence 1, 2 and 3 would be "inactive". [15:47:44] So you can call spdk_nvme_getrlr_get_ns() on an INactive namespace ? [15:47:51] I'll check [15:48:13] if it returns NULL, then we should fix the comment [15:49:53] yeah I think it's going to return a valid pointer [15:50:33] Really? I thought that's where those "other" guys' code failed... [15:51:56] I can't remember at the moment - but looking at the function we store the namespaces as an array of struct spdk_nvme_ns [15:51:58] Snooping through the uses of spdk_nvme_ctrlr_get_ns() in the SPDK code base, I consistently see it paired with spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() hence the assumption. [15:52:01] not pointers - full structs [15:52:14] and all the function does is return an index into that array [15:52:17] *** Joins: travis-ci (~travis-ci@ec2-3-95-205-53.compute-1.amazonaws.com) [15:52:18] (spdk/master) test: remove duplicate fio.py script file (Karol Latecki) [15:52:18] Diff URL: https://github.com/spdk/spdk/compare/02b0230296c3...b6abc16b0501 [15:52:18] *** Parts: travis-ci (~travis-ci@ec2-3-95-205-53.compute-1.amazonaws.com) () [15:52:24] if you go beyond max nsid, it will return null [15:54:11] jimharris, do you think I should rebase my entire chain with the tip of your or wait til its merged and just rebase on master? [15:54:13] Right. I guess the question then becomes if that array entry is bounds but is associated with an INactive namespace, is there something in the entry that indicates that, or can it be garbage, stale, etc. [15:54:46] peluse: good question - thinking... [15:56:23] i'd say leave them as-is for now - i'm guessing most of my patches will get a second +2 by tomorrow and then you can rebase from master [15:56:35] OK, cool. thanks [15:56:45] your patches are near the top of my review list so hopefully we can get at least the first couple of yours in shortly after that [15:57:11] thanks, I believe last time I checked (yesterday) all comments have been addressed [15:57:57] *** Joins: travis-ci (~travis-ci@ec2-23-20-87-228.compute-1.amazonaws.com) [15:57:58] (spdk/master) ocf: Added zeroing memory returned from mempool (Michal Mielewczyk) [15:57:58] Diff URL: https://github.com/spdk/spdk/compare/b6abc16b0501...b949f1318ef9 [15:57:58] *** Parts: travis-ci (~travis-ci@ec2-23-20-87-228.compute-1.amazonaws.com) () [15:58:21] bwalker: The enum spdk_nvme_ns_flags don't appear to reflect a value for whether the entry is unallocated. [15:58:49] *** Joins: travis-ci (~travis-ci@ec2-3-95-160-2.compute-1.amazonaws.com) [15:58:50] (spdk/master) rpc: add get_spdk_version rpc method (Chunyang Hui) [15:58:50] Diff URL: https://github.com/spdk/spdk/compare/b949f1318ef9...38902a5a270b [15:58:50] *** Parts: travis-ci (~travis-ci@ec2-3-95-160-2.compute-1.amazonaws.com) () [16:00:52] Perhaps one should use spdk_nvme_ns_is_active() to test for that? [16:01:18] yes that's the function that tests for that [16:01:28] but if we can make the API here more intuitive I'm open to changes [16:10:50] thanks jrlusby for the github issue - i'm responding to shuhei's comment now [16:23:31] its been suggested that we include the following compiler options for SPDK. They seem to work OK on my dev system, curious if anyone has looked at these before or has an opinion... [16:23:33] -fno-strict-overflow tells the compiler NOT to assume that signed overflow does not occur. [16:23:33] -fno-delete-null-pointer-checks tells the compiler NOT to assume that null pointer deference does not exist. [16:23:33] -fwrapv tells the compiler that signed overflow always wraps. [16:24:59] let's have vishal run his nvme performance stress test with those options to see if there's any performance impact [16:25:19] can you sync up with him? [16:29:51] can you first confirm that those aren't the defaults? [16:30:01] yup on both accounts [16:31:12] you're confirming that those are NOT the default settings? [16:31:15] just to be clear [16:32:59] *** Joins: travis-ci (~travis-ci@ec2-100-25-216-123.compute-1.amazonaws.com) [16:33:00] (spdk/master) blobstore: switch to spdk_*malloc(). (Darek Stojaczyk) [16:33:00] Diff URL: https://github.com/spdk/spdk/compare/38902a5a270b...530f481259ee [16:33:00] *** Parts: travis-ci (~travis-ci@ec2-100-25-216-123.compute-1.amazonaws.com) () [16:34:33] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [16:35:24] *** Joins: travis-ci (~travis-ci@ec2-3-88-3-71.compute-1.amazonaws.com) [16:35:25] (spdk/master) ocf: batched request processing in queue poller (Vitaliy Mysak) [16:35:25] Diff URL: https://github.com/spdk/spdk/compare/530f481259ee...2c651f101cb3 [16:35:25] *** Parts: travis-ci (~travis-ci@ec2-3-88-3-71.compute-1.amazonaws.com) () [16:48:00] @jimharris i succeeded in reproducing the failure with the steps you described [16:49:17] not sure if its worth mentioning in the github issue that it doesn't always happen, first time I tried the steps spdk survived unscathed and I brought the disk back in and everybody was happy [16:49:29] second attempt caused a hang though [16:49:53] definitely add a note in the github issue [16:50:04] what kind of fio workload were you running when you deleted the block device? [16:53:25] https://www.irccloud.com/pastebin/YdmoR9jv/ [16:57:33] I didnt follow the steps... "exactly" [16:58:53] I went through a few extra layers we have for our product rather than turning off our daemons and trying to connect to iscsi directly, though I mostly did it this way because I don't know the commands to mess with iscsi directly but I already know how to do it through our io stack [17:38:33] @Shuhei we can also talk here if you prefer [17:49:56] jimharris, jrlusby: [17:50:08] hi Shuhei! [17:50:40] unfortunately, i will be signing off here very shortly [17:50:48] I created aio bdev by pc.py construct_aio_bdev /dev/sdd aio0 and then echo 1 > /sys/block/sdd [17:51:21] No, [17:51:39] echo 1 > /sys/block/sdd/device/delete [17:51:58] I'm not sure this is correct operation. [17:52:05] during FIO runs. [17:52:29] I saw the FIO error but there was no hung. [17:53:02] I looked the configuration jrlusby posted. [17:53:41] major difference may be iodepth of FIO (8 and 1024), and number of iSCSI target (I used a target with 4 bdevs but jrlusby used 4 target with each bdev) [17:54:20] I'll try whatever but honestly this is the first time to use aio bdev. [17:54:31] So I'm afraid that my operation was wrong. [17:54:54] hi Shuhei - what kind of device is /dev/sdd? [17:55:34] and jrlusby - what kind of device are you using with aio? [17:55:35] HDD [17:55:43] I have NVMe SSD [17:56:03] i would have guessed that HDD would be the type of device most likely to reproduce this problem [17:56:28] but let's see what jrlusby was using - this could be key configuration difference [17:56:41] OK, make sense. [17:56:51] By the way, my operation of removal is correct? [17:57:08] i think so, yes [17:57:16] Thank you! [17:58:30] signing off now - if i have some extra time tomorrow i may try to reproduce this also [17:58:51] OK, thank you for your help! [18:02:17] the removal is correct @Shuhei [18:02:26] thats what I did to get the repro manually [18:02:54] but when I reproduced the error there were other processes running in the background that may have caused the actual issue [18:04:41] namely we have a daemon that gathers all sorts of status information about the system, including some info from iscsi_tgtd, and runs various commands based in response to certain statuses, so at a minimum theres various commands like `rpc.py get_portal_groups` and `rpc.py get_bdevs` running in the background regularly [18:06:04] we have hdds and ssds, i dont know off of the top of my head if sda was an hdd or an ssd [18:07:01] jrlusby: thank you, I think I have enough information now. [18:07:04] its definitely not an NVMe drive though [18:07:15] though all of this work is being done to eventually support NVMe [18:08:25] Please wait for my update, I hope I will give any update by your tomorrow morning. [18:08:57] Oh, I have a question [18:09:22] you used only randwrite? [18:09:42] I haven't tried randwrite, I tried randread and randrw once for each. [18:10:13] thats the fio I used during the manual repro [18:10:24] OK [18:10:51] And thanks for your input, I'll focus on HDD for now. [18:11:12] can you post the fio cmdline you're using when you're trying to repro? [18:12:22] I had used fio config file. [18:12:44] I'll change to use command line simply and will share that. [18:13:31] i can use config file [18:13:33] And I have used physical HDD and every try will need server reboot, so every try will take a few minute. [18:13:39] im just unfamiliar with the fio as a utility [18:13:45] Me too. [18:13:57] So no problem. [18:14:18] I will follow what you did [18:15:21] okay [18:15:33] I have a hour meeting 10 minutes later. [18:16:01] I'll restart after that. thank you for your help. [18:16:25] okay [18:17:02] for when you do get back, when I kicked the drive I didnt kick it via the handle created by the iscsiadm login, aka `/dev/sde`, i used the original block device handle `/dev/sda` [18:17:29] or more accurately `echo 1 > /sys/block/sda/device/delete` [18:17:42] I'm not sure if that matters [18:20:56] I deleted the backing device to AIO bdev as you did. [18:21:11] I didn't delete device visible from FIO. [18:22:30] there might be an extra step to delete the bdev that our daemon does in the background [18:24:37] I haven't managed to get a clean repro without the rest of the system running, but if you wanna see where im at heres a vague outline of the test script im trying to write. [18:24:51] https://www.irccloud.com/pastebin/EX57Tgl2/ [18:30:17] actually fk that script its garbage atm ill link it again when it vaguely does what I expect [18:30:18] *** Joins: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) [18:34:32] *** Quits: felipef (~felipef@cpc92310-cmbg19-2-0-cust421.5-4.cable.virginm.net) (Ping timeout: 250 seconds) [18:42:17] just need to figure out why my fio command is broken but heres what I got atm @Shuhei [18:42:29] https://www.irccloud.com/pastebin/ZqcaougB/test_iscsi_hang [19:20:53] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [19:47:01] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [20:02:49] *** Quits: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) (Ping timeout: 256 seconds) [21:37:05] *** Joins: Shuhei (caf6fc61@gateway/web/freenode/ip.202.246.252.97) [23:10:04] *** Quits: guerby (~guerby@april/board/guerby) (Remote host closed the connection) [23:14:54] *** Joins: guerby (~guerby@april/board/guerby) [23:26:00] Project autotest-nightly build #441: STILL FAILING in 25 min. See https://ci.spdk.io/spdk-jenkins for results. [23:39:23] Project autotest-nightly-failing build #310: STILL FAILING in 39 min. See https://ci.spdk.io/spdk-jenkins for results.