[00:48:56] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-khudaowrmrimqube) [01:09:56] *** Joins: tkulasek (86bfdc4e@gateway/web/freenode/ip.134.191.220.78) [02:08:08] *** Joins: travis-ci (~travis-ci@ec2-3-91-56-6.compute-1.amazonaws.com) [02:08:08] (spdk/master) test/vhost: Fix. Remove recursive common.sh sourcing (Pawel Kaminski) [02:08:08] Diff URL: https://github.com/spdk/spdk/compare/86d8214d157b...357ccdd8a418 [02:08:08] *** Parts: travis-ci (~travis-ci@ec2-3-91-56-6.compute-1.amazonaws.com) () [02:09:31] *** Joins: travis-ci (~travis-ci@ec2-34-239-109-223.compute-1.amazonaws.com) [02:09:32] (spdk/master) test/nvmf: improve nested lvol test (Jim Harris) [02:09:32] Diff URL: https://github.com/spdk/spdk/compare/357ccdd8a418...46fef66536a7 [02:09:32] *** Parts: travis-ci (~travis-ci@ec2-34-239-109-223.compute-1.amazonaws.com) () [02:17:58] *** Joins: travis-ci (~travis-ci@ec2-54-225-39-119.compute-1.amazonaws.com) [02:17:58] (spdk/master) lib/ftl: check config after geometry is retrieved (Konrad Sztyber) [02:17:58] Diff URL: https://github.com/spdk/spdk/compare/46fef66536a7...e382b437b91c [02:17:58] *** Parts: travis-ci (~travis-ci@ec2-54-225-39-119.compute-1.amazonaws.com) () [02:44:02] *** Joins: travis-ci (~travis-ci@ec2-35-174-155-151.compute-1.amazonaws.com) [02:44:03] (spdk/master) bdev: add test case for splitting iovs that exceeds child_iov (lorneli) [02:44:03] Diff URL: https://github.com/spdk/spdk/compare/e382b437b91c...598ba408a272 [02:44:03] *** Parts: travis-ci (~travis-ci@ec2-35-174-155-151.compute-1.amazonaws.com) () [09:28:08] *** Quits: tomzawadzki (uid327004@gateway/web/irccloud.com/x-khudaowrmrimqube) (Quit: Connection closed for inactivity) [10:00:54] *** Joins: travis-ci (~travis-ci@ec2-34-230-70-3.compute-1.amazonaws.com) [10:00:54] (spdk/master) lib/bdev: merge buffer allocation in get_buf/put_buf (Konrad Sztyber) [10:00:54] Diff URL: https://github.com/spdk/spdk/compare/598ba408a272...2fac8bc76bfa [10:00:54] *** Parts: travis-ci (~travis-ci@ec2-34-230-70-3.compute-1.amazonaws.com) () [10:01:20] *** Joins: travis-ci (~travis-ci@ec2-54-161-192-25.compute-1.amazonaws.com) [10:01:20] (spdk/master) Revert "scsi: Simplify removing LUN when no connection is for it" (Darek Stojaczyk) [10:01:20] Diff URL: https://github.com/spdk/spdk/compare/2fac8bc76bfa...122cc72cd960 [10:01:20] *** Parts: travis-ci (~travis-ci@ec2-54-161-192-25.compute-1.amazonaws.com) () [10:04:53] *** Joins: travis-ci (~travis-ci@ec2-35-174-155-151.compute-1.amazonaws.com) [10:04:53] (spdk/master) bdev/compress: use the queue pair for the current device (paul luse) [10:04:53] Diff URL: https://github.com/spdk/spdk/compare/122cc72cd960...634af37b463f [10:04:53] *** Parts: travis-ci (~travis-ci@ec2-35-174-155-151.compute-1.amazonaws.com) () [10:08:34] *** Joins: travis-ci (~travis-ci@ec2-3-95-189-230.compute-1.amazonaws.com) [10:08:34] (spdk/master) bdev/raid: drop max raids to 2 (paul luse) [10:08:34] Diff URL: https://github.com/spdk/spdk/compare/634af37b463f...45e17cb24269 [10:08:34] *** Parts: travis-ci (~travis-ci@ec2-3-95-189-230.compute-1.amazonaws.com) () [10:54:36] *** bwalker changes topic to 'Storage Performance Development Kit - https://spdk.io/. Join us in Slack instead: https://join.slack.com/t/spdk-team/shared_invite/enQtNjM0NDcxNzQyNzM3LWQ3M2FjZDgzMmU5MjkyZWU5MDRhMGZkNGY3ZjM1YjhhODkyZmRkYWNjODk1YTdiZDMxNzJjNzc0NTEyMTVhYTM' [11:17:07] *** Quits: tkulasek (86bfdc4e@gateway/web/freenode/ip.134.191.220.78) (Ping timeout: 256 seconds) [11:48:08] *** Joins: nagyz (ubuntu@nat/ibm/x-uksrduvvimfmtqmj) [11:48:12] hello folks [11:48:37] I've spent the day trying to modify the raid driver to do raid1 as well - I thought it's a simple exercise to learn how spdk bdevs operate. [11:49:56] in the write path I check the raid level, and if it's 1, I issue requests to all backing devices; in order to keep track of them, I've added a new field to the context using DPDK's rte_atomic32_t, so it's thread safe (although I think this might not be needed...) [11:50:51] then in the completion callback I check if the number of outstanding IOs is zero; if it is, I signal spdk_bdev_io_complete() on the parent IO spdk_bdev_io [11:51:06] yet when I do this I get an assert: [11:51:11] fio: bdev.c:3519: spdk_bdev_io_complete: Assertion `bdev_ch->io_outstanding > 0' failed. [11:51:27] is this the wrong approach? [11:52:22] so in SPDK all I/O are associated with a "channel" [11:52:26] each channel is only used from a single thread [11:53:08] I get that basically with a single thread on a single core the two IOs I issue would serialise [11:53:26] so you don't need to worry about thread safety as long as all of the I/O are completed on the same thread that they are submitted on (which is true for every other bdev, so will be true by default for your raid1) [11:53:43] each channel supports queue depth > 1, so they don't serialize [11:54:23] I see [11:54:31] I'm pretty new to SPDK just started playing with it last week :) [11:54:37] let me look up why that assert can hit [11:54:52] https://github.com/spdk/spdk/blob/master/lib/bdev/raid/bdev_raid.c#L330 [11:55:30] ok, so there is a mismatch in counting I/O that have been submitted vs completed [11:55:45] I think I can guess where that happens [11:55:49] just looking a bit more now [11:55:51] this is the write path I've modified - instead of issuing the spdk_bdev_writev_blocks() call only to raid_bdev->base_bdev_info[pd_idx].desc on channel raid_ch->base_channel[pd_idx], I loop through all backing bdevs and issue them to raid_bdev->base_bdev_info[i].desc on channel raid_ch->base_channel[i] [11:56:11] and then I only run this code path: [11:56:12] https://github.com/spdk/spdk/blob/master/lib/bdev/raid/bdev_raid.c#L277 [11:56:31] can you make sure you aren't completing the parent I/O multiple times by accident? [11:56:32] if all "generated" IOs are done, so the parent can be completed [11:57:13] I suspect the parent is getting completed twice too [11:58:08] let me add some debugging [11:58:17] I'm using fio via the fio_plugin to use bdev... to generate IO [11:58:20] you can put in a print any time io_outstanding is incremented or decremented [11:58:23] maybe not the most efficient approach [11:58:30] no using fio to send I/O is a great way to do this [11:58:33] try just running the bdevio tool [11:58:44] we also have bdevperf and bdevio which are more targeted tests [11:59:08] if this fails on just the first write I/O, then bdevio would be a simple test to make sure everything is basically working [11:59:27] aha, let me look at how to use that [11:59:37] it goes through some other test cases too - things like resets, unmap, etc. [12:00:25] I stopped building the tests as I had to add a Makefile hack to get DPDK's atomic handling in the code [12:00:33] and had no idea how to do the same for the tests :) [12:01:57] can I get bdevperf to issue a single IO request instead of the "run it for N seconds" thing? [12:02:33] not really - but the bdevio test will effectively do that [12:02:45] the first test is runs is do a 4k write followed by a 4k read [12:02:52] the first test *it* runs [12:05:26] I actually thought the assert means that it didn't get the completed call? is there a way to dump how many it thinks is outstanding? [12:06:30] ahahhah, how stupid am I :-) [12:06:49] I changed the if from "if (success)" to "if (success && all_done)" [12:06:57] now guess what, the else triggered the IO failure... :) [12:07:22] now fio succeeds! [12:07:59] great! [12:08:13] is there a reason nobody implemented RAID1 upstream? [12:08:18] would that be a welcome patch? [12:08:28] very welcome [12:08:37] I think we were waiting for you to do it [12:08:41] no reason really [12:08:42] haha [12:08:46] what bwalker said [12:08:48] :) [12:09:22] also I saw that the raid bdev code does not support >1 threads [12:09:33] but I guess that would be the 2nd challenge [12:10:33] where do you see that? I'm not super familiar with the raid 0 module sorry [12:11:38] at first glance it looks like it supports multiple threads to me [12:11:53] ah, right, that was the fio code that said thread=1 is mandatory [12:11:56] yeah - it supports multiple threads [12:12:06] thread=1 in fio doesn't mean 1 thread - everyone gets confused by that [12:12:15] that means "thread mode = true" [12:12:16] * nagyz too [12:12:17] fio thread=1 just means that every job is a new thread in one process [12:12:23] instead of a different process for every job [12:12:26] each job in fio can be either a thread or a process [12:12:31] we require thread mode [12:12:42] hah [12:12:45] TIL :) [12:12:58] I wish fio would change that, but it is what it is [12:14:29] so what I did right now is that I've added an rte_atomic32_t counter to raid_bdev_io to keep track of the "generated IOs" - eg writes to the backing devices [12:14:45] yeah that doesn't need to be atomic [12:14:56] the raid_bdev_io will only get touched from a single thread [12:15:13] at qd>1 can't two IOs's async callback thread stump on the same parent IO's raid_bdev_io at the same time? [12:15:17] you can do different raid_bdev_ios on different threads, but any given request will only be touched by a single thread for its lifetime [12:15:36] the children will get processed on the same thread as the parent [12:15:38] it's not the bdev_io for the callback from SPDK, but the actual bdev_io that gets passed to the async callback as an argument [12:15:47] I see [12:15:50] so their callbacks won't ever stomp each other [12:16:14] ok, so I can just change it to uint8_t and get rid of my DPDK usage [12:16:17] yep [12:16:46] wouldn't uint32_t be better for cache alignment and such? or it doesn't matter in practice? (the struct has 4x uint8_t just before) [12:17:21] I like the way you are thinking [12:17:34] usually I run 'pahole' on the object file [12:17:43] and look at the struct to see where the boundaries are [12:17:51] makes sense [12:18:03] you're right - you want it to be aligned and to not push stuff beyond a cache line [12:20:40] so once it runs with fio with randread/randwrite at qd>1 with verify=1, I guess I can submit it on gerrit. let's see if I can get there. [12:20:43] thank you guys for the quick responses [12:20:57] based on the archive bot data on the web the channel seemed a bit empty, so this was a very positive surprise! [12:21:14] (wanted to say qd>1 above, obviously) [12:22:35] i'd suggest running it through the bdevio tool as well [12:37:38] it's been empty the last week because we're moving to slack [12:37:43] link to slack team in motd [14:21:00] make [14:21:05] whops, wrong window,sorry:) [14:35:18] if I'd like to implement RAID5 let's say (for which I need to modify the actual data content - eg generate new IO) are there any helpers for that? [14:35:24] or any pointers where I should look for something like that [14:35:29] I guess the compression bdev is similar [15:00:13] I think I'll need to copy the iovec into a single buffer and issue that as a read/write - as I need to transform the actual bytes [15:01:50] nagyz - are you able to join our slack channel? That's where we're all at lately [15:01:54] otherwise I'll keep answering here [15:02:56] ok, let's continue there :-) [15:30:22] *** Quits: nagyz (ubuntu@nat/ibm/x-uksrduvvimfmtqmj) (Quit: leaving) [22:14:46] *** Joins: travis-ci (~travis-ci@ec2-3-83-116-64.compute-1.amazonaws.com) [22:14:46] (spdk/master) bdev: increase QoS granularity (Darek Stojaczyk) [22:14:46] Diff URL: https://github.com/spdk/spdk/compare/6be900dae8de...ca0cdbf269f7 [22:14:46] *** Parts: travis-ci (~travis-ci@ec2-3-83-116-64.compute-1.amazonaws.com) () [22:34:01] Yippee, build fixed! [22:34:02] Project autotest-nightly build #508: FIXED in 34 min. See https://dqtibwqq6s6ux.cloudfront.net for results. [22:35:24] *** Joins: travis-ci (~travis-ci@ec2-18-208-175-171.compute-1.amazonaws.com) [22:35:24] (spdk/master) vhost_ut: Add tests for spdk_vhost_vq_avail_ring_get (Seth Howell) [22:35:24] Diff URL: https://github.com/spdk/spdk/compare/ca0cdbf269f7...5904abae7915 [22:35:24] *** Parts: travis-ci (~travis-ci@ec2-18-208-175-171.compute-1.amazonaws.com) () [22:43:27] Project autotest-nightly-failing build #364: STILL FAILING in 43 min. See https://dqtibwqq6s6ux.cloudfront.net for results. [23:08:00] *** Joins: tomzawadzki (uid327004@gateway/web/irccloud.com/x-ndneqljsdugyergz)