[03:24:40] *** Joins: satyam (6a3324e8@gateway/web/freenode/ip.106.51.36.232) [03:26:53] *** Quits: satyam (6a3324e8@gateway/web/freenode/ip.106.51.36.232) (Client Quit) [08:44:33] *** Joins: sethhowe (~sethhowe@192.55.54.42) [09:53:48] *** Joins: Darren (c6ba0002@gateway/web/freenode/ip.198.186.0.2) [09:54:10] *** Darren is now known as Guest48477 [09:55:32] Question: I'm using SPDK to develop my own app against my HW. I [09:57:58] fast fingers. I've been using this with spdk 16.12 and 17.03 with no problems. With current tip though, I have trouble writing to 16 bit hw registers. 32bit read/writes are fine. 16-bit reads are fine. Any ideas what has changed in newer dpdk? [10:01:42] do you mean writing PCI config space? e.g. spdk_pci_device_cfg_write16()? [10:04:40] No. Mem bar [10:06:49] The mem bar is memory mapped. For example. I have a 16 bit register at offset 0x80. And another one at offset 0x82. If I write to the individually, the hw doesn't see the writes. If I combine the 2 values into a single 32-bit write, the hw sees it. [10:07:55] Previous release I used spdk17.03/dpdk 17.02. worked fine. Before that 16.12/16.11 and worked fine. [10:07:57] hmm, the only thing I can think of is that our DPDK submodule has a patch that maps memory BARs as write combining if they are capable [10:08:00] https://github.com/spdk/dpdk/commit/47668495fca617c430df34ea2cb733abaf984f91 [10:09:31] this will only apply if you are using uio_pci_generic - VFIO already maps wc by default as far as I understand [10:15:38] hmm. Let me undo that patch and see if that changes things. [10:28:57] backing out that patch got it working [10:38:05] interesting - so your device presumably sets the Prefetchable bit on the BAR, which is what controls whether Linux exposes resource%d_wc [10:40:20] it might be possible to add a read after the write to force the write to be flushed [10:43:02] yes. It is prefetchable. [10:44:11] This area is not in the perf path. ugly, but I could add a read after each write to these specific registers. [10:44:55] we could also probably expose a mechanism for DPDK PCI drivers to opt out of the write combining mapping [10:45:10] although I don't think you have any choice with VFIO, so it might not be worth the trouble [10:46:32] does any other OS(kernel) do this? we have been shipping this hw for many years and never had an issue. [10:53:04] I'm not sure; I would guess a normal kernel driver would have the choice of whether to map WC or not [10:57:47] I don't see this issue with kernel drivers in linux/windows/esx/solaris etc. [10:57:57] FYI, doing a read after the write worked also. [11:22:08] thanks for the help. Took awhile to chase down where the failure was. [12:16:11] *** Quits: Guest48477 (c6ba0002@gateway/web/freenode/ip.198.186.0.2) (Quit: Page closed) [13:40:57] peluse: I posted a comment on your valgrind patch; I think it's a good idea to move all of that into unittest.sh, but it needs a small tweak to work correctly with the build pool $SPDK_RUN_VALGRIND option [13:43:18] I'm working on it now [14:38:15] jimharris: last merge introduced a build failure - see https://review.gerrithub.io/#/c/369080/ [16:16:01] *** Joins: ziyeyang_ (~ziyeyang@134.134.139.76) [16:18:47] *** Quits: ziyeyang_ (~ziyeyang@134.134.139.76) (Client Quit) [17:24:46] drv, OK, circled back to the valgrind thing. See if that looks better :) [19:39:24] anyone else seeing issues with random CI failures with these in the signature: (I have a few patches that seem to have been hit by this randomly lately) [19:39:37] 125 run_test ./test/vhost/spdk_vhost.sh --integrity-blk [19:39:37] => 126 run_test ./test/vhost/spdk_vhost.sh --integrity [20:10:31] sheesh [20:56:31] *** Joins: ziyeyang_ (~ziyeyang@134.134.139.74) [20:56:47] I also faced this issue [20:56:59] For my GPT rpc tet [20:57:01] test [21:06:16] It is a random failure