Introduction
When debugging an spdk application using gdb we may need to view data structures in lists, e.g. information about bdevs or threads.
If, for example I have several bdevs, and I wish to get information on bdev by the name 'test_vols3', I will need to manually iterate over the list as follows:
(gdb) p g_bdev_mgr->bdevs->tqh_first->name
$5 = 0x7f7dcc0b21b0 "test_vols1"
(gdb) p g_bdev_mgr->bdevs->tqh_first->internal->link->tqe_next->name
$6 = 0x7f7dcc0b1a70 "test_vols2"
(gdb) p
g_bdev_mgr->bdevs->tqh_first->internal->link->tqe_next->internal->link->tqe_next->name
$7 = 0x7f7dcc215a00 "test_vols3"
(gdb) p
g_bdev_mgr->bdevs->tqh_first->internal->link->tqe_next->internal->link->tqe_next
$8 = (struct spdk_bdev *) 0x7f7dcc2c7c08
At this stage, we can start looking at the relevant fields of our bdev which now we know is in address 0x7f7dcc2c7c08.
This can be somewhat troublesome if there are 100 bdevs, and the one we need is 56th in the list...
Instead, we can use a gdb macro in order to get information about all the devices.
Examples:
Printing bdevs:
(gdb) spdk_print_bdevs
SPDK object of type struct spdk_bdev at 0x7f7dcc1642a8
((struct spdk_bdev*) 0x7f7dcc1642a8)
name 0x7f7dcc0b21b0 "test_vols1"
---------------
SPDK object of type struct spdk_bdev at 0x7f7dcc216008
((struct spdk_bdev*) 0x7f7dcc216008)
name 0x7f7dcc0b1a70 "test_vols2"
---------------
SPDK object of type struct spdk_bdev at 0x7f7dcc2c7c08
((struct spdk_bdev*) 0x7f7dcc2c7c08)
name 0x7f7dcc215a00 "test_vols3"
---------------
Finding a bdev by name:
(gdb) spdk_find_bdev test_vols1
test_vols1
SPDK object of type struct spdk_bdev at 0x7f7dcc1642a8
((struct spdk_bdev*) 0x7f7dcc1642a8)
name 0x7f7dcc0b21b0 "test_vols1"
Printing spdk threads:
(gdb) spdk_print_threads
SPDK object of type struct spdk_thread at 0x7fffd0008b50
((struct spdk_thread*) 0x7fffd0008b50)
name 0x7fffd00008e0 "reactor_1"
IO Channels:
SPDK object of type struct spdk_io_channel at 0x7fffd0052610
((struct spdk_io_channel*) 0x7fffd0052610)
name
ref 1
device 0x7fffd0008c80 (0x7fffd0008ce0 "nvmf_tgt")
---------------
SPDK object of type struct spdk_io_channel at 0x7fffd0056cd0
((struct spdk_io_channel*) 0x7fffd0056cd0)
name
ref 2
device 0x7fffd0056bf0 (0x7fffd0008e70 "test_vol1")
---------------
SPDK object of type struct spdk_io_channel at 0x7fffd00582e0
((struct spdk_io_channel*) 0x7fffd00582e0)
name
ref 1
device 0x7fffd0056c50 (0x7fffd0056cb0 "bdev_test_vol1")
---------------
SPDK object of type struct spdk_io_channel at 0x7fffd00583b0
((struct spdk_io_channel*) 0x7fffd00583b0)
name
ref 1
device 0x7fffd0005630 (0x7fffd0005690 "bdev_mgr")
---------------
Printing nvmf subsystems:
(gdb) spdk_print_nvmf_subsystems
SPDK object of type struct spdk_nvmf_subsystem at 0x7fffd0008d00
((struct spdk_nvmf_subsystem*) 0x7fffd0008d00)
name "nqn.2014-08.org.nvmexpress.discovery", '\000' <repeats 187 times>
nqn "nqn.2014-08.org.nvmexpress.discovery", '\000' <repeats 187 times>
ID 0
---------------
SPDK object of type struct spdk_nvmf_subsystem at 0x7fffd0055760
((struct spdk_nvmf_subsystem*) 0x7fffd0055760)
name "nqn.2016-06.io.spdk.umgmt:cnode1", '\000' <repeats 191 times>
nqn "nqn.2016-06.io.spdk.umgmt:cnode1", '\000' <repeats 191 times>
ID 1
Printing SPDK spinlocks:
In this example, the spinlock has been initialized and locked but has never been unlocked. After it is unlocked the first time the last unlocked stack will be present and the Locked by spdk_thread line will say not locked.
Breakpoint 2, spdk_spin_unlock (sspin=0x655110 <g_bdev_mgr+80>) at thread.c:2915
2915 struct spdk_thread *thread = spdk_get_thread();
(gdb) print *sspin
$2 = struct spdk_spinlock:
Locked by spdk_thread: 0x658080
Initialized at:
0x43e677 <spdk_spin_init+213> thread.c:2878
0x404feb <_bdev_init+16> /build/spdk/spdk-review-public/lib/bdev/bdev.c:116
0x44483d <__libc_csu_init+77>
0x7ffff62c9d18 <__libc_start_main+120>
0x40268e <_start+46>
Last locked at:
0x43e936 <spdk_spin_lock+436> thread.c:2909
0x40ca9c <bdev_name_add+129> /build/spdk/spdk-review-public/lib/bdev/bdev.c:3855
0x411a3c <bdev_register+641> /build/spdk/spdk-review-public/lib/bdev/bdev.c:6660
0x412e1e <spdk_bdev_register+24> /build/spdk/spdk-review-public/lib/bdev/bdev.c:7171
0x417895 <num_blocks_test+119> bdev_ut.c:878
0x7ffff7bc38cb <run_single_test.constprop+379>
0x7ffff7bc3b61 <run_single_suite.constprop+433>
0x7ffff7bc3f76 <CU_run_all_tests+118>
0x43351f <main+1439> bdev_ut.c:6295
0x7ffff62c9d85 <__libc_start_main+229>
0x40268e <_start+46>
Last unlocked at:
Print a single spinlock stack:
(gdb) print sspin->internal.lock_stack
$1 = struct sspin_stack:
0x40c6a1 <spdk_spin_lock+436> /build/spdk/spdk-review-public/lib/thread/thread.c:2909
0x413f48 <spdk_spin+552> thread_ut.c:1831
0x7ffff7bc38cb <run_single_test.constprop+379>
0x7ffff7bc3b61 <run_single_suite.constprop+433>
0x7ffff7bc3f76 <CU_run_all_tests+118>
0x4148fa <main+547> thread_ut.c:1948
0x7ffff62c9d85 <__libc_start_main+229>
0x40248e <_start+46>
Loading The gdb Macros
Copy the gdb macros to the host where you are about to debug. It is best to copy the file either to somewhere within the PYTHONPATH, or to add the destination directory to the PYTHONPATH. This is not mandatory, and can be worked around, but can save a few steps when loading the module to gdb.
From gdb, with the application core open, invoke python and load the modules.
In the example below, I copied the macros to the /tmp directory which is not in the PYTHONPATH, so I had to manually add the directory to the path.
(gdb) python
>import sys
>sys.path.append('/tmp')
>import gdb_macros
>end
(gdb) spdk_load_macros
Using the gdb Data Directory
On most systems, the data directory is /usr/share/gdb. The python script should be copied into the python/gdb/function (or python/gdb/command) directory under the data directory, e.g. /usr/share/gdb/python/gdb/function.
If the python script is in there, then the only thing you need to do when starting gdb is type "spdk_load_macros".
Using .gdbinit To Load The Macros
.gdbinit can also be used in order to run automatically run the manual steps above prior to starting gdb.
Example .gdbinit:
source /opt/km/install/tools/gdb_macros/gdb_macros.py
When starting gdb you still have to call spdk_load_macros.
Why Do We Need to Explicitly Call spdk_load_macros
The reason is that the macros need to use globals provided by spdk in order to iterate the spdk lists and build iterable representations of the list objects. This will result in errors if these are not available which is very possible if gdb is used for reasons other than debugging spdk core dumps.
In the example below, I attempted to load the macros when the globals are not available causing gdb to fail loading the gdb_macros:
(gdb) spdk_load_macros
Traceback (most recent call last):
File "/opt/km/install/tools/gdb_macros/gdb_macros.py", line 257, in invoke
spdk_print_threads()
File "/opt/km/install/tools/gdb_macros/gdb_macros.py", line 241, in __init__
threads = SpdkThreads()
File "/opt/km/install/tools/gdb_macros/gdb_macros.py", line 234, in __init__
super(SpdkThreads, self).__init__('g_threads', SpdkThread)
File "/opt/km/install/tools/gdb_macros/gdb_macros.py", line 25, in __init__
['tailq'])
File "/opt/km/install/tools/gdb_macros/gdb_macros.py", line 10, in __init__
self.list = gdb.parse_and_eval(self.list_pointer)
RuntimeError: No symbol table is loaded. Use the "file" command.
Error occurred in Python command: No symbol table is loaded. Use the "file"
command.
Macros available
- spdk_load_macros: load the macros (use –reload in order to reload them)
- spdk_print_bdevs: information about bdevs
- spdk_find_bdev: find a bdev (substring search)
- spdk_print_io_devices: information about io devices
- spdk_print_nvmf_subsystems: information about nvmf subsystems
- spdk_print_threads: information about threads
Adding New Macros
The list iteration macros are usually built from 3 layers:
- SpdkPrintCommand: inherits from gdb.Command and invokes the list iteration
- SpdkTailqList: Performs the iteration of a tailq list according to the tailq member implementation
- SpdkObject: Provides the str function so that the list iteration can print the object
Other useful objects:
- SpdkNormalTailqList: represents a list which has 'tailq' as the tailq object
- SpdkArr: Iteration over an array (instead of a linked list)