The spdk_top application is designed to resemble the standard top in that it provides a real-time insights into CPU cores usage by SPDK lightweight threads and pollers. Have you ever wondered which CPU core is used most by your SPDK instance? Are you building your own bdev or library and want to know if your code is running efficiently? Are your new pollers busy most of the time? The spdk_top application uses RPC calls to collect performance metrics and displays them in a report that you can analyze and determine if your code is running efficiently so that you can tune your implementation and get more from SPDK.
Why doesn't the classic top utility work for SPDK? SPDK uses a polled-mode design; a reactor thread running on each CPU core assigned to an SPDK application schedules SPDK lightweight threads and pollers to run on the CPU core. Therefore, the standard Linux top utility is not effective for analyzing the CPU usage for polled-mode applications like SPDK because it just reports that they are using 100% of the CPU resources assigned to them. The spdk_top utility was developed to analyze and report the CPU cycles used to do real work vs just polling for work. The utility relies on instrumentation added to pollers to track when they are doing work vs. polling for work. The spdk_top utility gets the fine grained metrics from the pollers, analyzes and report the metrics on a per poller, thread and core basis. This information enables users to identify CPU cores that are busy doing real work so that they can determine if the application needs more or less CPU resources.
spdk_top uses RPCs to communicate with the app it is viewing, so it will work only with those that run RPC server and support thread_get_stats, thread_get_pollers, framework_get_reactors methods. Apps currently meeting this criteria: spdk_tgt, nvmf_tgt, vhost, iscsi_tgt.
Before running spdk_top you need to run the SPDK application whose performance you want to analyze using spdk_top.
Run the spdk_top application
Menu at the bottom of SPDK top window shows many options for changing displayed data. Each menu item has a key associated with it in square brackets.
The threads tab displays a line item for each spdk thread. The information displayed shows:
By pressing ENTER key a pop-up window appears, showing above and a list of pollers running on selected thread (with poller name, type, run count and period). Pop-up then can be closed by pressing ESC key.
To learn more about spdk threads see Message Passing and Concurrency.
The pollers tab displays a line item for each poller. The information displayed shows:
Poller pop-up window can be displayed by pressing ENTER on a selected data row and displays above information. Pop-up can be closed by pressing ESC key.
The cores tab provides insights into how the application is using the CPU cores assigned to it. The information displayed for each core shows:
Pressing ENTER key makes a pop-up window appear, showing above information, along with a list of threads running on selected core. Cores details window allows to select a thread and display thread details pop-up on top of it. To close both pop-ups use ESC key.
Help window pop-up can be invoked by pressing 'h' key inside any tab. It contains explanations for each key used inside the spdk_top application.
Current scheduler information may be displayed with 'g' key inside all tabs. It contains scheduler name and period along with governor name.