kernel/block/blk-mq-tag.c, branch linux-6.19.y

blk-mq: use array manage hctx map instead of xarray

2025-11-28T16:09:19Z

After commit 4e5cc99e1e48 ("blk-mq: manage hctx map via xarray"), we use an xarray instead of array to store hctx, but in poll mode, each time in blk_mq_poll, we need use xa_load to find corresponding hctx, this introduce some costs. In my test, xa_load may cost 3.8% cpu. This patch revert previous change, eliminates the overhead of xa_load and can result in a 3% performance improvement. Signed-off-by: Fengnan Chang Signed-off-by: Jens Axboe

blk-mq: fix stale tag depth for shared sched tags in blk_mq_update_nr_requests()

2025-10-15T13:49:19Z

Commit 7f2799c546db ("blk-mq: cleanup shared tags case in blk_mq_update_nr_requests()") moves blk_mq_tag_update_sched_shared_tags() before q->nr_requests is updated, however, it's still using the old q->nr_requests to resize tag depth. Fix this problem by passing in expected new tag depth. Fixes: 7f2799c546db ("blk-mq: cleanup shared tags case in blk_mq_update_nr_requests()") Signed-off-by: Yu Kuai Reviewed-by: Ming Lei Reviewed-by: Nilay Shroff Reported-by: Chris Mason Link: https://lore.kernel.org/linux-block/20251014130507.4187235-2-clm@meta.com/ Signed-off-by: Jens Axboe

blk-mq: fix null-ptr-deref in blk_mq_free_tags() from error path

2025-09-23T07:35:52Z

blk_mq_free_tags() can be called after blk_mq_init_tags(), while tags->page_list is still not initialized, causing null-ptr-deref. Fix this problem by initializing tags->page_list at blk_mq_init_tags(), meanwhile, also free tags directly from error path because there is no srcu barrier. Fixes: ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback") Reported-by: syzbot+5c5d41e80248d610221f@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/68d1b079.a70a0220.1b52b.0000.GAE@google.com/ Signed-off-by: Yu Kuai Signed-off-by: Jens Axboe

blk-mq: Fix more tag iteration function documentation

2025-09-23T06:20:31Z

Commit 8ab30a331946 ("blk-mq: Drop busy_iter_fn blk_mq_hw_ctx argument") removed the hctx argument from the callback functions called by bt_for_each() and blk_mq_queue_tag_busy_iter(). Commit 2dd6532e9591 ("blk-mq: Drop 'reserved' arg of busy_tag_iter_fn") removed the 'reserved' argument of the busy_tag_iter_fn function pointer type. Bring the documentation of the tag iteration functions in sync with these changes. Cc: John Garry Cc: Ming Lei Cc: Christoph Hellwig Signed-off-by: Bart Van Assche Signed-off-by: Jens Axboe

blk-mq: Fix the blk_mq_tagset_busy_iter() documentation

2025-09-17T13:26:29Z

Commit 2dd6532e9591 ("blk-mq: Drop 'reserved' arg of busy_tag_iter_fn") removed the 'reserved' argument from tag iteration callback functions. Bring the blk_mq_tagset_busy_iter() documentation in sync with that change. Cc: Jens Axboe Cc: Christoph Hellwig Cc: Ming Lei Cc: John Garry Signed-off-by: Bart Van Assche Reviewed-by: Ming Lei Reviewed-by: Martin K. Petersen Signed-off-by: Jens Axboe

blk-mq: remove blk_mq_tag_update_depth()

2025-09-10T11:25:56Z

This helper is not used now. Signed-off-by: Yu Kuai Reviewed-by: Nilay Shroff Signed-off-by: Jens Axboe

blk-mq: cleanup shared tags case in blk_mq_update_nr_requests()

2025-09-10T11:25:56Z

For shared tags case, all hctx->sched_tags/tags are the same, it doesn't make sense to call into blk_mq_tag_update_depth() multiple times for the same tags. Signed-off-by: Yu Kuai Reviewed-by: Nilay Shroff Signed-off-by: Jens Axboe

blk-mq: check invalid nr_requests in queue_requests_store()

2025-09-10T11:25:56Z

queue_requests_store() is the only caller of blk_mq_update_nr_requests(), and blk_mq_update_nr_requests() is the only caller of blk_mq_tag_update_depth(), however, they all have checkings for nr_requests input by user. Make code cleaner by moving all the checkings to the top function: 1) nr_requests > reserved tags; 2) if there is elevator, 4 <= nr_requests <= 2048; 3) if elevator is none, 4 <= nr_requests <= tag_set->queue_depth; Meanwhile, case 2 is the only case tags can grow and -ENOMEM might be returned. Signed-off-by: Yu Kuai Reviewed-by: Nilay Shroff Signed-off-by: Jens Axboe

blk-mq: Replace tags->lock with SRCU for tag iterators

2025-09-08T14:05:32Z

Replace the spinlock in blk_mq_find_and_get_req() with an SRCU read lock around the tag iterators. This is done by: - Holding the SRCU read lock in blk_mq_queue_tag_busy_iter(), blk_mq_tagset_busy_iter(), and blk_mq_hctx_has_requests(). - Removing the now-redundant tags->lock from blk_mq_find_and_get_req(). This change fixes lockup issue in scsi_host_busy() in case of shost->host_blocked. Also avoids big tags->lock when reading disk sysfs attribute `inflight`. Reviewed-by: Hannes Reinecke Signed-off-by: Ming Lei Reviewed-by: Yu Kuai Reviewed-by: Martin K. Petersen Signed-off-by: Jens Axboe

blk-mq: Defer freeing of tags page_list to SRCU callback

2025-09-08T14:05:32Z

Tag iterators can race with the freeing of the request pages(tags->page_list), potentially leading to use-after-free issues. Defer the freeing of the page list and the tags structure itself until after an SRCU grace period has passed. This ensures that any concurrent tag iterators have completed before the memory is released. With this way, we can replace the big tags->lock in tags iterator code path with srcu for solving the issue. This is achieved by: - Adding a new `srcu_struct tags_srcu` to `blk_mq_tag_set` to protect tag map iteration. - Adding an `rcu_head` to `struct blk_mq_tags` to be used with `call_srcu`. - Moving the page list freeing logic and the `kfree(tags)` call into a new callback function, `blk_mq_free_tags_callback`. - In `blk_mq_free_tags`, invoking `call_srcu` to schedule the new callback for deferred execution. The read-side protection for the tag iterators will be added in a subsequent patch. Reviewed-by: Hannes Reinecke Reviewed-by: Yu Kuai Signed-off-by: Ming Lei Reviewed-by: Martin K. Petersen Signed-off-by: Jens Axboe