/* empty file to figure out endianness / word size */ nk rel='self' href='https://universe.0xinfinity.dev/distro/kernel/atom?h=linux-5.0.y'/> 2018-12-20T19:17:05Z sbitmap: add helpers for add/del wait queue handling 2018-12-20T19:17:05Z Jens Axboe axboe@kernel.dk 2018-12-20T15:49:00Z urn:sha1:9f6b7ef6c3ebe35be77b0ae3cf12e4d25ae80420 After commit 5d2ee7122c73, users of sbitmap that need wait queue handling must use the provided helpers. But we only added prepare_to_wait()/finish_wait() style helpers, add the equivalent add_wait_queue/list_del wrappers as we.. This is needed to ensure kyber plays by the sbitmap waitqueue rules. Tested-by: Ming Lei <ming.lei@redhat.com> Reviewed-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> sbitmap: fix sbitmap_for_each_set() 2018-12-04T00:03:58Z Omar Sandoval osandov@fb.com 2018-12-03T22:45:43Z urn:sha1:8c2def893afc60d88160d524acf345765cf0c447 We need to ignore bits in the cleared mask when iterating over all set bits. Fixes: ea86ea2cdced ("sbitmap: ammortize cost of clearing bits") Reported-by: Jens Axboe@kernel.dk> Signed-off-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> sbitmap: optimize wakeup check 2018-11-30T21:48:04Z Jens Axboe axboe@kernel.dk 2018-11-30T00:36:41Z urn:sha1:5d2ee7122c73be6a3b6bfe90d237e8aed737cfaa Even if we have no waiters on any of the sbitmap_queue wait states, we still have to loop every entry to check. We do this for every IO, so the cost adds up. Shift a bit of the cost to the slow path, when we actually have waiters. Wrap prepare_to_wait_exclusive() and finish_wait(), so we can maintain an internal count of how many are currently active. Then we can simply check this count in sbq_wake_ptr() and not have to loop if we don't have any sleepers. Convert the two users of sbitmap with waiting, blk-mq-tag and iSCSI. Reviewed-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> sbitmap: ammortize cost of clearing bits 2018-11-30T21:47:45Z Jens Axboe axboe@kernel.dk 2018-11-30T20:18:06Z urn:sha1:ea86ea2cdced20057da4d2c32965c1219c238197 sbitmap maintains a set of words that we use to set and clear bits, with each bit representing a tag for blk-mq. Even though we spread the bits out and maintain a hint cache, one particular bit allocated will end up being cleared in the exact same spot. This introduces batched clearing of bits. Instead of clearing a given bit, the same bit is set in a cleared/free mask instead. If we fail allocating a bit from a given word, then we check the free mask, and batch move those cleared bits at that time. This trades 64 atomic bitops for 2 cmpxchg(). In a threaded poll test case, half the overhead of getting and clearing tags is removed with this change. On another poll test case with a single thread, performance is unchanged. Reviewed-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> scsi: target: sbitmap: add seq_file forward declaration 2018-07-11T02:42:47Z Arnd Bergmann arnd@arndb.de 2018-07-06T20:19:07Z urn:sha1:14b470b56840dbb093abd71f214e9d63770c87b8 The target core runs into a warning in the linux/sbitmap.h file in some configurations: In file included from include/target/target_core_base.h:7, from drivers/target/target_core_fabric_lib.c:41: include/linux/sbitmap.h:331:46: error: 'struct seq_file' declared inside parameter list will not be visible outside of this definition or declaration [-Werror] void sbitmap_show(struct sbitmap *sb, struct seq_file *m); ^~~~~~~~ In general, headers should not depend on others being included first, so this fixes it with a forward declaration for that struct name, but we probably want to merge the patch through the scsi tree to help bisection. Fixes: 10e9cbb6b531 ("scsi: target: Convert target drivers to use sbitmap") Signed-off-by: Arnd Bergmann <arnd@arndb.de> Reviewed-by: Bart Van Assche <bart.vanassche@wdc.com> Reviewed-by: Omar Sandoval <osandov@fb.com> Reviewed-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> blk-mq: avoid starving tag allocation after allocating process migrates 2018-05-24T17:00:39Z Ming Lei ming.lei@redhat.com 2018-05-24T17:00:39Z urn:sha1:e6fc46498784e799d3eb95d83079180e413c4e7d When the allocation process is scheduled back and the mapped hw queue is changed, fake one extra wake up on previous queue for compensating wake up miss, so other allocations on the previous queue won't be starved. This patch fixes one request allocation hang issue, which can be triggered easily in case of very low nr_request. The race is as follows: 1) 2 hw queues, nr_requests are 2, and wake_batch is one 2) there are 3 waiters on hw queue 0 3) two in-flight requests in hw queue 0 are completed, and only two waiters of 3 are waken up because of wake_batch, but both the two waiters can be scheduled to another CPU and cause to switch to hw queue 1 4) then the 3rd waiter will wait for ever, since no in-flight request is in hw queue 0 any more. 5) this patch fixes it by the fake wakeup when waiter is scheduled to another hw queue Cc: <stable@vger.kernel.org> Reviewed-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Ming Lei <ming.lei@redhat.com> Modified commit message to make it clearer, and make it apply on top of the 4.18 branch. Signed-off-by: Jens Axboe <axboe@kernel.dk> sbitmap: fix missed wakeups caused by sbitmap_queue_get_shallow() 2018-05-10T17:27:36Z Omar Sandoval osandov@fb.com 2018-05-10T00:16:31Z urn:sha1:a327553965dede92587e6ccbe7df98dba36edcea The sbitmap queue wake batch is calculated such that once allocations start blocking, all of the bits which are already allocated must be enough to fulfill the batch counters of all of the waitqueues. However, the shallow allocation depth can break this invariant, since we block before our full depth is being utilized. Add sbitmap_queue_min_shallow_depth(), which saves the minimum shallow depth the sbq will use, and update sbq_calc_wake_batch() to take it into account. Acked-by: Paolo Valente <paolo.valente@linaro.org> Signed-off-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> sbitmap: use test_and_set_bit_lock()/clear_bit_unlock() 2018-02-28T19:23:35Z Omar Sandoval osandov@fb.com 2018-02-28T00:56:43Z urn:sha1:4ace53f1ed40a5cfee4bdd7614c8a8b2798227ad sbitmap_queue_get()/sbitmap_queue_clear() are used for allocating/freeing a resource, so they should provide acquire/release barrier semantics, respectively. sbitmap_get() currently contains a full barrier, which is unnecessary, so use test_and_set_bit_lock() instead of test_and_set_bit() (these are equivalent on x86_64). sbitmap_clear_bit() does not imply any barriers, which is incorrect, as accesses of the resource (e.g., request) could potentially get reordered to after the clear_bit(). Introduce sbitmap_clear_bit_unlock() and use it for sbitmap_queue_clear() (this only adds a compiler barrier on x86_64). The other existing user of sbitmap_clear_bit() (the blk-mq software queue pending map) is serialized through a spinlock and does not need this. Reported-by: Tejun Heo <tj@kernel.org> Acked-by: Tejun Heo <tj@kernel.org> Signed-off-by: Omar Sandoval <osandov@fb.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>