kernel/io_uring/zcrx.c, branch linux-rolling-stable

io_uring/zcrx: use READ_ONCE with user shared RQEs

2026-03-19T15:15:19Z

commit 531bb98a030cc1073bd7ed9a502c0a3a781e92ee upstream. Refill queue entries are shared with the user space, use READ_ONCE when reading them. Fixes: 34a3e60821ab9 ("io_uring/zcrx: implement zerocopy receive pp memory provider"); Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman

io_uring/zcrx: fix user_ref race between scrub and refill paths

2026-03-04T12:21:45Z

[ Upstream commit 003049b1c4fb8aabb93febb7d1e49004f6ad653b ] The io_zcrx_put_niov_uref() function uses a non-atomic check-then-decrement pattern (atomic_read followed by separate atomic_dec) to manipulate user_refs. This is serialized against other callers by rq_lock, but io_zcrx_scrub() modifies the same counter with atomic_xchg() WITHOUT holding rq_lock. On SMP systems, the following race exists: CPU0 (refill, holds rq_lock) CPU1 (scrub, no rq_lock) put_niov_uref: atomic_read(uref) - 1 // window opens atomic_xchg(uref, 0) - 1 return_niov_freelist(niov) [PUSH #1] // window closes atomic_dec(uref) - wraps to -1 returns true return_niov(niov) return_niov_freelist(niov) [PUSH #2: DOUBLE-FREE] The same niov is pushed to the freelist twice, causing free_count to exceed nr_iovs. Subsequent freelist pushes then perform an out-of-bounds write (a u32 value) past the kvmalloc'd freelist array into the adjacent slab object. Fix this by replacing the non-atomic read-then-dec in io_zcrx_put_niov_uref() with an atomic_try_cmpxchg loop that atomically tests and decrements user_refs. This makes the operation safe against concurrent atomic_xchg from scrub without requiring scrub to acquire rq_lock. Fixes: 34a3e60821ab ("io_uring/zcrx: implement zerocopy receive pp memory provider") Cc: stable@vger.kernel.org Signed-off-by: Kai Aizen [pavel: removed a warning and a comment] Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/zcrx: check unsupported flags on import

2026-03-04T12:21:43Z

[ Upstream commit 7496e658a76a61758b20e27cea8abcfeafe3aec4 ] The imoorted zcrx registration path checks for ZCRX_REG_IMPORT, as it should, but doesn't reject any unsupported flags. Fix that. Cc: stable@vger.kernel.org Fixes: 00d91481279fb ("io_uring/zcrx: share an ifq between rings") Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/zcrx: fix post open error handling

2026-03-04T12:21:43Z

[ Upstream commit 5d540e4508950c674d6feef1d95463d039bbf4f5 ] Closing a queue doesn't guarantee that all associated page pools are terminated right away, let the refcounting do the work instead of releasing the zcrx ctx directly. Cc: stable@vger.kernel.org Fixes: e0793de24a9f6 ("io_uring/zcrx: set pp memory provider for an rx queue") Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/zcrx: fix sgtable leak on mapping failures

2026-03-04T12:21:43Z

[ Upstream commit a983aae397767e9da931128ff2b5bf9066513ce3 ] In an unlikely case when io_populate_area_dma() fails, which could only happen on a PAGE_POOL_32BIT_ARCH_WITH_64BIT_DMA machine, io_zcrx_map_area() will have an initialised and not freed table. It was supposed to be cleaned up in the error path, but !is_mapped prevents that. Fixes: 439a98b972fbb ("io_uring/zcrx: deduplicate area mapping") Cc: stable@vger.kernel.org Reported-by: Jens Axboe Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin

io_uring/zcrx: fix rq flush locking

2026-02-02T15:19:43Z

zcrx needs to keep the rq lock for uref manipulations, for now move all zcrx_return_buffers() under the lock. Fixes: 475eb39b00478 ("io_uring/zcrx: add sync refill queue flushing") Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe

io_uring/zcrx: fix page array leak

2026-02-02T15:19:35Z

d9f595b9a65e ("io_uring/zcrx: fix leaking pages on sg init fail") fixed a page leakage but didn't free the page array, release it as well. Fixes: b84621d96ee02 ("io_uring/zcrx: allocate sgtable for umem areas") Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe

io_uring/zcrx: share an ifq between rings

2025-11-13T18:19:37Z

Add a way to share an ifq from a src ring that is real (i.e. bound to a HW RX queue) with other rings. This is done by passing a new flag IORING_ZCRX_IFQ_REG_IMPORT in the registration struct io_uring_zcrx_ifq_reg, alongside the fd of an exported zcrx ifq. Signed-off-by: David Wei Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe

io_uring/zcrx: add io_fill_zcrx_offsets()

2025-11-13T18:19:37Z

Add a helper io_fill_zcrx_offsets() that sets the constant offsets in struct io_uring_zcrx_offsets returned to userspace. Signed-off-by: David Wei Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe

io_uring/zcrx: export zcrx via a file

2025-11-13T18:19:37Z

Add an option to wrap a zcrx instance into a file and expose it to the user space. Currently, users can't do anything meaningful with the file, but it'll be used in a next patch to import it into another io_uring instance. It's implemented as a new op called ZCRX_CTRL_EXPORT for the IORING_REGISTER_ZCRX_CTRL registration opcode. Signed-off-by: David Wei Signed-off-by: Pavel Begunkov Signed-off-by: Jens Axboe