kernel/drivers/pci/iov.c, branch 0x221E-v0.0.1-v6.19

PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

2026-03-04T12:21:31Z

[ Upstream commit a5338e365c4559d7b4d7356116b0eb95b12e08d5 ] Commit 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV") tried to fix a race between the VF removal inside sriov_del_vfs() and concurrent hot unplug by taking the PCI rescan/remove lock in sriov_del_vfs(). Similarly the PCI rescan/remove lock was also taken in sriov_add_vfs() to protect addition of VFs. This approach however causes deadlock on trying to remove PFs with SR-IOV enabled because PFs disable SR-IOV during removal and this removal happens under the PCI rescan/remove lock. So the original fix had to be reverted. Instead of taking the PCI rescan/remove lock in sriov_add_vfs() and sriov_del_vfs(), fix the race that occurs with SR-IOV enable and disable vs hotplug higher up in the callchain by taking the lock in sriov_numvfs_store() before calling into the driver's sriov_configure() callback. Fixes: 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV") Reported-by: Benjamin Block Signed-off-by: Niklas Schnelle Signed-off-by: Bjorn Helgaas Reviewed-by: Benjamin Block Reviewed-by: Gerd Bayer Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20251216-revert_sriov_lock-v3-2-dac4925a7621@linux.ibm.com Signed-off-by: Sasha Levin

Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"

2026-03-04T12:21:31Z

[ Upstream commit 2fa119c0e5e528453ebae9e70740e8d2d8c0ed5a ] This reverts commit 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"), which causes a deadlock by recursively taking pci_rescan_remove_lock when sriov_del_vfs() is called as part of pci_stop_and_remove_bus_device(). For example with the following sequence of commands: $ echo > /sys/bus/pci/devices//sriov_numvfs $ echo 1 > /sys/bus/pci/devices//remove A trimmed trace of the deadlock on a mlx5 device is as below: zsh/5715 is trying to acquire lock: 000002597926ef50 (pci_rescan_remove_lock){+.+.}-{3:3}, at: sriov_disable+0x34/0x140 but task is already holding lock: 000002597926ef50 (pci_rescan_remove_lock){+.+.}-{3:3}, at: pci_stop_and_remove_bus_device_locked+0x24/0x80 ... Call Trace: [<00000259778c4f90>] dump_stack_lvl+0xc0/0x110 [<00000259779c844e>] print_deadlock_bug+0x31e/0x330 [<00000259779c1908>] __lock_acquire+0x16c8/0x32f0 [<00000259779bffac>] lock_acquire+0x14c/0x350 [<00000259789643a6>] __mutex_lock_common+0xe6/0x1520 [<000002597896413c>] mutex_lock_nested+0x3c/0x50 [<00000259784a07e4>] sriov_disable+0x34/0x140 [<00000258f7d6dd80>] mlx5_sriov_disable+0x50/0x80 [mlx5_core] [<00000258f7d5745e>] remove_one+0x5e/0xf0 [mlx5_core] [<00000259784857fc>] pci_device_remove+0x3c/0xa0 [<000002597851012e>] device_release_driver_internal+0x18e/0x280 [<000002597847ae22>] pci_stop_bus_device+0x82/0xa0 [<000002597847afce>] pci_stop_and_remove_bus_device_locked+0x5e/0x80 [<00000259784972c2>] remove_store+0x72/0x90 [<0000025977e6661a>] kernfs_fop_write_iter+0x15a/0x200 [<0000025977d7241c>] vfs_write+0x24c/0x300 [<0000025977d72696>] ksys_write+0x86/0x110 [<000002597895b61c>] __do_syscall+0x14c/0x400 [<000002597896e0ee>] system_call+0x6e/0x90 This alone is not a complete fix as it restores the issue the cited commit tried to solve. A new fix will be provided as a follow on. Fixes: 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV") Reported-by: Benjamin Block Signed-off-by: Niklas Schnelle Signed-off-by: Bjorn Helgaas Reviewed-by: Benjamin Block Acked-by: Gerd Bayer Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20251216-revert_sriov_lock-v3-1-dac4925a7621@linux.ibm.com Signed-off-by: Sasha Levin

PCI: Convert BAR sizes bitmasks to u64

2025-11-14T18:34:22Z

PCIe r7.0, sec 7.8.6, defines resizable BAR sizes beyond the currently supported maximum of 128TB, which will require more than u32 to store the entire bitmask. Convert Resizable BAR related functions to use u64 bitmask for BAR sizes to make the typing more future-proof. The support for the larger BAR sizes themselves is not added at this point. Signed-off-by: Ilpo Järvinen Signed-off-by: Bjorn Helgaas Reviewed-by: Christian König Link: https://patch.msgid.link/20251113180053.27944-12-ilpo.jarvinen@linux.intel.com

PCI: Add pci_rebar_size_supported() helper

2025-11-14T18:34:21Z

Many callers of pci_rebar_get_possible_sizes() are interested in finding out if a particular encoded BAR Size (PCIe r7.0, sec 7.8.6.3) is supported by the particular BAR. Add pci_rebar_size_supported() into PCI core to make it easy for the drivers to determine if the BAR size is supported or not. Use the new function in pci_resize_resource() and in pci_iov_vf_bar_set_size(). Signed-off-by: Ilpo Järvinen Signed-off-by: Bjorn Helgaas Reviewed-by: Christian König Reviewed-by: Andi Shyti Link: https://patch.msgid.link/20251113180053.27944-6-ilpo.jarvinen@linux.intel.com

PCI/IOV: Adjust ->barsz[] when changing BAR size

2025-11-14T18:32:47Z

pci_rebar_set_size() adjusts BAR size for both normal and IOV BARs. The struct pci_sriov keeps a cached copy of BAR size in ->barsz[] which is not adjusted by pci_rebar_set_size() but by pci_iov_resource_set_size(). pci_iov_resource_set_size() is called also from pci_resize_resource_set_size(). The current arrangement is problematic once BAR resize algorithm starts to roll back changes properly in case of a failure. The normal resource fitting algorithm rolls back resource size using the struct pci_dev_resource easily but also calling pci_resize_resource_set_size() or pci_iov_resource_set_size() to roll back BAR size would be an extra burden, whereas combining ->barsz[] update with pci_rebar_set_size() naturally rolls back it when restoring the old BAR size on a different layer of the BAR resize operation. Thus, rework pci_rebar_set_size() to also update ->barsz[]. Signed-off-by: Ilpo Järvinen Signed-off-by: Bjorn Helgaas Tested-by: Alex Bennée # AVA, AMD GPU Link: https://patch.msgid.link/20251113162628.5946-3-ilpo.jarvinen@linux.intel.com

PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV

2025-09-26T20:55:59Z

Before disabling SR-IOV via config space accesses to the parent PF, sriov_disable() first removes the PCI devices representing the VFs. Since commit 9d16947b7583 ("PCI: Add global pci_lock_rescan_remove()") such removal operations are serialized against concurrent remove and rescan using the pci_rescan_remove_lock. No such locking was ever added in sriov_disable() however. In particular when commit 18f9e9d150fc ("PCI/IOV: Factor out sriov_add_vfs()") factored out the PCI device removal into sriov_del_vfs() there was still no locking around the pci_iov_remove_virtfn() calls. On s390 the lack of serialization in sriov_disable() may cause double remove and list corruption with the below (amended) trace being observed: PSW: 0704c00180000000 0000000c914e4b38 (klist_put+56) GPRS: 000003800313fb48 0000000000000000 0000000100000001 0000000000000001 00000000f9b520a8 0000000000000000 0000000000002fbd 00000000f4cc9480 0000000000000001 0000000000000000 0000000000000000 0000000180692828 00000000818e8000 000003800313fe2c 000003800313fb20 000003800313fad8 #0 [3800313fb20] device_del at c9158ad5c #1 [3800313fb88] pci_remove_bus_device at c915105ba #2 [3800313fbd0] pci_iov_remove_virtfn at c9152f198 #3 [3800313fc28] zpci_iov_remove_virtfn at c90fb67c0 #4 [3800313fc60] zpci_bus_remove_device at c90fb6104 #5 [3800313fca0] __zpci_event_availability at c90fb3dca #6 [3800313fd08] chsc_process_sei_nt0 at c918fe4a2 #7 [3800313fd60] crw_collect_info at c91905822 #8 [3800313fe10] kthread at c90feb390 #9 [3800313fe68] __ret_from_fork at c90f6aa64 #10 [3800313fe98] ret_from_fork at c9194f3f2. This is because in addition to sriov_disable() removing the VFs, the platform also generates hot-unplug events for the VFs. This being the reverse operation to the hotplug events generated by sriov_enable() and handled via pdev->no_vf_scan. And while the event processing takes pci_rescan_remove_lock and checks whether the struct pci_dev still exists, the lack of synchronization makes this checking racy. Other races may also be possible of course though given that this lack of locking persisted so long observable races seem very rare. Even on s390 the list corruption was only observed with certain devices since the platform events are only triggered by config accesses after the removal, so as long as the removal finished synchronously they would not race. Either way the locking is missing so fix this by adding it to the sriov_del_vfs() helper. Just like PCI rescan-remove, locking is also missing in sriov_add_vfs() including for the error case where pci_stop_and_remove_bus_device() is called without the PCI rescan-remove lock being held. Even in the non-error case, adding new PCI devices and buses should be serialized via the PCI rescan-remove lock. Add the necessary locking. Fixes: 18f9e9d150fc ("PCI/IOV: Factor out sriov_add_vfs()") Signed-off-by: Niklas Schnelle Signed-off-by: Bjorn Helgaas Reviewed-by: Benjamin Block Reviewed-by: Farhan Ali Reviewed-by: Julian Ruess Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20250826-pci_fix_sriov_disable-v1-1-2d0bc938f2a3@linux.ibm.com

PCI/IOV: Allow drivers to control VF BAR size

2025-07-14T20:04:23Z

Drivers could leverage the fact that the VF BAR MMIO reservation is created for total number of VFs supported by the device by resizing the BAR to larger size when smaller number of VFs is enabled. Add pci_iov_vf_bar_set_size() to control the size and a pci_iov_vf_bar_get_sizes() helper to get the VF BAR sizes that will allow up to num_vfs to be successfully enabled with the current underlying reservation size. Signed-off-by: Michał Winiarski Signed-off-by: Bjorn Helgaas Reviewed-by: Ilpo Järvinen Link: https://patch.msgid.link/20250702093522.518099-6-michal.winiarski@intel.com

PCI/IOV: Check that VF BAR fits within the reservation

2025-07-14T20:04:23Z

When the resource representing a VF MMIO BAR reservation is created, its size is always large enough to accommodate the BAR of all SR-IOV Virtual Functions that can potentially be created (total VFs). If for whatever reason it's not possible to accommodate all VFs, the resource is not assigned and no VFs can be created. An upcoming change will allow VF BAR size to be modified by drivers at a later point in time, which means that the check for resource assignment is no longer sufficient. Add an additional check that verifies that the VF BAR for all enabled VFs fits within the underlying reservation resource. Signed-off-by: Michał Winiarski Signed-off-by: Bjorn Helgaas Reviewed-by: Ilpo Järvinen Link: https://patch.msgid.link/20250702093522.518099-5-michal.winiarski@intel.com

PCI/IOV: Allow IOV resources to be resized in pci_resize_resource()

2025-07-14T20:04:18Z

Similar to regular resizable BARs, VF BARs can also be resized. The capability layout is the same as PCI_EXT_CAP_ID_REBAR, which means we can reuse most of the implementation, the only difference being resource size calculation (which is multiplied by total VFs) and memory decoding (which is controlled by a separate VF MSE field in SR-IOV cap). Extend the pci_resize_resource() function to accept IOV resources. See PCIe r6.2, sec 7.8.7. Signed-off-by: Michał Winiarski Signed-off-by: Bjorn Helgaas Reviewed-by: Ilpo Järvinen Link: https://patch.msgid.link/20250702093522.518099-4-michal.winiarski@intel.com

PCI/IOV: Add pci_resource_num_to_vf_bar() to convert VF BAR number to/from IOV resource

2025-07-14T20:03:23Z

There are multiple places where conversions between IOV resources and corresponding VF BAR numbers are done. Extract the logic to pci_resource_num_from_vf_bar() and pci_resource_num_to_vf_bar() helpers. Suggested-by: Ilpo Järvinen Signed-off-by: Michał Winiarski Signed-off-by: Bjorn Helgaas Reviewed-by: Ilpo Järvinen Acked-by: Christian König Link: https://patch.msgid.link/20250702093522.518099-3-michal.winiarski@intel.com