kernel/arch/x86/boot/compressed/pgtable_64.c, branch linux-rolling-stable

x86/boot: Fix page table access in 5-level to 4-level paging transition

2025-12-18T12:59:34Z

[ Upstream commit eb2266312507d7b757859e2227aa5c4ba6280ebe ] When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues: - __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags. Bits above the physical address width of the system i.e. above __PHYSICAL_MASK_SHIFT) are also not masked. - The PGD entry is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW. Replace this with proper accessor functions: - native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask metadata out of CR3 (like SME or LAM bits). All remaining bits are real address bits or reserved and must be 0. - mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below 51, but above the max physical address are reserved and must be 0. Fixes: e9d0e6330eb8 ("x86/boot/compressed/64: Prepare new top-level page table for trampoline") Reported-by: Michael van der Westhuizen Reported-by: Tobias Fleig Co-developed-by: Kiryl Shutsemau Signed-off-by: Kiryl Shutsemau Signed-off-by: Usama Arif Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Ard Biesheuvel Acked-by: Dave Hansen Link: https://lore.kernel.org/r/a482fd68-ce54-472d-8df1-33d6ac9f6bb5@intel.com Signed-off-by: Sasha Levin

x86/mm/64: Make 5-level paging support unconditional

2025-05-17T08:38:16Z

Both Intel and AMD CPUs support 5-level paging, which is expected to become more widely adopted in the future. All major x86 Linux distributions have the feature enabled. Remove CONFIG_X86_5LEVEL and related #ifdeffery for it to make it more readable. Suggested-by: Borislav Petkov Signed-off-by: Kirill A. Shutemov Signed-off-by: Ingo Molnar Reviewed-by: Ard Biesheuvel Reviewed-by: Borislav Petkov (AMD) Cc: Dave Hansen Cc: H. Peter Anvin Cc: Linus Torvalds Link: https://lore.kernel.org/r/20250516123306.3812286-4-kirill.shutemov@linux.intel.com

x86/boot/compressed: Merge the local pgtable.h include into

2025-04-06T18:15:14Z

Merge the local include "pgtable.h" -which declares the API of the 5-level paging trampoline- into so that its implementation in la57toggle.S as well as the calling code can be decoupled from the traditional decompressor. Signed-off-by: Ard Biesheuvel Signed-off-by: Ingo Molnar Cc: David Woodhouse Cc: H. Peter Anvin Cc: Kees Cook Cc: Linus Torvalds Link: https://lore.kernel.org/r/20250401133416.1436741-9-ardb+git@google.com

x86/boot: Sanitize boot params before parsing command line

2025-03-06T21:02:39Z

The 5-level paging code parses the command line to look for the 'no5lvl' string, and does so very early, before sanitize_boot_params() has been called and has been given the opportunity to wipe bogus data from the fields in boot_params that are not covered by struct setup_header, and are therefore supposed to be initialized to zero by the bootloader. This triggers an early boot crash when using syslinux-efi to boot a recent kernel built with CONFIG_X86_5LEVEL=y and CONFIG_EFI_STUB=n, as the 0xff padding that now fills the unused PE/COFF header is copied into boot_params by the bootloader, and interpreted as the top half of the command line pointer. Fix this by sanitizing the boot_params before use. Note that there is no harm in calling this more than once; subsequent invocations are able to spot that the boot_params have already been cleaned up. Signed-off-by: Ard Biesheuvel Signed-off-by: Ingo Molnar Cc: "H. Peter Anvin" Cc: Linus Torvalds Cc: # v6.1+ Link: https://lore.kernel.org/r/20250306155915.342465-2-ardb+git@google.com Closes: https://lore.kernel.org/all/202503041549.35913.ulrich.gemkow@ikr.uni-stuttgart.de

x86: Do not include in several files

2024-01-30T14:17:24Z

Remove the include statement for from several files that don't require it and limit the exposure of those definitions within the Linux kernel code. [ bp: Massage commit message. ] Signed-off-by: Thomas Zimmermann Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Ard Biesheuvel Acked-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20240112095000.8952-5-tzimmermann@suse.de

x86/boot: Rename conflicting 'boot_params' pointer to 'boot_params_ptr'

2023-10-18T10:03:03Z

The x86 decompressor is built and linked as a separate executable, but it shares components with the kernel proper, which are either #include'd as C files, or linked into the decompresor as a static library (e.g, the EFI stub) Both the kernel itself and the decompressor define a global symbol 'boot_params' to refer to the boot_params struct, but in the former case, it refers to the struct directly, whereas in the decompressor, it refers to a global pointer variable referring to the struct boot_params passed by the bootloader or constructed from scratch. This ambiguity is unfortunate, and makes it impossible to assign this decompressor variable from the x86 EFI stub, given that declaring it as extern results in a clash. So rename the decompressor version (whose scope is limited) to boot_params_ptr. [ mingo: Renamed 'boot_params_p' to 'boot_params_ptr' for clarity ] Signed-off-by: Ard Biesheuvel Signed-off-by: Ingo Molnar Cc: linux-kernel@vger.kernel.org

x86/decompressor: Merge trampoline cleanup with switching code

2023-08-07T18:51:17Z

Now that the trampoline setup code and the actual invocation of it are all done from the C routine, the trampoline cleanup can be merged into it as well, instead of returning to asm just to call another C function. Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) Acked-by: Kirill A. Shutemov Link: https://lore.kernel.org/r/20230807162720.545787-16-ardb@kernel.org

x86/decompressor: Pass pgtable address to trampoline directly

2023-08-07T18:49:26Z

The only remaining use of the trampoline address by the trampoline itself is deriving the page table address from it, and this involves adding an offset of 0x0. So simplify this, and pass the new CR3 value directly. This makes the fact that the page table happens to be at the start of the trampoline allocation an implementation detail of the caller. Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20230807162720.545787-15-ardb@kernel.org

x86/decompressor: Only call the trampoline when changing paging levels

2023-08-07T18:48:09Z

Since the current and desired number of paging levels are known when the trampoline is being prepared, avoid calling the trampoline at all if it is clear that calling it is not going to result in a change to the number of paging levels. Given that the CPU is already running in long mode, the PAE and LA57 settings are necessarily consistent with the currently active page tables, and other fields in CR4 will be initialized by the startup code in the kernel proper. So limit the manipulation of CR4 to toggling the LA57 bit, which is the only thing that really needs doing at this point in the boot. This also means that there is no need to pass the value of l5_required to toggle_la57(), as it will not be called unless CR4.LA57 needs to toggle. Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) Acked-by: Kirill A. Shutemov Link: https://lore.kernel.org/r/20230807162720.545787-14-ardb@kernel.org

x86/decompressor: Call trampoline directly from C code

2023-08-07T18:46:57Z

Instead of returning to the asm calling code to invoke the trampoline, call it straight from the C code that sets it up. That way, the struct return type is no longer needed for returning two values, and the call can be made conditional more cleanly in a subsequent patch. This means that all callee save 64-bit registers need to be preserved and restored, as their contents may not survive the legacy mode switch. Signed-off-by: Ard Biesheuvel Signed-off-by: Borislav Petkov (AMD) Acked-by: Kirill A. Shutemov Link: https://lore.kernel.org/r/20230807162720.545787-13-ardb@kernel.org