arm64: entry: fix ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD

Overview

In the Linux kernel, the following vulnerability has been resolved:

arm64: entry: fix ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD

Currently the ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD workaround isn't quite right, as it is supposed to be applied after the last explicit memory access, but is immediately followed by an LDR.

The ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD workaround is used to handle Cortex-A520 erratum 2966298 and Cortex-A510 erratum 3117295, which are described in:

• https://developer.arm.com/documentation/SDEN2444153/0600/?lang=en
• https://developer.arm.com/documentation/SDEN1873361/1600/?lang=en

In both cases the workaround is described as:

| If pagetable isolation is disabled, the context switch logic in the | kernel can be updated to execute the following sequence on affected | cores before exiting to EL0, and after all explicit memory accesses: | | 1. A non-shareable TLBI to any context and/or address, including | unused contexts or addresses, such as a TLBI VALE1 Xzr. | | 2. A DSB NSH to guarantee completion of the TLBI.

The important part being that the TLBI+DSB must be placed "after all explicit memory accesses".

Unfortunately, as-implemented, the TLBI+DSB is immediately followed by an LDR, as we have:

| alternative_if ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD | tlbi vale1, xzr | dsb nsh | alternative_else_nop_endif | alternative_if_not ARM64_UNMAP_KERNEL_AT_EL0 | ldr lr, [sp, #S_LR] | add sp, sp, #PT_REGS_SIZE // restore sp | eret | alternative_else_nop_endif | | [ ... KPTI exception return path ... ]

This patch fixes this by reworking the logic to place the TLBI+DSB immediately before the ERET, after all explicit memory accesses.

The ERET is currently in a separate alternative block, and alternatives cannot be nested. To account for this, the alternative block for ARM64_UNMAP_KERNEL_AT_EL0 is replaced with a single alternative branch to skip the KPTI logic, with the new shape of the logic being:

| alternative_insn "b .L_skip_tramp_exit_@", nop, ARM64_UNMAP_KERNEL_AT_EL0 | [ ... KPTI exception return path ... ] | .L_skip_tramp_exit_@: | | ldr lr, [sp, #S_LR] | add sp, sp, #PT_REGS_SIZE // restore sp | | alternative_if ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD | tlbi vale1, xzr | dsb nsh | alternative_else_nop_endif | eret

The new structure means that the workaround is only applied when KPTI is not in use; this is fine as noted in the documented implications of the erratum:

| Pagetable isolation between EL0 and higher level ELs prevents the | issue from occurring.

... and as per the workaround description quoted above, the workaround is only necessary "If pagetable isolation is disabled".

Affected

• Linux >= 6e3ae2927b432a3b7c8374f14dbc1bd9ebe4372c < d9a9a4d8ebc2fa534c0eb3eef3dae45f5e61b3e3
• Linux >= 471470bc7052d28ce125901877dd10e4c048e513 < 58eb5c07f41704464b9acc09ab0707b6769db6c0
• Linux >= 471470bc7052d28ce125901877dd10e4c048e513 < baa0aaac16432019651e0d60c41cd34a0c3c3477
• Linux >= 471470bc7052d28ce125901877dd10e4c048e513 < 832dd634bd1b4e3bbe9f10b9c9ba5db6f6f2b97f
• Linux 32b0a4ffcaea44a00a61e40c0d1bcc50362aee25
• Linux >= 6.1.57 < 6.1.191
• Linux >= 6.5.7 < 6.6
• Linux 6.6

Remediation

Refer to the linked advisories for vendor-supplied fixes and affected version ranges.