kernel/arch/arm64/lib/csum.c, branch linux-6.2.y

arm64: csum: Disable KASAN for do_csum()

2020-04-15T20:36:41Z

do_csum() over-reads the source buffer and therefore abuses READ_ONCE_NOCHECK() to avoid tripping up KASAN. In preparation for READ_ONCE_NOCHECK() becoming a macro, and therefore losing its '__no_sanitize_address' annotation, just annotate do_csum() explicitly and fall back to normal loads. Cc: Mark Rutland Cc: Robin Murphy Signed-off-by: Will Deacon

arm64: csum: Optimise IPv6 header checksum

2020-03-09T18:08:25Z

Throwing our __uint128_t idioms at csum_ipv6_magic() makes it about 1.3x-2x faster across a range of microarchitecture/compiler combinations. Not much in absolute terms, but every little helps. Signed-off-by: Robin Murphy Signed-off-by: Catalin Marinas

arm64: csum: Fix pathological zero-length calls

2020-01-17T16:05:50Z

In validating the checksumming results of the new routine, I sadly neglected to test its not-checksumming results. Thus it slipped through that the one case where @buff is already dword-aligned and @len = 0 manages to defeat the tail-masking logic and behave as if @len = 8. For a zero length it doesn't make much sense to deference @buff anyway, so just add an early return (which has essentially zero impact on performance). Signed-off-by: Robin Murphy Signed-off-by: Will Deacon

arm64: Implement optimised checksum routine

2020-01-16T15:23:29Z

Apparently there exist certain workloads which rely heavily on software checksumming, for which the generic do_csum() implementation becomes a significant bottleneck. Therefore let's give arm64 its own optimised version - for ease of maintenance this foregoes assembly or intrisics, and is thus not actually arm64-specific, but does rely heavily on C idioms that translate well to the A64 ISA and the typical load/store capabilities of most ARMv8 CPU cores. The resulting increase in checksum throughput scales nicely with buffer size, tending towards 4x for a small in-order core (Cortex-A53), and up to 6x or more for an aggressive big core (Ampere eMAG). Reported-by: Lingyan Huang Tested-by: Lingyan Huang Signed-off-by: Robin Murphy Signed-off-by: Will Deacon