From 034dd340b08be1f2f0477ad16131d609f9dbd53c Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Sun, 30 Aug 2026 09:22:00 -0700 Subject: Merge tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace Pull tracing fixes from Steven Rostedt: - Fix error output of boot instance creation failure Currently if a boot instance creation fails, instead of printing out the name of the instance that failed, it prints "(null)". That is because it prints "cur_str" that had already been processed by strsep(). Print the saved name instead. While at it, print the error code of the failure. - Fix use-after-free for same named historgrams Histograms can be named so that they can be used in multiple events. But if the named histogram has a variable attached, the second event that uses the named histogram which duplicates it and needs to free the original after duplication leaves the old variable in place and still visible. If another histogram uses than variable, it will use the stale one which will try to reference the freed duplicate histogram and crash the kernel. Free the duplicate variables along with the duplicated histogram data. - Check return value of kthread_run() in event self test The events self tests uses a kthread for testing but does not check if it succeeded in creating a kthread. If the kthread creation were to fail, the code will still try to call kthread_stop() on the error returned. - Fix race between reading trace_pipe and updating subbuffer size If a user is reading the trace_pipe file at the same time they update the ring buffer sub-buffer size, can cause the trace_pipe read to read stale data. Add trace_access_lock() around updating the ring buffer sub-buffer size. - Fix eventfs_inode on failure path in creation of the events directory In the creation of the "events" directory, if after allocating the eventfs_inode a failure is detected, it calls cleanup_ei() which calls free_ei(). The free_ei() will test if eventfs_inode being freed has no children. It is a bug if it does. But on the failure case of the creation of the "events" directory, the children lists have not yet been initialized and the free will trigger a warning because list_empty() on an uninitialized list returns false. Move the initialization into init_ei() where it makes more sense and makes sure that a created eventfs_inode has its lists initialized upon creation. - Check return value of kthread_run() in ftrace direct sample code The sample code that shows how to use the ftrace direct calls does not test the return of kthread_run() to see if it succeeds. Return a failure if the kthread_run() doesn't succeed. - Clear user events state on fork in case of alloc failure On fork, the child gets a pointer to the parent's user events state. It makes a copy of it then updates the child's pointer to it. But if the allocation fails, the duplication function leaves the child with a pointer to its parent's descriptor. When the child cleans up its data, it will free the parent's descriptor while the parent is still using it. In the duplication function, set the child's user_event_mm to NULL before testing if the allocation succeeded, and when it exits it will not free the parent's descriptor. - Fix retry exhaustion in simple ring buffer reader swap simple_ring_buffer_swap_reader_page() starts with retry set to 8 and post-decrements it only after a failed link replacement. On the final attempt, a successful replacement leaves retry at zero, while a failed replacement leaves it at -1. But the check for success expects the retry value to be non-zero and exits with an error on zero. This is the opposite result. Fix it. - Fail nicely when the remote swap_reader_page() returns an error Currently, if the swap_reader_page() of a remote buffer fails, it triggers a WARN_ON_ONCE() and continues normally. Instead, have it exit with an error and a pr_warn() print instead of a full WARNING. * tag 'trace-v7.3-2' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: ring-buffer: Stop remote reader update when page swap fails tracing: Fix retry exhaustion in simple ring buffer reader swap tracing/user_events: Clear copied tracing state before fork duplication samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-multi-modify samples/ftrace: Fix kthread_stop() on ERR_PTR in ftrace-direct-modify eventfs: Initialize ei->children and ei->list in init_ei() tracing: Fix use-after-free in trace_pipe read on sub-buffer order change tracing: Fix crash passing ERR_PTR to kthread_stop() tracing: Fix use-after-free with same-name named triggers tracing: Fix logged instance name on creation failure --- arch/hexagon/lib/Makefile | 6 + arch/hexagon/lib/checksum.c | 178 +++++++++++ arch/hexagon/lib/divsi3.S | 67 ++++ arch/hexagon/lib/memcpy.S | 529 +++++++++++++++++++++++++++++++ arch/hexagon/lib/memcpy_likely_aligned.S | 56 ++++ arch/hexagon/lib/memset.S | 302 ++++++++++++++++++ arch/hexagon/lib/modsi3.S | 46 +++ arch/hexagon/lib/udivsi3.S | 38 +++ arch/hexagon/lib/umodsi3.S | 36 +++ 9 files changed, 1258 insertions(+) create mode 100644 arch/hexagon/lib/Makefile create mode 100644 arch/hexagon/lib/checksum.c create mode 100644 arch/hexagon/lib/divsi3.S create mode 100644 arch/hexagon/lib/memcpy.S create mode 100644 arch/hexagon/lib/memcpy_likely_aligned.S create mode 100644 arch/hexagon/lib/memset.S create mode 100644 arch/hexagon/lib/modsi3.S create mode 100644 arch/hexagon/lib/udivsi3.S create mode 100644 arch/hexagon/lib/umodsi3.S (limited to 'arch/hexagon/lib') diff --git a/arch/hexagon/lib/Makefile b/arch/hexagon/lib/Makefile new file mode 100644 index 000000000..107894c09 --- /dev/null +++ b/arch/hexagon/lib/Makefile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0-only +# +# Makefile for hexagon-specific library files. +# +obj-y = checksum.o memcpy.o memset.o memcpy_likely_aligned.o \ + divsi3.o modsi3.o udivsi3.o umodsi3.o diff --git a/arch/hexagon/lib/checksum.c b/arch/hexagon/lib/checksum.c new file mode 100644 index 000000000..ba50822a0 --- /dev/null +++ b/arch/hexagon/lib/checksum.c @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Checksum functions for Hexagon + * + * Copyright (c) 2010-2011, The Linux Foundation. All rights reserved. + */ + +/* This was derived from arch/alpha/lib/checksum.c */ + + +#include +#include + +#include +#include +#include +#include + + +/* Vector value operations */ +#define SIGN(x, y) ((0x8000ULL*x)<> 32); + /* 33 to 32 */ + result = (result & 0xffffffffUL) + (result >> 32); + return (__force __wsum)result; +} +EXPORT_SYMBOL(csum_tcpudp_nofold); + +/* + * Do a 64-bit checksum on an arbitrary memory area.. + * + * This isn't a great routine, but it's not _horrible_ either. The + * inner loop could be unrolled a bit further, and there are better + * ways to do the carry, but this is reasonable. + */ + +/* optimized HEXAGON intrinsic version, with over read fixed */ +unsigned int do_csum(const void *voidptr, int len) +{ + u64 sum0, sum1, x0, x1, *ptr8_o, *ptr8_e, *ptr8; + int i, start, mid, end, mask; + const char *ptr = voidptr; + unsigned short *ptr2; + unsigned int *ptr4; + + if (len <= 0) + return 0; + + start = 0xF & (16-(((int) ptr) & 0xF)) ; + mask = 0x7fffffffUL >> HEXAGON_R_cl0_R(len); + start = start & mask ; + + mid = len - start; + end = mid & 0xF; + mid = mid>>4; + sum0 = mid << 18; + sum1 = 0; + + if (start & 1) + sum0 += (u64) (ptr[0] << 8); + ptr2 = (unsigned short *) &ptr[start & 1]; + if (start & 2) + sum1 += (u64) ptr2[0]; + ptr4 = (unsigned int *) &ptr[start & 3]; + if (start & 4) { + sum0 = HEXAGON_P_vrmpyhacc_PP(sum0, + VR_NEGATE(0, 0, 1, 1)^((u64)ptr4[0]), + VR_SELECT(0, 0, 1, 1)); + sum0 += VR_SELECT(0, 0, 1, 0); + } + ptr8 = (u64 *) &ptr[start & 7]; + if (start & 8) { + sum1 = HEXAGON_P_vrmpyhacc_PP(sum1, + VR_NEGATE(1, 1, 1, 1)^(ptr8[0]), + VR_SELECT(1, 1, 1, 1)); + sum1 += VR_CARRY(0, 0, 1, 0); + } + ptr8_o = (u64 *) (ptr + start); + ptr8_e = (u64 *) (ptr + start + 8); + + if (mid) { + x0 = *ptr8_e; ptr8_e += 2; + x1 = *ptr8_o; ptr8_o += 2; + if (mid > 1) + for (i = 0; i < mid-1; i++) { + sum0 = HEXAGON_P_vrmpyhacc_PP(sum0, + x0^VR_NEGATE(1, 1, 1, 1), + VR_SELECT(1, 1, 1, 1)); + sum1 = HEXAGON_P_vrmpyhacc_PP(sum1, + x1^VR_NEGATE(1, 1, 1, 1), + VR_SELECT(1, 1, 1, 1)); + x0 = *ptr8_e; ptr8_e += 2; + x1 = *ptr8_o; ptr8_o += 2; + } + sum0 = HEXAGON_P_vrmpyhacc_PP(sum0, x0^VR_NEGATE(1, 1, 1, 1), + VR_SELECT(1, 1, 1, 1)); + sum1 = HEXAGON_P_vrmpyhacc_PP(sum1, x1^VR_NEGATE(1, 1, 1, 1), + VR_SELECT(1, 1, 1, 1)); + } + + ptr4 = (unsigned int *) &ptr[start + (mid * 16) + (end & 8)]; + if (end & 4) { + sum1 = HEXAGON_P_vrmpyhacc_PP(sum1, + VR_NEGATE(0, 0, 1, 1)^((u64)ptr4[0]), + VR_SELECT(0, 0, 1, 1)); + sum1 += VR_SELECT(0, 0, 1, 0); + } + ptr2 = (unsigned short *) &ptr[start + (mid * 16) + (end & 12)]; + if (end & 2) + sum0 += (u64) ptr2[0]; + + if (end & 1) + sum1 += (u64) ptr[start + (mid * 16) + (end & 14)]; + + ptr8 = (u64 *) &ptr[start + (mid * 16)]; + if (end & 8) { + sum0 = HEXAGON_P_vrmpyhacc_PP(sum0, + VR_NEGATE(1, 1, 1, 1)^(ptr8[0]), + VR_SELECT(1, 1, 1, 1)); + sum0 += VR_CARRY(0, 0, 1, 0); + } + sum0 = HEXAGON_P_vrmpyh_PP((sum0+sum1)^VR_NEGATE(0, 0, 0, 1), + VR_SELECT(0, 0, 1, 1)); + sum0 += VR_NEGATE(0, 0, 0, 1); + sum0 = HEXAGON_P_vrmpyh_PP(sum0, VR_SELECT(0, 0, 1, 1)); + + if (start & 1) + sum0 = (sum0 << 8) | (0xFF & (sum0 >> 8)); + + return 0xFFFF & sum0; +} diff --git a/arch/hexagon/lib/divsi3.S b/arch/hexagon/lib/divsi3.S new file mode 100644 index 000000000..783e09424 --- /dev/null +++ b/arch/hexagon/lib/divsi3.S @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2021, The Linux Foundation. All rights reserved. + */ + +#include + +SYM_FUNC_START(__hexagon_divsi3) + { + p0 = cmp.gt(r0,#-1) + p1 = cmp.gt(r1,#-1) + r3:2 = vabsw(r1:0) + } + { + p3 = xor(p0,p1) + r4 = sub(r2,r3) + r6 = cl0(r2) + p0 = cmp.gtu(r3,r2) + } + { + r0 = mux(p3,#-1,#1) + r7 = cl0(r3) + p1 = cmp.gtu(r3,r4) + } + { + r0 = mux(p0,#0,r0) + p0 = or(p0,p1) + if (p0.new) jumpr:nt r31 + r6 = sub(r7,r6) + } + { + r7 = r6 + r5:4 = combine(#1,r3) + r6 = add(#1,lsr(r6,#1)) + p0 = cmp.gtu(r6,#4) + } + { + r5:4 = vaslw(r5:4,r7) + if (!p0) r6 = #3 + } + { + loop0(1f,r6) + r7:6 = vlsrw(r5:4,#1) + r1:0 = #0 + } + .falign +1: + { + r5:4 = vlsrw(r5:4,#2) + if (!p0.new) r0 = add(r0,r5) + if (!p0.new) r2 = sub(r2,r4) + p0 = cmp.gtu(r4,r2) + } + { + r7:6 = vlsrw(r7:6,#2) + if (!p0.new) r0 = add(r0,r7) + if (!p0.new) r2 = sub(r2,r6) + p0 = cmp.gtu(r6,r2) + }:endloop0 + { + if (!p0) r0 = add(r0,r7) + } + { + if (p3) r0 = sub(r1,r0) + jumpr r31 + } +SYM_FUNC_END(__hexagon_divsi3) diff --git a/arch/hexagon/lib/memcpy.S b/arch/hexagon/lib/memcpy.S new file mode 100644 index 000000000..f8b3c02a8 --- /dev/null +++ b/arch/hexagon/lib/memcpy.S @@ -0,0 +1,529 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2010-2011, The Linux Foundation. All rights reserved. + */ + +/* + * Description + * + * library function for memcpy where length bytes are copied from + * ptr_in to ptr_out. ptr_out is returned unchanged. + * Allows any combination of alignment on input and output pointers + * and length from 0 to 2^32-1 + * + * Restrictions + * The arrays should not overlap, the program will produce undefined output + * if they do. + * For blocks less than 16 bytes a byte by byte copy is performed. For + * 8byte alignments, and length multiples, a dword copy is performed up to + * 96bytes + * History + * + * DJH 5/15/09 Initial version 1.0 + * DJH 6/ 1/09 Version 1.1 modified ABI to inlcude R16-R19 + * DJH 7/12/09 Version 1.2 optimized codesize down to 760 was 840 + * DJH 10/14/09 Version 1.3 added special loop for aligned case, was + * overreading bloated codesize back up to 892 + * DJH 4/20/10 Version 1.4 fixed Ldword_loop_epilog loop to prevent loads + * occurring if only 1 left outstanding, fixes bug + * # 3888, corrected for all alignments. Peeled off + * 1 32byte chunk from kernel loop and extended 8byte + * loop at end to solve all combinations and prevent + * over read. Fixed Ldword_loop_prolog to prevent + * overread for blocks less than 48bytes. Reduced + * codesize to 752 bytes + * DJH 4/21/10 version 1.5 1.4 fix broke code for input block ends not + * aligned to dword boundaries,underwriting by 1 + * byte, added detection for this and fixed. A + * little bloat. + * DJH 4/23/10 version 1.6 corrected stack error, R20 was not being restored + * always, fixed the error of R20 being modified + * before it was being saved + * Natural c model + * =============== + * void * memcpy(char * ptr_out, char * ptr_in, int length) { + * int i; + * if(length) for(i=0; i < length; i++) { ptr_out[i] = ptr_in[i]; } + * return(ptr_out); + * } + * + * Optimized memcpy function + * ========================= + * void * memcpy(char * ptr_out, char * ptr_in, int len) { + * int i, prolog, kernel, epilog, mask; + * u8 offset; + * s64 data0, dataF8, data70; + * + * s64 * ptr8_in; + * s64 * ptr8_out; + * s32 * ptr4; + * s16 * ptr2; + * + * offset = ((int) ptr_in) & 7; + * ptr8_in = (s64 *) &ptr_in[-offset]; //read in the aligned pointers + * + * data70 = *ptr8_in++; + * dataF8 = *ptr8_in++; + * + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * + * prolog = 32 - ((int) ptr_out); + * mask = 0x7fffffff >> HEXAGON_R_cl0_R(len); + * prolog = prolog & mask; + * kernel = len - prolog; + * epilog = kernel & 0x1F; + * kernel = kernel>>5; + * + * if (prolog & 1) { ptr_out[0] = (u8) data0; data0 >>= 8; ptr_out += 1;} + * ptr2 = (s16 *) &ptr_out[0]; + * if (prolog & 2) { ptr2[0] = (u16) data0; data0 >>= 16; ptr_out += 2;} + * ptr4 = (s32 *) &ptr_out[0]; + * if (prolog & 4) { ptr4[0] = (u32) data0; data0 >>= 32; ptr_out += 4;} + * + * offset = offset + (prolog & 7); + * if (offset >= 8) { + * data70 = dataF8; + * dataF8 = *ptr8_in++; + * } + * offset = offset & 0x7; + * + * prolog = prolog >> 3; + * if (prolog) for (i=0; i < prolog; i++) { + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * data70 = dataF8; + * dataF8 = *ptr8_in++; + * } + * if(kernel) { kernel -= 1; epilog += 32; } + * if(kernel) for(i=0; i < kernel; i++) { + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * data70 = *ptr8_in++; + * + * data0 = HEXAGON_P_valignb_PPp(data70, dataF8, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * dataF8 = *ptr8_in++; + * + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * data70 = *ptr8_in++; + * + * data0 = HEXAGON_P_valignb_PPp(data70, dataF8, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * dataF8 = *ptr8_in++; + * } + * epilogdws = epilog >> 3; + * if (epilogdws) for (i=0; i < epilogdws; i++) { + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * ptr8_out = (s64 *) &ptr_out[0]; *ptr8_out = data0; ptr_out += 8; + * data70 = dataF8; + * dataF8 = *ptr8_in++; + * } + * data0 = HEXAGON_P_valignb_PPp(dataF8, data70, offset); + * + * ptr4 = (s32 *) &ptr_out[0]; + * if (epilog & 4) { ptr4[0] = (u32) data0; data0 >>= 32; ptr_out += 4;} + * ptr2 = (s16 *) &ptr_out[0]; + * if (epilog & 2) { ptr2[0] = (u16) data0; data0 >>= 16; ptr_out += 2;} + * if (epilog & 1) { *ptr_out++ = (u8) data0; } + * + * return(ptr_out - length); + * } + * + * Codesize : 784 bytes + */ + + +#define ptr_out R0 /* destination pounter */ +#define ptr_in R1 /* source pointer */ +#define len R2 /* length of copy in bytes */ + +#define data70 R13:12 /* lo 8 bytes of non-aligned transfer */ +#define dataF8 R11:10 /* hi 8 bytes of non-aligned transfer */ +#define ldata0 R7:6 /* even 8 bytes chunks */ +#define ldata1 R25:24 /* odd 8 bytes chunks */ +#define data1 R7 /* lower 8 bytes of ldata1 */ +#define data0 R6 /* lower 8 bytes of ldata0 */ + +#define ifbyte p0 /* if transfer has bytes in epilog/prolog */ +#define ifhword p0 /* if transfer has shorts in epilog/prolog */ +#define ifword p0 /* if transfer has words in epilog/prolog */ +#define noprolog p0 /* no prolog, xfer starts at 32byte */ +#define nokernel p1 /* no 32byte multiple block in the transfer */ +#define noepilog p0 /* no epilog, xfer ends on 32byte boundary */ +#define align p2 /* alignment of input rel to 8byte boundary */ +#define kernel1 p0 /* kernel count == 1 */ + +#define dalign R25 /* rel alignment of input to output data */ +#define star3 R16 /* number bytes in prolog - dwords */ +#define rest R8 /* length - prolog bytes */ +#define back R7 /* nr bytes > dword boundary in src block */ +#define epilog R3 /* bytes in epilog */ +#define inc R15:14 /* inc kernel by -1 and defetch ptr by 32 */ +#define kernel R4 /* number of 32byte chunks in kernel */ +#define ptr_in_p_128 R5 /* pointer for prefetch of input data */ +#define mask R8 /* mask used to determine prolog size */ +#define shift R8 /* used to work a shifter to extract bytes */ +#define shift2 R5 /* in epilog to workshifter to extract bytes */ +#define prolog R15 /* bytes in prolog */ +#define epilogdws R15 /* number dwords in epilog */ +#define shiftb R14 /* used to extract bytes */ +#define offset R9 /* same as align in reg */ +#define ptr_out_p_32 R17 /* pointer to output dczero */ +#define align888 R14 /* if simple dword loop can be used */ +#define len8 R9 /* number of dwords in length */ +#define over R20 /* nr of bytes > last inp buf dword boundary */ + +#define ptr_in_p_128kernel R5:4 /* packed fetch pointer & kernel cnt */ + + .section .text + .p2align 4 + .global memcpy + .type memcpy, @function +memcpy: +{ + p2 = cmp.eq(len, #0); /* =0 */ + align888 = or(ptr_in, ptr_out); /* %8 < 97 */ + p0 = cmp.gtu(len, #23); /* %1, <24 */ + p1 = cmp.eq(ptr_in, ptr_out); /* attempt to overwrite self */ +} +{ + p1 = or(p2, p1); + p3 = cmp.gtu(len, #95); /* %8 < 97 */ + align888 = or(align888, len); /* %8 < 97 */ + len8 = lsr(len, #3); /* %8 < 97 */ +} +{ + dcfetch(ptr_in); /* zero/ptrin=ptrout causes fetch */ + p2 = bitsclr(align888, #7); /* %8 < 97 */ + if(p1) jumpr r31; /* =0 */ +} +{ + p2 = and(p2,!p3); /* %8 < 97 */ + if (p2.new) len = add(len, #-8); /* %8 < 97 */ + if (p2.new) jump:NT .Ldwordaligned; /* %8 < 97 */ +} +{ + if(!p0) jump .Lbytes23orless; /* %1, <24 */ + mask.l = #LO(0x7fffffff); + /* all bytes before line multiples of data */ + prolog = sub(#0, ptr_out); +} +{ + /* save r31 on stack, decrement sp by 16 */ + allocframe(#24); + mask.h = #HI(0x7fffffff); + ptr_in_p_128 = add(ptr_in, #32); + back = cl0(len); +} +{ + memd(sp+#0) = R17:16; /* save r16,r17 on stack6 */ + r31.l = #LO(.Lmemcpy_return); /* set up final return pointer */ + prolog &= lsr(mask, back); + offset = and(ptr_in, #7); +} +{ + memd(sp+#8) = R25:24; /* save r25,r24 on stack */ + dalign = sub(ptr_out, ptr_in); + r31.h = #HI(.Lmemcpy_return); /* set up final return pointer */ +} +{ + /* see if there if input buffer end if aligned */ + over = add(len, ptr_in); + back = add(len, offset); + memd(sp+#16) = R21:20; /* save r20,r21 on stack */ +} +{ + noprolog = bitsclr(prolog, #7); + prolog = and(prolog, #31); + dcfetch(ptr_in_p_128); + ptr_in_p_128 = add(ptr_in_p_128, #32); +} +{ + kernel = sub(len, prolog); + shift = asl(prolog, #3); + star3 = and(prolog, #7); + ptr_in = and(ptr_in, #-8); +} +{ + prolog = lsr(prolog, #3); + epilog = and(kernel, #31); + ptr_out_p_32 = add(ptr_out, prolog); + over = and(over, #7); +} +{ + p3 = cmp.gtu(back, #8); + kernel = lsr(kernel, #5); + dcfetch(ptr_in_p_128); + ptr_in_p_128 = add(ptr_in_p_128, #32); +} +{ + p1 = cmp.eq(prolog, #0); + if(!p1.new) prolog = add(prolog, #1); + dcfetch(ptr_in_p_128); /* reserve the line 64bytes on */ + ptr_in_p_128 = add(ptr_in_p_128, #32); +} +{ + nokernel = cmp.eq(kernel,#0); + dcfetch(ptr_in_p_128); /* reserve the line 64bytes on */ + ptr_in_p_128 = add(ptr_in_p_128, #32); + shiftb = and(shift, #8); +} +{ + dcfetch(ptr_in_p_128); /* reserve the line 64bytes on */ + ptr_in_p_128 = add(ptr_in_p_128, #32); + if(nokernel) jump .Lskip64; + p2 = cmp.eq(kernel, #1); /* skip ovr if kernel == 0 */ +} +{ + dczeroa(ptr_out_p_32); + /* don't advance pointer */ + if(!p2) ptr_out_p_32 = add(ptr_out_p_32, #32); +} +{ + dalign = and(dalign, #31); + dczeroa(ptr_out_p_32); +} +.Lskip64: +{ + data70 = memd(ptr_in++#16); + if(p3) dataF8 = memd(ptr_in+#8); + if(noprolog) jump .Lnoprolog32; + align = offset; +} +/* upto initial 7 bytes */ +{ + ldata0 = valignb(dataF8, data70, align); + ifbyte = tstbit(shift,#3); + offset = add(offset, star3); +} +{ + if(ifbyte) memb(ptr_out++#1) = data0; + ldata0 = lsr(ldata0, shiftb); + shiftb = and(shift, #16); + ifhword = tstbit(shift,#4); +} +{ + if(ifhword) memh(ptr_out++#2) = data0; + ldata0 = lsr(ldata0, shiftb); + ifword = tstbit(shift,#5); + p2 = cmp.gtu(offset, #7); +} +{ + if(ifword) memw(ptr_out++#4) = data0; + if(p2) data70 = dataF8; + if(p2) dataF8 = memd(ptr_in++#8); /* another 8 bytes */ + align = offset; +} +.Lnoprolog32: +{ + p3 = sp1loop0(.Ldword_loop_prolog, prolog) + rest = sub(len, star3); /* whats left after the loop */ + p0 = cmp.gt(over, #0); +} + if(p0) rest = add(rest, #16); +.Ldword_loop_prolog: +{ + if(p3) memd(ptr_out++#8) = ldata0; + ldata0 = valignb(dataF8, data70, align); + p0 = cmp.gt(rest, #16); +} +{ + data70 = dataF8; + if(p0) dataF8 = memd(ptr_in++#8); + rest = add(rest, #-8); +}:endloop0 +.Lkernel: +{ + /* kernel is at least 32bytes */ + p3 = cmp.gtu(kernel, #0); + /* last itn. remove edge effects */ + if(p3.new) kernel = add(kernel, #-1); + /* dealt with in last dword loop */ + if(p3.new) epilog = add(epilog, #32); +} +{ + nokernel = cmp.eq(kernel, #0); /* after adjustment, recheck */ + if(nokernel.new) jump:NT .Lepilog; /* likely not taken */ + inc = combine(#32, #-1); + p3 = cmp.gtu(dalign, #24); +} +{ + if(p3) jump .Lodd_alignment; +} +{ + loop0(.Loword_loop_25to31, kernel); + kernel1 = cmp.gtu(kernel, #1); + rest = kernel; +} + .falign +.Loword_loop_25to31: +{ + dcfetch(ptr_in_p_128); /* prefetch 4 lines ahead */ + if(kernel1) ptr_out_p_32 = add(ptr_out_p_32, #32); +} +{ + dczeroa(ptr_out_p_32); /* reserve the next 32bytes in cache */ + p3 = cmp.eq(kernel, rest); +} +{ + /* kernel -= 1 */ + ptr_in_p_128kernel = vaddw(ptr_in_p_128kernel, inc); + /* kill write on first iteration */ + if(!p3) memd(ptr_out++#8) = ldata1; + ldata1 = valignb(dataF8, data70, align); + data70 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(data70, dataF8, align); + dataF8 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata1; + ldata1 = valignb(dataF8, data70, align); + data70 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(data70, dataF8, align); + dataF8 = memd(ptr_in++#8); + kernel1 = cmp.gtu(kernel, #1); +}:endloop0 +{ + memd(ptr_out++#8) = ldata1; + jump .Lepilog; +} +.Lodd_alignment: +{ + loop0(.Loword_loop_00to24, kernel); + kernel1 = cmp.gtu(kernel, #1); + rest = add(kernel, #-1); +} + .falign +.Loword_loop_00to24: +{ + dcfetch(ptr_in_p_128); /* prefetch 4 lines ahead */ + ptr_in_p_128kernel = vaddw(ptr_in_p_128kernel, inc); + if(kernel1) ptr_out_p_32 = add(ptr_out_p_32, #32); +} +{ + dczeroa(ptr_out_p_32); /* reserve the next 32bytes in cache */ +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(dataF8, data70, align); + data70 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(data70, dataF8, align); + dataF8 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(dataF8, data70, align); + data70 = memd(ptr_in++#8); +} +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(data70, dataF8, align); + dataF8 = memd(ptr_in++#8); + kernel1 = cmp.gtu(kernel, #1); +}:endloop0 +.Lepilog: +{ + noepilog = cmp.eq(epilog,#0); + epilogdws = lsr(epilog, #3); + kernel = and(epilog, #7); +} +{ + if(noepilog) jumpr r31; + if(noepilog) ptr_out = sub(ptr_out, len); + p3 = cmp.eq(epilogdws, #0); + shift2 = asl(epilog, #3); +} +{ + shiftb = and(shift2, #32); + ifword = tstbit(epilog,#2); + if(p3) jump .Lepilog60; + if(!p3) epilog = add(epilog, #-16); +} +{ + loop0(.Ldword_loop_epilog, epilogdws); + /* stop criteria is lsbs unless = 0 then its 8 */ + p3 = cmp.eq(kernel, #0); + if(p3.new) kernel= #8; + p1 = cmp.gt(over, #0); +} + /* if not aligned to end of buffer execute 1 more iteration */ + if(p1) kernel= #0; +.Ldword_loop_epilog: +{ + memd(ptr_out++#8) = ldata0; + ldata0 = valignb(dataF8, data70, align); + p3 = cmp.gt(epilog, kernel); +} +{ + data70 = dataF8; + if(p3) dataF8 = memd(ptr_in++#8); + epilog = add(epilog, #-8); +}:endloop0 +/* copy last 7 bytes */ +.Lepilog60: +{ + if(ifword) memw(ptr_out++#4) = data0; + ldata0 = lsr(ldata0, shiftb); + ifhword = tstbit(epilog,#1); + shiftb = and(shift2, #16); +} +{ + if(ifhword) memh(ptr_out++#2) = data0; + ldata0 = lsr(ldata0, shiftb); + ifbyte = tstbit(epilog,#0); + if(ifbyte.new) len = add(len, #-1); +} +{ + if(ifbyte) memb(ptr_out) = data0; + ptr_out = sub(ptr_out, len); /* return dest pointer */ + jumpr r31; +} +/* do byte copy for small n */ +.Lbytes23orless: +{ + p3 = sp1loop0(.Lbyte_copy, len); + len = add(len, #-1); +} +.Lbyte_copy: +{ + data0 = memb(ptr_in++#1); + if(p3) memb(ptr_out++#1) = data0; +}:endloop0 +{ + memb(ptr_out) = data0; + ptr_out = sub(ptr_out, len); + jumpr r31; +} +/* do dword copies for aligned in, out and length */ +.Ldwordaligned: +{ + p3 = sp1loop0(.Ldword_copy, len8); +} +.Ldword_copy: +{ + if(p3) memd(ptr_out++#8) = ldata0; + ldata0 = memd(ptr_in++#8); +}:endloop0 +{ + memd(ptr_out) = ldata0; + ptr_out = sub(ptr_out, len); + jumpr r31; /* return to function caller */ +} +.Lmemcpy_return: + r21:20 = memd(sp+#16); /* restore r20+r21 */ +{ + r25:24 = memd(sp+#8); /* restore r24+r25 */ + r17:16 = memd(sp+#0); /* restore r16+r17 */ +} + deallocframe; /* restore r31 and incrment stack by 16 */ + jumpr r31 diff --git a/arch/hexagon/lib/memcpy_likely_aligned.S b/arch/hexagon/lib/memcpy_likely_aligned.S new file mode 100644 index 000000000..6a541fb90 --- /dev/null +++ b/arch/hexagon/lib/memcpy_likely_aligned.S @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2021, The Linux Foundation. All rights reserved. + */ + +#include + +SYM_FUNC_START(__hexagon_memcpy_likely_aligned_min32bytes_mult8bytes) + { + p0 = bitsclr(r1,#7) + p0 = bitsclr(r0,#7) + if (p0.new) r5:4 = memd(r1) + if (p0.new) r7:6 = memd(r1+#8) + } + { + if (!p0) jump:nt .Lmemcpy_call + if (p0) r9:8 = memd(r1+#16) + if (p0) r11:10 = memd(r1+#24) + p0 = cmp.gtu(r2,#64) + } + { + if (p0) jump:nt .Lmemcpy_call + if (!p0) memd(r0) = r5:4 + if (!p0) memd(r0+#8) = r7:6 + p0 = cmp.gtu(r2,#32) + } + { + p1 = cmp.gtu(r2,#40) + p2 = cmp.gtu(r2,#48) + if (p0) r13:12 = memd(r1+#32) + if (p1.new) r15:14 = memd(r1+#40) + } + { + memd(r0+#16) = r9:8 + memd(r0+#24) = r11:10 + } + { + if (p0) memd(r0+#32) = r13:12 + if (p1) memd(r0+#40) = r15:14 + if (!p2) jumpr:t r31 + } + { + p0 = cmp.gtu(r2,#56) + r5:4 = memd(r1+#48) + if (p0.new) r7:6 = memd(r1+#56) + } + { + memd(r0+#48) = r5:4 + if (p0) memd(r0+#56) = r7:6 + jumpr r31 + } + +.Lmemcpy_call: + jump memcpy + +SYM_FUNC_END(__hexagon_memcpy_likely_aligned_min32bytes_mult8bytes) diff --git a/arch/hexagon/lib/memset.S b/arch/hexagon/lib/memset.S new file mode 100644 index 000000000..e67304e3f --- /dev/null +++ b/arch/hexagon/lib/memset.S @@ -0,0 +1,302 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2011, The Linux Foundation. All rights reserved. + */ + + +/* HEXAGON assembly optimized memset */ +/* Replaces the standard library function memset */ + + + .macro HEXAGON_OPT_FUNC_BEGIN name + .text + .p2align 4 + .globl \name + .type \name, @function +\name: + .endm + + .macro HEXAGON_OPT_FUNC_FINISH name + .size \name, . - \name + .endm + +/* FUNCTION: memset (v2 version) */ +#if __HEXAGON_ARCH__ < 3 +HEXAGON_OPT_FUNC_BEGIN memset + { + r6 = #8 + r7 = extractu(r0, #3 , #0) + p0 = cmp.eq(r2, #0) + p1 = cmp.gtu(r2, #7) + } + { + r4 = vsplatb(r1) + r8 = r0 /* leave r0 intact for return val */ + r9 = sub(r6, r7) /* bytes until double alignment */ + if p0 jumpr r31 /* count == 0, so return */ + } + { + r3 = #0 + r7 = #0 + p0 = tstbit(r9, #0) + if p1 jump 2f /* skip byte loop */ + } + +/* less than 8 bytes to set, so just set a byte at a time and return */ + + loop0(1f, r2) /* byte loop */ + .falign +1: /* byte loop */ + { + memb(r8++#1) = r4 + }:endloop0 + jumpr r31 + .falign +2: /* skip byte loop */ + { + r6 = #1 + p0 = tstbit(r9, #1) + p1 = cmp.eq(r2, #1) + if !p0 jump 3f /* skip initial byte store */ + } + { + memb(r8++#1) = r4 + r3:2 = sub(r3:2, r7:6) + if p1 jumpr r31 + } + .falign +3: /* skip initial byte store */ + { + r6 = #2 + p0 = tstbit(r9, #2) + p1 = cmp.eq(r2, #2) + if !p0 jump 4f /* skip initial half store */ + } + { + memh(r8++#2) = r4 + r3:2 = sub(r3:2, r7:6) + if p1 jumpr r31 + } + .falign +4: /* skip initial half store */ + { + r6 = #4 + p0 = cmp.gtu(r2, #7) + p1 = cmp.eq(r2, #4) + if !p0 jump 5f /* skip initial word store */ + } + { + memw(r8++#4) = r4 + r3:2 = sub(r3:2, r7:6) + p0 = cmp.gtu(r2, #11) + if p1 jumpr r31 + } + .falign +5: /* skip initial word store */ + { + r10 = lsr(r2, #3) + p1 = cmp.eq(r3, #1) + if !p0 jump 7f /* skip double loop */ + } + { + r5 = r4 + r6 = #8 + loop0(6f, r10) /* double loop */ + } + +/* set bytes a double word at a time */ + + .falign +6: /* double loop */ + { + memd(r8++#8) = r5:4 + r3:2 = sub(r3:2, r7:6) + p1 = cmp.eq(r2, #8) + }:endloop0 + .falign +7: /* skip double loop */ + { + p0 = tstbit(r2, #2) + if p1 jumpr r31 + } + { + r6 = #4 + p0 = tstbit(r2, #1) + p1 = cmp.eq(r2, #4) + if !p0 jump 8f /* skip final word store */ + } + { + memw(r8++#4) = r4 + r3:2 = sub(r3:2, r7:6) + if p1 jumpr r31 + } + .falign +8: /* skip final word store */ + { + p1 = cmp.eq(r2, #2) + if !p0 jump 9f /* skip final half store */ + } + { + memh(r8++#2) = r4 + if p1 jumpr r31 + } + .falign +9: /* skip final half store */ + { + memb(r8++#1) = r4 + jumpr r31 + } +HEXAGON_OPT_FUNC_FINISH memset +#endif + + +/* FUNCTION: memset (v3 and higher version) */ +#if __HEXAGON_ARCH__ >= 3 +HEXAGON_OPT_FUNC_BEGIN memset + { + r7=vsplatb(r1) + r6 = r0 + if (r2==#0) jump:nt .L1 + } + { + r5:4=combine(r7,r7) + p0 = cmp.gtu(r2,#8) + if (p0.new) jump:nt .L3 + } + { + r3 = r0 + loop0(.L47,r2) + } + .falign +.L47: + { + memb(r3++#1) = r1 + }:endloop0 /* start=.L47 */ + jumpr r31 +.L3: + { + p0 = tstbit(r0,#0) + if (!p0.new) jump:nt .L8 + p1 = cmp.eq(r2, #1) + } + { + r6 = add(r0, #1) + r2 = add(r2,#-1) + memb(r0) = r1 + if (p1) jump .L1 + } +.L8: + { + p0 = tstbit(r6,#1) + if (!p0.new) jump:nt .L10 + } + { + r2 = add(r2,#-2) + memh(r6++#2) = r7 + p0 = cmp.eq(r2, #2) + if (p0.new) jump:nt .L1 + } +.L10: + { + p0 = tstbit(r6,#2) + if (!p0.new) jump:nt .L12 + } + { + r2 = add(r2,#-4) + memw(r6++#4) = r7 + p0 = cmp.eq(r2, #4) + if (p0.new) jump:nt .L1 + } +.L12: + { + p0 = cmp.gtu(r2,#127) + if (!p0.new) jump:nt .L14 + } + r3 = and(r6,#31) + if (r3==#0) jump:nt .L17 + { + memd(r6++#8) = r5:4 + r2 = add(r2,#-8) + } + r3 = and(r6,#31) + if (r3==#0) jump:nt .L17 + { + memd(r6++#8) = r5:4 + r2 = add(r2,#-8) + } + r3 = and(r6,#31) + if (r3==#0) jump:nt .L17 + { + memd(r6++#8) = r5:4 + r2 = add(r2,#-8) + } +.L17: + { + r3 = lsr(r2,#5) + if (r1!=#0) jump:nt .L18 + } + { + r8 = r3 + r3 = r6 + loop0(.L46,r3) + } + .falign +.L46: + { + dczeroa(r6) + r6 = add(r6,#32) + r2 = add(r2,#-32) + }:endloop0 /* start=.L46 */ +.L14: + { + p0 = cmp.gtu(r2,#7) + if (!p0.new) jump:nt .L28 + r8 = lsr(r2,#3) + } + loop0(.L44,r8) + .falign +.L44: + { + memd(r6++#8) = r5:4 + r2 = add(r2,#-8) + }:endloop0 /* start=.L44 */ +.L28: + { + p0 = tstbit(r2,#2) + if (!p0.new) jump:nt .L33 + } + { + r2 = add(r2,#-4) + memw(r6++#4) = r7 + } +.L33: + { + p0 = tstbit(r2,#1) + if (!p0.new) jump:nt .L35 + } + { + r2 = add(r2,#-2) + memh(r6++#2) = r7 + } +.L35: + p0 = cmp.eq(r2,#1) + if (p0) memb(r6) = r1 +.L1: + jumpr r31 +.L18: + loop0(.L45,r3) + .falign +.L45: + dczeroa(r6) + { + memd(r6++#8) = r5:4 + r2 = add(r2,#-32) + } + memd(r6++#8) = r5:4 + memd(r6++#8) = r5:4 + { + memd(r6++#8) = r5:4 + }:endloop0 /* start=.L45 */ + jump .L14 +HEXAGON_OPT_FUNC_FINISH memset +#endif diff --git a/arch/hexagon/lib/modsi3.S b/arch/hexagon/lib/modsi3.S new file mode 100644 index 000000000..9ea1c86ef --- /dev/null +++ b/arch/hexagon/lib/modsi3.S @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2021, The Linux Foundation. All rights reserved. + */ + +#include + +SYM_FUNC_START(__hexagon_modsi3) + { + p2 = cmp.ge(r0,#0) + r2 = abs(r0) + r1 = abs(r1) + } + { + r3 = cl0(r2) + r4 = cl0(r1) + p0 = cmp.gtu(r1,r2) + } + { + r3 = sub(r4,r3) + if (p0) jumpr r31 + } + { + p1 = cmp.eq(r3,#0) + loop0(1f,r3) + r0 = r2 + r2 = lsl(r1,r3) + } + .falign +1: + { + p0 = cmp.gtu(r2,r0) + if (!p0.new) r0 = sub(r0,r2) + r2 = lsr(r2,#1) + if (p1) r1 = #0 + }:endloop0 + { + p0 = cmp.gtu(r2,r0) + if (!p0.new) r0 = sub(r0,r1) + if (p2) jumpr r31 + } + { + r0 = neg(r0) + jumpr r31 + } +SYM_FUNC_END(__hexagon_modsi3) diff --git a/arch/hexagon/lib/udivsi3.S b/arch/hexagon/lib/udivsi3.S new file mode 100644 index 000000000..477f27b93 --- /dev/null +++ b/arch/hexagon/lib/udivsi3.S @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2021, The Linux Foundation. All rights reserved. + */ + +#include + +SYM_FUNC_START(__hexagon_udivsi3) + { + r2 = cl0(r0) + r3 = cl0(r1) + r5:4 = combine(#1,#0) + p0 = cmp.gtu(r1,r0) + } + { + r6 = sub(r3,r2) + r4 = r1 + r1:0 = combine(r0,r4) + if (p0) jumpr r31 + } + { + r3:2 = vlslw(r5:4,r6) + loop0(1f,r6) + } + .falign +1: + { + p0 = cmp.gtu(r2,r1) + if (!p0.new) r1 = sub(r1,r2) + if (!p0.new) r0 = add(r0,r3) + r3:2 = vlsrw(r3:2,#1) + }:endloop0 + { + p0 = cmp.gtu(r2,r1) + if (!p0.new) r0 = add(r0,r3) + jumpr r31 + } +SYM_FUNC_END(__hexagon_udivsi3) diff --git a/arch/hexagon/lib/umodsi3.S b/arch/hexagon/lib/umodsi3.S new file mode 100644 index 000000000..280bf06a5 --- /dev/null +++ b/arch/hexagon/lib/umodsi3.S @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (c) 2021, The Linux Foundation. All rights reserved. + */ + +#include + +SYM_FUNC_START(__hexagon_umodsi3) + { + r2 = cl0(r0) + r3 = cl0(r1) + p0 = cmp.gtu(r1,r0) + } + { + r2 = sub(r3,r2) + if (p0) jumpr r31 + } + { + loop0(1f,r2) + p1 = cmp.eq(r2,#0) + r2 = lsl(r1,r2) + } + .falign +1: + { + p0 = cmp.gtu(r2,r0) + if (!p0.new) r0 = sub(r0,r2) + r2 = lsr(r2,#1) + if (p1) r1 = #0 + }:endloop0 + { + p0 = cmp.gtu(r2,r0) + if (!p0.new) r0 = sub(r0,r1) + jumpr r31 + } +SYM_FUNC_END(__hexagon_umodsi3) -- cgit v1.3.1