simd/jsimd_arm64_neon.S


Log

Author Commit Date CI Message
mayeut cb88e5da 2016-09-20T21:06:24 ARM64 NEON: Fix another ABI conformance issue Based on https://github.com/mayeut/libjpeg-turbo/commit/98a5a9dc899aa9265858a3cbe0a96289a31a1322 with wordsmithing by DRC. In the AArch64 ABI, as in many others, it's forbidden to read/store data below the stack pointer. Some SIMD functions were doing just that (stack pointer misuse) when trying to preserve callee-saved registers, and this resulted in those registers being restored with incorrect contents under certain circumstances. This patch fixes that behavior, and callee-saved registers are now stored above the stack pointer throughout the function call. The patch also removes register saving in places where it is unnecessary for this ABI, or it makes use of unused scratch regiters instead of callee-saved registers. Fixes #97. Closes #101. Refer also to https://bugzilla.redhat.com/show_bug.cgi?id=1368569
DRC 1120ff29 2016-07-13T12:15:02 Fix AArch64 ABI conformance issue in SIMD code In the AArch64 ABI, the high (unused) DWORD of a 32-bit argument's register is undefined, so it was incorrect to use 64-bit instructions to transfer a JDIMENSION argument in the 64-bit NEON SIMD functions. The code worked thus far only because the existing compiler optimizers weren't smart enough to do anything else with the register in question, so the upper 32 bits happened to be all zeroes. The latest builds of Clang/LLVM have a smarter optimizer, and under certain circumstances, it will attempt to load-combine adjacent 32-bit integers from one of the libjpeg structures into a single 64-bit integer and pass that 64-bit integer as a 32-bit argument to one of the SIMD functions (which is allowed by the ABI, since the upper 32 bits of the 32-bit argument's register are undefined.) This caused the libjpeg-turbo regression tests to crash. This patch tries to use the Wn registers whenever possible. Otherwise, it uses a zero-extend instruction to avoid using the upper 32 bits of the 64-bit registers, which are not guaranteed to be valid for 32-bit arguments. Based on https://github.com/sebpop/libjpeg-turbo/commit/1fbae13021eb98f6fffdfaf8678fcdb00b0b04d9 Closes #91. Refer also to android-ndk/ndk#110 and https://llvm.org/bugs/show_bug.cgi?id=28393
DRC 123f7258 2016-05-24T10:23:56 Format copyright headers more consistently The IJG convention is to format copyright notices as: Copyright (C) YYYY, Owner. We try to maintain this convention for any code that is part of the libjpeg API library (with the exception of preserving the copyright notices from Cendio's code verbatim, since those predate libjpeg-turbo.) Note that the phrase "All Rights Reserved" is no longer necessary, since all Buenos Aires Convention signatories signed onto the Berne Convention in 2000. However, our convention is to retain this phrase for any files that have a self-contained copyright header but to leave it off of any files that refer to another file for conditions of distribution and use. For instance, all of the non-SIMD files in the libjpeg API library refer to README.ijg, and the copyright message in that file contains "All Rights Reserved", so it is unnecessary to add it to the individual files. The TurboJPEG code retains my preferred formatting convention for copyright notices, which is based on that of VirtualGL (where the TurboJPEG API originated.)
DRC bd49803f 2016-02-19T08:53:33 Use consistent/modern code formatting for pointers The convention used by libjpeg: type * variable; is not very common anymore, because it looks too much like multiplication. Some (particularly C++ programmers) prefer to tuck the pointer symbol against the type: type* variable; to emphasize that a pointer to a type is effectively a new type. However, this can also be confusing, since defining multiple variables on the same line would not work properly: type* variable1, variable2; /* Only variable1 is actually a pointer. */ This commit reformats the entirety of the libjpeg-turbo code base so that it uses the same code formatting convention for pointers that the TurboJPEG API code uses: type *variable1, *variable2; This seems to be the most common convention among C programmers, and it is the convention used by other codec libraries, such as libpng and libtiff.
DRC 8632f1b2 2016-02-09T00:38:58 ARM64: Avoid tbl instruction on Cortex-A53/A57 Full-color compression speedups relative to previous commits: Cortex-A53 (Nexus 5X), Android, 64-bit: 0.91-3.0% (avg. 1.8%) Cortex-A57 (Nexus 5X), Android, 64-bit: -0.35-1.5% (avg. 0.65%)
DRC 28f00bf2 2016-02-08T15:15:11 Fix iOS/ARMv8 build Broken by 46ecffa324be43aab80f6160dc57d98b0a54a704. gas-preprocessor.pl and/or the clang assembler apparently don't like default values in macro arguments, and we need to use a separate const section for each function (because of our use of adr, also necessitated by the broken clang assembler.)
DRC 46ecffa3 2016-02-07T22:05:56 ARM64: Avoid LD3/ST3 at run time, not compile time ... and only if ThunderX is detected. This can be easily expanded later on to include other CPUs that are known to suffer from slow LD3/ST3, but it doesn't make sense to disable LD3/ST3 for all non-Android Linux platforms just because ThunderX is slow.
DRC 219470d6 2016-02-07T20:36:02 ARM64 NEON SIMD implementation of Huffman encoding Full-color compression speedups relative to previous commits: Cortex-A53 (Nexus 5X), Android, 64-bit: 1.1-13% (avg. 6.0%) Cortex-A57 (Nexus 5X), Android, 64-bit: 0.0-22% (avg. 6.3%) Refer to #47 and #50 for discussion Closes #50 Note that this commit introduces a similar /proc/cpuinfo parser to that of the ARM32 implementation. It is used to specifically check whether the code is running on Cavium ThunderX and, if so, disable the ARM64 SIMD Huffman routines (which slow performance by an average of 8% on that CPU.) Based on: https://github.com/mayeut/libjpeg-turbo/commit/a8c282e5e5ac10a715d6d6a9ab22121982b485f6
DRC 15aaa7f7 2016-02-07T17:39:33 ARM SIMD: Comment tweaks
DRC cf888486 2016-02-02T23:17:06 Use consistent formatting in ARM NEON SIMD code There aren't really any best practices to follow here. I tried as best as I could to adopt a standard that would ease any future maintenance burdens. The basic tenets of that standard are: * Assembly instructions always start on Column 5, and operands always start on Column 21, except: - The instruction and operand can be indented (usually by 2 spaces) to indicate a separate instruction stream. - If the instruction is within an enclosing .if block in a macro, it should always be indented relative to the .if block. * Comments are placed with an eye toward readability. There are always at least 2 spaces between the end of a line of code and the associated in-line comment. Where it made sense, I tried to line up the comments in blocks, and some were shifted right to avoid overlap with neighboring instruction lines. Not an exact science. * Assembler directives and macros use 2-space indenting rules. .if blocks are indented relative to the macro, and code within the .if blocks is indented relative to the .if directive. * No extraneous spaces between operands. Lining up the operands vertically did not really improve readability-- personally, I think it made it worse, since my eye would tend to lose its place in the uniform columns of characters. Also, code with a lot of vertical alignment is really hard to maintain, since changing one line could necessitate changing a bunch of other lines to avoid spoiling the alignment. * No extraneous spaces in #defines or other directives. In general, the only extraneous spaces (other than indenting spaces) are between: - Instructions and operands - Operands and in-line comments This standard should be more or less in keeping with other formatting standards used within the project.
DRC cb49bb00 2016-02-02T23:10:27 Opt. ARM64 SIMD decompr. for in-order pipelines Decompression speedup relative to libjpeg-turbo 1.4.2 (ISLOW IDCT): 48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 60-113% (avg. 86%) Cortex-A53 (Nexus 5X), Android, 64-bit: 6.8-27% (avg. 14%) Cortex-A57 (Nexus 5X), Android, 64-bit: 2.0-14% (avg. 6.8%) Decompression speedup relative to libjpeg-turbo 1.4.2 (IFAST IDCT): 48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 51-98% (avg. 75%) Minimal speedup (1-5%) observed on iPhone 5S (Cortex-A7) NOTE: This commit avoids the st3 instruction for non-Android and non-Apple builds, which may cause a performance regression against libjpeg-turbo 1.4.x on ARM64 systems that are running plain Linux. Since ThunderX is the only platform known to suffer from slow ld3 and st3 instructions, it is probably better to check for the CPU type at run time and disable ld3/st3 only if ThunderX is detected. This commit also enables the use of ld3 on Android platforms, which should be a safe bet, at least for now. This speeds up compression on the afore-mentioned Nexus Cortex-A53 by 5.5-19% (avg. 12%) and on the Nexus Cortex-A57 by 1.2-14% (avg. 6.3%), relative to the previous commits. This commit also removes unnecessary macros. Refer to #52 for discussion. Closes #52. Based on: https://github.com/mayeut/libjpeg-turbo/commit/6bad905034e6e73b33ebf07a74a6b72f58319f62 https://github.com/mayeut/libjpeg-turbo/commit/488dd7bf1726e2f6af6e9294ccf77b729fec1f20 https://github.com/mayeut/libjpeg-turbo/commit/4f4d057c1fb31d643536e6effb46a5946e15c465 https://github.com/mayeut/libjpeg-turbo/commit/d3198afc43450989a4fc63d2dcbe3272c8a0a3c1
DRC d38b4f21 2016-01-16T01:53:32 Optimize ARM64 SIMD code for Cavium ThunderX Per @ssvb: ThunderX is an ARM64 chip that dedicates most of its transistor real estate to providing 48 cores, so each core is not as fast as a result. Each core is dual-issue & in-order for scalar instructions and has only a single-issue half-width NEON unit, so the peak throughput is one 128-bit instruction per 2 cycles. So careful instruction scheduling is important. Furthermore, ThunderX has an extremely slow implementation of ld2 and ld3, so this commit implements the equivalent of those instructions using ld1. Compression speedup relative to libjpeg-turbo 1.4.2: 48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 58-85% (avg. 74%) relative to jpeg-6b: 1.75-2.14x (avg. 1.95x) Refer to #49 and #51 for discussion. Closes #51. This commit also wordsmiths the ChangeLog entry (the ARMv8 SIMD implementation is "complete" only for compression-- it still lacks some decompression algorithms, as does the ARMv7 implementation.) Based on: https://github.com/mayeut/libjpeg-turbo/commit/9405b5fd031558113bdfeae193a2b14baa589a75 which is based on: https://github.com/libjpeg-turbo/libjpeg-turbo/commit/f561944ff70adef65bb36212913bd28e6a2926d6 https://github.com/libjpeg-turbo/libjpeg-turbo/commit/962c8ab21feb3d7fc2a7a1ec8d26f6b985bbb86f
DRC ec6941f7 2016-01-15T09:29:11 Complete the ARM64 NEON SIMD implementation This adds 64-bit NEON coverage for all of the algorithms that are covered by the 32-bit NEON implementation, except for h2v1 (4:2:2) fancy upsampling (used when decompressing 4:2:2 JPEG images.) It also adds 64-bit NEON SIMD coverage for: * slow integer forward DCT (compressor) * h2v2 (4:2:0) downsampling (compressor) * h2v1 (4:2:2) downsampling (compressor) which are not covered in the 32-bit implementation. Compression speedups relative to libjpeg-turbo 1.4.2: Apple A7 (iPhone 5S), iOS, 64-bit: 113-150% (reported) 48-core ThunderX (RunAbove ARM Cloud), Linux, 64-bit: 2.1-33% (avg. 15%) Refer to #44 and #49 for discussion This commit also removes the unnecessary if (simd_support & JSIMD_ARM_NEON) statements from the jsimd* algorithm functions. Since the jsimd_can*() functions check for the existence of NEON, the corresponding algorithm functions will never be called if NEON isn't available. Based on: https://github.com/mayeut/libjpeg-turbo/commit/dcd9d84f10fae192c0e3935818dc289bca9c3e29 https://github.com/mayeut/libjpeg-turbo/commit/b0d87b811f37bd560083deea8c6e7d704e5cd944 https://github.com/mayeut/libjpeg-turbo/commit/70cd5c8a493a67f4d54dd2067ae6dedb65d95389 https://github.com/mayeut/libjpeg-turbo/commit/3e58d9a064648503c57ec2650ee79880f749a52b https://github.com/mayeut/libjpeg-turbo/commit/837b19542f53fa81af83e6ba002d559877aaf597 https://github.com/mayeut/libjpeg-turbo/commit/73dc43ccc870c2e10ba893e9764b8e48d6836585 https://github.com/mayeut/libjpeg-turbo/commit/a82b71a261b4c0213f558baf4bc745f1c27356d8 https://github.com/mayeut/libjpeg-turbo/commit/c1b1188c2106d6ea7b76644b6023b57edeb602e1 https://github.com/mayeut/libjpeg-turbo/commit/305c89284e1bb222b34fbc7261f697a0cc452a41 https://github.com/mayeut/libjpeg-turbo/commit/7f443f99950b4d7d442b9b879648eca5273209bd https://github.com/mayeut/libjpeg-turbo/commit/4c2b53b77da5a20e30e2aadaeddb0efbfe24e06d Unified version with fixes: https://github.com/mayeut/libjpeg-turbo/commit/1004a3cd05870612a194b410efeaa1b4da76d246
DRC d70a5c12 2015-12-14T16:59:43 Remove unnecessary .arch directive in ARM64 code This directive was preventing the code from assembling using the integrated assembler in clang. Fixes #33
DRC 1e32fe31 2015-10-14T17:32:39 Replace INT32 with a new internal datatype (JLONG) These days, INT32 is a commonly-defined datatype in system headers. We cannot eliminate the definition of that datatype from jmorecfg.h, since the INT32 typedef has technically been part of the libjpeg API since version 5 (1994.) However, using INT32 internally is risky, because the inclusion of a particular header (Xmd.h, for instance) could change the definition of INT32 from long to int on 64-bit platforms and thus change the internal behavior of libjpeg-turbo in unexpected ways (for instance, failing to correctly set __INT32_IS_ACTUALLY_LONG to match the INT32 typedef-- perhaps as a result of including the wrong version of jpeglib.h-- could cause libjpeg-turbo to produce incorrect results.) The library has always been built in environments in which INT32 is effectively long (on Windows, long is always 32-bit, so effectively it's the same as int), so it makes sense to turn INT32 into an explicitly long datatype. This ensures that libjpeg-turbo will always behave consistently, regardless of the headers included at compile time. Addresses a concern expressed in #26.
DRC 62999d77 2014-12-19T15:36:39 Modify the ARM64 assembly file so that it uses only syntax that the clang assembler in XCode 5.x can understand. These changes should all be cosmetic in nature-- they do not change the meaning or readability of the code nor the ability to build it for Linux. Actually, the code is now more in compliance with the ARM64 programming manual. In addition to these changes, there were a couple of instructions that clang simply doesn't support, so gas-preprocessor.pl was modified so that it now converts those into equivalent instructions that clang can handle. git-svn-id: svn+ssh://svn.code.sf.net/p/libjpeg-turbo/code/branches/1.4.x@1450 632fc199-4ca6-4c93-a231-07263d6284db
DRC 0a9a2526 2014-08-29T01:53:17 Rename the ARM64 assembly file to match the C file git-svn-id: svn+ssh://svn.code.sf.net/p/libjpeg-turbo/code/trunk@1390 632fc199-4ca6-4c93-a231-07263d6284db