cmake: fix arm64 to not trigger armv4 ASM Reported here: https://github.com/libressl/portable/pull/935#issuecomment-1798345787