Bug#1128483: xsimd: drop arch-specific armhf build flags

[Drew Parsons]

Source: xsimd
Version: 14.0.0-5
Severity: normal
X-Debbugs-Cc: debian-arm at lists.debian.org
User: debian-arm at lists.debian.org
Usertags: armhf
Control: forwarded -1 https://github.com/xtensor-stack/xsimd/issues/1260

Bug#1128257 requested we drop the arch-specific build flags for armhf.
But we found they are still needed to manage failure with NEON code on
32-bit ARM, and instead updated them to -march=armv7-a+fp

However, the principle of not applying arch-specific build flags is
still reasonable in general. We should drop them if we can.

This requires help upstream to resolve the SIGABRT in 
  [batch]<xsimd::batch<unsigned char>>
on 32-bit ARM systems that claim neon support
(i.e. on armv8l, which is Aarch64 hardware but used by debian to provide armhf environments).

This is upstream issue #1260.