We don't need these Vulkan bits, so let's try not building them. This
might seem kind of petty, but given that we build dEQP four times for GL
(once for GL, once for GLES, once for EGL/X11, and once for
EGL/Wayland), it does actually make a measurable difference to the time
spent rebuilding dEQP.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33287>
This avoids a whole pile of nonsense that we don't need during the dEQP
build, which is helpful when you build it five times in a row. In
particular, not building the Vulkan validation layers means we don't
waste three minutes building a second copy of SPIRV-Tools which we never
use.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33287>
LLVM 15 is pretty old, and notably not supported by either ANGLE nor
Skia anymore. So let's move up to LLVM 19 using packages provided by
LLVM themselves, apart from PPC and ARMv7 which don't have builds.
The Skia build now requires a bunch of new warning exclusions; hopefully
most of these are no longer needed when we can upgrade Skia shortly.
The ci-deb-repo revision has also been bumped to get us a new version of
xtensor which builds with LLVM 19, and a version of spirv-tools which
also works with LLVM 19.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Closes: mesa/mesa#11538
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33137>
We now have to do five builds of dEQP: one each for tools, GL, GLES,
Vulkan (release branch), and Vulkan (main).
dEQP is not a quick build, with even the tools build taking ten minutes,
at least three of which go to just pulling the SPIRV-Tools source.
Building all five takes a cumulative 60 minutes (10 tools, 11 GL, 9
GLES, 15 VK release, 13 VK main). That doesn't give us a load of time to
build everything else we need.
Two hours is obviously a completely unreasonable amount of time to spend
doing builds, especially for jobs which shouldn't really exist. But the
yak shave to allow the container jobs to be able to run on non-b2c
hardware has been a long one, and with no big-bang solution turning up
any time soon, we need to be able to actually do rootfs builds.
So, let's do the worst possible thing and go for a gigantic timeout.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33137>
Rename the variable ANDROID_NDK to ANDROID_NDK_VERSION and only use it
to specify the version itself.
This makes it consistent with ANDROID_SDK_VERSION and it is also in
preparation for using the same variable to build ANGLE in the future.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33188>
When cross-building for Android meson fails to find supported compiler
arguments for the C++ cross-compiler, e.g.:
```
Compiler for C++ supports arguments -Wno-array-bounds: NO (cached)
Compiler for C++ supports arguments -Wno-overflow: NO
Compiler for C++ supports arguments -Wno-c++11-narrowing: NO (cached)
Compiler for C++ supports arguments -Wno-vla-cxx-extension: NO (cached)
```
This is due to an **unrelated** and more generic compilation failure
when testing for the supported arguments, e.g.:
```
Command line: `/tmp/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android34-clang++ -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -static-libstdc++ /home/ao2/Collabora/GOO0042/mesa/_build/meson-private/tmpv6_hke9l/testfile.cpp -o /home/ao2/Collabora/GOO0042/mesa/_build/meson-private/tmpv6_hke9l/output.obj -c -Wno-error=c99-designator -Wno-error=unused-variable -Wno-error=unused-but-set-variable -Wno-error=self-assign -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -Werror=implicit-function-declaration -Werror=unknown-warning-option -Werror=unused-command-line-argument -Werror=ignored-optimization-argument -Wvla-cxx-extension -Wno-vla-cxx-extension` -> 1
stderr:
clang++: error: argument unused during compilation: '-static-libstdc++' [-Werror,-Wunused-command-line-argument]
-----------
Compiler for C++ supports arguments -Wno-vla-cxx-extension: NO
```
The issue is caused by how the cross compiler is set up by
.gitlab-ci/container/create-android-cross-file.sh
Allow the cross compiler to still start even when the
`-Werror,-Wunused-command-line-argument` error occurs in order to be
able to actually detect other arguments:
```
Command line: `/tmp/android-ndk-r27c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android34-clang++ -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables --start-no-unused-arguments -static-libstdc++ --end-no-unused-arguments /home/ao2/Collabora/GOO0042/mesa/_build/meson-private/tmpm6eolxed/testfile.cpp -o /home/ao2/Collabora/GOO0042/mesa/_build/meson-private/tmpm6eolxed/output.obj -c -Wno-error=c99-designator -Wno-error=unused-variable -Wno-error=unused-but-set-variable -Wno-error=self-assign -D_FILE_OFFSET_BITS=64 -O0 -fpermissive -Werror=implicit-function-declaration -Werror=unknown-warning-option -Werror=unused-command-line-argument -Werror=ignored-optimization-argument -Wvla-cxx-extension -Wno-vla-cxx-extension` -> 0
Compiler for C++ supports arguments -Wno-vla-cxx-extension: YES
```
This makes argument detection work again, e.g:
```
Compiler for C++ supports arguments -Wno-array-bounds: YES (cached)
Compiler for C++ supports arguments -Wno-overflow: YES
Compiler for C++ supports arguments -Wno-c++11-narrowing: YES (cached)
Compiler for C++ supports arguments -Wno-vla-cxx-extension: YES (cached)
```
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/12441
Reviewed-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33013>
Also add a note in .gitlab-ci/container/debian/x86_64_test-android.sh
that DEBIAN_TEST_ANDROID_TAG needs to be updated when that file changes
too.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
Use a custom kernel when launching cuttlefish, this is needed for
-gpu_mode=venus.
And while at it also clean up the state from previous runs, this seems
to be required when passing a custom kernel.
In this change also pass -enable_bootanimation=false which makes
drm_virgl work, and pass -enable_minimal_mode=true to avoid using some
features which are not really needed.
Note that the file number limit has to be increased too in
.gitlab-ci/cuttlefish-runner.sh for cuttlefish to work properly.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
To test mesa on cuttlefish some special images tailored for upstream
mesa are needed, in particular these images may set properties or add
functionality that enable new use cases for upstream mesa on Android,
like for instance using venus via the vulkan.virtio mesa driver.
So instead of downloading the standard images from Google, get those
custom images from s3.freedesktop.org.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
The ./tools/buildutils/build_packages.sh script used by the
android-cuttlefish project built in debian/x86_64_test-android.sh uses
sudo, so install that as an EPHEMERAL dependency.
Also set SUDO_FORCE_REMOVE=yes when removing sudo to avoid some errors.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
In order to build the mesa llvmpipe driver for Android, llvm libraries
for Android are needed; force meson to use a fallback for llvm and
provide prebuilt libraries making them available to meson thanks to
a wrap file.
Enable other software drivers for the x86_64-linux-android build, but
also disable the amd and intel hardware drivers for now which don't
build correctly on Android.
Download the prebuilt LLVM libraries for Android in the container job
rather than in the build job, to avoid downloading some big artifacts
for every mesa version built for Android.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
In commit ad42be50c9 (ci/deqp: fully isolate deqp builds, 2024-03-17)
the deqp module version file was changed from something like
`/deqp/version-$deqp_api` to something like `/deqp-$deqp_api/version`
and this seems to cause some issues when building for Android:
-----------------------------------------------------------------------
[439/1301] Building CXX object external/amber/src/src/vulkan/CMakeFiles/libamberenginevulkan.dir/blas.cc.o
FAILED: external/amber/src/src/vulkan/CMakeFiles/libamberenginevulkan.dir/blas.cc.o
/tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ --target=x86_64-none-linux-android28 --sysroot=/tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot -DAMBER_CTS_VULKAN_HEADER=1 -DAMBER_ENABLE_CLSPV=0 -DAMBER_ENABLE_DXC=0 -DAMBER_ENABLE_LODEPNG=1 -DAMBER_ENABLE_RTTI=1 -DAMBER_ENABLE_SHADERC=0 -DAMBER_ENABLE_SPIRV_TOOLS=0 -DAMBER_ENGINE_DAWN=0 -DAMBER_ENGINE_VULKAN=1 -DDE_ANDROID_API=28 -DDE_ASSERT_FAILURE_CALLBACK -DDE_COMPILER=DE_COMPILER_CLANG -DDE_CPU=DE_CPU_X86_64 -DDE_FENV_ACCESS_ON="_Pragma(\"STDC FENV_ACCESS ON\")" -DDE_MINGW=0 -DDE_OS=DE_OS_ANDROID -DDE_PTR_SIZE=8 -I/tmp/VK-GL-CTS/external/amber/../vulkancts/framework/vulkan/generated/vulkan -I/tmp/VK-GL-CTS/framework/delibs/debase -I/tmp/VK-GL-CTS/framework/delibs/decpp -I/tmp/VK-GL-CTS/framework/delibs/depool -I/tmp/VK-GL-CTS/framework/delibs/dethread -I/tmp/VK-GL-CTS/framework/delibs/deutil -I/tmp/VK-GL-CTS/framework/delibs/destream -I/tmp/VK-GL-CTS/execserver -I/tmp/VK-GL-CTS/external/spirv-headers/src/include -I/tmp/VK-GL-CTS/external/amber/src/include -I/tmp/VK-GL-CTS/external/amber/src -I/tmp/deqp-vk -I/tmp/deqp-vk/external/vulkancts/framework/vulkan/generated/vulkan -Wall -Wextra -Wno-long-long -Wshadow -Wundef -Wconversion -Wno-sign-conversion -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -frtti -fexceptions -fwrapv -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -std=gnu++17 -fPIC -w -fno-exceptions -fvisibility=hidden -Wall -Wextra -Wno-cast-function-type-strict -Wno-padded -Wno-switch-enum -Wno-unknown-pragmas -Wno-unsafe-buffer-usage -pedantic-errors -Werror -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-format-pedantic -Wno-unknown-warning-option -Weverything -Wno-zero-as-null-pointer-constant -MD -MT external/amber/src/src/vulkan/CMakeFiles/libamberenginevulkan.dir/blas.cc.o -MF external/amber/src/src/vulkan/CMakeFiles/libamberenginevulkan.dir/blas.cc.o.d -o external/amber/src/src/vulkan/CMakeFiles/libamberenginevulkan.dir/blas.cc.o -c /tmp/VK-GL-CTS/external/amber/src/src/vulkan/blas.cc
In file included from /tmp/VK-GL-CTS/external/amber/src/src/vulkan/blas.cc:16:
In file included from /tmp/VK-GL-CTS/external/amber/src/src/vulkan/blas.h:19:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/vector:274:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/__bit_reference:14:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/bit:57:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/limits:105:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/type_traits:417:
In file included from /tmp/android-ndk-r25b/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/cstddef:37:
/tmp/deqp-vk/version:1:1: error: unknown type name 'dEQP'
dEQP base version vulkan-cts-1.3.10.0
^
-----------------------------------------------------------------------
This seems to be caused by `cstddef` from the Android NDK which includes
a file named `version`, i.e.:
-----------------------------------------------------------------------
#include <version>
-----------------------------------------------------------------------
To avoid the problem also add the deqp module name to the file name,
like: `/deqp-$deqp_api/deqp-$deqp_api-version`.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
Collect the mustpass lists also for the Android target, in particular
this makes vk-main.txt to be found again when running vulkan tests on
Android.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
The main branch of dEQP build fine for android but the GL and GLES
branches fail with errors like the following:
-----------------------------------------------------------------------
.../external/openglcts/modules/common/glcFramebufferBlitTests.cpp:715:26: error: no member named 'floor' in namespace 'std'
return (GLubyte)std::floor(f == 1.0f ? 255 : f * 255.0);
~~~~~^
.../external/openglcts/modules/common/glcFramebufferBlitTests.cpp:1491:14: error: no member named 'fabs' in namespace 'std'
if (std::fabs(actual - expected) > eps)
~~~~~^
2 errors generated.
-----------------------------------------------------------------------
Cherry-pick a couple of commit from main that fix the errors.
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
When adding patches wit git-am create the date format for
GIT_COMMITTER_DATE in a locale-agnostic format, this avoids potential
issues when running the scripts locally with a non-English locale, e.g.:
-----------------------------------------------------------------------
++ GIT_COMMITTER_DATE='gio 1 gen 1970, 01:00:00, CET'
++ git am
fatal: invalid date format: gio 1 gen 1970, 01:00:00, CET
-----------------------------------------------------------------------
Reviewed-by: Eric Engestrom <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30895>
The running container jobs scripts S3_JWT_FILE needs to be removed on
exit because the resulting image is committed and we don't want the auth
token to leak permanently in the fs.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31093>
In order to build the mesa llvmpipe driver for Android, LLVM libraries
for Android are needed, but building them each time a new upstream mesa
is built would be too much overhead.
So add a script to build LLVM libraries for Android and call it from the
debian/android_build job. The actual build will be performed only when
necessary and the resulting artifact will be uploaded to the S3 bucket
to be re-used when building mesa for Android.
The build process follows what has been documented in
docs/drivers/llvmpipe.rst
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31093>
Unset the compiler env vars in debian/android_build.sh used to
cross-build libelf for Android, to prevents those env vars to
inadvertently affect subsequent commands that might want to build for
the native host after the cross-builds.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31093>
Set up "${S3_JWT_FILE}" also for container jobs, this can be useful in
case some jobs want to save something to S3.
Note that for container jobs setting the file /s3_swt in the
`default:before_script` section would not work: this wold be "too early"
because, when using ci-templates, the cbuild invocation switches the
root filesystem before executing FDO_DISTRIBUTION_EXEC, resulting in the
file becoming unavailable after the switch.
So set up the file exactly in FDO_DISTRIBUTION_EXEC, before launching
the actual container script.
Do this using a new trampoline script
.gitlab-ci/container/container_job_trampoline.sh so that in the future
other tasks common to all container jobs can be added there.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31093>
The debian/x86_64_pyutils job incorporates references to rules from the
python-test and .container jobs, but currently these jobs don't have any
express rules. The .container job does extend jobs that have rules, but
these same rules are already included in debian/x86_64 because it
extends the .debian-container job.
Remove these empty rule references to make debian/x86_64_pyutils more
readable.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32340>
