The recent change to install kernel modules for AMD included a sed job to
disable kernel modules in the defconfig. This somehow broke booting on
a307, except the commit failed to bump the arm64_test tag so it wasn't
noticed until the next uprev. (I didn't notice when landing the next
change to that container to add the deqp runner, because I didn't get a
git conflict on rebasing my tag bump so I didn't bump the tag again to
pull in the kernel changes and catch the fail).
I've spent a while trying to debug what's happened (including what
*should* be a replication of the kernel build on my local db410c) and come
up empty. Just punt and disable the AMD kernel module changes on
baremetal to fix it. Bump every container using lava_build.sh to make
sure we don't screw anything up with the script changes.
Fixes: 60c5729d16 ("ci: Distribute ADMGPU driver to LAVA as a module")
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6971>
(cherry picked from commit bf576b449e)
I found the C++ runner hard to develop on, and we had stability issues and
outstanding feature needs that made me want something I felt good about
hacking on. Thus, Rewrite It In Rust of the deqp runner.
The new runner includes:
- Skip lists don't reshuffle the test list.
- Known-flake handling without resorting to skip lists (fixing our main CI
reliability issue on a3xx right now).
- Per-thread Vulkan shader caches should speed up VK CI runtime.
- Tracking of crashes separate from fails (so we can see progress on that
front).
- Logging of deqp stderr spam (particularly assertion failures!) in the CI
log.
- Integrated QPA filtering so we don't have bash perf issues for it.
- Logging of what caselist to go look at for a given error report (in red,
so it's easier to find in your CI log).
- The code is 1/3 unit tests, and easy to extend for more coverage.
- Non-LAVA CI runs create a failures.csv in artifacts that you can check
in as your deqp-*-fails.txt file.
- Test runtime is included in results.csv so you can debug how to speed up
your CI job.
- Pretty summary at the end of the run of slow/flaky/failed tests.
Since this is a new runner with a different RNG, the test groups are
shuffled one more time. This seems to result in some panfrost T720
stability issues (See its new deqp-panfrost-t720-flakes.txt), and one new
flake in freedreno a630.
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7434>
As it needs firmware to probe, and we cannot bundle it within the kernel
image because it is incompatible with the GPL.
Currently we rebind the driver after boot but that's slow and fragile,
as unloads of DRM drivers aren't generally tested.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7420>
I wasted a bunch of time today tracking down a spurious test results
change due to a driver invoking UB by running tests where NIR validation
had failed (instruction reading from components beyond vector size). If
we need to shrink our coverage to get runtimes down, it will still be
better to be catching validation errors in CI.
To keep the test jobs runtime under 10 minutes, I've split a530's gles2 to
two different jobs.
Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7203>
Due to scheduled physical lab maintenance to prepare for expansion and
better shard our device types across redundant infrastructure, these
device types will be inaccessible for approx. 3 hours. Disable them
until they are back.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6302>
Some changes unintendedly slipped into an unrelated commit before it was
merged.
This caused kernel modules to be built and installed in the ramdisk,
which caused some devices to fail to boot due to the ramdisk size limit
being surpassed.
These changes weren't in effect until a subsequent commit triggered a
rebuild of the ramdisks.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Fixes: a9560939e0 ("ci: Build-test Panfrost tools")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6167>
If another MR was merged while these were still running for the main
project, the result could be no updated images in the main project
registry (forcing a rebuild of the new images in all forked projects) or
an outdated Mesa website.
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6011>
It's something that was added to ease development, but that was supposed
to be removed before merging.
It also causes problems when arm-related jobs aren't enabled, as
arm_build is needed by these jobs but in that case isn't there.
Also extend from .ci-run-policy.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5802>
Place the kernel and ramdisk into a place in the file server so the URL
will only change when the contents also change.
Also put the Mesa build into a separate tarball so the ramdisk's
contents don't change every build.
With proper caching in place, all devices in the same farm need only to
download the mesa tarball once, saving time.
As we switch to MinIO for making kernels and rootfs available to LAVA
devices, we can stop using Docker to distribute them.
Instead, build when needed in separate jobs that push directly to MinIO,
from where LAVA devices can download them.
Signed-off-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5515>
Phase two of our network reconfiguration is happening this afternoon, so
we need to drop our RK3399 out for a little while. (Part of this
reconfiguration is to shard our devices across networks and racks, so
losing one part of our infrastructure doesn't mean losing any particular
device type.)
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5689>
Switches have been rewired, VLANs have been reconfigured, network
elements with non-functional remote management have been removed from
racks and thrown on desks in anger.
This reverts commit ae6e1aee7d1bd49ae494b8a25ca33d092a3a145a.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5301>
We're reconfiguring our Cambridge office lab networking and physical
setup for more scalability amongst other things. We can still run jobs
on one RK3399 at the peak outage, but we'll lose the T7x0 this morning,
so disable it until it's all back.
T820 is still disabled due to an unrelated BayLibre internal outage.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5296>
