Commit Graph

253 Commits

Author SHA1 Message Date
Valentine Burley
b49eaf0966 ci/lava: Consolidate piglit trace job definitions
Clean up LAVA job definitions.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34424>
2025-04-11 07:05:07 +00:00
Valentine Burley
87d58ea57a ci/piglit: Consolidate HWCI_TEST_SCRIPT for piglit traces
The HWCI_TEST_SCRIPT variable was always getting overwritten for these
definitions.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34424>
2025-04-11 07:05:06 +00:00
Valentine Burley
1aeedddbb6 ci/piglit: Drop redundant PIGLIT_PROFILES variable
PIGLIT_PROFILES was only used with the piglit-runner.sh script, which no
jobs were using anymore.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34424>
2025-04-11 07:05:06 +00:00
Guilherme Gallo
70f10dc327 ci/lava: Fetch kernel modules from overlay
LAVA supports overlays in the deploy action, so there is no need to
download them in lava_build.sh and bloat the rootfs file with it.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34352>
2025-04-05 08:42:20 +00:00
Vignesh Raman
7959250d1e s3_upload: improve url validation and error message
Ensure s3_upload correctly validates the S3 folder url by requiring it
to end with /. This prevents wrong uploads to invalid paths, such as
file urls. Also improve the error message.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34255>
2025-04-04 13:32:45 +00:00
Daniel Stone
3b6a40af53 ci: Make all job timeouts explicit
Enforce a default job timeout of 1 second, to make jobs which don't
explicitly specify a timeout insta-fail, rather than potentially hanging
around for an hour.

Container builds get the full hour as they can run long and are not run
in pre-merge context, and LAVA jobs also get the full hour as they have
multiple internal timeout mechanisms which aim to fast-fail jobs once
they actually start. However, as they just queue jobs to an external
host (shared with other projects like KernelCI), these timeouts aren't
reflected into the GitLab CI definitions.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34280>
2025-04-01 12:21:01 +00:00
Daniel Stone
24c86ff345 ci/lava: Fail faster when getting results
Previously we'd burn 4 minutes waiting for curl to bang its head
trying to get results which would never emerge. Change this so we don't
fail on a 404, and also pull the retry down quicker, so we'll spend a
maximum of 2 minutes waiting for a server to come back, and no minutes
when the results aren't there to be fetched.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34244>
2025-03-27 20:23:15 +00:00
Eric Engestrom
7178425ccf ci: replace broken s3cp command with a simple curl call
The current `s3cp` implementation does not work anymore after the
migration, and instead of fixing it and propagating the fix down to us,
it's simpler to directly use `curl`.

Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120>
2025-03-24 20:45:52 +00:00
Guilherme Gallo
fb224e9016 ci/lava: Fix LAVA lima jobs
lima uses a different version from other farms, where some log output
patches were not delivery yet, so let's use a temporary fix to make
those job traces look as nice as the other ones.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33994>
2025-03-13 00:45:59 -03:00
Guilherme Gallo
0330522e99 ci/lava: Fix LAVA lima jobs
lima uses a different version from other farms, where some log output
patches were not delivery yet, so let's use a temporary fix to make
those job traces look as nice as the other ones.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33994>
2025-03-13 03:31:31 +00:00
Guilherme Gallo
8fcc52b8d7 ci/lava: Don't print empty lines when changing sections
Make `print_log` section-aware to stop printing newlines whenever a
section changes.
This also caught a bug: the `handle_exception` was sending an exception
type to the `print_log`, now it is fixed.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
422e65557d ci/lava: Tweak timeouts
LAVA actions follow a hierarchical structure, where most subactions have
their timeouts overridden if the parent action supports a retry
mechanism, such as the `depthcharge-retry` action.

The timeout is calculated as: [1]

```
parent action timeout / failure_retry value
```

To adjust a subaction's timeout, we need to modify the nearest parent
action.

[1]
https://gitlab.collabora.com/lava/lava/-/blob/collabora/production/lava_dispatcher/action.py#L149

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
a33c0e1867 ci/lava: Split boot action into deploy and boot
The boot action was wrapping the deploy action, which could cause
timeout misalignment. For example, the boot `GitlabSection` timeout was
shorter than the deploy timeout in LAVA, leading to cases where LAVA
jobs were canceled during their own retry mechanism.

By splitting these actions, we can align the timeouts properly,
preventing interference and unnecessary job cancellations.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
d85af615f9 ci/lava: Remove depthcharge-start timeout
It has no effect, as it is overridden by depthcharge-retry timeout /
failure_retry.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
47659ddf70 ci: Simplify LAVA farm detection
Refactor the LAVA farm detection to use a simpler environment
variable-based approach:
- Remove the complex regex-based farm detection
- Replace LavaFarm enum with a simple string-based farm identification
- Update related tests and job definition logic
- Remove hypothesis testing dependency

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33888>
2025-03-08 02:45:02 +00:00
Guilherme Gallo
fbc55afbdf ci/lava: Properly detect VMWARE farm
This will make the job definition default to the UART format for vmware
jobs, as only Collabora's farm relies on the SSH job definition due to
the unreliable Chromebook UART in LAVA.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33874>
2025-03-04 16:46:17 +00:00
Guilherme Gallo
1dbebd2619 ci/lava: Add U-Boot action timeout for rockchip DUTs
Add a specific timeout for the U-Boot action in LAVA job definitions for
rockchip devices. This ensures sufficient time for U-Boot to download
the kernel and set up early network, preventing potential job failures
due to timeout constraints.

This behavior started to happen since LAVA 2025.02 version.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33839>
2025-03-04 01:17:50 +00:00
Guilherme Gallo
2b9e3e2fff ci/lava: xtrace the lava_job_submitter call
To make easier to debug issues and run locally.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33839>
2025-03-04 01:17:50 +00:00
Guilherme Gallo
1169f704d3 ci/lava: Propagate errors in SSH tests
The `lava_ssh_test_case` wrapper was missing the `set -e` shell option,
which made LAVA system interpret the job was succeeding, because the
`container` namespace was exiting normally, even though the `dut`
namespace was failing.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33839>
2025-03-04 01:17:50 +00:00
Guilherme Gallo
02a86b3284 ci/lava: Drop the repeating quotes on lava-test-case
LAVA was recently patched [1] with a fix on how parameters are parsed in
`lava-test-case`, so we don't need to repeat quotes to send the
arguments properly to it.

[1] 18c9cf7976

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33839>
2025-03-04 01:17:50 +00:00
Eric Engestrom
55bd96d038 ci: rename generate-env.sh to export-gitlab-job-env-for-dut.sh
This makes it clearer what this script does.

Reviewed-by: Vignesh Raman <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33037>
2025-02-12 14:25:30 +00:00
Dudemanguy
1aa21c27d4 treewide: remove unneeded executable bit in non-scripting files
Noticed one by chance and searched for any others with find that were
clearly not meant to be executable.

For the curious:

33aa039acf changed texstore.c to
executable.
ed176e2c71 introduced si_vpc.c and
si_vpc.h which have always been executable.
d0e5203855 changed lava-gitlab-ci.yml to
executable.
328c29d600 introduced OVR_multiview.xml as
executable.
ac912b3754 introduced
OVR_multiview_multisampled_render_to_texture.xml as exectuable.

Signed-off-by: Dudemanguy <random342@airmail.cc>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33436>
2025-02-06 16:51:15 -06:00
Valentine Burley
b2105fe162 ci/lava: Allow passing extra cmdline arguments
The LAVA_CMDLINE variable is appended to extra_nfsroot_args.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33282>
2025-02-05 14:01:03 +00:00
Detlev Casanova
8f4efda257 ci/fluster: Add radeonsi-raven-vaapi-fluster jobs
The CI job runs each supported decoder on 1 LAVA job:
- vp9
- h.264
- h.265

They each run the gstreamer VAAPI fluster decoder tests on a radeonsi
device on LAVA.

Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>
Co-authored-by: Sergi Blanch Torne <sergi@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26113>
2025-01-25 08:48:04 +00:00
Detlev Casanova
aa11f3d630 ci/fluster/lava: Add fluster in LAVA rootfs
It will be used in next commits to run fluster vaapi tests on AMD
devices on LAVA.

Signed-off-by: Detlev Casanova <detlev.casanova@collabora.com>
Co-authored-by: Sergi Blanch Torne <sergi@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26113>
2025-01-25 08:48:04 +00:00
Martin Roukala (né Peres)
660412c63d ci/init-stage2: use the common scripts from the build artifact
This unifies the behaviour between the LAVA, baremetal, and CI-Tron
farms by ensuring every job has access and runs the same scripts.

The init-* scripts are however still sourced from outside the build
artifact, hopefully not for too long.

Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33026>
2025-01-16 04:07:42 +00:00
Valentine Burley
61d9c47944 ci/lava: Use CI_JOB_TIMEOUT instead of separate variable
The CI_JOB_TIMEOUT variable is the GitLab-defined job timeout in
seconds.
Use this variable in LAVA instead of the separate JOB_TIMEOUT,
which was intended to represent the test phase timeout (job timeout
minus 5 minutes), but was often overlooked.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32609>
2024-12-18 09:23:27 +00:00
Deborah Brouwer
3bddf77f61 ci: separate python tests and artifacts
Currently the python-test job runs pytest and then creates python
related artifacts for use by other jobs. But the python-test job isn't
triggered by changes to the python scripts and libraries. Instead, changes
to python scripts and libraries can currently be merged without running
pytest and, if they do generate errors, they will only appear in ci
later when trying to generate the python artifacts.

Add pytest to the existing linting jobs and expand the rules so that it
will be triggered by changes to python scripts and libraries. Remove
pytest from the job that creates artifacts. This will ensure that the
artifacts are still valid while catching script/library errors before they
are merged.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32340>
2024-12-13 10:04:03 -08:00
Deborah Brouwer
b6435207ab ci: python-test rename artifacts
The current python-test job creates and compresses python related
artifacts for use by future jobs. The artifacts are currently named
`mesa-python-test` which is somewhat misleading because they are not
needed for testing python scripts or libraries.

Rename the artifacts generated by the python-test job to be more
descriptive of their purpose.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32340>
2024-12-13 10:04:03 -08:00
Valentine Burley
3bff52da4e ci: Drop lava-piglit:(x86_64|arm64) definitions
As part of the migration to deqp-runner suites, remove these
definitions to prevent the introduction of additional piglit
jobs without test suites.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32461>
2024-12-12 10:35:41 +00:00
Guilherme Gallo
93ce733425 ci/lava: Improve exception handling for job failures
Include detailed error messages when raising exceptions on LAVA job
failures to enhance debugging and error tracking.

Also, handle additional error types by extracting error messages from
metadata and retrying accordingly.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32163>
2024-11-21 04:10:52 +00:00
Guilherme Gallo
b2c2f0d187 ci/lava: Set default exit code to 1 for failed jobs
Sets the default exit code to 1 to ensure the GitLab job fails when the
LAVA job fails or is interrupted. Adds tests to verify the exit code is
correctly set based on the logs or the lack of them (unexpected
finishing: timeouts and canceling).

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32163>
2024-11-21 04:10:52 +00:00
Guilherme Gallo
bc86b73bbe ci/lava: Fix lava-tags parsing
python-fire auto-converts `item1,item2` into a tuple, but if there is a
dash `-` inside the argument, it treats it as a string.

Let's validate the data, when it comes as a `str` or `tuple`.

For more details, here are the tested scenarios:

| --lava-tags= | Type  | Value               |
|--------------|-------|---------------------|
| None         | bool  | True                |
| ''           | str   | ''                  |
| tag1         | str   | "tag1"              |
| tag1,        | tuple | ("tag1",)           |
| tag-1,tag-2  | str   | 'tag-1,tag-2'       |
| tag1,tag2    | tuple | ("tag1", "tag2")    |
| ','          | str   | ','                 |
| ',,'         | str   | ',,'                |
| 'tag1,,'     | str   | 'tag1,,'            |

See also:
https://google.github.io/python-fire/guide/#argument-parsing

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
ce78dcb24e ci/lava: Refactor is-rootfs-uploaded function
We don't need the /done anymore, because we have better job
dependencies. But the mainline-or-fork query is still helpful, so
refactor that out into a common helper we can reuse for other things.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
f44970173d ci/lava: Provide list of overlays to submitter
Instead of providing a hardcoded set of arguments, allow overlays to be
added to the submitter script. Passing Python dicts as a string
representation and relying on coercion from strings is far from great,
but fire doesn't give anything else, so.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
f32a2de26d ci/lava: Provide LAVA rootfs URL directly
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
2b3839c9c7 ci/lava: Use LAVA rootfs overlays for build/per-job
We compose the rootfs from a mixture of the base rootfs (exported from
the container build stage, currently lava_build.sh, which can be reused
as long as the container isn't rebuilt), the Mesa build overlay
(exported from the debian-* build job, which can be reused for every job
in that pipeline), and the per-job rootfs (containing job-specific
variables which cannot be reused).

Instead of having LAVA pull the base rootfs and separately downloading
the build/per-job parts on the DUT, get LAVA to compose the whole thing
by using overlays.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:26 +00:00
Daniel Stone
021d7d8b77 ci/lava: Remove duplicate build download
We already do it above.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:26 +00:00
Vignesh Raman
c3531d5fea ci: rename FORCE_KERNEL_TAG to EXTERNAL_KERNEL_TAG
FORCE_KERNEL_TAG allows testing kernel uprevs without rebuilding
containers by supplying an external kernel directly for booting on
hardware devices. Renaming it to EXTERNAL_KERNEL_TAG clarifies its
purpose, distinguishing it from KERNEL_TAG which rebuilds containers.

Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31795>
2024-10-28 02:18:27 +00:00
Daniel Stone
ef175ea594 ci: Make section emission really quiet
If you're emitting a section header under set -x, you will see:
+ section_start foo "foo"
+ x_off
[the section header]

This is kind of annoying. Instead of trying to squash it everywhere by
dancing around local set +x management, play some extremely stupid
tricks with shells to make sure we never emit it.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:43 +01:00
Daniel Stone
d171f47f44 ci/lava: Coalesce post-processed job information
We can combine a section _and_ a print.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
9be46b29f0 ci/lava: Print relative timestamps in sections
Follow what the shell executor does and print the time since the job
started in the section header.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
8ee6241a8c ci/hw: Wrap pre-test setup in collapsed section
Most people don't care about environment variables and starting Weston.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
970f37be09 ci/lava: Change default section colour to cyan
This matches the sections from the shell prints, which are quite nice.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
3c7b53e27c ci/lava: Be a little less enthusiastic with bold
Some things can just fade away into the background.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
dead2b7e62 ci/lava: Fix colour definitions
All the foreground colours pass 1 to ANSI SGR, which sets bold. The
other arguments are either a colour from 30-37 (passed directly), or
38;5;nnn, where nnn is an extended RGB colour. It looks like most of the
definitions were cargo-culted from FG_RED, which correctly sets an
extended colour, because the arguments there were being parsed as
setting blinking, followed by 197 which was ignored as unknown.

Fix them to just set the original definition.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
3068279280 ci/lava: Truncate printed times
We don't need to go down to the microsecond.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
65f05f2231 ci/lava: Explicitly pass UTC timezone
Rather than putting it into the environment, just pass it explicitly
every time we need a timestamp.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
2b4d468421 ci/lava: Hide more boot details into sections
Make sure we keep as much of the boot as we can behind sections, so
by default people only see the actual test run.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Daniel Stone
837004fa26 ci/lava: Rename lava_boot section
'LAVA boot' is less meaningful to normal people than explaining that
we're booting the hardware device under test.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00