Commit Graph

32 Commits

Author SHA1 Message Date
Guilherme Gallo
422e65557d ci/lava: Tweak timeouts
LAVA actions follow a hierarchical structure, where most subactions have
their timeouts overridden if the parent action supports a retry
mechanism, such as the `depthcharge-retry` action.

The timeout is calculated as: [1]

```
parent action timeout / failure_retry value
```

To adjust a subaction's timeout, we need to modify the nearest parent
action.

[1]
https://gitlab.collabora.com/lava/lava/-/blob/collabora/production/lava_dispatcher/action.py#L149

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
a33c0e1867 ci/lava: Split boot action into deploy and boot
The boot action was wrapping the deploy action, which could cause
timeout misalignment. For example, the boot `GitlabSection` timeout was
shorter than the deploy timeout in LAVA, leading to cases where LAVA
jobs were canceled during their own retry mechanism.

By splitting these actions, we can align the timeouts properly,
preventing interference and unnecessary job cancellations.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
d85af615f9 ci/lava: Remove depthcharge-start timeout
It has no effect, as it is overridden by depthcharge-retry timeout /
failure_retry.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
2025-03-10 05:44:25 +00:00
Guilherme Gallo
47659ddf70 ci: Simplify LAVA farm detection
Refactor the LAVA farm detection to use a simpler environment
variable-based approach:
- Remove the complex regex-based farm detection
- Replace LavaFarm enum with a simple string-based farm identification
- Update related tests and job definition logic
- Remove hypothesis testing dependency

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33888>
2025-03-08 02:45:02 +00:00
Guilherme Gallo
1dbebd2619 ci/lava: Add U-Boot action timeout for rockchip DUTs
Add a specific timeout for the U-Boot action in LAVA job definitions for
rockchip devices. This ensures sufficient time for U-Boot to download
the kernel and set up early network, preventing potential job failures
due to timeout constraints.

This behavior started to happen since LAVA 2025.02 version.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33839>
2025-03-04 01:17:50 +00:00
Valentine Burley
b2105fe162 ci/lava: Allow passing extra cmdline arguments
The LAVA_CMDLINE variable is appended to extra_nfsroot_args.

Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33282>
2025-02-05 14:01:03 +00:00
Guilherme Gallo
bc86b73bbe ci/lava: Fix lava-tags parsing
python-fire auto-converts `item1,item2` into a tuple, but if there is a
dash `-` inside the argument, it treats it as a string.

Let's validate the data, when it comes as a `str` or `tuple`.

For more details, here are the tested scenarios:

| --lava-tags= | Type  | Value               |
|--------------|-------|---------------------|
| None         | bool  | True                |
| ''           | str   | ''                  |
| tag1         | str   | "tag1"              |
| tag1,        | tuple | ("tag1",)           |
| tag-1,tag-2  | str   | 'tag-1,tag-2'       |
| tag1,tag2    | tuple | ("tag1", "tag2")    |
| ','          | str   | ','                 |
| ',,'         | str   | ',,'                |
| 'tag1,,'     | str   | 'tag1,,'            |

See also:
https://google.github.io/python-fire/guide/#argument-parsing

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
f44970173d ci/lava: Provide list of overlays to submitter
Instead of providing a hardcoded set of arguments, allow overlays to be
added to the submitter script. Passing Python dicts as a string
representation and relying on coercion from strings is far from great,
but fire doesn't give anything else, so.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
f32a2de26d ci/lava: Provide LAVA rootfs URL directly
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:27 +00:00
Daniel Stone
2b3839c9c7 ci/lava: Use LAVA rootfs overlays for build/per-job
We compose the rootfs from a mixture of the base rootfs (exported from
the container build stage, currently lava_build.sh, which can be reused
as long as the container isn't rebuilt), the Mesa build overlay
(exported from the debian-* build job, which can be reused for every job
in that pipeline), and the per-job rootfs (containing job-specific
variables which cannot be reused).

Instead of having LAVA pull the base rootfs and separately downloading
the build/per-job parts on the DUT, get LAVA to compose the whole thing
by using overlays.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
2024-10-31 18:00:26 +00:00
Daniel Stone
8ee6241a8c ci/hw: Wrap pre-test setup in collapsed section
Most people don't care about environment variables and starting Weston.

Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31602>
2024-10-20 11:32:42 +01:00
Martin Krastev
8bcd18c90e svga/ci: change DNS server for vmware jobs
Previous vmware-specific DNS server no longer valid.

Signed-off-by: Martin Krastev <martin.krastev@broadcom.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/30344>
2024-07-24 21:26:26 +00:00
Guilherme Gallo
41dd1c52b1 ci/lava: Fix cmdline for UART/fastboot devices
Fastboot devices need an indirection for creating a boot image via
`mkbootimg`, so we need to propagate the cmdline from LAVA and our extra
arguments to it properly.

This commit fixes it by retrieving the default cmdline from LAVA and
sending it, together with the `extra_nfsroot_args` to the `mkbootimg`
command.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/29611>
2024-06-07 22:03:22 +00:00
Guilherme Gallo
7101aecc53 ci: Use id_tokens for JWT auth
Fixes: #9180

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28916>
2024-04-25 20:45:53 +00:00
Martin Krastev
3d348e632e svga/ci: workaround vmware farm's inability to use public DNS 8.8.8.8
This workaround is termporary and will be dropped once the
farm is moved to its permament location.

Signed-off-by: Martin Krastev <martin.krastev@broadcom.com>
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28261>
2024-03-27 10:30:23 +00:00
Martin Krastev
54050d8844 svga/ci: land vmware mesa-ci lava farm
* add vmware farm, jobs and local rules
* add qemu lava deploy and boot actions
* add vmware-qemu traces job for llvmpipe device, as manual
* add vmware-vmx piglit job for svga device, as automatic
* add llvmpipe device piglit skip/fail lists
* add svga to GALLIUM_DRIVERS in debian-testing job
* add svga device fails/skips/flakes lists
* switch KERNEL_REPO/FORCE_KERNEL_TAG to own fork

Signed-off-by: Martin Krastev <martin.krastev@broadcom.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Reviewed-by: David Heidelberg <david.heidelberg@collabora.com>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28261>
2024-03-27 10:30:23 +00:00
David Heidelberg
5e44cee47d ci: inject gfx-ci/linux S3 artifacts without rebuilding containers
We need update kernel often. We need test kernel changes often.

Introduced `KERNEL_EXTERNAL_TAG` to differ between `KERNEL_TAG` which is
also used to rebuild the containers. We don't need rebuild containers
for the external kernel, so this way we don't have to.

Updating kernel goes wruuuuuum.

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23563>
2023-11-07 12:22:09 +00:00
Guilherme Gallo
fdd51ef584 ci/lava: Enable SSH by default in fastboot devices
It is only applied in Collabora farm, since we still haven't seen many
UART hangs in Lima farm.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25912>
2023-11-02 03:31:50 +00:00
Guilherme Gallo
654f7f783f ci/lava: Make SSH definition wrap the UART one
Simplify both UART and SSH job definitions module to share common
building blocks themselves.

- generate_lava_yaml_payload is now a LAVAJobDefinition method, so
  dropped the Strategy pattern between both modules
- if SSH is supported and UART is not enforced, default to SSH
- when SSH is enabled, wrap the last deploy action to run the SSH server
  and rewrite the test actions, which should not change due to the boot
  method
- create a constants module to load environment variables

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25912>
2023-11-02 03:31:50 +00:00
Guilherme Gallo
76922f8404 ci/lava: Create LAVAJobDefinition
To absorb complexity from the building blocks to generate job
definitions for each mode:
- fastboot-uart
- uboot-uart
- uboot-ssh

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25912>
2023-11-02 03:31:50 +00:00
Guilherme Gallo
af9273eb4f ci/lava: Fix imports formatting
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25912>
2023-11-02 03:31:50 +00:00
Guilherme Gallo
f7f2d26e3b ci/lava: Use project_name instead of hardcoded mesa
The LAVA job submitter is being used by other fd.o projects, such as
`drm/ci`, so let's make it generate more generic job definitions and
test cases.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25912>
2023-11-02 03:31:49 +00:00
David Heidelberg
3a4bdf26e6 ci: remove LAVA prefix from variables which can be used also elsewhere
At least these two can be easily used in bare-metal or Labgrid setups.

Currently I already have MR for implementing these for Labgrid.

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24665>
2023-08-17 13:25:46 +00:00
David Heidelberg
2af371b3f3 ci: replace MINIO_RESULTS_UPLOAD with S3_RESULTS_UPLOAD
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23527>
2023-06-10 01:31:16 +02:00
Guilherme Gallo
ebdf8a95b7 ci/lava: Force LAVA panfrost jobs to use UART
To ensure proper SSH functioning, the device IP should be added to the
LAVA device dictionary by setting device_ip. LAVA will then map the
value to lava-target-ip.

meson-g12b-a311d-khadas-vim3-cbg-4 has an IP in the dictionary, while
sun50i-h6-pine-h64-cbg-1 and meson-g12b-a311d-khadas-vim3-cbg-2 do not.

Since some devices are not yet properly configured, and device tag
fixing is not an option here, let's temporarily switch to a job
definition based on UART, until it gets fixed.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
2023-05-19 14:45:17 +00:00
Guilherme Gallo
4173e4b18f ci/lava: Hide JWT block during YAML dump
Make hide_sensitive_data work in a block fashion, not only hiding the
JWT line, since these tokens are huge, it may break the line when it
extrapolates the YAML dump width.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
2023-05-19 14:45:17 +00:00
Guilherme Gallo
8626a52637 ci/lava: Add bridge function for job definition
To use the supported job definition depending on some Mesa CI job
characteristics.

The strategy here, is to use LAVA with a containerized SSH session to
follow the job output, escaping from dumping data to the UART, which
proves to be error prone in some devices.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
2023-05-19 14:45:17 +00:00
David Heidelberg
cc6f2589ec ci/lava: add support for HDK 888 firmware
Acked-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22604>
2023-04-25 20:41:50 +00:00
David Heidelberg
5dd68b6ba6 ci/lava: implement fastboot support
Based on work from Tomeu Vizoso.

Acked-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22604>
2023-04-25 20:41:50 +00:00
Guilherme Gallo
a893b06691 ci/lava: Use f-strings in job definition
f-strings are little more concise than str.format() approach.
It makes the code a little more readable as well.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
2023-04-19 14:36:37 +00:00
Guilherme Gallo
cfe644a9e5 ci/lava: Use python-fire in job submitter
Cleanup argparse to use dataclasses+python-fire to give easier
maintenance to job submitter.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
2023-04-19 14:36:37 +00:00
Guilherme Gallo
6f6b892dca ci/lava: Move job definition stuff to another file
The LAVA job submitter is too big, let's reorganize it a little.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
2023-04-19 14:36:37 +00:00