The boot action was wrapping the deploy action, which could cause
timeout misalignment. For example, the boot `GitlabSection` timeout was
shorter than the deploy timeout in LAVA, leading to cases where LAVA
jobs were canceled during their own retry mechanism.
By splitting these actions, we can align the timeouts properly,
preventing interference and unnecessary job cancellations.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33906>
python-fire auto-converts `item1,item2` into a tuple, but if there is a
dash `-` inside the argument, it treats it as a string.
Let's validate the data, when it comes as a `str` or `tuple`.
For more details, here are the tested scenarios:
| --lava-tags= | Type | Value |
|--------------|-------|---------------------|
| None | bool | True |
| '' | str | '' |
| tag1 | str | "tag1" |
| tag1, | tuple | ("tag1",) |
| tag-1,tag-2 | str | 'tag-1,tag-2' |
| tag1,tag2 | tuple | ("tag1", "tag2") |
| ',' | str | ',' |
| ',,' | str | ',,' |
| 'tag1,,' | str | 'tag1,,' |
See also:
https://google.github.io/python-fire/guide/#argument-parsing
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31882>
Improves the error logging in the LAVA job submitter by capturing and
logging the exception message rather than just the exception type when a
job fails to run.
Additionally, introduces a clearer script interruption
message to aid in debugging and immediate understanding of job
submission failures.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28778>
This commit refactors the exception hierarchy to differentiate between
retriable and fatal errors in the CI pipeline, specifically within the
LAVA job submission process.
A new base class, `MesaCIRetriableException`, is introduced for
exceptions that should trigger a retry of the CI job, while
`MesaCIFatalException` is added for non-recoverable errors that halt the
process immediately.
Additionally, the logic for deciding whether a job should be retried or
not is updated to check for instances of `MesaCIRetriableException`,
improving the robustness and reliability of the CI job execution
strategy.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/28778>
We need update kernel often. We need test kernel changes often.
Introduced `KERNEL_EXTERNAL_TAG` to differ between `KERNEL_TAG` which is
also used to rebuild the containers. We don't need rebuild containers
for the external kernel, so this way we don't have to.
Updating kernel goes wruuuuuum.
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23563>
To use the supported job definition depending on some Mesa CI job
characteristics.
The strategy here, is to use LAVA with a containerized SSH session to
follow the job output, escaping from dumping data to the UART, which
proves to be error prone in some devices.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22870>
Refactor some pieces of the submitter to improve the clarity of the
functions and create a simple dictionary with aggregated data from the
submitter execution which will be dumped to a file when the script
exits.
Add support for the AutoSaveDict based structured logger as well, which
will come in a follow-up commit.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
Update the LogFollower class to improve section handling and provide a
history of sections encountered during log processing:
1. Add section_history attribute to store the history of encountered
GitlabSections.
2. Make LAVA job submitter use the section history feature to improve
structural logging.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
Let's make lava_job_submitter.py cleaner with only parsing and retry
mechanism capabilities.
Moved out from the submitter script:
1. proxy functions
- moved to lava.utils.lava_proxy.py
2. LAVAJob class definition
- moved to lava.utils.lava_job.py
- added structural logging capabilities into LAVAJob
- Implemented properties for job_id, is_finished, and status, with
corresponding setter methods that update the log dictionary.
- Added new methods show, get_lava_time, and refresh_log for improved
log handling and data retrieval.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22500>
Better error handling is more reliable.
Options:
-L, follow location
--retry, number of retries
--retry-all-errors, does not fail on ALL errors, that's why there is -f
-f, fail fast with no output at all on server errors
--retry-delay, make curl sleep this amount of time before each retry
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20788>
To debug the LAVA jobs locally, we have an option in the
lava_job_submitter script to ignore the JWT token to make it possible to
retry jobs without the need to get an unexpired token.
But this trick needs to modify the overlayed directory so that we would
need to download and extract it earlier in the run.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
LAVA uses XMLRPC to send jobs information and control, more specifically
it sends device logs via YAML dumps encoded in UTF-8 bytes.
In Python, we have xmlrpc.client.Binary class as the serializer
protocol, we get the logs wrapped by this class, which encodes the data
as UTF-8 bytes data.
We were converting the encoded data to a string via the `str` function,
but this led the loaded YAML data to use single quotes instead of double
quotes for string values that made special characters, such as `\x1b` to
be escaped as `\\x1b`.
With this fix, we can now drop one of the hacks that fixed the bash
colors.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20051>
