Commit Graph

112 Commits

Author SHA1 Message Date
Guilherme Gallo
daff21ef55 ci/lava: Make hung job status yellow
It will help to know what happened to a non-successful job.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
2022-07-08 12:26:05 +00:00
Guilherme Gallo
2c51b7a9c9 ci/lava: Detect R8152 issues preemptively and retry
Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:

```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```

Where <IP> and <USB> could be any IP and USB addresses, respectfully.

This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.

Closes: #6681

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
2022-07-08 12:26:05 +00:00
Guilherme Gallo
45a4b01427 ci/lava: Split lava_log into modules
This script is getting too big, it been hard to extend it.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>
2022-07-08 12:26:05 +00:00
Guilherme Gallo
20827dfa9b ci/lava: Update license header
Use SPDX to indicate the license.
Update authors of lava_job_submitter.py

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
3f3b0a28f2 ci/lava: Rename console color names
Use FG to indicate foreground colors

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
6ba2b33a8c ci/lava: Flexibilize section marker regexes
In some jobs, such as
https://gitlab.freedesktop.org/gallo/mesa/-/jobs/24904100, the kmsg is
interleaved with stderr/stdout in serial console, making it difficult to
confidently find the log messages to detect when the DUT is booting,
when the DUT is running etc.

Luckily, LAVA sends redundant messages about their signals. We can use
them to mitigate the chance of missing an interleaved message by being
more open to different messages, using the regex on both `debug` and
`target` LAVA log levels.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
868daded22 ci/lava: Highlight kernel messages in bold
This will serve to warn the user that those messages are processed
differently, e.g. the kmsgs does not trigger heartbeats and maybe
eventual targets of hint to retry the job immediately.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
29af421272 ci/lava: Don't print LAVA debug messages
Remove debug messages from the output in order to unclutter the log a
little more for the developers.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
02cac49721 ci/lava: heartbeat: don't consider kernel message logs
Currently, the submitter consider that every new log that comes from the
DUT console is a signal that the device is healthy, but maybe that is
not the case, since in some kernel hangs/failures, no output is
presented except from some kernel messages.

This commit bypass the heartbeat when the LogFollower detect a kernel
message. Any log line that does follow the kmsg pattern will make the
job labeled as healthy again.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
aa26a6ab72 ci/lava: Follow job execution via LogFollower
Now LogFollower is used to deal with the LAVA logs.

Moreover, this commit adds timeouts per Gitlab section, if a section
takes longer than expected, cancel the job and retry again.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
2569d7d7df ci/lava: Create LogFollower and move logging methods
- Create LogFollower to capture LAVA log and process it adding some
- GitlabSection and color treatment to it
- Break logs further, make new gitlab sections between testcases
- Implement LogFollower as ContextManager to deal with incomplete LAVA
  jobs.
- Use template method to simplify gitlab log sections management
- Fix sections timestamps

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00
Guilherme Gallo
c86ba3640f ci/lava: Create Gitlab log sections handler
Gitlab has support for collapsible sections, so it would be good to
create collapsed log sections for the LAVA setup logs. This way, the
Mesa developers to see only the execution of the scripts, instead of
LAVA messages clutter.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16323>
2022-07-07 00:28:53 +00:00