The job definitions for lava-related jobs are encapsulated in a directory,
while the other two farm managers were in the generic test directory. Having a
directory for ci-tron places it side by side with other farm managers. For
bare-metal, it has another advantage, as this encapsulates elements related to
this farm manager in a single place.
To maintain simplicity and consistency in file naming, the gitlab-ci file in
the lava directory is renamed as it has a prefix that corresponds with the
directory hosting it. The other farm equivalent files don't include this
duplication as a prefix, and there isn't such a prefix in any other case of
the CI.
Signed-off-by: Sergi Blanch-Torne <sergi.blanch.torne@collabora.com>
Acked-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35427>
Oops. We were being a bit too extensive with our rules and also skipping
the container stage for user (non-marge) pipelines in MRs unless
image-tags.yml had changed. We can't actually do this, because we might
still need to run the jobs to copy the containers into the user's
namespace, even if we don't need to rebuild them.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35491>
For pre-merge pipelines, we know that we don't need to run the
container-check jobs unless image-tags.yml has been modified. In this
case, we can just skip the container jobs, because we 'know' that the
containers are present in the upstream repo.
For downstream or off-main pipelines, we still need to run the check
job, because the changed-file tracking is not reliable on other
branches, and because we may need to copy the container image from the
upstream container storage into the user's namespace.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35203>
Compared to the existing Debian-based x86_64_pyutils container, this
Alpine-based variant reduces the image size by approximately 83%.
Include all the necessary python artifacts, including lava_job_submitter
in the container to avoid having to download them at the start of each
test job.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34980>
Firmware packages continue to grow in size, so stop installing them in
the test-base image.
The necessary firmware is now collected and uploaded per vendor in an
external repository.
LAVA devices can opt into optional firmware by specifying the name of the
archive via LAVA_FIRMWARE.
For bare-metal, Qualcomm firmware required for DUTs in the Google lab is
included in the baremetal image.
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/13051
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34861>
This whole thing of dumping the env var in a file and unsetting it so
that it wouldn't be visible in an env dump anymore? Yeah, we kinda
failed here 😅
Note: setting it to an empty string instead of unsetting it allows for
redundant `eval "$S3_JWT_FILE_SCRIPT"` calls without failing over an
unset variable.
Reported-by: @alatiera
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35050>
The EXTERNAL_KERNEL_TAG variable is no longer needed.
For LAVA and bare-metal, we can override the KERNEL_TAG variable to fetch
both the kernel image and modules from a different tag than the default
mainline gfx-ci/linux kernel.
For LAVA, this also avoids the issue where jobs using EXTERNAL_KERNEL_TAG
would still have mainline kernel modules downloaded by the LAVA overlay.
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34873>
Up until now, every project using CI-Tron had to write their own job
submission flow because CI-Tron itself was not providing any official
way of interacting with it via GitLab.
This however changed, and the solution is vastly superior to what we
have been using in Mesa:
* Ability to pass all the environment variables of the job to the DUT,
so no need to remember to add variables in
`export-gitlab-job-env-for-dut.sh` anymore
* No dependency on Mesa code, which means no need to wait on
python-artifacts and the ability to replicate a run by just copying
the job description outputted by the job \o/
* Ability to have as many initrd, HTTP, and TFTP artifacts as wanted
* Ability to expose a variable through a TFTP/HTTP endpoint or as an
initrd
* Ability to overwrite the platform environment (machine-specific FW)
* Ability to have as many kernel cmdline variables, all merged when
generating the final cmdline. This makes it easy to share some
snippets of cmdline between jobs
Transitioning from the custom to the generic template is however pretty
involved. This commit does the minimum changes needed to switch to the
new model, often simply replacing the B2C_ prefix with CI_TRON_.
Further renaming of "b2c" prefixes into "ci-tron" is left for future
commits.
Co-authored-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34795>
These stages are for the jobs that are skipped in merge pipelines,
automatically run in nightly pipelines, and are available to run
manually in other pipelines.
None of these ever run in post-merge pipelines.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34590>
Note that this code is executed on the generic FDo gitlab runners, not
in our docker images. This change is merely to avoid the confusion that
lead to the code in the previous commit.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34684>
Right now, `deploy-docs` (updating the docs.mesa3d.org website) is the
only job that can exist in that pipeline (along with `alpine/x86_64_build`,
but that job is always a no-op).
Right now this job can get stuck for a long time waiting for a runner,
but it's very short, and we kinda want the website to be updated quickly.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34554>
Enforce a default job timeout of 1 second, to make jobs which don't
explicitly specify a timeout insta-fail, rather than potentially hanging
around for an hour.
Container builds get the full hour as they can run long and are not run
in pre-merge context, and LAVA jobs also get the full hour as they have
multiple internal timeout mechanisms which aim to fast-fail jobs once
they actually start. However, as they just queue jobs to an external
host (shared with other projects like KernelCI), these timeouts aren't
reflected into the GitLab CI definitions.
Signed-off-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34280>
This affects:
- generic jobs (sanity, rustfmt, shader-db, docs, etc.)
- linux image builds
- linux mesa builds
- software renderer tests
- android tests
- virgl & venus tests
Marge pipelines get high priority, nightly pipelines get low priority,
and everything else is in between.
(Hardware test farms have their own mechanisms.)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34264>
That tag was supposed to allow these jobs to run faster, but these
runners are currently having disk issues, and the normal runners look
like they're plenty fast enough (at least right now since almost nobody
runs ci jobs ^^).
We might revert this later, but for now let's merge this to unblock CI.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120>
The current `s3cp` implementation does not work anymore after the
migration, and instead of fixing it and propagating the fix down to us,
it's simpler to directly use `curl`.
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/34120>
What we need is a way to tell GitLab "queue `build-only` jobs after
`build-for-tests` jobs have started", to make sure that `build-only`
jobs don't start before `build-for-tests` jobs and thus delays test jobs
and the overall pipeline.
The best I had found was "queue `build-only` jobs after *all* the
`build-for-tests` jobs have finished", but this introduces a larger
delay than we want, and causes `build-only` jobs to often be the last
ones to finish in a pipeline, after test jobs that respect the 15min
runtime limit.
Instead, we can tell GitLab "queue `build-only` jobs after the
`build-for-tests` jobs have been queued for X minutes", which is closer
to what we want, and in particular this ensures the correct order of
*starting* jobs as long as the CI is not overwhelmed and doesn't manage
to actually start a queued `build-for-tests` job within 5min, in which
case I'd argue we don't care about job order anymore because we have
bigger problems anyway and likely everything's going to timeout.
This also gets rid of the hard-to-maintain `.build-for-tests-jobs` list
of `needs:`, which also needed to be manually merged in half the jobs.
The trade-off is that we need to make a (shallow) copy of the
`.container+build-rules` list, that replaces all the `when: on_success`
with `when: delayed` + `start_in: 5 minutes`. This means that we'll need
to make sure the two lists of conditions remain identical, but this
seems more manageable; nevertheless, I added a comment to remind us.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33564>
The release branch contains only what was on the staging branch first,
so testing it again is a waste of resources.
To do this, we split the rule into specifically "default branch" and
"staging branch", and "release branch" gets dropped by virtue of no
longer being caught by any rule.
Cc: mesa-stable
Reviewed-by: Martin Roukala <None>
Reviewed-by: Dylan Baker <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33411>
It's too late to run all the tests by then, the release has been made
based on the staging pipelines results
Cc: mesa-stable
Reviewed-by: Martin Roukala <None>
Reviewed-by: Dylan Baker <None>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/33409>
In order to build the mesa llvmpipe driver for Android, LLVM libraries
for Android are needed, but building them each time a new upstream mesa
is built would be too much overhead.
So add a script to build LLVM libraries for Android and call it from the
debian/android_build job. The actual build will be performed only when
necessary and the resulting artifact will be uploaded to the S3 bucket
to be re-used when building mesa for Android.
The build process follows what has been documented in
docs/drivers/llvmpipe.rst
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/31093>
