mirror of https://github.com/apache/superset.git
feat(ci): no more docker builds on PR-related events (#27146)
This commit is contained in:
parent
127df24c08
commit
f1cd8cc263
|
@ -73,14 +73,6 @@ github:
|
||||||
- test-postgres (3.9)
|
- test-postgres (3.9)
|
||||||
- test-postgres (3.10)
|
- test-postgres (3.10)
|
||||||
- test-sqlite (3.9)
|
- test-sqlite (3.9)
|
||||||
- docker-build (dev, linux/amd64)
|
|
||||||
- docker-build (lean, linux/amd64)
|
|
||||||
- docker-build (py310, linux/arm64)
|
|
||||||
- docker-build (py310, linux/amd64)
|
|
||||||
- docker-build (websocket, linux/arm64)
|
|
||||||
- docker-build (websocket, linux/amd64)
|
|
||||||
- docker-build (dockerize, linux/arm64)
|
|
||||||
- docker-build (dockerize, linux/amd64)
|
|
||||||
|
|
||||||
required_pull_request_reviews:
|
required_pull_request_reviews:
|
||||||
dismiss_stale_reviews: false
|
dismiss_stale_reviews: false
|
||||||
|
|
|
@ -5,7 +5,8 @@ on:
|
||||||
branches:
|
branches:
|
||||||
- 'master'
|
- 'master'
|
||||||
pull_request:
|
pull_request:
|
||||||
types: [synchronize, opened, reopened, ready_for_review]
|
branches:
|
||||||
|
- 'master'
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
|
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
|
||||||
|
@ -17,15 +18,8 @@ jobs:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
strategy:
|
strategy:
|
||||||
matrix:
|
matrix:
|
||||||
build_preset: ["dev", "lean", "py310", "websocket", "dockerize"]
|
build_preset: |
|
||||||
platform: ["linux/amd64", "linux/arm64"]
|
${{ fromJson(github.event_name == 'pull_request' ? '["ci"]' : '["dev", "lean", "py310", "websocket", "dockerize"]') }}
|
||||||
exclude:
|
|
||||||
# disabling because slow! no python wheels for arm/py39 and
|
|
||||||
# QEMU is slow!
|
|
||||||
- build_preset: "dev"
|
|
||||||
platform: "linux/arm64"
|
|
||||||
- build_preset: "lean"
|
|
||||||
platform: "linux/arm64"
|
|
||||||
fail-fast: false
|
fail-fast: false
|
||||||
steps:
|
steps:
|
||||||
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
|
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
|
||||||
|
@ -46,4 +40,9 @@ jobs:
|
||||||
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
|
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||||
run: |
|
run: |
|
||||||
pip install click
|
pip install click
|
||||||
./scripts/build_docker.py ${{ matrix.build_preset }} ${{ github.event_name }} --platform ${{ matrix.platform }}
|
./scripts/build_docker.py \
|
||||||
|
${{ matrix.build_preset }} \
|
||||||
|
${{ github.event_name }} \
|
||||||
|
--build_context_ref "$RELEASE" $FORCE_LATEST \
|
||||||
|
--platform "linux/arm64" \
|
||||||
|
--platform "linux/amd64"
|
||||||
|
|
|
@ -19,7 +19,7 @@
|
||||||
# - Ensure that the job names in this workflow match exactly the names of the corresponding jobs in the main workflows.
|
# - Ensure that the job names in this workflow match exactly the names of the corresponding jobs in the main workflows.
|
||||||
# - This workflow should be kept as-is, without path-specific conditions.
|
# - This workflow should be kept as-is, without path-specific conditions.
|
||||||
|
|
||||||
name: No Operation Checks
|
name: no-op Checks
|
||||||
on: pull_request
|
on: pull_request
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
|
@ -52,3 +52,20 @@ jobs:
|
||||||
run: |
|
run: |
|
||||||
echo "This is a no-op step for python-lint to ensure a successful status."
|
echo "This is a no-op step for python-lint to ensure a successful status."
|
||||||
exit 0
|
exit 0
|
||||||
|
# section to be deleted after master merge
|
||||||
|
docker-build:
|
||||||
|
strategy:
|
||||||
|
matrix:
|
||||||
|
build_preset: ["dev", "lean", "py310", "websocket", "dockerize"]
|
||||||
|
platform: ["linux/amd64", "linux/arm64"]
|
||||||
|
exclude:
|
||||||
|
- build_preset: "dev"
|
||||||
|
platform: "linux/arm64"
|
||||||
|
- build_preset: "lean"
|
||||||
|
platform: "linux/arm64"
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: No-op for docker
|
||||||
|
run: |
|
||||||
|
echo "No-op to get rid of the docker-build checks reqs in .asf.yml"
|
||||||
|
exit 0
|
||||||
|
|
|
@ -1,47 +1,47 @@
|
||||||
# Docker Images and Tags
|
# Docker builds, images and tags
|
||||||
|
|
||||||
The Apache Superset community extensively uses Docker for development, release,
|
The Apache Superset community extensively uses Docker for development, release,
|
||||||
and productionizing Superset. This page details our Docker builds and tag naming
|
and productionizing Superset. This page details our Docker builds and tag naming
|
||||||
schemes to help users navigate our offerings.
|
schemes to help users navigate our offerings.
|
||||||
|
|
||||||
Images are built and pushed to the [Superset Docker Hub repository](
|
Images are built and pushed to the [Superset Docker Hub repository](
|
||||||
https://hub.docker.com/r/apache/superset). Different sets of images are created for:
|
https://hub.docker.com/r/apache/superset) using GitHub Actions.
|
||||||
|
Different sets of images are built and/or published at different times:
|
||||||
|
|
||||||
- **Published releases** (`release`): with tags like `3.0.0` and the `latest` tag.
|
- **Published releases** (`release`): published using
|
||||||
Those are multi-platform (arm+amd). More on that later.
|
tags like `3.0.0` and the `latest` tag.
|
||||||
- **Pull request iterations** (`pull_request`):, each identified by tags starting with a SHA like
|
- **Pull request iterations** (`pull_request`): for each pull request, while
|
||||||
`8a2f7d378ab13c156fa183d9284b607ed69f5ecc`, and `pr-3454`, referencing the pull
|
we actively build the docker to validate the build, we do
|
||||||
request ID.
|
not publish those images for security reasons, we simply `docker build --load`
|
||||||
- **Merges to the main branch** (`push`): resulting in new SHAs, with tags
|
- **Merges to the main branch** (`push`): resulting in new SHAs, with tags
|
||||||
prefixed with `master` for the latest `master` version.
|
prefixed with `master` for the latest `master` version.
|
||||||
|
|
||||||
Each CI build run has multiple builds for different purposes, identified by suffixes:
|
# Build presets
|
||||||
- **Build preset:** We offer various images for different needs:
|
|
||||||
|
We have a set of build "presets" that each represent a combination of
|
||||||
|
parameters for the build, mostly pointing to either different target layer
|
||||||
|
for the build, and/or base image.
|
||||||
|
|
||||||
|
Here are the build presets that are exposed through the `build_docker.py` script:
|
||||||
- `lean`: The default Docker image, including both frontend and backend. Tags
|
- `lean`: The default Docker image, including both frontend and backend. Tags
|
||||||
without a build_preset are lean builds, e.g., `latest`.
|
without a build_preset are lean builds, e.g., `latest`.
|
||||||
- `dev`: For development, with a headless browser and root access.
|
- `dev`: For development, with a headless browser, dev-related utilities and root access.
|
||||||
- `py310`, e.g., Py310: Similar to lean but with a different Python version (in this example, 3.10).
|
- `py310`, e.g., Py310: Similar to lean but with a different Python version (in this example, 3.10).
|
||||||
- `ci`: For certain CI workloads.
|
- `ci`: For certain CI workloads.
|
||||||
- `websocket`: For Superset clusters supporting advanced features.
|
- `websocket`: For Superset clusters supporting advanced features.
|
||||||
- `dockerize`: Used by Helm.
|
- `dockerize`: Used by Helm.
|
||||||
- **Platform:** We build for `linux/arm64` and `linux/amd64`. The `-arm` suffix
|
|
||||||
indicates ARM builds (e.g., `latest-arm`), while tags without a suffix are for
|
|
||||||
AMD (e.g., `latest`).
|
|
||||||
|
|
||||||
## Key Image Tags and Examples
|
## Key tags examples
|
||||||
|
|
||||||
- `latest`: The latest official release build, implicitly the lean build on
|
- `latest`: The latest official release build
|
||||||
`linux/amd64`.
|
|
||||||
- `latest-dev`: the `-dev` image of the latest official release build, with a
|
- `latest-dev`: the `-dev` image of the latest official release build, with a
|
||||||
headless browser and root access.
|
headless browser and root access.
|
||||||
- `master`: The latest build from the `master` branch, implicitly lean on
|
- `master`: The latest build from the `master` branch, implicitly the lean build
|
||||||
`linux/amd64`.
|
preset
|
||||||
- `master-dev`: Similar to `master` but includes a headless browser and root access.
|
- `master-dev`: Similar to `master` but includes a headless browser and root access.
|
||||||
- `pr-5252`: The latest commit in PR 5252.
|
- `pr-5252`: The latest commit in PR 5252.
|
||||||
- `30948dc401b40982cb7c0dbf6ebbe443b2748c1b-dev-arm`: A `linux/arm64` build for
|
- `30948dc401b40982cb7c0dbf6ebbe443b2748c1b-dev`: A build for
|
||||||
this specific SHA, which could be from a pull request, master merge, or release.
|
this specific SHA, which could be from a `master` merge, or release.
|
||||||
- `30948dc-dev-arm`: Same as above, but SHA truncated to 7 characters for a
|
|
||||||
shorter handle on the same image
|
|
||||||
- `websocket-latest`: The WebSocket image for use in a Superset cluster.
|
- `websocket-latest`: The WebSocket image for use in a Superset cluster.
|
||||||
|
|
||||||
For insights or modifications to the build matrix and tagging conventions,
|
For insights or modifications to the build matrix and tagging conventions,
|
||||||
|
@ -64,9 +64,9 @@ build times, larger images, lower layer cache hit rate, ...).
|
||||||
For production use cases, we recommend that you derive our `lean` image(s) and
|
For production use cases, we recommend that you derive our `lean` image(s) and
|
||||||
add database support for the database you need.
|
add database support for the database you need.
|
||||||
|
|
||||||
## On supporting arm64 AND amd64
|
## On supporting different platforms (namely arm64 AND amd64)
|
||||||
|
|
||||||
Only the release builds are multi-platform, supporting `linux/arm64`
|
Currently all automated builds are multi-platform, supporting both `linux/arm64`
|
||||||
and `linux/amd64`. This enables higher level constructs like `helm` and
|
and `linux/amd64`. This enables higher level constructs like `helm` and
|
||||||
docker-compose to point to these images and effectively be multi-platform
|
docker-compose to point to these images and effectively be multi-platform
|
||||||
as well.
|
as well.
|
||||||
|
|
Loading…
Reference in New Issue