2019-01-22 11:21:13 -05:00
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
2018-09-19 13:01:51 -04:00
2019-03-25 19:21:26 -04:00
# Apache Releases
2018-09-19 13:01:51 -04:00
2019-10-30 13:24:18 -04:00
Until things settle and we create scripts that streamline this,
you'll probably want to run these commands manually and understand what
2019-03-25 19:21:26 -04:00
they do prior to doing so.
2018-09-19 13:01:51 -04:00
2019-10-30 13:24:18 -04:00
For coordinating on releases, on operational topics that require more
synchronous communications, we recommend using the `#apache-releases` channel
2019-10-23 00:42:12 -04:00
on the Superset Slack. People crafting releases and those interested in
partaking in the process should join the channel.
2021-01-20 21:18:21 -05:00
## Release notes for recent releases
2021-08-30 02:06:24 -04:00
2022-04-07 05:52:20 -04:00
- [1.5 ](release-notes-1-5/README.md )
2022-01-28 08:11:19 -05:00
- [1.4 ](release-notes-1-4/README.md )
2021-08-30 02:06:24 -04:00
- [1.3 ](release-notes-1-3/README.md )
2021-06-27 16:29:08 -04:00
- [1.2 ](release-notes-1-2/README.md )
- [1.1 ](release-notes-1-1/README.md )
- [1.0 ](release-notes-1-0/README.md )
2021-01-20 21:18:21 -05:00
- [0.38 ](release-notes-0-38/README.md )
2019-10-23 09:43:04 -04:00
## Release setup (First Time Only)
2019-04-20 13:26:01 -04:00
2019-03-25 19:21:26 -04:00
First you need to setup a few things. This is a one-off and doesn't
need to be done at every release.
2018-09-19 13:01:51 -04:00
```bash
2019-07-04 01:33:14 -04:00
# Create PGP Key, and use your @apache .org email address
2019-03-25 19:21:26 -04:00
gpg --gen-key
2019-08-12 17:01:15 -04:00
2019-03-25 19:21:26 -04:00
# Checkout ASF dist repo
2019-04-15 15:48:51 -04:00
2020-12-18 07:04:33 -05:00
svn checkout https://dist.apache.org/repos/dist/dev/superset/ ~/svn/superset_dev
2019-04-15 15:48:51 -04:00
2020-12-18 07:04:33 -05:00
svn checkout https://dist.apache.org/repos/dist/release/superset/ ~/svn/superset
2019-03-25 19:21:26 -04:00
cd ~/svn/superset
2019-08-12 17:01:15 -04:00
2019-03-25 19:21:26 -04:00
# Add your GPG pub key to KEYS file. Replace "Maxime Beauchemin" with your name
2019-10-28 06:17:10 -04:00
export SUPERSET_PGP_FULLNAME="Maxime Beauchemin"
(gpg --list-sigs "${SUPERSET_PGP_FULLNAME}" & & gpg --armor --export "${SUPERSET_PGP_FULLNAME}" ) >> KEYS
2019-08-12 17:01:15 -04:00
2019-03-25 19:21:26 -04:00
# Commit the changes
svn commit -m "Add PGP keys of new Superset committer"
2021-06-20 12:35:05 -04:00
# push the changes
svn update
2018-09-19 13:01:51 -04:00
```
2022-05-27 09:09:12 -04:00
To minimize the risk of mixing up your local development environment, it's recommended to work on the
release in a different directory than where the devenv is located. In this example, we'll clone
the repo directly from the main `apache/superset` repo to a new directory `superset-release` :
```bash
cd < MY PROJECTS PATH >
git clone git@github.com:apache/superset.git superset-release
cd superset-release
```
We recommend setting up a virtual environment to isolate the python dependencies from your main
setup:
```bash
virtualenv venv
source venv/bin/activate
```
In addition, we recommend using the [`cherrytree` ](https://pypi.org/project/cherrytree/ ) tool for
automating cherry picking, as it will help speed up the release process. To install `cherrytree`
and other dependencies that are required for the release process, run the following commands:
```bash
pip install -r RELEASING/requirements.txt
```
2020-02-18 00:14:31 -05:00
## Setting up the release environment (do every time)
2021-01-21 10:52:48 -05:00
As the vote process takes a minimum of 72h, sometimes stretching over several weeks
of calendar time if votes don't pass, chances are
2020-02-18 00:14:31 -05:00
the same terminal session won't be used for crafting the release candidate and the
final release. Therefore, it's a good idea to do the following every time you
work on a new phase of the release process to make sure you aren't releasing
the wrong files/using wrong names. There's a script to help you set correctly all the
necessary environment variables. Change your current directory to `superset/RELEASING`
2020-07-11 12:00:02 -04:00
and execute the `set_release_env.sh` script with the relevant parameters:
2020-02-18 00:14:31 -05:00
2022-05-27 09:09:12 -04:00
Usage (MacOS/ZSH):
2022-06-02 09:07:57 -04:00
2020-02-18 00:14:31 -05:00
```bash
2022-05-27 09:09:12 -04:00
cd RELEASING
source set_release_env.sh < SUPERSET_RC_VERSION > < PGP_KEY_FULLNAME >
2020-10-28 14:17:33 -04:00
```
2022-05-27 09:09:12 -04:00
Usage (BASH):
2022-06-02 09:07:57 -04:00
2020-10-28 14:17:33 -04:00
```bash
2022-05-27 09:09:12 -04:00
. set_release_env.sh < SUPERSET_RC_VERSION > < PGP_KEY_FULLNAME >
2020-10-28 14:17:33 -04:00
```
Example:
2022-06-02 09:07:57 -04:00
2020-10-28 14:17:33 -04:00
```bash
2022-05-27 09:09:12 -04:00
source set_release_env.sh 1.5.1rc1 myid@apache.org
2020-02-18 00:14:31 -05:00
```
2022-05-27 09:09:12 -04:00
The script will output the exported variables. Here's example for 1.5.1rc1:
2020-02-18 00:14:31 -05:00
```
2022-05-27 09:09:12 -04:00
-------------------------------
2021-01-21 10:52:48 -05:00
Set Release env variables
2022-05-27 09:09:12 -04:00
SUPERSET_VERSION=1.5.1
2021-01-21 10:52:48 -05:00
SUPERSET_RC=1
2022-05-27 09:09:12 -04:00
SUPERSET_GITHUB_BRANCH=1.5
SUPERSET_PGP_FULLNAME=villebro@apache.org
SUPERSET_VERSION_RC=1.5.1rc1
SUPERSET_RELEASE=apache-superset-1.5.1
SUPERSET_RELEASE_RC=apache-superset-1.5.1rc1
SUPERSET_RELEASE_TARBALL=apache-superset-1.5.1-source.tar.gz
SUPERSET_RELEASE_RC_TARBALL=apache-superset-1.5.1rc1-source.tar.gz
SUPERSET_TMP_ASF_SITE_PATH=/tmp/incubator-superset-site-1.5.1
-------------------------------
2020-02-18 00:14:31 -05:00
```
2019-10-28 06:17:10 -04:00
## Crafting a source release
2020-02-09 20:53:56 -05:00
When crafting a new minor or major release we create
2020-07-11 12:00:02 -04:00
a branch named with the release MAJOR.MINOR version (on this example 0.37).
2020-02-09 20:53:56 -05:00
This new branch will hold all PATCH and release candidates
2019-10-28 06:17:10 -04:00
that belong to the MAJOR.MINOR version.
2022-05-27 09:09:12 -04:00
### Creating an initial minor release (e.g. 1.5.0)
2019-10-28 06:17:10 -04:00
The MAJOR.MINOR branch is normally a "cut" from a specific point in time from the master branch.
2022-05-27 09:09:12 -04:00
When creating the initial minor release (e.g. 1.5.0), create a new branch:
2022-06-02 09:07:57 -04:00
2022-05-27 09:09:12 -04:00
```bash
git checkout master
git pull
git checkout -b ${SUPERSET_GITHUB_BRANCH}
git push origin $SUPERSET_GITHUB_BRANCH
```
Note that this initializes a new "release cut", and is NOT needed when creating a patch release
(e.g. 1.5.1).
### Creating a patch release (e.g. 1.5.1)
When getting ready to bake a patch release, simply checkout the relevant branch:
2019-10-28 06:17:10 -04:00
2020-07-11 12:00:02 -04:00
```bash
2022-05-27 09:09:12 -04:00
git checkout master
git pull
git checkout ${SUPERSET_GITHUB_BRANCH}
2020-07-11 12:00:02 -04:00
```
2022-05-27 09:09:12 -04:00
### Cherry picking
It is customary to label PRs that have been introduced after the cut with the label
`v<MAJOR>.<MINOR>` . For example, for any PRs that should be included in the 1.5 branch, the
label `v1.5` should be added.
To see how well the labelled PRs would apply to the current branch, run the following command:
```bash
cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} ${SUPERSET_GITHUB_BRANCH}
```
This requires the presence of an environment variable `GITHUB_TOKEN` . Alternatively,
you can pass the token directly via the `--access-token` parameter (`-at` for short).
#### Happy path: no conflicts
This will show how many cherries will apply cleanly. If there are no conflicts, you can simply apply all cherries
by adding the `--no-dry-run` flag (`-nd` for short):
```bash
cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd ${SUPERSET_GITHUB_BRANCH}
```
#### Resolving conflicts
If there are conflicts, you can issue the following command to apply all cherries up until the conflict automatically, and then
break by adding the `-error-mode break` flag (`-e break` for short):
```bash
cherrytree bake -r apache/superset -m master -l v${SUPERSET_GITHUB_BRANCH} -nd -e break ${SUPERSET_GITHUB_BRANCH}
```
After applying the cleanly merged cherries, `cherrytree` will specify the SHA of the conflicted cherry. To resolve the conflict,
simply issue the following command:
```bash
git cherry-pick < SHA >
```
Then fix all conflicts, followed by
```bash
git add -u # add all changes
git cherry-pick --continue
```
After this, rerun all the above steps until all cherries have been picked, finally pushing all new commits to the release branch
on the main repo:
```bash
git push
```
### Updating changelog
2020-07-11 12:00:02 -04:00
Next, update the `CHANGELOG.md` with all the changes that are included in the release.
2022-05-27 09:09:12 -04:00
Make sure the branch has been pushed to `origin` to ensure the changelog generator
2020-10-30 03:50:54 -04:00
can pick up changes since the previous release.
2022-05-27 09:09:12 -04:00
Similar to `cherrytree` , the change log script requires a github token, either as an env var
(`GITHUB_TOKEN`) or as the parameter `--access_token` .
#### Initial release (e.g. 1.5.0)
When generating the changelog for an initial minor relese, you should compare with
the previous release (in the example, the previous release branch is `1.4` , so remember to
update it accordingly):
2019-11-19 10:44:38 -05:00
```bash
2022-05-27 09:09:12 -04:00
python changelog.py --previous_version 1.4 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
2019-11-19 10:44:38 -05:00
```
2021-04-15 11:58:46 -04:00
You can get a list of pull requests with labels started with blocking, risk, hold, revert and security by using the parameter `--risk` .
Example:
2022-06-02 09:07:57 -04:00
2021-04-15 11:58:46 -04:00
```bash
python changelog.py --previous_version 0.37 --current_version 0.38 changelog --access_token {GITHUB_TOKEN} --risk
```
2022-05-27 09:09:12 -04:00
The script will checkout both branches, compare all the PRs, and output the lines that are needed to be added to the
`CHANGELOG.md` file in the root of the repo. Remember to also make sure to update the branch id (with the above command
`1.5` needs to be changed to `1.5.0` )
2020-10-30 03:50:54 -04:00
2019-11-27 10:28:37 -05:00
Then, in `UPDATING.md` , a file that contains a list of notifications around
deprecations and upgrading-related topics,
make sure to move the content now under the `Next Version` section under a new
section for the new release.
2022-05-27 09:09:12 -04:00
#### Patch release (e.g. 1.5.1)
To compare the forthcoming patch release with the latest release from the same branch, set
`--previous_version` as the tag of the previous release (in this example `1.5.0` ; remember to update accordingly)
```bash
python changelog.py --previous_version 1.5.0 --current_version ${SUPERSET_GITHUB_BRANCH} changelog
```
### Set version number
2019-10-28 06:17:10 -04:00
2022-05-27 09:09:12 -04:00
Finally, bump the version number on `superset-frontend/package.json` (replace with whichever version is being released excluding the RC version):
```
2021-01-21 10:52:48 -05:00
"version": "0.38.0"
2019-11-15 08:44:07 -05:00
```
2019-10-28 06:17:10 -04:00
2020-04-06 02:21:31 -04:00
Commit the change with the version number, then git tag the version with the release candidate and push to the branch:
```
2021-01-21 10:52:48 -05:00
# add changed files and commit
git add ...
git commit ...
# push new tag
git tag ${SUPERSET_VERSION_RC}
2022-05-27 09:09:12 -04:00
git push origin ${SUPERSET_VERSION_RC}
2020-04-06 02:21:31 -04:00
```
2019-10-28 06:17:10 -04:00
2022-06-02 09:07:57 -04:00
### Create a release on Github
After submitting the tag, follow the steps [here ](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository ) to create the release. Use the vote email text as the content for the release description. Make sure to check the "This is a pre-release" checkbox for release canditates. You can check previous releases if you need an example.
2019-10-23 09:43:04 -04:00
## Preparing the release candidate
The first step of preparing an Apache Release is packaging a release candidate
2019-11-19 10:44:38 -05:00
to be voted on. Make sure you have correctly prepared and tagged the ready to ship
2020-02-09 20:53:56 -05:00
release on Superset's repo (MAJOR.MINOR branch), the following script will clone
2019-11-19 10:44:38 -05:00
the tag and create a signed source tarball from it:
2019-10-23 09:43:04 -04:00
```bash
2021-01-21 10:52:48 -05:00
# make_tarball will use the previously set environment variables
# you can override by passing arguments: make_tarball.sh <SUPERSET_VERSION> <SUPERSET_VERSION_RC> "<PGP_KEY_FULLNAME>"
./make_tarball.sh
2019-10-23 09:43:04 -04:00
```
2019-03-25 19:21:26 -04:00
2019-11-19 10:44:38 -05:00
Note that `make_tarball.sh` :
2019-10-23 09:43:04 -04:00
2022-05-27 09:09:12 -04:00
- By default, the script assumes you have already executed an SVN checkout to `$HOME/svn/superset_dev` .
2022-06-02 09:07:57 -04:00
This can be overridden by setting `SUPERSET_SVN_DEV_PATH` environment var to a different svn dev directory
2019-11-19 10:44:38 -05:00
- Will refuse to craft a new release candidate if a release already exists on your local svn dev directory
- Will check `package.json` version number and fails if it's not correctly set
2019-10-23 09:43:04 -04:00
2019-11-19 10:44:38 -05:00
### Build and test the created source tarball
2019-10-23 09:43:04 -04:00
2020-02-18 00:14:31 -05:00
To build and run the **local copy** of the recently created tarball:
2022-06-02 09:07:57 -04:00
2019-10-23 09:43:04 -04:00
```bash
2021-01-21 10:52:48 -05:00
# Build and run a release candidate tarball
./test_run_tarball.sh local
# you should be able to access localhost:5001 on your browser
# login using admin/admin
2019-03-25 19:21:26 -04:00
```
2019-10-23 09:43:04 -04:00
### Shipping to SVN
2019-04-20 13:26:01 -04:00
2019-04-15 15:48:51 -04:00
Now let's ship this RC into svn's dev folder
2019-03-25 19:21:26 -04:00
```bash
2021-01-21 10:52:48 -05:00
cd ~/svn/superset_dev/
svn add ${SUPERSET_VERSION_RC}
svn commit -m "Release ${SUPERSET_VERSION_RC}"
2021-07-03 09:33:46 -04:00
svn update
2019-03-25 19:21:26 -04:00
```
2019-11-19 10:44:38 -05:00
### Build and test from SVN source tarball
2019-11-15 08:44:07 -05:00
2020-02-18 00:14:31 -05:00
To build and run the recently created tarball **from SVN** :
2022-06-02 09:07:57 -04:00
2019-11-15 08:44:07 -05:00
```bash
2021-01-21 10:52:48 -05:00
# Build and run a release candidate tarball
./test_run_tarball.sh
# you should be able to access localhost:5001 on your browser
# login using admin/admin
2019-11-15 08:44:07 -05:00
```
2019-10-30 13:24:18 -04:00
### Voting
2022-06-02 09:07:57 -04:00
2019-10-30 13:24:18 -04:00
Now you're ready to start the [VOTE] thread. Here's an example of a
previous release vote thread:
https://lists.apache.org/thread.html/e60f080ebdda26896214f7d3d5be1ccadfab95d48fbe813252762879@< dev.superset.apache.org >
2019-11-25 05:01:31 -05:00
To easily send a voting request to Superset community, still on the `superset/RELEASING` directory:
```bash
2021-01-21 10:52:48 -05:00
# Note: use Superset's virtualenv
2022-06-02 09:07:57 -04:00
(venv)$ python generate_email.py vote_pmc
2019-11-25 05:01:31 -05:00
```
2022-06-02 09:07:57 -04:00
The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version and release candidate number are fetched from the previously set environment variables.
2019-11-25 05:01:31 -05:00
2019-10-30 13:24:18 -04:00
Once 3+ binding votes (by PMC members) have been cast and at
least 72 hours have past, you can post a [RESULT] thread:
https://lists.apache.org/thread.html/50a6b134d66b86b237d5d7bc89df1b567246d125a71394d78b45f9a8@%3Cdev.superset.apache.org%3E
2019-11-25 05:01:31 -05:00
To easily send the result email, still on the `superset/RELEASING` directory:
```bash
2021-01-21 10:52:48 -05:00
# Note: use Superset's virtualenv
2022-06-02 09:07:57 -04:00
python generate_email.py result_pmc
2019-11-25 05:01:31 -05:00
```
The script will interactively ask for extra information needed to fill out the email template. Based on the
2020-02-09 20:53:56 -05:00
voting description, it will generate a passing, non passing or non conclusive email.
2022-06-02 09:07:57 -04:00
Here's an example:
2019-11-25 05:01:31 -05:00
2021-01-21 10:52:48 -05:00
```
A List of people with +1 binding vote (ex: Max,Grace,Krist): Daniel,Alan,Max,Grace
A List of people with +1 non binding vote (ex: Ville): Ville
A List of people with -1 vote (ex: John):
2019-11-25 05:01:31 -05:00
```
2022-06-02 09:07:57 -04:00
The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version and release candidate number are fetched from the previously set environment variables.
2019-11-27 10:28:37 -05:00
2019-10-23 09:43:04 -04:00
### Validating a release
2019-04-20 13:26:01 -04:00
https://www.apache.org/info/verification.html
## Publishing a successful release
2021-01-21 10:52:48 -05:00
Upon a successful vote, you'll have to copy the folder into the non-"dev/" folder.
2022-06-02 09:07:57 -04:00
2019-04-15 15:48:51 -04:00
```bash
2021-01-21 10:52:48 -05:00
cp -r ~/svn/superset_dev/${SUPERSET_VERSION_RC}/ ~/svn/superset/${SUPERSET_VERSION}/
cd ~/svn/superset/
# Rename the RC (0.34.1rc1) to the actual version being released (0.34.1)
for f in ${SUPERSET_VERSION}/*; do mv "$f" "${f/${SUPERSET_VERSION_RC}/${SUPERSET_VERSION}}"; done
svn add ${SUPERSET_VERSION}
svn commit -m "Release ${SUPERSET_VERSION}"
2021-07-03 09:33:46 -04:00
svn update
2019-10-23 09:43:04 -04:00
```
Then tag the final release:
2022-06-02 09:07:57 -04:00
2019-10-23 09:43:04 -04:00
```bash
2021-01-21 10:52:48 -05:00
# Go to the root directory of the repo, e.g. `~/src/superset`
cd ~/src/superset/
# make sure you're on the correct branch (e.g. 0.34)
git branch
# Create the release tag
git tag -f ${SUPERSET_VERSION}
2022-03-18 11:27:57 -04:00
# push the tag to the remote
2022-05-27 09:09:12 -04:00
git push origin ${SUPERSET_VERSION}
2019-04-15 15:48:51 -04:00
```
2019-11-27 10:28:37 -05:00
### Update CHANGELOG and UPDATING on superset
Now that we have a final Apache source release we need to open a pull request on Superset
with the changes on `CHANGELOG.md` and `UPDATING.md` .
2019-08-12 17:01:15 -04:00
2019-11-05 03:44:06 -05:00
### Publishing a Convenience Release to PyPI
2019-11-27 10:28:37 -05:00
2020-04-17 11:28:10 -04:00
Using the final release tarball, unpack it and run `./pypi_push.sh` .
2022-04-26 12:17:15 -04:00
This script will build the JavaScript bundle and echo the twine command
2020-04-17 11:28:10 -04:00
allowing you to publish to PyPI. You may need to ask a fellow committer to grant
2019-11-05 03:44:06 -05:00
you access to it if you don't have access already. Make sure to create
an account first if you don't have one, and reference your username
while requesting access to push packages.
2019-11-27 10:28:37 -05:00
### Announcing
2019-09-10 13:36:07 -04:00
2019-11-27 10:28:37 -05:00
Once it's all done, an [ANNOUNCE] thread announcing the release to the dev@ mailing list is the final step.
```bash
2021-01-21 10:52:48 -05:00
# Note use Superset's virtualenv
2022-06-02 09:07:57 -04:00
python generate_email.py announce
2019-11-27 10:28:37 -05:00
```
2022-06-02 09:07:57 -04:00
The script will generate the email text that should be sent to dev@superset.apache.org using an email client. The release version is fetched from the previously set environment variables.
2022-04-26 12:17:15 -04:00
### GitHub Release
2020-07-01 11:14:09 -04:00
2022-04-26 12:17:15 -04:00
Finally, so the GitHub UI reflects the latest release, you should create a release from the
2021-01-06 06:45:19 -05:00
tag corresponding with the new version. Go to https://github.com/apache/superset/tags,
2020-07-01 11:14:09 -04:00
click the 3-dot icon and select `Create Release` , paste the content of the ANNOUNCE thread in the
release notes, and publish the new release.
2020-12-17 13:50:44 -05:00
At this point, a GitHub action will run that will check whether this release's version number is higher than the current 'latest' release. If that condition is true, this release sha will automatically be tagged as `latest` so that the most recent release can be referenced simply by using the 'latest' tag instead of looking up the version number. The existing version number tag will still exist, and can also be used for reference.