2015-07-20 19:29:16 -04:00
# Contributing
Contributions are welcome and are greatly appreciated! Every
little bit helps, and credit will always be given.
2018-09-19 13:01:51 -04:00
## Table of Contents
- [Types of Contributions ](#types-of-contributions )
- [Report Bugs ](#report-bugs )
- [Fix Bugs ](#fix-bugs )
- [Implement Features ](#implement-features )
- [Improve Documentation ](#improve-documentation )
- [Add Translations ](#add-translations )
- [Submit Feedback ](#submit-feedback )
- [Ask Questions ](#ask-questions )
- [Pull Request Guidelines ](#pull-request-guidelines )
- [Local development ](#local-development )
- [Documentation ](#documentation )
- [Flask server ](#flask-server )
- [Frontend assets ](#frontend-assets )
- [Testing ](#testing )
- [JavaScript testing ](#javascript-testing )
- [Integration testing ](#integration-testing )
- [Linting ](#linting )
- [Translating ](#translating )
- [Enabling language selection ](#enabling-language-selection )
- [Extracting new strings for translation ](#extracting-new-strings-for-translation )
- [Creating a new language dictionary ](#creating-a-new-language-dictionary )
- [Tips ](#tips )
- [Adding a new datasource ](#adding-a-new-datasource )
- [Creating a new visualization type ](#creating-a-new-visualization-type )
- [Adding a DB migration ](#adding-a-db-migration )
- [Merging DB migrations ](#merging-db-migrations )
2015-07-20 19:29:16 -04:00
## Types of Contributions
### Report Bugs
2018-09-19 13:01:51 -04:00
Report bugs through GitHub. If you are reporting a bug, please include:
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
- Your operating system name and version.
- Any details about your local setup that might be helpful in troubleshooting.
- Detailed steps to reproduce the bug.
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
When posting Python stack traces, please quote them using
[Markdown blocks ](https://help.github.com/articles/creating-and-highlighting-code-blocks/ ).
2017-06-21 13:12:34 -04:00
2015-07-20 19:29:16 -04:00
### Fix Bugs
2018-09-19 13:01:51 -04:00
Look through the GitHub issues for bugs. Anything tagged with `bug` is
2015-07-20 19:29:16 -04:00
open to whoever wants to implement it.
### Implement Features
Look through the GitHub issues for features. Anything tagged with
2018-09-19 13:01:51 -04:00
`feature` or `starter_task` is open to whoever wants to implement it.
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
### Improve Documentation
2015-07-20 19:29:16 -04:00
2016-11-10 02:08:22 -05:00
Superset could always use better documentation,
whether as part of the official Superset docs,
2015-07-20 19:29:16 -04:00
in docstrings, `docs/*.rst` or even on the web as blog posts or
2018-09-19 13:01:51 -04:00
articles. See [Documentation ](#documentation ) for more details.
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
### Add Translations
2018-08-25 19:31:03 -04:00
2018-09-19 13:01:51 -04:00
If you are proficient in a non-English language, you can help translate text strings from Superset's UI. You can jump in to the existing language dictionaries at `superset/translations/<language_code>/LC_MESSAGES/messages.po` , or even create a dictionary for a new language altogether. See [Translating ](#translating ) for more details.
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
### Submit Feedback
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
The best way to send feedback is to file an issue on GitHub. If you are proposing a feature:
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
- Explain in detail how it would work.
- Keep the scope as narrow as possible, to make it easier to implement.
- Remember that this is a volunteer-driven project, and that contributions are welcome :)
2018-03-14 19:50:25 -04:00
2018-09-19 13:01:51 -04:00
### Ask Questions
2017-11-02 16:49:47 -04:00
2018-09-19 13:01:51 -04:00
There is a dedicated [`apache-superset` tag ](https://stackoverflow.com/questions/tagged/apache-superset ) on [StackOverflow ](https://stackoverflow.com/ ). Please use it when asking questions.
2015-07-20 19:29:16 -04:00
2017-06-04 21:00:33 -04:00
## Pull Request Guidelines
Before you submit a pull request from your forked repo, check that it
meets these guidelines:
1. The pull request should include tests, either as doctests,
unit tests, or both.
2018-04-10 18:59:44 -04:00
2. Run `tox` and resolve all errors and test failures.
3. If the pull request adds functionality, the docs should be updated
2017-06-04 21:00:33 -04:00
as part of the same PR. Doc string are often sufficient, make
sure to follow the sphinx compatible standards.
2018-08-07 16:22:19 -04:00
4. The pull request should work for Python 2.7 and Python 3.6.
2017-06-04 21:00:33 -04:00
``from __future__ import`` will be required in every `.py` file soon.
2018-04-10 18:59:44 -04:00
5. If the pull request adds a Python dependency include it in `setup.py`
denoting any specific restrictions and in `requirements.txt` pinned to a
specific version which ensures that the application build is deterministic.
6. Please rebase and resolve all conflicts before submitting.
7. Please ensure the necessary checks pass and that code coverage does not
decrease.
2018-04-06 19:14:14 -04:00
8. If you are asked to update your pull request with some changes there's
2017-06-04 21:00:33 -04:00
no need to create a new one. Push your changes to the same branch.
2018-09-19 13:01:51 -04:00
## Local development
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
First, [fork the repository on GitHub ](https://help.github.com/articles/about-forks/ ), then clone it. You can clone the main repository directly instead, but you won't be able to send pull requests.
```bash
git clone git@github.com:your-username/incubator-superset.git
cd incubator-superset
```
### Documentation
The latest documentation and tutorial are available at https://superset.incubator.apache.org/.
2017-01-25 14:41:39 -05:00
Contributing to the official documentation is relatively easy, once you've setup
your environment and done an edit end-to-end. The docs can be found in the
`docs/` subdirectory of the repository, and are written in the
[reStructuredText format ](https://en.wikipedia.org/wiki/ReStructuredText ) (.rst).
If you've written Markdown before, you'll find the reStructuredText format familiar.
Superset uses [Sphinx ](http://www.sphinx-doc.org/en/1.5.1/ ) to convert the rst files
in `docs/` to the final HTML output users see.
2018-03-14 19:50:25 -04:00
Finally, to make changes to the rst files and build the docs using Sphinx,
2017-01-26 12:07:23 -05:00
you'll need to install a handful of dependencies from the repo you cloned:
2017-01-25 14:41:39 -05:00
2018-09-19 13:01:51 -04:00
```bash
pip install -r docs/requirements.txt
```
2017-01-25 14:41:39 -05:00
To get the feel for how to edit and build the docs, let's edit a file, build
the docs and see our changes in action. First, you'll want to
[create a new branch ](https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging )
to work on your changes:
2018-09-19 13:01:51 -04:00
```bash
git checkout -b changes-to-docs
```
2017-01-25 14:41:39 -05:00
2018-09-19 13:01:51 -04:00
Now, go ahead and edit one of the files under `docs/` , say `docs/tutorial.rst` - change
it however you want. Check out the
2017-01-25 14:41:39 -05:00
[ReStructuredText Primer ](http://docutils.sourceforge.net/docs/user/rst/quickstart.html )
for a reference on the formatting of the rst files.
2018-09-19 13:01:51 -04:00
Once you've made your changes, run this command to convert the docs into HTML:
2017-01-25 14:41:39 -05:00
2018-09-19 13:01:51 -04:00
```bash
make html
```
2017-01-25 14:41:39 -05:00
You'll see a lot of output as Sphinx handles the conversion. After it's done, the
2018-09-19 13:01:51 -04:00
HTML Sphinx generated should be in `docs/_build/html` . Navigate there
2017-01-25 14:41:39 -05:00
and start a simple web server so we can check out the docs in a browser:
2018-09-19 13:01:51 -04:00
```bash
cd docs/_build/html
python -m SimpleHTTPServer
```
2017-01-25 14:41:39 -05:00
This will start a small Python web server listening on port 8000. Point your
2018-09-19 13:01:51 -04:00
browser to http://localhost:8000, find the file
2017-01-25 14:41:39 -05:00
you edited earlier, and check out your changes!
If you've made a change you'd like to contribute to the actual docs, just commit
your code, push your new branch to Github:
2018-09-19 13:01:51 -04:00
```bash
git add docs/tutorial.rst
git commit -m 'Awesome new change to tutorial'
git push origin changes-to-docs
```
2017-01-25 14:41:39 -05:00
Then, [open a pull request ](https://help.github.com/articles/about-pull-requests/ ).
2018-09-19 13:01:51 -04:00
#### Images
2017-01-25 14:41:39 -05:00
If you're adding new images to the documentation, you'll notice that the images
referenced in the rst, e.g.
.. image:: _static/img/tutorial/tutorial_01_sources_database.png
2018-09-19 13:01:51 -04:00
aren't actually stored in that directory. Instead, you should add and commit
images (and any other static assets) to the `superset/assets/images` directory.
When the docs are deployed to https://superset.incubator.apache.org/, images
are copied from there to the `_static/img` directory, just like they're referenced
2017-01-25 14:41:39 -05:00
in the docs.
2018-09-19 13:01:51 -04:00
For example, the image referenced above actually lives in `superset/assets/images/tutorial` . Since the image is moved during the documentation build process, the docs reference the image in `_static/img/tutorial` instead.
2017-01-25 14:41:39 -05:00
2018-09-19 13:01:51 -04:00
#### API documentation
2017-01-25 14:41:39 -05:00
2018-09-19 13:01:51 -04:00
Generate the API documentation with:
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
```bash
pip install -r docs/requirements.txt
python setup.py build_sphinx
```
2016-04-08 20:57:24 -04:00
2018-09-19 13:01:51 -04:00
### Flask server
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
Make sure your machine meets the [OS dependencies ](https://superset.incubator.apache.org/installation.html#os-dependencies ) before following these steps.
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
```bash
# Create a virtual environemnt and activate it (recommended)
virtualenv venv
source venv/bin/activate
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
# Install external dependencies
pip install -r requirements.txt
# Install Superset in editable (development) mode
pip install -e .
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
# Create an admin user
fabmanager create-admin --app superset
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
# Initialize the database
superset db upgrade
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
# Create default roles and permissions
superset init
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
# Load some data to play with
superset load_examples
2016-01-13 12:03:27 -05:00
2018-09-19 13:01:51 -04:00
# Start the Flask web server (but see below for frontend asset compilation)
superset runserver -d
```
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
#### Logging to the browser console
2018-04-13 00:48:17 -04:00
2018-09-19 13:01:51 -04:00
This feature is only available on Python 3. When debugging your application, you can have the server logs sent directly to the browser console:
2018-04-13 00:48:17 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset runserver -d --console-log
```
2018-04-13 00:48:17 -04:00
You can log anything to the browser console, including objects:
2018-09-19 13:01:51 -04:00
```python
from superset import app
app.logger.error('An exception occurred!')
app.logger.info(form_data)
```
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
### Frontend assets
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
Frontend assets (JavaScript, CSS, and images) must be compiled in order to properly display the web UI. The `superset/assets` directory contains all NPM-managed front end assets. Note that there are additional frontend assets bundled with Flask-Appbuilder (e.g. jQuery and bootstrap); these are not managed by NPM, and may be phased out in the future.
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
First, be sure you are using recent versions of NodeJS and npm. Using [nvm ](https://github.com/creationix/nvm ) to manage them is recommended.
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
Install third-party dependencies listed in `package.json` :
2016-02-27 10:47:56 -05:00
2017-07-19 04:17:56 -04:00
```bash
2018-09-19 13:01:51 -04:00
# From the root of the repository
cd superset/assets
# Install yarn, a replacement for `npm install`
2017-07-19 04:17:56 -04:00
npm install -g yarn
2018-09-19 13:01:51 -04:00
# Install dependencies
yarn install
2016-02-27 10:47:56 -05:00
```
2018-09-19 13:01:51 -04:00
Finally, to compile frontend assets, run any of the following commands.
2016-02-27 10:47:56 -05:00
2018-09-04 13:39:05 -04:00
```bash
2018-09-19 13:01:51 -04:00
# Start a watcher that recompiles your assets as you modify them (reload your browser to see changes)
npm run dev
2017-02-16 20:28:35 -05:00
2018-09-19 13:01:51 -04:00
# Compile the Javascript and CSS in production/optimized mode for official releases
2016-02-27 10:47:56 -05:00
npm run prod
2018-09-19 13:01:51 -04:00
# Copy a conf file from the frontend to the backend
npm run sync-backend
2016-02-27 10:47:56 -05:00
```
2018-09-19 13:01:51 -04:00
#### Webpack dev server
2016-02-27 10:47:56 -05:00
2018-09-19 13:01:51 -04:00
Alternatively, you can run the Webpack dev server, which runs on port 9000 and proxies non-asset requests to the Flask server on port 8088. After pointing your browser to it, updates to asset sources will be reflected in-browser without a refresh.
2018-09-04 13:39:05 -04:00
```bash
2018-09-19 13:01:51 -04:00
# Run the dev server
2018-09-04 13:39:05 -04:00
npm run dev-server
2018-09-19 13:01:51 -04:00
# Run the dev server on a non-default port
npm run dev-server -- --port=9001
2018-09-04 13:39:05 -04:00
2018-09-19 13:01:51 -04:00
# Run the dev server proxying to a Flask server on a non-default port
2018-09-04 13:39:05 -04:00
npm run dev-server -- --supersetPort=8081
```
2018-09-19 13:01:51 -04:00
#### Upgrading NPM packages
2018-03-23 18:31:47 -04:00
2018-09-19 13:01:51 -04:00
After adding or upgrading an NPM package by changing `package.json` , you must run `yarn install` , which will regenerate the `yarn.lock` file. Then, be sure to commit the new `yarn.lock` so that other users' builds are reproducible. See [the Yarn docs ](https://yarnpkg.com/blog/2016/11/24/lockfiles-for-all/ ) for more information.
2018-03-23 18:31:47 -04:00
2016-01-13 12:03:27 -05:00
## Testing
2018-09-19 13:01:51 -04:00
2018-04-10 18:59:44 -04:00
All tests are carried out in [tox ](http://tox.readthedocs.io/en/latest/index.html )
a standardized testing framework mostly for Python (though we also used it for Javascript).
All python tests can be run with any of the tox [environments ](http://tox.readthedocs.io/en/latest/example/basic.html#a-simple-tox-ini-default-environments ), via,
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
```bash
tox -e < environment >
```
2018-04-10 18:59:44 -04:00
i.e.,
2017-06-21 15:52:04 -04:00
2018-09-19 13:01:51 -04:00
```bash
tox -e py27
tox -e py36
```
2015-07-20 19:29:16 -04:00
2018-04-10 18:59:44 -04:00
Alternatively, you can run all tests in a single file via,
2018-03-14 19:50:25 -04:00
2018-09-19 13:01:51 -04:00
```bash
tox -e < environment > -- tests/test_file.py
```
2017-10-12 23:53:37 -04:00
2018-04-10 18:59:44 -04:00
or for a specific test via,
2018-03-14 19:50:25 -04:00
2018-09-19 13:01:51 -04:00
```bash
tox -e < environment > -- tests/test_file.py:TestClassName.test_method_name
```
2018-04-10 18:59:44 -04:00
Note that the test environment uses a temporary directory for defining the
SQLite databases which will be cleared each time before the group of test
commands are invoked.
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
### JavaScript testing
2016-07-19 19:14:04 -04:00
We use [Mocha ](https://mochajs.org/ ), [Chai ](http://chaijs.com/ ) and [Enzyme ](http://airbnb.io/enzyme/ ) to test Javascript. Tests can be run with:
2018-09-19 13:01:51 -04:00
```bash
cd superset/assets/javascripts
npm install
npm run test
```
### Integration testing
2016-07-19 19:14:04 -04:00
2018-09-07 13:26:51 -04:00
We use [Cypress ](https://www.cypress.io/ ) for integration tests. Tests can be run by `tox -e cypress` . To open Cypress and explore tests first setup and run test server:
2018-09-19 13:01:51 -04:00
```bash
export SUPERSET_CONFIG=tests.superset_test_config
superset db upgrade
superset init
superset load_test_users
superset load_examples
superset runserver
```
2018-09-07 13:26:51 -04:00
2018-09-19 13:01:51 -04:00
Run Cypress tests:
2018-09-07 13:26:51 -04:00
2018-09-19 13:01:51 -04:00
```bash
cd /superset/superset/assets
npm run build
npm run cypress run
```
2018-09-07 13:26:51 -04:00
2018-09-19 13:01:51 -04:00
### Linting
2016-07-19 19:14:04 -04:00
2015-07-20 19:29:16 -04:00
Lint the project with:
2018-09-19 13:01:51 -04:00
```bash
# for python
tox -e flake8
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
# for javascript
tox -e eslint
```
2016-03-05 15:41:22 -05:00
2018-09-19 13:01:51 -04:00
## Translating
2016-03-05 15:41:22 -05:00
2018-09-19 13:01:51 -04:00
We use [Babel ](http://babel.pocoo.org/en/latest/ ) to translate Superset. In Python files, we import the magic `_` function using:
2015-07-20 19:29:16 -04:00
2018-09-19 13:01:51 -04:00
```python
from flask_babel import lazy_gettext as _
```
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
then wrap our translatable strings with it, e.g. `_('Translate me')` . During extraction, string literals passed to `_` will be added to the generated `.po` file for each language for later translation.
At runtime, the `_` function will return the translation of the given string for the current language, or the given string itself if no translation is available.
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
In JavaScript, the technique is similar: we import `t` (simple translation), `tn` (translation containing a number), and `TCT` (translating entire React Components).
js translation -- performance improvment (#3390)
* Chinese page
* Using react-intl-universal to improve multi language in react page
* Using react-intl-universal to improve multi language in react page
* react_intl_universal
* change
* change
* change
* change
* change
* change
* change
* merge
* multiple page in js
* merge
* merge
* merge
* merge
* Js Translations
* JS Translation
* JS Translations
* Js translation
* JS translations
* JS translations
* Js translaion
* JS en Translation
* JS Translation
* upgrade document
Fixing the damn build (#3179)
* Fixing the build
* Going deeper
[bugfix] only filterable columns should show up in FilterBox list (#3105)
* [bugfix] only filterable columns should show up in FilterBox list
* Touchups
Datasource cannot be empty (#3035)
add title description to model view (#3045)
* add title description to model view
* add missing import
Add 'show/hide totals' option to pivot table vis (#3101)
[bugfix] numeric value for date fields in table viz (#3036)
Bug was present only when using the NOT GROUPED BY option
fixes https://github.com/ApacheInfra/superset/issues/3027
fix hive.fetch_logs (#2968)
add Zalando to the list of organizations (#3171)
docs: fixup installation examples code indentation (#3169)
[bugfix] fix bar order (#3180)
[bugfix] visualize flow error: 'Metric x is not valid' (#3181)
The metric name in the frontend doesn't match the one generated on the
backend. It turns out the explore view will default to the first
metric so specifying one isn't needed.
Fix the segment interval for pulling metadata (#3174)
The end of the interval would be on the truncated today date, which
means that you will exclude today. If your realtime ingestion job
runs shorter than a day, the metadata cannot be pulled from the
druid cluster.
Bump cryptography to 1.9 (#3065)
As 1.7.2 doesn't compile here with openssl 1.1.0f
Escaping the user's SQL in the explore view (#3186)
* Escaping the user's SQL in the explore view
When executing SQL from SQL Lab, we use a lower level API to the
database which doesn't require escaping the SQL. When going through
the explore view, the stack chain leading to the same method may need
escaping depending on how the DBAPI driver is written, and that is the
case for Presto (and perhaps other drivers).
* Using regex to avoid doubling doubles
[sqllab] improve Hive support (#3187)
* [sqllab] improve Hive support
* Fix "Transport not open" bug
* Getting progress bar to show
* Bump pyhive to 0.4.0
* Getting [Track Job] button to show
* Fix testzz
Add BigQuery engine specifications (#3193)
As contributed by @mxmzdlv on issue #945
[bugfix] fix merge conflict that broke Hive support (#3196)
Adding 'apache' to docs (#3194)
[druid] Allow custom druid postaggregators (#3146)
* [druid] Allow custom druid postaggregators
Also, fix the postaggregation for approxHistogram quantiles so it adds
the dependent field and that can show up in the graphs/tables.
In general, postAggregators add significant power, we should probably
support including custom postAggregators. Plywood has standard
postAggregators here, and a customAggregator escape hatch that allows
you to define custom postAggregators.
This commit adds a similar capability for Superset and a additional
field/fields/fieldName breakdown of the typical naming for dependent
aggregations, which should make it significantly easier to develop
approxHistogram and custom postAggregation-required dashboards.
* [druid] Minor style cleanup in tests file.
* [druid] Apply code review suggestions
* break out CustomPostAggregator into separate class. This just cleans
up the creation of the postaggregator a little bit.
* minor style issues.
* move the function around so the git diff is more readable
add combine config for metrics in pivot table (#3086)
* add combine config for metrics in pivot table
* change method to stack/unstack
* update backendSync
Autofocus search input in VizTypeControl modal onEnter (#2929)
Speed up JS build time (#3203)
Also bumping a few related libs
JS Translation
JS translations
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
[docs] update url in CONTRIBUTING.md (#3212)
[sqllab/cosmetics] add margin-top for labels in query history (#3222)
[explore] nvd3 sort values in rich tooltip (#3197)
[sqllab] fix UI shows 'The query returned no results' momentarily (#3214)
this is visible when running async queries between the fetching and
success state as the rows are getting cached in the component
[explore] DatasourceControl to pick datasource in modal (#3210)
* [explore] DatasourceControl to pick datasource in modal
Makes it easier to change datasource, also makes it such that the list
of all datasources doesn't need to be loaded upfront.
* Adding more metadata
* Js translation
* js tran
* js trans
* js trans
* js tran
* js trans
* js trans
* js tran
* js translation
* js trans
* js translation
* try load language pack async
* Backend translations things
* create language pack inside common data
* performance improvement for js i18n.
- js bundle should not contain localized content
- we populate translation content from server-side, in boostrap.common.language_pack
- in client-side, use promise to wrap around translation content. text will be translated after translation content arrived/parsed.
- fix linting
* fix Timer unit test
* 1. add global hook for all tests, to make translation pack avaialble before each test starts.
2. fix unit test for Timer component
3. remove noused method get_locale, and modules
4. fix page reload after user change page language
* parse and build i18n dictionary as a module
* fix sync-backend task, which should run without DOM
2017-09-20 15:37:33 -04:00
2018-09-19 13:01:51 -04:00
```javascript
import {t, tn, TCT} from locales;
```
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
### Enabling language selection
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
Add the `LANGUAGES` variable to your `superset_config.py` . Having more than one
option inside will add a language selection dropdown to the UI on the right side
of the navigation bar.
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
```python
LANGUAGES = {
'en': {'flag': 'us', 'name': 'English'},
'fr': {'flag': 'fr', 'name': 'French'},
'zh': {'flag': 'cn', 'name': 'Chinese'},
}
```
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
### Extracting new strings for translation
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
```bash
fabmanager babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _ -k __ -k t -k tn -k tct
```
2016-05-02 13:50:23 -04:00
You can then translate the strings gathered in files located under
2016-11-10 02:08:22 -05:00
`superset/translation` , where there's one per language. For the translations
2018-09-19 13:01:51 -04:00
to take effect:
2016-05-02 13:50:23 -04:00
2018-09-19 13:01:51 -04:00
```bash
# In the case of JS translation, we need to convert the PO file into a JSON file, and we need the global download of the npm package po2json.
npm install -g po2json
fabmanager babel-compile --target superset/translations
# Convert the en PO file into a JSON file
po2json -d superset -f jed1.x superset/translations/en/LC_MESSAGES/messages.po superset/translations/en/LC_MESSAGES/messages.json
```
2016-09-21 12:52:05 -04:00
2018-09-19 13:01:51 -04:00
If you get errors running `po2json` , you might be running the Ubuntu package with the same
name, rather than the NodeJS package (they have a different format for the arguments). If
there is a conflict, you may need to update your `PATH` environment variable or fully qualify
the executable path (e.g. `/usr/local/bin/po2json` instead of `po2json` ).
js translation -- performance improvment (#3390)
* Chinese page
* Using react-intl-universal to improve multi language in react page
* Using react-intl-universal to improve multi language in react page
* react_intl_universal
* change
* change
* change
* change
* change
* change
* change
* merge
* multiple page in js
* merge
* merge
* merge
* merge
* Js Translations
* JS Translation
* JS Translations
* Js translation
* JS translations
* JS translations
* Js translaion
* JS en Translation
* JS Translation
* upgrade document
Fixing the damn build (#3179)
* Fixing the build
* Going deeper
[bugfix] only filterable columns should show up in FilterBox list (#3105)
* [bugfix] only filterable columns should show up in FilterBox list
* Touchups
Datasource cannot be empty (#3035)
add title description to model view (#3045)
* add title description to model view
* add missing import
Add 'show/hide totals' option to pivot table vis (#3101)
[bugfix] numeric value for date fields in table viz (#3036)
Bug was present only when using the NOT GROUPED BY option
fixes https://github.com/ApacheInfra/superset/issues/3027
fix hive.fetch_logs (#2968)
add Zalando to the list of organizations (#3171)
docs: fixup installation examples code indentation (#3169)
[bugfix] fix bar order (#3180)
[bugfix] visualize flow error: 'Metric x is not valid' (#3181)
The metric name in the frontend doesn't match the one generated on the
backend. It turns out the explore view will default to the first
metric so specifying one isn't needed.
Fix the segment interval for pulling metadata (#3174)
The end of the interval would be on the truncated today date, which
means that you will exclude today. If your realtime ingestion job
runs shorter than a day, the metadata cannot be pulled from the
druid cluster.
Bump cryptography to 1.9 (#3065)
As 1.7.2 doesn't compile here with openssl 1.1.0f
Escaping the user's SQL in the explore view (#3186)
* Escaping the user's SQL in the explore view
When executing SQL from SQL Lab, we use a lower level API to the
database which doesn't require escaping the SQL. When going through
the explore view, the stack chain leading to the same method may need
escaping depending on how the DBAPI driver is written, and that is the
case for Presto (and perhaps other drivers).
* Using regex to avoid doubling doubles
[sqllab] improve Hive support (#3187)
* [sqllab] improve Hive support
* Fix "Transport not open" bug
* Getting progress bar to show
* Bump pyhive to 0.4.0
* Getting [Track Job] button to show
* Fix testzz
Add BigQuery engine specifications (#3193)
As contributed by @mxmzdlv on issue #945
[bugfix] fix merge conflict that broke Hive support (#3196)
Adding 'apache' to docs (#3194)
[druid] Allow custom druid postaggregators (#3146)
* [druid] Allow custom druid postaggregators
Also, fix the postaggregation for approxHistogram quantiles so it adds
the dependent field and that can show up in the graphs/tables.
In general, postAggregators add significant power, we should probably
support including custom postAggregators. Plywood has standard
postAggregators here, and a customAggregator escape hatch that allows
you to define custom postAggregators.
This commit adds a similar capability for Superset and a additional
field/fields/fieldName breakdown of the typical naming for dependent
aggregations, which should make it significantly easier to develop
approxHistogram and custom postAggregation-required dashboards.
* [druid] Minor style cleanup in tests file.
* [druid] Apply code review suggestions
* break out CustomPostAggregator into separate class. This just cleans
up the creation of the postaggregator a little bit.
* minor style issues.
* move the function around so the git diff is more readable
add combine config for metrics in pivot table (#3086)
* add combine config for metrics in pivot table
* change method to stack/unstack
* update backendSync
Autofocus search input in VizTypeControl modal onEnter (#2929)
Speed up JS build time (#3203)
Also bumping a few related libs
JS Translation
JS translations
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
[docs] update url in CONTRIBUTING.md (#3212)
[sqllab/cosmetics] add margin-top for labels in query history (#3222)
[explore] nvd3 sort values in rich tooltip (#3197)
[sqllab] fix UI shows 'The query returned no results' momentarily (#3214)
this is visible when running async queries between the fetching and
success state as the rows are getting cached in the component
[explore] DatasourceControl to pick datasource in modal (#3210)
* [explore] DatasourceControl to pick datasource in modal
Makes it easier to change datasource, also makes it such that the list
of all datasources doesn't need to be loaded upfront.
* Adding more metadata
* Js translation
* js tran
* js trans
* js trans
* js tran
* js trans
* js trans
* js tran
* js translation
* js trans
* js translation
* try load language pack async
* Backend translations things
* create language pack inside common data
* performance improvement for js i18n.
- js bundle should not contain localized content
- we populate translation content from server-side, in boostrap.common.language_pack
- in client-side, use promise to wrap around translation content. text will be translated after translation content arrived/parsed.
- fix linting
* fix Timer unit test
* 1. add global hook for all tests, to make translation pack avaialble before each test starts.
2. fix unit test for Timer component
3. remove noused method get_locale, and modules
4. fix page reload after user change page language
* parse and build i18n dictionary as a module
* fix sync-backend task, which should run without DOM
2017-09-20 15:37:33 -04:00
2018-09-19 13:01:51 -04:00
### Creating a new language dictionary
js translation -- performance improvment (#3390)
* Chinese page
* Using react-intl-universal to improve multi language in react page
* Using react-intl-universal to improve multi language in react page
* react_intl_universal
* change
* change
* change
* change
* change
* change
* change
* merge
* multiple page in js
* merge
* merge
* merge
* merge
* Js Translations
* JS Translation
* JS Translations
* Js translation
* JS translations
* JS translations
* Js translaion
* JS en Translation
* JS Translation
* upgrade document
Fixing the damn build (#3179)
* Fixing the build
* Going deeper
[bugfix] only filterable columns should show up in FilterBox list (#3105)
* [bugfix] only filterable columns should show up in FilterBox list
* Touchups
Datasource cannot be empty (#3035)
add title description to model view (#3045)
* add title description to model view
* add missing import
Add 'show/hide totals' option to pivot table vis (#3101)
[bugfix] numeric value for date fields in table viz (#3036)
Bug was present only when using the NOT GROUPED BY option
fixes https://github.com/ApacheInfra/superset/issues/3027
fix hive.fetch_logs (#2968)
add Zalando to the list of organizations (#3171)
docs: fixup installation examples code indentation (#3169)
[bugfix] fix bar order (#3180)
[bugfix] visualize flow error: 'Metric x is not valid' (#3181)
The metric name in the frontend doesn't match the one generated on the
backend. It turns out the explore view will default to the first
metric so specifying one isn't needed.
Fix the segment interval for pulling metadata (#3174)
The end of the interval would be on the truncated today date, which
means that you will exclude today. If your realtime ingestion job
runs shorter than a day, the metadata cannot be pulled from the
druid cluster.
Bump cryptography to 1.9 (#3065)
As 1.7.2 doesn't compile here with openssl 1.1.0f
Escaping the user's SQL in the explore view (#3186)
* Escaping the user's SQL in the explore view
When executing SQL from SQL Lab, we use a lower level API to the
database which doesn't require escaping the SQL. When going through
the explore view, the stack chain leading to the same method may need
escaping depending on how the DBAPI driver is written, and that is the
case for Presto (and perhaps other drivers).
* Using regex to avoid doubling doubles
[sqllab] improve Hive support (#3187)
* [sqllab] improve Hive support
* Fix "Transport not open" bug
* Getting progress bar to show
* Bump pyhive to 0.4.0
* Getting [Track Job] button to show
* Fix testzz
Add BigQuery engine specifications (#3193)
As contributed by @mxmzdlv on issue #945
[bugfix] fix merge conflict that broke Hive support (#3196)
Adding 'apache' to docs (#3194)
[druid] Allow custom druid postaggregators (#3146)
* [druid] Allow custom druid postaggregators
Also, fix the postaggregation for approxHistogram quantiles so it adds
the dependent field and that can show up in the graphs/tables.
In general, postAggregators add significant power, we should probably
support including custom postAggregators. Plywood has standard
postAggregators here, and a customAggregator escape hatch that allows
you to define custom postAggregators.
This commit adds a similar capability for Superset and a additional
field/fields/fieldName breakdown of the typical naming for dependent
aggregations, which should make it significantly easier to develop
approxHistogram and custom postAggregation-required dashboards.
* [druid] Minor style cleanup in tests file.
* [druid] Apply code review suggestions
* break out CustomPostAggregator into separate class. This just cleans
up the creation of the postaggregator a little bit.
* minor style issues.
* move the function around so the git diff is more readable
add combine config for metrics in pivot table (#3086)
* add combine config for metrics in pivot table
* change method to stack/unstack
* update backendSync
Autofocus search input in VizTypeControl modal onEnter (#2929)
Speed up JS build time (#3203)
Also bumping a few related libs
JS Translation
JS translations
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
[docs] update url in CONTRIBUTING.md (#3212)
[sqllab/cosmetics] add margin-top for labels in query history (#3222)
[explore] nvd3 sort values in rich tooltip (#3197)
[sqllab] fix UI shows 'The query returned no results' momentarily (#3214)
this is visible when running async queries between the fetching and
success state as the rows are getting cached in the component
[explore] DatasourceControl to pick datasource in modal (#3210)
* [explore] DatasourceControl to pick datasource in modal
Makes it easier to change datasource, also makes it such that the list
of all datasources doesn't need to be loaded upfront.
* Adding more metadata
* Js translation
* js tran
* js trans
* js trans
* js tran
* js trans
* js trans
* js tran
* js translation
* js trans
* js translation
* try load language pack async
* Backend translations things
* create language pack inside common data
* performance improvement for js i18n.
- js bundle should not contain localized content
- we populate translation content from server-side, in boostrap.common.language_pack
- in client-side, use promise to wrap around translation content. text will be translated after translation content arrived/parsed.
- fix linting
* fix Timer unit test
* 1. add global hook for all tests, to make translation pack avaialble before each test starts.
2. fix unit test for Timer component
3. remove noused method get_locale, and modules
4. fix page reload after user change page language
* parse and build i18n dictionary as a module
* fix sync-backend task, which should run without DOM
2017-09-20 15:37:33 -04:00
2018-09-19 13:01:51 -04:00
To create a dictionary for a new language, run the following, where `LANGUAGE_CODE` is replaced with
the language code for your target language, e.g. `es` (see [Flask AppBuilder i18n documentation ](https://flask-appbuilder.readthedocs.io/en/latest/i18n.html ) for more details):
js translation -- performance improvment (#3390)
* Chinese page
* Using react-intl-universal to improve multi language in react page
* Using react-intl-universal to improve multi language in react page
* react_intl_universal
* change
* change
* change
* change
* change
* change
* change
* merge
* multiple page in js
* merge
* merge
* merge
* merge
* Js Translations
* JS Translation
* JS Translations
* Js translation
* JS translations
* JS translations
* Js translaion
* JS en Translation
* JS Translation
* upgrade document
Fixing the damn build (#3179)
* Fixing the build
* Going deeper
[bugfix] only filterable columns should show up in FilterBox list (#3105)
* [bugfix] only filterable columns should show up in FilterBox list
* Touchups
Datasource cannot be empty (#3035)
add title description to model view (#3045)
* add title description to model view
* add missing import
Add 'show/hide totals' option to pivot table vis (#3101)
[bugfix] numeric value for date fields in table viz (#3036)
Bug was present only when using the NOT GROUPED BY option
fixes https://github.com/ApacheInfra/superset/issues/3027
fix hive.fetch_logs (#2968)
add Zalando to the list of organizations (#3171)
docs: fixup installation examples code indentation (#3169)
[bugfix] fix bar order (#3180)
[bugfix] visualize flow error: 'Metric x is not valid' (#3181)
The metric name in the frontend doesn't match the one generated on the
backend. It turns out the explore view will default to the first
metric so specifying one isn't needed.
Fix the segment interval for pulling metadata (#3174)
The end of the interval would be on the truncated today date, which
means that you will exclude today. If your realtime ingestion job
runs shorter than a day, the metadata cannot be pulled from the
druid cluster.
Bump cryptography to 1.9 (#3065)
As 1.7.2 doesn't compile here with openssl 1.1.0f
Escaping the user's SQL in the explore view (#3186)
* Escaping the user's SQL in the explore view
When executing SQL from SQL Lab, we use a lower level API to the
database which doesn't require escaping the SQL. When going through
the explore view, the stack chain leading to the same method may need
escaping depending on how the DBAPI driver is written, and that is the
case for Presto (and perhaps other drivers).
* Using regex to avoid doubling doubles
[sqllab] improve Hive support (#3187)
* [sqllab] improve Hive support
* Fix "Transport not open" bug
* Getting progress bar to show
* Bump pyhive to 0.4.0
* Getting [Track Job] button to show
* Fix testzz
Add BigQuery engine specifications (#3193)
As contributed by @mxmzdlv on issue #945
[bugfix] fix merge conflict that broke Hive support (#3196)
Adding 'apache' to docs (#3194)
[druid] Allow custom druid postaggregators (#3146)
* [druid] Allow custom druid postaggregators
Also, fix the postaggregation for approxHistogram quantiles so it adds
the dependent field and that can show up in the graphs/tables.
In general, postAggregators add significant power, we should probably
support including custom postAggregators. Plywood has standard
postAggregators here, and a customAggregator escape hatch that allows
you to define custom postAggregators.
This commit adds a similar capability for Superset and a additional
field/fields/fieldName breakdown of the typical naming for dependent
aggregations, which should make it significantly easier to develop
approxHistogram and custom postAggregation-required dashboards.
* [druid] Minor style cleanup in tests file.
* [druid] Apply code review suggestions
* break out CustomPostAggregator into separate class. This just cleans
up the creation of the postaggregator a little bit.
* minor style issues.
* move the function around so the git diff is more readable
add combine config for metrics in pivot table (#3086)
* add combine config for metrics in pivot table
* change method to stack/unstack
* update backendSync
Autofocus search input in VizTypeControl modal onEnter (#2929)
Speed up JS build time (#3203)
Also bumping a few related libs
JS Translation
JS translations
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
js translation
fix issue 3204 (#3205)
[bugfix] capture Hive job_id pre-url transformation (#3213)
[docs] update url in CONTRIBUTING.md (#3212)
[sqllab/cosmetics] add margin-top for labels in query history (#3222)
[explore] nvd3 sort values in rich tooltip (#3197)
[sqllab] fix UI shows 'The query returned no results' momentarily (#3214)
this is visible when running async queries between the fetching and
success state as the rows are getting cached in the component
[explore] DatasourceControl to pick datasource in modal (#3210)
* [explore] DatasourceControl to pick datasource in modal
Makes it easier to change datasource, also makes it such that the list
of all datasources doesn't need to be loaded upfront.
* Adding more metadata
* Js translation
* js tran
* js trans
* js trans
* js tran
* js trans
* js trans
* js tran
* js translation
* js trans
* js translation
* try load language pack async
* Backend translations things
* create language pack inside common data
* performance improvement for js i18n.
- js bundle should not contain localized content
- we populate translation content from server-side, in boostrap.common.language_pack
- in client-side, use promise to wrap around translation content. text will be translated after translation content arrived/parsed.
- fix linting
* fix Timer unit test
* 1. add global hook for all tests, to make translation pack avaialble before each test starts.
2. fix unit test for Timer component
3. remove noused method get_locale, and modules
4. fix page reload after user change page language
* parse and build i18n dictionary as a module
* fix sync-backend task, which should run without DOM
2017-09-20 15:37:33 -04:00
2018-09-19 13:01:51 -04:00
```bash
pip install -r superset/translations/requirements.txt
pybabel init -i superset/translations/messages.pot -d superset/translations -l LANGUAGE_CODE
```
2016-09-21 12:52:05 -04:00
2018-09-19 13:01:51 -04:00
Then, [extract strings for the new language ](#extracting-new-strings-for-translation ).
2017-11-01 23:13:22 -04:00
2018-09-19 13:01:51 -04:00
## Tips
### Adding a new datasource
2016-09-21 12:52:05 -04:00
2016-11-10 02:08:22 -05:00
1. Create Models and Views for the datasource, add them under superset folder, like a new my_models.py
2016-09-21 12:52:05 -04:00
with models for cluster, datasources, columns and metrics and my_views.py with clustermodelview
and datasourcemodelview.
2018-09-19 13:01:51 -04:00
1. Create DB migration files for the new models
2016-09-21 12:52:05 -04:00
2018-09-19 13:01:51 -04:00
1. Specify this variable to add the datasource model and from which module it is from in config.py:
2016-09-21 12:52:05 -04:00
For example:
2018-09-19 13:01:51 -04:00
```python
ADDITIONAL_MODULE_DS_MAP = {'superset.my_models': ['MyDatasource', 'MyOtherDatasource']}
```
2016-09-21 12:52:05 -04:00
2016-11-10 02:08:22 -05:00
This means it'll register MyDatasource and MyOtherDatasource in superset.my_models module in the source registry.
2017-06-22 15:37:22 -04:00
2018-09-19 13:01:51 -04:00
### Creating a new visualization type
2017-06-22 15:37:22 -04:00
Here's an example as a Github PR with comments that describe what the
different sections of the code do:
2017-08-01 11:54:35 -04:00
https://github.com/apache/incubator-superset/pull/3013
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
### Adding a DB migration
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
1. Alter the model you want to change. This example will add a `Column` Annotations model.
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
[Example commit ](https://github.com/apache/incubator-superset/commit/6c25f549384d7c2fc288451222e50493a7b14104 )
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
1. Generate the migration file
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset db migrate -m 'add_metadata_column_to_annotation_model.py'
```
2017-08-15 19:37:40 -04:00
2018-09-19 13:01:51 -04:00
This will generate a file in `migrations/version/{SHA}_this_will_be_in_the_migration_filename.py` .
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
[Example commit ](https://github.com/apache/incubator-superset/commit/d3e83b0fd572c9d6c1297543d415a332858e262 )
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
1. Upgrade the DB
2018-04-06 19:14:14 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset db upgrade
```
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
The output should look like this:
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
```
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 1a1d627ebd8e -> 40a0a483dd12, add_metadata_column_to_annotation_model.py
```
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
1. Add column to view
2018-03-27 18:48:57 -04:00
2018-09-19 13:01:51 -04:00
Since there is a new column, we need to add it to the AppBuilder Model view.
2018-07-23 12:43:09 -04:00
2018-09-19 13:01:51 -04:00
[Example commit ](https://github.com/apache/incubator-superset/pull/5745/commits/6220966e2a0a0cf3e6d87925491f8920fe8a3458 )
2018-07-23 12:43:09 -04:00
2018-09-19 13:01:51 -04:00
### Merging DB migrations
2018-07-23 12:43:09 -04:00
2018-09-19 13:01:51 -04:00
When two DB migrations collide, you'll get an error message like this one:
2018-07-23 12:43:09 -04:00
```
2018-09-19 13:01:51 -04:00
alembic.util.exc.CommandError: Multiple head revisions are present for
given argument 'head'; please specify a specific target
revision, '< branchname > @head' to narrow to a specific head,
or 'heads' for all heads`
2018-07-23 12:43:09 -04:00
```
2018-09-19 13:01:51 -04:00
To fix it:
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
1. Get the migration heads
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset db heads
```
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
This should list two or more migration hashes.
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
1. Create a new merge migration
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset db merge {HASH1} {HASH2}
```
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
1. Upgrade the DB to the new checkpoint
2018-09-12 13:16:58 -04:00
2018-09-19 13:01:51 -04:00
```bash
superset db upgrade
```