pt
/
superset
mirror of https://github.com/apache/superset.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add docs for GLOBAL_ASYNC_QUERIES (SIP-39) (#12573)

This commit is contained in:
Rob DiCiuccio 2021-01-19 13:57:25 -08:00 committed by GitHub
parent 853d34beba
commit 10c2b09942
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 50 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
CONTRIBUTING.md
View File

@ -83,6 +83,7 @@ little bit helps, and credit will always be given.
- [Adding a DB migration](#adding-a-db-migration)
- [Merging DB migrations](#merging-db-migrations)
- [SQL Lab Async](#sql-lab-async)
- [Async Chart Queries](#async-chart-queries)
- [Chart Parameters](#chart-parameters)
- [Datasource & Chart Type](#datasource--chart-type)
- [Time](#time)
@ -978,6 +979,29 @@ Note that:
to your production environment, and use the similar broker as well as
results backend configuration
### Async Chart Queries
It's possible to configure database queries for charts to operate in `async` mode. This is especially useful for dashboards with many charts that may otherwise be affected by browser connection limits. To enable async queries for dashboards and Explore, the following dependencies are required:
- Redis 5.0+ (the feature utilizes [Redis Streams](https://redis.io/topics/streams-intro))
- Cache backends enabled via the `CACHE_CONFIG` and `DATA_CACHE_CONFIG` config settings
- Celery workers configured and running to process async tasks
The following configuration settings are available for async queries (see config.py for default values)
- `GLOBAL_ASYNC_QUERIES` (feature flag) - enable or disable async query operation
- `GLOBAL_ASYNC_QUERIES_REDIS_CONFIG` - Redis connection info
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX` - the prefix used with Redis Streams
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` - the maximum number of events for each user-specific event stream (FIFO eviction)
- `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE` - the maximum number of events for all users (FIFO eviction)
- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME` - the async query feature uses a [JWT](https://tools.ietf.org/html/rfc7519) cookie for authentication, this setting is the cookie's name
- `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE` - JWT cookie secure option
- `GLOBAL_ASYNC_QUERIES_JWT_SECRET` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security
- `GLOBAL_ASYNC_QUERIES_TRANSPORT` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions
- `GLOBAL_ASYNC_QUERIES_POLLING_DELAY` - the time (in ms) between polling requests
More information on the async query feature can be found in [SIP-39](https://github.com/apache/superset/issues/9190).
## Chart Parameters
Chart parameters are stored as a JSON encoded string the `slices.params` column and are often referenced throughout the code as form-data. Currently the form-data is neither versioned nor typed as thus is somewhat free-formed. Note in the future there may be merit in using something like [JSON Schema](https://json-schema.org/) to both annotate and validate the JSON object in addition to using a Mypy `TypedDict` (introduced in Python 3.8) for typing the form-data in the backend. This section serves as a potential primer for that work.

26
docs/installation.rst
View File

@ -1113,6 +1113,32 @@ serialization. This can be disabled by setting ``RESULTS_BACKEND_USE_MSGPACK = F
in your configuration, should any issues arise. Please clear your existing results
cache store when upgrading an existing environment.
**Async queries for dashboards and Explore**
It's also possible to configure database queries for charts to operate in `async` mode.
This is especially useful for dashboards with many charts that may otherwise be affected
by browser connection limits. To enable async queries for dashboards and Explore, the
following dependencies are required:
- Redis 5.0+ (the feature utilizes `Redis Streams <https://redis.io/topics/streams-intro>`_)
- Cache backends enabled via the ``CACHE_CONFIG`` and ``DATA_CACHE_CONFIG`` config settings
- Celery workers configured and running to process async tasks
The following configuration settings are available for async queries (see config.py for default values)
- ``GLOBAL_ASYNC_QUERIES`` (feature flag) - enable or disable async query operation
- ``GLOBAL_ASYNC_QUERIES_REDIS_CONFIG`` - Redis connection info
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX`` - the prefix used with Redis Streams
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT`` - the maximum number of events for each user-specific event stream (FIFO eviction)
- ``GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE`` - the maximum number of events for all users (FIFO eviction)
- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME`` - the async query feature uses a `JWT <https://tools.ietf.org/html/rfc7519>`_ cookie for authentication, this setting is the cookie's name
- ``GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE`` - JWT cookie secure option
- ``GLOBAL_ASYNC_QUERIES_JWT_SECRET`` - JWT's use a secret key to sign and validate the contents. This value should be at least 32 bytes and have sufficient randomness for proper security
- ``GLOBAL_ASYNC_QUERIES_TRANSPORT`` - currently the only available option is (HTTP) `polling`, but support for a WebSocket will be added in future versions
- ``GLOBAL_ASYNC_QUERIES_POLLING_DELAY`` - the time (in ms) between polling requests
More information on the async query feature can be found in `SIP-39 <https://github.com/apache/superset/issues/9190>`_.
**Important notes**
* It is important that all the worker nodes and web servers in