2019-01-22 11:21:13 -05:00
|
|
|
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
or more contributor license agreements. See the NOTICE file
|
|
|
|
distributed with this work for additional information
|
|
|
|
regarding copyright ownership. The ASF licenses this file
|
|
|
|
to you under the Apache License, Version 2.0 (the
|
|
|
|
"License"); you may not use this file except in compliance
|
|
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
|
|
software distributed under the License is distributed on an
|
|
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
|
|
KIND, either express or implied. See the License for the
|
|
|
|
specific language governing permissions and limitations
|
|
|
|
under the License.
|
|
|
|
|
2016-10-26 14:09:27 -04:00
|
|
|
SQL Lab
|
|
|
|
=======
|
|
|
|
|
|
|
|
SQL Lab is a modern, feature-rich SQL IDE written in
|
|
|
|
`React <https://facebook.github.io/react/>`_.
|
|
|
|
|
2018-04-13 13:23:27 -04:00
|
|
|
------
|
|
|
|
|
2020-02-09 20:53:56 -05:00
|
|
|
.. image:: _static/images/screenshots/sqllab.png
|
2018-04-13 13:23:27 -04:00
|
|
|
|
|
|
|
------
|
2016-10-26 14:09:27 -04:00
|
|
|
|
|
|
|
Feature Overview
|
|
|
|
----------------
|
|
|
|
- Connects to just about any database backend
|
|
|
|
- A multi-tab environment to work on multiple queries at a time
|
2016-11-10 02:08:22 -05:00
|
|
|
- A smooth flow to visualize your query results using Superset's rich
|
2016-10-26 14:09:27 -04:00
|
|
|
visualization capabilities
|
|
|
|
- Browse database metadata: tables, columns, indexes, partitions
|
|
|
|
- Support for long-running queries
|
2017-06-01 00:26:17 -04:00
|
|
|
|
2018-12-01 12:34:49 -05:00
|
|
|
- uses the `Celery distributed queue <http://www.celeryproject.org/>`_
|
2016-10-26 14:09:27 -04:00
|
|
|
to dispatch query handling to workers
|
|
|
|
- supports defining a "results backend" to persist query results
|
2017-06-01 00:26:17 -04:00
|
|
|
|
2016-10-26 14:09:27 -04:00
|
|
|
- A search engine to find queries executed in the past
|
|
|
|
- Supports templating using the
|
|
|
|
`Jinja templating language <http://jinja.pocoo.org/docs/dev/>`_
|
|
|
|
which allows for using macros in your SQL code
|
|
|
|
|
2016-11-09 11:42:13 -05:00
|
|
|
Extra features
|
|
|
|
--------------
|
|
|
|
- Hit ``alt + enter`` as a keyboard shortcut to run your query
|
2016-10-26 14:09:27 -04:00
|
|
|
|
|
|
|
Templating with Jinja
|
|
|
|
---------------------
|
|
|
|
|
|
|
|
.. code-block:: sql
|
|
|
|
|
|
|
|
SELECT *
|
|
|
|
FROM some_table
|
2019-08-07 12:35:16 -04:00
|
|
|
WHERE partition_key = '{{ presto.first_latest_partition('some_table') }}'
|
2016-10-26 14:09:27 -04:00
|
|
|
|
|
|
|
Templating unleashes the power and capabilities of a
|
|
|
|
programming language within your SQL code.
|
|
|
|
|
|
|
|
Templates can also be used to write generic queries that are
|
|
|
|
parameterized so they can be re-used easily.
|
|
|
|
|
|
|
|
|
|
|
|
Available macros
|
|
|
|
''''''''''''''''
|
|
|
|
|
|
|
|
We expose certain modules from Python's standard library in
|
2016-11-10 02:08:22 -05:00
|
|
|
Superset's Jinja context:
|
2017-12-19 18:55:58 -05:00
|
|
|
|
2016-10-26 14:09:27 -04:00
|
|
|
- ``time``: ``time``
|
|
|
|
- ``datetime``: ``datetime.datetime``
|
2020-09-04 11:37:14 -04:00
|
|
|
- ``uuid1``: ``uuid1``
|
|
|
|
- ``uuid3``: ``uuid3``
|
|
|
|
- ``uuid4``: ``uuid4``
|
|
|
|
- ``uuid5``: ``uuid5``
|
2016-10-26 14:09:27 -04:00
|
|
|
- ``random``: ``random``
|
|
|
|
- ``relativedelta``: ``dateutil.relativedelta.relativedelta``
|
|
|
|
|
|
|
|
`Jinja's builtin filters <http://jinja.pocoo.org/docs/dev/templates/>`_ can be also be applied where needed.
|
|
|
|
|
2020-05-19 00:27:25 -04:00
|
|
|
.. autoclass:: superset.jinja_context.ExtraCache
|
|
|
|
:members:
|
2017-12-19 18:55:58 -05:00
|
|
|
|
2018-09-17 12:36:47 -04:00
|
|
|
.. autofunction:: superset.jinja_context.filter_values
|
|
|
|
|
2019-07-03 12:54:03 -04:00
|
|
|
.. autoclass:: superset.jinja_context.PrestoTemplateProcessor
|
|
|
|
:members:
|
|
|
|
|
|
|
|
.. autoclass:: superset.jinja_context.HiveTemplateProcessor
|
|
|
|
:members:
|
|
|
|
|
2017-12-19 18:55:58 -05:00
|
|
|
Extending macros
|
|
|
|
''''''''''''''''
|
|
|
|
|
2021-01-06 06:45:19 -05:00
|
|
|
As mentioned in the `Installation & Configuration <https://superset.apache.org/installation.html#installation-configuration>`_ documentation,
|
2017-12-19 18:55:58 -05:00
|
|
|
it's possible for administrators to expose more more macros in their
|
|
|
|
environment using the configuration variable ``JINJA_CONTEXT_ADDONS``.
|
|
|
|
All objects referenced in this dictionary will become available for users
|
|
|
|
to integrate in their queries in **SQL Lab**.
|
2019-09-17 12:09:40 -04:00
|
|
|
|
2020-04-07 16:00:42 -04:00
|
|
|
Customize templating
|
|
|
|
''''''''''''''''''''
|
|
|
|
|
2021-01-06 06:45:19 -05:00
|
|
|
As mentioned in the `Installation & Configuration <https://superset.apache.org/installation.html#sql-lab>`__ documentation,
|
2020-04-07 16:00:42 -04:00
|
|
|
it's possible for administrators to overwrite Jinja templating with your customized
|
|
|
|
template processor using the configuration variable ``CUSTOM_TEMPLATE_PROCESSORS``.
|
|
|
|
The template processors referenced in the dictionary will overwrite default Jinja template processors
|
|
|
|
of the specified database engines.
|
|
|
|
|
2019-09-17 12:09:40 -04:00
|
|
|
Query cost estimation
|
|
|
|
'''''''''''''''''''''
|
|
|
|
|
|
|
|
Some databases support ``EXPLAIN`` queries that allow users to estimate the cost
|
|
|
|
of queries before executing this. Currently, Presto is supported in SQL Lab. To
|
|
|
|
enable query cost estimation, add the following keys to the "Extra" field in the
|
|
|
|
database configuration:
|
|
|
|
|
2019-12-03 04:42:57 -05:00
|
|
|
.. code-block:: text
|
2019-09-17 12:09:40 -04:00
|
|
|
|
|
|
|
{
|
|
|
|
"version": "0.319",
|
2019-12-03 04:42:57 -05:00
|
|
|
"cost_estimate_enabled": true
|
2019-09-17 12:09:40 -04:00
|
|
|
...
|
|
|
|
}
|
|
|
|
|
|
|
|
Here, "version" should be the version of your Presto cluster. Support for this
|
|
|
|
functionality was introduced in Presto 0.319.
|
2019-10-17 09:54:02 -04:00
|
|
|
|
2019-11-04 14:08:00 -05:00
|
|
|
You also need to enable the feature flag in your `superset_config.py`, and you
|
|
|
|
can optionally specify a custom formatter. Eg:
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
2019-11-05 13:55:35 -05:00
|
|
|
def presto_query_cost_formatter(cost_estimate: List[Dict[str, float]]) -> List[Dict[str, str]]:
|
2019-11-04 14:08:00 -05:00
|
|
|
"""
|
|
|
|
Format cost estimate returned by Presto.
|
|
|
|
|
|
|
|
:param cost_estimate: JSON estimate from Presto
|
|
|
|
:return: Human readable cost estimate
|
|
|
|
"""
|
|
|
|
# Convert cost to dollars based on CPU and network cost. These coefficients are just
|
|
|
|
# examples, they need to be estimated based on your infrastructure.
|
2019-11-05 13:55:35 -05:00
|
|
|
cpu_coefficient = 2e-12
|
|
|
|
network_coefficient = 1e-12
|
2019-11-04 14:08:00 -05:00
|
|
|
|
2019-11-05 13:55:35 -05:00
|
|
|
cost = 0
|
|
|
|
for row in cost_estimate:
|
|
|
|
cost += row.get("cpuCost", 0) * cpu_coefficient
|
|
|
|
cost += row.get("networkCost", 0) * network_coefficient
|
2019-11-04 14:08:00 -05:00
|
|
|
|
2019-11-05 13:55:35 -05:00
|
|
|
return [{"Cost": f"US$ {cost:.2f}"}]
|
2019-11-04 14:08:00 -05:00
|
|
|
|
|
|
|
|
2021-04-08 14:05:59 -04:00
|
|
|
FEATURE_FLAGS = {
|
2019-11-05 13:55:35 -05:00
|
|
|
"ESTIMATE_QUERY_COST": True,
|
|
|
|
"QUERY_COST_FORMATTERS_BY_ENGINE": {"presto": presto_query_cost_formatter},
|
|
|
|
}
|
2019-11-04 14:08:00 -05:00
|
|
|
|
2019-10-17 09:54:02 -04:00
|
|
|
.. _ref_ctas_engine_config:
|
|
|
|
|
|
|
|
Create Table As (CTAS)
|
|
|
|
''''''''''''''''''''''
|
|
|
|
|
|
|
|
You can use ``CREATE TABLE AS SELECT ...`` statements on SQLLab. This feature can be toggled on
|
|
|
|
and off at the database configuration level.
|
|
|
|
|
|
|
|
Note that since ``CREATE TABLE..`` belongs to a SQL DDL category. Specifically on PostgreSQL, DDL is transactional,
|
|
|
|
this means that to properly use this feature you have to set ``autocommit`` to true on your engine parameters:
|
|
|
|
|
2019-12-03 04:42:57 -05:00
|
|
|
.. code-block:: text
|
2019-10-17 09:54:02 -04:00
|
|
|
|
|
|
|
{
|
|
|
|
...
|
|
|
|
"engine_params": {"isolation_level":"AUTOCOMMIT"},
|
|
|
|
...
|
|
|
|
}
|