2016-03-16 23:25:41 -04:00
Installation & Configuration
============================
Getting Started
---------------
2017-02-23 16:47:15 -05:00
Superset is tested against Python `` 2.7 `` and Python `` 3.4 `` .
Airbnb currently uses 2.7.* in production. We do not plan on supporting
Python `` 2.6 `` .
2016-03-16 23:25:41 -04:00
2016-04-01 11:33:28 -04:00
OS dependencies
---------------
2016-11-10 02:08:22 -05:00
Superset stores database connection information in its metadata database.
2016-04-01 11:33:28 -04:00
For that purpose, we use the `` cryptography `` Python library to encrypt
connection passwords. Unfortunately this library has OS level dependencies.
You may want to attempt the next step
2016-11-10 02:08:22 -05:00
("Superset installation and initialization") and come back to this step if
2016-04-01 11:33:28 -04:00
you encounter an error.
Here's how to install them:
2016-04-01 11:37:19 -04:00
For **Debian** and **Ubuntu** , the following command will ensure that
the required dependencies are installed: ::
2016-04-01 11:33:28 -04:00
2016-10-05 16:00:38 -04:00
sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-pip libsasl2-dev libldap2-dev
2016-04-01 11:33:28 -04:00
2016-04-01 11:37:19 -04:00
For **Fedora** and **RHEL-derivatives** , the following command will ensure
that the required dependencies are installed: ::
2016-06-02 15:34:36 -04:00
2016-04-06 21:11:24 -04:00
sudo yum upgrade python-setuptools
2017-02-27 13:19:33 -05:00
sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel libsasl2-devel openldap-devel
2016-04-01 11:37:19 -04:00
2016-04-06 21:11:24 -04:00
**OSX** , system python is not recommended. brew's python also ships with pip ::
2016-04-01 11:37:19 -04:00
2016-04-06 21:11:24 -04:00
brew install pkg-config libffi openssl python
2017-04-26 17:20:27 -04:00
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography==1.7.2
2016-04-01 11:37:19 -04:00
**Windows** isn't officially supported at this point, but if you want to
2016-04-06 21:11:24 -04:00
attempt it, download `get-pip.py <https://bootstrap.pypa.io/get-pip.py> `_ , and run `` python get-pip.py `` which may need admin access. Then run the following: ::
2016-04-01 11:33:28 -04:00
C:\> pip install cryptography
2016-04-06 11:23:27 -04:00
# You may also have to create C:\Temp
C:\> md C:\Temp
2016-08-09 18:56:36 -04:00
Python virtualenv
-----------------
2016-11-10 02:08:22 -05:00
It is recommended to install Superset inside a virtualenv. Python 3 already ships virtualenv, for
2016-08-09 18:56:36 -04:00
Python 2 you need to install it. If it's packaged for your operating systems install it from there
otherwise you can install from pip: ::
pip install virtualenv
You can create and activate a virtualenv by: ::
# virtualenv is shipped in Python 3 as pyvenv
virtualenv venv
. ./venv/bin/activate
On windows the syntax for activating it is a bit different: ::
venv\Scripts\activate
Once you activated your virtualenv everything you are doing is confined inside the virtualenv.
To exit a virtualenv just type `` deactivate `` .
2016-04-01 11:33:28 -04:00
2016-10-05 00:59:40 -04:00
Python's setup tools and pip
----------------------------
Put all the chances on your side by getting the very latest `` pip ``
and `` setuptools `` libraries.::
pip install --upgrade setuptools pip
2016-11-10 02:08:22 -05:00
Superset installation and initialization
2016-11-10 14:27:56 -05:00
----------------------------------------
2016-11-10 02:08:22 -05:00
Follow these few simple steps to install Superset.::
2016-03-16 23:25:41 -04:00
2016-11-10 02:08:22 -05:00
# Install superset
pip install superset
2016-03-16 23:25:41 -04:00
2017-01-10 12:06:38 -05:00
# Create an admin user (you will be prompted to set username, first and last name before setting a password)
2016-11-10 02:08:22 -05:00
fabmanager create-admin --app superset
2016-03-16 23:25:41 -04:00
# Initialize the database
2016-11-10 02:08:22 -05:00
superset db upgrade
2016-03-16 23:25:41 -04:00
# Load some data to play with
2016-11-10 02:08:22 -05:00
superset load_examples
2016-03-16 23:25:41 -04:00
2016-10-07 13:57:26 -04:00
# Create default roles and permissions
2016-11-10 02:08:22 -05:00
superset init
2016-10-07 13:57:26 -04:00
2017-01-04 17:43:56 -05:00
# Start the web server on port 8088, use -p to bind to another port
superset runserver
2016-06-16 12:15:40 -04:00
# To start a development web server, use the -d switch
2016-11-10 02:08:22 -05:00
# superset runserver -d
2016-03-16 23:25:41 -04:00
After installation, you should be able to point your browser to the right
2016-04-06 23:13:04 -04:00
hostname:port `http://localhost:8088 <http://localhost:8088> `_ , login using
2016-03-16 23:25:41 -04:00
the credential you entered while creating the admin account, and navigate to
`Menu -> Admin -> Refresh Metadata` . This action should bring in all of
2016-11-10 02:08:22 -05:00
your datasources for Superset to be aware of, and they should show up in
2016-03-16 23:25:41 -04:00
`Menu -> Datasources` , from where you can start playing with your data!
2016-11-10 02:08:22 -05:00
Please note that *gunicorn* , Superset default application server, does not
2016-10-05 15:14:09 -04:00
work on Windows so you need to use the development web server.
The development web server though is not intended to be used on production systems
so better use a supported platform that can run *gunicorn* .
2016-06-20 12:24:49 -04:00
Configuration behind a load balancer
------------------------------------
2016-11-10 02:08:22 -05:00
If you are running superset behind a load balancer or reverse proxy (e.g. NGINX
2016-06-20 12:24:49 -04:00
or ELB on AWS), you may need to utilise a healthcheck endpoint so that your
2016-11-10 02:08:22 -05:00
load balancer knows if your superset instance is running. This is provided
2016-06-20 12:24:49 -04:00
at `` /health `` which will return a 200 response containing "OK" if the
webserver is running.
2016-09-20 15:24:15 -04:00
If the load balancer is inserting X-Forwarded-For/X-Forwarded-Proto headers, you
2016-11-10 02:08:22 -05:00
should set `ENABLE_PROXY_FIX = True` in the superset config file to extract and use
2016-09-20 15:24:15 -04:00
the headers.
2016-03-16 23:25:41 -04:00
Configuration
-------------
To configure your application, you need to create a file (module)
2016-11-10 02:08:22 -05:00
`` superset_config.py `` and make sure it is in your PYTHONPATH. Here are some
2016-03-16 23:25:41 -04:00
of the parameters you can copy / paste in that configuration module: ::
#---------------------------------------------------------
2016-11-10 02:08:22 -05:00
# Superset specific config
2016-03-16 23:25:41 -04:00
#---------------------------------------------------------
ROW_LIMIT = 5000
2016-11-10 02:08:22 -05:00
SUPERSET_WORKERS = 4
2016-03-16 23:25:41 -04:00
2016-11-10 02:08:22 -05:00
SUPERSET_WEBSERVER_PORT = 8088
2016-03-16 23:25:41 -04:00
#---------------------------------------------------------
#---------------------------------------------------------
# Flask App Builder configuration
#---------------------------------------------------------
# Your App secret key
SECRET_KEY = '\2\1thisismyscretkey\1\2\e\y\y\h'
2016-04-01 19:41:02 -04:00
# The SQLAlchemy connection string to your database backend
# This connection defines the path to the database that stores your
2016-11-10 02:08:22 -05:00
# superset metadata (slices, connections, tables, dashboards, ...).
2016-04-01 19:41:02 -04:00
# Note that the connection information to connect to the datasources
# you want to explore are managed directly in the web UI
2016-11-10 02:08:22 -05:00
SQLALCHEMY_DATABASE_URI = 'sqlite:////path/to/superset.db'
2016-03-16 23:25:41 -04:00
# Flask-WTF flag for CSRF
CSRF_ENABLED = True
2016-08-17 11:04:39 -04:00
# Set this API key to enable Mapbox visualizations
MAPBOX_API_KEY = ''
2016-03-16 23:25:41 -04:00
This file also allows you to define configuration parameters used by
2016-11-10 02:08:22 -05:00
Flask App Builder, the web framework used by Superset. Please consult
2016-03-16 23:25:41 -04:00
the `Flask App Builder Documentation
<http://flask-appbuilder.readthedocs.org/en/latest/config.html> `_
2016-11-10 02:08:22 -05:00
for more information on how to configure Superset.
2016-03-16 23:25:41 -04:00
2016-08-10 11:15:19 -04:00
Please make sure to change:
2016-11-10 02:08:22 -05:00
* *SQLALCHEMY_DATABASE_URI* , by default it is stored at *~/.superset/superset.db*
2016-08-10 11:15:19 -04:00
* *SECRET_KEY* , to a long random string
2016-04-08 11:44:28 -04:00
Database dependencies
---------------------
2016-11-10 02:08:22 -05:00
Superset does not ship bundled with connectivity to databases, except
2016-04-08 11:44:28 -04:00
for Sqlite, which is part of the Python standard library.
You'll need to install the required packages for the database you
want to use as your metadata database as well as the packages needed to
2016-11-10 02:08:22 -05:00
connect to the databases you want to access through Superset.
2016-04-08 11:44:28 -04:00
Here's a list of some of the recommended packages.
+---------------+-------------------------------------+-------------------------------------------------+
| database | pypi package | SQLAlchemy URI prefix |
+===============+=====================================+=================================================+
| MySQL | `` pip install mysqlclient `` | `` mysql:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
| Postgres | `` pip install psycopg2 `` | `` postgresql+psycopg2:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
| Presto | `` pip install pyhive `` | `` presto:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
| Oracle | `` pip install cx_Oracle `` | `` oracle:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
| sqlite | | `` sqlite:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2017-03-21 16:59:29 -04:00
| Redshift | `` pip install sqlalchemy-redshift `` | `` postgresql+psycopg2:// `` |
2016-04-08 11:44:28 -04:00
+---------------+-------------------------------------+-------------------------------------------------+
| MSSQL | `` pip install pymssql `` | `` mssql:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2016-06-21 12:41:24 -04:00
| Impala | `` pip install impyla `` | `` impala:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2016-07-21 20:51:30 -04:00
| SparkSQL | `` pip install pyhive `` | `` jdbc+hive:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2016-11-14 18:46:24 -05:00
| Greenplum | `` pip install psycopg2 `` | `` postgresql+psycopg2:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2017-04-03 11:22:40 -04:00
| Athena | `` pip install "PyAthenaJDBC>1.0.9" ` ` | ` ` awsathena+jdbc:// `` |
+---------------+-------------------------------------+-------------------------------------------------+
2017-04-10 18:37:14 -04:00
| Vertica | `` pip install | ` ` vertica+vertica_python:// `` |
| | sqlalchemy-vertica-python`` | |
+---------------+-------------------------------------+-------------------------------------------------+
2016-04-08 11:44:28 -04:00
Note that many other database are supported, the main criteria being the
existence of a functional SqlAlchemy dialect and Python driver. Googling
the keyword `` sqlalchemy `` in addition of a keyword that describes the
database you want to connect to should get you to the right place.
2017-04-03 11:22:40 -04:00
(AWS) Athena
------------
This currently relies on an unreleased future version of `PyAthenaJDBC <https://github.com/laughingman7743/PyAthenaJDBC> `_ . If you're adventurous or simply impatient, you can install directly from git: ::
pip install git+https://github.com/laughingman7743/PyAthenaJDBC@support_sqlalchemy
The connection string for Athena looks like this ::
awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...
Where you need to escape/encode at least the s3_staging_dir, i.e., ::
s3://... -> s3%3A//...
2016-03-16 23:25:41 -04:00
Caching
-------
2016-11-10 02:08:22 -05:00
Superset uses `Flask-Cache <https://pythonhosted.org/Flask-Cache/> `_ for
2016-03-16 23:25:41 -04:00
caching purpose. Configuring your caching backend is as easy as providing
2016-11-10 02:08:22 -05:00
a `` CACHE_CONFIG `` , constant in your `` superset_config.py `` that
2016-03-16 23:25:41 -04:00
complies with the Flask-Cache specifications.
2016-07-22 12:45:51 -04:00
Flask-Cache supports multiple caching backends (Redis, Memcached,
SimpleCache (in-memory), or the local filesystem). If you are going to use
2017-04-05 13:07:36 -04:00
Memcached please use the `pylibmc` client library as `python-memcached` does
2016-08-17 00:35:31 -04:00
not handle storing binary data correctly. If you use Redis, please install
2017-04-05 13:07:36 -04:00
the `redis <https://pypi.python.org/pypi/redis> `_ Python package: ::
pip install redis
2016-03-16 23:25:41 -04:00
2016-11-10 02:08:22 -05:00
For setting your timeouts, this is done in the Superset metadata and goes
2016-03-16 23:25:41 -04:00
up the "timeout searchpath", from your slice configuration, to your
data source's configuration, to your database's and ultimately falls back
into your global default defined in `` CACHE_CONFIG `` .
2016-04-06 11:45:27 -04:00
Deeper SQLAlchemy integration
-----------------------------
It is possible to tweak the database connection information using the
parameters exposed by SQLAlchemy. In the `` Database `` edit view, you will
find an `` extra `` field as a `` JSON `` blob.
.. image :: _static/img/tutorial/add_db.png
2016-04-08 11:44:28 -04:00
:scale: 30 %
2016-04-06 11:45:27 -04:00
This JSON string contains extra configuration elements. The `` engine_params ``
object gets unpacked into the
`sqlalchemy.create_engine <http://docs.sqlalchemy.org/en/latest/core/engines.html#sqlalchemy.create_engine> `_ call,
while the `` metadata_params `` get unpacked into the
`sqlalchemy.MetaData <http://docs.sqlalchemy.org/en/rel_1_0/core/metadata.html#sqlalchemy.schema.MetaData> `_ call. Refer to the SQLAlchemy docs for more information.
2016-04-14 01:32:03 -04:00
Schemas (Postgres & Redshift)
-----------------------------
2016-04-06 11:45:27 -04:00
2016-04-14 01:32:03 -04:00
Postgres and Redshift, as well as other database,
use the concept of **schema** as a logical entity
2016-11-10 02:08:22 -05:00
on top of the **database** . For Superset to connect to a specific schema,
2016-04-14 01:32:03 -04:00
there's a **schema** parameter you can set in the table form.
2016-04-06 11:45:27 -04:00
2016-04-12 00:41:32 -04:00
SSL Access to databases
-----------------------
This example worked with a MySQL database that requires SSL. The configuration
may differ with other backends. This is what was put in the `` extra ``
parameter ::
2016-04-14 01:32:03 -04:00
2016-04-12 00:41:32 -04:00
{
"metadata_params": {},
"engine_params": {
"connect_args":{
"sslmode":"require",
"sslrootcert": "/path/to/my/pem"
}
}
}
2016-03-16 23:25:41 -04:00
Druid
-----
* From the UI, enter the information about your clusters in the
2016-04-04 13:36:51 -04:00
`` Admin->Clusters `` menu by hitting the + sign.
2016-03-16 23:25:41 -04:00
* Once the Druid cluster connection information is entered, hit the
2016-04-04 13:36:51 -04:00
`` Admin->Refresh Metadata `` menu item to populate
2016-03-16 23:25:41 -04:00
* Navigate to your datasources
2016-04-06 21:02:15 -04:00
2016-11-10 02:08:22 -05:00
Note that you can run the `` superset refresh_druid `` command to refresh the
2016-04-06 21:02:15 -04:00
metadata from your Druid cluster(s)
2016-06-02 15:34:36 -04:00
CORS
-----
The extra CORS Dependency must be installed:
2016-11-10 02:08:22 -05:00
superset[cors]
2016-06-02 15:34:36 -04:00
2016-11-10 02:08:22 -05:00
The following keys in `superset_config.py` can be specified to configure CORS:
2016-06-02 15:34:36 -04:00
* `` ENABLE_CORS `` : Must be set to True in order to enable CORS
* `` CORS_OPTIONS `` : options passed to Flask-CORS (`documentation <http://flask-cors.corydolphin.com/en/latest/api.html#extension>` )
2016-12-14 12:39:59 -05:00
MIDDLEWARE
----------
Superset allows you to add your own middleware. To add your own middleware, update the `` ADDITIONAL_MIDDLEWARE `` key in
your `superset_config.py` . `` ADDITIONAL_MIDDLEWARE `` should be a list of your additional middleware classes.
For example, to use AUTH_REMOTE_USER from behind a proxy server like nginx, you have to add a simple middleware class to
add the value of `` HTTP_X_PROXY_REMOTE_USER `` (or any other custom header from the proxy) to Gunicorn's `` REMOTE_USER ``
environment variable: ::
class RemoteUserMiddleware(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
user = environ.pop('HTTP_X_PROXY_REMOTE_USER', None)
environ['REMOTE_USER'] = user
return self.app(environ, start_response)
ADDITIONAL_MIDDLEWARE = [RemoteUserMiddleware, ]
*Adapted from http://flask.pocoo.org/snippets/69/*
2016-04-06 21:02:15 -04:00
Upgrading
---------
Upgrading should be as straightforward as running::
2016-11-10 02:08:22 -05:00
pip install superset --upgrade
superset db upgrade
superset init
2016-08-19 18:27:35 -04:00
2016-09-19 18:28:10 -04:00
SQL Lab
-------
SQL Lab is a powerful SQL IDE that works with all SQLAlchemy compatible
databases out there. By default, queries are run in a web request, and
may eventually timeout as queries exceed the maximum duration of a web
2016-11-10 02:08:22 -05:00
request in your environment, whether it'd be a reverse proxy or the Superset
2016-09-19 18:28:10 -04:00
server itself.
In the modern analytics world, it's not uncommon to run large queries that
run for minutes or hours.
To enable support for long running queries that
execute beyond the typical web request's timeout (30-60 seconds), it is
necessary to deploy an asynchronous backend, which consist of one or many
2016-11-10 02:08:22 -05:00
Superset worker, which is implemented as a Celery worker, and a Celery
2016-09-19 18:28:10 -04:00
broker for which we recommend using Redis or RabbitMQ.
It's also preferable to setup an async result backend as a key value store
that can hold the long-running query results for a period of time. More
details to come as to how to set this up here soon.
2016-08-19 18:27:35 -04:00
2016-10-26 14:09:27 -04:00
SQL Lab supports templating in queries, and it's possible to override
the default Jinja context in your environment by defining the
2016-11-10 02:08:22 -05:00
`` JINJA_CONTEXT_ADDONS `` in your superset configuration. Objects referenced
2016-10-26 14:09:27 -04:00
in this dictionary are made available for users to use in their SQL.
2016-08-19 18:27:35 -04:00
Making your own build
---------------------
2016-11-10 02:08:22 -05:00
For more advanced users, you may want to build Superset from sources. That
2016-08-19 18:27:35 -04:00
would be the case if you fork the project to add features specific to
your environment.::
2016-11-10 02:08:22 -05:00
# assuming $SUPERSET_HOME as the root of the repo
cd $SUPERSET_HOME/superset/assets
2016-08-19 18:27:35 -04:00
npm install
2017-01-24 11:36:14 -05:00
npm run build
2016-11-10 02:08:22 -05:00
cd $SUPERSET_HOME
2016-08-19 18:27:35 -04:00
python setup.py install
2017-03-03 20:09:54 -05:00
Blueprints
----------
`Blueprints are Flask's reusable apps <http://flask.pocoo.org/docs/0.12/blueprints/> `_ .
Superset allows you to specify an array of Blueprints
an array of Blueprints in your `` superset_config `` module. Here's
an example on how this can work with a simple Blueprint. By doing
so, you can expect Superset to serve a page that says "OK"
at the `` /simple_page `` url. This can allow you to run other things such
as custom data visualization applications alongside Superset, on the
same server.
..code ::
from flask import Blueprint
simple_page = Blueprint('simple_page', __name__,
template_folder='templates')
@simple_page.route('/', defaults={'page': 'index'})
@simple_page.route('/<page>')
def show(page):
return "Ok"
BLUEPRINTS = [simple_page]