Druid sometimes returns error message that are contained in "<>", as in
`<urlopen error [Errno 61] Connection refused>`. Since Superset's
approach is often to bubble up messages coming from external library,
it's impossible to predict whether it will contain special characters.
There are some cases where our error handling does return some html
(presto?),
but we should manage that upstream. Plus the current setup has security concerns,
so let's move away from that.
fillna would miss out on identifying STRING columns for Druid and
replace None in string columns with a numeric `0`. This
mixed type column would confuse
pandas down the line on some operations like `df.pivot_table`.
Though flask has a builtin web debugger, ipdb some times still work
better. So I think add ipdb to dev dependencies is a good option for
people who prefer CLI debugging.
* cast postgresql date_trunc() to timestamp without time zone to prevent pandas error
* fix formatting for flake8
* change cast to timezone conversion instead
* Update installation.rst for Ubuntu 16.04 LTS
Ubuntu 16.04 by default install python2.7 alongside with python 3.5 and set python2.7 as default. If you have created a virtualenv with python3.5 compilation fails due to wrong python-dev library installed.
If you install ``python3.5-dev`` the build for the wheel package of ``cryptography`` run fine.
* Add options in D3_TIME_FORMAT_OPTIONS for non-english Time Format.
Added '%d/%m/%Y' and '%d/%m/%Y %H:%M:%S' as valid drop-down option for Axis Format
* [cli] permission cleanup on 'superset init'
FAB sometimes creates NULL/None permissions in the database,
presumably a race condition when multiple gunicorn workers start at the
same time, and those create issues raising "AttributeError: 'NoneType'
object has no attribute 'name'"
* Linting
Ubuntu 16.04 by default install python2.7 alongside with python 3.5 and set python2.7 as default. If you have created a virtualenv with python3.5 compilation fails due to wrong python-dev library installed.
If you install ``python3.5-dev`` the build for the wheel package of ``cryptography`` run fine.
* Use the query_obj as the basis for the cache key
When we recently moved from hashing form_data to define the cache_key
towards using the rendered query instead,
it made is such that non deterministic form
control values like relative times specified in "from" and "until" time
bound resulted in making those miss cache 100% of the time.
Here we move away from using the rendered query and using the query_obj
instead.
* Deprecating using form_data in templates
Filters applied to deck_multi will be passed down to layers as.
If the column isn't set as "filterable", the filter is ignored.
Also note that Dashboard configuration in regards to
"filter_immune_slices" and such will be disregarded in this context as
it isn't the dashboard controller passing down the filter and that
context is not easily accessible here.
Moving from having the user define an interceptor function that operates
on one object at a time.
By passing the entire array, it's possible to do multiple pass where
needed. A common pattern might be to figure out the max value in order
to define a scaler function. That's only possible if dealing with the
whole array.
* conditional check on datatype of results before converting to df
fix type checking
fix conditional checks
remove trailing whitespace and fix df_data fallback def
actually remove trailing whitespace
generalized type check to check all columns for dict
refactor dict col check
* move df conversion to helper and add unit test
add missing newlines
another missing newline
fix quotes
more quote fixes
Currently, even though `get_sqla_engine` calls get memoized, engines are
still short lived since they are attached to an models.Database ORM
object. All engines created through this method have the scope of a web
request.
Knowing that the SQLAlchemy objects are short lived means that
a related connection pool would also be short lived and mostly useless.
I think it's pretty rare that connections get reused within the context
of a view or Celery worker task.
We've noticed on Redshift that Superset was leaving many connections
opened (hundreds). This is probably due to a combination of the current
process not garbage collecting connections properly, and perhaps the
absence of connection timeout on the redshift side of things. This
could also be related to the fact that we experience web requests timeouts
(enforced by gunicorn) and that process-killing may not allow SQLAlchemy
to clean up connections as they occur (which this PR may not help
fixing...)
For all these reasons, it seems like the right thing to do to use
NullPool for external connection (but not for our connection to the metadata
db!).
Opening the PR for conversation. Putting this query into our staging
today to run some tests.