Commit Graph

731 Commits

Author SHA1 Message Date
John Bodley 54c6ddbdb7
chore(security): Renaming schemas_accessible_by_user (#10030)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-11 11:29:43 -07:00
Will Barrett 4ef26993e2
feat: Add owners and physical/virtual status to Datasets API list endpoint (#10025)
* Add owners and physical/virtual status to Datasets API list endpoint

* black

* Update type definition for DatasourceKind

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* Fixin' tests and mypy

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-06-10 22:04:40 +03:00
Erik Ritter e17da58a39
feat: add replace option to hive csv upload (#9764) 2020-06-10 08:57:33 -07:00
Ville Brofeldt 526ee3c3df
fix: flaky test_copy_dash test in dashboard_tests.py (#10021) 2020-06-09 13:29:22 +03:00
Ville Brofeldt 315518d2d2
feat: add support for query offset (#10010)
* feat: add support for query offset

* Address comments and add new tests
2020-06-09 11:46:28 +03:00
John Bodley 91517a56a3
style(mypy): Spit-and-polish pass (#10001)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-07 08:53:46 -07:00
Ville Brofeldt 5c4d4f16b3
feat: finalize Word Cloud move to new chart data endpoint (#9975)
* remove word cloud from viz.py

* Fix Run in SQL Lab

* remove deprecated python tests

* break out legacy endpoint type into function

* Break out exploreChart from exportChart and implement results type

* Fix jest tests and refactor accordingly

* lint

* Rename v1 payload function

* Add dashboard id to v1 chart data request url params

* Add support for domain sharding to v1 chart data request
2020-06-05 14:08:46 +03:00
John Bodley 244677cf5e
style(mypy): Enforcing typing for superset (#9943)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-03 15:26:12 -07:00
Xiang Fu 1d9dbcd9dc
fix: pinot select query logic (#9954)
* Update how pinot handles selection logic

* Change DATETIMECONVERT argument to use single quote for literals
2020-06-02 23:18:35 -07:00
John Bodley 1001c6d5f4
Revert "[caching] Using request context rather than globals" (#9969)
This reverts commit 90cd3889ac.

Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-06-02 16:03:33 -07:00
Ville Brofeldt 38a6bd79da
feat: expand new chart data endpoint coverage (#9903)
* feat: implement new chart API for additional components

* Fix python tests

* Fix tests

* Fix lint

* fix camel case error in requestParams

* lint

* fix samples row limit

* Add samples row limit to config

* remove unnecessary code

* lint

* Address review comments
2020-06-02 10:47:28 +03:00
John Bodley b296a0f250
[mypy] Enforcing typing for superset.utils (#9905)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-05-27 22:57:30 -07:00
Erik Ritter c4040a2ae0
fix: annotation layer json (#9915)
* fix: annotation layer json

* attempt to add a test

* [tests] Fixing test

Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-05-27 09:02:03 -07:00
Bogdan 3e8e441bfc
feat: implement csv upload configuration func for the schema enforcement (#9734)
* Implement csv upload func for schema enforcement

Implement function controlled csv upload schema

Refactor + fix tests

Fixing hive as well

* Add explore_db to the extras

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-05-21 13:49:53 -07:00
Ville Brofeldt a43a1d6303
feat: Add new result formats and types to chart data API (#9841)
* feat: Add new result formats and types to chart data API

* lint

* Linting

* Add language to query payload

* Fix tests

* simplify tests
2020-05-20 21:36:14 +03:00
michellethomas 368c85de1b
fix: Removing the logic to add timeseries_limit_metric to the data for table (#9832)
* Removing the logic to add timeseries_limit_metric to the data for table viz

* Also make the change in viz_sip_38

* Fix tests

Co-authored-by: michelle_thomas <michelle.thomas@airbnb.com>
2020-05-20 10:20:54 -07:00
John Bodley c691a07948
Revert "[sqllab] fix, strip comments before parsing statements" (#9862)
This reverts commit 949c4eae28.

Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-05-20 09:54:17 -07:00
Erik Ritter c7c76c6a99
build: Add python 3.8 to the build matrix (#9827) 2020-05-17 14:49:51 -07:00
Daniel Vaz Gaspar 3cc540019f
fix(mssql): reverts #9644 and displays a better error msg (#9752) 2020-05-14 17:00:02 +01:00
Daniel Vaz Gaspar 4427d65717
tests(celery): improve celery tests infra (#9775) 2020-05-14 16:58:30 +01:00
Erik Ritter d02f2d1fa7
feat: return security errors in the SIP-40 format (#9796) 2020-05-13 17:10:37 -07:00
Ville Brofeldt b532c4609f
chore: bump python dependencies (#9786)
* chore: bump pyhton dependencies

* bump flask-testing

* Add note to UPDATING and bump flask-testing in requirements-dev.txt

* fix old sqlparse test case that is fixed with sqlparse==0.3.1

* Fix migration and sqlparse set test

* downgrade alembic

* Downgrade sqlalchemy-utils

* pin sqlalchemy-utils due to regression in 0.36.5

* Refine comment in setup.py

* make cachelib an explicit requirement
2020-05-13 18:49:34 +03:00
Erik Ritter 83ec7365a7
feat: convert backend chart errors to the new error type (#9753) 2020-05-12 18:15:53 -07:00
John Bodley 04e3a4a72d
[caching] Using request context rather than globals (#9715)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-05-11 15:55:05 -07:00
Ville Brofeldt a8d5a8aaa2
fix: make tests work individually (#9782) 2020-05-11 11:48:22 -07:00
Ville Brofeldt 42b10aecae
Add QueryContext deserialization test (#9778)
* Add QueryContext deserialization test

* deserialize using marshmallow and assert error dict
2020-05-11 14:10:14 +03:00
Bogdan 903217f64d
Fix SQL Lab schema permission checks (#9756)
Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-05-08 14:59:49 -07:00
Daniel Vaz Gaspar 453806fc4c
perf(dataset): improve performance on get list (#9739) 2020-05-06 11:48:32 +01:00
Daniel Vaz Gaspar 0d85d25314
docs(api): improve openapi documentation for dash, charts and queries (#9724) 2020-05-05 14:42:18 +01:00
Bogdan 11972ed60d
Fix bad test (#9735)
Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-05-04 23:52:18 -07:00
ʈᵃᵢ 4c522ac65f
[sqllab] fix, strip comments before parsing statements (#9692) 2020-05-04 19:47:29 +03:00
Daniel Vaz Gaspar 2b59075d57
tests(engine_specs): full bigquery engine coverage (#9702) 2020-04-30 18:31:15 +01:00
John Bodley 3b0f8e9c8a
[sql] Adding lighweight Table class (#9649)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-30 08:38:02 -07:00
Bogdan 283504479e
Fix ENABLE_ROW_LEVEL_SECURITY flag usage (#9689)
Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-04-29 23:57:06 +03:00
John Bodley 7f89f12c4a
[debug] Debugging caching issue (#9665)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-29 12:16:47 -07:00
Daniel Vaz Gaspar 527bee5051
tests(engine_specs): full postgres engine coverage (#9682) 2020-04-29 18:20:25 +01:00
Daniel Vaz Gaspar f13ba2561c
fix(tests): custom filter flaky tests on dash and charts (#9679) 2020-04-29 08:36:11 +01:00
Lily Kuang b27a81e0da
[logging] deprecation notices for SQLite (#9662)
* [logging] deprecation notices for SQLite

* [logging] use warning for SQLite notice

* [logging] update grammar
2020-04-28 14:28:09 -07:00
Ville Brofeldt a52cfcd234
feat: Add geospatial post processing operations (#9661)
* feat: Add geospatial post processing operations

* Linting

* Refactor

* Add tests

* Improve docs

* Address comments

* fix latitude/longitude mixup

* fix: bad refactor by pycharm
2020-04-28 20:15:16 +03:00
Daniel Vaz Gaspar f07ca7d836
fix(database): test connection error message for module not found (#9634) 2020-04-27 09:24:41 +01:00
Daniel Vaz Gaspar 516bdf6db1
fix(mssql): apply limit and set alias for functions (#9644) 2020-04-27 09:23:08 +01:00
Lily Kuang b2720074bd
[datesets] feat: add statsd to datasets api (#9577)
* add statsd to tables api

* [datasets] update export endpoint

* [datasets] update charts api test
2020-04-24 16:38:29 +01:00
John Bodley 955a4fe792
[fix] Automatically add relevant Jinja methods to cache key if present (#9572)
* [fix] Adding URL params to cache key if present

* [cache] Wrapping Jinja methods

Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-24 08:18:22 -07:00
John Bodley 8ae92b50ad
[fix] Fixing regression from #9161 (#9641)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-23 22:51:33 -07:00
Ville Brofeldt a6cedaaa87
chore: Improve chart data API + schemas + tests (#9599)
* Make all fields optional in QueryObject and fix having_druid schema

* fix: datasource type sql to table

* lint

* Add missing fields

* Refactor tests

* Linting

* Refactor query context fixtures

* Add typing to test func
2020-04-23 14:30:48 +03:00
Ville Brofeldt 85778f5bec
fix: broken IS NULL and IS NOT NULL operator (#9613)
* fix: broken is null and is not null operator

* add unit tests

* Rename filter operator enum
2020-04-22 19:11:45 +03:00
ʈᵃᵢ 0b999e3b91
[charts] adds new filters ui (#9530)
* [charts] adds new filters ui

* move null check to be more visible

* better filter lists and async filter functionality
2020-04-21 12:04:11 -07:00
Lily Kuang 7cefc89c64
[charts] feat: add statsd to charts api (#9571)
* add statsd to charts api

* update test for charts api

* [charts] add statsd asserts wrapper

* [charts] update api test

* removed white space
2020-04-21 19:57:42 +01:00
John Bodley 237ac59474
[fix] Fixing issue with Jinja filter_value (#9582)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-20 09:44:45 -07:00
Ville Brofeldt 0472c11157
fix: Add deprecated fields to QueryObject schema (#9579)
* fix: Add deprecated fields to QueryObject schema

* linting
2020-04-18 20:28:57 +03:00
Ville Brofeldt 8e439b1115
chore: Add OpenAPI docs to /api/v1/chart/data EP (#9556)
* Add OpenAPI docs to /api/v1/chart/data EP

* Add missing fields to QueryObject, fix linting and unit test errors

* Fix unit test errors

* abc

* Fix errors uncovered by schema validation and add unit test for invalid payload

* Add schema for response

* Remove unnecessary pylint disable
2020-04-17 16:44:16 +03:00
John Bodley 074e3653e2
[fix] warm up cache error handling (#9560)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-04-16 15:19:32 -07:00
Daniel Vaz Gaspar 7b11b44abe
[dashboard] New, add statsd metrics to the API (#9519) 2020-04-16 10:54:45 +01:00
Bogdan 3574a3a4fb
Make email parsing more robust (#9523)
* Make email parsing more robust

* Address comment

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-04-15 16:58:03 +03:00
Aliaksei Kushniarevich ef5e11f45b
[copy] fix: Row Level Security get_rls_filters func SELECT statement (#9541)
* fix: Row Level Security get_rls_filters func SELECT statement

* More general RowLevelSecurityTests case to avoid improper ids matching
2020-04-15 16:41:54 +03:00
Daniel Vaz Gaspar d81f720502
[thumbnails] API and celery task for dashboards and charts (#8947) 2020-04-15 09:40:14 +01:00
Jianchao Yang 7b4b0d1c61
[Build] Add Github workflows (#9517)
As an replacement to Travis CI.
2020-04-14 13:38:09 -07:00
Daniel Vaz Gaspar 925d8b5315
[dashboard] Fix, improve test for custom filter (#9521) 2020-04-14 10:10:14 +01:00
Ville Brofeldt 9447381549
deprecate groupby controls in query_obj (#9366)
* Deprecate groupby from query_obj

* Fix query_object bug

* Fix histogram

* Remove groupby from legacy druid connector and fix first batch of unit tests

* Deprecate some unnecessary tests + fix a few others

* Address comments

* hide SIP-38 changes behind feature flag

* Break out further SIP-38 related tests

* Reslove test errors

* Add feature flag to QueryContext

* Resolve tests and bad rebase

* Backport recent changes from viz.py and fix broken DeckGL charts

* Fix bad rebase

* backport #9522 and address comments
2020-04-14 12:06:10 +03:00
Daniel Vaz Gaspar 622561f756
[charts] New, custom filter for name OR description (#9492)
* [charts] New, custom filter for name OR description

* Improve test
2020-04-11 07:25:54 +01:00
Bogdan 02b3fb1d63
Add visualization flow to the CTA queries (#9370)
* Visualize after running Ctas query (#4)

* Visualize after running Ctas query.

Stub for table viz

Work on the viz

Add JS functionality for Ctas viz

Add test for the table viz flow

Fix test

Add js test for the ctas viz

* Fix tests

* Resolve comments

* Leverate tmpSchema in the query object

* Fix i18n string

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-04-11 00:46:46 +03:00
Ville Brofeldt a8ce3bccdf
feat: Add post processing to QueryObject (#9427)
* Add post processing to QueryObject

* Simplify sort signature and require explicit sort order

* Add new operations and unit tests

* linting

* Address comments

* Simplify test method names

* Address comments

* Linting

* remove unnecessary logic

* Apply strict whitelisting to all getattr calls

* Add checking of rolling_type_options and add/improve docs
2020-04-10 20:50:11 +03:00
Daniel Vaz Gaspar b39e78fca5
[dashboards] New, tittle and slug OR filter (#9435)
* [dashboards] New, tittle and slug OR filter

* Update requirements, because of prison bump

* Tests

* Fix tests

* Avoid like filter on empty string value

* merge master brings strict typing to the table
2020-04-09 11:14:28 +01:00
Daniel Vaz Gaspar bb80ceaccc
[dashboards] Fix, update dashboard owners not propagating to charts o… (#9484)
* [dashboards] Fix, update dashboard owners not propagating to charts owners

* Add tests

* Fix tests

* better naming
2020-04-09 09:37:37 +01:00
Daniel Vaz Gaspar 0610c489bb
[query] Migrate api v1 query to new location (#9479)
* [query] Migrate api v1 query to new location

* Improved errors and tests

* [query] nit and revert undesired change

* [query] lint

* [query] revert nan change

* Update superset/queries/api.py

Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* Update tests/queries/api_tests.py

Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* change endpoint location to charts

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-04-09 09:05:49 +01:00
Daniel Vaz Gaspar 4485800e21
[tests] refactor, change datasets and charts to it's own folder (#9487) 2020-04-08 08:44:35 +01:00
dandanhub 72f051f3ce
[sqllab] Add CUSTOM_TEMPLATE_PROCESSOR config (#9376)
Co-authored-by: Dandan Shi <dshi@dropbox.com>
2020-04-07 21:00:42 +01:00
David Aaron Suddjian 5e535062da
Filter owners select by text input (#9337)
* filter owners select by text input

* use rison

* fix backend owners filter logic

* use fullname, not username on owners inputs

* fix some tests

* fixing tests

* deterministic tests

* appease linter

* add back search by username

* more comprehensive filter test

* add clarifying text

* formatting...
2020-04-07 09:09:02 -07:00
Daniel Vaz Gaspar c0807c1af7
[query] New, readonly API (#9403)
* [query] New, readonly API

* Update tests/queries/api_tests.py

Co-Authored-By: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-04-06 11:18:32 +01:00
Jianchao Yang 801e2f1777
chore: bump black to 19.10b0 and mypy to 0.770 (#9378)
* Bump black to 19.10b0

* Upgrade mypy to 0.770

* Update how inline type is defined
2020-04-04 13:23:18 -07:00
Daniel Vaz Gaspar 752de8fe9d
[dashboards] Fix, API update slug uniqueness refusing empty string (#9417)
* [dashboards] Fix, API update slug uniqueness refusing empty string

* [dashboards] tests
2020-03-30 16:10:08 +01:00
Ville Brofeldt 499f9c8fca
feat: add SSL certificate validation for Druid (#9396)
* feat: add SSL certificate feature

* Address comments

* don't mutate extras

* Address comments and add polish

* Add further polish
2020-03-27 19:07:07 +02:00
Daniel Vaz Gaspar 81971967c3
[charts] New, bulk delete API endpoint (#9387)
* [charts] New, bulk delete API endpoint

* [charts] Fix, typos

* [charts] Fix wrong model name
2020-03-27 09:31:01 +00:00
Daniel Vaz Gaspar 40bc1b33d2
[dataset] New, export API endpoint (#9367)
* [dataset] New, export API endpoint

* Fix, lint
2020-03-27 09:30:23 +00:00
John Bodley 19da9042ff
[cache] Cleaning up viz/cache logic (#9375)
Co-authored-by: John Bodley <john.bodley@airbnb.com>
2020-03-26 09:22:06 -07:00
Grace Guo 6b0f62a36e
[explore view] fix long query issue from Run in SQL LAB Button (#9345)
* [explore view] fix long query issue from Run in SQL LAB Button

* SQL Lab page needs to take the post form data, too

* fix variable names

* updated payload dict, rename hidden form

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
2020-03-25 13:15:52 -07:00
Daniel Vaz Gaspar ccb22dc976
[dataset] columns and metrics API (nested) (#9268)
* [dataset] columns and metrics API (nested)

* [dataset] tests and validation

* [datasets] Fix, revert list field name to database_name
2020-03-24 17:24:08 +00:00
Daniel Vaz Gaspar f51ab59748
[charts] Refactor API using SIP-35 (#9329)
* [charts] Refactor charts API using SIP-35

* [charts] Fix, copy pasta

* [charts] simplify
2020-03-24 10:05:11 +00:00
Maxime Beauchemin 98a71be80b
feat: [explore] don't save filters inherited from a dashboard (#9340)
* feat: [explore] don't save filters inherited from a dashboard

When navigating to explore from a dashboard context, the current
dashboard filter(s) are passed along to explore so that the context is
kept. So say you're filtering on "country=Romania", in your dashboard
and pivot to explore, that filter is still there and keep on exploring.

Now a common issue is that you'll want to make some tweak to your chart
that are unrelated to the filter, say toggling the legend off for
instance, and then save it. Now you back to your dashboard and even
though you started with an "all countries" dashboard, with a global
filter on country, now that one chart is stuck on "Romania". Typically
you notice this when filtering on something else, say "Italy" and then
that one chart now has two mutually exclusive filters, and show "No data".

Now, the fix is to flag the filter as "extra" (that's the not-so-good internal
name we use for these inherited filters) and make it clear that that
specific filter is special and won't be saved when saving the chart.

* fix build
2020-03-23 23:05:00 -07:00
Daniel Vaz Gaspar c34df6b7b3
[dashboard] Refactor API using SIP-35 (#9315)
* [dashboard] Refactor API using SIP-35

* [dashboard] Fix, import

* [dashboard] more tests

* [dashboards] a misc of improvements

* [charts] Fix, DAO and tests

* [dashboards] small exceptions refactor

* [dashboards] lint

* [dashboards] Improves comments on base classes

* [dashboards] lint
2020-03-20 16:32:03 +00:00
Ville Brofeldt f1edcebc79
fix: handle list of lists from fetch_data (#9322)
* fix: handle list of lists from fetch_data

* Address comments
2020-03-18 22:04:26 +02:00
Grace Guo adebd40d30
[cache warm_up] warm_up slice with dashboard default_filters (#9311)
* [cache warm_up] warm_up slice with dashboard default_filters

* update Celery warmup tasks

* fix code review comments

* add try catch and type checking for parsed dash metadata

* extra code review fix
2020-03-18 08:21:10 -07:00
Ville Brofeldt 982c234a50
fix: remove character set and collate column info by default (#9316)
* fix: remove character set and collate column info by default

* lint

* remove collation and charset info before compile
2020-03-17 21:34:39 +02:00
Maxime Beauchemin 6cf36c91ea
fix: big number to handle NULL as it did in the past (#9314) 2020-03-17 10:46:59 -07:00
ʈᵃᵢ 5767fb15cd
[datasets] new, listview (react) (#9197)
* [datasets] new, react listview

* add hidden columns to support filtering by columns not rendered

* throw exception if config is incorrect

* fix database filter

* update endpoints to point to datasets; fix translation strings

* move Link into src/components

* add add new record button to datasets listview
2020-03-13 21:35:00 +02:00
Erik Ritter f80fadff0e
Reduce dashboard bootstrap payload (#9284) 2020-03-13 12:18:22 -07:00
Daniel Vaz Gaspar 78ba7d52f6
[dashboard][api] Fix, PUT publish/draft to not clean slug and owners (#9279) 2020-03-12 15:35:27 +00:00
Ville Brofeldt 3682702e91
fix: add connection testing params for snowflake (#9272)
* fix: add connection testingt params for snowflake

* Linting
2020-03-11 06:51:57 +02:00
Grace Guo 724b8a3c31
[fix] copy filter_scopes with duplicate charts (#9271) 2020-03-10 10:23:15 -07:00
Maxime Beauchemin c04d6163e7
feat: add rolling window support to 'Big Number with Trendline' viz (#9107)
* Rolling big number

* addressing comment
2020-03-10 10:19:12 -07:00
Maxime Beauchemin 116200cf73
fix: change database save in DatasourceEditor (#9255)
* fix: change database save in DatasourceEditor

This addresses the issue where pointing a datasource to another database
in the datasource editor is not reflected.

Also addresses:
- a minorcosmetic issue in the datasource editor.
- user/owners list not getting populated

* tests
2020-03-10 09:20:37 -07:00
Daniel Vaz Gaspar 52c59d6890
[datasets] new, API using command pattern (#9129)
* [datasets] new, API using command pattern

* [datasets] tests and improvements

* [datasets] lint

* [database] address comments

* [datasets] lint

* [datasets] Address PR comments

* [dataset] Fix, dataset expects a Dict now

* [dataset] lint and optional commits

* [dataset] mypy

* [dataset] Fix, license and parent class

* [dataset] Make CRUD DAO raise exceptions
2020-03-08 09:13:08 +00:00
Erik Ritter f34e9e393a
Replace No data error with help message (#9249) 2020-03-06 16:11:01 -08:00
Grace Guo 000a038af1
[fix] use filter_scopes in dashboard warmup strategy (#9235)
* [fix] use filter_scopes in dashboard warmup strategy

* remove unnecessary comment
2020-03-05 17:50:32 -08:00
Ville Brofeldt ef2ebbd570
Add option to specify type specific date truncation functions (#9238) 2020-03-05 07:25:50 +02:00
Ville Brofeldt 969bc87431
fix: Oracle fetch_query and datetime conversion (#9240) 2020-03-04 23:43:28 +02:00
Ville Brofeldt 7a91498cf1
fix: share column type matching between model and result set (#9161)
* Share column type matching between model and result set

* Address comments
2020-03-04 16:33:22 +02:00
Bogdan 4e1fa95035
Make schema name for the CTA queries and limit configurable (#8867)
* Make schema name configurable

Fixing unit tests

Fix table quoting

Mypy

Split tests out for sqlite

Grant more permissions for mysql user

Postgres doesn't support if not exists

More logging

Commit for table creation

Priviliges for postgres

Update tests

Resolve comments

Lint

No limits for the CTA queries if configures

* CTA -> CTAS and dict -> {}

* Move database creation to the .travis file

* Black

* Move tweaks to travis db setup

* Remove left over version

* Address comments

* Quote table names in the CTAS queries

* Pass tmp_schema_name for the query execution

* Rebase alembic migration

* Switch to python3 mypy

* SQLLAB_CTA_SCHEMA_NAME_FUNC -> SQLLAB_CTAS_SCHEMA_NAME_FUNC

* Black
2020-03-03 19:52:20 +02:00
David Aaron Suddjian e01f24f833
Prevent database connections to sqlite (#9218)
* prevent database connections to sqlite

* tweaks and tests

* add entry to UPDATING.md
2020-03-02 13:13:11 -08:00
Grace Guo ccd6e44edf
refactor copy filter_scopes and add tests (#9224)
* refactor copy filter_scopes and add tests

* fix review comments
2020-03-02 12:55:30 -08:00
John Bodley 7b2c2e8a33
[fix] Adding SIP-15 support for the query context (#9219) 2020-03-02 10:10:54 -08:00
michellethomas 4f73f8a1f9
[Bug Fix] Returning timeseries_limit_metric in table viz get_data (#9196)
* Returning timeseries_limit_metric in table viz get_data

* Fixing issue with include_time field

* Reformatting and adding a test

* Changing if/else structure

* Reformatting
2020-02-25 21:34:36 -08:00
Daniel Vaz Gaspar 27b371a485
[core] Fix, sanitize errors returned from testconn (#9178) 2020-02-24 10:40:51 +00:00
altef dee16de03e
[SIP-29] Add support for row-level security (#8699)
* Support and apply filters.

* Added the UI for row level security, and moved it all under SQLA in order to access the Table model more easily.

* Added a row level security filter documentation entry.

* Accidentally added two new lines to this file.

* Blacked and iSorted, hopefully.  Also, sometimes g.user may not be set.

* Another isort, and handling g not having a user attribute another way.

* Let's try this again #CI tests.

* Adjusted import order for isort; I was sure I'd already done this..

* Row level filters should be wrapped in parentheses in case one contains an OR.

* Oops, did not think that would change Black's formatting.

* Changes as per @mistercrunch.

* RLS filters are now many-to-many with Roles.

* Updated documentation to reflect RLS filters supporting multiple rows.

* Let's see what happens when I set it to the previous revision ID

* Updated from upstream.

* There was a pylint error.

* Added RLS ids to the cache keys; modified documentation; added template processing to RLS filters.

* A new migration was merged in.

* Removed RLS cache key from query_object.

* RLS added to the cache_key from query_context.

* Changes as per @etr2460.

* Updating entry for RLS pull request.

* Another migration to skip.

* Changes as per @serenajiang.

* Blacked.

* Blacked and added some attributes to check for.

* Changed to a manual query as per @mistercrunch.

* Blacked.

* Another migration in the meantime.

* Black wanted some whitespace changes.

* AttributeError: 'AnonymousUserMixin' object has no attribute 'id'.

* Oops, did hasattr backwards.

* Changes as per @mistercrunch.

* Doesn't look like text us required here anymore.

* Changes as per @dpgaspar

* Two RLS tests.

* Row level security is now disabled by default via the feature flag ENABLE_ROW_LEVEL_SECURITY.

* New head to revise.

* Changed the comment.
2020-02-22 11:21:31 +02:00
Daniel Vaz Gaspar e55fe43ca6
[database] Fix, tables API endpoint (#9144) 2020-02-20 10:15:22 +00:00
Will Barrett 38f3fd0c9f
Add feature flags to control query sharing, KV exposure (#9120)
* Add feature flags to control query sharing, KV exposure

* Add tests, fix bug

* Skip test for kv endpoints when they are disabled

* ESLint fixes

* Remove unnecessary binds

* Fix eslint errors

* Add note to UPDATING.md RE: new feature flag options

* Use expanded version of RBAC

* Enable KV_STORE and SHARE_QUERIES_VIA_KV_STORE feature flags in the test environment

* Fix black
2020-02-19 09:51:50 -08:00
Grace Guo 84b42d28b9
[dashboard] use filter_scopes metadata when import old dashboard (#9145)
* [dashboard] update filter_scopes metadata when import old dashboard

* fix review comments
2020-02-19 09:06:52 -08:00
Rob DiCiuccio 0992445ae9
Catch TypeError on PyArrow array instantiation (#9139) 2020-02-17 08:34:00 -08:00
John Bodley 9f7466ef90
[fix] Fix table viz column order (#9122) 2020-02-16 22:51:35 -08:00
John Bodley a7e433a512
[mypy] Disallowing implicit optional (#9150) 2020-02-16 22:34:15 -08:00
Daniel Vaz Gaspar f993bdc7ef
[database] new, select star API migration (#9054) 2020-02-14 14:30:49 +00:00
Craig Rueda 607cfd1f29
Moving away from using the root logger everywhere (#9099)
* Moving away from using the root logger everywhere

* self.logger -> logger
2020-02-07 23:38:48 -08:00
Rob DiCiuccio 161d211c07
SQL Lab: Use numpy structured arrays, fallback to JSON serialization (#9096)
* Use numpy structured arrays, fallback to JSON serialization

* Explicitly cast data as list when creating numpy array
2020-02-06 17:25:22 -08:00
John Bodley 8a138fbd03
[tox] Allowing running of specific tests (#9097) 2020-02-06 16:45:37 -08:00
ʈᵃᵢ e5f5eed425
[chart] new, list view (react) (#8999)
* [charts] new, list view (react)

* DRY up template rendering

* fix i18n

* lint package.json
2020-02-05 12:42:42 -08:00
Will Barrett 47b5780c5c
Wrap tagging endpoints in a feature flag (disabled by default) (#9078)
* Wrap tagging endpoints in a feature flag (disabled by default)

* Add tests for new feature flag

* Reuse existing feature flag

* Add TAGGING_SYSTEM to feature flags defaulted to False
2020-02-05 11:10:13 -08:00
Daniel Vaz Gaspar 916d184076
[query] deprecate can_only_access_owned_queries (#9046) 2020-02-05 08:58:11 +00:00
Will Barrett fc1c9428e3
Do not show stacktraces on some intentionally-thrown errors (#9056)
* Do not show stacktraces on some intentionally-thrown errors

* Fix pylint error

* Fix JavaScript looking for the wrong key in error response from connection check

* format -> f-string

* Run black because PyCharm missed it on the last save
2020-02-05 07:01:43 +02:00
John Bodley 848c565c8c
[sip-15] Enabling SIP-15 by default (#9017) 2020-02-03 10:37:29 -08:00
Daniel Vaz Gaspar d5dbfc886d
[database] new, API table metadata (#9002)
Migrates and completely removes: '/superset/table/<database_id>/<table_name>/<schema>/'
- Guarantees database filtering based on the user permissions
- Full OpenAPI spec
- Better error handling
2020-01-29 09:51:06 +00:00
Daniel Vaz Gaspar fb17acd122
[api] fix, set default columns to just id when not defined (#8982) 2020-01-29 09:41:40 +00:00
Rob DiCiuccio 2fc3d84b5a
Serialize nested columns as JSON strings (#9007) 2020-01-28 15:50:36 -08:00
John Bodley dc60db2a43
[fix] Reverting metic logic from #8901 (#9030) 2020-01-27 22:28:42 -08:00
Daniel Vaz Gaspar 1f21bf877f
[log] fix, log model view permissions (#8993)
Limit MVC access to admin role only and limit REST API permissions
2020-01-26 12:15:57 +00:00
Erik Ritter 68e85ab1b6 [Caching] Ensure cache is always created (#9020)
* [Caching] Ensure cache is always created

* Update cache_manager.py

* Refactoring cache typing
2020-01-25 22:49:05 +00:00
Maxime Beauchemin 315a11dfe2
fix: shut off unneeded endpoints (#8960)
* fix: shut off all uneeded endpoints

We recently added a new feature to FAB allowing to whitelist the needed
endpoints in ModelView and ModelRestApi.

First, we set our base wrapper class to an empty set, forcing each
class inheriting from it to explicitely turn on the endpoints that
Superset intends to use.

Second, we go ModelView by ModelView to whitelist the actual endpoints
used in the app.

Notes:
* as a result a large set of [unneeded] permissions should be cleaned up
* outside of the "private" use of endpoints in the app, people that have
  been using endpoints in their environment for other purposes may
  experience loss of functionality

* Tweaking

* Reduce the amount of endpoints using white lists

* Fix, included needed endpoints for dashboard and druid

* Drying things up

* fixes

* limiting more endpoints

* Read only on some FAB model views

* fixing some tests

* fixes

* Fixing more tests

* Addressing comments

* Drying up route_methods

* further drying

Co-authored-by: Daniel Vaz Gaspar <danielvazgaspar@gmail.com>
2020-01-23 11:25:15 -05:00
Daniel Vaz Gaspar d02cf2f509 [dashboards] New, API for Bulk delete (#8972)
* [dashboards] New, API for Bulk delete

* [dashboards] Tests

* [dashboards] Fix not found on multiple delete

* [dashboards] Bulk delete partial deletes on not owned dashes

* [dashboards] Improve OpenAPI spec and tests

* [dashboards] Test for bad request

* [dashboards] i18n

* [dashboards] black

* [dashboard] make bulk all or nothing

* [dashboard] Log on sqlalchemy error

* [dashboard] Log on sqlalchemy error
2020-01-21 17:14:53 -05:00
Daniel Vaz Gaspar 74158694c5 [charts] New, REST API (#8917)
* [charts] New REST API

* [charts] Small improvements

* [charts] Fix, lint

* [charts] Tests and datasource validation

* [charts] Fix, lint

* [charts] DRY post schemas

* [charts] lint and improve type declarations

* [charts] DRY owned REST APIs

* [charts] Small fixes

* [charts] More tests

* [charts] Tests and DRY

* [charts] Tests for update

* [charts] More tests

* [charts] Fix, isort

* [charts] DRY and improve quality

* [charts] DRY and more tests

* [charts] Refactor base for api and schemas

* [charts] Fix bug on partial updates for dashboards

* [charts] Fix missing apache license

* black app.py after merge

* [charts] Fix, missing imports and black

* [api] Log on sqlalchemy error

* [api] isort
2020-01-21 13:04:52 -05:00
Ville Brofeldt c087a48d52 fix: add datasource.changed_on to cache_key (#8901)
* Add datasource.changed_on to cache_key and add+fix related unit tests

* Add note to UPDATING.md

* Remove redundant comment about metric names
2020-01-16 00:51:13 -05:00
ʈᵃᵢ 7b97764dbc [dashboard] New, list view (react) (#8845)
* adds dashboard listview component

* use new api

* use json over rison

* lint

* adds seperate dashboard list view

* edit and delete actions

* fix lint ignore

* fix common_bootstrap_payload is now a function

* fix license

* fix pylint

* isort

* fix tests

* lint

* lint ts

* fix js tests

* fix double import from bad rebase

* fix indent error

* lookup permissions

* generic permission lookup

* get tslint to pass

* adds js specs

* lint

* fix rebase

* lint

* lint again

* fix type errors preventing build

* adds more specs

* fix tslint error

* fix null check

* remove unecessary code

* use translations provided by api

* more translations

* linting

* fix spec

* i18n

* fix register order
2020-01-16 00:48:55 -05:00
Daniel Vaz Gaspar 123246fca6
[dashboards] New, export api (#8941)
* [dashboards] Multiple exports

* [dashboards] Fix, mulexport permission missing

* [dashboards] Test for security filtered export

* [dashboards] Address PR comments
2020-01-15 18:09:55 +00:00
serenajiang 1f6f4ed879 [db migration] change datasources-clusters foreign key to cluster_id (#8576)
* [db migration] change datasources foreign key to cluster_id

* address pr comments

* address pr comment, fix ci
2020-01-13 11:02:36 -08:00
Rob DiCiuccio 7ff18f060c Ensure proper JSON serialization of numpy.ndarray (#8946) 2020-01-10 13:24:03 -05:00
John Bodley 2d456e88eb
[fix] Enforce the query result to contain a data-frame (#8935) 2020-01-08 11:50:26 -08:00
Daniel Vaz Gaspar 2726f21cbc
[dashboard] Fix, prevent delete and update on dashes not owned (#8911) 2020-01-06 15:29:39 +00:00
Rob DiCiuccio 6537d5ed8c Replace pandas.DataFrame with PyArrow.Table for nullable int typing (#8733)
* Use PyArrow Table for query result serialization

* Cleanup dev comments

* Additional cleanup

* WIP: tests

* Remove explicit dtype logic from db_engine_specs

* Remove obsolete  column property

* SupersetTable column types

* Port SupersetDataFrame methods to SupersetTable

* Add test for nullable boolean columns

* Support datetime values with timezone offsets

* Black formatting

* Pylint

* More linting/formatting

* Resolve issue with timezones not appearing in results

* Types

* Enable running of tests in tests/db_engine_specs

* Resolve application context errors

* Refactor and add tests for pyodbc.Row conversion

* Appease isort, regardless of isort:skip

* Re-enable RESULTS_BACKEND_USE_MSGPACK default based on benchmarks

* Dataframe typing and nits

* Renames to reduce ambiguity
2020-01-03 11:55:39 -05:00
Craig Rueda 4ce5add5df Making test_config and main config the same (#8838) 2020-01-02 13:57:54 -05:00
Rob DiCiuccio 8fc814fc62 Enable running of tests in tests/db_engine_specs (#8902)
* Enable running of tests in tests/db_engine_specs

* Resolve application context errors

* Refactor and add tests for pyodbc.Row conversion

* Appease isort, regardless of isort:skip
2019-12-31 14:45:33 +02:00
Ville Brofeldt 5b690f9411
chore: refactor, add typing and fix uncovered errors (#8900)
* Add type annotations and fix inconsistencies

* Address review comments

* Remove incorrect typing of jsonable obj
2019-12-31 09:26:23 +02:00
Daniel Vaz Gaspar 3d9181d270
[dashboard] New, get releated owners and slices (#8872) 2019-12-20 10:16:57 +00:00
David Aaron Suddjian 016f202423 Refactor Dashboard and Slice models (#8820)
* refactor dashboard and slice models

* appease various linters

* remove shortcuts & import indirection

* appease mypy

* fix bad imports

* lint

* address various issues

* ignore type issue

* remove unused imports

* lint
2019-12-18 11:40:45 -08:00
David Aaron Suddjian 7a68cb7ca0 factor out datasource_access_request model (#8809) 2019-12-17 16:17:49 -08:00
Daniel Vaz Gaspar 2f2ac00a09
[dashboard] feat: REST API (#8694) 2019-12-16 21:10:33 +00:00
Will Barrett b87f21fe1f Move function in utils folder to fix circular import issues (#8814)
* Move function in utils folder to fix circular import issues

* CI fixes - black, isort
2019-12-13 16:44:56 +00:00
Will Barrett ed54f6ec8d Require the SQLAlchemy URI on the database model (#8720)
* Require the SQLAlchemy URI when creating a database

* Add migration to make dbs.sqlalchemy_uri not-nullable

* Fixes for black, isort, tests

* Alter migration to use current revision from master as downgrade target

* Update tests to support new db constraint

* black
2019-12-11 14:05:23 -08:00
Bogdan e6be51953c Avoid circular dependency between superset config and superset package (#8254)
Resolve comments

Avoid circular dependency between superset config and superset package

Resolve comments
2019-12-10 09:17:34 +00:00
Will Barrett 7f8db7ab8d Re-enable pylint for common, connector, and test files (#8774)
* re-enable pylint for superset/common/query_context.py

* Re-enable pylint for superset/connectors/base/views.py

* re-enable pylint for superset/connectors/connector_registry.py

* Re-enable pylint for tests/fixtures/pyodbcRow.py

* re-enable pylint for superset/cli.py

* Fix for mypy

* Improve variable naming

* A few more naming fixes
2019-12-09 22:32:54 -08:00
Beto Dealmeida 5477f42740
Limit payload returned when using SQLLAB_BACKEND_PERSISTENCE (#8769)
* Limit queries

* WIP

* Adding unit test

* WIP

* Add unit test

* Fix test

* Try to fix remote unit tests

* Fix unit test

* Fix  for postgres
2019-12-09 16:12:40 -08:00
Dinesh Sawant cae0583e0f Support timeFormat extraction function column for Druid in Filter (#8728) 2019-12-07 17:54:37 -08:00
Hossein Torabi b053d66ea7 fix urlib opener bug (#8788) 2019-12-07 16:07:09 -08:00