Commit Graph

851 Commits

Author SHA1 Message Date
Rob DiCiuccio
4d329071a1
feat(SIP-39): Async query support for charts (#11499)
* Generate JWT in Flask app

* Refactor chart data API query logic, add JWT validation and async worker

* Add redis stream implementation, refactoring

* Add chart data cache endpoint, refactor QueryContext caching

* Typing, linting, refactoring

* pytest fixes and openapi schema update

* Enforce caching be configured for async query init

* Async query processing for explore_json endpoint

* Add /api/v1/async_event endpoint

* Async frontend for dashboards [WIP]

* Chart async error message support, refactoring

* Abstract asyncEvent middleware

* Async chart loading for Explore

* Pylint fixes

* asyncEvent middleware -> TypeScript, JS linting

* Chart data API: enforce forced_cache, add tests

* Add tests for explore_json endpoints

* Add test for chart data cache enpoint (no login)

* Consolidate set_and_log_cache and add STORE_CACHE_KEYS_IN_METADATA_DB flag

* Add tests for tasks/async_queries and address PR comments

* Bypass non-JSON result formats for async queries

* Add tests for redux middleware

* Remove debug statement

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* Skip force_cached if no queryObj

* SunburstViz: don't modify self.form_data

* Fix failing annotation test

* Resolve merge/lint issues

* Reduce polling delay

* Fix new getClientErrorObject reference

* Fix flakey unit tests

* /api/v1/async_event: increment redis stream ID, add tests

* PR feedback: refactoring, configuration

* Fixup: remove debugging

* Fix typescript errors due to redux upgrade

* Update UPDATING.md

* Fix failing py tests

* asyncEvent_spec.js -> asyncEvent_spec.ts

* Refactor flakey Python 3.7 mock assertions

* Fix another shared state issue in Py tests

* Use 'sub' claim in JWT for user_id

* Refactor async middleware config

* Fixup: restore FeatureFlag boolean type

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2020-12-10 20:21:56 -08:00
Beto Dealmeida
475f59cb1c
feat: confirm overwrite when importing (#11982)
* feat: confirm overwrite when importing

* Skip flaky test
2020-12-10 14:50:10 -08:00
Kasia Kucharczyk
4da47f1635
tests: added fixture with energy usage instead of example (#11308)
* Added energy usage fixture. Updated dashboard utils and unicode fixture with new method parameters.

* Add energy fixture to tests/access_tests.py

* Add energy fixture to tests/core_tests.py

* Add energy fixture to tests/dashboard_tests.py

* Add energy fixture to tests/datasets/api_tests.py

* Add energy fixture to tests/db_engine_specs/base_engine_spec_tests.py

* Add energy fixture to tests/import_export_tests.py

* Add energy fixture to tests/model_tests.py

* Add energy fixture to tests/query_context_tests.py

* Add energy fixture to tests/security_tests.py

* Add energy fixture to tests/charts/api_tests.py

* Changed formatting of slices' parameters in energy usage fixture

* Removed loading energy udage data from test conf file

* Add energy fixture to tests/databases/api_tests.py

* Fixes after review: removed isort:skip, load_charts->load_energy_charts, removed unused import.

* Added energy fixture to tests/charts/commands_tests.py and retrieving proper Slice by name

* Fixed charts/api_tests.py to use energy_usage from fixtures

* Fixed datasets/commands_tests.py to retrieve dataset by name and use energy_usage fixture

* Changed energy usage data to generated data and fixed chart tests which was checking energy usage data
2020-12-09 12:02:29 -08:00
Daniel Vaz Gaspar
1e3aaab590
fix(reports): validator_config, report state machine, working_timeout (#11890)
* fix(reports): expect more exceptions and fix validator config

* use a state pattern on command reports

* use a state pattern on command reports continue

* fix multiple heads

* fix unittests

* add more tests

* fix api tests after enum rename

* fix alembic multiple heads

* fix tests

* fix fixture cleanup

* fix mysql tests

* fix initial and not found state

* fix schema, and private public methods, addressing comments

* add new col to the API
2020-12-09 18:19:07 +00:00
Beto Dealmeida
33325f9fa6
feat: add modal to import charts (#11956)
* WIP

* Splat props
2020-12-07 18:33:59 -08:00
Beto Dealmeida
2b9695c520
feat: add modal to import databases (#11884)
* feat: add modal to import databases

* Fix test

* Improve hook

* Remove log and needless store.

* Change JS functions
2020-12-07 11:22:45 -08:00
Beto Dealmeida
66cd565bff
feat: add Postgres SQL validator (#11538)
* Add Postgres SQL validator

* Strip line number from message

* Add unit tests

* Run tests only with postgres backend

* Add dep

* Add dep to bashlib
2020-12-04 19:17:23 -08:00
Erik Ritter
77d362d306
fix: Remove expensive logs table migration (#11920) 2020-12-04 07:59:28 -08:00
Ville Brofeldt
327a2817d3
feat: add event and interval annotation support to chart data ep (#11665)
* feat: add event and interval annotation support to chart data ep

* add tests + refactor fixtures

* use chart dao
2020-12-04 14:40:31 +02:00
Beto Dealmeida
e0288bf76b
chore: add unit tests for the current import functionality (#11786)
* chore: add unit tests for the current import functionality

* Improve comment

* Fix unit test
2020-12-03 14:48:43 -08:00
Bogdan
0396c705d4
chore: Support SET & SHOW commands as read only SQL commands (#11868)
* Support SET & SHOW commands as read only SQL commands

* Move is_readonly definition into the engine spec

* Rename & use super()

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-12-03 10:44:11 -08:00
Daniel Vaz Gaspar
71c567bf3a
feat(css templates): security perm simplification (#11856)
* feat: security converge css templates

* fix security tests

* fix JS test

* add migration

* black and fix migration
2020-12-03 08:45:32 +00:00
Lily Kuang
a76eadd838
feat: report/alert list CRUD view (#11802) 2020-12-01 17:57:15 -08:00
Kasia Kucharczyk
462b39d762
fix: update chart doesn't remove all connections to dashboards (#11830)
* Added possibility to not change dashboards connection when chart is changed

* Added two chart unit tests which checks if dashboards properly change on update
2020-11-30 09:48:14 -08:00
Hugh A. Miles II
0689738e7c
feat: Sqllab to Explore UX improvements api changes (#11836) 2020-11-30 09:23:11 -08:00
Daniel Vaz Gaspar
9dd33d5566
feat(saved queries): security perm simplification (#11764)
* feat(saved queries): security perm simplification

* migration script and frontend

* add downgrade procedure

* downgrade procedure, inferred from the upgrade data

* fix JS test

* improve code

* add tests for role migration

* more tests and a short description for each one

* Improve readability

* simplify dataclass creation

* fix dataclass
2020-11-30 13:07:39 +00:00
Daniel Vaz Gaspar
bac84a3aac
fix: delete chart, dashboards, dbs with assoc reports (#11801)
* fix: delete chart or dashboards with assoc reports

* database constraint to reports and tests

* add tests for dashboards and database

* fix exceptions default text
2020-11-26 08:45:49 +00:00
Beto Dealmeida
c354e7e0ab
fix: remove root dir from ZIP bundle (#11805)
* fix: remove root dir from ZIP bundle

* Fix lint

* Fix tests

* Improve tests

* Fix dashboard as well
2020-11-25 11:47:48 -08:00
Daniel Vaz Gaspar
f27ebc4be5
feat: new reports scheduler (#11711)
* feat(reports): scheduler and delivery system

* working version

* improvements and fix grace_period

* add tests and fix bugs

* fix report API test

* test MySQL test fail

* delete-orphans

* fix MySQL tests

* address comments

* lint
2020-11-25 08:50:30 +00:00
Beto Dealmeida
501b9d47c5
feat: API endpoint to import dashboards (#11751)
* ImportChartsCommand

* feat: add a command to import dashboards

* feat: API endpoint to import dashboards

* Add dispatcher

* Raise specific exception

* Fix test

* Remove print calls

* Add logging when passing
2020-11-24 22:45:35 -08:00
Erik Ritter
c0224aa928
feat: add certification info to table selector (#11785) 2020-11-24 12:13:52 -08:00
Beto Dealmeida
25345bea64
feat: add a command to import dashboards (#11749)
* feat: add a command to import dashboards

* Fix lint

* Remove print()
2020-11-23 16:28:26 -08:00
ʈᵃᵢ
fbe4a6622e
feat: SQL preview modal for Query History (#11634) 2020-11-20 16:01:06 -08:00
Beto Dealmeida
a3a2a68f01
feat: API endpoint to import charts (#11744)
* ImportChartsCommand

* feat: API endpoint to import charts

* Add dispatcher

* Fix docstring
2020-11-20 14:40:27 -08:00
Beto Dealmeida
2f4f87795d
feat: add a command to import charts (#11743)
* ImportChartsCommand

* Fix type
2020-11-20 14:20:13 -08:00
Ville Brofeldt
7ae8cd07cc
fix: do not drop calculated column on metadata sync (#11731) 2020-11-18 08:33:14 -08:00
Beto Dealmeida
cce716a821
feat: API endpoints to upload dataset/db (#11728)
* feat: API endpoints to upload dataset/db

* Fix method call
2020-11-17 14:49:33 -08:00
Rob DiCiuccio
01d15f5368
feat(templating): Safer Jinja template processing (#11704)
* Enable safer Jinja template processing

* Allow JINJA_CONTEXT_ADDONS with SAFE_JINJA_PROCESSING

* Make template processor initialization less magical, refactor classes

* Consolidat Jinja logic, remove config flag in favor of sane defaults

* Restore previous ENABLE_TEMPLATE_PROCESSING default

* Add recursive type checking, update tests

* remove erroneous config file

* Remove TableColumn models from template context

* pylint refactoring

* Add entry to UPDATING.md

* Resolve botched merge conflict

* Update docs on running single python test

* Refactor template context checking to support engine-specific methods
2020-11-17 11:55:47 -08:00
Beto Dealmeida
7bc353f8a8
feat: new import commands for dataset and databases (#11670)
* feat: commands for importing databases and datasets

* Refactor code
2020-11-16 17:11:20 -08:00
Bogdan
a16a2b87c7
chore: statds metrics for cache invalidation (#11701)
* Add more logging to the cache invalidation

* Address linter & tests

* Fix linter

Co-authored-by: bogdan kyryliuk <bogdankyryliuk@dropbox.com>
2020-11-16 06:48:26 -08:00
Jesse Yang
4cfcaebb61
feat: rename TABLE_NAMES_CACHE_CONFIG to DATA_CACHE_CONFIG (#11509)
* feat: rename TABLE_NAMES_CACHE_CONFIG to DATA_CACHE_CONFIG

The corresponding cache will now also cache the query results.

* Slice use DATA_CACHE_CONFIG CACHE_DEFAULT_TIMEOUT

* Add test for default cache timeout

* rename FAR_FUTURE to ONE_YEAR_IN_SECS
2020-11-13 22:35:10 -08:00
Jesse Yang
ec8ccd4cf1
feat: keep modal open when saving database failed (#11618) 2020-11-12 16:10:14 -08:00
ʈᵃᵢ
432e5ab460
feat: Query History CRUD list view (#11574) 2020-11-12 11:55:13 -10:00
Daniel Vaz Gaspar
12cb27f5cb
feat: new reports models api (#11606)
* feat: new report schedule models

* lint and unique constraint

* support sqlite

* fix sqlite

* add audit mixin and minor fixes

* feat(api): alerts and reports REST API

* feat: new report schedule models

* lint and unique constraint

* support sqlite

* fix sqlite

* add audit mixin and minor fixes

* feat(api): alerts and reports REST API

* draft working version

* add tests

* test

* black

* remove copy pasta

* solve dashboard object representation being used on cache

* tests and custom filter

* fix PUT has PATCH on active field

* create feature flag

* fix lint

* address comments
2020-11-12 13:21:01 -08:00
Grace Guo
0338b48fdd
fix (dashboard): remove default filters parameters in dashboard url (#11661) 2020-11-12 13:00:27 -08:00
Jesse Yang
542d2e3b06
chore: clean up a debug line from #11653 (#11663) 2020-11-12 08:37:49 -08:00
Ville Brofeldt
2718909314
feat: add ECharts BoxPlot chart (#11199)
* feat: add ECharts BoxPlot chart

* lint

* fix cypress tests

* lint

* remove viz.py shim

* bump plugin package

* skip non-legacy plugin cypress dashboard tests

* fix cypress tests

* disable cypress tests for non-leagcy charts

* fix bad rebase

* use midpoint interpolation for quartile calculation

* bump packages and add support for no groupby

* whitespace

* whitespace

* linting

* fix tests

* xit mathjs load test

* bump mathjs to 8.0.1

* disable cypress filter test for v1 charts
2020-11-12 10:01:04 +02:00
Jesse Yang
302c9603c4
fix: database connection validation when creation (#11653) 2020-11-12 08:00:38 +00:00
Beto Dealmeida
45738ffc1d
chore: consolidate datasource import logic (#11533)
* Consolidate dash import logic

* WIP

* Add license

* Fix lint

* Retrigger tests

* Fix lint
2020-11-11 22:04:16 -08:00
Beto Dealmeida
4a070cfceb
chore: consolidate dashboard import logic (#11529)
* Consolidate dash import logic

* Fix lint

* Remove Slice.import_obj

* Remove unused import

* Fix log
2020-11-11 14:50:18 -08:00
Ville Brofeldt
77dff0e4e0
fix(chart-data): ignore orderby on sample result type (#11656) 2020-11-11 16:27:56 +02:00
Beto Dealmeida
51712bf7ce
fix: CommandInvalidError had invalid init (#11635) 2020-11-10 07:41:03 -08:00
Ville Brofeldt
600a6fa92a
chore(rls): move to feature flag and disable related view (#11575)
* chore(rls): move to feature flag and disable related view

* rename feature flag
2020-11-05 12:19:48 +02:00
Lily Kuang
eef4809978
feat: annotation edit modal with antd datepicker (#11500) 2020-11-03 12:59:13 -08:00
Daniel Vaz Gaspar
6f2e36dd1b
fix(tests): flaky test with dttm on Query API (#11526)
* fix(tests): flaky test wirh dttm on Query API
2020-11-03 09:01:17 +00:00
ʈᵃᵢ
edb9619731
refactor: reduce number of api calls needed to fetch favorite status for charts and dashboards (#11502) 2020-11-02 21:26:14 -08:00
Beto Dealmeida
fd10c47bc6
chore: remove sanitize (#11532)
* chore: remove sanitize

* Fix lint
2020-11-02 15:13:44 -08:00
Daniel Vaz Gaspar
34ae29e59b
feat: query REST API more fields and tests (#11482)
* feat: query REST API more fields and tests

* fix missing field
2020-10-31 10:18:34 +00:00
Moriah Kreeger
01ddbd0697
feat: annotation layers modal + filters (#11494) 2020-10-30 15:51:46 -07:00
Beto Dealmeida
fbcfaacda3
feat: create base class for export commands (#11463)
* Add UUID to saved_query

* Reuse function from previous migration

* Point to new head

* feat: add backend to export saved queries using new format

* Rename ImportMixin to ImportExportMixin

* Create base class for exports

* Add saved queries as well

* Add constant, small fixes

* Fix wrong import

* Fix lint
2020-10-30 11:52:11 -07:00