* Add check for calls to cache_key_wrapper to avoid unavoidable compilation of query
* Add fetch_values_predicate to check
* Only check relevant attributes
* Address nit
* Add typing to db_engine_specs
* Add more type annotations and docstrings
* Add docstrings and typing to sql_parse and db_engine_specs
* Refine select_star
* Fix execute and add more docstrings
* Revert kwargs change from execute
* Remove redundant or
* Align view and table getter schema types
* Fix return type of latest_partition
* Remove some typing from presto
* Improve docstring for __extract_from_token
* [database] Fix, Removes the limit for the page size, Bump FAB to 2.1.8
Old FAB API had no limits by default, this will keep this behaviour
but only for this endpoint
* [sqllab] Add test for database API
* [sqllab] Add test for database API
* [sqllab] Include page zero on request
* [sqllab] Fix, Black and requirements
* [sqllab] Make database API return unlimited results
* [sqllab] just a test
* [sqllab] Bump FAB to 2.1.9
* [sqllab] Remove unused import
* 7620: Start removing dependencies on requests
* Patch urllib.request.urlopen instead of requests.get
* Try to fix flake8
* More work on flake8 import errors
* First attempt at using urllib with cookies
* Fix pylint/flake8
* Fix test_deliver_slice_csv_attachment
* Fix test_deliver_slice_csv_inline
* Import requests and pydruid conditionally, remove dependency on prison
* Fix flake errors
* Fix load_examples
* Please flake
* Skip tests depending on optional deps
* Try to please flake
* Address review comments
* Remove Druid-related UI
* Revert "Remove Druid-related UI"
This reverts commit d7e0f166cc3f3dd2496b4a666e177f0c191aeb0f.
* Skip a few tests more
* Put imports in right order
* Apply black patch
* Please flake
* Please black, silence flake
* Use flake8 silencing the right way
* Add deps for CI
* [cli] New, command line option to create or set a db URI
* [tests] New, get or create db on utils
* [tests] flake8 and black
* [tests] Fix, remove dummy comments
* [cli] dummy commit to re-trigger build
* [cli] dummy commit to re-trigger build
* Create query_obj for every filter
* Deprecate is_prequery and prequeries from query_obj
* Fix tests
* Fix typos and remove redundant ; from sql
* Add typing to namedtuples and move all query str logic to one place
* Fix unit test
* Allow to pass user for dashboard import cli
Dashboard import assign current user from flask context during import.
But in case of cli import there is no flask user and imported charts
don't have an owner which prevents ability to edit them.
* Reset ownership on dashboard import
For overriding existing charts it requires `owners` property to be set.
* Add tests for reset ownership
* Use ORM to decode dashboards json
Creating instances using ORM allows to normally work with relations
* Fix test_import_dashboard_1_slice test
Previously tests used side-effect of slices import which kept id from
json on insert into db.
* [api] [database] New, migrate to new FAB API side by side
* [database] view and api module reorganization
* [style] Fix, flake8
* [test] [database] Fix, tests for new database view structure
* [style] [database] black
* [database] Fix, missing apache license
* Migrate databaasync api to the new api
* flake8
* More efficient api call
* Revert querySearch to old API, test
* dummy commit
* revert config.py commit
* Remove unused import
* Remove old API view
* Remove new API view
* Add database filter
* Add database filter
* Add database filter
* Remove comments on js
* Fix JS test
* Fix pylint
* QuerySearch new databaseasync API endpoint
* Fix, query search new api endpoint
* Reintroduce old API for smooth side by side migration
* Better naming and use common hooks for view and API
* black it
* Fix, database mixin must come first
* Fix, lint
* fix(sql-lab explore): fixed response data and header type to be aligned
* fix(sql-lab explore): fixed test error caused by incorrect test at master
* fix(sql-lab explore): reformatted
* fix(sql-lab explore): removed unused import
* Remove unnecessary fields from dashboard exported json
This commit makes export respect export_fields and doesn't export
unnecessary relations (like users with passwords hashes) which are
ignored during the import.
* Allow to import dashboard without position_json
In case charts were added from chart-edit page and wasn't re-ordered on
the dashboard position_json field is empty and import was failing with
error.
* Update export/import tests
* [WiP] improve load_examples
related to #7472, longer term we will generate the examples by exporting
them into tarball as in #7472. In the meantime, we need this subset of
the features:
* allowing specifying an alternate database connection for examples
* allowing a --only-metadata flag to `load_examples` to load only
dashboard and chart definitions, no actual data is loaded
* Improve logging
* Rename data->examples
* Load only if not exist
* By default do not load, add a force flag
* fix build
* set published to true
* Allow users to publish dashboards
* Rework publish dashboards feature
- The eye next to the title has been replaced with a [draft] badge
- Published status is now toggled in the Header Action Dropdown
- CRUD list shows published status
* Fix linter errors
* Update javascript tests
* Add tests and change DashboardFilter
Add some tests to make sure the published status is rendered and
Make it so that users cannot see dashboards that are published
if they don't have access to any of the slices within
* Fix some linter errors
* Remove commas from core.py
* Fix some failing tests
* More linter errors I introduced
* Fix more linter errors I introduced
* update alembic migration
* Update design of publish dash feature
* Upgrade migration version
* Secure publish endpoint
* Remove bad quotes
* Give publish span its own style
* fix publish rendering
* Add new test for publish feature
* Update migration
* update slug in test
* Update migration
* Address reviwer comments
* Fix linter errors
* Add licenses
* Remove fetchPublished(), use bootstrap data
* Update migration
* Update croniter to existing version
* Fix linter errors
* Upgrade DB Revisions
* Fix flake8 linter error
* Set all dashboards to published on migration
* Migration proper line spacing
* Fix migration to work with postgres
* UPDATE statement works with postgresql and sqlite hopefully
* Update wording to kick off travis
Fixing a few failing examples, examples were referencing metrics that
were getting auto-generated in the past (like `sum__num`),
that we now prevent from getting created.
* Add extra_require for bigquery to setup.py
* Refactor df_to_db and add df upload capability for BigQuery
* Fix unit tests and clarify kwarg logic
* Fix flake8 errors
* Add minimum versions for bigquery dependencies
* wrap to_gbq in try-catch block and raise error if pandas-gbq is missing
* Fix linting error and make error more generic
* fix: SqlaColumn.type overflow on mysql
Hitting a new error when loading examples around a datatype that exceeds
the 32 chars limit on SqlaColumn.type
Type includes the 'COLLATE utf8mb4_general_ci' suffix which is too
verbose and not needed in that context.
* fix tests
* feat: rough check in for Presto rows and arrays
* fix: presto arrays
* fix: return selected and expanded columns
* fix: add helper methods and unit tests
* fix: only allow exploration of selected columns
* fix: address Beto's comments and add more unit tests
* Move schema name handling in table names from frontend to backend
* Rename all_schema_names to get_all_schema_names
* Fix js errors
* Fix additional js linting errors
* Refactor datasource getters and fix linting errors
* Update js unit tests
* Add python unit test for get_table_names method
* Add python unit test for get_table_names method
* Fix js linting error
* Bugfix to SQL Lab to support tables and schemas with characters that require quoting
* Remove debugging prints
* Add uri encoding to secondary tables call
* Quote schema names for presto
* Quote selected_schema on Snowflake, MySQL and Hive
* Remove redundant parens
* Add python unit tests
* Add js unit test
* Fix flake8 linting error
* Merge lastest from master into lyft-release-sp8 (#7405)
* filter out all nan series (#7313)
* improve not rich tooltip (#7345)
* Create issue_label_bot.yaml (#7341)
* fix: do not save colors without a color scheme (#7347)
* [wtforms] Strip leading/trailing whitespace (#7084)
* [schema] Updating the datasources schema (#5451)
* limit tables/views returned if schema is not provided (#7358)
* limit tables/views returned if schema is not provided
* fix typo
* improve code performance
* handle the case when table name or view name does not present a schema
* Add type anno (#7342)
* Updated local dev instructions to include missing step
* First pass at type annotations
* [schema] Updating the base column schema (#5452)
* Update 937d04c16b64_update_datasources.py (#7361)
* Feature flag for client cache (#7348)
* Feature flag for client cache
* Fix integration test
* Revert "Fix integration test"
This reverts commit 58434ab98a.
* Feature flag for client cache
* Fix integration tests
* Add feature flag to config.py
* Add another feature check
* Fix more integration tests
* Fix raw HTML in SliceAdder (#7338)
* remove backendSync.json (#7331)
* [bubbles] issue when using duplicated metrics (#7087)
* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)
* SUPERSET-8: Update text in docs copyright footer (#7360)
* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04
* SUPERSET-8: Extra text in docs copyright footer
* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)
* Store last selected dashboard in sessionStorage (#7181)
* Store last selected dashboard in sessionStorage
* Fix tests
* [schema] Updating the base metric schema (#5453)
* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)
* feat: see Presto row and array data types (#7391)
* feat: see Presto row and array data types
* fix: address PR comments
* fix: lint and build issues
* fix: add types
* Incorporate feedback from initial PR (prematurely merged to lyft-release-sp8) (#7415)
* add stronger type hints where possible
* fix: lint issues and add select_star func in Hive
* add missing pkg init
* fix: build issues
* fix: pylint issues
* fix: use logging instead of print
* feat: view presto row objects in data grid
* fix: address feedback
* fix: spacing
* Merge lastest from master into lyft-release-sp8 (#7405)
* filter out all nan series (#7313)
* improve not rich tooltip (#7345)
* Create issue_label_bot.yaml (#7341)
* fix: do not save colors without a color scheme (#7347)
* [wtforms] Strip leading/trailing whitespace (#7084)
* [schema] Updating the datasources schema (#5451)
* limit tables/views returned if schema is not provided (#7358)
* limit tables/views returned if schema is not provided
* fix typo
* improve code performance
* handle the case when table name or view name does not present a schema
* Add type anno (#7342)
* Updated local dev instructions to include missing step
* First pass at type annotations
* [schema] Updating the base column schema (#5452)
* Update 937d04c16b64_update_datasources.py (#7361)
* Feature flag for client cache (#7348)
* Feature flag for client cache
* Fix integration test
* Revert "Fix integration test"
This reverts commit 58434ab98a.
* Feature flag for client cache
* Fix integration tests
* Add feature flag to config.py
* Add another feature check
* Fix more integration tests
* Fix raw HTML in SliceAdder (#7338)
* remove backendSync.json (#7331)
* [bubbles] issue when using duplicated metrics (#7087)
* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04 (#7359)
* SUPERSET-8: Update text in docs copyright footer (#7360)
* SUPERSET-7: Docker compose config version breaks on Ubuntu 16.04
* SUPERSET-8: Extra text in docs copyright footer
* [schema] Adding commits and removing unnecessary foreign-key definitions (#7371)
* Store last selected dashboard in sessionStorage (#7181)
* Store last selected dashboard in sessionStorage
* Fix tests
* [schema] Updating the base metric schema (#5453)
* Fix NoneType bug & fill the test recipients with original recipients if empty (#7365)
* feat: see Presto row and array data types (#7391)
* feat: see Presto row and array data types
* fix: address PR comments
* fix: lint and build issues
* fix: add types
* add stronger type hints where possible
* fix: lint issues and add select_star func in Hive
* add missing pkg init
* fix: build issues
* fix: pylint issues
* fix: use logging instead of print
* Bump FAB to 2.0.0
* [tests] whitelist SecurityApi login and refresh endpoints
* [style] Fix, C812 missing trailing commas
* [security] Remove SUPERSET_UPDATE_PERMS flag
Registering sources needs to be performed after the views are
initialized on UPDATE_PERMS=False configuration
* [docs] New, FAB_UPDATE_PERMS and flask fab cli
* [docs] Fix, db upgrade needs to come first, create-admin needs a db
* [cli] New, superset init bootstraps all permissions for FAB and Superset
* [style] Fix, flakes
* Adding permission for can_only_access_owned_queries
* Fixing lint adding typing to variable
* Adding test for queryview and enabling /queryview/api/read
* Fixing issues with python typing
* Sparkline dates aren't formatting in Time Series Table (#6976)
* Exclude venv for python linter to ignore
* Fix NaN error
* Fix the white background shown in SQL editor on drag (#7021)
This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends.
* Show tooltip with time frame (#6979)
* Fix time filter control (#6978)
* Enhancement of query context and object. (#6962)
* added more functionalities for query context and object.
* fixed cache logic
* added default value for groupby
* updated comments and removed print
(cherry picked from commit d5b9795f87)
* [fix] /superset/slice/id url is too long (#6989)
(cherry picked from commit 6a4d507ab6)
* [WIP] fix user specified JSON metadata not updating dashboard on refresh (#7027)
(cherry picked from commit cc58f0e661)
* feat: add ability to change font size in big number (#7003)
* Add ability to change font sizes in Big Number
* rename big number to header
* Add comment to clarify font size values
* Allow LIMIT to be specified in parameters (#7052)
* [fix] Cursor jumping when editing chart and dashboard titles (#7038)
(cherry picked from commit fc1770f7b7)
* Changing time table viz to pass formatTime a date (#7020)
(cherry picked from commit 7f3c145b1f)
* [db-engine-spec] Aligning Hive/Presto partition logic (#7007)
(cherry picked from commit 05be866117)
* [fix] explore chart from dashboard missed slice title (#7046)
(cherry picked from commit a6d48d4052)
* fix inaccurate data calculation with adata rolling and contribution (#7035)
(cherry picked from commit 0782e831cd)
* Adding warning message for sqllab save query (#7028)
(cherry picked from commit ead3d48133)
* [datasource] Ensuring consistent behavior of datasource editing/saving. (#7037)
* Update datasource.py
* Update datasource.py
(cherry picked from commit c771625f10)
* [csv-upload] Fixing message encoding (#6971)
(cherry picked from commit 48431ab5b9)
* [sql-parse] Fixing LIMIT exceptions (#6963)
(cherry picked from commit 3e076cb60b)
* Adding custom control overrides (#6956)
* Adding extraOverrides to line chart
* Updating extraOverrides to fit with more cases
* Moving extraOverrides to index.js
* Removing webpack-merge in package.json
* Fixing metrics control clearing metric
(cherry picked from commit e6194051f4)
* [sqlparse] Fixing table name extraction for ill-defined query (#7029)
(cherry picked from commit 07c340cf82)
* [missing values] Removing replacing missing values (#4905)
(cherry picked from commit 61add606ca)
* [SQL Lab] Improved query and results tabs rendering reliability (#7082)
closes#7080
(cherry picked from commit 9b58e9f492)
* Fix filter_box migration PR #6523 (#7066)
* Fix filter_box migration PR #6523
* Fix druid-related bug
(cherry picked from commit b210742ad2)
* SQL editor layout makeover (#7102)
This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board
(cherry picked from commit 71f1bbd2ec)
* [forms] Fix handling of NULLs
(cherry picked from commit e83a07d3df)
* handle null column_name in sqla and druid models
(cherry picked from commit 2ff721ae07)
* Use metric name instead of metric in filter box (#7106)
(cherry picked from commit 003364e74e)
* Bump python lib croniter to an existing version (#7132)
Package maintainers should really never delete packages, but it appears
this happened with croniter and resulted in breaking our builds.
This PR bumps to a more recent existing version of the library
(cherry picked from commit 215ed392a1)
* Revert PR #6933 (#7162)
* Celery worker for warming up cache
* Remove testing changes
* Add documentation
* Fix lint
* WIP dashboard filters
* Use new cache so it works with dashboards
* Add more unit tests, fix old ones
* Fix flake8 and docs
* Sparkline dates aren't formatting in Time Series Table (#6976)
* Exclude venv for python linter to ignore
* Fix NaN error
* Changing time table viz to pass formatTime a date (#7020)
(cherry picked from commit 7f3c145b1f)
* SQL editor layout makeover (#7102)
This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board
(cherry picked from commit 71f1bbd2ec)
* Celery worker for warming up cache
* Remove testing changes
* Add documentation
* Fix lint
* WIP dashboard filters
* Use new cache so it works with dashboards
* Add more unit tests, fix old ones
* Fix flake8 and docs
* Fix bad merge and pylint
* Enable color consistency in a dashboard
Moved actions, minor UI, allowed dashboard copy
Fix linting errors
Undo unintentional change
Updated and added unit tests
Fail quietly if package has not been updated
Fail quietly on dashboard copy if package is old
* Update packages
* Remove unnecessary code
* Addressed Grace's comments
* Small fix for item key
* Reset chart's color during exploration
* Do not reset chart form data when exploring chart
* Add ability to inject statsd client; some py test/reqs updates
- Updated the metrics logger to allow construction with an existing
statsd client, so that it can be configured by external systems or libs.
- added requirements to requirements-dev.txt which are needed to run
tests-eg coverage, nose
- removed dependency on mock lib, it is in python stdlib now
- updated tox.ini to remove the now-superfluous deps
* add license to test file, and remove blank line at EOF
* Bump sqla to >=1.3.1
* Refine mssql column types to only use N-prefixing when necessary
* make join explicit
* replace set with list
* Add additional test case for N-prefix
* Replace engine with dialect and fix linting error
* Remove unneeded import
* Allow for dynamic feature flags
Giving more control over feature flags, allowing administrator to define
custom logic around whether features are enabled for particular users /
roles.
The exposed function can be used for things like:
* progressive rollout of features (1%, 5%, 50%, 100%)
* experimentation
* role-based feature affectation (only admins see a particular feature)
* fix build
* Addressing comments
* Addressing @hughhh's comments
* Force quotes on non-expression time grains on postgres
* Change to or semantics
* Return lower case column name as unmutated
* Add testcases for postgres time grains
* Make expression MixedCase
* Merge dataframe and column name mutation logic, add flag for disabling column aliases and add column name length checking
* Remove custome mutate_label from oracle spec
* Move hashing from mutate_label() to make_label_compatible()
* Remove empty line
* Make label mutating and truncating more robust
* Rename variables and make proposed changes from review
* Always execute labels_expected codepath
* Fix linting error
* Add comments and fix subquery errors
* Refine column compatibility
* Simplify label assignment
* Add unit tests for BQ and Oracle
* Linting
* Adding dropdown to DatasourceControl and ability to change datasource
* Style fixes
* Adding unit tests for datasource/get endpoint
* Fixing issue with dropdown overflow and style changes
* Fixing issues rebasing metadata button and fixing sort for datasource with no name
Summary: We want to allow grouping by on expressions, including the
ability to select expressions (without group-by). The UI ain't good yet
.. it is not at the feature parity of say adhoc filters/metrics that
come with a nice text-box to edit the SQL. But
this suffices for my usecase for now.
(The UI would have to be redone ... but that would require some refactoring
to merge the AdhocFilter/Metric/Popover stuff such that it can be used
for this use case too.)
Also fixed a bug in the SelectControl freeForm selection.
Allow selecting all columns easily: Added a new Select-All special item
that users can select to make all the options in the selectable show
up.
The ability to group by arbitrary expressions is useful because now two
users don't need to create custom computed-fields for this.
- Rename the default feature flags key in `config.py` to DEFAULT_FEATURE_FLAGS
- Merge default feature flags with user defined ones allowing the latter to overwrite the former
- Expose feature_flags for both server and client to use
- Add a utility method for checking whether a feature flag is on on server side
* Add licenses
* More licenses
* Ignore messages.json as they are generated
* More licenses
* Also typescript
* Fix alignment
* Add to svg
* Many more licenses
* more licenses
* Better excludes
* Add licenses to docs and md files
* Pre-finalize licenses
* Fix newlines
* Finalize all sourde licenses
* Fix lint
* Making thrift, pyhive and tableschema as extra_requires
Looking at the dependency tree for license related questions, I noticed
that tableschema had a huge tree, and only people running Hive really
need it. Making this as well as pyhive and thrift optional.
Also bumping some python dependencies
* Run pip-compile
* Removing refs to past.builtins (from future lib)
* Add thrift
* [cosmetic] CRUD's list from 'Add Filter' to 'Filter Results'
* Add global new button
* A better viz type selector
* lint
* Move view to new module, add unit test
* 'Filter Results' -> 'Filter'
* db merge
* Filter -> 'Filter List'
* [WiP] deprecate auto-generated metrics & fix examples
Picking up some leftover pieces not needed anymore since the
MetricsControl
* db merge
* db merge
* fix migration
* Creating metrics required for tests
* [scheduled reports] Add support for scheduled reports
* Scheduled email reports for slice and dashboard visualization
(attachment or inline)
* Scheduled email reports for slice data (CSV attachment on inline table)
* Each schedule has a list of recipients (all of them can receive a single mail,
or separate mails)
* All outgoing mails can have a mandatory bcc - for audit purposes.
* Each dashboard/slice can have multiple schedules.
In addition, this PR also makes a few minor improvements to the celery
infrastructure.
* Create a common celery app
* Added more celery annotations for the tasks
* Introduced celery beat
* Update docs about concurrency / pools
* [scheduled reports] - Debug mode for scheduled emails
* [scheduled reports] - Ability to send test mails
* [scheduled reports] - Test email functionality - minor improvements
* [scheduled reports] - Rebase with master. Minor fixes
* [scheduled reports] - Add warning messages
* [scheduled reports] - flake8
* [scheduled reports] - fix rebase
* [scheduled reports] - fix rebase
* [scheduled reports] - fix flake8
* [scheduled reports] Rebase in prep for merge
* Fixed alembic tree after rebase
* Updated requirements to latest version of packages (and tested)
* Removed py2 stuff
* [scheduled reports] - fix flake8
* [scheduled reports] - address review comments
* [scheduled reports] - rebase with master
* Use py3's f-strings instead of s.format(**locals())
In light of the bug reported here
https://github.com/apache/incubator-superset/issues/6347, which seems
like an odd `.format()` issue in py3, I greped and replaced all
instances of `.format(**locals())` using py3's f-strings
* lint
* fix tests
* Adding cypress tests for showing erros, time range filter, and verbose name
* Adding tests for time range without ago
* Adding test to verify error message showing
* [SIP-5] Open a new /api/v1/query endpoint that takes query_obj
- Introduce a new handle_superset_exception decorator to avoid repeating the logic for catching SupersetExceptions
- Create a query_obj_backfill method that takes form_data and constructs a query_obj that will be constructed in the client in the future. Use the backfill in explore_json.
- Create a new /api/v1/query endpoint that takes query_obj only and returns the payload data. Note the query_obj is constructed in the client. The endpoint currently only handles query_obj for table view viz (we'll be adding support to new viz types as we go).
- Unit test to verify the new endpoint for table view
* fix tests and lint errors
* - Move the new query endpoint into its own api.py view.
- Create QueryObject and QueryContext class to encapsulate query_object to be built from the client and additional info (e.g. datasource) needed to get the data payload for a given query
- Remove the query_obj_backfill as we'll start building the first query_object on the client so it no longer makes sense to have a short-lived backfill for the matter of days.
* Fixing lint and test errors
* Fixing additional lint error from the previous rebase.
* fixing additional lint error
* addressing additional pr comments
* Make /query accept a list of queries in the query_context object.
* fixing a lint error
* - Move time_shift based calculation and since, until check into util
- Add typing info for get_since_until
- Add new unit tests to verify time_shift calculation and the since until check
* Add separate limit setting for SqlLab
Use separate param for wrap sql
Get query limit from config
unit tests for limit control rendering in sql editor
py unit test
pg tests
Add max rows limit
Remove concept of infinity, always require defined limits
consistency
Assert on validation errors instead of tooltip
fix unit tests
attempt persist state
pr comments and linting
* load configs in via common param
* default to 1k
* [utils] gathering/refactoring into a "utils/" folder
Moving current utils.py into utils/core.py and moving other *util*
modules under this new "utils/" as well.
Following steps include eroding at "utils/core.py" and breaking it down
into smaller modules.
* Improve tests
* Make loading examples in scope for tests
* Remove test class attrs examples_loaded and requires_examples
* feat: import/export dashboards via cli
* style: fixed lint error
* test: added test for import and export util
* test: removing import test as it is causing integrity issues
Import is a wrapper around exist functionality so we can go ahead without a test or mock the actual db operation using https://docs.python.org/3/library/unittest.mock.html
And validate the wrapper operations only.
* test: remove test data file
* test: removed usage of reserved keyword id
* Add schema level access control on csv upload
* add db migrate merge point
* fix flake 8
* fix test
* remove unnecessary db migration
* fix flake
* nit
* fix test for test_schemas_access_for_csv_upload_endpoint
* fix test_csv_import test
* use security_manager to check whether schema is allowed to be accessed
* bring security manager to the party
* flake8 & repush to retrigger test
* address comments
* remove trailing comma
* Adds new macro to get filter values from "filters" and "extra_filters"
Adds test for filter_values macro
Adds doco for filter_values
Changes filter_values return type to be a list rather than string
* Makes return value type consistent
- filter_values always return a list
* Bug: fixing async syntax for python 3.7
Rename async to async_ so superset installs for python 3.7.
* Addressing PR comments. Use kwargs instead of explicitly specifying async_ so downstream engines (e.g. PyHive) that supports async can choose to use the async_ in pythonwq3.7 and async in <=python3.6
* addressing additional pr comments
* Field names in big query can contain only alphanumeric and underscore
* bad quote
* better place for mutating labels
* lint
* bug fix thanks to mistercrunch
* lint
* lint again
* Fixing issue with extra params in formData
* Pass in param use_slice_data to decide whether to use slice data
* Fixing core_tests to not use explore_json overrides
* Improve URLs for Chart and Dashboard ModelViews
Prior to this, the ModelView for Chart and Dashboard would be
at `/slicemodelview/list/` and `/dashboardmodelview/list/`.
Now we have cleaner URLs at `/chart/list/` and `/dashboard/list/`
* Fix unrelated js lint
* addressing comments
* Add function to fix dataframe column case
* Fix broken handle_nulls method
* Add case sensitivity option to dedup
* Refactor function definition and call location
* Remove added blank line
* Move df column rename logit to db_engine_spec
* Remove redundant variable
* Update comments in db_engine_specs
* Tie df adjustment to db_engine_spec class attribute
* Fix dedup error
* Linting
* Check for db_engine_spec attribute prior to adjustment
* Rename case sensitivity flag
* Linting
* Remove function that was moved to db_engine_specs
* Get metrics names from utils
* Remove double import and rename dedup variable
* [sql lab] simplify the visualize flow
The "visualize flow" linking SQL Lab to the "explore view" has never
worked so great for people, here's a list of issues:
* it's not really clear to users that their query is wrapped as a
subquery, and the explore view runs queries on top of it
* lint + fix tests
* Addressing comments
* Add interim grains
* Refactor and add blacklist
* Change PT30M to PT0.5H
* Linting
* Linting
* Add time grain addons to config.py and refactor engine spec logic
* Remove redundant import and clean up config.py
* Fix bad rebase
* Implement changes proposed by @betodealmeida
* Revert removal of name from Grain
* Linting
* [sql lab] extract Hive error messages
So pyhive returns an exception object with a stringified thrift error
object. This PR uses a regex to extract the errorMessage portion of that
string.
* Unit test
* Visualization Unicode bug fix
* Fix the build (#5403)
The travis build has been failing for 2 reasons recently
* pylint takes > 10 minutes without outputing
* bad merge confict auto resolve in controls.jsx
* Visualization Unicode bug fix
* Migrate flask_script to the Flask built-in click.
Flask 0.11 is the built-in integration of the click command line interface.
Flask-Migrate support for the new Flask CLI based on Click after Release 2.0.0.
* Resolved merge conflicts.
* Fixed issue introduced from bad merge.
* Fixed flake8 errors, added build to excluded flake8 stuff.
* * Moved the FlaskGroup declaration to the driver script.
* Moved shell context definition to cli.py
* Switched shell context definition to use decorator.
* Moved create_app definition to cli.py
* Fixed InvocationError with a wrapped function
* Added extra newlines between functions
* Removed flask-script dependency.
* Improve database type inference
Python's DBAPI isn't super clear and homogeneous on the
cursor.description specification, and this PR attempts to improve
inferring the datatypes returned in the cursor.
This work started around Presto's TIMESTAMP type being mishandled as
string as the database driver (pyhive) returns it as a string. The work
here fixes this bug and does a better job at inferring MySQL and Presto types.
It also creates a new method in db_engine_specs allowing for other
databases engines to implement and become more precise on type-inference
as needed.
* Fixing tests
* Adressing comments
* Using infer_objects
* Removing faulty line
* Addressing PrestoSpec redundant method comment
* Fix rebase issue
* Fix tests
* fetch datasources from broker endpoint when refresh new datasources
* remove get_base_coordinator_url as out of use
* add broker_endpoint in get_test_cluster_obj
* Added support for URLShortLinkButton to work for the dashboard case
* Fix lint errors and test
* Change references to 'slice' to 'chart'.
* Add unit tests to improve coverage
* Fixing lint errors
* Refactor to make URLShortLink more generic. Remove history modification code, redirect should be handling this.
* Remove history modification code, redirect should be handling this
* Generate a shorter link without the directory, and delegate default linked to the contents of window.location
* Fix lint errors
* Fix test_shortner test to check for new pattern
* Remove usage of addHistory to manipulate explore shortlink redirection
* Address build failure and using better practices for shortlink defaults
* Fixing alphabetical order
* More syntax mistakes
* Revert explore view history changes
* Fix use of component props, & rebase
* Initial test
* Save
* Working version
* Use since/until from payload
* Option to prefix metric name
* Rename LineMultiLayer to MultiLineViz
* Add more styles
* Explicit nulls
* Add more x controls
* Refactor to reuse nvd3_vis
* Fix x ticks
* Fix spacing
* Fix for druid datasource
* Rename file
* Small fixes and cleanup
* Fix margins
* Add proper thumbnails
* Align yaxis1 and yaxis2 ticks
* Improve code
* Trigger tests
* Move file
* Small fixes plus example
* Fix unit test
* Remove SQL and Filter sections
* Fix percent_metrics ZeroDivisionError and can not get metrics with label issue
* convert iterator to list
* get percentage metrics with get_metric_label method
* Adding tests case for expression type metrics
* Simplify expression
* [sql lab] a better approach at limiting queries
Currently there are two mechanisms that we use to enforce the row
limiting constraints, depending on the database engine:
1. use dbapi's `cursor.fetchmany()`
2. wrap the SQL into a limiting subquery
Method 1 isn't great as it can result in the database server storing
larger than required result sets in memory expecting another fetch
command while we know we don't need that.
Method 2 has a positive side of working with all database engines,
whether they use LIMIT, ROWNUM, TOP or whatever else since sqlalchemy
does the work as specified for the dialect. On the downside though
the query optimizer might not be able to optimize this as much as an
approach that doesn't use a subquery.
Since most modern DBs use the LIMIT syntax, this adds a regex approach
to modify the query and force a LIMIT clause without using a subquery
for the database that support this syntax and uses method 2 for all
others.
* Fixing build
* Fix lint
* Added more tests
* Fix tests
* Force lowercase column names for Snowflake and Oracle
* Force lowercase column names for Snowflake and Oracle
* Remove lowercasing of DB2 columns
* Remove DB2 lowercasing
* Fix test cases
* add extraction fn support for Druid queries
* bump pydruid version to get extraction fn commits
* update and add tests for druid for filters with extraction fns
* conform to flake8 rules
* fix flake8 issues
* bump pyruid version for extraction function features
* [bugfix] temporal columns with expression fail
error msg: "local variable 'literal' referenced before assignment"
Error occurs [only] when using temporal column defined as a SQL
expression.
Also noticed that examples were using `granularity` instead of using
`granularity_sqla` as they should. Fixed that here.
* Add tests
* [WiP] [explore] proper filtering of NULLs and ''
TODO: handling of Druid equivalents
* Unit tests
* Some refactoring
* [druid] fix 'Unorderable types' when col has nuls
Error "unorderable types: str() < int()" occurs when grouping by a
numerical Druid colummn that contains null values.
* druid/pydruid returns strings in the datafram with NAs for nulls
* Superset has custom logic around get_fillna_for_col that fills in the
NULLs based on declared column type (FLOAT here), so now we have a mixed
bag of type in the series
* pandas chokes on pivot_table or groupby operations as it cannot sorts
mixed types
The approach here is to stringify and fillna('<NULL>') to get a
consistent series.
* typo
* Fix druid_func tests
* Addressing more comments
* last touches
* [sql lab] preserve schema through visualize flow
https://github.com/apache/incubator-superset/pull/4696 got tangled
into refactoring views out of views/core.py and onto views/sql_lab.py
This is the same PR without the refactoring.
* Fix lint
* [bugfix] convert metrics to numeric in dataframe
It appears sometimes the dbapi driver and pandas's read_sql fail at
returning the proper numeric types for metrics and they show up as
`object` in the dataframe. This results in "No numeric types to
aggregate" errors when trying to perform aggregations or pivoting in
pandas.
This PR looks for metrics in dataframes that are typed as "object"
and uses pandas' to_numeric to convert.
* Fix tests
* Remove all iteritems
* Allowing limit ordering by post-aggregation metrics
* don't overwrite og dictionaries
* update tests
* python3 compat
* code review comments, add tests, implement it in groupby as well
* python 3 compat for unittest
* more self
* Throw exception when get aggregations is called with postaggs
* Treat adhoc metrics as another aggregation
* move access permissions methods to security manager
* consolidate all security methods into SupersetSecurityManager
* update security method calls
* update calls from tests
* move get_or_create_main_db to utils
* raise if supersetsecuritymanager is not extended
* rename sm to security_manager
* [Explore] Save url parameters when user save slices
* remove print
(cherry picked from commit bd9ecbe)
* add unit test
(cherry picked from commit 0f350ad)
* wrapping all request params into url_params
(cherry picked from commit 17197c1)
* Remove comments from queries in SQL Lab that break Explore view
This fixes an issue where comments on the last line of the source query comment out the closing parenthesis of the subquery.
* Add test case for SqlaTable with query with comment
This test ensures that comments in the query are removed when calling SqlaTable.get_from_clause().
* Add missing blank line class definition (PEP8)
* [Explore view] Use POST method for charting requests
* fix per code review comments
* more code review fixes
* code review fix: remove duplicated calls for getting values from request
* [Explore view] Use POST method for charting requests
* fix per code review comments
* more code review fixes
* code review fix: remove duplicated calls for getting values from request
* Use the query_obj as the basis for the cache key
When we recently moved from hashing form_data to define the cache_key
towards using the rendered query instead,
it made is such that non deterministic form
control values like relative times specified in "from" and "until" time
bound resulted in making those miss cache 100% of the time.
Here we move away from using the rendered query and using the query_obj
instead.
* Deprecating using form_data in templates
* conditional check on datatype of results before converting to df
fix type checking
fix conditional checks
remove trailing whitespace and fix df_data fallback def
actually remove trailing whitespace
generalized type check to check all columns for dict
refactor dict col check
* move df conversion to helper and add unit test
add missing newlines
another missing newline
fix quotes
more quote fixes
* Fix USA's state geojson for 'Country Map' visualization
Turns out the ISO codes were missing from the geojson file, this adds it
and uses human-readable indents.
* using proper ISO codes
* Linting
New linting rules started applying, I'm guessing a new version of
pylint?
* Adding full Annotation Framework
* Viz types
* Re organizing native annotations
* liniting
* Bug fix
* Handle no data
* Cleanup
* Refactor slice form_data to data
Before this PR the only way to query lat/long is in the shape of 2
columns that contains lat and long.
Now we're adding 2 more options:
* a single column that has lat and long with a delimiter in between
* support for geohashes - geohashes are cool
* change reference for slices to chart
* change profile page reference
* change reference for Associated Slices
* change back to single quotes
* fix other single quotes
* linting
* last one
* fix test
A bug in to_dict(orient="records") in pandas/core/frame.py prevents
datetimes with time zones to be worked with. This works around the
issue in superset by re-implementing the logic of pandas in the
correct way. Until pandas fixes the issue this code should stay.
https://github.com/pandas-dev/pandas/issues/18372
This closes#1929
Ville Brofeldt
Dave Smith
Beto Dealmeida
Maxime Beauchemin
serenajiang
Daniel Vaz Gaspar
John Bodley
Erik Ritter
Gianluca Ciccarelli
Maxim Sukharev
Arpit
Rob DiCiuccio
Tresdon Jones
Kim Truong
michellethomas
Jim Horng
Alex Berghage
Raphaël Merx
agrawaldevesh
Christine Chambers
bolkedebruin
Chinh Nguyen
Mahendra M
Junda Yang
Jiaji Zhou
timifasubaa
Christine Chambers
Jeffrey Wang
Grace Guo
yamyamyuo
Manuel Silva
Sumedh Sakdeo
JamshedRahman
gbates101
Chris Williams
Yuance.Li
Tamika Tannis
Timi Fasubaa
Gabe Lyons
Yongjie Zhao
jasnovak
Hugh A. Miles II
Riccardo Magliocchetti
John Bodley
Jeff Niu
Marcus Levine
fabianmenges
Nicolas Bonnotte