2017-07-07 14:11:25 -04:00
|
|
|
Concepts
|
|
|
|
|
======================================
|
2017-07-07 13:54:13 -04:00
|
|
|
|
2017-07-07 14:11:25 -04:00
|
|
|
pull various static files into postgres and do basic transformation without losing the original document
|
|
|
|
|
or getting into custom code for each scenario
|
|
|
|
|
|
2017-07-07 14:12:05 -04:00
|
|
|
## Storage
|
2017-07-07 14:11:25 -04:00
|
|
|
all records are jsonb
|
|
|
|
|
applied mappings are in associated jsonb documents
|
|
|
|
|
|
2017-07-07 14:12:05 -04:00
|
|
|
## Import
|
2017-07-07 14:11:25 -04:00
|
|
|
`COPY` function utilized
|
|
|
|
|
|
2017-07-07 14:12:05 -04:00
|
|
|
## Mappings
|
2017-07-07 14:11:25 -04:00
|
|
|
1. regular expressions are used to extract pieces of the json objects
|
|
|
|
|
2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document
|
|
|
|
|
|
2017-07-07 14:12:05 -04:00
|
|
|
## Transformation tools
|
2017-07-07 14:11:25 -04:00
|
|
|
* `COPY`
|
|
|
|
|
* `regexp_matches()`
|
|
|
|
|
|
2017-07-07 14:12:05 -04:00
|
|
|
## Difficulties
|
2017-07-07 14:11:25 -04:00
|
|
|
Non standard file formats will require additional logic
|
2017-07-07 15:59:26 -04:00
|
|
|
example: PNC loan balance and collateral CSV files
|
|
|
|
|
1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
|
|
|
|
|
2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources
|
