pt/tps
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

add event record example

Browse Source
This commit is contained in:
2017-07-24 23:13:34 -04:00
parent fbba0a1ada
commit 4c6f4f04b0
3 changed files with 59 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
LICENSE
View File

@ -1,21 +1,21 @@
MIT License
Copyright (c) 2017
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
MIT License
Copyright (c) 2017
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

7
event_log.pgsql Normal file
View File

@ -0,0 +1,7 @@
SELECT
'MODULE',
'2017-06-01'::DATE pdate,
'2017-06-01'::DATE tdate,
$${"attribute1":"value","attribute2":"value","attribute3":"value","attribute4":"value","attribute5":"value","attribute5":"value"}$$::jsonb,
$${"account":"amount"}$$::jsonb ledger

62
readme.md
View File

@ -1,32 +1,32 @@
Concepts
======================================
pull various static files into postgres and do basic transformation without losing the original document
or getting into custom code for each scenario
## Storage
all records are jsonb
applied mappings are in associated jsonb documents
## Import
`COPY` function utilized
## Mappings
1. regular expressions are used to extract pieces of the json objects
2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document
## Transformation tools
* `COPY`
* `regexp_matches()`
## Difficulties
Non standard file formats will require additional logic
example: PNC loan balance and collateral CSV files
1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources
## Interface
maybe start out in excel until it gets firmed up
* list existing mappings
* apply mappings to see what results come back
Concepts
======================================
pull various static files into postgres and do basic transformation without losing the original document
or getting into custom code for each scenario
## Storage
all records are jsonb
applied mappings are in associated jsonb documents
## Import
`COPY` function utilized
## Mappings
1. regular expressions are used to extract pieces of the json objects
2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document
## Transformation tools
* `COPY`
* `regexp_matches()`
## Difficulties
Non standard file formats will require additional logic
example: PNC loan balance and collateral CSV files
1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources
## Interface
maybe start out in excel until it gets firmed up
* list existing mappings
* apply mappings to see what results come back
* experiment with new mappings