pt/tps
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

add event record example

Browse Source
This commit is contained in:
2017-07-24 23:13:34 -04:00
parent fbba0a1ada
commit 4c6f4f04b0
3 changed files with 59 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
LICENSE
View File

@ -1,21 +1,21 @@
MIT License MIT License
Copyright (c) 2017 Copyright (c) 2017
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions: furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software. copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE. SOFTWARE.

7
event_log.pgsql Normal file
View File

@ -0,0 +1,7 @@
SELECT
'MODULE',
'2017-06-01'::DATE pdate,
'2017-06-01'::DATE tdate,
$${"attribute1":"value","attribute2":"value","attribute3":"value","attribute4":"value","attribute5":"value","attribute5":"value"}$$::jsonb,
$${"account":"amount"}$$::jsonb ledger

62
readme.md
View File

@ -1,32 +1,32 @@
Concepts Concepts
====================================== ======================================
pull various static files into postgres and do basic transformation without losing the original document pull various static files into postgres and do basic transformation without losing the original document
or getting into custom code for each scenario or getting into custom code for each scenario
## Storage ## Storage
all records are jsonb all records are jsonb
applied mappings are in associated jsonb documents applied mappings are in associated jsonb documents
## Import ## Import
`COPY` function utilized `COPY` function utilized
## Mappings ## Mappings
1. regular expressions are used to extract pieces of the json objects 1. regular expressions are used to extract pieces of the json objects
2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document 2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document
## Transformation tools ## Transformation tools
* `COPY` * `COPY`
* `regexp_matches()` * `regexp_matches()`
## Difficulties ## Difficulties
Non standard file formats will require additional logic Non standard file formats will require additional logic
example: PNC loan balance and collateral CSV files example: PNC loan balance and collateral CSV files
1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV 1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources 2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources
## Interface ## Interface
maybe start out in excel until it gets firmed up maybe start out in excel until it gets firmed up
* list existing mappings * list existing mappings
* apply mappings to see what results come back * apply mappings to see what results come back
* experiment with new mappings * experiment with new mappings