diff --git a/LICENSE b/LICENSE index 8864d4a..911ea4a 100644 --- a/LICENSE +++ b/LICENSE @@ -1,21 +1,21 @@ -MIT License - -Copyright (c) 2017 - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to deal -in the Software without restriction, including without limitation the rights -to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all -copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. +MIT License + +Copyright (c) 2017 + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/event_log.pgsql b/event_log.pgsql new file mode 100644 index 0000000..fba1daf --- /dev/null +++ b/event_log.pgsql @@ -0,0 +1,7 @@ +SELECT + 'MODULE', + '2017-06-01'::DATE pdate, + '2017-06-01'::DATE tdate, + $${"attribute1":"value","attribute2":"value","attribute3":"value","attribute4":"value","attribute5":"value","attribute5":"value"}$$::jsonb, + $${"account":"amount"}$$::jsonb ledger + diff --git a/readme.md b/readme.md index dec35a5..6a326a9 100644 --- a/readme.md +++ b/readme.md @@ -1,32 +1,32 @@ -Concepts -====================================== - -pull various static files into postgres and do basic transformation without losing the original document -or getting into custom code for each scenario - -## Storage -all records are jsonb -applied mappings are in associated jsonb documents - -## Import -`COPY` function utilized - -## Mappings -1. regular expressions are used to extract pieces of the json objects -2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document - -## Transformation tools -* `COPY` -* `regexp_matches()` - -## Difficulties -Non standard file formats will require additional logic -example: PNC loan balance and collateral CSV files -1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV -2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources - -## Interface -maybe start out in excel until it gets firmed up -* list existing mappings - * apply mappings to see what results come back +Concepts +====================================== + +pull various static files into postgres and do basic transformation without losing the original document +or getting into custom code for each scenario + +## Storage +all records are jsonb +applied mappings are in associated jsonb documents + +## Import +`COPY` function utilized + +## Mappings +1. regular expressions are used to extract pieces of the json objects +2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document + +## Transformation tools +* `COPY` +* `regexp_matches()` + +## Difficulties +Non standard file formats will require additional logic +example: PNC loan balance and collateral CSV files +1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV +2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources + +## Interface +maybe start out in excel until it gets firmed up +* list existing mappings + * apply mappings to see what results come back * experiment with new mappings \ No newline at end of file