pt/tps
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
96 Commits 6 Branches 0 Tags 2 MiB
PLpgSQL 67.8%
JavaScript 12.9%
CSV 10.4%
HTML 8.7%
Shell 0.2%
75d511f6a4
Go to file
Paul Trowbridge 75d511f6a4 regex modification template
2017-10-19 17:39:29 -04:00
build_json.xlsx add discover transactions 2017-10-07 15:45:48 -04:00
coa.md add credit cards 2017-10-13 09:18:48 -04:00
col_balance.pgsql add calc columns to collateral 2017-10-19 13:10:36 -04:00
event_log.md expirement with different formats and add a receipt 2017-09-20 00:12:37 -04:00
event_log.pgsql link in header 2017-08-24 23:46:15 -04:00
LICENSE add event record example 2017-07-24 23:13:34 -04:00
loan_bal.pgsql add running collateral & loan balances 2017-10-19 11:20:52 -04:00
log.md add excel file with json build macro 2017-09-18 21:07:22 -04:00
map_rm.pgsql regex modification template 2017-10-19 17:39:29 -04:00
new_format.pgsql build macro for creating record 2017-10-03 01:47:30 -04:00
readme.md add event record example 2017-07-24 23:13:34 -04:00
rebuild_pg.cmd add rebuild script 2017-10-13 11:41:43 -04:00
rec.json updates 2017-08-24 22:57:21 -04:00
srce.pgsql revise import to work with a set of unique keys in determining which rows to insert. prior used many of the same key, creating large duplication on join to tps 2017-10-19 13:10:25 -04:00
transaction_range.pgsql add order by clause 2017-10-19 09:25:07 -04:00
ubm_backup.cmd dont export tps.trans table 2017-10-19 09:24:55 -04:00
ubm_data.sql alter regex defs to have a retention flag indicating if the return value shoudl be mapped or thrown away 2017-10-19 17:37:46 -04:00
ubm_schema.sql add jsonb object aggregate functions, change regex to jsonb, remove seq from primary key of map_rm, add foreign key to map_rv pointing map_rm 2017-10-19 17:38:58 -04:00

readme.md

Concepts

pull various static files into postgres and do basic transformation without losing the original document or getting into custom code for each scenario

Storage

all records are jsonb applied mappings are in associated jsonb documents

Import

COPY function utilized

Mappings

  1. regular expressions are used to extract pieces of the json objects
  2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document

Transformation tools

  • COPY
  • regexp_matches()

Difficulties

Non standard file formats will require additional logic example: PNC loan balance and collateral CSV files

  1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
  2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources

Interface

maybe start out in excel until it gets firmed up

  • list existing mappings
    • apply mappings to see what results come back
  • experiment with new mappings