pt/tps
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
146 Commits 6 Branches 0 Tags 2 MiB
PLpgSQL 67.8%
JavaScript 12.9%
CSV 10.4%
HTML 8.7%
Shell 0.2%
afa16ce925
Go to file
Paul Trowbridge afa16ce925 add items to log and create summary pivot
2017-10-28 02:35:24 -04:00
build_json.xlsx add items to log and create summary pivot 2017-10-28 02:35:24 -04:00
coa.md add credit cards 2017-10-13 09:18:48 -04:00
col_balance.pgsql added partion by major item, and then sort on import id 2017-10-25 10:45:24 -04:00
dcard_bal.pgsql add running discover card balance 2017-10-26 00:16:46 -04:00
do_map_g_option.pgsql add column to event_log. demonstact GL extraction and load initial row 2017-10-26 23:38:36 -04:00
do_map.pgsql merge wk 2017-10-26 18:46:42 -04:00
event_log.md expirement with different formats and add a receipt 2017-09-20 00:12:37 -04:00
event_log.pgsql link in header 2017-08-24 23:46:15 -04:00
evt_log_gl_extract.pgsql build interpretation of evt.log for running totals 2017-10-27 01:44:41 -04:00
LICENSE add event record example 2017-07-24 23:13:34 -04:00
loan_bal.pgsql added partion by major item, and then sort on import id 2017-10-25 10:45:24 -04:00
log.md add excel file with json build macro 2017-09-18 21:07:22 -04:00
map_rm_template.pgsql add column to event_log. demonstact GL extraction and load initial row 2017-10-26 23:38:36 -04:00
map_rm.pgsql update maps for first 20, get rid of pretty and tag on summary of mappings 2017-10-19 23:57:03 -04:00
readme.md update readme 2017-10-26 17:34:45 -04:00
rebuild_pg.cmd add rebuild script 2017-10-13 11:41:43 -04:00
rec.json updates 2017-08-24 22:57:21 -04:00
sqitch.conf add sqitch 2017-10-25 01:35:21 -04:00
sqitch.plan add sqitch 2017-10-25 01:35:21 -04:00
srce_defn.pgsql try using correlated subquery for unique list of keys, is pretty slow 2017-10-25 12:05:28 -04:00
srce_template.pgsql copy source template from pt branch 2017-10-25 10:44:57 -04:00
srce_unq.pgsql rename fle to srce_unq 2017-10-25 00:57:41 -04:00
srce.pgsql merge wk 2017-10-26 18:46:42 -04:00
summary.xlsx add items to log and create summary pivot 2017-10-28 02:35:24 -04:00
trans_log_template.pgsql catch up with wk branch 2017-10-25 00:42:22 -04:00
transaction_range.pgsql add order by clause 2017-10-19 09:25:07 -04:00
ubm_backup.cmd dont export tps.trans table 2017-10-19 09:24:55 -04:00
ubm_data.sql add items to log and create summary pivot 2017-10-28 02:35:24 -04:00
ubm_schema.sql add column to event_log. demonstact GL extraction and load initial row 2017-10-26 23:38:36 -04:00

readme.md

Concepts

pull various static files into postgres and do basic transformation without losing the original document or getting into custom code for each scenario

the is an in-between for an foreign data wrapper & custom programming

Storage

all records are jsonb applied mappings are in associated jsonb documents

Import

COPY function utilized

Mappings

  1. regular expressions are used to extract pieces of the json objects
  2. the results of the regular expressions are bumped up against a list of basic mappings and written to an associated jsonb document

a target represents a whole scenario that needs matched. it can contain several regex expressions. if one fails, then no match is attempted because it coudl result in a false positive based on the @> oeprator used at join time this probably isn't correctly implemented

Transformation tools

  • COPY
  • regexp_matches()

Difficulties

Non standard file formats will require additional logic example: PNC loan balance and collateral CSV files

  1. External: Anything not in CSV should be converted external to Postgres and then imported as CSV
  2. Direct: Outside logic can be setup to push new records to tps.trans direct from non-csv fornmated sources or fdw sources

Interface

maybe start out in excel until it gets firmed up

  • list existing mappings
    • apply mappings to see what results come back
  • experiment with new mappings