Commit Graph

127 Commits

Author SHA1 Message Date
Paul Trowbridge
2d6212e3cd add index to srce and couple import jobs 2017-10-27 16:26:00 -04:00
Paul Trowbridge
273b7c9673 limit mapper to allj records are null 2017-10-27 15:50:42 -04:00
Paul Trowbridge
fac52588ae update map to exclude 6 digits at end of company name 2017-10-27 14:18:57 -04:00
Paul Trowbridge
f88584f835 convert map flag to be at the regex item level 2017-10-27 13:51:08 -04:00
Paul Trowbridge
417bdca27d move all current map_rm into update scripts for editing 2017-10-27 13:23:08 -04:00
Paul Trowbridge
b94948b1b9 add new mappings and clean up old ones 2017-10-27 13:12:45 -04:00
Paul Trowbridge
2829887a57 get rid of bad maps 2017-10-27 13:08:03 -04:00
Paul Trowbridge
30da16dd0a add map definition query 2017-10-27 13:07:52 -04:00
Paul Trowbridge
4a62ca84b1 implement sequence of target, and insure that the link to the map does not return duplicates 2017-10-27 13:06:44 -04:00
Paul Trowbridge
6b08ea8fb8 add new maps, need to convert OBI & RFB maps to a single target 2017-10-26 17:36:03 -04:00
Paul Trowbridge
3924776161 update readme 2017-10-26 17:34:45 -04:00
Paul Trowbridge
9623853387 add map to parse all ACH transactions 2017-10-26 17:34:37 -04:00
Paul Trowbridge
7eba5981ef trim retained values 2017-10-26 17:34:21 -04:00
Paul Trowbridge
6a90cc4b32 add maps 2017-10-26 14:31:37 -04:00
Paul Trowbridge
2448ccf641 add new map, add column for all json, and apply index 2017-10-26 13:39:50 -04:00
Paul Trowbridge
bf8e901ffd add intermediate selects, add coalesce to target aggregation since some items are SQL null, then apply a strip_nulls, but exclude from subsequent step. add update line at end 2017-10-26 13:32:14 -04:00
Paul Trowbridge
fb93f73b6c create another mapping approach to better handle option g on regexp_ functions. 2017-10-26 12:15:58 -04:00
Paul Trowbridge
1af3f05a1e overhaul of do_map, add seperate column for parse versus mapped, accomodate rexexp_replace 2017-10-25 16:28:42 -04:00
Paul Trowbridge
a01c5c1a04 add mapping template 2017-10-25 15:21:13 -04:00
Paul Trowbridge
4a85732890 edit function to recognize slice of array, ad bank schema to hold types there, add function to json regex definition to accomodate differntiating extracts from replaces 2017-10-25 12:07:04 -04:00
Paul Trowbridge
8929b4eea9 try using correlated subquery for unique list of keys, is pretty slow 2017-10-25 12:05:28 -04:00
Paul Trowbridge
ba4a404980 added partion by major item, and then sort on import id 2017-10-25 10:45:24 -04:00
Paul Trowbridge
44373bc616 copy source template from pt branch 2017-10-25 10:44:57 -04:00
Paul Trowbridge
65ee6d703b modify to print log info 2017-10-25 10:44:32 -04:00
Paul Trowbridge
6056491ace merge srce changes from pt branch 2017-10-25 09:07:51 -04:00
Paul Trowbridge
937cf38af1 updates in progress 2017-10-24 17:37:20 -04:00
Paul Trowbridge
57fa123adf add transaction log 2017-10-23 10:13:32 -04:00
Paul Trowbridge
51fbec1244 push inserts into do block so they can consume local variables, need to setup log inserts 2017-10-23 10:12:37 -04:00
Paul Trowbridge
b3775210b5 merge pt branch updates 2017-10-20 10:09:55 -04:00
Paul Trowbridge
ccd685ca22 implement retain flag to drop keys used only for linking to the appropriate map value, but not acutally included in the map 2017-10-19 18:02:52 -04:00
Paul Trowbridge
9001c44b46 regex mapping script 2017-10-19 17:39:52 -04:00
Paul Trowbridge
75d511f6a4 regex modification template 2017-10-19 17:39:29 -04:00
Paul Trowbridge
2979fd3437 add jsonb object aggregate functions, change regex to jsonb, remove seq from primary key of map_rm, add foreign key to map_rv pointing map_rm 2017-10-19 17:38:58 -04:00
Paul Trowbridge
351b83a206 alter regex defs to have a retention flag indicating if the return value shoudl be mapped or thrown away 2017-10-19 17:37:46 -04:00
Paul Trowbridge
5369a39f34 add mapping tables & data 2017-10-19 14:13:57 -04:00
Paul Trowbridge
f203c7010a add calc columns to collateral 2017-10-19 13:10:36 -04:00
Paul Trowbridge
e68b3e0349 revise import to work with a set of unique keys in determining which rows to insert. prior used many of the same key, creating large duplication on join to tps 2017-10-19 13:10:25 -04:00
Paul Trowbridge
2a2652e397 qualify the order by id 2017-10-19 11:31:54 -04:00
Paul Trowbridge
f0838b28c0 retain the id number in the json rec, and order by id asc before inserting to trans. if mult records per key, this helps sort 2017-10-19 11:22:03 -04:00
Paul Trowbridge
2190f55acb add running collateral & loan balances 2017-10-19 11:20:52 -04:00
Paul Trowbridge
774540a83e remove assn & coll 2017-10-19 10:30:40 -04:00
Paul Trowbridge
e03a5cdbac backup 2017-10-19 10:29:33 -04:00
Paul Trowbridge
e0f8bad70a add ADPRP 2017-10-19 09:53:52 -04:00
Paul Trowbridge
1ae5298557 add index on tps.rec 2017-10-19 09:25:42 -04:00
Paul Trowbridge
dcf2fcbb6c update srce definitions 2017-10-19 09:25:31 -04:00
Paul Trowbridge
bc3cecec09 add order by clause 2017-10-19 09:25:07 -04:00
Paul Trowbridge
995eb181c1 dont export tps.trans table 2017-10-19 09:24:55 -04:00
Paul Trowbridge
e0163382f9 added sources 2017-10-19 08:49:33 -04:00
Paul Trowbridge
419017f98c switch to PNCC 2017-10-18 14:11:57 -04:00
Paul Trowbridge
748d6ddb3c merge summary of records not inserted from pt branch 2017-10-18 14:10:07 -04:00