add throughts on new source definition
This commit is contained in:
		
							parent
							
								
									ba6a06e30a
								
							
						
					
					
						commit
						d3114d5759
					
				
							
								
								
									
										42
									
								
								readme.md
									
									
									
									
									
								
							
							
						
						
									
										42
									
								
								readme.md
									
									
									
									
									
								
							@ -66,16 +66,38 @@ source definition
 | 
			
		||||
----------------------------------------------------------------------
 | 
			
		||||
 | 
			
		||||
* **load data**
 | 
			
		||||
    * parsing function reference
 | 
			
		||||
        * csv_from_client --> load file dialog, send content as post body to backend, backend sends array of json as argument to database?
 | 
			
		||||
        * custom_pnc_parse --> load file dialog, send content as post body to backend, parse out data to array of json and send to db as large json object?
 | 
			
		||||
        * _note_ : the database will have to store in the json a reference to these functions which the browser will need to read in order to act on an import request
 | 
			
		||||
 | 
			
		||||
        * the brwosers role is to extract the contents of a file and send them as a post body to the backend for processing under target function (/parse_csv&q=source)
 | 
			
		||||
 | 
			
		||||
    * constraints
 | 
			
		||||
    * the brwosers role is to extract the contents of a file and send them as a post body to the backend for processing under target function `based on srce defintion`
 | 
			
		||||
        * the backend builds a json array of all the rows to be added and sends as an argument to a database insert function
 | 
			
		||||
            * build constraint key `based on srce definition`
 | 
			
		||||
            * handle violations
 | 
			
		||||
            * increment global key list (this may not be possible depending on if a json with variable length arrays can be traversed)
 | 
			
		||||
            * build an import log
 | 
			
		||||
            * run maps (as opposed to relying on trigger)
 | 
			
		||||
* **read data**
 | 
			
		||||
    * top level key to table as type?
 | 
			
		||||
    * function that returns a string that is a list of columns?
 | 
			
		||||
    * custom function to read and return the table?
 | 
			
		||||
    * view that `uses the source definiton` to extrapolate a table?
 | 
			
		||||
    * a materialized table is built `based on the source definition` and any addtional regex?
 | 
			
		||||
        * add regex = alter table add column with historic updates?
 | 
			
		||||
        * no primary key?
 | 
			
		||||
        * every document must work out to one row
 | 
			
		||||
 | 
			
		||||
{
 | 
			
		||||
    "name":"sample_name",
 | 
			
		||||
    "source":"client_file",
 | 
			
		||||
    "loading_function":"csv"
 | 
			
		||||
    "constrain":[
 | 
			
		||||
        "{date}",
 | 
			
		||||
        "{level1,item3,field}"
 | 
			
		||||
    ],
 | 
			
		||||
    "table_schema":[
 | 
			
		||||
        {
 | 
			
		||||
            "top_level_key":"column"name",
 | 
			
		||||
            "type":"numeric"
 | 
			
		||||
        },
 | 
			
		||||
        {
 | 
			
		||||
            "top_level_key":"column"name",
 | 
			
		||||
            "type":"numeric"
 | 
			
		||||
        }
 | 
			
		||||
    ]
 | 
			
		||||
}
 | 
			
		||||
 | 
			
		||||
 | 
			
		||||
		Loading…
	
		Reference in New Issue
	
	Block a user