ccm:data_import
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revision | |||
ccm:data_import [2020/09/11 08:52] – [Filenames] genadmin | ccm:data_import [2020/09/11 08:59] (current) – genadmin | ||
---|---|---|---|
Line 1: | Line 1: | ||
Our Import tool is fairly flexible but does have some specific requirements when supplying data to us. Please read the information below in its entirety. | Our Import tool is fairly flexible but does have some specific requirements when supplying data to us. Please read the information below in its entirety. | ||
- | File format | + | File format |
- | All lines within a CSV must have the same number of delimiters, | + | All lines within a CSV must have the same number of delimiters, |
+ | |||
+ | The actual | ||
+ | |||
+ | Where a data element would contain the separator for CSV, a comma, it should be removed and not supplied. | ||
+ | |||
+ | Some characters are prohibited regardless of quoting or escaping and those are defined below under Prohibited Characters | ||
- | The actual names of the columns are for readability only but the column order must remain consistent. If you wish to change the order and/or add/ | ||
The column data is defined below. Where data is not mandatory it can be optionally supplied, but if supplied must be valid. | The column data is defined below. Where data is not mandatory it can be optionally supplied, but if supplied must be valid. | ||
Line 57: | Line 62: | ||
The Transfer Filename format should be identified and defined clearly so our import process knows what its looking to import. A good example would be vendorname_YYYYMMDD.csv. | The Transfer Filename format should be identified and defined clearly so our import process knows what its looking to import. A good example would be vendorname_YYYYMMDD.csv. | ||
+ | |||
+ | The FOLDER into which you upload the files is critical in determining the destination of the data on our systems so you must ensure data is loaded to the correct folder. | ||
If you will be supplying multiple varying files then an example would be vendorname_filetype_YYYYMMDD.csv | If you will be supplying multiple varying files then an example would be vendorname_filetype_YYYYMMDD.csv | ||
Line 66: | Line 73: | ||
When you have constructed a file in the format above and wish to begin the process of supply then the following should be observed. | When you have constructed a file in the format above and wish to begin the process of supply then the following should be observed. | ||
- | The Transfer Method should be identified and defined. We can accept files provided to SFTP and we will setup a specific | + | The Transfer Method should be identified and defined. We can accept files provided to Secure FTP and we will setup a specific |
The Transfer Schedule should be identified so we know how often to check for new data. | The Transfer Schedule should be identified so we know how often to check for new data. | ||
Line 74: | Line 81: | ||
===== Testing Phase ===== | ===== Testing Phase ===== | ||
- | A sample file should be supplied to the agreed | + | A sample file should be supplied to the agreed |
We will then code our import process to read this file and setup the mapping between columns and our database. We will validate each column as supplied and feedback. | We will then code our import process to read this file and setup the mapping between columns and our database. We will validate each column as supplied and feedback. | ||
- | We will produce a data return based on this test file (if required) and deliver this to the SFTP where it can be collected and validated by the data vendor. | + | Import Notifications and Fallout can be advised by FTP or email and this will be setup as required. |
+ | |||
+ | We will produce a data return based on this test file (if required) and deliver this to the Secure FTP where it can be collected and validated by the data vendor. | ||
The above process should be repeated as many times as necessary to ensure that both parties are confident that the process is correct. | The above process should be repeated as many times as necessary to ensure that both parties are confident that the process is correct. | ||
Line 84: | Line 93: | ||
===== Production Phase ===== | ===== Production Phase ===== | ||
- | Once testing is complete then we can begin live operations on a date mutually agreed upon. All test files/data should be purged from the SFTP and system before this date. | + | Once testing is complete then we can begin live operations on a date mutually agreed upon. All test files/data should be purged from the Secure FTP and system before this date. |
During the first week of live operation, log files should be examined to ensure no post testing issues have arisen. | During the first week of live operation, log files should be examined to ensure no post testing issues have arisen. |
/home/gen.uk/domains/wiki.gen.uk/public_html/data/attic/ccm/data_import.1599814366.txt.gz · Last modified: 2020/09/11 08:52 by genadmin