How to use the MP-Desktop Import Processor
Introduction
The import processor acts as a holding area for data that has failed to import correctly due to incorrect resolutions and/or validation. Data in the Import Processor is not accessible from the main Database Explorer or Datasets.
Open the import processor using the ‘Data Entry’ button on the toolbar, or the link under ‘MP-Desktop Today’.
Select an import that has failed from the drop-down list. There is a refresh list button to show files that have been imported since the Import Processor was opened.
Colour coding is used to show what needs to be resolved and why it has been retained in the Import Processor.
Items to be resolved
Work through the file and manually resolve any problems, reject the entire file, or attempt to accept the file again.
To manually resolve rows double-click the row.
In this example, the Variable name ‘Turbidity' is not resolved. Select the matching Variable, in this case ‘Turbidity (NTU)’ from the drop-down list, or add a new one using the 'Add' button. If resolving Sample Points, the list can be filtered by Sample Point Group to allow for easier location of the correct point.
Once you have resolved the point, you can choose how to apply it from the Auto Resolve tab. By default, your resolution will apply to the file that you are currently working on. However, you can set this to be the single record or all outstanding files (batches).
Once all the resolutions are made, press ‘Accept’ and the data will be imported for use in reporting and other output.
Duplicates
Duplicate data is related to the properties of the data point, not based on whether the record is the same value as another. Duplicate status is based upon data having the same Date/Time, Sample Point, Variable, Data Source, Sample Type, Lab Ref and Sample Ref, as another piece of data already in the database or duplicated in the same file.
Data in this situation can be made unique by assigning a different Sample Type, Lab Reference or Sample Reference to differentiate the of data.
Duplicate data is stored in the Import Processor and highlighted in grey. Users can then either reject the data or accept it, overwriting current information stored within the database leaving one reading.
A data point may be flagged as a duplicate if it existed in the database prior to the import of the specific file, or if the same reading existed twice or more in the same file – MonitorPro Desktop would import the first reading that it came to, making the rest duplicates.
How Duplicates are managed in the import process can also be controlled via the properties of the Auto Importer. For example, Duplicates can be set to be automatically discarded from the Import Processor.
Invalid Values
Invalid values highlighted in red are typically invalid dates. That is MonitorPro has not been able to distinguish the value as a true date. Different date formats can be recognised by MonitorPro including different international standards. If the date can be read and understood in Excel as a date then it will equally be able to be determined in MonitorPro. Also note the settings on the Auto Importer for the expected date format match those being imported (such as mm/dd/yyyy, or dd/mm/yyyy). It's likely these dates will need to be fixed manually and it can often be easier to correct or format the file before reporting if there is a number to fix.
Unusual Data
Unusual data checks can be set in System Options and related to further checks around dates. Any dates which are in the future will be held in the Import Processor as unusual. Equally any dates which are older than the prescribed limit (1 year by default) will be highlighted as unusual. For importing historical batched of data first change the System Options setting.
Physical Limits
Physical limit validation is completed based on the settings of each Variable. If data is above or below the set limits, data will be held in the Import Processor. As this is intended to be an absolute hard limit, data outside this range cannot be imported. If the Physical Limits are too restrictive they will need to be amended from the properties of the Variable and then refreshing the Import Processor will allow the data to be accepted.
File Recheck
If Aliases have been made or other object resolutions have been done outside of the Import Processor since the file was imported the Recheck button will apply them ready for import.
Discard File
Discarding a file if applicable if the number of resolutions is excessive and the simple solution is to reimport the file, or if the entire file was imported by error (perhaps duplicated).
Note
If some data was imported and some held for resolution, discarding the file ONLY discards those unresolved and committed records. Any data which was imported will still be in the database.
Accepting
Accepting the data will attempt to import the data which has subsequently been resolved by the user. Anything still unresolved will not be imported and will remain in the Import Processor until fixed or the file discarded.
Sample Reference Auto Reconciliation
The MonitorPro Desktop Auto-Importer is able to reconcile and resolve Sample References to Sample Points. This is useful where sample anonymity is required, and the laboratory or other external testing organisation is sent only a Sample Reference, without location (Sample Point) information. Sample References and their respective Sample Points are managed through the Monitoring Scheduler system. For further information see the section on Monitoring Schedules.