Skip to content
English
Contact us
Go to ehsdata.com
  • There are no suggestions because the search field is empty.

MP-Web Auto Importer

How to use the MP-Web Auto Importer

Overview

The Auto Importer is a highly configurable import process consisting of folder watchers monitoring incoming files to process. Importers can be configured to support several file types and layouts. This is the core mechanism for importing routine lab data, logger files, or other consistent and known file types. 

As a part of the Auto Import setup, a path to a folder and the expected import file layout is entered. The MonitorPro Service will be installed which runs the auto imports (and the associated tasks such as the Calculator, Scheduled Task and Monitoring Scheduler).

Creating an Auto Importer

General Tab

Name

Enter a meaningful name for the importer

Location

The folder path where the importer will look for new files when run.

Note

For hosted clients, please contact support@ehsdata.com to assist with setting up a new Importer path in your system. 

Import Type

The type of Importer, List, Grid, etc.


Date

The date format can be set for multi-country support, most commonly to allow for US-based mm/dd/yyyy formats.

Import Culture

Used to set how values will be treated concerning the decimal separator. 

Column Separator

Specifies the column delimiters for the incoming file (default is a comma).

Incoming data files are often .csv format. csv often is understood to be 'comma separated' meaning that in the file new columns are defined by a comma as a separator. This can be seen if the file is opened in Notepad for example.

However, csv can mean 'character separated' with a different character used to separate columns, for example, semi-colon or tab.  This is common, especially in countries that use a decimal comma instead of a decimal point.

String Delimiter

Defines what should be treated as a text string within the file.  'Wrapping' the data within a file is a common approach as it stops characters, such as a comma or semi column being read as a column separator. Without a string, delimiter files could be misread. This is only very rarely changed from double-quotes.

Record Comment

An entry here will be added to every data reading's comment field

Defaults

The Defaults tab allows you to specify default values for key information that is not present in the file and provides further details about the file format.

Example

Files from a weather station will not normally contain a column for 'Sample Point'.  The Sample Point can therefore be pre-determined (for this import only) and thereafter any files imported from this folder (that are missing a sample point column) will default to ‘Weather Station’ in MonitorPro. 

Behaviour

The behaviour tab allows users to specify specific rules for each importer. When creating a new importer, a default selection of options represents the most common scenarios. 
 
Auto-create new objects in the database where no match is found

If the importer cannot match a name within the file to a name within the database, it checks this option to see what to do next. If checked, the importer will create an object that cannot be resolved automatically. 

Warning

This setting is almost only ever used to create new databases, although there are some exceptions where it can be used carefully for ongoing data input. The data coming in via this importer needs to be well understood and consistent otherwise this setting can quickly ruin a well-ordered database.

This setting is almost only used to create new databases, although there are some exceptions where it can be used carefully for ongoing data input. The data coming in via this importer needs to be well understood and consistent; otherwise, this setting can quickly ruin a well-ordered database.

Automatically commit all valid records to database

Typically, users will want valid records in a file to be imported, and only those that are not and need resolution to be held in the Import Processor. If there is an error when importing data, the importer checks this flag to see if it should do so or alternatively hold back the entire contents of the file. 

The remaining sub-options of this are

Retain the complete file in the Import Processor. This setting will retain the entire file in the Import Processor if any records need resolution. This is typically only used if the data is very consistent, and any problem with the file might indicate a wider issue for which holding the entire file back is prudent. 

If any Columns can't be resolved, place the complete file in the Import Processor. This is Similar to the above, but only headers that are not resolved will retain the entire file, not just individual data resolutions.

Clear any uncommitted records for these files from the Import Processor once done. This setting clears any remaining records from the Import Processor that do not automatically resolve. It can also be used to discard unwanted data from a file that isn't required for import. 

Caution

This setting will discard any unresolved data, which is useful to not bloat the Import Processor with unwanted data when the imports are consistent. Where the data is not consistent, caution should be used as this setting will discard anything not automatically resolved.

Discard any duplicate records for these files from the Import Processor once done. As above, this setting will clear any valid but duplicate records from the Import Processor.

Tip

This setting is convenient when importing logger files. The configuration can be set up so that for example 3 hrs of data is imported for every 1 hr. If the system is working as expected 2 hrs of data will be consistently discarded as duplicates, and 1 hr imported. If there is a problem with the delivery of the data then the overlap of data can fill the gap without the need to manually make up for the interruption.  

Do not automatically overwrite duplicate records to the database. Data with the same Location, Variable, Data Source, Date/Time, Sample Type, Sample Ref and Lab Ref is considered a unique record. If imported data is seen as a duplicate, this setting controls how it is managed. Typically, users would want duplicate records to be held in the Import Processor for review, as this may indicate an issue, or simply that the data has been imported before. 

Do not automatically overwrite duplicate records to the database. Data with the same Location, Variable, Data Source, Date/Time, Sample Type, is considered a partially unique record. If imported data is seen as a partial duplicate, this setting controls how it is managed. Typically, users would want duplicate records to be held in the Import Processor for review, as this may indicate an issue, or simply that the data has been imported before with a different reference. 

Automatically approve any records committed via this auto import. MonitorPro has a mechanism for setting data to a state of 'unapproved' on import. This means data will not be automatically present in standard Datasets if in a state of unapproved. Data can be manually approved or automatically approved based on set rules and workflow, to then show in standard Datasets and onward reporting. Unless this workflow is in place, this should be ticked to approve data on import. 

Auto-detect variable & units over multiple header rows. Tick this box if the data file has variable names split over more than one row and is on by default.

Note

When ticked the importer will identify the first variable name row and append the values found in the next line to each name. E.g. the variable ‘Temperature’ may have the related unit (°C) recorded on the next line. The importer would return Temperature (°C) as the variable name.

Apply sample reconciliation rules where sample information is present. Data can be resolved on import without Site/Sample Point information, provided a Sample Reference, previously set up in Monitorpro, is in the file.  


Assume blank entries in appropriate columns are the same as the previous row. This setting will copy down records from above in a file to blank rows below. This is only typical for a small number of logger files where data is provided in a single column, but is assumed for all the rows of the file. 

Embedded Files

Non-data files can also be imported into MonitorPro via the Auto Importer. This can be done without linking, and the file(s) can be manually associated with objects or data in the database later. To make the process more automated, specific rules can help link these non-data files (typically images or PDF documents) to objects in the database.

If you plan to use a variety of these methods, it would be advisable to create separate import folders named suitably to guide users as to the rules applied to each.

Filter

This list will denote the file types the rules should apply to.

File Split

This list defines how MonitorPro should split the filename string to extract the text that matches the applicable object.

Minimum character match

This helps reduce false positive matches by limiting the number of characters used to match. 

Note

In the following scenarios, the file name will be checked based on separating words or 'strings' as distinct matches. File names that include a space, an underscore, or an open or closing bracket (parentheses) will be used to distinguish the strings and then match to the Objects (Sample Point, Sample Ref, Lab Ref).  

For example, 'ALS_01012022_REF12345' will generate three strings to be checked for a match. In this example, it's likely that 'REF12345' is the part required, and this will be checked against all the Sample Ref or Lab Refs (depending on the setting(s)). In this example, the other strings will likely not match; however, all the strings will be checked for the chosen objects, hence why care needs to be given to avoid false positive results.

Options

Link the file to a batch of data brought in with the same name (but a different extension)

A non-data file can be imported with a name matching the imported data file. In this scenario, the file will be linked with the batch (file) of data. 


Sample Point

This option is used to match non-data files to Sample Points by naming the file the same as the Long or Short name of a Sample Point in MonitorPro

Sample Reference

This option will use the Sample Reference in the non-data file name to match with the data, including the corresponding Sample Reference.

Lab Reference

This option will use the Lab Reference in the non-data file name to match with the data, including the corresponding Lab Reference. 

Import Macro

The Import Macro tab can contain code to pre-process files which do not fit a standard MonitorPro format and convert them to a standard format to import. 

This can be the case with certain loggers, which, although flexible, the importer cannot resolve natively even with the settings mentioned previously in this section. EHS Data has a library of these macros for a number of common logger types. Please contact your Account Manager or support@ehsdata.com for options.