How do I make a Data Integration?

Data Integrations are automations from data sources into the platform.

In order to establish a Data Integration, the user must first make a Data Connection and then a Data Source.  After the Source is established, the user can add the Data integration.

Data Integrations are found under Sources on the Navigation Panel.

Video Example

Select Data Connections from the Data Connections screen.

Select the green plus button.

In order to establish the integration, the user has to select the data source and if desired, cleansing script. A blue check will highlight the selected source.

After making the selection desired, select .

Enter a name and select save.

In order to see the data, use the drop down selection under "Run Data Integration" to Preview the data.

PetroVisor will display a "Running" while it runs the integration.

The preview will display information about the integration.

The summary information displays:

  • Values Processed:  Number of lines of data imported
  • Duration:  Length of time to import data
  • Mappings:  Searches the columns for data
  • Processed: Number of lines of data imported
  • Cleaned:  Number of lines of data cleaned (notice in the example, no data was cleaned)
  • Not Recognized:  Number of lines that could not be read
  • Empty: Number of lines with no data
  • Existing:  Number of lines already available
  • Multiples:  Number of lines with the same information, same entity-signal-timestep combination (aggregation type is the same as provided in the mapping)
  • Out of Scope:  Number of lines skipped as the data was outside of the scope set
  • Imported:  Number of lines of data imported

After clicking the "x" in the top of the preview screen, the information will be saved in the home screen for the integration.  To view the data, simply select the run from the list.

Notice in the list, the user can see the Data Sources and Cleansing scripts in the top left.  The user can also see the options in the top right.

Under Options, selections can be made for:

  • Skip Existing Data - When this item is checked, none of the existing data will be updated and new data will be added to the end of your current data.
  • Delete Data In Scope - When this item is check, it removes all existing data within the specified scope for all mapped signals before importing new data.
  • Scope (set up in the Library)
  • Start and End date for pulling in integration
  • Time step for how often data should be pulled
  • Start and End depth for data 

 

Those options are set on the Cloud Admin site at the Workspace Summary Level.