How do I connect my data sources?

For self-hosted, cloud-based, or third party data sources, Datagration engineers require connection details as well as credentials to connect to the source and query or pull data.

For self-hosted data sources, Datagration can provide IPs to whitelist or can help setup a site-to-site VPN to securely connect to the appropriate network where the data source is hosted.  For cloud-based data sources, the server or web URL and credentials are typically all it takes.

Database connections will require the database type (e.g. MS SQL, PostgreSQL, etc), server address, database name, a port number (if necessary), username, and password.  A brief overview or documentation of the schema is also helpful.

API connections will require the API's root endpoint as well as credentials.  These credentials are dependent on the API and may also include a client ID and secret along with the username and password.  Documentation of the available endpoints is also most helpful.

If creating credentials for the platform, we suggest using a name starting with the client company name / PetroVisor / EcoVisor / Datagration, and ending with 'readyonly' since these credentials are strictly used to query or pull data.  Examples include {client-name}-readonly, petrovisor-readonly, dg-readonly, etc.

The platform can also read in flat files.  For these type of data sources, the platform can use an email flow or (preferably) connect via SFTP.