Use Case “Receive Data from external or internal source”

Example of HTTP connector setup

First step for working with data sources is setting up an Integration with the required data source. For than, you need to select one of the available connection types and then setup the necessary parameters as described in specific connection setup guides here: HTTP, LDAP, Oracle, MS SQL, PostgreSQL.

After than, you need to load API schema as shown on the left in HTTP setup example.

Setting up Cron for your Components

Once connector is setup, you need to navigate to your Component’s General settings. We will use Cron option to setup timed launch of our dataflow so it is executed every 5 minutes.

Setup “Get values from connector” step

Now we’re going to setup our Input dataflow. Go to your data flow, add new stage and add a step of “Get values from connector” activity type. Set it up selecting your connector type (HTTP) , your query path and your query type as shown on the left. You can then press the “i” icon to see the data your dataflow step will receive from the connector.

{
  "type": "object",
  "properties": {
    "id": {
      "type": "integer"
    },
    "birthDate": {
      "type": "string",
      "format": "date-time"
    },
    "firstName": {
      "type": "string"
    },
    "middleName": {
      "type": "string"
    },
    "secondName": {
      "type": "string"
    },
    "departmentId": {
      "type": "integer"
    },
    "noMiddleName": {
      "type": "boolean"
    }
  }
}

Definition settings

Now that you have the data that you’re going to receive from the connector via “Get values from connector” step, you need to make sure your component have all the relevant properties in which you’re going to store data received from connector. For example, we will add the following properties: id (type: integer), firstName (type: string), SecondName (type: string).

Next we will add a new Data flow step of the “Store entry over bus” type, and then we will set it up using Fields mapping as shown below.

Setting up “Store entry over bus” step

Once this is done, you need to save and publish your component which is now ready to use.

Use Case “Filtering data using specified criteria”

Select “Filter source” step

Extending the previous scenario, we will now add filtering of the results we’ve received from the external connector, using Dataflow activity of the ”Filter source” type.

Set up “Filter source” step and the dataflow

Once you’ve added “Filter source” step as shown above, set it up using Pencil icon and then filling in the fields as described here. In Src field select a property for which you want to filter. In Operator field set one of the available filtering parameters such as Equal, Not equal, Great, Less. Finally, in Filter value field fill in the value you’d like to use to filter the source field with. Also, add “Store entry over bus” step to make sure you save your updated data once it is filtered.

Use Case “Joining data from different sources”

Setting up a new “Get values from connector” step

To further extend previous scenario, we will demonstrate use of the “Join models”, For that we will add a new “Get values from connector” step, and set it up as shown on the left. It is important that Query path will be different from the once we’ve setup earlier.

Once the step has been set, press the “i” icon to see the new data schema.

{
  "type": "object",
  "properties": {
    "id": {
      "type": "integer"
    },
    "name": {
      "type": "string"
    },
    "managerId": {
      "type": "integer"
    }
  }
}

Now you can go to Component’s data model **** if you need to add missing properties using Definition menu.

Selecting “Join models” step

Add a new “Join models” step, and set it up as shown below. Left step should contain our first data source, Right step should contain our second data source. Left Key and Right key - should contain primary keys of data sources, and Map setting should be used to join data from the two sources.