Optional Input and Output connectors to Import and Export

A list of requested enhancements to DataSlave

Moderators: Tom, ian

Optional Input and Output connectors to Import and Export

Postby blackice » Sat Nov 04, 2006 11:51 am


In my world of ETL, the acquisition and export of data doesn't mean that's where the process stops. I would assume many others may fall into this situation as well. For example, DataSlave has standard Import and Export objects, but that assumes the process starts and ends there. For example, I read in a .csv file, validate, transform, then export to a data warehouse.

However, before I can read in the .csv file, I have to sometimes acquire or create it. For example, I have FTP, HTTP and OFX import and export objects for DataSlave created using the SDK. So, one map will actually start by going to an FTP site and retrieving a file. Another one uses HTTP/HTTPS to obtain the datafile from a financial institution.

For export, after validation and transformation, I store the data in rawfeed SQL tables, and then there are several stored procedures, DTS packages, etc. that handle additional processing.

However, even though I can do this in my objects, I drop those objects onto the map editor and that's it. In terms of function priority, it has no connection to the other main objects, so when the map is executed, does it execute first?

It would be incredibly useful if an import object had an \"optional\" input connector, and an export object had an optional \"output\" connector, such that the level of what happens within a map could be expanded to handle such items. My FTP object now has the data file, and could trigger the import object to begin once completed. As for export, the optional output connector could trigger downstream processes.

Perhaps this is implemented somehow within the SDK and I haven't found it, but I don't think so. But it's incredibly useful to contain all of this within a single map, as that map is a true complete ETL definition. I'm pretty sure I can figure out a way to make it happen by making several custom import and export objects that adhere to that and then just make them available to others who might want or need to do the same thing, but that capability would greatly expand DataSlave, or so I would think.

Just some thoughts.

Kind Regards,
Matt R. Cole
Posts: 25
Joined: Fri Nov 03, 2006 7:06 pm

Return to Wishlist

Who is online

Users browsing this forum: No registered users and 1 guest