Text File Schema mapping

A list of requested enhancements to DataSlave

Moderators: Tom, ian

Text File Schema mapping

Postby ceejayemm » Fri May 11, 2007 9:20 am

I have just had to map a very large text file (10M records but only 10 columns) to a Read Text File object. The data file has fixed length records and fixed length fields within the record HOWEVER there are no spaces between the records (in this case the data file was a Cognos Powerhouse Portable Subfile). To map just 10 columns has taken me the best part of 3 hours as the wizard did not find the columns and I have had to map the schema manually. After adding each new field / column row there is a significant pause, after entering the field name another long pause and then after entering the field size yet another long pause.

Would it be possible to enter all the field mappings and then for the mapping process to do its thing in terms of redrawing the column headings, identifying the fields etc in one go when I click an Ok button (or something similar) or at the very least allow all the details about each field to be entered before the data map is redrawn ? This would save significant time when trying to map files of this type.
ceejayemm
 
Posts: 22
Joined: Mon Apr 23, 2007 1:29 pm
Location: London, UK

Delays when working with the Read Text Object

Postby ian » Mon May 14, 2007 6:42 am

You raise two questions and I would like to address each in turn.

Fixed length data will not normally have any spaces or other delimiters between the data. It is then up the user to designate the length of each field. This takes a little while, but is made worse by the problem discussed below and the large data file.

There is a know problem with DataSlave when working with large files, which instigates and unnecessary scan to the end of the file. This is now fixed in release 2.1. To work around this problem can I suggest that you create a small file from say the first 50 rows of your data. Use this to develop your map and then when complete rename the file to then process the big file.
Last edited by ian on Wed Dec 12, 2007 1:39 pm, edited 1 time in total.
ian
 
Posts: 364
Joined: Sat Dec 18, 2004 8:13 am
Location: UK

Postby ceejayemm » Tue May 15, 2007 7:42 am

Thanks Ian, can you say when the next version will be available as information in this forum (posted June 2005) suggests the next version will be v2.0 which I already have. Is it also possible to say a bit more about what will be in the next version ? Will there be any input / output objects to support the use of Oracle databases ?

Chris
ceejayemm
 
Posts: 22
Joined: Mon Apr 23, 2007 1:29 pm
Location: London, UK

Version 2.1

Postby ian » Tue May 15, 2007 11:49 am

The current release of DataSlave, 2.1, now includes the following new features:

    Support for MySQL
    Support for Oracle
    Support for DataFlex
    A new Dot NET Script Objects at three levels (Map, Validation and Transform)
    The auto-size of the data grid is now optional improving performance with very large data sets.

There is an issue with the Text Read object scanning to the end of the file when displaying the properties wizard. This is now fixed
Last edited by ian on Wed Dec 12, 2007 1:40 pm, edited 1 time in total.
ian
 
Posts: 364
Joined: Sat Dec 18, 2004 8:13 am
Location: UK

Ian

Postby blackice » Thu May 17, 2007 11:18 am

Regarding your comment that fixed length data will not normally have any spaces, it actually does in many instances, most notably data coming from mainframes. The spaces are inserted to ensure the data field does come to a specific length. I have this in alot of my custodian feeds.

Kind Regards,
Matt R. Cole
blackice
 
Posts: 25
Joined: Fri Nov 03, 2006 7:06 pm

Spaces

Postby ian » Thu May 17, 2007 11:27 am

Sorry I was not clear. Yes the data may have many spaces to make fields fixed length. What I meant that it is not safe to regard the space as a field delimiter as it may not be present or may be present in numbers.
ian
 
Posts: 364
Joined: Sat Dec 18, 2004 8:13 am
Location: UK


Return to Wishlist

Who is online

Users browsing this forum: No registered users and 2 guests

cron