Filename Variable - Does not get saved while executing Map

Support for DataSlave

Moderators: Tom, ian

Filename Variable - Does not get saved while executing Map

Postby mdesi » Mon Mar 14, 2011 12:24 am

I am importing a number of csv files with variable number of records in each file using a data map. To keep an audit trail the 'filename' of the csv file is written with each record read to a database.

When I execute the data map, in batch mode, for some record the filename is not written successfully - the field is blank. If I change the batch size and run the datamap the count of the records with the blank filename field is different. If the batch size is set to 1 then the filename is not written at all.

I presume the filename variable gets destroyed when a new batch is run while there may be still records to be processed from an open file.

Is there an alternative method to write the filename to the output.

Thanks,

Malik
mdesi
 
Posts: 3
Joined: Thu Feb 17, 2011 10:15 am

Re: Filename Variable - Does not get saved while executing M

Postby ian » Thu Mar 17, 2011 11:38 am

Can I ask what version of DataSlave you are using?

Also can you tell me how many rows there are in the data set?
ian
 
Posts: 364
Joined: Sat Dec 18, 2004 8:13 am
Location: UK

Re: Filename Variable - Does not get saved while executing M

Postby mdesi » Thu Mar 17, 2011 3:18 pm

The Map was run using versions 2.2.2 and now 2.2.4 with similar results.

The files that are processed may be empty or contain up to about 1,400 rows. 1,572 files with approximately 500,000 records were processed using batch sizes of 5,000 and 20,000. In the above case all files processed had record count of greater than 2.
mdesi
 
Posts: 3
Joined: Thu Feb 17, 2011 10:15 am

Re: Filename Variable - Does not get saved while executing M

Postby mdesi » Thu Mar 31, 2011 2:48 pm

Is there an update to the post?
mdesi
 
Posts: 3
Joined: Thu Feb 17, 2011 10:15 am

Re: Filename Variable - Does not get saved while executing M

Postby ian » Fri Nov 29, 2013 9:06 am

This issue is fixed in the current release, 2.2.6. Please remember that if you look at the data in the grid after the map has executed only the last batch will be shown. To test that the correct filename is written to the data you will need to write the data out to a file and check this.

Apologies for the very late reply. For some reason the post only just as displayed.
ian
 
Posts: 364
Joined: Sat Dec 18, 2004 8:13 am
Location: UK


Return to Support

Who is online

Users browsing this forum: No registered users and 2 guests

cron