0001804: Merge tFileInputCSV and tFileInputDelimited components

(0004210)
plegall (reporter)
2007-09-27 16:06

I try to reproduce the problem but I can't (with TOS 2.2.0RC1), maybe I don't understand the problem. Can you tell me what you expect to find in the "Field Separator" field of tFileInputDelimited_1 component?

(0004235)
maxsom (reporter)
2007-09-28 07:14

I created a test job and tested in 2.2.0M1, then imported to 2.2.0M2, and 2.2.0RC1 and the issue exists in all three versions.

The defined tFileInputDelimited should have the escape char and text enclosed fields being pulled from the repository but it does not. I have had to change jobs that previously used tFileInputDelimited components to use tFileInputCSV components instead.

From a repository perspective you really only add metadata for a delim file type (confusing as you have two separate job components) and there is no real difference between CSV / Delim. From a job component perspective there are both types the delim type only has row and field separator whereas the CSV also has text enclosure and escape character.

Whatever the case in previous versions < 2.1.1 CSV and Delim worked the same and they do not now. It's not a big deal but these issues should be handled better for users moving from version to version like someone upgraded from 2.0 to 2.1.4 as it might break their jobs...

I guess ultimately I don't see a reason for the two types of delimited components (delim/csv). If we are going to keep both component types and the delim components continue to be a subset of CSV then we should notice when the repository contains more defined elements than we can handle and notify the user that the component and repository are mismatched.

See additional screenshots and uploaded job for additional info.

(0004256)
plegall (reporter)
2007-09-28 15:57

Reminder sent to: cantoine, ccarbone, fbonan, nrousseau

Gentlemen, what's your opinion abou maxsom point of view? should we have two wizards: one to map tFID properties and another to map tFIC properties?

(0004273)
nrousseau (administrator)
2007-09-29 05:16

I agree that have 2 different components can be confusing as there is not so much difference if we just see the parameters.

I think it should be possible and interesting to have only one component to do both. (but now it will be only in the version 2.3 if we do of course...)
I don't really like the idea about have 2 different repository... it will be even more confusing.

(0004293)
ccarbone (administrator)
2007-09-30 16:45
edited on: 2007-09-30 16:47

I'm not agree to have 2 distinct wizards to handle CSV and delimited.
I'm agree with nrousseau : only one wizard, only one component!
This component should display an option to choice between delimited or CSV (in the 2nd case, escapechar and textenclosure fields appear).

Be careful, only one component but we keep our 2 different algorithms (CSV and Delimited) to have the best performance code when delimited algorithm is sufficient.

(0004294)
ccarbone (administrator)
2007-09-30 16:46

Reminder sent to: mhirt

(0004298)
plegall (reporter)
2007-10-01 10:09

Ok to have a single component, how should we name it? (fbonan, your opinion is strongly welcomed at this point of the discussion ;-)

(0004301)
plegall (reporter)
2007-10-01 11:29

I change the entry severity from "minor" to "feature"

(0004491)
plegall (reporter)
2007-10-09 16:50

After discussion with fbonan, the name of the merged component is tFileInputDelimited.

(0004517)
plegall (reporter)
2007-10-10 17:37
edited on: 2007-10-10 17:48

Added on trunk in [Subversion, tos] r6249

Issue History
Date Modified	Username	Field	Change
2007-08-15 10:12	maxsom	New Issue
2007-08-15 10:13	maxsom	File Added: 2.2.0M2_tFileInputDelimited_Missing_Props.png
2007-08-18 17:28	ccarbone	generated language	=> perl
2007-08-18 17:28	ccarbone	Assigned To	=> plegall
2007-08-18 17:28	ccarbone	Status	new => feedback from Talend
2007-09-27 16:06	plegall	Note Added: 0004210
2007-09-28 06:45	maxsom	File Added: 2.2.0M2_tFileInputCSV_1_Correct_Props.png
2007-09-28 07:14	maxsom	Note Added: 0004235
2007-09-28 07:14	maxsom	File Added: 2.2.0M2_tFileInputDelimited_Repository_Settings.png
2007-09-28 07:15	maxsom	File Added: Job_CNV_DEBUG.png
2007-09-28 07:15	maxsom	File Added: Job_CNV_DEBUG2.png
2007-09-28 07:18	maxsom	File Added: Job_CNV_Debug.zip
2007-09-28 15:55	plegall	Assigned To	plegall => nrousseau
2007-09-28 15:57	plegall	Issue Monitored: cantoine
2007-09-28 15:57	plegall	Issue Monitored: ccarbone
2007-09-28 15:57	plegall	Issue Monitored: fbonan
2007-09-28 15:57	plegall	Note Added: 0004256
2007-09-29 05:16	nrousseau	Note Added: 0004273
2007-09-30 16:45	ccarbone	Note Added: 0004293
2007-09-30 16:46	ccarbone	Issue Monitored: mhirt
2007-09-30 16:46	ccarbone	Note Added: 0004294
2007-09-30 16:47	ccarbone	Note Edited: 0004293
2007-09-30 23:23	mhirt	Assigned To	nrousseau => plegall
2007-09-30 23:23	mhirt	Category	??? unknown ??? => Perl components
2007-09-30 23:23	mhirt	Summary	Component tFileInputDelimited Missing Repository Properties => Merge tFileInputCSV and tFileInputDelimited components
2007-09-30 23:23	mhirt	Issue cloned	0002139
2007-09-30 23:23	mhirt	Relationship added	related to 0002139
2007-10-01 10:09	plegall	Note Added: 0004298
2007-10-01 11:29	plegall	Note Added: 0004301
2007-10-01 11:29	plegall	Severity	minor => feature
2007-10-09 16:50	plegall	Note Added: 0004491
2007-10-09 16:50	plegall	Status	feedback from Talend => assigned
2007-10-10 17:36	plegall	Status	assigned => resolved/needs doc
2007-10-10 17:36	plegall	Resolution	open => fixed
2007-10-10 17:36	plegall	Fixed in Version	=> 2.3.0M1
2007-10-10 17:37	plegall	Note Added: 0004517
2007-10-10 17:48	plegall	Note Edited: 0004517
2007-11-22 09:23	plegall	Status	resolved/needs doc => closed

Notes
(0004210) plegall (reporter) 2007-09-27 16:06	I try to reproduce the problem but I can't (with TOS 2.2.0RC1), maybe I don't understand the problem. Can you tell me what you expect to find in the "Field Separator" field of tFileInputDelimited_1 component?

(0004235) maxsom (reporter) 2007-09-28 07:14	I created a test job and tested in 2.2.0M1, then imported to 2.2.0M2, and 2.2.0RC1 and the issue exists in all three versions. The defined tFileInputDelimited should have the escape char and text enclosed fields being pulled from the repository but it does not. I have had to change jobs that previously used tFileInputDelimited components to use tFileInputCSV components instead. From a repository perspective you really only add metadata for a delim file type (confusing as you have two separate job components) and there is no real difference between CSV / Delim. From a job component perspective there are both types the delim type only has row and field separator whereas the CSV also has text enclosure and escape character. Whatever the case in previous versions < 2.1.1 CSV and Delim worked the same and they do not now. It's not a big deal but these issues should be handled better for users moving from version to version like someone upgraded from 2.0 to 2.1.4 as it might break their jobs... I guess ultimately I don't see a reason for the two types of delimited components (delim/csv). If we are going to keep both component types and the delim components continue to be a subset of CSV then we should notice when the repository contains more defined elements than we can handle and notify the user that the component and repository are mismatched. See additional screenshots and uploaded job for additional info.

(0004256) plegall (reporter) 2007-09-28 15:57	Reminder sent to: cantoine, ccarbone, fbonan, nrousseau Gentlemen, what's your opinion abou maxsom point of view? should we have two wizards: one to map tFID properties and another to map tFIC properties?

(0004273) nrousseau (administrator) 2007-09-29 05:16	I agree that have 2 different components can be confusing as there is not so much difference if we just see the parameters. I think it should be possible and interesting to have only one component to do both. (but now it will be only in the version 2.3 if we do of course...) I don't really like the idea about have 2 different repository... it will be even more confusing.

(0004293) ccarbone (administrator) 2007-09-30 16:45 edited on: 2007-09-30 16:47	I'm not agree to have 2 distinct wizards to handle CSV and delimited. I'm agree with nrousseau : only one wizard, only one component! This component should display an option to choice between delimited or CSV (in the 2nd case, escapechar and textenclosure fields appear). Be careful, only one component but we keep our 2 different algorithms (CSV and Delimited) to have the best performance code when delimited algorithm is sufficient.

(0004294) ccarbone (administrator) 2007-09-30 16:46	Reminder sent to: mhirt

(0004298) plegall (reporter) 2007-10-01 10:09	Ok to have a single component, how should we name it? (fbonan, your opinion is strongly welcomed at this point of the discussion ;-)

(0004301) plegall (reporter) 2007-10-01 11:29	I change the entry severity from "minor" to "feature"

(0004491) plegall (reporter) 2007-10-09 16:50	After discussion with fbonan, the name of the merged component is tFileInputDelimited.

(0004517) plegall (reporter) 2007-10-10 17:37 edited on: 2007-10-10 17:48	Added on trunk in [Subversion, tos] r6249