Planet Talend

Datamining type

2008-11-04T20:42:48+00:00

In Talend Open Profiler, when you create a column analysis, you can see a combo box near each column in the editor which represents the data mining type of the column. What is it? And what is it useful for?

The available data mining types are

nominal
interval
unstructured text
other

Not all indicators (or metrics) can be computed on all kind of data. These data mining types helps Talend Open Profiler to choose the appropriate metrics for the column.

Nominal (also called “categorical” sometimes) means that the data can serve as label. For example, the type of a column called “WEATHER” with values: “sun”, “cloud”, “rain” would be nominal. In Talend Open Profiler, textual data are set to nominal data mining type.

But it happens that data such as “52200″, “75014″ are nominal data too although they are represented by numbers. In fact, a column called “POSTAL_CODE” could have these values. It is clear for the user that these data are of nominal type because they identify a postal code in France. Computing mathematical quantities such as the average on these data is a non sense. In that case, the user should set the data mining type of this column to “nominal”, because there is currently no way to automatically guess the correct type in Talend Open Profiler in such a case.
The same is true for primary or foreign key data. Keys are most of the time numerical data, but their data mining type is “nominal”.

“Interval” data mining type is used for numerical data and time data. Difference between two values, averages can be computed on this kind of data. In databases, sometimes numerical quantities are stored in textual fields. With Talend Open Profiler, it’s possible to declare a textual column (e.g. a column of type VARCHAR) as an interval. In that case, the data should be treated as numerical data and summary statistics should be available. Currently, it’s not yet implemented because there is not yet an interface which allows the user to specify the format of the data. But this feature is planned for a future release.

The other two data mining types are not usual data mining types. In data mining we find sometimes the types “ordinal” and “ratio”.

The reason is that for the indicators currently available in Talend Open Profiler, these two types are not needed. Instead we added two other types to handle textual data. For example, a column “COMMENT” which contains text is not a nominal data, but still we could be interested in seeing the duplicate values of this column. Or we could implement metrics specific to text mining (but this is not for the current release…).

And finally, we also have the type “other” which design a data which Talend Open Profiler does not know how to handle yet.

How to compute a median in SQL

2008-10-28T21:46:47+00:00

In Talend Open Profiler, we generate SQL queries to get statistical informations. Among the currently available indicators, the median is one of the most difficult to compute. Nevertheless this indicator is worth computing because it is more stable than the mean indicator (average). By stable, I mean that it is less influenced by extremal values. This is not the case with the average which can vary a lot when extremal values exist.

I found several ways to compute the median depending on the database type. The most simple is for example with Oracle 10g which provides a MEDIAN function, so that your query writes
SELECT MEDIAN(salary) FROM employee

But for other databases, things begin to be more tricky. Let’s take MySQL first. One way to compute the median is the following:
SELECT AVG(salary) FROM ( SELECT salary FROM employee WHERE salary IS NOT NULL ORDER by salary ASC LIMIT p, n) T
where p=1 and n=N/2-1 when the number of non null rows N is even, or p=2 and n=(N-1)/2 when N is odd.

For Postgresql, the query is similar to the MySQL query and uses LIMIT too.
SELECT AVG(salary) FROM ( SELECT salary FROM employee WHERE salary IS NOT NULL ORDER by salary ASC LIMIT n OFFSET p) T
This query can also be used on MySQL but not on old versions of MySQL (before 5.0).
For Oracle 9i, the MEDIAN function does not exists and we must use the PERCENTILE_CONT function:
SELECT PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY salary) FROM employee
For DB2, the following query is used in Talend Open Profiler:
SELECT AVG(salary) FROM ( SELECT salary, COUNT(*) OVER( ) total, CAST(COUNT(*) OVER( ) AS DECIMAL)/2 mid, CEIL(CAST(COUNT(*) OVER( ) AS DECIMAL)/2) next, ROW_NUMBER() OVER ( ORDER BY salary) rn FROM employee ) x WHERE ( MOD(total,2) = 0 AND rn IN ( mid, mid+1 ) ) OR ( MOD(total,2) = 1 AND rn = next )
For Microsoft SQL Server, we used the TOP clause as follows
SELECT AVG(CAST(salary AS NUMERIC)) FROM ( SELECT TOP n salary FROM ( SELECT TOP m salary FROM employee WHERE salary IS NOT NULL ORDER BY salary ASC ) AS FOO ORDER BY salary DESC ) AS BAR
where n is given as in the MySQL case and m=n+p (p being given above for the MySQL case).

Up to now, the only way I found for computing the median on Sybase ASE is the following:
SELECT AVG(CAST (salary AS NUMERIC)) FROM ( SELECT DISTINCT salary FROM ( SELECT salary FROM employee UNION ALL SELECT salary FROM employee ) STT WHERE (SELECT COUNT(salary) FROM employee) <= (SELECT COUNT(salary) FROM ( SELECT salary FROM employee UNION ALL SELECT salary FROM employee ) AS SOU WHERE SOU.salary <= STT.salary) AND (SELECT COUNT(salary) FROM employee) <= (SELECT COUNT(salary) FROM ( SELECT salary FROM employee UNION ALL SELECT salary FROM employee ) AS SUR WHERE SUR.salary >= STT.salary) ) T
This query makes heavy use of correlated subqueries and I hope to find a more efficient way to compute a median on this database.

Median can be computed by other approaches. Temporary tables could be used or cursors. But Talend Open Profiler must only use SELECT statements because a data profiler could not have the permissions to create a table on a database and the use of cursors is too complex for this tool.

Talend Free Webinars, Case Studies and White Papers

2008-10-22T15:07:03+00:00

New sections have been added on our website that you will probably find of interest:

- the new On Demand Webinars section, where replays of selected past Webinars are now available. The Free Talend Webinar are recorded and you can see it when you wan’t ! This section is a complement of the calendar of the real time webinar.

- a reformatted Talend Customers section, with links to case studies (most of them are available in several languages) and a rotation of customer and partners logos.

- the new white paper library, featuring WPs and analyst reports.

All the documents are provide in French and in English.

talend

Source: ocarbone.free.fr

When memory matters, multiple keys hash

2008-10-16T22:41:00+00:00

One more time I need to load a huge number of data in memory with a Perl hash. See previous post about this in When memory matters. This time the hash key is made of several values. The value corresponding to each key is an array of scalar data. I have 6 string values for the key and 10 numeric float values for the corresponding value.

My data looks like this:

x5;7Sm;aI8w;2bNuW;lcDogi;pBTAdSKxha;2.73718;6.86649;0.49441;6.97707;2.62658;0.29478;3.97692;1.76468;4.86533;7.82656
sR;Ndu;PvlX;k3IJu;ZTwnAN;jTmtvmlqru;3.03941;1.84579;3.66076;7.14519;8.18409;5.87612;0.56569;1.49385;8.77644;6.98281
jm;cQL;ZXul;J0NIC;UwBz4S;O9VfAA74ao;6.20567;0.16487;3.02578;9.16104;0.14110;8.24097;8.23658;3.53050;7.11429;8.31212
3s;gId;wjXk;jJ3X2;7C833a;tzlTSHet4Y;8.08147;2.62935;9.00534;9.98493;5.17147;4.15576;8.60537;0.41931;9.79758;6.75664

Here comes my basic script. The hash key is made of 6 first columns, the values are the next 10 columns. In the next examples, only code between mark 1 and mark 2 will change.

#!/usr/bin/perl

use strict;
use warnings;

use Time::HiRes qw(gettimeofday tv_interval);
use Sys::Statistics::Linux::Processes;

my $lxs = Sys::Statistics::Linux::Processes->new;
$lxs->init;

my $start = [gettimeofday];
my %cache = ();

open(my $ifh, '<'.$ARGV[0])
    or die 'cannot open input file';

while (<$ifh>) {
    chomp;
    my @fields = split ';', $_;

    # mark 1
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[6];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[7];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[8];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[9];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[10];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[11];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[12];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[13];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[14];
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]}{operation1}{sum} = $fields[15];
    # mark 2
}

# use Data::Dumper;
# print Dumper(\%cache);

close($ifh);

my $stop = [gettimeofday];
my $stat = $lxs->get;

printf(
    "time: %.1f seconds, memory : %uM\n",
    tv_interval($start, $stop),
    $stat->{$$}{vsize} / (1024 * 1024)
);

It corresponds to the way Talend Open Studio currently stores the aggregation hash, so you may understand what "operation1" and "sum" mean :-). Here comes a second way to do it:

    # mark 1
    $cache{$fields[0]}{$fields[1]}{$fields[2]}{$fields[3]}{$fields[4]}{$fields[5]} = [
        $fields[6],
        $fields[7],
        $fields[8],
        $fields[9],
        $fields[10],
        $fields[11],
        $fields[12],
        $fields[13],
        $fields[14],
        $fields[15],
    ];
    # mark 2

and a third one, very compact:

    # mark 1
    $cache{join($;, @fields[0..5])} = join($;, @fields[6..15]);
    # mark 2

I've performed the test on a 1 million lines file :

$ perl hash_memory1.pl /tmp/agg.in; perl hash_memory2.pl /tmp/agg.in; perl hash_memory3.pl /tmp/agg.in
time: 35.4 seconds, memory : 991M
time: 22.1 seconds, memory : 1205M
time: 15.7 seconds, memory : 197M

As hash_memory1.pl is the current "way to do it" in Talend Open Studio, I take it as the reference. The second method is 1.6 times faster, but 1.2 times more memory consuming. Third method is 2.25 times faster and 5 times less memory consuming. Huge improvement, really!

According to the limit of 3GB per process on a Linux x86 32bits architecture, it means that before we could have 3 millions keys and with this improvement we can have 15 million lines in memory.

krunner history

2008-10-16T07:36:09+00:00

krunner is the small linux application launcher that pops-up when you hit Alt+F2. Its behavior in kde4 changed from the behavior of minicli of kde-3.5. In order to get the previous behavior back, right-click on the text field of krunner and change the text completion mode to “short automatic”.

I never knew this menu before today. That’s why I give this tip here.

Want to create a startup?

2008-10-16T07:35:17+00:00

Before you start to create your startup, listen to (or read) Bill Inmon’s advice.

Some things that you must keep in mind are:

be pragmatic: “build what you can sell”, “95% of the startup resources will go to marketing and sales.”
protect your work “intellectual property is the backbone of the startup”
be ready for competition “competition is lurking behind every shadow and every bush”
business is business, don’t keep unproductive people “Get rid of unproductive people quickly”
Remember that a lot of money is needed to start because there is a “long way to go before there is any income”
But avoid venture capitalists at all costs
“be flexible” and adapt your business plan
…

I have not created a startup, but as I work in a startup, I can see how these advice are more or less followed by my employers. And these advice apply to open source editors too as soon as they need to make money.

Talend Open Studio v3.0.0: discover the essential knowledge

2008-10-14T23:01:20+00:00

Talend Open Studio v3.0.0 has just been released. Please download it now and check all that this new release has to offer! To discover all the new features, please visit the changelog page.

To discover the essential knowledge, Talend Open Studio documentation is split into two guides:

The User Guide provide general use information and describe the Graphical Interface User
The Reference Guide of Talend components describe the use of all components and include several use cases

To turn theory into practice, Talend propose several resources :

Tutorials (video) published in TalendForge
Job Design Scenarios described in our documentation
A Demo Project included in Talend Open Studio

This 3 resources are independent, the examples files are not common and the use cases are not the same.

To have ready-to-use Job Designs at your disposal, import the Demo Project include in Talend Open Studio.

In this release, the demo project are updated.

How to import the Demo Project?

Simply follow the steps below:

Launch Talend Open Studio
In the login window, click on Manage and click on Demo
Choose Java or Perl Demo Project
Click on Finish to complete the operation
Choose the TALENDDEMOJAVA or TALENDDEMOPERL on the “Existing” list
Click on Open

How to install the example files?

In the view Repository, open the node Job Designs and Contexts:

The default folder used to store the example files is “C// “. To change this value, let’s double click on globalContext (node Contexts), Next and let’s go in the panel Value as Tree to change the value of the var “defaultDir”
Open the job named beforeRunJobs and execute it to install the example files

Additional setting: Open the Metadata/Db Connections/demoMysql and change the settings to establish the connection with your database.

How to take advantage of the Demo Project?

Open each Job Design and press F6 to run the job. If you wan’t some help about a component, select it and press F1. In the view Run Job, 2 features are available: Traces and Statistics.

The Statistics feature is available to observe the repartition of the data:

The Traces feature is available to observe the transformation of the data:

Want to talk about Talend Open Studio? Need help to use the software? Want to make suggestions and discuss with the dev team? Forum is the right place!

talend

Source: ocarbone.free.fr

Talend Open Studio: free technical workshops

2008-10-14T00:16:55+00:00

The best way to demonstrate to our potential users the value that Talend’s technology can bring to them is to have them try it themselves. So we are getting on the road and organizing a series of free technical workshops in a number of cities in France, Switzerland and Belgium.

During 3 hours the participants to the workshops will receive complimentary training on Talend Open Studio - and at the end of the workshop they will leave with a USB key containing the data integration jobs they have developed.

Next year, we’ll be taking this concept to other countries and continents - we have also been running several workshops in Germany, which have been very successful.

Check out the calendar and sign up now:

23rd of october - Lille
24th of october - Luxembourg City
28th of october - Genève
29th of october - Genève

Following the numerous requests received, I am please to announce you that the Talend Discovery Roadshow will make a stop in Brussels on the 25th of November 2008. We will make there 2 workshops in French and English.

If you wish to register for this workshop, or simply suggest another location, please click on this link.

talend

Source: ocarbone.free.fr

Data quality blogs and resources

2008-10-10T13:41:22+00:00

The best data quality blogs are now listed in one place: Data Quality Pro Blog Finder

This post explains how the blogs are evaluated and shows that independent bloggers have more interactions with the community than vendor bloggers.

Data Quality Tools vendors are listed here.

Latest news are aggregated here and Data quality events are here.

What’s in your databases?

2008-10-02T20:54:37+00:00

Often you only know approximately what’s in your databases. Data profiling tools can help you to get a better idea of your database content. The goal of a data profiler is not to analyze your data in depth but to give you at a glance the main features of your data. Especially, data profilers can give you information about missing data, duplicates, badly formatted data, invalid data (out of range, incorrect business pattern…)

Talend Open Profiler (TOP) can help you to explore your data. The latest version is the 1.1.0. Its official documentation is available here. A lot of other informations can be found on the Data Quality Pro website which also made a 21 page tutorial for addressing your data quality issues with Talend Open Profiler and their free DQ Pattern analyser.

This tutorial shows you how to use TOP to explore your data and gives a lot of tips about how to interpret your profiling results. And this is really important, because you can profile easily your data and produce nice graphics with TOP, but if you don’t know what to do once you obtained the results, then profiling your data did not really help you to enhance your data quality. The tutorial also presents a very useful function called “DQ Pattern analyser” that lists the patterns existing in the data. It helps you to quickly see what’s wrong with your data and permits to identify rare occurences.
This function does not exists yet in TOP, but it will be implemented for the next version along with other new features.

By the way, if you are missing a feature, it’s time to tell Talend’s team which new feature would be great to be in TOP.