Clover - ETL tool for data integration

Basic concept

clover.ETL is a data transformation and data integration tool. It is platform-independent and resource-efficient. Due to high scalability it can be used on low-cost PCs as well as high-end multi-processors servers.

clover.ETL's flexibility allows for easy design and implementation of data migration solutions, data transformation applications and consolidation of data sources between dissimilar and heterogeneous systems in your organization.

clover.ETL will work with any structured data. Whether it is stored in text files (fixed length records, CSV, XML), produced by MS Excel, dBase or FoxPro, kept in binary format or contained in modern relational database systems ? clover.ETL will allow for the combination, transformation and movement of all data from any such source.

Since clover.ETL natively supports UNICODE character set, it has no issues with processing data in any character coding / pages, be it ASCII, Latin 2, CP-1250 or simplified Chinese. Clover.ETL will handle them all.

clover.ETL is enhanced with the visual design of data transformations. clover.GUI. It allows for the easy design of any data manipulating application through the suitable combination of standard predefined ETL components using visual editor. The flexibility of clover.ETL allows an experienced programmer to complement visual design with 'classic' program sections, thus enhancing functionality of standard components. Such code still remains fully and directly transferable to any supported platform.

Typical applications for clover.ETL

• Data migration
• Data collection
• ETL for Data Warehouses
• Data integration

Licensing policies

Clover.ETL is an Open Source tool distributed under dual license (commercial & LGPL) which allows total transparency and control over the tool, as complete source code for the engine is available to all customers and end-users. With Clover.ETL you are not faced with a frequent risk of lock-down to a specific proprietary technology, where the user is at the mercy of the supplier for debugging and enhancements of functionality.

Support and services

In order to use any software tool in a professional environment, it is necessary to have competent support and service to provide bug fixes or application enhancements and to have expert consultants at hand who have practical knowledge and experience with the tool. Customer support as well as trainig and consulting services for Clover.ETL/Clover.GUI are offered by OpenSys company .

Technical description

clover.ETL is written in Java, which insures it's full compatibility with all platforms running Java Virtual Machine (JVM). Up to date clover.ETL was succesfully used on Linux, AIX, Solaris, HP-UX, Windows, AS/400 and others.

clover.ETL is a multithreaded application and as such it can efficiently utilize multi-processor systems as well as single-CPU, and low cost PC.

clover.ETL is based on the so-called Transformation Graph. Transformation Graph is a set of certain specialized components, which are interlinked via data pipes. Every such component performs certain operation (database extracts, output to Excel file, data sort, filtering, FTP transfer, etc.). Data processed by clover.ETL flows through the transformation graph and is step-by-step transformed into the required format. While performing the transformation, data can be merged, sorted, split or enriched in many other ways.

Every component uses its own independent thread and the entire system is implemented with the so-called pipeline-parallelism. Once a component is done with processing a record, it will pass it on to the adjacent interconnected component, which in turn will perform its operation. This process is repeated until the final result is achieved. Individual records progress through the transformation graph in a way, similar to a manufacturing process, where sub-assemblies are combined forming a final product. One of the main advantages of such architecture is that requirement for operational memory is mandated by the transformation graph rather than amount of data presented for processing.

Schema of clover.ETL architecture

clover.ETL is a server application complemented with the Visual Transformation Editor - clover.GUI. The GUI tools allows for creating and modifying data transformation in a graphical format where the developer selects from a pallet of existing components and interconnects them on the worksheet into transformation graph fulfilling the required task. Visual design essentially simplifies and streamlines the entire process. The result is a self-documenting 'code', which can be executed on any of the supported platforms.

It is possible to create partial transformation (sub-graphs or 'macros') which are re-used later as building blocks in more complex transformations. Such approach follows the encapsulation principle of Object Oriented Programming.

Using the visual editor allows for the use of the drag'n'drop method to compose a graph while selecting from a pallet of available components. These need to be further interconnected via input/output ports.

Individual graph branches can be suitably phased and thus executed in a preset order. It is also possible to create conditional branches, generated and executed if a specified condition is met.

To enhance the comfort of editing the graph, functions like Undo, Copy, Paste or Zoom are also available. Using vertical and horizontal slide bars, it is possible to create graphs bigger then actual screen size. The view of the entire graph is then still just a click away (the viewing zoom will be automatically adjusted)

The individual components will be configured using a parameter set. Entering the parameters is done via dialog box. The parameter values can be as simple as numerical/text strings, and a selection from a combobox or as complex as composing a re-formatting code and data structure of the component output.

Each component can be appropriately labeled and described in its individual text window, which further describes purpose and behavior of the component in each given graph.