Please note: All new versions will be located at GitHub under Waarp new project name :

http://waarp.github.com/Waarp/

The project still remains Open Source under GPL V3.

 

 

GoldanGate is an open source file transfer monitor in Java. It unifies several protocols (FTP, HTTP and proprietary protocols) in an efficient and secured way. Its purpose is to enable bridging between several protocols and to enable dynamic pre or post action on transfer or other commands.

 

 

GoldenGate is composed of several parts, most of them could be used separately without the full package:

• an open FTP Server: it can be used in a stand alone version to fit your needs.

• an open HTTP Server: this component would be at some point dedicated to the project.

• an open proprietary file transfer monitor, OpenR66: it can be used in a stand alone version to fit your needs.

• a kernel for common actions (GoldenGate) that could be taken when merging any part of the above.

The start idea was to be able to receive through several protocols some files (or not files but data) and to transmit them into a system not directly accessible, but without breaking the global vision of the data transfer (where it is, what status, and so on...).

A standard FTP server (or SFTP or FTPS) do not allow to make post action (and of course neither pre action) when a file transfer occurs. So the reason of this new project to be able to imply business actions that can be whatever the need when an event occurs in the transfer protocol.

 

For instance, when a file is to be transmitted through an FTP service, perhaps some tests should be done to see if the next business step (another application) is ready to accept transfers. When a file is transmitted, an action can occur like for instance checking of integrity (business vision) or retransmission to another business partner (using eventually a different protocol). Also one possibly wants to follow any transfers (or other actions defined in the protocols) and to log them somewhere in its own format, either to just be able to look at what happened or even to restart some actions to correct wrong situation in production.

 

To be able to do all of that, we decided to implement the most common used file transfer protocols (FTP, HTTP) in such a way they can fulfill those behaviours. FTP was the first step. HTTP for now is simply a bunch of JSP and classes attached to this JSP. SSH is planned.

 

Of course, if they can do all the stuff we present before, they can also do simple thing like standard FTP server.

 

Why a proprietary protocol therefore ? Well, all of those protocols suffers from pitfalls like:

• When a file is transfered, even if it is by block (FTP BLOCK mode), it does not have any control on each block transfered. So if something goes wrong on the network, it could happened that the sender or the receiver will not see what happens. The proprietary protocol will implement such a check on each transfered block.

• When a file transfered ends up abnormally, it does not have true facility to enable to restart from where it falls. Even the FTP REST command is not enough since who knows at which step the error occurs. And asking for the size of the remote file could be not enough. The proprietary protocol will implement such a restart facility on failed transfer.

• In production context, if you have to send many files, how do you follow those transfers and check them? Some FTP clients does such thing, however they generally don't have the hability to enable planned production. The proprietary protocol will implement such control, planning, and adding some functionalities like limitating (or not) the number of concurrent file transfers.

• Of course, all of the behaviours we described before for GoldenGate will be implemented in this proprietary protocol.

What kind of proprietary protocol it will be? It will be a proprietary protocol that mimic the CFT PESIT one plus some extension, including in the protocol itself (block checking and restarting), but also to enable specific functionalities like MD5 block control (optional), restart possibilities, duplexing network connection, SSL support, database support, ...

 

This project is in production in the French Ministery of Finances to enable file transfers from an FTP protocol (client side) to an OpenR66 protocol (server side protocol) since end of 2009.