Subject | Re: [firebird-support] Re: FB suitability for Consolidated Database of 15 GB. |
---|---|
Author | Sudheer Palaparambil |
Post date | 2011-10-11T02:18:59Z |
Hi,
SymmetricDS is an asynchronous data replication software package that
supports multiple subscribers
and bi-directional synchronization. It uses web and database technologies to
replicate tables between
relational databases, in near real time if desired. The software was
designed to scale for a large number of
databases, work across low-bandwidth connections, and withstand periods of
network outage. The
software can be installed as a standalone process, as a web application in a
Java application server, or it
can be embedded into another Java application.
A single installation of SymmetricDS attached to a target database is called
a node. A node is initialized
by a properties file and is configured by inserting configuration data into
a series of database tables. It
then creates database triggers on the application tables to be synchronized
so that database events are
captured for delivery to other SymmetricDS nodes
SymmetricDS supports synchronization across different database platforms
through the concept of
Database Dialects. A Database Dialect is an abstraction layer that
SymmetricDS uses to insulate the main
synchronization logic from database-specific implementation details.
SymmetricDS is extendable through extension points. Extension points are
custom, reusable Java code
that are configured via XML.
It captures the column values and record them in a table named sym_data.
The triggers participate in the
same transaction as the data change, and they record a unique transaction ID
in sym_data also. Changes are
batched together, given a batch ID, and sent to the target system. They
are played back exactly as they
occurred, in the same order. The batch is committed together for
efficiency. All changes for the same
transaction ID are guaranteed to be committed together.
SymmetricDS has features found in both replication and synchronization
products. It can be configured to
create replicas with complete data sets, it is aware of transactions, and it
replays data changes without
merges. But instead of synchronously locking transactions, it allows all
data changes to happen, then it
periodically synchronizes the data to other databases. It's strengths are
being able to sync a large
number of databases (1000's) and allowing for clients to be temporarily
disconnected. It's weaknesses
are a higher latency (than transaction log parsers) and less mature conflict
resolution.
Here are some features:
* triggers guarantee change data capture without special drivers for clients
* driven by configuration, with advanced features supported through a Java
API
* horizontal and vertical data filtering and sub-setting and transformation
* synchronization is scheduled at intervals or event driven
* deployment can be to an application server, stand alone, or embedded in an
application
* clients can be disconnect for periods of time
* web-based protocol that is compact and efficient and works with firewalls
* data changes replayed exactly with no staging or merging of operations
* database and operating system agnostic
Root and node properties can be tweaked to control how data flows.
SymmetricDS supports the concept of channels of data. Data synchronization
is defined at the table (or
table subset) level, and each managed table can be assigned to a channel
that helps control the flow of
data. A channel is a category of data that can be enabled, prioritized and
synchronized independently of
other channels. For example, in a retail environment, users may be waiting
for inventory documents to
update while a promotional sale event updates a large number of items. If
processed in order, the item
updates would delay the inventory updates even though the data is unrelated.
By assigning changes to the
item tables to an item channel and inventory tables' changes to an inventory
channel, the changes are
processed independently so inventory can get through despite the large
amount of item data.
It supports Firebird 2.0 and above. A UDF is needed by the database
triggers. SymmetricDS includes the
required UDF library, called SYM_UDF, in both source form (as a C program)
and as pre-compiled libraries
for both Windows and Linux.
Deployment Options
------------------------------
� Web application archive (WAR) deployed to an application server
This option means packaging a WAR file and deploying to your favorite web
server, like Apache
Tomcat. It's a little more work, but you can configure the web server to do
whatever you need.
SymmetricDS can also be embedded in an existing web application, if desired.
� Standalone service that embeds Jetty web server
This option means running the sym command line, which launches the built-in
Jetty web server.
This is a simple option because it is already provided, but you lose the
flexibility to configure the
web server any further.
� Embedded as a Java library in an application
This option means you must write a wrapper Java program that runs
SymmetricDS. You would
probably use Jetty web server, which is also embeddable. You could bring up
an embedded
database like Derby or H2. You could configure the web server, database, or
SymmetricDS to do
whatever you needed, but it's also the most work of the three options
discussed thus far.
� Grails Application
A Grails SymmetricDS plugin is provided at the default Grails plugin site.
This option ends up
being a WAR deployment, but allows for the use of the Grails SDK for
configuring and building
the deployment. The plugin also provides Gorm (Hibernate) access to many of
the core database
tables.
The only trouble I faced is the sym_udf entry point could not be found
problem. This was solved by
instsalling MS VC++ 2008 redistributable.
The SymmetricDS forum is also very responsive. There are also some samples
with the distribution.
Regards.
Sudheer Palaparmbil
SymmetricDS is an asynchronous data replication software package that
supports multiple subscribers
and bi-directional synchronization. It uses web and database technologies to
replicate tables between
relational databases, in near real time if desired. The software was
designed to scale for a large number of
databases, work across low-bandwidth connections, and withstand periods of
network outage. The
software can be installed as a standalone process, as a web application in a
Java application server, or it
can be embedded into another Java application.
A single installation of SymmetricDS attached to a target database is called
a node. A node is initialized
by a properties file and is configured by inserting configuration data into
a series of database tables. It
then creates database triggers on the application tables to be synchronized
so that database events are
captured for delivery to other SymmetricDS nodes
SymmetricDS supports synchronization across different database platforms
through the concept of
Database Dialects. A Database Dialect is an abstraction layer that
SymmetricDS uses to insulate the main
synchronization logic from database-specific implementation details.
SymmetricDS is extendable through extension points. Extension points are
custom, reusable Java code
that are configured via XML.
It captures the column values and record them in a table named sym_data.
The triggers participate in the
same transaction as the data change, and they record a unique transaction ID
in sym_data also. Changes are
batched together, given a batch ID, and sent to the target system. They
are played back exactly as they
occurred, in the same order. The batch is committed together for
efficiency. All changes for the same
transaction ID are guaranteed to be committed together.
SymmetricDS has features found in both replication and synchronization
products. It can be configured to
create replicas with complete data sets, it is aware of transactions, and it
replays data changes without
merges. But instead of synchronously locking transactions, it allows all
data changes to happen, then it
periodically synchronizes the data to other databases. It's strengths are
being able to sync a large
number of databases (1000's) and allowing for clients to be temporarily
disconnected. It's weaknesses
are a higher latency (than transaction log parsers) and less mature conflict
resolution.
Here are some features:
* triggers guarantee change data capture without special drivers for clients
* driven by configuration, with advanced features supported through a Java
API
* horizontal and vertical data filtering and sub-setting and transformation
* synchronization is scheduled at intervals or event driven
* deployment can be to an application server, stand alone, or embedded in an
application
* clients can be disconnect for periods of time
* web-based protocol that is compact and efficient and works with firewalls
* data changes replayed exactly with no staging or merging of operations
* database and operating system agnostic
Root and node properties can be tweaked to control how data flows.
SymmetricDS supports the concept of channels of data. Data synchronization
is defined at the table (or
table subset) level, and each managed table can be assigned to a channel
that helps control the flow of
data. A channel is a category of data that can be enabled, prioritized and
synchronized independently of
other channels. For example, in a retail environment, users may be waiting
for inventory documents to
update while a promotional sale event updates a large number of items. If
processed in order, the item
updates would delay the inventory updates even though the data is unrelated.
By assigning changes to the
item tables to an item channel and inventory tables' changes to an inventory
channel, the changes are
processed independently so inventory can get through despite the large
amount of item data.
It supports Firebird 2.0 and above. A UDF is needed by the database
triggers. SymmetricDS includes the
required UDF library, called SYM_UDF, in both source form (as a C program)
and as pre-compiled libraries
for both Windows and Linux.
Deployment Options
------------------------------
� Web application archive (WAR) deployed to an application server
This option means packaging a WAR file and deploying to your favorite web
server, like Apache
Tomcat. It's a little more work, but you can configure the web server to do
whatever you need.
SymmetricDS can also be embedded in an existing web application, if desired.
� Standalone service that embeds Jetty web server
This option means running the sym command line, which launches the built-in
Jetty web server.
This is a simple option because it is already provided, but you lose the
flexibility to configure the
web server any further.
� Embedded as a Java library in an application
This option means you must write a wrapper Java program that runs
SymmetricDS. You would
probably use Jetty web server, which is also embeddable. You could bring up
an embedded
database like Derby or H2. You could configure the web server, database, or
SymmetricDS to do
whatever you needed, but it's also the most work of the three options
discussed thus far.
� Grails Application
A Grails SymmetricDS plugin is provided at the default Grails plugin site.
This option ends up
being a WAR deployment, but allows for the use of the Grails SDK for
configuring and building
the deployment. The plugin also provides Gorm (Hibernate) access to many of
the core database
tables.
The only trouble I faced is the sym_udf entry point could not be found
problem. This was solved by
instsalling MS VC++ 2008 redistributable.
The SymmetricDS forum is also very responsive. There are also some samples
with the distribution.
Regards.
Sudheer Palaparmbil
On Mon, Oct 10, 2011 at 11:00 PM, raja_s_patil <kpr.rspatil@...>wrote:
> **
>
>
>
>
> --- In firebird-support@yahoogroups.com, Thomas Steinmaurer <ts@...>
> wrote:
> >
> > >> Pls have a look at the open source project
> > >> http://symmetricds.codehaus.org/ for synchronization. I have been
> using this
> > >> for the last 2 years
> > >> with Firebird database. This works fantastically. You can sync both
> ways, HO
> > >> to branch as well as branch to HO.
> > >>
> > >
> > > Yet another option for me. I visited the link you mentioned and I am
> impressed with general Description. The idea of two tier replication is
> simply fantastic. Since you are using it for last two years can you give us
> general idea about it ? like how configuration is done, how is response time
> (since its in Java), Can it function with Delphi application ? etc. etc.
> Please also mention any problems you faced may be initially and resolved
> later.
> >
> > I'm not using it, but interesting products. Check out Firebird-specific
> information here:
> >
> http://symmetricds.codehaus.org/guide/2.4/html-single/user-guide.html#ap02-firebird
> >
> >
> > A mandatory UDF library and limiting BLOB columns to 16K might be
> > show-stoppers.
> >
> >
>
> Thanks Thomas Steinmaurer for making me aware of probable pitfall.
> regarding using this product, I was addressing Mr. Sudheer, It was not
> referring you. Anyways I will keep this option open as you have suggested, I
> will discuss with client and arrive at final decision. After all he is going
> to pay for licenses So his opinion is vital in going for a particular
> product. We will be just evaluating technical aspect of all probable
> solution and put forward a technical comparison as well cost benefit
> analysis, ultimately he has to decide who will own his Money. If he decides
> to drop the project all together, all our efforts are going to go to drain.
> What to do, its part of life, cant help, but we can at most put all our
> efforts in anticipation to to be successful.
>
> Thanks and warm regards
>
> R. S. Patil
>
>
>
[Non-text portions of this message have been removed]