All,

I want to present a general description on the design and implementation
of the external engines branch. The branch is not complete, but should
not have more not-documented architectural changes (except of what we
agree here). I've changed some things of the draft spec. I sent here
early this year. It now also includes a generic plugin mechanism. So let
discuss it again to have an agreement on all points.

The design goals was:
- Procedures, triggers and functions (will call all them as "routines"
here) are supported, capable of doing everything and more (example:
reusability, automated tasks on various parameters and trigger
variables, get table name of an executing trigger, etc) than a PSQL
routine can do.
- External routines are able to access the "current" database and
transaction and others databases.
- User code still uses the ISC/GDS API. This guarantees that every
library and component set of all languages are able to work inside the
database without a major rewrite.
- A totally new (object oriented, easily usable in C++ but still direct
[1] usable in C and some others languages, compatible across different
[2] C++ compilers and runtime boundaries in the memory space) interface
for the new things (i.e., the routines integration). ISC/GDS API is not
extended.
[1] Depending on decoding of pure virtual class layout.
[2] Depending on standardized platform ABI and compilers respecting it.
Firebird and EE plugin compiled with MSVC and example library compiled
with G++ is working.


Syntax:

{ CREATE [ OR ALTER ] | RECREATE | ALTER } PROCEDURE <name>
[ ( <parameter list> ) ]
[ RETURNS ( <parameter list> ) ]
ENTRY_POINT '<entry point>'
LANGUAGE <language>

DECLARE EXTERNAL FUNCTION <name>
[ <parameter list> ]
RETURNS <data type>
ENTRY_POINT '<entry point>'
LANGUAGE <language>

ALTER EXTERNAL FUNCTION <name>
[ ENTRY_POINT '<entry point>' ]
[ LANGUAGE <language> ]

[ Note: Objective of this branch was not fix legacy problems, but our
functions is not well defined as the procedures are. For example,
functions parameters doesn't have names. Instead of introduce CREATE
FUNCTION syntax, I just extended the current syntax of UDFs statements.
Since this work is target for V3, I think the general enhancements
(parameter names, grants, owner) to functions should be done direct in
HEAD. Functions (stored and external) and UDFs are going to be alterable
to each other anyway, and parameter names for external functions are not
*required*, so I don't see a problem. ]

{ CREATE [ OR ALTER ] | RECREATE | ALTER } TRIGGER <name>
...
ENTRY_POINT '<entry point>'
LANGUAGE <language>


Examples:

create procedure gen_rows (
start_n integer not null,
end_n integer not null
) returns (
n integer not null
)
entry_point 'cppeng_example!gen_rows'
language cpp;

declare external function wait_event
varchar(31) character set ascii
returns integer
entry_point 'cppeng_example!wait_event'
language cpp;

create trigger persons_replicate
after insert on persons
entry_point 'cppeng_example!replicate!ds1'
language cpp;


ODS changes:
- Increased the length of RDB$EXTERNAL_NAME domain (previously used only
in RDB$FUNCTIONS and RDB$FILTERS) from 31 to 255 bytes
- Added RDB$LANGUAGE to tables RDB$FUNCTIONS, RDB$PROCEDURES and
RDB$TRIGGERS
- Added RDB$ENTRYPOINT to tables RDB$PROCEDURES and RDB$TRIGGERS


Here are links to latest version of files that worth discuss. Please
verify them:

FirebirdApi.h is the generic new C++ API. These classes was designed to
future replace the ISC API in mind. Only the necessary classes and
methods for external engines was created -
http://firebird.cvs.sourceforge.net/firebird/firebird2/src/include/FirebirdApi.h?view=markup&pathrev=B2_5_ExtEngines

FirebirdPluginApi.h is the interface with plugins. Every kind of plugins
should use it. You see there how plugins read generic engine
informations as well how they register extra functionality (through
factories) -
http://firebird.cvs.sourceforge.net/firebird/firebird2/src/include/FirebirdPluginApi.h?view=markup&pathrev=B2_5_ExtEngines

FirebirdExternalApi.h has all the things related to External Engines -
http://firebird.cvs.sourceforge.net/firebird/firebird2/src/include/FirebirdExternalApi.h?view=markup&pathrev=B2_5_ExtEngines

CppEngExample.cpp has three examples, demonstrating how to write
external C++ functions, procedures and triggers -
http://firebird.cvs.sourceforge.net/firebird/firebird2/examples/plugins/CppEngExample.cpp?view=markup&pathrev=B2_5_ExtEngines

The files are all using int's to represent sizes, positions and whatever
matters. Though I prefer int's, there is no problem if you all decides
to using uint's. I also plan to remove all Error parameters where errors
are expected to never happen (Values::getCount, for example).


Firebird Plugin API:

When Firebird is initializing, it opens all *.conf files from
<fbroot>/plugins. For each plugin_module tag found, it constructs a
Plugin object, reads the corresponding plugin_config tag and inserts all
config information in the object.

It then gets the attribute value of plugin_module/filename, load it as a
dynamic (shared) library and calls the exported function firebirdPlugin
(PluginEntryPoint prototype) passing the Plugin object as parameter.

The plugin library may save the plugin object and call they methods
later. The object and all pointers returned by it are valid until the
plugin is unloaded (done through OS unload of the dynamic library) when
Firebird is shutting down.

Inside the plugin entry point (firebirdPlugin), the plugin may register
extra functionality that may be obtained by Firebird when required.
Currently only External Engines may be registered through
Plugin::setExternalEngineFactory.

Example plugin configuration file:

<language CPP>
plugin_module CPP_engine
</language>

<plugin_module CPP_engine>
filename $(this)/cpp_engine
plugin_config CPP_config
</plugin_module>

<plugin_config CPP_config>
path $(this)/cpp
</plugin_config>

Note that the language tag is ignored at this stage. Only plugin_module
and plugin_config are read. The dynamic library extension may be
ommitted, and $(this) expands to the directory of the .conf file.

Plugins access Firebird databases through the client library read from
Plugin::getLibraryName method. This method may return different
filenames depending on the server architecture, and may even return
NULL. Currently it returns:
Architecture -> File
Embedded -> The embedded library
Windows SS -> fbserver executable [3]
Windows CS/SC -> fb_inet_server executable [3]
POSIX CS/SC -> The embedded library
POSIX SS -> NULL [application should open it through dlopen(NULL)] [3]
[3] The functions are exported direct in the executable. Not well
know/used technique, but works in Windows and POSIX.


External Engines API:

Entry points are opaque strings to Firebird. They are recognized by
specific external engines. A external engine is the implementation of a
language. Languages are declared in config files (possibly in the same
file as a plugin, like in the config example present here).

When Firebird wants to load an external routine into its metadata cache,
it gets (if not already done for the database [4]) the external engine
through the plugin external engine factory and ask it for the routine.
The plugin used is the one referenced by the attribute plugin_module of
the routine's language.
[4] This is in Super-Server. In [Super-]Classic, different attachments
to one database creates multiple metadata caches and hence multiple
external engine instances.


The C++ (CPP) engine:

Entry points of the C++ engine are defined as following:
'<module name>!<routine name>!<misc info>'

The <module name> is used to locate the library, <routine name> is used
to locate the routine registered by the given module, and <misc info> is
an user defined string passed to the routine and can read there. "!<misc
info>" may be ommitted.

Modules available to the C++ engine should be in a directory listed
through the path attribute of the correspondent plugin_config tag. In
our example config file, the line below makes available to the C++
engine all libraries present on <fbroot>/plugins/cpp.
path $(this)/cpp

The user library should include CppEngine.h and link with the cpp_engine
library. Routines are easily defined and registered using some macros,
but nothing prevent you of doing things manually.
A example routine library is implemented in examples/plugins, showing
how to write functions, selectable procedures and triggers. Also it
shows how to interact with the current database through the ISC API.

The C++ routines state (i.e., member variables) are shared between
multiple invocations of the same routine until it's unloaded from the
metadata cache. But note that the C++ engine isolates the instances per
session, different from the raw interface that shares instances by
multiple sessions in SuperServer. This allows the routines to create and
cache prepared statements for the current database.

By default, C++ routines uses the same character set specified by the
client. They can modify it overriding the getCharSet method. The chosen
character set is valid for communications with the ISC library as well
as the communications done through the FirebirdExternal API.

Comments, please...


Adriano