// some workarounds for words contain `++`
// https://github.com/asciidoctor/asciidoctor/issues/1208
:cpp: C++
:gpp: g++

This directory contains a sample extension module code to bridge
Gauche and {cpp} libraries.

The two files, link:mqueue.h[] and link:mqueue.cpp[], are our hypothetical
external {cpp} library.  It implements a simple message queue.
It is independent from Gauche.   Our mission here is, given this
mqueue library, to write a Gauche binding for it.

The following files are needed for our extension:

[horizontal]
link:Makefile.in[]::		template makefile
link:configure[]::		"configure" script
link:mqueue_glue.cpp[]::	a bridge (glue) between {cpp} library and Gauche
link:mqueue_glue.h[]::		ditto
link:mqueue_lib.scm[]::		Scheme binding definition
link:test.scm[]::		unit test
link:example/mqueue-cpp.scm[]::	Scheme module definition

The skeleton of those files can be generated by
`gauche-package` command, like this:

[source,console]
----
% gauche-package generate mqueue-cpp example.mqueue-cpp
----

I renamed the generated `mqueue_cpp.h`, `mqueue_cpp.c`, `mqueue_cpplib.scm`
to more descriptive names, `mqueue_glue.h`, `mqueue_glue.cpp`,
and `mqueue_lib.scm`, respectively.

The generated files don't know about {cpp}, so you have to tweak link:configure[]
and link:Makefile.in[] to enable {cpp}.

1. Add (`cf-prog-cxx`) in link:configure[].  This tries to figure out how
to call {cpp} compiler, and sets up a substitution variable `CXX`.
+
+NOTE:+ the {cpp} compiler has to be 'compatible' with the C
compiler used to build Gauche, e.g. it must accept the
same set of command-line options to generate a shared
library.  Gcc and {gpp} are such compatible ones.

2. In link:Makefile.in[], added the following line to receive the {cpp}
compiler name `configure.in` finds.
+
[source,Makefile]
----
CXX      = @CXX@
----

3. Also in link:Makefile.in[], when builing `mqueue_cpp.so`, give `--c++=$(CXX)`
option to the `gauche-package` script to have it use the {cpp} compiler
instead of the default C compiler.
+
[source,Makefile]
----
mqueue_cpp.$(SOEXT): $(mqueue_SRCS) $(mqueue_HDRS)
        $(GAUCHE_PACKAGE) compile --c++=$(CXX) --verbose mqueue_cpp $(mqueue_SRCS)
----


== Glue code

Now, let's take a look at the 'glue' code.

The header file link:mqueue_glue.h[] begins with including both the Gauche
interface and the library interface.

[source,c++]
----
#include <gauche.h>
#include <gauche/extend.h>
#include "mqueue.h"
----

It is important to include `+<gauche.h>+` *before any other system
header files*.  Since `gauche.h` includes some configuration
information (from `gauche/config.h`) which may affect the definitions
of the standard header files; one example is `_FILE_OFFSET_BITS`,
which is 32 on most 32bit Linux by default, but Gauche sets it to 64,
that causes the system library calls to call 64bit versions.

We need a Gauche class that represents `MQueue` object in {cpp} world.
We're going to use Gauche's foreign pointer feature to implement it.
The pointer to the Scheme `<mqueue>` class is held in this external
variable.  It is set in the extension initialization routine.

[source,c++]
----
extern ScmClass *MQueueClass;
----

Some conveninece macros.  The `<mqueue>` object in Scheme world is
wrapped in `ScmForeignPointer.`  'Unboxing' is to retrieve the {cpp}
`+MQueue*+` object from it, and 'boxing' is to wrap {cpp} object by
`ScmForeignPointer`.

[source,c++]
----
#define MQUEUE_P(obj)      SCM_XTYPEP(obj, MQueueClass)
#define MQUEUE_UNBOX(obj)  SCM_FOREIGN_POINTER_REF(MQueue*, obj)
#define MQUEUE_BOX(ptr)    Scm_MakeForeignPointer(MQueueClass, ptr)
----

Next comes the declaration of the extension initialization routine.
It is important for it to be declared/defined in `"C"` linkage scope.
Macros `SCM_DECL_BEGIN` and `SCM_DECL_END` ensures it (they are the same
as `extern "C" {` and `}`.)

[source,c++]
----
extern void Scm_Init_mqueue_cpp();
----

The source link:mqueue_glue.cpp[] implements a bridging machinery.  The
important part is this statement:

[source,c++]
----
MQueueClass =
    Scm_MakeForeignPointerClass(mod, "<mqueue>",
                                mqueue_print,
                                mqueue_cleanup,
                                SCM_FOREIGN_POINTER_KEEP_IDENTITY|SCM_FOREIGN_POINTER_MAP_NULL);
----

This creates a new Scheme class, named `<mqueue>`, as a foreign pointer
class and stores its pointer to `MQueueClass`.  Scheme variable `<mqueue>`
is bound to the newly created class in the module `mod`.

The second argument is a print procedure, called whenever the
instance of `<mqueue>` is printed.

The tricky part here is the 'cleanup' procedure.  If you pass
a cleanup procedure, it is called when the Scheme `<mqueue>` object
is about to be GC-ed, so that you can free the resources in the {cpp}
world.  Although it sounds simple, it requires more attention than
its superficial simplicity.  If you do something wrong here,
you'll get a nasty bug which is very hard to track.

Typically, there are a few cases with regard to the foreign resource
management.

Case 1::
You allocate the foreign object via Gauche's allocator.
(`SCM_NEW` etc.)   In this case, Gauche's GC can take care
of deallocation, so you don't need specify 'cleanup' procedure
except you have other resources to free (e.g. file descriptors).

anchor:frm-case-2[]Case 2::
The foreign object is allocated by the foreign library, and
it is only pointed from the Scheme world.  In this case,
you are responsible to free the foreign object when there's
no reference from the Scheme world to it.  You can use cleanup
procedure to do so.
+
Note that the cleanup procedure is called when there's
no reference to the Scheme `<mqueue>` object here, but it doesn't
necessarily means that the {cpp} `MQueue` object pointed by it can be
freed -- what if there are other Scheme `<mqueue>` object which
points to the same {cpp} `MQueue` object?  Gauche's GC can't detect
such case, since the {cpp} `MQueue` object is outside the scope
of Gauche's memory management.
+
There's no universal solution for it, but Gauche provides a
convenience mehchanism that works in typical cases.  If you
give a flag `SCM_FOREIGN_POINTER_KEEP_IDENTITY` to the
`Scm_MakeForeignPointerClass`, then Gauche guarantees that
`Scm_MakeForeignPointer` returns exactly same Scheme object
for the same pointer (internally, it uses a weak hash table to
map the `+void*+` pointer value to the created Scheme object).
If your code never assigns the wrapped foreign pointer to
other Scheme objects, then you can be sure that whenever the
Scheme `<mqueue>` object is GC-ed, the {cpp} `MQueue` object pointed
by it can also be destroyed.

Case 3::
The foreign object is allocated by the foreign library, and
it may be pointed from both Scheme and {cpp} worlds.  In this case,
you cannot destroy the foreign object even when the Scheme
object is GC-ed, since there might be a pointer in {cpp} world
that points to the same foreign object.  Gauche's GC cannot
track it.
+
This is very difficult problem indeed.  Usually, if this is
the case, such foreign library provides some sort of resource
management infrastructure such as reference counting.  If so,
you can drop the reference count in the cleanup routine,
provided that you do increment the reference count in the
boxing routine.
+
(`SCM_FOREIGN_POINTER_KEEP_IDENTITY`, for there can be more than
one Scheme object that points to the same foreign object).
+
Unfortunately this won't work sometimes; there are cases that
you also have to pass a Scheme object to the foreign library
(e.g. closures for callback), and sometimes they form a loop
of reference, spanning into both Scheme and foreign library
world.  But it is beyond the scope of this document and I'll
leave the further discussion for some other time.


The use of `SCM_FOREIGN_POINTER_KEEP_IDENTITY` is convenient even
for case 1 or 3, since you can use `eq?` to test the pointer identity
of the foreign object.  However, using it incurs overhead of
bookkeeping.

The other flag, `SCM_FOREIGN_POINTER_MAP_NULL`, is a convenience flag.
If you specify this flag, `Scm_MakeForeignPointer` returns `SCM_FALSE`
if `NULL` is given.  It is handy if you call a foreign routine that
may return `NULL`; with this flag, such case can be seen from
Scheme as if the foreign routine returned Scheme `#f`.
(Be careful, however, that when you use this flag, you cannot assume
the returned value of `Scm_MakeForeignPointer(KLASS, PTR)` is the
instance of `KLASS`.  If you play with the returned value in C code
you have to check its type.)


== Stub code

The link:mqueue_lib.scm[] file binds the Scheme procedures and the
foreign library functions.   It looks like a Scheme source, and you
can write Scheme code in it as well, but it also contains several forms
that can't be executed if you load it directly to Gauche.

Such forms needs to be _precompiled_ by Gauche's AOT compiler.
The `gauche-package compile` command takes care of it.

Let's see those forms specific to be precompiled.  Open link:mqueue_lib.scm[]
in a separete window and read along.

The first form, `(in-module example.mqueue-cpp)`, sets up the current
module much like `select-module`.  However, unlike the latter, `in-module`
doesn't need the specified module to exist beforehand.  You can't
just `(use example.mqueue-cpp)`, since it will try to load the compiled
extension, which is exactly what we're about to produce.

A quick side note: If you put Scheme code in it (which will also
be precompiled), be aware that you need to `use` dependent modules
after `in-module` form; since we don't read
`(define-module example.mqueue-cpp ...)`.   In the example file,

The second form is `(inline-stub ...)`.  It is the way to write
C/C++ interface code inside Scheme source.  The code inside
`inline-stub` isn't Scheme at all; we call it _stub code_ for
historical reasons.

The first thing in the stub code is `(.include "mqueue_glue.h")`, which
is C/C++ `#include "mqueue_glue.h"`.

The `declare-stub-type` directive tells the stub generator about the
`<mqueue>` class you create in this extension.

[source,scheme]
----
(declare-stub-type <mqueue> "MQueue*" "mqueue"
  "MQUEUE_P" "MQUEUE_UNBOX" "MQUEUE_BOX")
----

The arguments are: Scheme name, C type name, description (used in
error messages), C function or macro to check the type, an unboxer,
and a boxer.

NOTE: I'm planning to have better way to define foreign types.
Consider this `declare-stub-type` as temporary solution.

The `define-cproc` directive defines the foreign function interface:

[source,scheme]
----
(define-cproc SCHEME-NAME (ARGSPEC ...) ::RETURN-TYPE
  CLAUSE ...)
----

`SCHEME-NAME` becomes the name of the Scheme function.  `ARGSPEC`
specifies the arguments and its type.

----
ARGSPEC := name::type
----

To see the exact meanings of type and how it is mapped in the
C world, peek the source of `src/genstub` (search "Type handling").
You can use `:rest`, `:optional` and `:keyword` a la Common Lisp.

`::RETURN-TYPE` specifies what type of value the function body
returns.  If the function doesn't need to generate a value,
specify `::<void>`.  The Scheme function returns `#<undef>` then.
If the function generates a `ScmObj` value, you can omit `::RETURN-TYPE`;
or give `::<top>`.   If you give other types, such as `::<int>`,
the stub generator can generate a code to convert C `int` to
Scheme `integer`.

`CLAUSE` gives the information about how to call the C stuff.
You can write C/C++ code in S-exression (CiSE) here.  Consult
Gauche document for details of CiSE.

Other clause type worth to mention here is `catch` clause.
It handles {cpp} exceptions.

Gauche has its own exception handling system, and although both
can coexist, it is not allowed that one nonlocal exit jumps the
dynamic environment of the other.  For example, Suppose you call {cpp}
function (2) from Scheme (1), which in turn calls Scheme function (3),
which calls {cpp} function (4), which calls {cpp} function (5).
And the {cpp} function (5) raised an exception.

   Scheme(1) --> C++(2) --> Scheme(3) --> C++(4) --> C++(5)
                                                        throw!

It is fine as far as the exception is caught within (5) or (4).

   Scheme(1) --> C++(2) --> Scheme(3) --> C++(4) --> C++(5)
                                           catch <----- throw


However, it is not allowed to catch the exception in the function (2),
since it will jump the Gauche's exception frame set up for Scheme (3).

   Scheme(1) --> C++(2) --> Scheme(3) --> C++(4) --> C++(5)
                   catch <----------------------------- throw

This fact mandates that if you call a {cpp} function that may throw
an exception, you have to catch it within the stub routine.
The typical way is to convert the caught exception to Gauche's
exception.  The Gauche's exception can then be caught in Scheme's
`guard` form.

   Scheme(1) --> C++(2) --> Scheme(3) --> C++(4) --> C++(5)
                                            catch <---- throw
                              guard <----- raise

For the convenience, you can write a `catch` spec within the stub
description, such as this:

[source,scheme]
----
(define-cproc mqueue-pop! (mq::<mqueue>) ::<const-cstring>
   (catch ("MQueueException& e"
           (Scm_Error "mqueue-pop!: %s" (ref e reason (c_str)))))
   (return (SCM_MAKE_STR_COPYING (ref (-> mq (popMessage)) (c_str)))))
----

The `catch` spec causes the body of stub function to be surrounded
by try, and appropriate catch clauses to be generated.  The generated
code roughly will be something like this:

[source,c++]
----
try {
  const char* result;
  result = mq->popMessage().c_str();
  return SCM_MAKE_STR_COPYING(result);
}
catch (MQueueException& e) {
  Scm_Error("mqueue-pop!: %s", e.reason.c_str());
}
catch (std::exception& e) {
  Scm_Error("mqueue-pop!: %s", e.what());
}
catch (...) {
  Scm_Error("C++ exception is thrown in mqueue-pop!");
}
----

Note that mere existence of `catch` spec causes the last two
catch clause (`std::exception` and `+...+`) to be generated.


== Caveats

Be always careful about the ownership of resources.  If you're
within Gauche's world, most things are taken care of by its garbage
collector.  But once you step into the foreign land, it's up
to you again to make sure all resources are managed.

Especially, make sure the memory owned by Gauche is always visible
from Gauche.  For example, if you specify `<const-cstring>` in the
argument type, Gauche converts Scheme string to NUL-terminated
C string, but Gauche still owns the resulted string.  So, if the
foreign function retains the passed pointer within itself, such as
this fictious code:

[source,c++]
----
/* foreign code */
static const char *ss;

void foo(const char *s)
{
  ss = s;
}
----

Then, it is wrong to write a stub function like this:

[source,scheme]
----
;; stub function
(define-cproc foo (s::<const-cstring>) ::<void>
  (call "foo"))
----

It compiles, but the string passed to `foo` is stored
in the location that Gauche doesn't know, so later the string body
is GC-ed, leaving the foreign pointer dangling.

The more subtle case is to pass the foreign object pointer, which
itself is allocated via foreign allocator, to the foreign function.
If we adopt the <<#frm-case-2,Case 2>> scheme described above, the foreign object
would be destroyed when its wrapping Scheme object is GC-ed, even
if the foreign object itself isn't allocated by Gauche.
