= Precompile (a.k.a. ahead-of-time compile) Scheme code

NOTE: This is a feature still in development and very likely
change over time.
Especially, we usually keep ABI compatibility across micro versions
so that you don't need to recompile extensions for every new
release, but we don't guarantee that for precompiled binary yet --
if you precompile your library to distribute, let users know
that they need to recompile it for new Gauche releases, even if
it's just a micro version update.

As a script engine, the default operating mode of Gauche is to read
Scheme source code at run-time, compile it on-the-fly and execute.
However, that incurs some overhead of loading and compiling, and
you might want to cut the time if you have tons of Scheme code.

Gauche has a feature to compile Scheme code and dump the result
as C code, which can be compiled by a C compiler into DSO and dlopen-ed
at runtime.  At this moment, we don't really "compile Scheme to C" --
what we do is to compile Scheme into Gauche's VM instruction sequence,
as we do at run-time, and to dump the result as a C static data structure.
Thus run-time performance won't improve by precompilation, but you do
cut the compile time, including macro expansion, and the parsing time.
In future, we may improve performance of precompiled code, for
we can spend more time on optimization.

== Things to consider before precompiling

1. The precompiler needs to make sense of source statically (i.e. without
executing it), so we recommend the source to follow certain constraints:
+
** Each scm source begins with a `define-module` form.
** One scm source defines one module.

2. In the normal operation, Gauche compiles and executes each
toplevel form.  It allows you to use a value computed at
run-time to be used in the macro expansion of the subsequent
toplevel forms in the same file.  It doesn't work in precompilation,
since macro expansion precedes execution.  This limitation
includes procedure definitions.   For example:
+
[source,scheme]
----
(define (quote-it x) `(quote ,x))

(define-syntax quote-self
  (er-macro-transformer
    (^[f r c] (quote-it f))))

(define (test)
  (quote-self a b c))
----
+
This works by loading the source, but won't work in precompilation,
since the expansion of `quote-self` requires to call `quote-it`,
which is only available at run-time.
+
You need to split the file, one contains the values the macro
expander depends, and another contains the forms that uses the
macro.  The macro definition itself can be in either file.
(If the macro expander does not depend on the runtime value
computed in the same file, you can put the macro definition
and macro use in the same file.)

3. The `cond-expand` macro is handy to switch architecture-dependent
code.  Since it is a macro, it's expanded at the precompile time,
so the features are selected based on the platform you're
precompiling, not the platform you actually run the program on.
That means you can't simply precompile a Scheme file and carry over
the generated C file to other platform, if the Scheme file needs
to select architecture-dependent part with cond-expand.  C code
itself is portable, but the unportable selection is to be done
before C file is created.
+
If you need to distribute generated C code and you need to use
`cond-expand`, just extract the part that uses `cond-expand` into
a Scheme file and keep it from being precompiled.


== The first step

Let's begin with the simplest case.  Precompiling a single Scheme file.

Suppose we have `greet.scm`:

[source,scheme]
---------------------------------
(define-module greet
  (export greeting))
(select-module greet)

(define *message* '("Hello, " ".  How are you?"))

(define (greeting who)
  #"~(car *message*)~|who|~(cadr *message*)")
---------------------------------

Run `tools/precomp` subcommand as follows.  The `-e` option is important -- that
tells `precomp` to create a C file suitable for Gauche extension module
on its own.

[source,console]
----
$ gosh tools/precomp -e greet.scm
----

Now you see two files are generated: `greet.c` and `greet.sci`

[source,console]
----
$ ls
greet.c  greet.sci  greet.scm
----

The `.sci` suffix stands for **SC**heme **I**nterface, which contains a small
Scheme code fragment for module definitions, plus forms that can't be
compiled to C, and the call to dynamic-load that loads DSO file:

[source,console]
----
$ cat greet.sci
;; generated automatically.  DO NOT EDIT
#!no-fold-case
(define-module greet (export greeting))
(select-module greet)
(dynamic-load "greet")
----

Precompiled DSOs are always loaded via its interface file -- it resolves
module dependency.  That means you have to distribute the interface
file along the DSO.  You don't usually see `+*.sci+`, though, since
`gauche-install` command can rename `+*.sci+` to `+*.scm+` while it copies
the file (with `-C` option).  We use `+*.sci+` suffix only because we
want to be able to generate the file in the same directory as
the source `+*.scm+` file.

Next, you can compile `greet.c` to generate greet.so (or `greet.dylib`
or `greet.dll`, depending on your platform):

[source,console]
----
$ gauche-package compile greet greet.c
$ ls
greet.c  greet.o  greet.sci  greet.scm  greet.so
----

`gauche-package compile` command knows which C compiler and flags
are used when Gauche is compiled, where the Gauche header files are,
etc.  You have to compile the `+*.c+` file with the same compiler and
same `CFLAGS`, otherwise Gauche can't safely dlopen the resulting DSO.
You can give extra flags to the `gauche-package compile`; just type
this command with no arguments and it tells the usage.

Now, let's try the precompiled module.  We give `-I.` option to `gosh`
so that it can find `greet.sci` and `greet.so`.  We also give `-fload-verbose`
option to check if it really loads the precompiled file, not the
source file:

[source,console]
----
$ gosh -I. -fload-verbose
;;Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/interactive.scm...
;;  Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/common-macros.scm...
;;  Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/defvalues.scm...
gosh> (use greet)
;;Loading ./greet.sci...
;;  Dynamically Loading ./greet.so...
gosh> (greeting "John")
"Hello, John.  How are you?"
----

Woohoo!  Note that `+*.sci+` takes precedence over `+*.scm+`, so even we have
both `greet.sci` and `greet.scm`, gosh loads `greet.sci` (and subsequently,
`greet.so`).

In practice, you have to install `+*.sci+` file and `+*.so+` file to
appropriate places so that other's code can find them.  The standard
places can be queried to with `gauche-config` command:

[source,console]
----
$ gauche-config --sitelibdir
/usr/share/gauche-0.9/site/lib
$ gauche-config --sitearchdir
/usr/lib/gauche-0.9/site/x86_64-pc-linux-gnu
----

You can use gauche-install to copy the files:

[source,console]
----
$ gauche-install -T `gauche-config --sitelibdir` -C greet.sci
$ gauche-install -T `gauche-config --sitearchdir` greet.so
----

The `-C` option renames `greet.sci` to `greet.scm` in the target directory.

(If you use the template makefile generated by `gauche-package generate`,
it has the skeleton to set up the installation destination.)


== Multiple source files

The previous section shows converting one Scheme source into one DSO.
However, usually you want to make one DSO per library, which may
contain lots of source files.

Suppose you have the following source code structure, and `foo.scm`
uses sub files as `foo.boo` and `foo.woo`.

----
foo.scm
foo/boo.scm
foo/woo.scm
----

You have to generate C file for each Scheme source, then compile and
link all together.

[source,console]
----
$ gosh tools/precomp -I. -e foo.scm foo/boo.scm foo/woo.scm
$ ls
foo/  foo--boo.c  foo--woo.c  foo.c  foo.sci  foo.scm
$ ls foo
boo.sci  boo.scm  woo.sci  woo.scm
----

Note the `-I.` option, which allows `precomp` to find
`foo/util1.scm` and `foo/util2.scm` referred from `foo.scm`.

You see C files are generated in the current directory, while
SCI files are on the side of each file under the subdirectory.

Now you can compile those C files into a single DSO:

[source,console]
----
$ gauche-package compile foo foo.c foo--boo.c foo--woo.c
$ ls
foo/        foo--boo.o  foo--woo.o  foo.o    foo.scm
foo--boo.c  foo--woo.c  foo.c       foo.sci  foo.so
----

You still have an SCI file per each Scheme source, but there's only
one DSO and that can be loaded at once.


== But I don't want a bunch of SCI files!

The reason we have one `+*.sci+` file per one `+*.scm+` file is to guarantee
the consitent behavior between source form and precompiled form.
With keeping the interface file in the same relative path as the source,
we can guarantee the library user can say not only `(use foo)`,
but also `(use foo.boo)` and `(use foo.woo)`.  The latter two still
work, since we have `foo/boo.sci` and `foo/woo.sci`, with appropriate
initialization in it.

However, if you have hundreds of source files and you *know*
the external user will only use the toplevel module and not
directly `use` the submodules, then you can consolidate those
SCI files into single one.  Just give `--single-interface`
option to precomp.

[source,console]
----
$ gosh tools/precomp -I. -e --single-interface foo.scm foo/boo.scm foo/woo.scm
$ ls
foo/  foo--boo.c  foo--woo.c  foo.c  foo.sci  foo.scm
$ ls foo
boo.scm  woo.scm
----

Having this in `foo.sci`, you only need it and `foo.so` for the library
to work:

[source,console]
----
$ gosh -I. -fload-verbose
;;Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/interactive.scm...
;;  Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/common-macros.scm...
;;  Loading /usr/share/gauche-0.9/0.9.5/lib/gauche/defvalues.scm...
gosh> (use foo)
;;Loading ./foo.sci...
;;  Dynamically Loading ./foo.so...
----
