8.1.1 Library directory - data containers

Generic container operations

Some data containers have similar properties; for example, lists, vectors and hash tables can be seen as a collection of data. So it is handy to have generic operators, such as applying a procedure to all the elements.

Gauche provides such mechanism to a certain degree, mainly using its object system.

• Collection - Generic functions applicable for unordered set of values. See gauche.collection - Collection framework.
• Sequence - Generic functions applicable for ordered set of values. See gauche.sequence - Sequence framework.
• Dictionary - Generic functions to handle dictionary, that is, a mapping from keys to values. See gauche.dictionary - Dictionary framework.
• Relation - Generic functions to handle relations (in a sense of Codd’s definition). See util.relation - Relation framework.
• Comprehension - This is a collection of macros very handy to construct and traverse collections/sequences in concise code. See srfi.42 - Eager comprehensions.

Container implementations

• List - the universal data structure. You want to check Pairs and lists and scheme.list - R7RS lists
• Vector - a one-dimensional array of arbitrary Scheme values. See Vectors and scheme.vector - R7RS vectors. If you need a wide range of index, but the actual data is sparse, you might want to look at Sparse vectors.
• Uniform vector - a special kind of vectors that can hold limited types of values (e.g. integers representable in 8bits). It tends to be used in performance sensitive applications, such as graphics. See Uniform vectors.
• Array - multi-dimensional arrays that can hold arbitrary Scheme values. See gauche.array - Arrays.
• Uniform array - multi-dimensional arrays that can hold limited types of values. This is also supported by gauche.array - Arrays.
• String - a sequence of characters. See Strings and srfi.13 - String library. Gauche handles multibyte strings— see Multibyte strings for the details.
• Character set - a set of characters. See Character Sets and scheme.charset - R7RS character sets.
• Hash table - hash tables. See Hashtables. For very large hash tables (millions of entries), Sparse tables may provide better memory footprint.
• Balanced tree - If you need to order keys in a dictionary, you can use treemaps. See Treemaps.
• Immutable map - Sometimes immutable dictionary is handy. Internally it implements a functional balanced tree. See data.imap - Immutable map.
• Queue - Both fast and thread-safe queues are provided in data.queue - Queue. Thread-safe queues can also be used as synchronized messaging channel.
• Heap - See data.heap - Heap.
• Ring buffer - Space-efficient ring buffer. See data.ring-buffer - Ring buffer.
• Cache - Various cache algorithm implementations. See data.cache - Cache.
• Record - a simple data structure. Although Gauche’s object system can be used to define arbitrary data structures, you might want to look at gauche.record - Record types and util.record - SLIB-compatible record type, for they are more portable and potentially more efficient.
• Stream - you can implement cool lazy algorithms with it. See util.stream - Stream library.
• Trie - Another tree structure for efficient common-prefix search. See data.trie - Trie.
• Database interface - dbm interface can be used as a persistent hash table; see dbm - Generic DBM interface. For generic RDBMS interface, see dbi - Database independent access layer.

8.1.2 Library directory - string and character

Basic string operations are covered in Strings and srfi.13 - String library. A string is also a sequence of characters, so you can apply methods in gauche.collection - Collection framework and gauche.sequence - Sequence framework.

Character and character set operations are covered in Characters, Character Sets, and scheme.charset - R7RS character sets.

If you scan or build strings sequentially, do not use index access. String ports (see String ports) provides more efficient, and elegant way.

You can use regular expressions to search and extract character sequences from strings; see Regular expressions.

If you need to deal with low-level (i.e. byte-level) representation of strings, Uniform vectors has some tools to convert strings and byte vectors back and forth.

Are you dealing with a structure higher than a mere sequence of characters? Then take a look at text.* modules. text.parse - Parsing input stream has some basic scanners. text.tr - Transliterate characters implements a feature similar to Unix’s tr(1). You can take diff of two texts; see text.diff - Calculate difference of text streams. And if you want to construct large text from string fragments, do not use string-append—see text.tree - Lazy text construction.

Last but not least, Gauche has support of various character encoding schemes. See gauche.charconv - Character Code Conversion for the basic utilities. Most higher-level functions such as open-input-file can take :encoding keyword argument to perform character conversion implicitly. Also see Multibyte scripts if you write Scheme program in non-ASCII characters. If you want to process Gauche source code which may contain "encoding" magic comment, see Coding-aware ports. Gauche also has GNU gettext compatible module (text.gettext - Localized messages) if you need localization.

8.1.3 Library directory - data exchange

Most useful programs need to communicate with outside world (other programs or humans). That involves reading the external data into your program understanding whatever format the data is in, and also writing the data in the format the others can understand.

Lots of network-related external formats are defined in RFC, and there are corresponding rfc.* module that handle some of them. See rfc.822 - RFC822 message parsing, for example, to handle the pervasive RFC2822 message format. Or, JSON can be handled by rfc.json - JSON parsing and construction.

When you exchange table-formatted data, one of the easiest way may be the plain text, one row per line, and columns are separated by some specific characters (e.g. comma). See text.csv - CSV tables for basic parser/writer for them.

Oh, and nowadays every business user wants XML, right? You know they are just S-expressions with extra redundancy and pointy parentheses. So why don’t you read XML as if they’re S-exprs, process them with familiar cars and cdrs and maps, then write them out with extra redundancy and pointy parens? Module sxml.ssax (sxml.ssax - Functional XML parser) implements SAX XML parser, with which you can parse XML and process them on the fly, or convert it to SXML, S-expression XML. You can query SXML using SXPath, an XPath counterparts of S-expression (sxml.sxpath - SXML query language). You can output all kinds of XML and HTML using the SXML serializer (sxml.serializer - Serializing XML and HTML from SXML).

(But you know most web services nowadays also talks JSON, and that’s much lighter and handier than XML. See rfc.json - JSON parsing and construction).

It is planned that various file format handling routines would be available as file.* modules, though we have none ready yet. If you plan to write one, please go ahead and let us know!

8.1.4 Library directory - files

Files and directories. Roughly speaking, there are two places you want to look at.

Filesystems, in the core, has routines close to the underlying OS provides. If you have experience with Unix system programming you’ll find familiar function names there. The fcntl functionality is splitted to gauche.fcntl (gauche.fcntl - Low-level file operations), FYI.

Also you definitely want to look at file.util (file.util - Filesystem utilities), which implements higher-level routines on top of system-level ones.

8.1.5 Library directory - processes and threads

Process-related routines also come in two levels.

The gauche.process module provides high-level routines (gauche.process - High-level process interface); you can pipe the data into and out of child processes easily, for example.

Gauche core provides the primitive fork and exec interface as well as the convenient system call (see Process management). Use them when you want a precise control over what you’re doing.

Gauche has preemptive threads on most Unix platforms including OSX. Check out gauche.threads - Threads for the basic thread support, including primitive mutexes. The data.queue module (see data.queue - Queue) provides thread-safe queue that can also be handy for synchronization. Thread pool is available in control.thread-pool (see control.thread-pool - Thread pools).

8.1.6 Library directory - networking

We have multi-layer abstraction here. At the bottom, we have APIs corresponding to socket-level system calls. In the middle, a convenience library that automates host name lookups, connection and shutdown, etc. On top of them we have several modules that handles specific protocols (e.g. http).

The gauche.net module (gauche.net - Networking) provides the bottom and middle layer. For the top layer, look for rfc.* modules, e.g. rfc.http (rfc.http - HTTP client). More protocol support is coming (there are rfc.ftp and rfc.imap4 written by users, which are waiting for being integrated into Gauche—maybe in next release).

There’s a plan of even higher level of libraries, under the name net.*, which will abstract more than one network protocols. The planned ones include sending emails, or universal resource access by uri. Code contributions are welcome.

8.1.7 Library directory - input and output

8.1.8 Library directory - time

8.1.9 Library directory - bits and bytes

Binary I/O

As the bottom level, Gauche includes primitive byte I/O (read-byte, write-byte) as well as block I/O (read-uvector, read-uvector!, write-uvector) in its core. (See Reading data, Output, and Uvector block I/O).

As the middle level, the module binary.io (binary.io - Binary I/O) has routines to retrieve specific datatype with optional endian specification.

And as the top level, the module binary.pack (binary.pack - Packing binary data) allows packing and unpacking structured binary data, a la Perl’s pack/unpack.

Bit manipulation

Gauche core provides basic bitshift and mask operations (see Basic bitwise operations). SRFI-151 has comprehensive bitwise operations (see scheme.bitwise - R7RS bitwise operations).