The trivial-mmap Reference Manual

Table of Contents

The trivial-mmap Reference Manual

This is the trivial-mmap Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Apr 19 18:06:07 2021 GMT+0.

1 Introduction

Trivial-MMAP

Trivial-MMAP is a Common Lisp library that aims to provide a high-level, easy-to-use API for working with memory-mapped files.

The library exports the following symbols:

• mmap-file a function that maps a file into memory and returns an object containing a pointer to the memory-mapped file along with other meta data;
• munmap-file a function that removes a memory-mapped file from memory;
• mmap-read-char a helper function that facilitates reading a character from a memory-mapped file;
• mmap-read-byte a helper function for reading one byte from a memory-mapped file;
• with-mmap-file a helper macro that ensures that a memory-mapped file is safely unmapped (using munmap-file function) after we are done with it.

In additions, the following are the exported symbols of the predicate and accessor functions for dealing with the memory-mapped file object returned by the mmap-file function:

• mmapped-file-p a predicate function that returns true if a variable is a memory-mapped file object (and false otherwise);
• mmapped-file-pointer an accessor function that, given a memory-mapped file object, returns a raw pointer address of the file;
• mmapped-file-size an accessor function that, given a memory-mapped file object, returns the size of the file;
• mmapped-file-offset an accessor function that, given a memory-mapped file object, returns the offset into the file.

Trivial-MMAP is based on code snippets in a blog post by Nicolas Martyanoff.

Getting Started

The following code snippet shows how to use Trivial-MMAP's with-mmap-file and mmap-read-char to read one character at a time from a file containing Twitter social graph (780 MB uncompressed).

CL-USER> (time
          (trivial-mmap:with-mmap-file (mmapped-file "/home/zodmaner/twitter_rv_15066953.net")
                    (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file)
                                   (trivial-mmap:mmapped-file-size mmapped-file)) do
                       (assert (characterp (trivial-mmap:mmap-read-char mmapped-file))))))
Evaluation took:
  3.400 seconds of real time
  3.400000 seconds of total run time (3.360000 user, 0.040000 system)
  100.00% CPU
  8,479,536,815 processor cycles
  65,952 bytes consed

The little toy benchmark above was run on my laptop (an i5 ThinkPad X230 (Ivy Bridge) with 8 GB of RAM and an SSD). For comparison, reading the same file using with-open-file and read-char takes around 14 seconds of real time on the same machine.

Of course, you could avoid the with-mmap-file macro if you want to. The following code snippet is equivalent to the above, but only use the functions provided by Trivial-MMAP:

(time
 (let ((mmapped-file (trivial-mmap:mmap-file "/home/zodmaner/twitter_rv_15066953.net")))
  (unwind-protect
       (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file)
                      (trivial-mmap:mmapped-file-size mmapped-file)) do
         (assert (characterp (trivial-mmap:mmap-read-char mmapped-file))))
    (trivial-mmap:munmap-file mmapped-file))))

Although from my experiences the version that uses the with-mmap-file macro is usually around 10% to 15% faster compared to the version that did not use the macro.

Dependencies

• Alexandria
• Osicat

Limitations

Of course, this library is far from being completed. Here is the list of some of the current limitations/missing features:

• Currently the library only supports reading data from memory-mapped files.
• The library only supports Linux. It may work on MacOS, but definitely not on Windows.

As always, patches and bug reports are more then welcome!

Author & Maintainer

Smith Dhumbumroong (zodmaner@gmail.com)

License

Public Domain

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 trivial-mmap

Author

Smith Dhumbumroong <zodmaner@gmail.com>

License

Public Domain

Description

A library providing an easy-to-use API for working with memory-mapped files.

Dependencies

• alexandria
• osicat

Source

trivial-mmap.asd (file)

Components

• package.lisp (file)
• low-level-interface.lisp (file)
• trivial-mmap.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 trivial-mmap.asd

Location

trivial-mmap.asd

Systems

trivial-mmap (system)

3.1.2 trivial-mmap/package.lisp

Parent

trivial-mmap (system)

Location

package.lisp

Packages

trivial-mmap

3.1.3 trivial-mmap/low-level-interface.lisp

Dependency

package.lisp (file)

Parent

trivial-mmap (system)

Location

low-level-interface.lisp

Internal Definitions

• %mmap-file (function)
• %mmap-read-byte (function)
• %mmap-read-char (function)
• %munmap-file (function)
• %with-mmap-file (macro)

3.1.4 trivial-mmap/trivial-mmap.lisp

Dependency

low-level-interface.lisp (file)

Parent

trivial-mmap (system)

Location

trivial-mmap.lisp

Exported Definitions

• mmap-file (function)
• mmap-read-byte (function)
• mmap-read-char (function)
• mmapped-file-offset (function)
• (setf mmapped-file-offset) (function)
• mmapped-file-p (function)
• mmapped-file-pointer (function)
• (setf mmapped-file-pointer) (function)
• mmapped-file-size (function)
• (setf mmapped-file-size) (function)
• munmap-file (function)
• with-mmap-file (macro)

Internal Definitions

• copy-mmapped-file (function)
• make-mmapped-file (function)
• mmapped-file (structure)

4 Packages

Packages are listed by definition order.

4.1 trivial-mmap

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• mmap-file (function)
• mmap-read-byte (function)
• mmap-read-char (function)
• mmapped-file-offset (function)
• (setf mmapped-file-offset) (function)
• mmapped-file-p (function)
• mmapped-file-pointer (function)
• (setf mmapped-file-pointer) (function)
• mmapped-file-size (function)
• (setf mmapped-file-size) (function)
• munmap-file (function)
• with-mmap-file (macro)

Internal Definitions

• %mmap-file (function)
• %mmap-read-byte (function)
• %mmap-read-char (function)
• %munmap-file (function)
• %with-mmap-file (macro)
• copy-mmapped-file (function)
• make-mmapped-file (function)
• mmapped-file (structure)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Macros

Macro: with-mmap-file (MMAPPED-FILE FILENAME) &body BODY

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

5.1.2 Functions

Function: mmap-file FILENAME

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmap-read-byte MMAPPED-FILE &optional EOF-VALUE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmap-read-char MMAPPED-FILE &optional EOF-VALUE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-offset INSTANCE

Function: (setf mmapped-file-offset) VALUE INSTANCE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-p OBJECT

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-pointer INSTANCE

Function: (setf mmapped-file-pointer) VALUE INSTANCE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-size INSTANCE

Function: (setf mmapped-file-size) VALUE INSTANCE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: munmap-file MMAPPED-FILE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

5.2 Internal definitions

5.2.1 Macros

Macro: %with-mmap-file (POINTER-TO-MMAP-FILE FILE-SIZE FILENAME &key FILE-LENGTH MAPPING-TYPE OFFSET) &body BODY

Uses MMAP-FILE to maps a FILENAME into memory.

POINTER-TO-MMAP-FILE and FILE-SIZE are bound to values that MMAP-FILE returns after we apply it to FILENAME.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

5.2.2 Functions

Function: %mmap-file FILENAME &key FILE-LENGTH MAPPING-TYPE OFFSET

Maps a FILENAME into memory.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %mmap-read-byte POINTER-TO-MMAP-FILE OFFSET

Reads and returns one byte from a memory-mapped file pointed to by the POINTER-TO-MMAP-FILE pointer, offset by OFFSET bytes.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %mmap-read-char POINTER-TO-MMAP-FILE OFFSET

Reads and returns a character from a memory-mapped file pointed to by the POINTER-TO-MMAP-FILE pointer, offset by OFFSET bytes.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %munmap-file POINTER-TO-MMAP-FILE FILE-SIZE

Removes a mapping at the address (with the range of FILE-SIZE) that the POINTER-TO-MMAP-FILE pointer points to from memory.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: copy-mmapped-file INSTANCE

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: make-mmapped-file &key (POINTER POINTER) (SIZE SIZE) (OFFSET OFFSET)

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

5.2.3 Structures

Structure: mmapped-file ()

Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots

Slot: pointer

Initform

(cffi-sys:null-pointer)

Readers

mmapped-file-pointer (function)

Writers

(setf mmapped-file-pointer) (function)

Slot: size

Type

fixnum

Initform

0

Readers

mmapped-file-size (function)

Writers

(setf mmapped-file-size) (function)

Slot: offset

Type

fixnum

Initform

0

Readers

mmapped-file-offset (function)

Writers

(setf mmapped-file-offset) (function)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types