The trivial-mmap Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The trivial-mmap Reference Manual

This is the trivial-mmap Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Apr 19 18:06:07 2021 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

Trivial-MMAP

Trivial-MMAP is a Common Lisp library that aims to provide a high-level, easy-to-use API for working with memory-mapped files.

The library exports the following symbols:

In additions, the following are the exported symbols of the predicate and accessor functions for dealing with the memory-mapped file object returned by the mmap-file function:

Trivial-MMAP is based on code snippets in a blog post by Nicolas Martyanoff.

Getting Started

The following code snippet shows how to use Trivial-MMAP's with-mmap-file and mmap-read-char to read one character at a time from a file containing Twitter social graph (780 MB uncompressed).

CL-USER> (time
          (trivial-mmap:with-mmap-file (mmapped-file "/home/zodmaner/twitter_rv_15066953.net")
                    (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file)
                                   (trivial-mmap:mmapped-file-size mmapped-file)) do
                       (assert (characterp (trivial-mmap:mmap-read-char mmapped-file))))))
Evaluation took:
  3.400 seconds of real time
  3.400000 seconds of total run time (3.360000 user, 0.040000 system)
  100.00% CPU
  8,479,536,815 processor cycles
  65,952 bytes consed

The little toy benchmark above was run on my laptop (an i5 ThinkPad X230 (Ivy Bridge) with 8 GB of RAM and an SSD). For comparison, reading the same file using with-open-file and read-char takes around 14 seconds of real time on the same machine.

Of course, you could avoid the with-mmap-file macro if you want to. The following code snippet is equivalent to the above, but only use the functions provided by Trivial-MMAP:

(time
 (let ((mmapped-file (trivial-mmap:mmap-file "/home/zodmaner/twitter_rv_15066953.net")))
  (unwind-protect
       (loop while (< (trivial-mmap:mmapped-file-offset mmapped-file)
                      (trivial-mmap:mmapped-file-size mmapped-file)) do
         (assert (characterp (trivial-mmap:mmap-read-char mmapped-file))))
    (trivial-mmap:munmap-file mmapped-file))))

Although from my experiences the version that uses the with-mmap-file macro is usually around 10% to 15% faster compared to the version that did not use the macro.

Dependencies

Limitations

Of course, this library is far from being completed. Here is the list of some of the current limitations/missing features:

As always, patches and bug reports are more then welcome!

Author & Maintainer

Smith Dhumbumroong (zodmaner@gmail.com)

License

Public Domain


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 trivial-mmap

Author

Smith Dhumbumroong <zodmaner@gmail.com>

License

Public Domain

Description

A library providing an easy-to-use API for working with memory-mapped files.

Dependencies
Source

trivial-mmap.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 trivial-mmap.asd

Location

trivial-mmap.asd

Systems

trivial-mmap (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 trivial-mmap/package.lisp

Parent

trivial-mmap (system)

Location

package.lisp

Packages

trivial-mmap


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.3 trivial-mmap/low-level-interface.lisp

Dependency

package.lisp (file)

Parent

trivial-mmap (system)

Location

low-level-interface.lisp

Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

3.1.4 trivial-mmap/trivial-mmap.lisp

Dependency

low-level-interface.lisp (file)

Parent

trivial-mmap (system)

Location

trivial-mmap.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 trivial-mmap

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Macros

Macro: with-mmap-file (MMAPPED-FILE FILENAME) &body BODY
Package

trivial-mmap

Source

trivial-mmap.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Functions

Function: mmap-file FILENAME
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmap-read-byte MMAPPED-FILE &optional EOF-VALUE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmap-read-char MMAPPED-FILE &optional EOF-VALUE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-offset INSTANCE
Function: (setf mmapped-file-offset) VALUE INSTANCE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-p OBJECT
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-pointer INSTANCE
Function: (setf mmapped-file-pointer) VALUE INSTANCE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: mmapped-file-size INSTANCE
Function: (setf mmapped-file-size) VALUE INSTANCE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: munmap-file MMAPPED-FILE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)


Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Macros

Macro: %with-mmap-file (POINTER-TO-MMAP-FILE FILE-SIZE FILENAME &key FILE-LENGTH MAPPING-TYPE OFFSET) &body BODY

Uses MMAP-FILE to maps a FILENAME into memory.

POINTER-TO-MMAP-FILE and FILE-SIZE are bound to values that MMAP-FILE returns after we apply it to FILENAME.

Package

trivial-mmap

Source

low-level-interface.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Functions

Function: %mmap-file FILENAME &key FILE-LENGTH MAPPING-TYPE OFFSET

Maps a FILENAME into memory.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %mmap-read-byte POINTER-TO-MMAP-FILE OFFSET

Reads and returns one byte from a memory-mapped file pointed to by the POINTER-TO-MMAP-FILE pointer, offset by OFFSET bytes.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %mmap-read-char POINTER-TO-MMAP-FILE OFFSET

Reads and returns a character from a memory-mapped file pointed to by the POINTER-TO-MMAP-FILE pointer, offset by OFFSET bytes.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: %munmap-file POINTER-TO-MMAP-FILE FILE-SIZE

Removes a mapping at the address (with the range of FILE-SIZE) that the POINTER-TO-MMAP-FILE pointer points to from memory.

Package

trivial-mmap

Source

low-level-interface.lisp (file)

Function: copy-mmapped-file INSTANCE
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Function: make-mmapped-file &key (POINTER POINTER) (SIZE SIZE) (OFFSET OFFSET)
Package

trivial-mmap

Source

trivial-mmap.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Structures

Structure: mmapped-file ()
Package

trivial-mmap

Source

trivial-mmap.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots
Slot: pointer
Initform

(cffi-sys:null-pointer)

Readers

mmapped-file-pointer (function)

Writers

(setf mmapped-file-pointer) (function)

Slot: size
Type

fixnum

Initform

0

Readers

mmapped-file-size (function)

Writers

(setf mmapped-file-size) (function)

Slot: offset
Type

fixnum

Initform

0

Readers

mmapped-file-offset (function)

Writers

(setf mmapped-file-offset) (function)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   T  
Index Entry  Section

F
File, Lisp, trivial-mmap.asd: The trivial-mmap․asd file
File, Lisp, trivial-mmap/low-level-interface.lisp: The trivial-mmap/low-level-interface․lisp file
File, Lisp, trivial-mmap/package.lisp: The trivial-mmap/package․lisp file
File, Lisp, trivial-mmap/trivial-mmap.lisp: The trivial-mmap/trivial-mmap․lisp file

L
Lisp File, trivial-mmap.asd: The trivial-mmap․asd file
Lisp File, trivial-mmap/low-level-interface.lisp: The trivial-mmap/low-level-interface․lisp file
Lisp File, trivial-mmap/package.lisp: The trivial-mmap/package․lisp file
Lisp File, trivial-mmap/trivial-mmap.lisp: The trivial-mmap/trivial-mmap․lisp file

T
trivial-mmap.asd: The trivial-mmap․asd file
trivial-mmap/low-level-interface.lisp: The trivial-mmap/low-level-interface․lisp file
trivial-mmap/package.lisp: The trivial-mmap/package․lisp file
trivial-mmap/trivial-mmap.lisp: The trivial-mmap/trivial-mmap․lisp file

Jump to:   F   L   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   %   (  
C   F   M   W  
Index Entry  Section

%
%mmap-file: Internal functions
%mmap-read-byte: Internal functions
%mmap-read-char: Internal functions
%munmap-file: Internal functions
%with-mmap-file: Internal macros

(
(setf mmapped-file-offset): Exported functions
(setf mmapped-file-pointer): Exported functions
(setf mmapped-file-size): Exported functions

C
copy-mmapped-file: Internal functions

F
Function, %mmap-file: Internal functions
Function, %mmap-read-byte: Internal functions
Function, %mmap-read-char: Internal functions
Function, %munmap-file: Internal functions
Function, (setf mmapped-file-offset): Exported functions
Function, (setf mmapped-file-pointer): Exported functions
Function, (setf mmapped-file-size): Exported functions
Function, copy-mmapped-file: Internal functions
Function, make-mmapped-file: Internal functions
Function, mmap-file: Exported functions
Function, mmap-read-byte: Exported functions
Function, mmap-read-char: Exported functions
Function, mmapped-file-offset: Exported functions
Function, mmapped-file-p: Exported functions
Function, mmapped-file-pointer: Exported functions
Function, mmapped-file-size: Exported functions
Function, munmap-file: Exported functions

M
Macro, %with-mmap-file: Internal macros
Macro, with-mmap-file: Exported macros
make-mmapped-file: Internal functions
mmap-file: Exported functions
mmap-read-byte: Exported functions
mmap-read-char: Exported functions
mmapped-file-offset: Exported functions
mmapped-file-p: Exported functions
mmapped-file-pointer: Exported functions
mmapped-file-size: Exported functions
munmap-file: Exported functions

W
with-mmap-file: Exported macros

Jump to:   %   (  
C   F   M   W  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   O   P   S  
Index Entry  Section

O
offset: Internal structures

P
pointer: Internal structures

S
size: Internal structures
Slot, offset: Internal structures
Slot, pointer: Internal structures
Slot, size: Internal structures

Jump to:   O   P   S  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   M   P   S   T  
Index Entry  Section

M
mmapped-file: Internal structures

P
Package, trivial-mmap: The trivial-mmap package

S
Structure, mmapped-file: Internal structures
System, trivial-mmap: The trivial-mmap system

T
trivial-mmap: The trivial-mmap system
trivial-mmap: The trivial-mmap package

Jump to:   M   P   S   T