The zcdb Reference Manual

Table of Contents

The zcdb Reference Manual

This is the zcdb Reference Manual, version 1.0.4, generated automatically by Declt version 3.0 "Montgomery Scott" on Wed Mar 25 19:32:35 2020 GMT+0.

1 Introduction

The ZCDB project is for reading and writing cdb files from Common
Lisp.

It is available under a BSD-like license. See the file LICENSE.txt for
details.

For full documentation, see http://www.xach.com/lisp/zcdb/ or the
doc/index.html file in this archive.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 zcdb

Author

Zach Beane <xach@xach.com>

License

MIT

Description

Read and write cdb files, as specified in http://cr.yp.to/cdb.html

Version

1.0.4

Source

zcdb.asd (file)

Components

• package.lisp (file)
• cdb.lisp (file)
• reading.lisp (file)
• writing.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 zcdb.asd

Location

zcdb.asd

Systems

zcdb (system)

3.1.2 zcdb/package.lisp

Parent

zcdb (system)

Location

package.lisp

Packages

zcdb

3.1.3 zcdb/cdb.lisp

Dependency

package.lisp (file)

Parent

zcdb (system)

Location

cdb.lisp

Internal Definitions

• +initial-hash-value+ (constant)
• cdb-hash (function)
• make-growable-vector (function)

3.1.4 zcdb/reading.lisp

Dependency

cdb.lisp (file)

Parent

zcdb (system)

Location

reading.lisp

Exported Definitions

• lookup (function)
• map-cdb (function)

Internal Definitions

• lookup-record-at (function)
• make-octet-vector (function)
• read-cdb-u32 (function)
• stream-lookup (function)
• stream-map-cdb (function)
• table-slot-lookup (function)

3.1.5 zcdb/writing.lisp

Dependency

reading.lisp (file)

Parent

zcdb (system)

Location

writing.lisp

Exported Definitions

• add-record (function)
• with-output-to-cdb (macro)

Internal Definitions

• *empty-record-pointer* (special variable)
• *pointer-padding* (special variable)
• bucket-hash-vector (function)
• buckets (method)
• (setf buckets) (method)
• call-with-output-to-cdb (function)
• cdb-writer (class)
• end-of-records-position (method)
• (setf end-of-records-position) (method)
• entries (method)
• (setf entries) (method)
• entry-count (generic function)
• entry-count (method)
• finish-cdb-writer (function)
• hash-table-bucket (class)
• hash-value (method)
• (setf hash-value) (method)
• output (method)
• (setf output) (method)
• record-pointer (class)
• record-position (method)
• (setf record-position) (method)
• slot-count (generic function)
• slot-count (method)
• table-position (method)
• (setf table-position) (method)
• write-bucket-hash-table (function)
• write-cdb-u32 (function)
• write-hash-tables (function)
• write-pointers (function)

4 Packages

Packages are listed by definition order.

4.1 zcdb

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• add-record (function)
• lookup (function)
• map-cdb (function)
• with-output-to-cdb (macro)

Internal Definitions

• *empty-record-pointer* (special variable)
• *pointer-padding* (special variable)
• +initial-hash-value+ (constant)
• bucket-hash-vector (function)
• buckets (generic function)
• buckets (method)
• (setf buckets) (method)
• (setf buckets) (generic function)
• call-with-output-to-cdb (function)
• cdb-hash (function)
• cdb-writer (class)
• end-of-records-position (generic function)
• end-of-records-position (method)
• (setf end-of-records-position) (method)
• (setf end-of-records-position) (generic function)
• entries (generic function)
• entries (method)
• (setf entries) (method)
• (setf entries) (generic function)
• entry-count (generic function)
• entry-count (method)
• finish-cdb-writer (function)
• hash-table-bucket (class)
• hash-value (generic function)
• hash-value (method)
• (setf hash-value) (method)
• (setf hash-value) (generic function)
• lookup-record-at (function)
• make-growable-vector (function)
• make-octet-vector (function)
• output (generic function)
• output (method)
• (setf output) (method)
• (setf output) (generic function)
• read-cdb-u32 (function)
• record-pointer (class)
• record-position (generic function)
• record-position (method)
• (setf record-position) (method)
• (setf record-position) (generic function)
• slot-count (generic function)
• slot-count (method)
• stream-lookup (function)
• stream-map-cdb (function)
• table-position (generic function)
• table-position (method)
• (setf table-position) (method)
• (setf table-position) (generic function)
• table-slot-lookup (function)
• write-bucket-hash-table (function)
• write-cdb-u32 (function)
• write-hash-tables (function)
• write-pointers (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Macros

Macro: with-output-to-cdb (CDB FILE TEMP-FILE) &body BODY

Evaluate BODY with CDB bound to a CDB-WRITER object. The CDB in progress is written to TEMP-FILE, and then when the CDB is successfully written, TEMP-FILE is renamed to FILE. For atomic operation, FILE and TEMP-FILE must be on the same filesystem.

Package

zcdb

Source

writing.lisp (file)

5.1.2 Functions

Function: add-record KEY VALUE CDB-WRITER

Add KEY and VALUE to a cdb file. KEY and VALUE should both be (unsigned-byte 8) vectors.

Package

zcdb

Source

writing.lisp (file)

Function: lookup KEY CDB

Package

zcdb

Source

reading.lisp (file)

Function: map-cdb FUNCTION CDB

Package

zcdb

Source

reading.lisp (file)

5.2 Internal definitions

5.2.1 Constants

Constant: +initial-hash-value+

Package

zcdb

Source

cdb.lisp (file)

5.2.2 Special variables

Special Variable: *empty-record-pointer*

Package

zcdb

Source

writing.lisp (file)

Special Variable: *pointer-padding*

Package

zcdb

Source

writing.lisp (file)

5.2.3 Functions

Function: bucket-hash-vector BUCKET

Create a hash vector for a bucket. A hash vector has 2x the entries of the bucket, and is initialized to an empty record pointer. The high 24 bits of the hash value of a record pointer, mod the size of the vector, is used as a starting slot, and the vector is walked (wrapping at the end) to find the first free slot for positioning each record pointer entry.

Package

zcdb

Source

writing.lisp (file)

Function: call-with-output-to-cdb CDB-PATHNAME TEMP-PATHNAME FUN

Call FUN with one argument, a CDB-WRITER instance to which records can be added with ADD-RECORD.

Package

zcdb

Source

writing.lisp (file)

Function: cdb-hash OCTETS

http://cr.yp.to/cdb/cdb.txt

Package

zcdb

Source

cdb.lisp (file)

Function: finish-cdb-writer CDB-WRITER

Write the trailing hash tables and leading table pointers to the cdb file.

Package

zcdb

Source

writing.lisp (file)

Function: lookup-record-at POSITION KEY STREAM

Package

zcdb

Source

reading.lisp (file)

Function: make-growable-vector &key SIZE ELEMENT-TYPE

Package

zcdb

Source

cdb.lisp (file)

Function: make-octet-vector SIZE

Package

zcdb

Source

reading.lisp (file)

Function: read-cdb-u32 STREAM

Package

zcdb

Source

reading.lisp (file)

Function: stream-lookup KEY STREAM

Package

zcdb

Source

reading.lisp (file)

Function: stream-map-cdb FUNCTION STREAM

Package

zcdb

Source

reading.lisp (file)

Function: table-slot-lookup KEY HASH TABLE-POSITION INITIAL-SLOT SLOT-COUNT STREAM

Package

zcdb

Source

reading.lisp (file)

Function: write-bucket-hash-table BUCKET STREAM

Write BUCKET’s hash table vector to STREAM.

Package

zcdb

Source

writing.lisp (file)

Function: write-cdb-u32 U32 STREAM

Write an (unsigned-byte 32) value to STREAM in little-endian order.

Package

zcdb

Source

writing.lisp (file)

Function: write-hash-tables CDB-WRITER

Write the traililng hash tables to the end of the cdb file. Initializes the position of the buckets in the process.

Package

zcdb

Source

writing.lisp (file)

Function: write-pointers CDB-WRITER

Write the leading hash table pointers to the beginning of the cdb file. Must be called after WRITE-HASH-TABLES, or the positions won’t be available.

Package

zcdb

Source

writing.lisp (file)

5.2.4 Generic functions

Generic Function: buckets OBJECT

Generic Function: (setf buckets) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: buckets (CDB-WRITER cdb-writer)

automatically generated reader method

Source

writing.lisp (file)

Method: (setf buckets) NEW-VALUE (CDB-WRITER cdb-writer)

automatically generated writer method

Source

writing.lisp (file)

Generic Function: end-of-records-position OBJECT

Generic Function: (setf end-of-records-position) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: end-of-records-position (CDB-WRITER cdb-writer)

automatically generated reader method

Source

writing.lisp (file)

Method: (setf end-of-records-position) NEW-VALUE (CDB-WRITER cdb-writer)

automatically generated writer method

Source

writing.lisp (file)

Generic Function: entries OBJECT

Generic Function: (setf entries) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: entries (HASH-TABLE-BUCKET hash-table-bucket)

Method: (setf entries) NEW-VALUE (HASH-TABLE-BUCKET hash-table-bucket)

A vector of record-pointers.

Source

writing.lisp (file)

Generic Function: entry-count OBJECT

Package

zcdb

Source

writing.lisp (file)

Methods

Method: entry-count (OBJECT hash-table-bucket)

Generic Function: hash-value OBJECT

Generic Function: (setf hash-value) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: hash-value (RECORD-POINTER record-pointer)

Method: (setf hash-value) NEW-VALUE (RECORD-POINTER record-pointer)

The hash value of the record key.

Source

writing.lisp (file)

Generic Function: output OBJECT

Generic Function: (setf output) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: output (CDB-WRITER cdb-writer)

automatically generated reader method

Source

writing.lisp (file)

Method: (setf output) NEW-VALUE (CDB-WRITER cdb-writer)

automatically generated writer method

Source

writing.lisp (file)

Generic Function: record-position OBJECT

Generic Function: (setf record-position) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: record-position (RECORD-POINTER record-pointer)

Method: (setf record-position) NEW-VALUE (RECORD-POINTER record-pointer)

The file position at which the record is stored.

Source

writing.lisp (file)

Generic Function: slot-count OBJECT

Package

zcdb

Source

writing.lisp (file)

Methods

Method: slot-count (OBJECT hash-table-bucket)

Generic Function: table-position OBJECT

Generic Function: (setf table-position) NEW-VALUE OBJECT

Package

zcdb

Methods

Method: table-position (HASH-TABLE-BUCKET hash-table-bucket)

Method: (setf table-position) NEW-VALUE (HASH-TABLE-BUCKET hash-table-bucket)

The file position at which this table is (eventually) slotted.

Source

writing.lisp (file)

5.2.5 Classes

Class: cdb-writer ()

Package

zcdb

Source

writing.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• output (method)
• output (method)
• end-of-records-position (method)
• end-of-records-position (method)
• buckets (method)
• buckets (method)

Direct slots

Slot: buckets

Initargs

:buckets

Readers

buckets (generic function)

Writers

(setf buckets) (generic function)

Slot: end-of-records-position

Initargs

:end-of-records-position

Readers

end-of-records-position (generic function)

Writers

(setf end-of-records-position) (generic function)

Slot: output

Initargs

:output

Readers

output (generic function)

Writers

(setf output) (generic function)

Direct Default Initargs

Initarg	Value
:end-of-records-position	2048
:buckets	(map-into (make-array 256) (lambda nil (make-instance (quote zcdb::hash-table-bucket))))

Class: hash-table-bucket ()

During construction of the CDB, record pointers are
accumulated into one of 256 hash table buckets, depending on the low 8 bits of the hash value of the key. At the end of record writing, these buckets are used to write out hash table vectors at the end of the file, and write pointers to the hash table vectors at the start of the file.

Package

zcdb

Source

writing.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• print-object (method)
• slot-count (method)
• entry-count (method)
• entries (method)
• entries (method)
• table-position (method)
• table-position (method)

Direct slots

Slot: table-position

The file position at which this table is (eventually) slotted.

Initargs

:table-position

Readers

table-position (generic function)

Writers

(setf table-position) (generic function)

Slot: entries

A vector of record-pointers.

Initargs

:entries

Readers

entries (generic function)

Writers

(setf entries) (generic function)

Direct Default Initargs

Initarg	Value
:table-position	0
:entries	(zcdb::make-growable-vector)

Class: record-pointer ()

Every key/value record written to a CDB has a
corresponding record pointer, which tracks the key’s hash value and the record’s position in the data file. When all records have been written to the file, these record pointers are organized into hash tables at the end of the cdb file.

Package

zcdb

Source

writing.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• print-object (method)
• record-position (method)
• record-position (method)
• hash-value (method)
• hash-value (method)

Direct slots

Slot: hash-value

The hash value of the record key.

Initargs

:hash-value

Readers

hash-value (generic function)

Writers

(setf hash-value) (generic function)

Slot: record-position

The file position at which the record is stored.

Initargs

:record-position

Readers

record-position (generic function)

Writers

(setf record-position) (generic function)

Direct Default Initargs

Initarg	Value
:hash-value	0
:record-position	0

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types