The cl-messagepack Reference Manual

Table of Contents

The cl-messagepack Reference Manual

This is the cl-messagepack Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Fri Jun 26 10:22:41 2020 GMT+0.

1 Introduction

-- markdown --

CL-MESSAGEPACK

A Common Lisp implementation of the MessagePack (http://msgpack.org/) serialization/deserialization format, implemented according to http://wiki.msgpack.org/display/MSGPACK/Format+specification.

Depends on flexi-streams and babel. Floating point values are supported only on SBCL currently.

Status: first draft encoder and decoder implemented, added extensions for some Lisp data types (see below), simple tests.

Extensions to the Message Pack specification

(C0) 'NIL'

This translates to NIL in Lisp directly, but see C2 (False) below, too.

(C2) 'False'

On encoding this can be achieved via :false; when this is encountered during decoding, NIL is returned to Lisp as long *use-false* is kept NIL.

Earlier implementation of CL-MESSAGEPACK

The previous version used the bytes #xC4, #xC5, #xC6, and #xC7 to encode a 'pure' cons (a cons whose CDR is not a cons), symbols, symbols as a number (via a lookup table), and rationals.

As these encodings are no longer allowed by the MSGPACK specification this functionality has been removed; you can use *EXTENDED-TYPES* to achieve similar things, though.

Extended Types

MSGPACK allows for a range of "Extended Types"; these consist of one of the bytes #xC7 to #xC9 resp. #xD4 to #xD8, a one-byte type number, and an array of bytes for the data (which can optionally be interpreted as an integer ID).

A simple use case is eg. to identify pieces of data across a messagepack-RPC channel (like eg. http://github.com/neovim/neovim does):

(defparameter *my-type-list*
  (messagepack:define-extension-types
    '(:numeric
      0
      Buffer
      Window
      Tabpage
      ...)))

(defparameter *my-lookup-table*
  (make-array 10 :adjustable t :initial-element nil))

(let ((messagepack:*extended-types* *my-type-list*))
      (messagepack:*lookup-table* *my-lookup-table*)
  (messagepack:decode-stream stream))

Now receiving an reply with an item of extended type 0 will automatically build an instance of the class BUFFER, and the ID slot will be filled with the received ID, so that passing that instance to another query can be converted into a matching messagepack extended type element.

This provides type-safe communication across this RPC link.

The classes don't have to be defined ahead of time; the call to DEFINE-EXTENSION-TYPES will create their definition if needed.

Please remember that only the id gets transmitted; if you want to get the same object (with same as in EQ), you'll need to make sure that the correct object is looked up again; this is what *LOOKUP-TABLE* above is for. Remember to bind that per RPC-connection to avoid duplicate IDs, and to invalidate it if the remote process changes!

For more advanced usage CL-MESSAGEPACK provides a base class EXTENSION-TYPE that can be used to define classes with more slots:

(defclass type1 (cl-messagepack:extension-type)
   ( slots... ))

Please note that encoding is currently limited to the #xC7 byte, and therefore imposes a 255 byte limit for the byte array.

Testing

Copy the cl-messagepack directory to the local-projects directory of your Quicklisp install, then

(ql:quickload :cl-messagepack)
(asdf:test-system :cl-messagepack)

in a REPL (tested under SBCL and CCL under Linux x64).

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 cl-messagepack

Author

Miron Brezuleanu

License

Simplified BSD License

Description

A Common-Lisp implementation of Message Pack serialization.

Dependencies

• flexi-streams
• babel
• closer-mop

Source

cl-messagepack.asd (file)

Components

• package.lisp (file)
• cl-messagepack.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 cl-messagepack.asd

Location

cl-messagepack.asd

Systems

cl-messagepack (system)

3.1.2 cl-messagepack/package.lisp

Parent

cl-messagepack (system)

Location

package.lisp

Packages

• messagepack
• messagepack-sym

3.1.3 cl-messagepack/cl-messagepack.lisp

Dependency

package.lisp (file)

Parent

cl-messagepack (system)

Location

cl-messagepack.lisp

Exported Definitions

• *decode-bin-as-string* (special variable)
• *decoder-prefers-alists* (special variable)
• *decoder-prefers-lists* (special variable)
• *encode-alist-as-map* (special variable)
• *extended-types* (special variable)
• *int->symbol* (special variable)
• *lookup-table* (special variable)
• *symbol->int* (special variable)
• decode (function)
• decode-stream (function)
• define-extension-types (function)
• encode (function)
• encode-stream (function)
• extension-type (class)
• get-symbol-int-table (function)
• make-lookup-table (function)
• symbol-to-extension-type (function)
• with-symbol-int-table (macro)
• write-hex (function)

Internal Definitions

• *use-false* (special variable)
• alistp (function)
• as-numeric (method)
• (setf as-numeric) (method)
• decode-array (function)
• decode-byte-array (function)
• decode-cons (function)
• decode-map (function)
• decode-rational (function)
• decode-string (function)
• decode-symbol (function)
• decode-symbol-as-number (function)
• decode-with (method)
• (setf decode-with) (method)
• encodable-as-integer (function)
• encode-array (function)
• encode-cons (function)
• encode-each (function)
• encode-float (function)
• encode-hash (function)
• encode-integer (function)
• encode-rational (function)
• encode-raw-bytes (function)
• encode-sequence-length (function)
• encode-string (function)
• encode-symbol (function)
• encode-with (method)
• (setf encode-with) (method)
• extension-type-description (class)
• extension-type-id (method)
• (setf extension-type-id) (method)
• is-byte-array (function)
• load-big-endian (macro)
• lookup-table-find (function)
• lookup-table-insert (function)
• make-hash (function)
• mkstr (function)
• mksymb (function)
• parse-big-endian (function)
• plistp (function)
• pure-cons (function)
• reg-class (method)
• (setf reg-class) (method)
• sb16->ub16 (function)
• sb32->ub32 (function)
• sb64->ub64 (function)
• sb8->ub8 (function)
• sbcl-encode-float (function)
• signed-unsigned-convertors (macro)
• speed-for-size (function)
• store-big-endian (macro)
• try-encode-ext-type (function)
• type-number (method)
• (setf type-number) (method)
• typed-data (function)
• ub16->sb16 (function)
• ub32->sb32 (function)
• ub64->sb64 (function)
• ub8->sb8 (function)

4 Packages

Packages are listed by definition order.

4.1 messagepack

Source

package.lisp (file)

Nickname

mpk

Use List

• messagepack-sym
• common-lisp

Exported Definitions

• *decode-bin-as-string* (special variable)
• *decoder-prefers-alists* (special variable)
• *decoder-prefers-lists* (special variable)
• *encode-alist-as-map* (special variable)
• *extended-types* (special variable)
• *int->symbol* (special variable)
• *lookup-table* (special variable)
• *symbol->int* (special variable)
• decode (function)
• decode-stream (function)
• define-extension-types (function)
• encode (function)
• encode-stream (function)
• extension-type (class)
• get-symbol-int-table (function)
• make-lookup-table (function)
• symbol-to-extension-type (function)
• with-symbol-int-table (macro)
• write-hex (function)

Internal Definitions

• *use-false* (special variable)
• alistp (function)
• as-numeric (generic function)
• as-numeric (method)
• (setf as-numeric) (method)
• (setf as-numeric) (generic function)
• decode-array (function)
• decode-byte-array (function)
• decode-cons (function)
• decode-map (function)
• decode-rational (function)
• decode-string (function)
• decode-symbol (function)
• decode-symbol-as-number (function)
• decode-with (generic function)
• decode-with (method)
• (setf decode-with) (method)
• (setf decode-with) (generic function)
• encodable-as-integer (function)
• encode-array (function)
• encode-cons (function)
• encode-each (function)
• encode-float (function)
• encode-hash (function)
• encode-integer (function)
• encode-rational (function)
• encode-raw-bytes (function)
• encode-sequence-length (function)
• encode-string (function)
• encode-symbol (function)
• encode-with (generic function)
• encode-with (method)
• (setf encode-with) (method)
• (setf encode-with) (generic function)
• extension-type-description (class)
• extension-type-id (generic function)
• extension-type-id (method)
• (setf extension-type-id) (method)
• (setf extension-type-id) (generic function)
• is-byte-array (function)
• load-big-endian (macro)
• lookup-table-find (function)
• lookup-table-insert (function)
• make-hash (function)
• mkstr (function)
• mksymb (function)
• parse-big-endian (function)
• plistp (function)
• pure-cons (function)
• reg-class (generic function)
• reg-class (method)
• (setf reg-class) (method)
• (setf reg-class) (generic function)
• sb16->ub16 (function)
• sb32->ub32 (function)
• sb64->ub64 (function)
• sb8->ub8 (function)
• sbcl-encode-float (function)
• signed-unsigned-convertors (macro)
• speed-for-size (function)
• store-big-endian (macro)
• try-encode-ext-type (function)
• type-number (generic function)
• type-number (method)
• (setf type-number) (method)
• (setf type-number) (generic function)
• typed-data (function)
• ub16->sb16 (function)
• ub32->sb32 (function)
• ub64->sb64 (function)
• ub8->sb8 (function)

4.2 messagepack-sym

Source

package.lisp (file)

Used By List

messagepack

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Special variables

Special Variable: *decode-bin-as-string*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *decoder-prefers-alists*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *decoder-prefers-lists*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *encode-alist-as-map*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *extended-types*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *int->symbol*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *lookup-table*

Package

messagepack

Source

cl-messagepack.lisp (file)

Special Variable: *symbol->int*

Package

messagepack

Source

cl-messagepack.lisp (file)

5.1.2 Macros

Macro: with-symbol-int-table TABLES &body BODY

Package

messagepack

Source

cl-messagepack.lisp (file)

5.1.3 Functions

Function: decode BYTE-ARRAY

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-stream STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: define-extension-types ARGS

This function defines types for the MessagePack extension type system (#xD4 to #xD8, and #xC7 to #xC9), and returns a list of them
that can be bound to *EXTENSION-TYPES*.
128 different types can be available simultaneously at any one time.

This function takes integers, flags, and/or closures as arguments; these get used as items for the next arguments.
* Integers define which type number to use next.
* Flags for decoding:
:BYTE-ARRAY - return the bytes as array. Default.
:NUMERIC - return value in DATA as a number. Only for fixextN. * A symbol associates the current type number to this type;
this type should be derived from MESSAGEPACK-EXT-TYPE, as
to have a correct MESSAGEPACK:ID slot.

Example:
(defvar *my-extension-types*
(define-extension-types :numeric
5 ’buffer ’block
8 ’cursor))
Eg., the type 6 would then return (MAKE-BLOCK ’ID <content>).

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode DATA

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-stream DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: get-symbol-int-table TABLE

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: make-lookup-table ()

Returns something that can be used for *LOOKUP-TABLE*.

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: symbol-to-extension-type NUM SYM DECODE-AS

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: write-hex DATA

Package

messagepack

Source

cl-messagepack.lisp (file)

5.1.4 Classes

Class: extension-type ()

Base type for Ext-Types.

Package

messagepack

Source

cl-messagepack.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• print-object (method)
• extension-type-id (method)
• extension-type-id (method)

Direct slots

Slot: id

Type

(or integer (array (unsigned-byte 8) *))

Initargs

:id

Initform

(error "need an id")

Readers

extension-type-id (generic function)

Writers

(setf extension-type-id) (generic function)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *use-false*

Package

messagepack

Source

cl-messagepack.lisp (file)

5.2.2 Macros

Macro: load-big-endian STREAM BYTE-COUNT

Package

messagepack

Source

cl-messagepack.lisp (file)

Macro: signed-unsigned-convertors SIZE

Package

messagepack

Source

cl-messagepack.lisp (file)

Macro: store-big-endian NUMBER STREAM BYTE-COUNT

Package

messagepack

Source

cl-messagepack.lisp (file)

5.2.3 Functions

Function: alistp L

Alist predicate

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-array LENGTH STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-byte-array LENGTH STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-cons STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-map LENGTH STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-rational STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-string LENGTH STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-symbol STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: decode-symbol-as-number STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encodable-as-integer DATA

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-array DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-cons DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-each DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-float DATA STREAM &optional DROP-PREFIX

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-hash DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-integer DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-rational DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-raw-bytes DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-sequence-length DATA STREAM SHORT-PREFIX SHORT-LENGTH TYPECODE-16 TYPECODE-32

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-string DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: encode-symbol DATA STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: is-byte-array DATA-TYPE

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: lookup-table-find TYPE ID

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: lookup-table-insert TYPE ID OBJ

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: make-hash DATA

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: mkstr &rest ARGS

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: mksymb &rest ARGS

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: parse-big-endian BYTE-ARRAY

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: plistp L

Plist predicate.

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: pure-cons DATA

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: sb16->ub16 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: sb32->ub32 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: sb64->ub64 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: sb8->ub8 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: sbcl-encode-float DATA STREAM &optional DROP-PREFIX

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: speed-for-size SIZE

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: try-encode-ext-type OBJ STREAM

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: typed-data TYPE-NUM BYTES

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: ub16->sb16 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: ub32->sb32 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: ub64->sb64 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Function: ub8->sb8 ()

Package

messagepack

Source

cl-messagepack.lisp (file)

5.2.4 Generic functions

Generic Function: as-numeric OBJECT

Generic Function: (setf as-numeric) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: as-numeric (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf as-numeric) NEW-VALUE (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated writer method

Source

cl-messagepack.lisp (file)

Generic Function: decode-with OBJECT

Generic Function: (setf decode-with) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: decode-with (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf decode-with) NEW-VALUE (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated writer method

Source

cl-messagepack.lisp (file)

Generic Function: encode-with OBJECT

Generic Function: (setf encode-with) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: encode-with (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf encode-with) NEW-VALUE (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated writer method

Source

cl-messagepack.lisp (file)

Generic Function: extension-type-id OBJECT

Generic Function: (setf extension-type-id) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: extension-type-id (EXTENSION-TYPE extension-type)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf extension-type-id) NEW-VALUE (EXTENSION-TYPE extension-type)

automatically generated writer method

Source

cl-messagepack.lisp (file)

Generic Function: reg-class OBJECT

Generic Function: (setf reg-class) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: reg-class (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf reg-class) NEW-VALUE (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated writer method

Source

cl-messagepack.lisp (file)

Generic Function: type-number OBJECT

Generic Function: (setf type-number) NEW-VALUE OBJECT

Package

messagepack

Methods

Method: type-number (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated reader method

Source

cl-messagepack.lisp (file)

Method: (setf type-number) NEW-VALUE (EXTENSION-TYPE-DESCRIPTION extension-type-description)

automatically generated writer method

Source

cl-messagepack.lisp (file)

5.2.5 Classes

Class: extension-type-description ()

Package

messagepack

Source

cl-messagepack.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• print-object (method)
• reg-class (method)
• reg-class (method)
• as-numeric (method)
• as-numeric (method)
• decode-with (method)
• decode-with (method)
• encode-with (method)
• encode-with (method)
• type-number (method)
• type-number (method)

Direct slots

Slot: type-number

Type

(integer 0 127)

Initargs

:type-number

Readers

type-number (generic function)

Writers

(setf type-number) (generic function)

Slot: encode-with

Type

function

Initargs

:encode-with

Readers

encode-with (generic function)

Writers

(setf encode-with) (generic function)

Slot: decode-with

Type

function

Initargs

:decode-with

Readers

decode-with (generic function)

Writers

(setf decode-with) (generic function)

Slot: as-numeric

Type

(member t nil)

Initargs

:as-numeric

Readers

as-numeric (generic function)

Writers

(setf as-numeric) (generic function)

Slot: reg-class

Initargs

:reg-class

Readers

reg-class (generic function)

Writers

(setf reg-class) (generic function)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types