The marshal Reference Manual

Table of Contents

The marshal Reference Manual

This is the marshal Reference Manual, version 1.3, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 12:31:24 2020 GMT+0.

1 Introduction

cl-marshal

Simple and fast marshalling of Lisp datastructures. Convert any object into a string representation, put it on a stream an revive it from there. Only minimal changes required to make your CLOS objects serializable. Actually you only need to add 1 method per baseclass.

Just to make that clear: project is not dead - the functionality is simply finished and working.

License

MIT License. See included LICENSE file.

Usage

Basic Usage

For this ASDF needs to be installed and has to have access to marshal.asd. Otherwise do a load *.lisp/tests-.lisp.

(require :marshal)

Serialization of simple examples:

$ (ms:marshal (list 1 2 3 "Foo" "Bar" (make-array '(3) :initial-contents '(a b c))))
--> (:PCODE 1
          (:LIST 1 1 2 3 (:SIMPLE-STRING 2 "Foo") (:SIMPLE-STRING 3 "Bar")
          (:ARRAY 4 (3) T (A B C))))

Deserialization:

$ (ms:unmarshal '(:PCODE 1
      (:LIST 1 1 2 3 (:SIMPLE-STRING 2 "Foo") (:SIMPLE-STRING 3 "Bar")
      (:ARRAY 4 (3) T (A B C)))))
--> (1 2 3 "Foo" "Bar" #(A B C))

That means that a

(ms:unmarshal (ms:marshal myobject))

returns a deep clone of myobject.

Advanced Usage

Definition of a class

 (defclass ship ()
   ((name :initform "" :initarg :name :accessor name)
    (dimensions :initform '(:width 0 :length 0) :initarg :dimensions :accessor dimensions)
    (course :initform 0 :initarg :course :accessor course)
    (cruise :initform 0 :initarg :cruise :accessor cruise) ; shall be transient
    (dinghy :initform NIL :initarg :dinghy :accessor dinghy)) ; another ship -> ref
   (:documentation "A democlass. Some 'persistent slots', one transient.
  Some numbers, string, lists and object references."))


(defparameter ark (make-instance 'ship :name "Ark" :course 360
                            :dimensions '(:width 30 :length 90)))

Let's try to serialize this:

$ (ms:marshal ark)
--> (:PCODE 1 NIL)

Actually nothing happens.

Next we define the method class-persistent-slots for this class. The method has to be defined in the package :marshal.

(defmethod ms:class-persistent-slots ((self ship))
  '(name dimensions course dinghy))
--> #<STANDARD-METHOD MARSHAL:CLASS-PERSISTENT-SLOTS (SHIP) {1002B16B31}

Note that the slot cruise is not listed. Therefore it will not be serialized.

$ (ms:marshal ark)
-->  (:PCODE 1
            (:OBJECT 1 SHIP (:SIMPLE-STRING 2 "Ark") (:LIST 3 :WIDTH 30 :LENGTH 90) 360
            (:LIST 4)))

Fine. Try a (ms:unmarshal (ms:marshal ark)) and you will get a clone of the object ark.

Let's define some subclasses (yes, it's Lisp, we use multiple inheritance here).

(defclass sailingship (ship)
  ((sailarea :initform 0 :initarg :sailarea :accessor sailarea))
  )

(defclass motorship (ship)
  ((enginepower :initform 0 :initarg :enginepower :accessor enginepower))
)

(defclass motorsailor (motorship sailingship)
  ()
)

Some instances:

 (defparameter ship2 (make-instance 'sailingship :name "Pinta" :course 270 :cruise 9
			   :dimensions '(:width 7 :length 21) :sailarea 400))
 (defparameter ship3 (make-instance 'motorship :name "Titanic" :course 320 :cruise 21
			   :dimensions '(:width 28 :length 269) :enginepower 51000))
 (defparameter ship4 (make-instance 'motorsailor  :name "Krusenstern" :course 180
			   :cruise 17.4 :dimensions '(:width 12 :length 82)
			   :sailarea 3400 :enginepower 2000))

Let's try

$ (ms:marshal ship4)
--> (:PCODE 1
           (:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
               (:LIST 3 :WIDTH 12 :LENGTH 82) 180 (:LIST 4)))

Note that the slots to be marshalled are determined by the function ms:class-peristant-slots of the baseclass ship.

One last class and an instance. please note the backreference to another ship. That's a circular reference.

(defclass dinghy (ship)
  ((aboard :initform NIL :initarg :aboard :accessor aboard)) ; another ship -> circular ref
)

(defparameter ship5 (make-instance 'dinghy :name "Gig" :course 320 :cruise 5
			:dimensions '(:width 2 :length 6) :aboard ship4))
(setf (dinghy ship4) ship5)

$ (ms:marshal ship4)
--> (:PCODE 1
           (:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
              (:LIST 3 :WIDTH 12 :LENGTH 82) 180
              (:OBJECT 4 DINGHY (:SIMPLE-STRING 5 "Gig") (:LIST 6 :WIDTH 2 :LENGTH 6)
                320 (:LIST 7))))

We see the reference to the dhingy (the "sub ship"), but not the backreference. Simply, because so far the back link aboard is still transient.

(defmethod ms:class-persistent-slots ((self dinghy))
  (append (call-next-method) '(aboard)))

$ (marshall ship4)
--> (:PCODE 1
         (:OBJECT 1 MOTORSAILOR (:SIMPLE-STRING 2 "Krusenstern")
             (:LIST 3 :WIDTH 12 :LENGTH 82) 180
             (:OBJECT 4 DINGHY (:SIMPLE-STRING 5 "Gig") (:LIST 6 :WIDTH 2 :LENGTH 6)
                320 (:LIST 7) (:REFERENCE 1))))

Brilliant! References, circles et. are working regardless these are references from and to lists, objects, hashtables, array etc.

Understanding the Implementation

Everything is in the package :marshal, nickname :ms.

To enable the serialization of a class, you need to specialise the method ms:class-persistent-slots for this class, or one of its baseclasses. This method must reside in the package :marshal! It has to return a list of slotnames. These slots will be serialized.

A call to ms:marshal on an object will generate the string (actually a sexp) representation. It will try to find the method ms:class-persistent-slots to see which slots have to be serialized.

A call to ms:unmarshal with sexp generated by a ms:marshal will revive an object.

You may have different implementations of classes to be serialized and deserialized. For examples different classes on the endpoints of a net work connections. Or simply different classes as time passes between the persistent storage of a serialization and its retrieval. It is important to understand that the classes that are serialized and the one of the object that will be deserialized need to have the same name and need to have the same slotnames as listed in ms:class-persistent-slots.

If you define a method ms:initialize-unmarshalled-instance for your class, then this method will be called in the end of the deserialization process. This gives you the chance to initialize transient slots, that were not serialized, or to do other initialization tricks.

There is a function called 'coding-idiom', that defines the language of the marshalling. The default vocabulary is quite verbose. In case you are going to send the objects through a network, you may want to change that to a shorter set of verbs. Well, I think there are better ways to speed that up, e.g. by adding a nginx proxy with automaic gzip compression in front of your lisp webserver. Anyway, you will find an alternative, shorter implementation in coding-idiom.lisp, it is fairly straight-forward.

Installation

The most simple and recommended way to install cl-marshal is by using Quicklisp. If you installed Quicklisp a simple

(ql:quickload :marshal)

will download the package and actually load it. You only need to do this once per machine. Later a

(require :marshal)

will be enough.

Alternatively you may get the code with:

git clone git://github.com/wlbr/cl-marshal.git

Either you add this to your asdf repository, then you will only need to do a (require :marshal) in your source.

Or, you may put the source in a subdirectory of your project and add the file marshal.asd wih its full path to your own asdf definition.

Or, you may put the source in a subdirectory of your project and load the file "marshal.asd" directly. After that a (asdf:load-system "marshal") should be sufficient.

Or, as a kind of worst case, you simply do a direct (load <file>) of the files package.lisp, coding-idiom.lisp, marshal.lisp, and unmarshal.lisp.

Dependencies

None except for asdf.

xlunit for the unit tests only (the tests are not included in the asdf system).

Testing

Tested with SBCL and CCL. No rocket science required, should run in any environment.

A set of unit tests is included in tests.lisp.

Reporting problems

If you run into any trouble or find bugs, please report them via the Github issue tracker.

History

First written as encode/decode for CLOS objects only during a diploma thesis in '95. Major rework/enhancements during a research project in the end of the '90s. Refactoring in 2011 when revisiting Lisp.

Contributors

Written by Michael Wolber. Major fixes and enhancements by Christoph Oechslein.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 marshal

Author

Michael Wolber <mwolber@gmx.de>

License

MIT

Description

marshal: Simple (de)serialization of Lisp datastructures.

Version

1.3

Source

marshal.asd (file)

Components

• package.lisp (file)
• utils.lisp (file)
• serialization-format.lisp (file)
• coding-idiom.lisp (file)
• marshal.lisp (file)
• unmarshal.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 marshal.asd

Location

marshal.asd

Systems

marshal (system)

3.1.2 marshal/package.lisp

Parent

marshal (system)

Location

package.lisp

Packages

• marshal
• utils
• serialization-format

3.1.3 marshal/utils.lisp

Dependency

package.lisp (file)

Parent

marshal (system)

Location

utils.lisp

Exported Definitions

• circular-list-p (function)
• dotted-list-p (function)
• proper-list-p (function)

3.1.4 marshal/serialization-format.lisp

Dependency

utils.lisp (file)

Parent

marshal (system)

Location

serialization-format.lisp

Exported Definitions

• array-elements-type (function)
• array-sizes (function)
• array-values (function)
• class-slots-values (function)
• data-type (function)
• ht-hash-fn (function)
• ht-rehash-size (function)
• ht-rehash-threshold (function)
• ht-size (function)
• ht-test-fn (function)
• ht-values (function)
• id (function)
• list-values (function)
• object-class-name (function)
• object-package-name (function)
• reference-id (function)
• simple-string-value (function)
• string-adjustable-p (function)
• string-fill-pointer (function)
• string-value (function)

3.1.5 marshal/coding-idiom.lisp

Dependency

serialization-format.lisp (file)

Parent

marshal (system)

Location

coding-idiom.lisp

Exported Definitions

*idiom-table* (special variable)

Internal Definitions

• +reference-placeholder+ (constant)
• coding-idiom (function)

3.1.6 marshal/marshal.lisp

Dependency

coding-idiom.lisp (file)

Parent

marshal (system)

Location

marshal.lisp

Exported Definitions

• class-persistant-slots (generic function)
• class-persistant-slots (method)
• class-persistent-slots (generic function)
• class-persistent-slots (method)
• marshal (generic function)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)

Internal Definitions

• %walk-list (function)
• genkey (generic function)
• genkey (method)
• getvalue (generic function)
• getvalue (method)
• hashtable (method)
• (setf hashtable) (method)
• marshal-simple-string (generic function)
• marshal-simple-string (method)
• marshal-string (function)
• next-key (method)
• (setf next-key) (method)
• persist-hashtable (class)
• setvalue (generic function)
• setvalue (method)

3.1.7 marshal/unmarshal.lisp

Dependency

marshal.lisp (file)

Parent

marshal (system)

Location

unmarshal.lisp

Exported Definitions

• initialize-unmarshalled-instance (generic function)
• initialize-unmarshalled-instance (method)
• unmarshal (generic function)
• unmarshal (method)

Internal Definitions

• make-circle-hash (function)
• second-pass-list (function)
• token-reference-p (function)
• unmarshal-fn (generic function)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)

4 Packages

Packages are listed by definition order.

4.1 marshal

Source

package.lisp (file)

Nickname

ms

Use List

common-lisp

Exported Definitions

• *idiom-table* (special variable)
• class-persistant-slots (generic function)
• class-persistant-slots (method)
• class-persistent-slots (generic function)
• class-persistent-slots (method)
• initialize-unmarshalled-instance (generic function)
• initialize-unmarshalled-instance (method)
• marshal (generic function)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• marshal (method)
• unmarshal (generic function)
• unmarshal (method)

Internal Definitions

• %walk-list (function)
• +reference-placeholder+ (constant)
• coding-idiom (function)
• genkey (generic function)
• genkey (method)
• getvalue (generic function)
• getvalue (method)
• hashtable (generic function)
• hashtable (method)
• (setf hashtable) (method)
• (setf hashtable) (generic function)
• make-circle-hash (function)
• marshal-simple-string (generic function)
• marshal-simple-string (method)
• marshal-string (function)
• next-key (generic function)
• next-key (method)
• (setf next-key) (method)
• (setf next-key) (generic function)
• persist-hashtable (class)
• second-pass-list (function)
• setvalue (generic function)
• setvalue (method)
• token-reference-p (function)
• unmarshal-fn (generic function)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)
• unmarshal-fn (method)

4.2 utils

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• circular-list-p (function)
• dotted-list-p (function)
• proper-list-p (function)

4.3 serialization-format

Source

package.lisp (file)

Nickname

fmt

Use List

common-lisp

Exported Definitions

• array-elements-type (function)
• array-sizes (function)
• array-values (function)
• class-slots-values (function)
• data-type (function)
• ht-hash-fn (function)
• ht-rehash-size (function)
• ht-rehash-threshold (function)
• ht-size (function)
• ht-test-fn (function)
• ht-values (function)
• id (function)
• list-values (function)
• object-class-name (function)
• object-package-name (function)
• reference-id (function)
• simple-string-value (function)
• string-adjustable-p (function)
• string-fill-pointer (function)
• string-value (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Special variables

Special Variable: *idiom-table*

Definition of the vocabulary of the generated serialization. You can increase verbosity or compactness by choosing your own ’language’. key= access code used inside the programs source code
value= generated identifier.

Package

marshal

Source

coding-idiom.lisp (file)

5.1.2 Functions

Function: array-elements-type EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: array-sizes EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: array-values EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: circular-list-p A-LIST

Package

utils

Source

utils.lisp (file)

Function: class-slots-values EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: data-type EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: dotted-list-p L

Package

utils

Source

utils.lisp (file)

Function: ht-hash-fn EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: ht-rehash-size EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: ht-rehash-threshold EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: ht-size EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: ht-test-fn EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: ht-values EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: id EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: list-values EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: object-class-name EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: object-package-name EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: proper-list-p L

Package

utils

Source

utils.lisp (file)

Function: reference-id EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: simple-string-value EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: string-adjustable-p EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: string-fill-pointer EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

Function: string-value EXPR

Package

serialization-format

Source

serialization-format.lisp (file)

5.1.3 Generic functions

Generic Function: class-persistant-slots CLASS

Defines the slots that will be serialized. Has to return list of valid slotnames. If this is a nested list, then the elements of the second level
need to be pairs of slot and accessors.

Package

marshal

Source

marshal.lisp (file)

Methods

Method: class-persistant-slots (CLASS standard-object)

Generic Function: class-persistent-slots CLASS

Defines the slots that will be serialized. Has to return list of valid slotnames. If this is a nested list, then the elements of the second level
need to be pairs of slot and accessors.

Package

marshal

Source

marshal.lisp (file)

Methods

Method: class-persistent-slots (CLASS standard-object)

Generic Function: initialize-unmarshalled-instance OBJECT

Called as the last step of the deserialization of an object. !Must return the object!!

Package

marshal

Source

unmarshal.lisp (file)

Methods

Method: initialize-unmarshalled-instance OBJECT

Called as the last step of the deserialization of an object. !Must return the object!!

Generic Function: marshal THING &optional CIRCLE-HASH

Generates an sexp when called with an object. The sexp can be used to send it over a network or to store it in a database etc.

Package

marshal

Source

marshal.lisp (file)

Methods

Method: marshal (HASH-TABLE hash-table) &optional CIRCLE-HASH

Method: marshal (OBJECT string) &optional CIRCLE-HASH

Method: marshal (ARRAY array) &optional CIRCLE-HASH

Method: marshal (LIST list) &optional CIRCLE-HASH

Method: marshal (OBJECT standard-object) &optional CIRCLE-HASH

Method: marshal THING &optional CIRCLE-HASH around

Method: marshal THING &optional CIRCLE-HASH

Generic Function: unmarshal THING

Returns an object when called with a wellformed marshal sexp.

Package

marshal

Source

unmarshal.lisp (file)

Methods

Method: unmarshal THING

5.2 Internal definitions

5.2.1 Constants

Constant: +reference-placeholder+

Package

marshal

Source

coding-idiom.lisp (file)

5.2.2 Functions

Function: %walk-list SEQUENCE OUTPUT CKEY KEY-IDIOM CIRCLE-HASH

Package

marshal

Source

marshal.lisp (file)

Function: coding-idiom KEY

Definition of the vocabulary of the generated serialization.You can increase verbosity or compactness by choosing your own ’language’. Simply define your own vocabulary and redefine the variable ms:*idiom-table*.

Package

marshal

Source

coding-idiom.lisp (file)

Function: make-circle-hash ()

Package

marshal

Source

unmarshal.lisp (file)

Function: marshal-string OBJECT CIRCLE-HASH

Package

marshal

Source

marshal.lisp (file)

Function: second-pass-list VERSION TOKEN CIRCLE-HASH &optional MAX-DEPTH

Package

marshal

Source

unmarshal.lisp (file)

Function: token-reference-p TOKEN

Package

marshal

Source

unmarshal.lisp (file)

5.2.3 Generic functions

Generic Function: genkey SELF

Package

marshal

Source

marshal.lisp (file)

Methods

Method: genkey (SELF persist-hashtable)

Generic Function: getvalue SELF KEY

Package

marshal

Source

marshal.lisp (file)

Methods

Method: getvalue (SELF persist-hashtable) KEY

Generic Function: hashtable OBJECT

Generic Function: (setf hashtable) NEW-VALUE OBJECT

Package

marshal

Methods

Method: hashtable (PERSIST-HASHTABLE persist-hashtable)

automatically generated reader method

Source

marshal.lisp (file)

Method: (setf hashtable) NEW-VALUE (PERSIST-HASHTABLE persist-hashtable)

automatically generated writer method

Source

marshal.lisp (file)

Generic Function: marshal-simple-string OBJECT CIRCLE-HASH

Package

marshal

Source

marshal.lisp (file)

Methods

Method: marshal-simple-string OBJECT CIRCLE-HASH

Generic Function: next-key OBJECT

Generic Function: (setf next-key) NEW-VALUE OBJECT

Package

marshal

Methods

Method: next-key (PERSIST-HASHTABLE persist-hashtable)

automatically generated reader method

Source

marshal.lisp (file)

Method: (setf next-key) NEW-VALUE (PERSIST-HASHTABLE persist-hashtable)

automatically generated writer method

Source

marshal.lisp (file)

Generic Function: setvalue SELF KEY VALUE

Package

marshal

Source

marshal.lisp (file)

Methods

Method: setvalue (SELF persist-hashtable) KEY VALUE

Generic Function: unmarshal-fn VERSION TYPE TOKEN &optional CIRCLE-HASH

Package

marshal

Source

unmarshal.lisp (file)

Methods

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql string)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql simple-string)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql hash-table)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql array)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql clist)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql dlist)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql list)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql object)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) (TYPE (eql reference)) TOKEN &optional CIRCLE-HASH

Method: unmarshal-fn (VERSION (eql 1)) TYPE TOKEN &optional CIRCLE-HASH around

Method: unmarshal-fn (VERSION (eql 1)) TYPE TOKEN &optional CIRCLE-HASH

5.2.4 Classes

Class: persist-hashtable ()

Package

marshal

Source

marshal.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• setvalue (method)
• getvalue (method)
• genkey (method)
• initialize-instance (method)
• next-key (method)
• next-key (method)
• hashtable (method)
• hashtable (method)

Direct slots

Slot: hashtable

Readers

hashtable (generic function)

Writers

(setf hashtable) (generic function)

Slot: next-key

Initform

0

Readers

next-key (generic function)

Writers

(setf next-key) (generic function)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types