The packet Reference Manual

Table of Contents

The packet Reference Manual

This is the packet Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Wed Mar 25 18:51:19 2020 GMT+0.

1 Introduction

PACKET

PACKET is a simple library for defining C-style structures and packing/unpacking them into Common Lisp arrays.

It is typically used for handling UDP packets, but could also be used for unpacking e.g. mmap'ed data files.

Packet type definitions

Packet comes with some pre-defined intrinsic data types:

• 8, 16, 32 and 64 bit signed/unsigned integers (:uint8, :uint16, :uint32, :uint64 ...)
• 32/64-bit floats (encoding/decoding provided by IEEE-FLOATS package)
• characters and wide characters
• arrays of primitives, such as strings

Users define their packet type and associated CLOS class using:

(defpacket name
  ((slot-name slot-type &rest slot-options)
   ...)
  &rest options)

This expands to a defclass for the structure and code for defining the new packet type.

The slot-type can either be a symbol referring to a previously defined type (either primitive or a previous defpacket) or it can be a form (slot-type length) which means an array of length of slot-type objects.

The slot-options are passed into the defclass slot specifier.

Options can include:

(:packing <integer packing>) ;; sets the packing width for the slots
(:size <integer size>)       ;; sets the total packet buffer size

All other options are passed as options to defclass.

Usage

Make a packet buffer using:

(pack object type) 

This returns the buffer filled in with information.

Extract an object from a buffer using:

(unpack buffer type)

Utilities

Along with the serialization functions, there are also several other utilities which provide commonly required functionality:

• Bitwise flags:

(defflags *example-flags*
  ((:myflag1 0 "My flag1")
   (:myflag2 1 "My flag2")))
(pack-flags '(:myflag1 :myflag2) *example-flags*)

• Enums:

(defenum *example-enum*
  ((:a 0 "A")
   (:b 1 "B")))
(enum :a *example-enum*)

• Hexdump output:

(hd #(1 2 3 4))

Both the flags and enums expand to eval-when forms so that they can be computed at compile time e.g. like:

(defun test ()
  (list #.(enum :a *example-enum*)))

Example

We are trying to send packets to a UDP server (written in C) expecting something like

struct person_s {
  char name[20];
  uint32_t flags;
};

struct people_s {
  uint32_t magic;
  int32_t npeople;
  struct person_s people[20];
  uint32_t flags;
};

We define our packet type using:

(defconstant +magic-number+ #x1E2B3A4D)

(defpacket person-s
  ((name (:string 20) :initform "" :initarg :name)
   (flags :uint32 :initform 0 :initarg :flags))
  (:documentation "Person structure."))

(defpacket people-s 
  ((magic :uint32 :initform +magic-number+)
   (npeople :int32 :initform 0 :initarg :npeople)
   (people (person-s 20) :initform nil :initarg :people)
   (flags :uint32 :initform 0 :initarg :flags))
  (:documentation "Collection of people."))

;; pack an instance into a buffer
(pack (make-instance 'people-s 
                     :npeople 2
                     :people (list (make-instance 'person-s :name "frank" :flags 123)
		                   (make-instance 'person-s :name "james" :flags 321))
                     :flags 456)
      'people-s)
-> #(77 58 43 30 2 0 0 0 102 114 97 110 107 ... )

;; unpack a buffer into an instance
(unpack buffer 'people-s)
-> #<PEOPLE-S {24CB5061}>

See example.lisp for more simple usages.

TODO

• Unions?
• Convert pack-object into a generic function

Frank James March 2014

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 packet

Author

Frank James <frank.a.james@gmail.com>

License

BSD

Description

Simple binary serialization library.

Dependency

ieee-floats

Source

packet.asd (file)

Components

• package.lisp (file)
• utils.lisp (file)
• packet.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 packet.asd

Location

packet.asd

Systems

packet (system)

3.1.2 packet/package.lisp

Parent

packet (system)

Location

package.lisp

Packages

packet

3.1.3 packet/utils.lisp

Dependency

package.lisp (file)

Parent

packet (system)

Location

utils.lisp

Exported Definitions

• defenum (macro)
• defflags (macro)
• enum (function)
• enum-id (function)
• enum-p (function)
• flag-p (function)
• hd (function)
• pack-flags (function)
• pad (function)
• pad* (function)
• subseq* (function)
• unpack-flags (function)
• usb8 (function)
• usb8* (function)

3.1.4 packet/packet.lisp

Dependency

utils.lisp (file)

Parent

packet (system)

Location

packet.lisp

Exported Definitions

• defpacket (macro)
• defpacket* (macro)
• list-types (function)
• pack (function)
• type-size (function)
• unpack (function)

Internal Definitions

• %define-alias (function)
• %define-type (function)
• *default-packing* (special variable)
• *type-definitions* (special variable)
• bytes (function)
• compute-real-slots (function)
• define-alias (macro)
• define-type (macro)
• get-type (function)
• merge-bytes (function)
• pack-array (function)
• pack-bytes (function)
• pack-object (function)
• pack-string (function)
• pack-type (function)
• pack-wstring (function)
• round-offset (function)
• type-packer (function)
• type-unpacker (function)
• unpack-array (function)
• unpack-bytes (function)
• unpack-object (function)
• unpack-string (function)
• unpack-type (function)
• unpack-wstring (function)

4 Packages

Packages are listed by definition order.

4.1 packet

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions

• defenum (macro)
• defflags (macro)
• defpacket (macro)
• defpacket* (macro)
• enum (function)
• enum-id (function)
• enum-p (function)
• flag-p (function)
• hd (function)
• list-types (function)
• pack (function)
• pack-flags (function)
• pad (function)
• pad* (function)
• subseq* (function)
• type-size (function)
• unpack (function)
• unpack-flags (function)
• usb8 (function)
• usb8* (function)

Internal Definitions

• %define-alias (function)
• %define-type (function)
• *default-packing* (special variable)
• *type-definitions* (special variable)
• bytes (function)
• compute-real-slots (function)
• define-alias (macro)
• define-type (macro)
• get-type (function)
• merge-bytes (function)
• pack-array (function)
• pack-bytes (function)
• pack-object (function)
• pack-string (function)
• pack-type (function)
• pack-wstring (function)
• round-offset (function)
• type-packer (function)
• type-unpacker (function)
• unpack-array (function)
• unpack-bytes (function)
• unpack-object (function)
• unpack-string (function)
• unpack-type (function)
• unpack-wstring (function)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Macros

Macro: defenum NAME ENUMS

Define a list of enums

Package

packet

Source

utils.lisp (file)

Macro: defflags NAME FLAGS &optional DOCUMENTATION

Macro to define a set of flags

Package

packet

Source

utils.lisp (file)

Macro: defpacket NAME SLOTS &rest OPTIONS

Macro to define the CLOS class and packet type.

SLOTS should be a list of format (SLOT-NAME SLOT-TYPE &rest SLOT-OPTIONS).

SLOT-NAME should be a symbol used for refering to the CLOS class slot.

SLOT-TYPE should be a symbol refering to a previously defined packet type (forward references are forbidden because we need to know the total packet size at definition time).
Use (SLOT-TYPE LENGTH) for arrays.

SLOT-OPTIONS are passed to the corresponding defclass slot specifier.

OPTIONS can contain (:packing PACKING) and (:size SIZE). These are used to
set the packing width and total packet size. If not specified PACKING defaults to *DEFAULT-PACKING*, SIZE defaults to the total size of packet type. If specified, SIZE may not be less than this, but can be more, to allow for unused space at the end of the structure.

All other options are passed to defclass.

Package

packet

Source

packet.lisp (file)

Macro: defpacket* NAME SLOTS &rest OPTIONS

Like DEFPACKET but defines a DEFSTRUCT instead of a DEFCLASS.

Each slot should be of the form (slot-name slot-type &optional initial-value).

Initial value defaults to 0.

Package

packet

Source

packet.lisp (file)

5.1.2 Functions

Function: enum ENUM ENUMS

Package

packet

Source

utils.lisp (file)

Function: enum-id NUMBER ENUMS

Package

packet

Source

utils.lisp (file)

Function: enum-p NUMBER ENUM ENUMS

Package

packet

Source

utils.lisp (file)

Function: flag-p NUMBER FLAG-NAME FLAGS

Package

packet

Source

utils.lisp (file)

Function: hd DATA

Hexdump output

Package

packet

Source

utils.lisp (file)

Function: list-types ()

List all defined PACKET types.

Package

packet

Source

packet.lisp (file)

Function: pack OBJECT TYPE

Make a packet buffer from the initial object.

Package

packet

Source

packet.lisp (file)

Function: pack-flags FLAG-NAMES FLAGS

Combine flags

Package

packet

Source

utils.lisp (file)

Function: pad ARRAY LEN

Pad array with zeros if its too short

Package

packet

Source

utils.lisp (file)

Function: pad* ARRAY &optional WIDTH

Pad to a length multiple of WIDTH

Package

packet

Source

utils.lisp (file)

Function: subseq* SEQUENCE START &optional LEN

Subsequence with length

Package

packet

Source

utils.lisp (file)

Function: type-size TYPE

Get the total size in bytes for this type.

Package

packet

Source

packet.lisp (file)

Function: unpack BUFFER TYPE

Unpack a buffer into an object. Returns any extra (unused) bytes as the 2nd value

Package

packet

Source

packet.lisp (file)

Function: unpack-flags NUMBER FLAGS

Split the number into its flags.

Package

packet

Source

utils.lisp (file)

Function: usb8 &rest SEQUENCES

Make an (unsigned byte 8) vector from the sequences

Package

packet

Source

utils.lisp (file)

Function: usb8* &rest NUMBERS

Make an (unsigned-byte 8) vector from the numbers

Package

packet

Source

utils.lisp (file)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *default-packing*

Default packing boundary.

Package

packet

Source

packet.lisp (file)

Special Variable: *type-definitions*

Global table storing type definitions.

Package

packet

Source

packet.lisp (file)

5.2.2 Macros

Macro: define-alias ALIAS NAME

Package

packet

Source

packet.lisp (file)

Macro: define-type NAME ((POBJECT PBUFFER PSTART) &body PBODY) ((UBUFFER USTART) &body UBODY) SIZE

Package

packet

Source

packet.lisp (file)

5.2.3 Functions

Function: %define-alias ALIAS NAME

Define an alias for a type. Allows refering to the same type by a different name.

Package

packet

Source

packet.lisp (file)

Function: %define-type NAME PACK-HANDLER UNPACK-HANDLER SIZE

Intern a new type definition.

PACK-HANDLER is a function with lambda-list (object buffer start)
and should encode the object into the buffer starting at offset START.

UNPACK-HANDLE is a function with lambda-list (buffer start)
and should extract information about the desired object starting at offset START. It should return the object.

SIZE is the total number of bytes this object consumes.

Package

packet

Source

packet.lisp (file)

Function: bytes INTEGER SIZE

Expand an INTEGER into its consituent number of SIZE bytes.

Package

packet

Source

packet.lisp (file)

Function: compute-real-slots SLOTS PACKING &optional SIZE

Compute the offsets of each slot. We use this information in the closures generated by DEFPACKET to pack/unpack the object.

Package

packet

Source

packet.lisp (file)

Function: get-type NAME

Find the type definition.

Package

packet

Source

packet.lisp (file)

Function: merge-bytes BYTES &optional SIGNED

Convert a list of bytes into a signed/unsigned integer

Package

packet

Source

packet.lisp (file)

Function: pack-array ARRAY TYPE BUFFER START

Pack an array of object into the buffer.

Package

packet

Source

packet.lisp (file)

Function: pack-bytes BYTES BUFFER START

Pack a list of bytes into a buffer

Package

packet

Source

packet.lisp (file)

Function: pack-object OBJECT SLOTS BUFFER START

Pack an OBJECT specified by SLOTS into the BUFFER starting at offset START.

Package

packet

Source

packet.lisp (file)

Function: pack-string STRING LENGTH BUFFER START

Pack a string into the buffer.

Package

packet

Source

packet.lisp (file)

Function: pack-type OBJECT TYPE BUFFER START

Pack an object into the buffer.

Package

packet

Source

packet.lisp (file)

Function: pack-wstring STRING LENGTH BUFFER START

Pack a string into the buffer.

Package

packet

Source

packet.lisp (file)

Function: round-offset OFFSET PACKING

Round the offset to the nearest packing boundary

Package

packet

Source

packet.lisp (file)

Function: type-packer TYPE

Get the packing handler for this type.

Package

packet

Source

packet.lisp (file)

Function: type-unpacker TYPE

Get the unpacking handler for this type.

Package

packet

Source

packet.lisp (file)

Function: unpack-array LENGTH TYPE BUFFER START

Extract an array of objects from the buffer.

Package

packet

Source

packet.lisp (file)

Function: unpack-bytes BUFFER START SIZE &optional SIGNED

Get an integer from the buffer.

Package

packet

Source

packet.lisp (file)

Function: unpack-object OBJECT SLOTS BUFFER START

Unpack the SLOTS of OBJECT from BUFFER starting at offset START.

Package

packet

Source

packet.lisp (file)

Function: unpack-string LENGTH BUFFER START

Unpack a string from the buffer.

Package

packet

Source

packet.lisp (file)

Function: unpack-type TYPE BUFFER START

Unpack an objcet from the buffer.

Package

packet

Source

packet.lisp (file)

Function: unpack-wstring LENGTH BUFFER START

Unpack a string from the buffer.

Package

packet

Source

packet.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types