The cl-beanstalk Reference Manual

Table of Contents

The cl-beanstalk Reference Manual

This is the cl-beanstalk Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 12:05:02 2020 GMT+0.

1 Introduction

CL-BEANSTALK, an Implementation of the beanstalk v1.4.2 protocol in Common Lisp

CL-BEANSTALK implements the version 1.4.2 of the Beanstalk queuing service protocol. It allows splitting programs into "worker" parts that execute units of work asynchronously, by fetching them from the beanstalk daemon.

Installing CL-BEANSTALK

CL-BEANSTALK requires three CL libraries to run:

1. usocket
2. flexi-streams
3. split-sequence

All of these libraries are available via quicklisp.

Getting cl-beanstalk from quicklisp

On the lisp REPL, with quicklisp loaded, run:

    (ql:quickload :cl-beanstalk)

And you're done!

Getting cl-beanstalk from source

git clone git://github.com/antifuchs/cl-beanstalk.git

Then, add the directory to your asdf system definition directory. After this, asdf:load-system should be able to find and load cl-beanstalk.

Using CL-BEANSTALK

First, all protocol functions reside in the BEANSTALK: package. Their names are the same as in the beanstalk protocol documentation.

Second, there are several error conditions specified in the protocol document, and all of these map to a specific lisp condition. All error conditions are subtypes of BEANSTALK:BEANSTALK-ERROR. There are several such error conditions, for an exhaustive list see the file package.lisp.

Connecting to the queue server and a little example

To start using CL-BEANSTALK, you need to connect to a beanstalk daemon. To do this, it provides functions BEANSTALK:CONNECT and BEANSTALK:DISCONNECT, and the macro BEANSTALK:WITH-BEANSTALK-CONNECTION:

(beanstalk:with-beanstalk-connection (conn "localhost" 11300)
  ;; Put jobs into a specific tube:
  (beanstalk:use conn "URLs-to-fetch")
  ;; Put two pieces of work into the tube, both at priority 100, 
  ;; no delay, with 180 seconds maximum running time:
  (beanstalk:put conn 100 0 180 "http://boinkor.net/")
  (beanstalk:put conn 100 0 180 "http://planet.lisp.org/"))

Getting work from the queue server

Workers need to get their units of work from somewhere, and they need to tell the queue server that they have performed (or failed to perform that work). Here's a simple (meaning, untested) worker implementation that fetches URLs from the tube in the last example and handles them in some way:

(beanstalk:with-beanstalk-connection (conn "localhost" 11300)
	  ;; Get jobs from a specific tube:
	  (beanstalk:watch conn "URLs-to-fetch")
	  (loop 
	    ;; reserve the next job:
	    (multiple-value-bind (url id status) (beanstalk:reserve conn :timeout 1)
              ;; If no new jobs arrived within 1 second, quit:
              (when (eql status :timed-out) (return))
              (handle-response (drakma:http-request url))
              ;; The job is done, remove it from the queue:
              (beanstalk:delete conn id))))

Note that this code can run on several worker processes (indeed, worker machines) at once, and perform several thousands units of work in parallel.

For a complete reference, see the beanstalk protocol documentation.

Conditions and errors

Whenever it makes sense to do so, a function signals a non-error condition to highlight an untypical event: For example, if a job is buried instead of put on the "ready" queue because the server can't reverve enough memory, that results in a condition being signaled via WARN or SIGNAL.

These functions will signal out-of-band information from the queue server:

• BEANSTALK:PUT and BEANSTALK:RELEASE will signal beanstalk:buried-job if the queue server ran out of memory while allocating a bigger priority queue. This (along with the third return value, :buried) indicates that the job was not put on the "ready" queue, but is now buried.
• BEANSTALK:RESERVE will signal beanstalk:deadline-soon indicating that a job previously reserved in this worker process (a job that was neither deleted nor released) job is within 1 second of its allocated time to run deadline. The worker can choose to extend the deadline by applying BEANSTALK:TOUCH to the reserved job ID.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 cl-beanstalk

Description

An interface to the beanstalk queue server

Dependencies

• usocket
• flexi-streams
• split-sequence

Source

cl-beanstalk.asd (file)

Components

• package.lisp (file)
• beanstalk.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 cl-beanstalk.asd

Location

cl-beanstalk.asd

Systems

cl-beanstalk (system)

3.1.2 cl-beanstalk/package.lisp

Parent

cl-beanstalk (system)

Location

package.lisp

Packages

• beanstalk-internal
• beanstalk

3.1.3 cl-beanstalk/beanstalk.lisp

Dependency

package.lisp (file)

Parent

cl-beanstalk (system)

Location

beanstalk.lisp

Exported Definitions

• bury (function)
• connect (function)
• delete (function)
• disconnect (function)
• ignore (function)
• kick (function)
• list-tube-used (function)
• list-tubes (function)
• list-tubes-watched (function)
• peek (function)
• peek-buried (function)
• peek-delayed (function)
• peek-ready (function)
• put (function)
• quit (function)
• release (function)
• reserve (function)
• stats (function)
• stats-job (function)
• stats-tube (function)
• touch (function)
• use (function)
• watch (function)
• with-beanstalk-connection (macro)
• yaml-parsing-failed (condition)

Internal Definitions

• *debug-protocol* (special variable)
• *yaml-external-format* (special variable)
• bad-message-format (condition)
• bad-reply (condition)
• beanstalk-connection (class)
• beanstalk-error (condition)
• beanstalkd-draining (condition)
• beanstalkd-internal-error (condition)
• beanstalkd-out-of-memory (condition)
• buried-job (condition)
• buried-job-id (method)
• command-reply-case (macro)
• converse (function)
• convert-message-out (generic function)
• convert-message-out (method)
• convert-message-out (method)
• deadline-soon (condition)
• document (method)
• expected-crlf (condition)
• length-below-200 (function)
• name-of (method)
• (setf name-of) (method)
• parse-yaml-data (function)
• pri (type)
• proto-debug (macro)
• proto-debug* (function)
• read-data-reply (function)
• read-reply (function)
• reply (method)
• send-command (function)
• socket-of (method)
• (setf socket-of) (method)
• stream-of (method)
• (setf stream-of) (method)
• time-spec (type)
• tube-name (type)
• unknown-command (method)
• unknown-command (condition)

4 Packages

Packages are listed by definition order.

4.1 beanstalk-internal

Source

package.lisp (file)

Use List

common-lisp

Internal Definitions

• *debug-protocol* (special variable)
• *yaml-external-format* (special variable)
• bad-message-format (condition)
• bad-reply (condition)
• beanstalk-connection (class)
• beanstalk-error (condition)
• beanstalkd-draining (condition)
• beanstalkd-internal-error (condition)
• beanstalkd-out-of-memory (condition)
• buried-job (condition)
• buried-job-id (generic function)
• buried-job-id (method)
• command-reply-case (macro)
• converse (function)
• convert-message-out (generic function)
• convert-message-out (method)
• convert-message-out (method)
• deadline-soon (condition)
• document (generic function)
• document (method)
• expected-crlf (condition)
• length-below-200 (function)
• name-of (generic function)
• name-of (method)
• (setf name-of) (method)
• (setf name-of) (generic function)
• parse-yaml-data (function)
• pri (type)
• proto-debug (macro)
• proto-debug* (function)
• read-data-reply (function)
• read-reply (function)
• reply (generic function)
• reply (method)
• send-command (function)
• socket-of (generic function)
• socket-of (method)
• (setf socket-of) (method)
• (setf socket-of) (generic function)
• stream-of (generic function)
• stream-of (method)
• (setf stream-of) (method)
• (setf stream-of) (generic function)
• time-spec (type)
• tube-name (type)
• unknown-command (generic function)
• unknown-command (method)
• unknown-command (condition)

4.2 beanstalk

Source

package.lisp (file)

Exported Definitions

• bury (function)
• connect (function)
• delete (function)
• disconnect (function)
• ignore (function)
• kick (function)
• list-tube-used (function)
• list-tubes (function)
• list-tubes-watched (function)
• peek (function)
• peek-buried (function)
• peek-delayed (function)
• peek-ready (function)
• put (function)
• quit (function)
• release (function)
• reserve (function)
• stats (function)
• stats-job (function)
• stats-tube (function)
• touch (function)
• use (function)
• watch (function)
• with-beanstalk-connection (macro)
• yaml-parsing-failed (condition)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Macros

Macro: with-beanstalk-connection (CONNECTION HOST PORT &rest INITARGS) &body BODY

Open a beanstalk connection to HOST:PORT for as long as control is inside BODY.

Package

beanstalk

Source

beanstalk.lisp (file)

5.1.2 Functions

Function: bury CONNECTION ID PRI

Package

beanstalk

Source

beanstalk.lisp (file)

Function: connect HOST PORT &rest INITARGS

Connect to a beanstalk daemon running on HOST:PORT.

Package

beanstalk

Source

beanstalk.lisp (file)

Function: delete CONNECTION ID

Package

beanstalk

Source

beanstalk.lisp (file)

Function: disconnect CONNECTION

Disconnect the beanstalk connection.

Package

beanstalk

Source

beanstalk.lisp (file)

Function: ignore CONNECTION TUBE

Package

beanstalk

Source

beanstalk.lisp (file)

Function: kick CONNECTION BOUND

Package

beanstalk

Source

beanstalk.lisp (file)

Function: list-tube-used CONNECTION

Package

beanstalk

Source

beanstalk.lisp (file)

Function: list-tubes CONNECTION

Package

beanstalk

Source

beanstalk.lisp (file)

Function: list-tubes-watched CONNECTION

Package

beanstalk

Source

beanstalk.lisp (file)

Function: peek CONNECTION ID &key EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: peek-buried CONNECTION &key EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: peek-delayed CONNECTION &key EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: peek-ready CONNECTION &key EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: put CONNECTION PRI DELAY TTR MESSAGE &key EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: quit CONNECTION

Alias for BEANSTALK:DISCONNECT.

Package

beanstalk

Source

beanstalk.lisp (file)

Function: release CONNECTION ID PRI DELAY

Package

beanstalk

Source

beanstalk.lisp (file)

Function: reserve CONNECTION &key TIMEOUT EXTERNAL-FORMAT

Package

beanstalk

Source

beanstalk.lisp (file)

Function: stats CONNECTION

Package

beanstalk

Source

beanstalk.lisp (file)

Function: stats-job CONNECTION ID

Package

beanstalk

Source

beanstalk.lisp (file)

Function: stats-tube CONNECTION TUBE

Package

beanstalk

Source

beanstalk.lisp (file)

Function: touch CONNECTION ID

Package

beanstalk

Source

beanstalk.lisp (file)

Function: use CONNECTION NAME

Package

beanstalk

Source

beanstalk.lisp (file)

Function: watch CONNECTION TUBE

Package

beanstalk

Source

beanstalk.lisp (file)

5.1.3 Conditions

Condition: yaml-parsing-failed ()

Package

beanstalk

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Direct methods

document (method)

Direct slots

Slot: document

Initargs

:document

Readers

document (generic function)

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *debug-protocol*

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Special Variable: *yaml-external-format*

Package

beanstalk-internal

Source

beanstalk.lisp (file)

5.2.2 Macros

Macro: command-reply-case (REPLY CONNECTION COMMAND DATA-PRODUCER &rest ARGS) &body CLAUSES

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Macro: proto-debug DATUM &rest ARGS

Package

beanstalk-internal

Source

beanstalk.lisp (file)

5.2.3 Functions

Function: converse CONNECTION COMMAND DATA-PRODUCER &rest ARGS

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: length-below-200 STRING

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: parse-yaml-data N-BYTES STREAM STRING-FIELDS

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: proto-debug* DATUM &rest ARGS

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: read-data-reply BYTES-AS-STRING STREAM EXTERNAL-FORMAT

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: read-reply CONNECTION

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Function: send-command CONNECTION NAME &rest ARGS

Package

beanstalk-internal

Source

beanstalk.lisp (file)

5.2.4 Generic functions

Generic Function: buried-job-id CONDITION

Package

beanstalk-internal

Methods

Method: buried-job-id (CONDITION buried-job)

Source

beanstalk.lisp (file)

Generic Function: convert-message-out MESSAGE CONNECTION EXTERNAL-FORMAT

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Methods

Method: convert-message-out (MESSAGE string) CONN EXTERNAL-FORMAT

Method: convert-message-out (MESSAGE vector) CONN EXTERNAL-FORMAT

Generic Function: document CONDITION

Package

beanstalk-internal

Methods

Method: document (CONDITION yaml-parsing-failed)

Source

beanstalk.lisp (file)

Generic Function: name-of OBJECT

Generic Function: (setf name-of) NEW-VALUE OBJECT

Package

beanstalk-internal

Methods

Method: name-of (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated reader method

Source

beanstalk.lisp (file)

Method: (setf name-of) NEW-VALUE (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated writer method

Source

beanstalk.lisp (file)

Generic Function: reply CONDITION

Package

beanstalk-internal

Methods

Method: reply (CONDITION bad-reply)

Source

beanstalk.lisp (file)

Generic Function: socket-of OBJECT

Generic Function: (setf socket-of) NEW-VALUE OBJECT

Package

beanstalk-internal

Methods

Method: socket-of (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated reader method

Source

beanstalk.lisp (file)

Method: (setf socket-of) NEW-VALUE (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated writer method

Source

beanstalk.lisp (file)

Generic Function: stream-of OBJECT

Generic Function: (setf stream-of) NEW-VALUE OBJECT

Package

beanstalk-internal

Methods

Method: stream-of (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated reader method

Source

beanstalk.lisp (file)

Method: (setf stream-of) NEW-VALUE (BEANSTALK-CONNECTION beanstalk-connection)

automatically generated writer method

Source

beanstalk.lisp (file)

Generic Function: unknown-command CONDITION

Package

beanstalk-internal

Methods

Method: unknown-command (CONDITION unknown-command)

Source

beanstalk.lisp (file)

5.2.5 Conditions

Condition: bad-message-format ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Direct subclasses

expected-crlf (condition)

Condition: bad-reply ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

condition (condition)

Direct methods

reply (method)

Direct slots

Slot: reply

Initargs

:reply

Readers

reply (generic function)

Condition: beanstalk-error ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

condition (condition)

Direct subclasses

• beanstalkd-out-of-memory (condition)
• beanstalkd-draining (condition)
• beanstalkd-internal-error (condition)
• bad-message-format (condition)
• unknown-command (condition)
• yaml-parsing-failed (condition)

Condition: beanstalkd-draining ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Condition: beanstalkd-internal-error ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Condition: beanstalkd-out-of-memory ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Condition: buried-job ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

warning (condition)

Direct methods

buried-job-id (method)

Direct slots

Slot: id

Initargs

:id

Readers

buried-job-id (generic function)

Condition: deadline-soon ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

warning (condition)

Condition: expected-crlf ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

bad-message-format (condition)

Condition: unknown-command ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

beanstalk-error (condition)

Direct methods

unknown-command (method)

Direct slots

Slot: unknown-command

Initargs

beanstalk-internal::command

Readers

unknown-command (generic function)

5.2.6 Classes

Class: beanstalk-connection ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• print-object (method)
• name-of (method)
• name-of (method)
• stream-of (method)
• stream-of (method)
• socket-of (method)
• socket-of (method)

Direct slots

Slot: socket

Initargs

:socket

Readers

socket-of (generic function)

Writers

(setf socket-of) (generic function)

Slot: stream

Initargs

:stream

Readers

stream-of (generic function)

Writers

(setf stream-of) (generic function)

Slot: name

Initargs

:name

Readers

name-of (generic function)

Writers

(setf name-of) (generic function)

5.2.7 Types

Type: pri ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Type: time-spec ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Type: tube-name ()

Package

beanstalk-internal

Source

beanstalk.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types