The psychiq Reference Manual

Table of Contents

The psychiq Reference Manual

This is the psychiq Reference Manual, version 0.1.0, generated automatically by Declt version 3.0 "Montgomery Scott" on Wed Nov 04 14:55:57 2020 GMT+0.

1 Introduction

Psychiq

Psychiq provides background job processing for Common Lisp applications inspired by Ruby's Sidekiq.

Warning

This software is still ALPHA quality. The APIs will be likely to change.

Getting Started

Writing a worker

(psy:connect-toplevel :host "localhost" :port 6379)

(defclass my-worker (psy:worker) ())
(defmethod psy:perform ((worker my-worker) &rest args)
  ;; Do something
  )

The worker class is commonly written in an individual ASDF system because it must be shared with clients and servers.

Enqueueing (Client)

;; Enqueueing to "default" queue
(psy:enqueue 'my-worker '("arg1" "arg2"))

;; Enqueueing the job after 300 seconds.
(psy:enqueue-in-sec 300 'my-worker '("arg1" "arg2"))

;; Enqueueing a large number of jobs at a time
;; This is useful if you are pushing tens of thousands of jobs or more
(psy:enqueue-bulk 'my-worker '("arg1" "arg2") '("another" "one") ...)

The arguments must be simple JSON datatypes which can be serialized with Jonathan.

Starting processing (Server)

Psychiq provides a Roswell script for starting processing:

$ psychiq --host localhost --port 6379 --system myapp-workers

$ psychiq -h
Usage: psychiq [option...]

Options:
    -o, --host HOST                 Redis server host (default: localhost)
    -p, --port PORT                 Redis server port (default: 6379)
    -d, --db DBNUM                  Redis server db (default: 0)
    -q, --queue QUEUE[,WEIGHT]      Queues to process with optional weights (several -q's allowed)
    -c, --concurrency INT           Processor threads to use (default: 25)
    -s, --system SYSTEM             ASDF system to load before starting (several -s's allowed)
    -v, --verbose                   Print more verbose output
    -n, --namespace NAMESPACE       Redis namespace (default: "psychiq")
    -V, --version                   Print version
    -h, --help                      Show help

Worker options

(defclass my-worker (psy:worker) ())

;; Specify max retry attempts. (default: 25)
(defmethod psy:worker-max-retries ((worker my-worker))
  1000)

;; Use a named queue to push. (default: "default")
(defmethod psy:worker-queue-name ((worker my-worker))
  "my-worker-queue")

;; Disable jobs going to the dead job queue. (default: T)
(defmethod psy:worker-use-dead-queue-p ((worker my-worker))
  nil)

;; Whether to save any error backtrace in the retry payload. (default: NIL)
(defmethod psy:worker-use-backtrace-p ((worker my-worker))
  t)

Signals

• INT: graceful shutdown, waits for all processors are idle.
• TERM: shutdown immediately

Error Handling

When getting an error while performing a job, Psychiq will add the job to the "retry" queue. Jobs in the "retry" queue will be retried automatically with an exponential backoff. After 25 attempts, Psychiq move the job to the "dead" queue.

Web UI

Since the data structure which Psychiq stores in Redis is compatible with Ruby's Resque/Sidekiq, Sidekiq's Web UI can be used.

# config.ru
require 'sidekiq'

Sidekiq.configure_client do |config|
  config.redis = { :size => 1, url: 'redis://localhost:6379', namespace: 'psychiq' }
end

require 'sidekiq/web'
run Sidekiq::Web

$ rackup config.ru

It will be up at http://127.0.0.1:9292 which allows you to see processes and can retry failed jobs manually.

Requirements

• SBCL (compiled with sb-thread) or Clozure CL
• Redis
• Roswell for command-line launcher script.

Installation

$ ros install psychiq

Author

• Eitaro Fukamachi (e.arrows@gmail.com)

Copyright

Copyright (c) 2015-2017 Eitaro Fukamachi (e.arrows@gmail.com)

License

Licensed under the LLGPL License.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 psychiq

Author

Eitaro Fukamachi

License

LLGPL

Description

Redis-backed job queueing system

Version

0.1.0

Dependencies

• cl-redis
• jonathan
• local-time
• cl-reexport
• bordeaux-threads
• dissect
• vom
• uiop
• alexandria
• sb-posix (for feature sbcl)

Source

psychiq.asd (file)

Components

• src (module)
• src/specials.lisp (file)

3 Modules

Modules are listed depth-first from the system components tree.

3.1 psychiq/src

Dependency

src/specials.lisp (file)

Parent

psychiq (system)

Location

src/

Components

• psychiq.lisp (file)
• client.lisp (file)
• launcher.lisp (file)
• core (module)
• launcher-core (module)
• middleware (module)
• util (module)

3.2 psychiq/src/core

Dependency

util (module)

Parent

src (module)

Location

src/core/

Components

• connection.lisp (file)
• worker.lisp (file)
• queue.lisp (file)
• coder.lisp (file)

3.3 psychiq/src/launcher-core

Dependencies

• core (module)
• middleware (module)
• util (module)

Parent

src (module)

Location

src/launcher/

Components

• processor.lisp (file)
• manager.lisp (file)
• scheduled.lisp (file)
• heartbeat.lisp (file)

3.4 psychiq/src/middleware

Dependencies

• core (module)
• util (module)

Parent

src (module)

Location

src/middleware/

Components

• retry-jobs.lisp (file)
• logging.lisp (file)

3.5 psychiq/src/util

Parent

src (module)

Location

src/util/

Components

• util.lisp (file)
• assoc.lisp (file)
• redis.lisp (file)
• concurrency.lisp (file)

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.

4.1 Lisp

4.1.1 psychiq.asd

Location

psychiq.asd

Systems

psychiq (system)

4.1.2 psychiq/src/psychiq.lisp

Dependencies

• core (module)
• client.lisp (file)

Parent

src (module)

Location

src/psychiq.lisp

Packages

psychiq

4.1.3 psychiq/src/client.lisp

Dependency

core (module)

Parent

src (module)

Location

src/client.lisp

Packages

psychiq.client

Exported Definitions

• all-dead-jobs (function)
• all-queues (function)
• all-retries (function)
• cleanup-processes (function)
• delete-queue (function)
• dequeue (function)
• enqueue (function)
• enqueue-bulk (function)
• enqueue-in-sec (function)
• peek-queue (function)
• queue-empty-p (function)
• queue-length (function)
• retry-length (function)
• slice-queue (function)
• stats (function)

Internal Definitions

decode-objects (function)

4.1.4 psychiq/src/launcher.lisp

Dependencies

• core (module)
• launcher-core (module)

Parent

src (module)

Location

src/launcher.lisp

Packages

psychiq.launcher

Exported Definitions

• kill (function)
• launcher (structure)
• launcher-status (function)
• run (function)
• start (function)
• stop (function)
• wait-for (function)

Internal Definitions

• %make-launcher (function)
• copy-launcher (function)
• launcher-heartbeat (function)
• (setf launcher-heartbeat) (function)
• launcher-manager (function)
• (setf launcher-manager) (function)
• launcher-p (function)
• launcher-scheduled (function)
• (setf launcher-scheduled) (function)
• make-launcher (function)

4.1.5 psychiq/src/core/connection.lisp

Parent

core (module)

Location

src/core/connection.lisp

Packages

psychiq.connection

Exported Definitions

• *connection* (special variable)
• close-connection (generic function)
• close-connection (method)
• connect (function)
• connect-toplevel (function)
• connectedp (generic function)
• connectedp (method)
• connection (class)
• disconnect (function)
• disconnect-toplevel (function)
• ensure-connected (function)
• make-connection (function)
• open-connection (generic function)
• open-connection (method)
• reconnect (function)
• with-connection (macro)

Internal Definitions

• redis-connection (method)
• (setf redis-connection) (method)

4.1.6 psychiq/src/core/worker.lisp

Parent

core (module)

Location

src/core/worker.lisp

Packages

psychiq.worker

Exported Definitions

• decode-job (function)
• encode-job (function)
• perform (generic function)
• perform (method)
• worker (class)
• worker-max-retries (generic function)
• worker-max-retries (method)
• worker-queue-name (generic function)
• worker-queue-name (method)
• worker-use-backtrace-p (generic function)
• worker-use-backtrace-p (method)
• worker-use-dead-queue-p (generic function)
• worker-use-dead-queue-p (method)

4.1.7 psychiq/src/core/queue.lisp

Dependency

coder.lisp (file)

Parent

core (module)

Location

src/core/queue.lisp

Packages

psychiq.queue

Exported Definitions

• dequeue-from-queue (function)
• enqueue-to-queue (function)
• enqueue-to-scheduled-queue (function)

4.1.8 psychiq/src/core/coder.lisp

Parent

core (module)

Location

src/core/coder.lisp

Packages

psychiq.coder

Exported Definitions

• decode-object (function)
• encode-object (function)

4.1.9 psychiq/src/launcher-core/processor.lisp

Parent

launcher-core (module)

Location

src/launcher/processor.lisp

Packages

psychiq.launcher.processor

Exported Definitions

• fetch-job (generic function)
• fetch-job (method)
• finalize (generic function)
• finalize (method)
• kill (generic function)
• kill (method)
• make-processor (function)
• perform-job (generic function)
• perform-job (method)
• process-job (generic function)
• process-job (method)
• processor (structure)
• processor-connection (function)
• (setf processor-connection) (function)
• processor-id (function)
• (setf processor-id) (function)
• processor-manager (function)
• (setf processor-manager) (function)
• processor-processing (function)
• (setf processor-processing) (function)
• processor-status (function)
• (setf processor-status) (function)
• processor-timeout (function)
• (setf processor-timeout) (function)
• run (generic function)
• run (method)
• start (generic function)
• start (method)
• stop (generic function)
• stop (method)
• wait-for (generic function)
• wait-for (method)

Internal Definitions

• %make-processor (function)
• copy-processor (function)
• processor-down (function)
• (setf processor-down) (function)
• processor-p (function)
• processor-queues (function)
• (setf processor-queues) (function)
• processor-thread (function)
• (setf processor-thread) (function)

4.1.10 psychiq/src/launcher-core/manager.lisp

Dependency

processor.lisp (file)

Parent

launcher-core (module)

Location

src/launcher/manager.lisp

Packages

psychiq.launcher.manager

Exported Definitions

• finalize (method)
• kill (method)
• make-manager (function)
• manager (structure)
• manager-children (function)
• (setf manager-children) (function)
• manager-count (function)
• (setf manager-count) (function)
• manager-queues (function)
• (setf manager-queues) (function)
• manager-stat-failed (function)
• (setf manager-stat-failed) (function)
• manager-stat-processed (function)
• (setf manager-stat-processed) (function)
• manager-stopped-p (function)
• (setf manager-stopped-p) (function)
• process-job (method)
• run (method)
• start (method)
• stop (method)
• wait-for (method)

Internal Definitions

• %make-manager (function)
• copy-manager (function)
• make-child-processors (function)
• manager-db (function)
• (setf manager-db) (function)
• manager-host (function)
• (setf manager-host) (function)
• manager-lock (function)
• (setf manager-lock) (function)
• manager-make-processor-fn (function)
• (setf manager-make-processor-fn) (function)
• manager-p (function)
• manager-port (function)
• (setf manager-port) (function)
• processor-died (function)
• processor-stopped (function)

4.1.11 psychiq/src/launcher-core/scheduled.lisp

Parent

launcher-core (module)

Location

src/launcher/scheduled.lisp

Packages

psychiq.launcher.scheduled

Exported Definitions

• kill (function)
• make-scheduled (function)
• scheduled (structure)
• scheduled-status (function)
• (setf scheduled-status) (function)
• scheduled-thread (function)
• (setf scheduled-thread) (function)
• start (function)
• stop (function)

Internal Definitions

• %make-scheduled (function)
• copy-scheduled (function)
• enqueue-jobs (function)
• scaled-poll-interval (function)
• scheduled-connection (function)
• (setf scheduled-connection) (function)
• scheduled-p (function)

4.1.12 psychiq/src/launcher-core/heartbeat.lisp

Dependencies

• manager.lisp (file)
• processor.lisp (file)

Parent

launcher-core (module)

Location

src/launcher/heartbeat.lisp

Packages

psychiq.launcher.heartbeat

Exported Definitions

• heartbeat (function)
• heartbeat (structure)
• kill (function)
• make-heartbeat (function)
• start (function)
• stop (function)

Internal Definitions

• clear-heartbeat (function)
• copy-heartbeat (function)
• heartbeat-db (function)
• (setf heartbeat-db) (function)
• heartbeat-host (function)
• (setf heartbeat-host) (function)
• heartbeat-manager (function)
• (setf heartbeat-manager) (function)
• heartbeat-p (function)
• heartbeat-port (function)
• (setf heartbeat-port) (function)
• heartbeat-stopped-p (function)
• (setf heartbeat-stopped-p) (function)
• heartbeat-thread (function)
• (setf heartbeat-thread) (function)
• run (function)

4.1.13 psychiq/src/middleware/retry-jobs.lisp

Parent

middleware (module)

Location

src/middleware/retry-jobs.lisp

Packages

psychiq.middleware.retry-jobs

Exported Definitions

*psychiq-middleware-retry-jobs* (special variable)

Internal Definitions

• *dead-max-jobs* (special variable)
• *dead-timeout-in-seconds* (special variable)
• attempt-retry (function)
• backtrace (function)
• delay-for (function)
• send-to-morgue (function)

4.1.14 psychiq/src/middleware/logging.lisp

Parent

middleware (module)

Location

src/middleware/logging.lisp

Packages

psychiq.middleware.logging

Exported Definitions

*psychiq-middleware-logging* (special variable)

4.1.15 psychiq/src/util/util.lisp

Dependencies

• assoc.lisp (file)
• redis.lisp (file)
• concurrency.lisp (file)

Parent

util (module)

Location

src/util/util.lisp

Packages

psychiq.util

Exported Definitions

• generate-random-id (function)
• getpid (function)
• machine-identity (function)
• symbol-name-with-package (function)

4.1.16 psychiq/src/util/assoc.lisp

Parent

util (module)

Location

src/util/assoc.lisp

Packages

psychiq.util.assoc

Exported Definitions

• aget (function)
• (setf aget) (function)

4.1.17 psychiq/src/util/redis.lisp

Parent

util (module)

Location

src/util/redis.lisp

Packages

psychiq.util.redis

Exported Definitions

• omit-redis-prefix (function)
• redis-key (function)
• with-redis-transaction (macro)

4.1.18 psychiq/src/util/concurrency.lisp

Parent

util (module)

Location

src/util/concurrency.lisp

Packages

psychiq.util.concurrency

Exported Definitions

• get-value (function)
• (setf get-value) (function)
• make-thread-safe-variable (function)
• reset-value (function)

Internal Definitions

• copy-thread-safe-variable (function)
• thread-safe-variable (structure)
• thread-safe-variable-initial-value (function)
• (setf thread-safe-variable-initial-value) (function)
• thread-safe-variable-lock (function)
• (setf thread-safe-variable-lock) (function)
• thread-safe-variable-p (function)
• thread-safe-variable-value (function)
• (setf thread-safe-variable-value) (function)

4.1.19 psychiq/src/specials.lisp

Parent

psychiq (system)

Location

src/specials.lisp

Packages

psychiq.specials

Exported Definitions

• *default-max-retry-attempts* (special variable)
• *default-queue-name* (special variable)
• *default-redis-host* (special variable)
• *default-redis-port* (special variable)
• *psychiq-namespace* (special variable)

5 Packages

Packages are listed by definition order.

5.1 psychiq

Source

psychiq.lisp (file)

Nickname

psy

Use List

common-lisp

5.2 psychiq.client

Source

client.lisp (file)

Use List

• psychiq.specials
• psychiq.util
• common-lisp

Exported Definitions

• all-dead-jobs (function)
• all-queues (function)
• all-retries (function)
• cleanup-processes (function)
• delete-queue (function)
• dequeue (function)
• enqueue (function)
• enqueue-bulk (function)
• enqueue-in-sec (function)
• peek-queue (function)
• queue-empty-p (function)
• queue-length (function)
• retry-length (function)
• slice-queue (function)
• stats (function)

Internal Definitions

decode-objects (function)

5.3 psychiq.launcher

Source

launcher.lisp (file)

Use List

• psychiq.specials
• common-lisp

Exported Definitions

• kill (function)
• launcher (structure)
• launcher-status (function)
• run (function)
• start (function)
• stop (function)
• wait-for (function)

Internal Definitions

• %make-launcher (function)
• copy-launcher (function)
• launcher-heartbeat (function)
• (setf launcher-heartbeat) (function)
• launcher-manager (function)
• (setf launcher-manager) (function)
• launcher-p (function)
• launcher-scheduled (function)
• (setf launcher-scheduled) (function)
• make-launcher (function)

5.4 psychiq.connection

Source

connection.lisp (file)

Use List

• psychiq.specials
• common-lisp

Exported Definitions

• *connection* (special variable)
• close-connection (generic function)
• close-connection (method)
• connect (function)
• connect-toplevel (function)
• connectedp (generic function)
• connectedp (method)
• connection (class)
• disconnect (function)
• disconnect-toplevel (function)
• ensure-connected (function)
• make-connection (function)
• open-connection (generic function)
• open-connection (method)
• reconnect (function)
• with-connection (macro)

Internal Definitions

• redis-connection (generic function)
• redis-connection (method)
• (setf redis-connection) (method)
• (setf redis-connection) (generic function)

5.5 psychiq.worker

Source

worker.lisp (file)

Use List

• psychiq.specials
• psychiq.util
• common-lisp

Exported Definitions

• decode-job (function)
• encode-job (function)
• perform (generic function)
• perform (method)
• worker (class)
• worker-max-retries (generic function)
• worker-max-retries (method)
• worker-queue-name (generic function)
• worker-queue-name (method)
• worker-use-backtrace-p (generic function)
• worker-use-backtrace-p (method)
• worker-use-dead-queue-p (generic function)
• worker-use-dead-queue-p (method)

5.6 psychiq.queue

Source

queue.lisp (file)

Use List

• psychiq.util
• common-lisp

Exported Definitions

• dequeue-from-queue (function)
• enqueue-to-queue (function)
• enqueue-to-scheduled-queue (function)

5.7 psychiq.coder

Source

coder.lisp (file)

Use List

common-lisp

Exported Definitions

• decode-object (function)
• encode-object (function)

5.8 psychiq.launcher.processor

Source

processor.lisp (file)

Use List

• psychiq.specials
• psychiq.util
• common-lisp

Used By List

psychiq.launcher.manager

Exported Definitions

• fetch-job (generic function)
• fetch-job (method)
• finalize (generic function)
• finalize (method)
• finalize (method)
• kill (generic function)
• kill (method)
• kill (method)
• make-processor (function)
• perform-job (generic function)
• perform-job (method)
• process-job (generic function)
• process-job (method)
• process-job (method)
• processor (structure)
• processor-connection (function)
• (setf processor-connection) (function)
• processor-id (function)
• (setf processor-id) (function)
• processor-manager (function)
• (setf processor-manager) (function)
• processor-processing (function)
• (setf processor-processing) (function)
• processor-status (function)
• (setf processor-status) (function)
• processor-timeout (function)
• (setf processor-timeout) (function)
• run (generic function)
• run (method)
• run (method)
• start (generic function)
• start (method)
• start (method)
• stop (generic function)
• stop (method)
• stop (method)
• wait-for (generic function)
• wait-for (method)
• wait-for (method)

Internal Definitions

• %make-processor (function)
• copy-processor (function)
• processor-down (function)
• (setf processor-down) (function)
• processor-p (function)
• processor-queues (function)
• (setf processor-queues) (function)
• processor-thread (function)
• (setf processor-thread) (function)

5.9 psychiq.launcher.manager

Source

manager.lisp (file)

Use List

• psychiq.specials
• psychiq.launcher.processor
• psychiq.util
• common-lisp

Exported Definitions

• make-manager (function)
• manager (structure)
• manager-children (function)
• (setf manager-children) (function)
• manager-count (function)
• (setf manager-count) (function)
• manager-queues (function)
• (setf manager-queues) (function)
• manager-stat-failed (function)
• (setf manager-stat-failed) (function)
• manager-stat-processed (function)
• (setf manager-stat-processed) (function)
• manager-stopped-p (function)
• (setf manager-stopped-p) (function)

Internal Definitions

• %make-manager (function)
• copy-manager (function)
• make-child-processors (function)
• manager-db (function)
• (setf manager-db) (function)
• manager-host (function)
• (setf manager-host) (function)
• manager-lock (function)
• (setf manager-lock) (function)
• manager-make-processor-fn (function)
• (setf manager-make-processor-fn) (function)
• manager-p (function)
• manager-port (function)
• (setf manager-port) (function)
• processor-died (function)
• processor-stopped (function)

5.10 psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Use List

• psychiq.specials
• psychiq.util
• common-lisp

Exported Definitions

• kill (function)
• make-scheduled (function)
• scheduled (structure)
• scheduled-status (function)
• (setf scheduled-status) (function)
• scheduled-thread (function)
• (setf scheduled-thread) (function)
• start (function)
• stop (function)

Internal Definitions

• %make-scheduled (function)
• copy-scheduled (function)
• enqueue-jobs (function)
• scaled-poll-interval (function)
• scheduled-connection (function)
• (setf scheduled-connection) (function)
• scheduled-p (function)

5.11 psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Use List

• psychiq.util
• common-lisp

Exported Definitions

• heartbeat (function)
• heartbeat (structure)
• kill (function)
• make-heartbeat (function)
• start (function)
• stop (function)

Internal Definitions

• clear-heartbeat (function)
• copy-heartbeat (function)
• heartbeat-db (function)
• (setf heartbeat-db) (function)
• heartbeat-host (function)
• (setf heartbeat-host) (function)
• heartbeat-manager (function)
• (setf heartbeat-manager) (function)
• heartbeat-p (function)
• heartbeat-port (function)
• (setf heartbeat-port) (function)
• heartbeat-stopped-p (function)
• (setf heartbeat-stopped-p) (function)
• heartbeat-thread (function)
• (setf heartbeat-thread) (function)
• run (function)

5.12 psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Use List

• psychiq.util
• psychiq.specials
• common-lisp

Exported Definitions

*psychiq-middleware-retry-jobs* (special variable)

Internal Definitions

• *dead-max-jobs* (special variable)
• *dead-timeout-in-seconds* (special variable)
• attempt-retry (function)
• backtrace (function)
• delay-for (function)
• send-to-morgue (function)

5.13 psychiq.middleware.logging

Source

logging.lisp (file)

Use List

• psychiq.util
• common-lisp

Exported Definitions

*psychiq-middleware-logging* (special variable)

5.14 psychiq.util

Source

util.lisp (file)

Use List

common-lisp

Used By List

• psychiq.launcher.heartbeat
• psychiq.launcher.scheduled
• psychiq.launcher.manager
• psychiq.launcher.processor
• psychiq.middleware.logging
• psychiq.middleware.retry-jobs
• psychiq.client
• psychiq.queue
• psychiq.worker

Exported Definitions

• generate-random-id (function)
• getpid (function)
• machine-identity (function)
• symbol-name-with-package (function)

5.15 psychiq.util.assoc

Source

assoc.lisp (file)

Use List

common-lisp

Exported Definitions

• aget (function)
• (setf aget) (function)

5.16 psychiq.util.redis

Source

redis.lisp (file)

Use List

common-lisp

Exported Definitions

• omit-redis-prefix (function)
• redis-key (function)
• with-redis-transaction (macro)

5.17 psychiq.util.concurrency

Source

concurrency.lisp (file)

Use List

common-lisp

Exported Definitions

• get-value (function)
• (setf get-value) (function)
• make-thread-safe-variable (function)
• reset-value (function)

Internal Definitions

• copy-thread-safe-variable (function)
• thread-safe-variable (structure)
• thread-safe-variable-initial-value (function)
• (setf thread-safe-variable-initial-value) (function)
• thread-safe-variable-lock (function)
• (setf thread-safe-variable-lock) (function)
• thread-safe-variable-p (function)
• thread-safe-variable-value (function)
• (setf thread-safe-variable-value) (function)

5.18 psychiq.specials

Source

src/specials.lisp (file)

Use List

common-lisp

Used By List

• psychiq.launcher
• psychiq.launcher.scheduled
• psychiq.launcher.manager
• psychiq.launcher.processor
• psychiq.middleware.retry-jobs
• psychiq.client
• psychiq.worker
• psychiq.connection

Exported Definitions

• *default-max-retry-attempts* (special variable)
• *default-queue-name* (special variable)
• *default-redis-host* (special variable)
• *default-redis-port* (special variable)
• *psychiq-namespace* (special variable)

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

6.1 Exported definitions

6.1.1 Special variables

Special Variable: *connection*

Package

psychiq.connection

Source

connection.lisp (file)

Special Variable: *default-max-retry-attempts*

Package

psychiq.specials

Source

src/specials.lisp (file)

Special Variable: *default-queue-name*

Package

psychiq.specials

Source

src/specials.lisp (file)

Special Variable: *default-redis-host*

Package

psychiq.specials

Source

src/specials.lisp (file)

Special Variable: *default-redis-port*

Package

psychiq.specials

Source

src/specials.lisp (file)

Special Variable: *psychiq-middleware-logging*

Package

psychiq.middleware.logging

Source

logging.lisp (file)

Special Variable: *psychiq-middleware-retry-jobs*

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Special Variable: *psychiq-namespace*

Package

psychiq.specials

Source

src/specials.lisp (file)

6.1.2 Macros

Macro: with-connection CONN &body BODY

Package

psychiq.connection

Source

connection.lisp (file)

Macro: with-redis-transaction &body BODY

Package

psychiq.util.redis

Source

redis.lisp (file)

6.1.3 Functions

Function: aget ALIST KEY &key TEST

Function: (setf aget) VAL ALIST KEY &key TEST

Package

psychiq.util.assoc

Source

assoc.lisp (file)

Function: all-dead-jobs ()

Package

psychiq.client

Source

client.lisp (file)

Function: all-queues ()

Package

psychiq.client

Source

client.lisp (file)

Function: all-retries ()

Package

psychiq.client

Source

client.lisp (file)

Function: cleanup-processes ()

Clean up dead processes recorded in Redis.

Package

psychiq.client

Source

client.lisp (file)

Function: connect &rest INITARGS &key HOST PORT DB

Package

psychiq.connection

Source

connection.lisp (file)

Function: connect-toplevel &rest INITARGS &key HOST PORT DB

Package

psychiq.connection

Source

connection.lisp (file)

Function: decode-job JOB-INFO

Package

psychiq.worker

Source

worker.lisp (file)

Function: decode-object PAYLOAD

Package

psychiq.coder

Source

coder.lisp (file)

Function: delete-queue QUEUE

Package

psychiq.client

Source

client.lisp (file)

Function: dequeue &optional QUEUE-OR-QUEUES TIMEOUT

Package

psychiq.client

Source

client.lisp (file)

Function: dequeue-from-queue QUEUE-OR-QUEUES &key TIMEOUT

Package

psychiq.queue

Source

queue.lisp (file)

Function: disconnect CONN

Package

psychiq.connection

Source

connection.lisp (file)

Function: disconnect-toplevel ()

Package

psychiq.connection

Source

connection.lisp (file)

Function: encode-job WORKER-CLASS ARGS

Package

psychiq.worker

Source

worker.lisp (file)

Function: encode-object OBJECT

Package

psychiq.coder

Source

coder.lisp (file)

Function: enqueue WORKER-CLASS &optional ARGS

Package

psychiq.client

Source

client.lisp (file)

Function: enqueue-bulk WORKER-CLASS JOB-ARGS &rest MORE-JOB-ARGS

Package

psychiq.client

Source

client.lisp (file)

Function: enqueue-in-sec INTERVAL WORKER-CLASS &optional ARGS

Package

psychiq.client

Source

client.lisp (file)

Function: enqueue-to-queue QUEUE JOB-INFO

Package

psychiq.queue

Source

queue.lisp (file)

Function: enqueue-to-scheduled-queue JOB-INFO AT

Package

psychiq.queue

Source

queue.lisp (file)

Function: ensure-connected CONN

Package

psychiq.connection

Source

connection.lisp (file)

Function: generate-random-id &optional LENGTH

Package

psychiq.util

Source

util.lisp (file)

Function: get-value VAR

Function: (setf get-value) NEW-VALUE VAR

Package

psychiq.util.concurrency

Source

concurrency.lisp (file)

Function: getpid ()

Package

psychiq.util

Source

util.lisp (file)

Function: heartbeat HEARTBEAT MACHINE-IDENTITY JSON

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: kill LAUNCHER

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: kill SCHEDULED

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: kill HEARTBEAT

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: launcher-status LAUNCHER

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: machine-identity ()

Package

psychiq.util

Source

util.lisp (file)

Function: make-connection &rest INITARGS &key HOST PORT DB

Package

psychiq.connection

Source

connection.lisp (file)

Function: make-heartbeat &key (HOST HOST) (PORT PORT) (DB DB) (THREAD THREAD) (STOPPED-P STOPPED-P) (MANAGER MANAGER)

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: make-manager &key HOST PORT DB QUEUES COUNT TIMEOUT

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: make-processor &key HOST PORT DB QUEUES MANAGER TIMEOUT

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: make-scheduled &key HOST PORT DB

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: make-thread-safe-variable INITIAL-VALUE &key LOCK &aux VALUE

Package

psychiq.util.concurrency

Source

concurrency.lisp (file)

Function: manager-children INSTANCE

Function: (setf manager-children) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-count INSTANCE

Function: (setf manager-count) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-queues INSTANCE

Function: (setf manager-queues) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-stat-failed INSTANCE

Function: (setf manager-stat-failed) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-stat-processed INSTANCE

Function: (setf manager-stat-processed) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-stopped-p INSTANCE

Function: (setf manager-stopped-p) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: omit-redis-prefix KEY &rest PREFIXES

Package

psychiq.util.redis

Source

redis.lisp (file)

Function: peek-queue QUEUE &optional START COUNT

Package

psychiq.client

Source

client.lisp (file)

Function: processor-connection INSTANCE

Function: (setf processor-connection) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: processor-id INSTANCE

Function: (setf processor-id) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: processor-manager INSTANCE

Function: (setf processor-manager) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: processor-processing INSTANCE

Function: (setf processor-processing) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: processor-status INSTANCE

Function: (setf processor-status) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: processor-timeout INSTANCE

Function: (setf processor-timeout) VALUE INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: queue-empty-p QUEUE

Package

psychiq.client

Source

client.lisp (file)

Function: queue-length QUEUE

Package

psychiq.client

Source

client.lisp (file)

Function: reconnect CONN

Package

psychiq.connection

Source

connection.lisp (file)

Function: redis-key &rest KEYS

Package

psychiq.util.redis

Source

redis.lisp (file)

Function: reset-value VAR

Package

psychiq.util.concurrency

Source

concurrency.lisp (file)

Function: retry-length ()

Package

psychiq.client

Source

client.lisp (file)

Function: run &rest INITARGS &key HOST PORT DB CONCURRENCY QUEUE

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: scheduled-status INSTANCE

Function: (setf scheduled-status) VALUE INSTANCE

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: scheduled-thread INSTANCE

Function: (setf scheduled-thread) VALUE INSTANCE

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: slice-queue QUEUE START &optional END

Package

psychiq.client

Source

client.lisp (file)

Function: start LAUNCHER

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: start SCHEDULED

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: start HEARTBEAT

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: stats ()

Package

psychiq.client

Source

client.lisp (file)

Function: stop LAUNCHER

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: stop SCHEDULED

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: stop HEARTBEAT

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: symbol-name-with-package SYMBOL

Package

psychiq.util

Source

util.lisp (file)

Function: wait-for LAUNCHER

Package

psychiq.launcher

Source

launcher.lisp (file)

6.1.4 Generic functions

Generic Function: close-connection CONN

Package

psychiq.connection

Source

connection.lisp (file)

Methods

Method: close-connection (CONN connection)

Generic Function: connectedp CONN

Package

psychiq.connection

Source

connection.lisp (file)

Methods

Method: connectedp (CONN connection)

Generic Function: fetch-job PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: fetch-job (PROCESSOR processor)

Generic Function: finalize PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: finalize (PROCESSOR processor) after

Source

manager.lisp (file)

Method: finalize (PROCESSOR processor)

Generic Function: kill PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: kill (MANAGER manager)

Source

manager.lisp (file)

Method: kill (PROCESSOR processor)

Generic Function: open-connection CONN

Package

psychiq.connection

Source

connection.lisp (file)

Methods

Method: open-connection (CONN connection)

Generic Function: perform WORKER &rest ARGS

Package

psychiq.worker

Source

worker.lisp (file)

Methods

Method: perform (WORKER worker) &rest ARGS

Generic Function: perform-job PROCESSOR QUEUE WORKER &rest ARGS

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: perform-job (PROCESSOR processor) QUEUE WORKER &rest ARGS

Generic Function: process-job PROCESSOR QUEUE JOB-INFO

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: process-job (PROCESSOR processor) QUEUE JOB-INFO around

Source

manager.lisp (file)

Method: process-job (PROCESSOR processor) QUEUE JOB-INFO

Generic Function: run PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: run (PROCESSOR processor) around

Source

manager.lisp (file)

Method: run (PROCESSOR processor)

Generic Function: start PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: start (MANAGER manager)

Source

manager.lisp (file)

Method: start (PROCESSOR processor)

Generic Function: stop PROCESSOR

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: stop (MANAGER manager)

Source

manager.lisp (file)

Method: stop (PROCESSOR processor)

Generic Function: wait-for OBJECT

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Methods

Method: wait-for (MANAGER manager)

Source

manager.lisp (file)

Method: wait-for (PROCESSOR processor)

Generic Function: worker-max-retries WORKER

Package

psychiq.worker

Source

worker.lisp (file)

Methods

Method: worker-max-retries (WORKER worker)

Generic Function: worker-queue-name WORKER

Package

psychiq.worker

Source

worker.lisp (file)

Methods

Method: worker-queue-name (WORKER worker)

Generic Function: worker-use-backtrace-p WORKER

Package

psychiq.worker

Source

worker.lisp (file)

Methods

Method: worker-use-backtrace-p (WORKER worker)

Generic Function: worker-use-dead-queue-p WORKER

Package

psychiq.worker

Source

worker.lisp (file)

Methods

Method: worker-use-dead-queue-p (WORKER worker)

6.1.5 Structures

Structure: heartbeat ()

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots

Slot: host

Readers

heartbeat-host (function)

Writers

(setf heartbeat-host) (function)

Slot: port

Readers

heartbeat-port (function)

Writers

(setf heartbeat-port) (function)

Slot: db

Readers

heartbeat-db (function)

Writers

(setf heartbeat-db) (function)

Slot: thread

Readers

heartbeat-thread (function)

Writers

(setf heartbeat-thread) (function)

Slot: stopped-p

Initform

t

Readers

heartbeat-stopped-p (function)

Writers

(setf heartbeat-stopped-p) (function)

Slot: manager

Readers

heartbeat-manager (function)

Writers

(setf heartbeat-manager) (function)

Structure: launcher ()

Package

psychiq.launcher

Source

launcher.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

print-object (method)

Direct slots

Slot: manager

Readers

launcher-manager (function)

Writers

(setf launcher-manager) (function)

Slot: scheduled

Readers

launcher-scheduled (function)

Writers

(setf launcher-scheduled) (function)

Slot: heartbeat

Readers

launcher-heartbeat (function)

Writers

(setf launcher-heartbeat) (function)

Structure: manager ()

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

• wait-for (method)
• kill (method)
• stop (method)
• start (method)

Direct slots

Slot: host

Readers

manager-host (function)

Writers

(setf manager-host) (function)

Slot: port

Readers

manager-port (function)

Writers

(setf manager-port) (function)

Slot: db

Readers

manager-db (function)

Writers

(setf manager-db) (function)

Slot: queues

Initform

(quote nil)

Readers

manager-queues (function)

Writers

(setf manager-queues) (function)

Slot: count

Initform

25

Readers

manager-count (function)

Writers

(setf manager-count) (function)

Slot: children

Initform

(quote nil)

Readers

manager-children (function)

Writers

(setf manager-children) (function)

Slot: lock

Initform

(bordeaux-threads:make-recursive-lock)

Readers

manager-lock (function)

Writers

(setf manager-lock) (function)

Slot: stopped-p

Initform

t

Readers

manager-stopped-p (function)

Writers

(setf manager-stopped-p) (function)

Slot: stat-processed

Readers

manager-stat-processed (function)

Writers

(setf manager-stat-processed) (function)

Slot: stat-failed

Readers

manager-stat-failed (function)

Writers

(setf manager-stat-failed) (function)

Slot: make-processor-fn

Readers

manager-make-processor-fn (function)

Writers

(setf manager-make-processor-fn) (function)

Structure: processor ()

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

• finalize (method)
• process-job (method)
• run (method)
• perform-job (method)
• process-job (method)
• wait-for (method)
• kill (method)
• stop (method)
• start (method)
• finalize (method)
• run (method)
• fetch-job (method)
• print-object (method)

Direct slots

Slot: id

Initform

(psychiq.util:generate-random-id 9)

Readers

processor-id (function)

Writers

(setf processor-id) (function)

Slot: connection

Type

psychiq.connection:connection

Readers

processor-connection (function)

Writers

(setf processor-connection) (function)

Slot: queues

Type

list

Initform

(quote nil)

Readers

processor-queues (function)

Writers

(setf processor-queues) (function)

Slot: manager

Readers

processor-manager (function)

Writers

(setf processor-manager) (function)

Slot: thread

Readers

processor-thread (function)

Writers

(setf processor-thread) (function)

Slot: status

Initform

:stopped

Readers

processor-status (function)

Writers

(setf processor-status) (function)

Slot: timeout

Initform

5

Readers

processor-timeout (function)

Writers

(setf processor-timeout) (function)

Slot: down

Readers

processor-down (function)

Writers

(setf processor-down) (function)

Slot: processing

Readers

processor-processing (function)

Writers

(setf processor-processing) (function)

Structure: scheduled ()

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Direct superclasses

structure-object (structure)

Direct slots

Slot: connection

Readers

scheduled-connection (function)

Writers

(setf scheduled-connection) (function)

Slot: thread

Readers

scheduled-thread (function)

Writers

(setf scheduled-thread) (function)

Slot: status

Initform

:stopped

Readers

scheduled-status (function)

Writers

(setf scheduled-status) (function)

6.1.6 Classes

Class: connection ()

Package

psychiq.connection

Source

connection.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• connectedp (method)
• close-connection (method)
• open-connection (method)
• print-object (method)
• redis-connection (method)
• redis-connection (method)

Direct slots

Slot: host

Type

string

Initargs

:host

Initform

psychiq.specials:*default-redis-host*

Slot: port

Type

integer

Initargs

:port

Initform

psychiq.specials:*default-redis-port*

Slot: db

Type

(or integer null)

Initargs

:db

Slot: redis

Type

(or redis:redis-connection null)

Readers

redis-connection (generic function)

Writers

(setf redis-connection) (generic function)

Class: worker ()

Package

psychiq.worker

Source

worker.lisp (file)

Direct superclasses

standard-object (class)

Direct methods

• worker-use-backtrace-p (method)
• worker-queue-name (method)
• worker-use-dead-queue-p (method)
• worker-max-retries (method)
• perform (method)

6.2 Internal definitions

6.2.1 Special variables

Special Variable: *dead-max-jobs*

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Special Variable: *dead-timeout-in-seconds*

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

6.2.2 Functions

Function: %make-launcher &key (MANAGER MANAGER) (SCHEDULED SCHEDULED) (HEARTBEAT HEARTBEAT)

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: %make-manager &key HOST PORT DB QUEUES COUNT LOCK &aux STAT-PROCESSED STAT-FAILED

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: %make-processor &key (ID ID) (CONNECTION CONNECTION) (QUEUES QUEUES) (MANAGER MANAGER) (THREAD THREAD) (STATUS STATUS) (TIMEOUT TIMEOUT) (DOWN DOWN) (PROCESSING PROCESSING)

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: %make-scheduled &key (CONNECTION CONNECTION) (THREAD THREAD) (STATUS STATUS)

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: attempt-retry QUEUE WORKER JOB-INFO E

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Function: backtrace &optional COUNT-TO-REMOVE

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Function: clear-heartbeat HEARTBEAT

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: copy-heartbeat INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: copy-launcher INSTANCE

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: copy-manager INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: copy-processor INSTANCE

Package

psychiq.launcher.processor

Source

processor.lisp (file)

Function: copy-scheduled INSTANCE

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: copy-thread-safe-variable INSTANCE

Package

psychiq.util.concurrency

Source

concurrency.lisp (file)

Function: decode-objects OBJECTS

Package

psychiq.client

Source

client.lisp (file)

Function: delay-for RETRY-COUNT

Package

psychiq.middleware.retry-jobs

Source

retry-jobs.lisp (file)

Function: enqueue-jobs NOW

Package

psychiq.launcher.scheduled

Source

scheduled.lisp (file)

Function: heartbeat-db INSTANCE

Function: (setf heartbeat-db) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-host INSTANCE

Function: (setf heartbeat-host) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-manager INSTANCE

Function: (setf heartbeat-manager) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-p OBJECT

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-port INSTANCE

Function: (setf heartbeat-port) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-stopped-p INSTANCE

Function: (setf heartbeat-stopped-p) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: heartbeat-thread INSTANCE

Function: (setf heartbeat-thread) VALUE INSTANCE

Package

psychiq.launcher.heartbeat

Source

heartbeat.lisp (file)

Function: launcher-heartbeat INSTANCE

Function: (setf launcher-heartbeat) VALUE INSTANCE

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: launcher-manager INSTANCE

Function: (setf launcher-manager) VALUE INSTANCE

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: launcher-p OBJECT

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: launcher-scheduled INSTANCE

Function: (setf launcher-scheduled) VALUE INSTANCE

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: make-child-processors MANAGER

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: make-launcher &key HOST PORT DB CONCURRENCY QUEUE INTERVAL

Package

psychiq.launcher

Source

launcher.lisp (file)

Function: manager-db INSTANCE

Function: (setf manager-db) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)

Function: manager-host INSTANCE

Function: (setf manager-host) VALUE INSTANCE

Package

psychiq.launcher.manager

Source

manager.lisp (file)