Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the swank-crew Reference Manual, version 1.3, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 12:38:00 2018 GMT+0.
| • Introduction: | What swank-crew is all about | |
| • Systems: | The systems documentation | |
| • Files: | The files documentation | |
| • Packages: | The packages documentation | |
| • Definitions: | The symbols documentation | |
| • Indexes: | Concepts, functions, variables and data types |
Swank Crew
==========
Swank Crew is a distributed computation framework written in Common Lisp that
uses Swank Client to evaluate forms on remote hosts.
Swank Crew works with Emacs and Slime to make developing distributed
applications easier. Typically, you interactively develop code on a master
machine using Slime. Code on the master uses the Swank Crew API to farm out
work to a gang of worker machines. If code on a worker generates an exception,
a Slime stack trace window pops up and you can interactively debug the problem
using your existing Slime connection to the master.
The Swank Crew API
==================
connect-workers Create a worker pool containing a set of worker
Lisps.
disconnect-workers Disconnect all the workers in a worker pool.
eval-form-all-workers Evaluate a form on all the workers in a worker pool.
eval-form-repeatedly Evaluate a form repeatedly on the workers in a pool
and collect the results on the master.
eval-repeatedly-async-state Evaluate a form repeatedly on the workers in a pool
while asynchronously updating the workers' state.
parallel-mapcar Use the workers in a worker pool to evaluate a
function on all the elements of a list.
parallel-reduce Use the workers in a worker pool to evaluate a
function on all the elements of a list. As results
return to the master, call a function repeatedly to
reduce them to one value.
For more information, see the documentation strings for the above functions in
master.lisp and the example code in swank-crew_test.lisp.
Installation
============
Swank Crew functions best when used with Slime. In order to implement the
distributed Slime debugging features described above, you must patch swank.lisp
in the directory where you installed Slime:
patch path/to/slime/swank.lisp swank.lisp-patch
The change to swank.lisp implements the forwarding of Swank protocol messages
from worker machines through the master's Swank server back to your Slime
client. It's needed to debug problems on worker machines using your Slime
connection to the master.
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
| • The swank-crew system: |
Robert Brown
New BSD license. See the copyright messages in individual files.
Distributed master/worker computing framework.
Swank Crew is a framework for developing distributed master/worker applications. It uses Slime’s Swank protocol to transport data between machines, making the debugging of distributed applications easier.
1.3
swank-crew.asd (file)
Files are sorted by type and then listed depth-first from the systems components trees.
| • Lisp files: |
| • The swank-crew.asd file: | ||
| • The swank-crew/package.lisp file: | ||
| • The swank-crew/master.lisp file: | ||
| • The swank-crew/worker.lisp file: |
Next: The swank-crew/package<dot>lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
swank-crew.asd
swank-crew (system)
Next: The swank-crew/master<dot>lisp file, Previous: The swank-crew<dot>asd file, Up: Lisp files [Contents][Index]
swank-crew (system)
package.lisp
Next: The swank-crew/worker<dot>lisp file, Previous: The swank-crew/package<dot>lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
swank-crew (system)
master.lisp
Previous: The swank-crew/master<dot>lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
swank-crew (system)
worker.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
| • The swank-crew package: |
Evaluate expressions on remote Lisps using the Swank protocol.
package.lisp (file)
Definitions are sorted by export status, category, package, and then by lexicographic order.
| • Exported definitions: | ||
| • Internal definitions: |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
| • Exported special variables: | ||
| • Exported functions: | ||
| • Exported classes: |
Next: Exported functions, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
master.lisp (file)
Next: Exported classes, Previous: Exported special variables, Up: Exported definitions [Contents][Index]
Makes Swank connections to all the workers in HOST/PORT-ALIST and returns a WORKER-POOL containing them. HOST/PORT-ALIST is a list of (host-name . port) pairs
master.lisp (file)
Closes the Swank connections of all connected workers in WORKER-POOL.
master.lisp (file)
Evaluates FORM on all workers in WORKER-POOL. When RESULT-DONE is non-NIL, it must be a function of two arguments. In this case RESULT-DONE is called as each worker returns a result with two arguments, a non-negative integer representing the order in which the work was dispatched and the worker’s result. If REPLAY-REQUIRED is true, which is the default, FORM will be remembered and evaluated again for side effects on any new worker that joins POOL.
master.lisp (file)
Evaluates FORM, which must return a function of no arguments, on WORKER-COUNT workers in WORKER-POOL, then arranges for the workers to repeatedly call the function to create RESULT-COUNT results, which are returned in a list. WORKER-COUNT defaults to the number of workers in WORKER-POOL.
master.lisp (file)
Evaluates FORM, which must return a function of one argument, on WORKER-COUNT
workers in WORKER-POOL, then arranges for the workers to repeatedly call the
work function and send its results back to the master. The work function is
passed a state argument, initially set to INITIAL-STATE, that the master can
update asynchronously as it receives new results.
On the master, the work state is initialized to INITIAL-STATE and UPDATE-STATE is called repeatedly to process results received from the workers. UPDATE-STATE is called with two arguments, the current work state and a list containing all unprocessed work results. UPDATE-STATE should return three values: the new work state, a boolean indicating whether the computation should end, and a boolean indicating whether the latest work state should be distributed to workers. When UPDATE-STATE’s second return value is true, EVAL-REPEATEDLY-ASYNC-STATE tells the workers to stop and returns the latest work state.
master.lisp (file)
Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. Results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is provided, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are the position of the work in LIST and the worker’s result.
master.lisp (file)
Traverses LIST, calling MAKE-WORK on each element to create a form that is
then passed to a remote worker in WORKER-POOL for evaluation. As results are
returned, REDUCER, a binary function, is used to combine successive results in
the manner of REDUCE. INITIAL-VALUE is used as the starting value for the
reduction computation. The form
(parallel-reduce worker-pool make-work list initial-value reducer)
is equivalent to
(reduce reducer
(mapcar (lambda (x) (eval (funcall make-work x))) list)
:initial-value initial-value)
master.lisp (file)
Returns the total number of workers in WORKER-POOL.
master.lisp (file)
Previous: Exported functions, Up: Exported definitions [Contents][Index]
A pool of Swank workers to which Lisp forms can be sent for evaluation.
master.lisp (file)
standard-object (class)
The worker pool’s ID.
symbol
(gentemp (string-upcase (format nil "~a-~d-" (machine-instance) (osicat-posix:getpid))) "keyword")
id (generic function)
Host name of the master.
string
(machine-instance)
master-host-name (generic function)
Port number for Swank remote execution requests to the master.
(or swank-crew::port null)
swank-crew:*rex-port*
rex-port (generic function)
Vector containing all workers in the worker pool.
vector
#()
workers (generic function)
(setf %workers) (generic function)
Lock protecting IDLE-WORKERS, WORKER-AVAILABLE, and DISCONNECTING.
(bordeaux-threads:make-lock "worker pool")
lock (generic function)
List of currently idle workers.
list
(quote nil)
idle-workers (generic function)
(setf idle-workers) (generic function)
Condition signaled when a worker becomes idle.
(bordeaux-threads:make-condition-variable)
worker-available (generic function)
Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.
disconnecting (generic function)
(setf disconnecting) (generic function)
Lock that protects REPLAY-FORMS.
(bordeaux-threads:make-lock "replay forms lock")
replay-forms-lock (generic function)
List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.
list
(quote nil)
replay-forms (generic function)
(setf replay-forms) (generic function)
Previous: Exported definitions, Up: Definitions [Contents][Index]
| • Internal constants: | ||
| • Internal special variables: | ||
| • Internal functions: | ||
| • Internal generic functions: | ||
| • Internal classes: | ||
| • Internal types: |
Next: Internal special variables, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
Seconds between attempts to reconnect workers.
master.lisp (file)
Next: Internal functions, Previous: Internal constants, Up: Internal definitions [Contents][Index]
List containing an EVALUATION instance for each running computation.
master.lisp (file)
Lock protecting access to *EVALS*.
master.lisp (file)
Counter used to generate a unique ID for EVALUATION instances.
master.lisp (file)
Lock protecting access to *EVALUATION-ID*.
master.lisp (file)
The last form evaluated by the worker.
worker.lisp (file)
The value of *RANDOM-STATE* right before the worker started evaluating *LAST-FORM-EVALED*.
worker.lisp (file)
Thunk created by the current invocation of REPEATEDLY-EVALUATE to produce results for an EVAL-FORM-REPEATEDLY request on the master. This variable is useful for debugging.
worker.lisp (file)
Mapping from worker pool IDs to the number of replay forms we have evaluated on this client for that pool.
worker.lisp (file)
Lock protecting access to *REPLAY-FORMS-COUNTS*.
worker.lisp (file)
Mapping from worker pool IDs to active worker pools.
master.lisp (file)
Lock protecting access to *WORKER-POOLS*.
master.lisp (file)
Next: Internal generic functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
Adds EVAL to the list of in-progress computations.
master.lisp (file)
Adds FORM to the set of forms that need to be evaluated on a new worker when it joins WORKER-POOL.
master.lisp (file)
Allocates and returns an idle worker from WORKER-POOL that is connected. If WORKER-POOL is being shut down, then NIL is returned.
master.lisp (file)
Allocates a connected worker from WORKER-POOL and sends it FORM to evaluate. Returns the worker that was successfully allocated, or NIL if WORKER-POOL is shutting down. SET-WORKER is a function that is called to tell the WORKER-DONE continuation what worker is evaluating FORM.
master.lisp (file)
Evaluates FORM, which must return a work function of one argument, then calls that function repeatedly to produce results, each time passing it the current computation state. At first this state is INITIAL-STATE, but the master may update the state asynchronously. Each work result is sent to the machine called MASTER-HOST-NAME by making a Swank connection to its MASTER-REX-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to process results and on the worker to update the state.
worker.lisp (file)
Handles incoming results for ASYNC-EVAL by calling UPDATE-STATE whenever ASYNC-EVAL holds unprocessed results. UPDATE-STATE is called with two arguments, the work state and a list of the unprocessed results.
master.lisp (file)
Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the async evaluation request identified by ID.
master.lisp (file)
Ensures that the current client is up to date, that it has evaluated all POOL-COUNT replay forms associated with the pool identified by WORKER-POOL-ID. If it is necessary to evaluate forms in order to catch up, they are fetched by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME.
worker.lisp (file)
Sets all debugging global variables to NIL.
worker.lisp (file)
Returns an expression that when evaluated returns the result of evaluating FORM. In addition, the returned expression arranges to update the values of *LAST-FORM-EVALED* and *LAST-RANDOM-STATE* for ease of debugging.
worker.lisp (file)
Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. When RETAIN-WORKERS is true, each form is evaluated on a unique worker. Otherwise, workers are reused and may process multiple forms. In either case, the results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is not NIL, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are an integer counter, indicating the work form’s position in LIST, and the result of evaluating the form. REPLAY-REQUIRED indicates whether new workers will have to perform the work generated by MAKE-WORK before being considered to be caught up.
master.lisp (file)
Returns a form that when evaluated on a worker in WORKER-POOL ensures that the worker is caught up then evaluates FORM and returns its result. REPLAY-REQUIRED indicates whether new workers must evaluate FORM before being considered to be caught up.
master.lisp (file)
Iterates over the members of LIST calling MAKE-WORK on each one. If RESULT-DONE is not NIL, it must be a function of two arguments. In this case, after each application of MAKE-WORK to a LIST member, RESULT-DONE is called with the member’s position in LIST and MAKE-WORK’s result.
master.lisp (file)
Evaluates FORM, but first ensures that this worker has evaluated all POOL-COUNT replay forms associated with the worker pool identified by WORKER-POOL-ID on the master. When catching up is required, fetches forms by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME. REPLAY-REQUIRED indicates whether FORM may need to be replayed in order to bring a worker up to date.
worker.lisp (file)
Fetches from the master and then evaluates all forms required to catch up with other workers in the pool identified by WORKER-POOL-ID on the master. All replay forms after the first LOCAL-COUNT forms are fetched by making a Swank connection to MASTER-REX-PORT on host MASTER-HOST-NAME.
worker.lisp (file)
Returns the running EVAL instance identified by ID, or NIL if no computation with that ID is currently running.
master.lisp (file)
Returns the worker pool identified by WORKER-POOL-ID.
master.lisp (file)
Changes the state of WORKER to idle, so that it’s available for allocation.
master.lisp (file)
Called when the connection to DISCONNECTED-WORKER, a member of WORKER-POOL, is closed because of a call to SLIME-CLOSE, the death of the worker, or because of a communications error. Moves all uncompleted work intended for DISCONNECTED-WORKER to another idle connected worker in WORKER-POOL.
master.lisp (file)
master.lisp (file)
Returns T if WORKER-POOL is NIL or contains no workers.
master.lisp (file)
Reconnects disconnected workers in WORKER-POOL.
master.lisp (file)
Stores RESULT as one of the values produced by the async evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call ASYNC-RESULT with additional results.
master.lisp (file)
Stores RESULT as one of the values produced by the repeated evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call RECORD-REPEATED-RESULT with additional results.
master.lisp (file)
Removes EVAL from the list of in-progress computations.
master.lisp (file)
Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the repeated evaluation request identified by ID.
master.lisp (file)
Evaluates FORM, which must return a function of no arguments, then calls that function repeatedly to produce results. Each result is sent to the machine called MASTER-HOST-NAME by making a Swank connection to its MASTER-REX-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to correctly record the result.
worker.lisp (file)
Creates a Slime connection to MASTER-HOST-NAME on port MASTER-REX-PORT, then repeatedly calls SEND-RESULT with the new connection as argument. Returns when SEND-RESULT returns NIL.
worker.lisp (file)
For a worker that has executed EVALUATED-COUNT of the replay forms associated with the worker-pool identified by WORKER-POOL-ID, returns a list of the forms the worker needs to evaluate in order to be completely up to date.
master.lisp (file)
Returns the number of idle, busy, and disconnected workers in WORKER-POOL. This function executes without locking WORKER-POOL, so it may return inconsistent information.
master.lisp (file)
Next: Internal classes, Previous: Internal functions, Up: Internal definitions [Contents][Index]
Vector containing all workers in the worker pool.
master.lisp (file)
Information used when connecting to this worker.
master.lisp (file)
Creates a connection to a worker’s Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.
master.lisp (file)
When non-NIL, an open Swank connection to the worker.
master.lisp (file)
Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.
master.lisp (file)
Is the computation done?
master.lisp (file)
Condition variable notified when computation is done.
master.lisp (file)
Host the worker is running on.
master.lisp (file)
Unique ID for this running evaluation request.
master.lisp (file)
The worker pool’s ID.
master.lisp (file)
List of currently idle workers.
master.lisp (file)
Lock protecting the access to this instance.
master.lisp (file)
Lock protecting IDLE-WORKERS, WORKER-AVAILABLE, and DISCONNECTING.
master.lisp (file)
Lock protecting the mutable data of this worker.
master.lisp (file)
Host name of the master.
master.lisp (file)
Port on which the worker’s Swank server is listening for connections.
master.lisp (file)
Reconnects to a Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.
master.lisp (file)
List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.
master.lisp (file)
Lock that protects REPLAY-FORMS.
master.lisp (file)
List holding returned, but unprocessed, results.
master.lisp (file)
Vector holding returned results.
master.lisp (file)
Condition notified when new results are available.
master.lisp (file)
Position where the next result will be recorded.
master.lisp (file)
Port number for Swank remote execution requests to the master.
master.lisp (file)
Function called to record which worker is evaluating a form.
master.lisp (file)
State of the asynchronous computation, updated from results.
master.lisp (file)
Counter incremented each time STATE is updated.
master.lisp (file)
Condition signaled when a worker becomes idle.
master.lisp (file)
Vector containing all workers in the worker pool.
master.lisp (file)
Next: Internal types, Previous: Internal generic functions, Up: Internal definitions [Contents][Index]
Data needed to process incoming asynchronous evaluation results.
master.lisp (file)
evaluation (class)
List holding returned, but unprocessed, results.
list
(quote nil)
results (generic function)
(setf results) (generic function)
Condition notified when new results are available.
(bordeaux-threads:make-condition-variable)
results-available (generic function)
State of the asynchronous computation, updated from results.
:state
state (generic function)
(setf state) (generic function)
Counter incremented each time STATE is updated.
(integer 0)
0
state-counter (generic function)
(setf state-counter) (generic function)
Information needed when connecting to a worker’s Swank server.
master.lisp (file)
standard-object (class)
Host the worker is running on.
string
:host-name
host-name (generic function)
(setf host-name) (generic function)
Port on which the worker’s Swank server is listening for connections.
swank-crew::port
:port
port (generic function)
(setf port) (generic function)
Stores the data needed to process incoming evaluation results.
master.lisp (file)
standard-object (class)
Unique ID for this running evaluation request.
integer
:id
(bordeaux-threads:with-lock-held (swank-crew::*evaluation-id-lock*) (incf swank-crew::*evaluation-id*))
id (generic function)
Lock protecting the access to this instance.
(bordeaux-threads:make-lock "evaluation")
lock (generic function)
Is the computation done?
boolean
done (generic function)
(setf done) (generic function)
Condition variable notified when computation is done.
(bordeaux-threads:make-condition-variable)
done-condition (generic function)
Stores the data needed to process an incoming repeated eval result.
master.lisp (file)
evaluation (class)
Vector holding returned results.
simple-vector
:results
results (generic function)
Position where the next result will be recorded.
com.google.base:vector-index
0
results-position (generic function)
(setf results-position) (generic function)
A remote Lisp running a Swank server.
master.lisp (file)
standard-object (class)
Lock protecting the mutable data of this worker.
(bordeaux-threads:make-lock "worker")
lock (generic function)
Information used when connecting to this worker.
swank-crew::connect-info
:connect-info
connect-info (generic function)
Function called to record which worker is evaluating a form.
function
(function identity)
set-worker (generic function)
(setf set-worker) (generic function)
When non-NIL, an open Swank connection to the worker.
(or null swank-client:swank-connection)
connection (generic function)
(setf connection) (generic function)
Previous: Internal classes, Up: Internal definitions [Contents][Index]
A non-privileged TCP/IP port number.
master.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
| • Concept index: | ||
| • Function index: | ||
| • Variable index: | ||
| • Data type index: |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
| Jump to: | F L S |
|---|
| Jump to: | F L S |
|---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
| Jump to: | (
A C D E F G H I L M N P R S U W |
|---|
| Jump to: | (
A C D E F G H I L M N P R S U W |
|---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
| Jump to: | *
+
C D H I L M P R S W |
|---|
| Jump to: | *
+
C D H I L M P R S W |
|---|
Previous: Variable index, Up: Indexes [Contents][Index]
| Jump to: | A C E P R S T W |
|---|
| Jump to: | A C E P R S T W |
|---|