This is the swank-crew Reference Manual, version 1.5, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Nov 06 15:00:11 2023 GMT+0.
The main system appears first, followed by any subsystem dependency.
swank-crewDistributed master/worker computing framework.
Robert Brown <robert.brown@gmail.com>
New BSD license. See the copyright messages in individual files.
Swank Crew is a framework for developing distributed master/worker applications. It uses Slime’s Swank protocol to transport data between machines, making the debugging of distributed applications easier.
1.5
com.google.base (system).
bordeaux-threads (system).
swank-client (system).
package.lisp (file).
master.lisp (file).
worker.lisp (file).
Files are sorted by type and then listed depth-first from the systems components trees.
swank-crew/master.lisppackage.lisp (file).
swank-crew (system).
connect-workers (function).
disconnect-workers (function).
eval-form-all-workers (function).
eval-form-repeatedly (function).
eval-repeatedly-async-state (function).
initialize-instance (method).
parallel-mapcar (function).
parallel-reduce (function).
print-object (method).
worker-count (function).
worker-pool (class).
(setf %workers) (writer method).
*evals* (special variable).
*evals-lock* (special variable).
*evaluation-id* (special variable).
*evaluation-id-lock* (special variable).
*worker-pools* (special variable).
*worker-pools-lock* (special variable).
+worker-reconnect-interval+ (constant).
add-eval (function).
add-evaluated-form (function).
allocate-worker (function).
allocate-worker-and-evaluate (function).
async-eval (class).
async-results-loop (function).
async-work-form (function).
connect-info (reader method).
connect-info (class).
connect-worker (generic function).
connection (reader method).
(setf connection) (writer method).
disconnecting (reader method).
(setf disconnecting) (writer method).
dispatch-work (function).
done (reader method).
(setf done) (writer method).
done-condition (reader method).
ensure-caught-up-then-evaluate (function).
eval-on-master (function).
evaluation (class).
find-eval (function).
find-worker-pool (function).
free-worker (function).
handle-connection-closed (function).
host-name (reader method).
(setf host-name) (writer method).
id (reader method).
id (reader method).
idle-workers (reader method).
(setf idle-workers) (writer method).
lock (reader method).
lock (reader method).
lock (reader method).
make-worker-pool (function).
master-host-name (reader method).
master-swank-port (reader method).
no-workers-p (function).
port (reader method).
(setf port) (writer method).
port (type).
reconnect-worker (generic function).
reconnect-workers (function).
record-async-result (function).
record-repeated-result (function).
remove-eval (function).
repeated-eval (class).
repeated-work-form (function).
replay-forms (reader method).
(setf replay-forms) (writer method).
replay-forms-lock (reader method).
results (reader method).
results (reader method).
(setf results) (writer method).
results-available (reader method).
results-position (reader method).
(setf results-position) (writer method).
set-worker (reader method).
(setf set-worker) (writer method).
state (reader method).
(setf state) (writer method).
state-counter (reader method).
(setf state-counter) (writer method).
unevaluated-replay-forms (function).
worker (class).
worker-available (reader method).
worker-counts (function).
workers (reader method).
swank-crew/worker.lisppackage.lisp (file).
swank-crew (system).
*last-form-evaled* (special variable).
*last-random-state* (special variable).
*last-repeated-eval-work-function* (special variable).
*replay-forms-counts* (special variable).
*replay-forms-counts-lock* (special variable).
async-evaluate (function).
catch-up-if-necessary (function).
clear-debugging-info (function).
debugging-form (function).
evaluate-form (function).
fetch-and-evaluate (function).
repeatedly-evaluate (function).
send-many-results (function).
Packages are listed by definition order.
swank-crewEvaluate expressions on remote Lisps using the Swank protocol.
common-lisp.
connect-workers (function).
disconnect-workers (function).
eval-form-all-workers (function).
eval-form-repeatedly (function).
eval-repeatedly-async-state (function).
parallel-mapcar (function).
parallel-reduce (function).
worker-count (function).
worker-pool (class).
(setf %workers) (generic writer).
*evals* (special variable).
*evals-lock* (special variable).
*evaluation-id* (special variable).
*evaluation-id-lock* (special variable).
*last-form-evaled* (special variable).
*last-random-state* (special variable).
*last-repeated-eval-work-function* (special variable).
*replay-forms-counts* (special variable).
*replay-forms-counts-lock* (special variable).
*worker-pools* (special variable).
*worker-pools-lock* (special variable).
+worker-reconnect-interval+ (constant).
add-eval (function).
add-evaluated-form (function).
allocate-worker (function).
allocate-worker-and-evaluate (function).
async-eval (class).
async-evaluate (function).
async-results-loop (function).
async-work-form (function).
catch-up-if-necessary (function).
clear-debugging-info (function).
connect-info (generic reader).
connect-info (class).
connect-worker (generic function).
connection (generic reader).
(setf connection) (generic writer).
debugging-form (function).
disconnecting (generic reader).
(setf disconnecting) (generic writer).
dispatch-work (function).
done (generic reader).
(setf done) (generic writer).
done-condition (generic reader).
ensure-caught-up-then-evaluate (function).
eval-on-master (function).
evaluate-form (function).
evaluation (class).
fetch-and-evaluate (function).
find-eval (function).
find-worker-pool (function).
free-worker (function).
handle-connection-closed (function).
host-name (generic reader).
(setf host-name) (generic writer).
id (generic reader).
idle-workers (generic reader).
(setf idle-workers) (generic writer).
lock (generic reader).
make-worker-pool (function).
master-host-name (generic reader).
master-swank-port (generic reader).
no-workers-p (function).
port (generic reader).
(setf port) (generic writer).
port (type).
reconnect-worker (generic function).
reconnect-workers (function).
record-async-result (function).
record-repeated-result (function).
remove-eval (function).
repeated-eval (class).
repeated-work-form (function).
repeatedly-evaluate (function).
replay-forms (generic reader).
(setf replay-forms) (generic writer).
replay-forms-lock (generic reader).
results (generic reader).
(setf results) (generic writer).
results-available (generic reader).
results-position (generic reader).
(setf results-position) (generic writer).
send-many-results (function).
set-worker (generic reader).
(setf set-worker) (generic writer).
state (generic reader).
(setf state) (generic writer).
state-counter (generic reader).
(setf state-counter) (generic writer).
unevaluated-replay-forms (function).
worker (class).
worker-available (generic reader).
worker-counts (function).
workers (generic reader).
Definitions are sorted by export status, category, package, and then by lexicographic order.
Makes Swank connections to all the workers in HOST/PORT-ALIST and returns a WORKER-POOL containing them. HOST/PORT-ALIST is a list of (host-name . port) pairs. MASTER-HOST-NAME and MASTER-SWANK-PORT are a host name and Swank port that workers can use to return results to the master.
Closes the Swank connections of all connected workers in WORKER-POOL.
Evaluates FORM on all workers in WORKER-POOL. When RESULT-DONE is non-NIL, it must be a function of two arguments. In this case RESULT-DONE is called as each worker returns a result with two arguments, a non-negative integer representing the order in which the work was dispatched and the worker’s result. If REPLAY-REQUIRED is true, which is the default, FORM will be remembered and evaluated again for side effects on any new worker that joins POOL.
Evaluates FORM, which must return a function of no arguments, on WORKER-COUNT workers in WORKER-POOL, then arranges for the workers to repeatedly call the function to create RESULT-COUNT results, which are returned in a list. WORKER-COUNT defaults to the number of workers in WORKER-POOL.
Evaluates FORM, which must return a function of one argument, on WORKER-COUNT
workers in WORKER-POOL, then arranges for the workers to repeatedly call the
work function and send its results back to the master. The work function is
passed a state argument, initially set to INITIAL-STATE, that the master can
update asynchronously as it receives new results.
On the master, the work state is initialized to INITIAL-STATE and UPDATE-STATE is called repeatedly to process results received from the workers. UPDATE-STATE is called with two arguments, the current work state and a list containing all unprocessed work results. UPDATE-STATE should return three values: the new work state, a boolean indicating whether the computation should end, and a boolean indicating whether the latest work state should be distributed to workers. When UPDATE-STATE’s second return value is true, EVAL-REPEATEDLY-ASYNC-STATE tells the workers to stop and returns the latest work state.
Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. Results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is provided, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are the position of the work in LIST and the worker’s result.
Traverses LIST, calling MAKE-WORK on each element to create a form that is
then passed to a remote worker in WORKER-POOL for evaluation. As results are
returned, REDUCER, a binary function, is used to combine successive results in
the manner of REDUCE. INITIAL-VALUE is used as the starting value for the
reduction computation. The form
(parallel-reduce worker-pool make-work list initial-value reducer)
is equivalent to
(reduce reducer
(mapcar (lambda (x) (eval (funcall make-work x))) list)
:initial-value initial-value)
Returns the total number of workers in WORKER-POOL.
worker-pool) &key) ¶worker-pool) stream) ¶Prints WORKER-POOL to STREAM. This function runs without locking WORKER-POOL, so it may output inconsistent information.
A pool of Swank workers to which Lisp forms can be sent for evaluation.
The worker pool’s ID.
symbol
(com.google.base:missing-argument)
:id
id.
This slot is read-only.
Host name of the master. Used by workers to return results to the master.
string
(com.google.base:missing-argument)
:master-host-name
This slot is read-only.
Port on the master on which a Swank server is listening. Used by workers to return results to the master.
swank-crew::port
(com.google.base:missing-argument)
:master-swank-port
This slot is read-only.
Vector containing all workers in the worker pool.
vector
#()
Lock protecting IDLE-WORKERS, WORKER-AVAILABLE, and DISCONNECTING.
(bordeaux-threads:make-lock "worker pool")
lock.
This slot is read-only.
List of currently idle workers.
list
(quote nil)
Condition signaled when a worker becomes idle.
(bordeaux-threads:make-condition-variable)
This slot is read-only.
Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.
Lock that protects REPLAY-FORMS.
(bordeaux-threads:make-lock "replay forms lock")
This slot is read-only.
List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.
list
(quote nil)
Seconds between attempts to reconnect workers.
List containing an EVALUATION instance for each running computation.
Lock protecting access to *EVALS*.
Counter used to generate a unique ID for EVALUATION instances.
Lock protecting access to *EVALUATION-ID*.
The last form evaluated by the worker.
The value of *RANDOM-STATE* right before the worker started evaluating *LAST-FORM-EVALED*.
Thunk created by the current invocation of REPEATEDLY-EVALUATE to produce results for an EVAL-FORM-REPEATEDLY request on the master. This variable is useful for debugging.
Mapping from worker pool IDs to the number of replay forms we have evaluated on this client for that pool.
Lock protecting access to *REPLAY-FORMS-COUNTS*.
Mapping from worker pool IDs to active worker pools.
Lock protecting access to *WORKER-POOLS*.
Adds EVAL to the list of in-progress computations.
Adds FORM to the set of forms that need to be evaluated on a new worker when it joins WORKER-POOL.
Allocates and returns an idle worker from WORKER-POOL that is connected. If WORKER-POOL is being shut down, then NIL is returned.
Allocates a connected worker from WORKER-POOL and sends it FORM to evaluate. Returns the worker that was successfully allocated, or NIL if WORKER-POOL is shutting down. SET-WORKER is a function that is called to tell the WORKER-DONE continuation what worker is evaluating FORM.
Evaluates FORM, which must return a work function of one argument, then calls that function repeatedly to produce results, each time passing it the current computation state. At first this state is INITIAL-STATE, but the master may update the state asynchronously. Each work result is sent to host MASTER-HOST-NAME by making a Swank connection to port MASTER-SWANK-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to process results and on the worker to update the state.
Handles incoming results for ASYNC-EVAL by calling UPDATE-STATE whenever ASYNC-EVAL holds unprocessed results. UPDATE-STATE is called with two arguments, the work state and a list of the unprocessed results.
Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the async evaluation request identified by ID.
Ensures that the current client is up to date, that it has evaluated all POOL-COUNT replay forms associated with the pool identified by WORKER-POOL-ID. If it is necessary to evaluate forms in order to catch up, they are fetched by making a Swank connection to host MASTER-HOST-NAME on port MASTER-SWANK-PORT.
Sets all debugging global variables to NIL.
Returns an expression that when evaluated returns the result of evaluating FORM. In addition, the returned expression arranges to update the values of *LAST-FORM-EVALED* and *LAST-RANDOM-STATE* for ease of debugging.
Traverses LIST, calling MAKE-WORK on each element to create a form that is then passed to a remote worker in WORKER-POOL for evaluation. When RETAIN-WORKERS is true, each form is evaluated on a unique worker. Otherwise, workers are reused and may process multiple forms. In either case, the results of evaluating each form are collected into a list, which is returned when every remote worker is done. If RESULT-DONE is not NIL, then it must be a function of two arguments. In this case RESULT-DONE is called on the master as each worker returns a result. The arguments to RESULT-DONE are an integer counter, indicating the work form’s position in LIST, and the result of evaluating the form. REPLAY-REQUIRED indicates whether new workers will have to perform the work generated by MAKE-WORK before being considered to be caught up.
Returns a form that when evaluated on a worker in WORKER-POOL ensures that the worker is caught up then evaluates FORM and returns its result. REPLAY-REQUIRED indicates whether new workers must evaluate FORM before being considered to be caught up.
Iterates over the members of LIST calling MAKE-WORK on each one. If RESULT-DONE is not NIL, it must be a function of two arguments. In this case, after each application of MAKE-WORK to a LIST member, RESULT-DONE is called with the member’s position in LIST and MAKE-WORK’s result.
Evaluates FORM, but first ensures that this worker has evaluated all POOL-COUNT replay forms associated with the worker pool identified by WORKER-POOL-ID on the master. When catching up is required, fetches forms by making a Swank connection to host MASTER-HOST-NAME on port MASTER-SWANK-PORT. REPLAY-REQUIRED indicates whether FORM may need to be replayed in order to bring a worker up to date.
Fetches from the master and then evaluates all forms required to catch up with other workers in the pool identified by WORKER-POOL-ID on the master. All replay forms after the first LOCAL-COUNT forms are fetched by making a Swank connection to host MASTER-HOST-NAME on port MASTER-SWANK-PORT.
Returns the running EVAL instance identified by ID, or NIL if no computation with that ID is currently running.
Returns the worker pool identified by WORKER-POOL-ID.
Changes the state of WORKER to idle, so that it’s available for allocation.
Called when the connection to DISCONNECTED-WORKER, a member of WORKER-POOL, is closed because of a call to SLIME-CLOSE, the death of the worker, or because of a communications error. Moves all uncompleted work intended for DISCONNECTED-WORKER to another idle connected worker in WORKER-POOL.
Returns T if WORKER-POOL is NIL or contains no workers.
Reconnects disconnected workers in WORKER-POOL.
Stores RESULT as one of the values produced by the async evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call ASYNC-RESULT with additional results.
Stores RESULT as one of the values produced by the repeated evaluation identified by ID. Returns a boolean indicating whether the worker should continue to call RECORD-REPEATED-RESULT with additional results.
Removes EVAL from the list of in-progress computations.
Returns a form for evaluation on a client of WORKER-POOL that ensures the client is caught up and then evaluates FORM for the repeated evaluation request identified by ID.
Evaluates FORM, which must return a function of no arguments, then calls that function repeatedly to produce results. Each result is sent to host MASTER-HOST-NAME by making a Swank connection on port MASTER-SWANK-PORT and remotely evaluating an expression that records the result. ID is used on the master machine to correctly record the result.
Creates a Slime connection to host MASTER-HOST-NAME on port MASTER-SWANK-PORT, then repeatedly calls SEND-RESULT with the new connection as argument. Returns when SEND-RESULT returns NIL.
For a worker that has executed EVALUATED-COUNT of the replay forms associated with the worker-pool identified by WORKER-POOL-ID, returns a list of the forms the worker needs to evaluate in order to be completely up to date.
Returns the number of idle, busy, and disconnected workers in WORKER-POOL. This function executes without locking WORKER-POOL, so it may return inconsistent information.
worker-pool)) ¶Vector containing all workers in the worker pool.
Creates a connection to a worker’s Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.
connect-info) close-handler) ¶worker-pool)) ¶worker-pool)) ¶Set non-NIL when the worker pool is being torn down to tell the reconnector thread it should exit.
evaluation)) ¶evaluation)) ¶Is the computation done?
done.
evaluation)) ¶Condition variable notified when computation is done.
connect-info)) ¶connect-info)) ¶Host the worker is running on.
evaluation)) ¶Unique ID for this running evaluation request.
id.
worker-pool)) ¶The worker pool’s ID.
id.
worker-pool)) ¶worker-pool)) ¶List of currently idle workers.
evaluation)) ¶Lock protecting access to DONE, DONE-CONDITION, and other mutable slots.
lock.
worker-pool)) ¶Lock protecting IDLE-WORKERS, WORKER-AVAILABLE, and DISCONNECTING.
lock.
worker-pool)) ¶Host name of the master. Used by workers to return results to the master.
worker-pool)) ¶Port on the master on which a Swank server is listening. Used by workers to return results to the master.
connect-info)) ¶connect-info)) ¶Port on which the worker’s Swank server is listening for connections.
port.
Reconnects to a Swank server using information in CONNECT-INFO. Passes the thunk CLOSE-HANDLER to SWANK-CLIENT:SLIME-CONNECT, so that it is invoked when the connection closes.
connect-info) close-handler) ¶worker-pool)) ¶worker-pool)) ¶List containing all forms passed to EVAL-FORM-ALL-WORKERS that need to be replayed on new workers when they join the pool.
worker-pool)) ¶Lock that protects REPLAY-FORMS.
async-eval)) ¶async-eval)) ¶List holding returned, but unprocessed, results.
repeated-eval)) ¶Vector holding returned results.
async-eval)) ¶Condition notified when new results are available.
repeated-eval)) ¶repeated-eval)) ¶Position where the next result will be recorded.
async-eval)) ¶async-eval)) ¶State of the asynchronous computation, updated from results.
async-eval)) ¶async-eval)) ¶Counter incremented each time STATE is updated.
worker-pool)) ¶Condition signaled when a worker becomes idle.
worker-pool)) ¶Vector containing all workers in the worker pool.
Data needed to process incoming asynchronous evaluation results.
List holding returned, but unprocessed, results.
list
(quote nil)
Condition notified when new results are available.
(bordeaux-threads:make-condition-variable)
This slot is read-only.
State of the asynchronous computation, updated from results.
(com.google.base:missing-argument)
:state
Counter incremented each time STATE is updated.
(integer 0)
0
Information needed when connecting to a worker’s Swank server.
Host the worker is running on.
string
(com.google.base:missing-argument)
:host-name
Stores the data needed to process incoming evaluation results.
(setf done).
done.
done-condition.
id.
lock.
Unique ID for this running evaluation request.
integer
(bordeaux-threads:with-lock-held (swank-crew::*evaluation-id-lock*) (incf swank-crew::*evaluation-id*))
:id
id.
This slot is read-only.
Lock protecting access to DONE, DONE-CONDITION, and other mutable slots.
(bordeaux-threads:make-lock "evaluation")
lock.
This slot is read-only.
Condition variable notified when computation is done.
(bordeaux-threads:make-condition-variable)
This slot is read-only.
Stores the data needed to process an incoming repeated eval result.
Vector holding returned results.
simple-vector
(com.google.base:missing-argument)
:results
This slot is read-only.
Position where the next result will be recorded.
swank-crew::vector-index
0
A remote Lisp running a Swank server.
Information used when connecting to this worker.
swank-crew::connect-info
(com.google.base:missing-argument)
:connect-info
This slot is read-only.
Lock protecting SET-WORKER and CONNECTION.
(bordeaux-threads:make-lock "worker")
lock.
This slot is read-only.
Function called to record which worker is evaluating a form.
function
(function identity)
When non-NIL, an open Swank connection to the worker.
(or null swank-client:swank-connection)
| Jump to: | (
A C D E F G H I L M N P R S U W |
|---|
| Jump to: | (
A C D E F G H I L M N P R S U W |
|---|
| Jump to: | *
+
C D H I L M P R S W |
|---|
| Jump to: | *
+
C D H I L M P R S W |
|---|
| Jump to: | A C E F M P R S T W |
|---|
| Jump to: | A C E F M P R S T W |
|---|