Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the monomyth Reference Manual, version 0.3.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 14:27:23 2020 GMT+0.
• Introduction | What monomyth is all about | |
• Systems | The systems documentation | |
• Modules | The modules documentation | |
• Files | The files documentation | |
• Packages | The packages documentation | |
• Definitions | The symbols documentation | |
• Indexes | Concepts, functions, variables and data types |
Monomyth is a distributed data processing system built using Common Lisp and based heavily on Broadway. It is designed to split the messaging systems into two, one defined and controlled by Monomyth, and one defined and controlled by the user. The messaging controlled by the user pertains only to data, which moves between persisted data streams and node threads. The structure and manipulation of this data is largely defined by the user (though there are data stream specific aspects certain node types might handle). Monomyth itself handles all aspects of system orchestration via the Monomyth Orchestration Protocol (MMOP). The work itself is done on a group of distributed workers that use concurrent, user defined nodes to process the data and are controlled by a single master server.
Design documentation can be found at design/doc.org
.
To run the tests you will need to have roswell
and qlot
installed, the tests
have been verified on SBCL 2.0.3.
Then perform the following:
source test_env.sh
docker-compose up -d
qlot install
qlot exec ./bin/test.ros
The example right now is pretty arbitrary, both in set up and the computations
involved.
In one process start qlot exec ./bin/example-master.ros
, and then in a few
others run qlot exec ./bin/example-worker.ros
.
Once all the workers have connected to the master (you will see log entries),
press the return button in the first process.
This project has its basic architecture set up, but lacks most of the functionality needed for a 1.0 release.
The features likely to be targeted for a 1.0 release:
Next: Modules, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
• The monomyth system |
Paul Ricks
MPL 2.0
A distributed data processing library for CL
0.3.1
monomyth.asd (file)
src (module)
Modules are listed depth-first from the system components tree.
• The monomyth/src module |
monomyth (system)
src/
Files are sorted by type and then listed depth-first from the systems components trees.
• Lisp files |
Next: The monomyth/src/node-recipe․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
monomyth.asd
monomyth (system)
Next: The monomyth/src/mmop․lisp file, Previous: The monomyth․asd file, Up: Lisp files [Contents][Index]
src (module)
src/node-recipe.lisp
Next: The monomyth/src/mmop-worker․lisp file, Previous: The monomyth/src/node-recipe․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/mmop.lisp
handle-libzmq-error (macro)
Next: The monomyth/src/mmop-master․lisp file, Previous: The monomyth/src/mmop․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/mmop-worker.lisp
Next: The monomyth/src/mmop-control․lisp file, Previous: The monomyth/src/mmop-worker․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/mmop-master.lisp
Next: The monomyth/src/node․lisp file, Previous: The monomyth/src/mmop-master․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/mmop-control.lisp
Next: The monomyth/src/rmq-node․lisp file, Previous: The monomyth/src/mmop-control․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/node.lisp
Next: The monomyth/src/rmq-node-recipe․lisp file, Previous: The monomyth/src/node․lisp file, Up: Lisp files [Contents][Index]
node.lisp (file)
src (module)
src/rmq-node.lisp
Next: The monomyth/src/worker․lisp file, Previous: The monomyth/src/rmq-node․lisp file, Up: Lisp files [Contents][Index]
node-recipe.lisp (file)
src (module)
src/rmq-node-recipe.lisp
Next: The monomyth/src/rmq-worker․lisp file, Previous: The monomyth/src/rmq-node-recipe․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/worker.lisp
Next: The monomyth/src/master․lisp file, Previous: The monomyth/src/worker․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/rmq-worker.lisp
Next: The monomyth/src/dsl․lisp file, Previous: The monomyth/src/rmq-worker․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/master.lisp
Previous: The monomyth/src/master․lisp file, Up: Lisp files [Contents][Index]
src (module)
src/dsl.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
Next: The monomyth/mmop package, Previous: Packages, Up: Packages [Contents][Index]
node-recipe.lisp (file)
Next: The monomyth/mmop-worker package, Previous: The monomyth/node-recipe package, Up: Packages [Contents][Index]
mmop.lisp (file)
handle-libzmq-error (macro)
Next: The monomyth/mmop-master package, Previous: The monomyth/mmop package, Up: Packages [Contents][Index]
mmop-worker.lisp (file)
mmop-w
Next: The monomyth/mmop-control package, Previous: The monomyth/mmop-worker package, Up: Packages [Contents][Index]
mmop-master.lisp (file)
mmop-m
Next: The monomyth/node package, Previous: The monomyth/mmop-master package, Up: Packages [Contents][Index]
mmop-control.lisp (file)
mmop-c
Next: The monomyth/rmq-node package, Previous: The monomyth/mmop-control package, Up: Packages [Contents][Index]
node.lisp (file)
Next: The monomyth/rmq-node-recipe package, Previous: The monomyth/node package, Up: Packages [Contents][Index]
rmq-node.lisp (file)
Next: The monomyth/worker package, Previous: The monomyth/rmq-node package, Up: Packages [Contents][Index]
rmq-node-recipe.lisp (file)
Next: The monomyth/rmq-worker package, Previous: The monomyth/rmq-node-recipe package, Up: Packages [Contents][Index]
worker.lisp (file)
Next: The monomyth/master package, Previous: The monomyth/worker package, Up: Packages [Contents][Index]
rmq-worker.lisp (file)
Next: The monomyth/dsl package, Previous: The monomyth/rmq-worker package, Up: Packages [Contents][Index]
master.lisp (file)
Previous: The monomyth/master package, Up: Packages [Contents][Index]
dsl.lisp (file)
Definitions are sorted by export status, category, package, and then by lexicographic order.
• Exported definitions | ||
• Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
• Exported constants | ||
• Exported special variables | ||
• Exported macros | ||
• Exported functions | ||
• Exported generic functions | ||
• Exported conditions | ||
• Exported structures | ||
• Exported classes |
Next: Exported special variables, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-worker.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-worker.lisp (file)
Next: Exported macros, Previous: Exported constants, Up: Exported definitions [Contents][Index]
mmop.lisp (file)
mmop.lisp (file)
Next: Exported functions, Previous: Exported special variables, Up: Exported definitions [Contents][Index]
Defines all classes, methods, and functions for a new node type.
dsl.lisp (file)
Takes a list of plist (:name :fn :batch-size) and turns them into rmq nodes
that work in sequential order.
All recipes are also set up to be loaded into master server via the add-recipes method.
dsl.lisp (file)
Next: Exported generic functions, Previous: Exported macros, Up: Exported definitions [Contents][Index]
acks a message as complete
rmq-node.lisp (file)
adds a recipe to the master records
master.lisp (file)
uses a master to tell a worker to shutdown via MMOP.
No state in the master is currently changes, returns t if the call seems to have
been sent, nil otherwise
master.lisp (file)
attempts to start a node of type-id on one of the masters workers. returns t if it works, nil otherwise
master.lisp (file)
turns the linear list of edges into queue names
dsl.lisp (file)
rmq-node.lisp (file)
rmq-worker.lisp (file)
turns a serialized buffer back into a recipe
node-recipe.lisp (file)
gets a message off the source queue and changes the message to be a string
(as opposed to a byte array)
return :success t with the :result if things go well
rmq-node.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
takes a bunch of symbols and combines them
dsl.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
nacks a message as incomplete, re-queues message if asked to
rmq-node.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
pulls down a message designed for the control server and attempts to translate it into the correct adt
mmop-control.lisp (file)
pulls down a message designed for the master router socket and attempts to translate it into an equivalent struct
mmop-master.lisp (file)
Pulls down all message frames
mmop.lisp (file)
pulls down a message designed for the worker dealer socket and attempts to translate it into an equivalent struct
mmop-worker.lisp (file)
mmop-master.lisp (file)
rmq-node.lisp (file)
rmq-node.lisp (file)
main event loop for the worker
worker.lisp (file)
sends a message to the specified queue
rmq-node.lisp (file)
translates a message struct into frames, and then sends them
mmop.lisp (file)
Helper function that sends a set of frames as single message
mmop.lisp (file)
turns the recipe into a buffer acceptable to ZMQ
node-recipe.lisp (file)
builds a new connection, sets up the socket, and logs in defaults are the local rabbit-mq defaults
rmq-node.lisp (file)
mmop-master.lisp (file)
starts up all worker threads and the router loop for load balancing
master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
master.lisp (file)
master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
Next: Exported conditions, Previous: Exported functions, Up: Exported definitions [Contents][Index]
add all recipes created by the define system macro
master.lisp (file)
uses a worker and recipe to build a new node
worker.lisp (file)
takes a message struct and builds a list of zmq frames to be sent
mmop.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-worker.lisp (file)
mmop-worker.lisp (file)
actions to take if the returned plist contains :success false
expects an explanation under :error in the result
the step can be :pull, :transform, or :place
the result is the full payload sent by the last step
node.lisp (file)
rmq-node.lisp (file)
mmop.lisp (file)
mmop.lisp (file)
uses the node type to name the fail queue in a deterministic manor
rmq-node-recipe.lisp (file)
takes a recipe and produces a unique node name
node-recipe.lisp (file)
node.lisp (file)
node.lisp (file)
the batch size that is passed directly to the node if not set uses the default (10)
node-recipe.lisp (file)
node ’type’, often used to name nodes
node-recipe.lisp (file)
number of items to pull in pull-items at a time
node.lisp (file)
name of the node
node.lisp (file)
the node type corresponds to the node recipe type
node.lisp (file)
places the finished items on the message bus
takes the entire payload sent by transorm
assumes that the items are passed under :items
should return a plist with one of the slots as :success
node.lisp (file)
rmq-node.lisp (file)
tells the node to pull count items from the message bus should return a plist with one of the slots as :success
node.lisp (file)
rmq-node.lisp (file)
automatically generated reader method
rmq-node-recipe.lisp (file)
automatically generated reader method
rmq-node-recipe.lisp (file)
rabbit mq host
rmq-worker.lisp (file)
automatically generated reader method
rmq-worker.lisp (file)
rabbit mq port
rmq-worker.lisp (file)
rabbit mq username
rmq-worker.lisp (file)
runs an entire operation start to finish
node.lisp (file)
graceful shutdown of the node
node.lisp (file)
closes the channel and then destroys the connections.
note that this means that once an rmq-node is shutdown, it cannot be started up again
rmq-node.lisp (file)
starts the worker processes, by the time this method is done it should be okay start a node
worker.lisp (file)
performs any initial start up to ensure the node is working as corrected. The build worker thread option exists for testing purposes
node.lisp (file)
opens a channel using the nodes connections after setting up the socket.
also ensures all three queues are up and sets up basic consume for the source queue
rmq-node.lisp (file)
stops the worker processes and frees all resources and connections
worker.lisp (file)
the core transform function
node.lisp (file)
extracts the items from the pull step (provided it was successful) and applies the transform function to each one
node.lisp (file)
rmq-node.lisp (file)
a unique worker name set before startup
worker.lisp (file)
Next: Exported structures, Previous: Exported generic functions, Up: Exported definitions [Contents][Index]
an error that happens in the mmop protocol
mmop.lisp (file)
error (condition)
:message
(quote (error "mmop-error message must be set"))
mmop-error/message (generic function)
:version
(quote "undefined")
mmop-error/version (generic function)
an internal node error, handled by run-iteration
node.lisp (file)
error (condition)
the step the node failed on
should be :place, :transform, or :pull if handle failure will take it
:step
(quote (error "node error step must be set"))
node-error/step (generic function)
:message
(quote (error "node error message must be set"))
node-error/message (generic function)
the items to be reprocessed
:items
(quote nil)
node-error/items (generic function)
Next: Exported classes, Previous: Exported conditions, Up: Exported definitions [Contents][Index]
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
json-info-response-v0%0 (function)
(setf json-info-response-v0%0) (function)
string
(error "unspecified field.")
json-info-response-v0%1 (function)
(setf json-info-response-v0%1) (function)
mmop-control.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
json-info-response-v0%0 (function)
(setf json-info-response-v0%0) (function)
the master system only has two fields, a table of worker identifiers to worker infos, and a table of node type symbols to node recipes
master.lisp (file)
structure-object (structure)
(stmx::tstruct-slot-initialize (quote monomyth/master:master) (quote monomyth/master::workers) (quote (make-instance (quote stmx.util:thash-table) :test (quote equal))))
%stmx-impl/tstruct/master-workers (function)
(setf %stmx-impl/tstruct/master-workers) (function)
(stmx::tstruct-slot-initialize (quote monomyth/master:master) (quote monomyth/master::recipes) (quote (make-instance (quote stmx.util:thash-table) :test (quote equal))))
%stmx-impl/tstruct/master-recipes (function)
(setf %stmx-impl/tstruct/master-recipes) (function)
(stmx::tstruct-slot-initialize (quote monomyth/master:master) (quote monomyth/master::context) (quote (pzmq:ctx-new)))
%stmx-impl/tstruct/master-context (function)
(setf %stmx-impl/tstruct/master-context) (function)
(stmx:transactional t)
(stmx::tstruct-slot-initialize (quote monomyth/master:master) (quote monomyth/master::running) (quote t))
%stmx-impl/tstruct/master-running (function)
(setf %stmx-impl/tstruct/master-running) (function)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
ping-v0%0 (function)
(setf ping-v0%0) (function)
mmop-control.lisp (file)
sent-mmop (structure)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
pong-v0%0 (function)
(setf pong-v0%0) (function)
mmop-control.lisp (file)
received-mmop (structure)
mmop-worker.lisp (file)
algebraic-data-type (structure)
mmop-master.lisp (file)
algebraic-data-type (structure)
mmop-control.lisp (file)
algebraic-data-type (structure)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
recipe-info-v0%0 (function)
(setf recipe-info-v0%0) (function)
mmop-control.lisp (file)
sent-mmop (structure)
mmop-worker.lisp (file)
algebraic-data-type (structure)
mmop-master.lisp (file)
algebraic-data-type (structure)
mmop-control.lisp (file)
algebraic-data-type (structure)
mmop-worker.lisp (file)
received-mmop (structure)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
shutdown-worker-v0%0 (function)
(setf shutdown-worker-v0%0) (function)
mmop-worker.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-failure-v0%0 (function)
(setf start-node-failure-v0%0) (function)
string
(error "unspecified field.")
start-node-failure-v0%1 (function)
(setf start-node-failure-v0%1) (function)
string
(error "unspecified field.")
start-node-failure-v0%2 (function)
(setf start-node-failure-v0%2) (function)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
start-node-failure-v0%0 (function)
(setf start-node-failure-v0%0) (function)
string
(error "unspecified field.")
start-node-failure-v0%1 (function)
(setf start-node-failure-v0%1) (function)
string
(error "unspecified field.")
start-node-failure-v0%2 (function)
(setf start-node-failure-v0%2) (function)
string
(error "unspecified field.")
start-node-failure-v0%3 (function)
(setf start-node-failure-v0%3) (function)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-request-failure-v0%0 (function)
(setf start-node-request-failure-v0%0) (function)
string
(error "unspecified field.")
start-node-request-failure-v0%1 (function)
(setf start-node-request-failure-v0%1) (function)
integer
(error "unspecified field.")
start-node-request-failure-v0%2 (function)
(setf start-node-request-failure-v0%2) (function)
mmop-control.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
start-node-request-failure-v0%0 (function)
(setf start-node-request-failure-v0%0) (function)
integer
(error "unspecified field.")
start-node-request-failure-v0%1 (function)
(setf start-node-request-failure-v0%1) (function)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-request-success-v0%0 (function)
(setf start-node-request-success-v0%0) (function)
mmop-control.lisp (file)
received-mmop (structure)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
start-node-request-v0%0 (function)
(setf start-node-request-v0%0) (function)
string
(error "unspecified field.")
start-node-request-v0%1 (function)
(setf start-node-request-v0%1) (function)
mmop-control.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-request-v0%0 (function)
(setf start-node-request-v0%0) (function)
mmop-worker.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-success-v0%0 (function)
(setf start-node-success-v0%0) (function)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
start-node-success-v0%0 (function)
(setf start-node-success-v0%0) (function)
string
(error "unspecified field.")
start-node-success-v0%1 (function)
(setf start-node-success-v0%1) (function)
mmop-worker.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
start-node-v0%0 (function)
(setf start-node-v0%0) (function)
monomyth/node-recipe:node-recipe
(error "unspecified field.")
start-node-v0%1 (function)
(setf start-node-v0%1) (function)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
start-node-v0%0 (function)
(setf start-node-v0%0) (function)
monomyth/node-recipe:node-recipe
(error "unspecified field.")
start-node-v0%1 (function)
(setf start-node-v0%1) (function)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
stop-worker-request-failure-v0%0 (function)
(setf stop-worker-request-failure-v0%0) (function)
string
(error "unspecified field.")
stop-worker-request-failure-v0%1 (function)
(setf stop-worker-request-failure-v0%1) (function)
integer
(error "unspecified field.")
stop-worker-request-failure-v0%2 (function)
(setf stop-worker-request-failure-v0%2) (function)
mmop-control.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
stop-worker-request-failure-v0%0 (function)
(setf stop-worker-request-failure-v0%0) (function)
integer
(error "unspecified field.")
stop-worker-request-failure-v0%1 (function)
(setf stop-worker-request-failure-v0%1) (function)
mmop-master.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
stop-worker-request-success-v0%0 (function)
(setf stop-worker-request-success-v0%0) (function)
mmop-control.lisp (file)
received-mmop (structure)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
stop-worker-request-v0%0 (function)
(setf stop-worker-request-v0%0) (function)
string
(error "unspecified field.")
stop-worker-request-v0%1 (function)
(setf stop-worker-request-v0%1) (function)
mmop-control.lisp (file)
sent-mmop (structure)
string
(error "unspecified field.")
stop-worker-request-v0%0 (function)
(setf stop-worker-request-v0%0) (function)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
worker-info-v0%0 (function)
(setf worker-info-v0%0) (function)
mmop-control.lisp (file)
sent-mmop (structure)
mmop-worker.lisp (file)
sent-mmop (structure)
mmop-master.lisp (file)
received-mmop (structure)
string
(error "unspecified field.")
worker-ready-v0%0 (function)
(setf worker-ready-v0%0) (function)
Previous: Exported structures, Up: Exported definitions [Contents][Index]
base node class for the monomyth flow system
node.lisp (file)
transactional-object (class)
rmq-node (class)
name of the node
:name
(format nil "node-~a" (uuid:make-v4-uuid))
node/node-name (generic function)
the node type corresponds to the node recipe type
:type
(error "node type must be set")
node/type (generic function)
number of items to pull in pull-items at a time
:batch-size
10
node/batch-size (generic function)
transactional condition that allows for safe shutdown
t
node/running (generic function)
(setf node/running) (generic function)
everything the systems needs to make a new node
node-recipe.lisp (file)
standard-object (class)
rmq-node-recipe (class)
node ’type’, often used to name nodes
:type
(error "recipe type must be set")
node-recipe/type (generic function)
the batch size that is passed directly to the node if not set uses the default (10)
:batch-size
10
node-recipe/batch-size (generic function)
a node type specially designed to work with rabbit mq
rmq-node.lisp (file)
node (class)
the rmq connection, there should only be one per machine. Due to the library we are using, there will be one per node
:conn
(error "connection must be supplied")
rmq-node/conn (generic function)
:exchange
""
rmq-node/exchange (generic function)
:source
(error "source queue must be set")
rmq-node/source-queue (generic function)
:dest
(error "destination queue must be set")
rmq-node/dest-queue (generic function)
:fail
(error "failure queue must be set")
rmq-node/fail-queue (generic function)
recipe with the added fields needed to build an rmq-node
rmq-node-recipe.lisp (file)
node-recipe (class)
:source
(error "source queue must be set")
rmq-node-recipe/source-queue (generic function)
:dest
(error "destination queue must be set")
rmq-node-recipe/dest-queue (generic function)
a worker designed to work directly with rabbit mq nodes
rmq-worker.lisp (file)
worker (class)
rabbit mq host
:host
(error "rmq host must be set")
rmq-worker/host (generic function)
rabbit mq port
:port
(error "rmq port must be set")
rmq-worker/port (generic function)
rabbit mq username
:username
(error "rmq username must be set")
rmq-worker/username (generic function)
:password
(error "rmq password")
rmq-worker/password (generic function)
defines a single machine with its own threads, nodes, and connections
worker.lisp (file)
standard-object (class)
rmq-worker (class)
the zmq context used for the MMOP
(pzmq:ctx-new)
worker/context (generic function)
the zmq dealer socket used for the MMOP
worker/socket (generic function)
(setf worker/socket) (generic function)
a unique worker name set before startup
(monomyth/worker::name-worker)
worker/name (generic function)
the MMOP version the worker is using
monomyth/mmop:*mmop-v0*
worker/mmop-version (generic function)
a hash table of node names to nodes
(make-hash-table :test (function equal))
worker/nodes (generic function)
Previous: Exported definitions, Up: Definitions [Contents][Index]
• Internal special variables | ||
• Internal macros | ||
• Internal functions | ||
• Internal generic functions | ||
• Internal structures |
Next: Internal macros, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
rmq-node.lisp (file)
master.lisp (file)
dsl.lisp (file)
rmq-node.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
dsl.lisp (file)
master.lisp (file)
Next: Internal functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
wraps a zmq call in an error handler
mmop.lisp (file)
wraps a rmq call to look for a standard error
rmq-node.lisp (file)
Next: Internal generic functions, Previous: Internal macros, Up: Internal definitions [Contents][Index]
master.lisp (file)
master.lisp (file)
(setf %stmx-impl/notx/master-running) (setf expander)
master.lisp (file)
%stmx-impl/notx/master-running (function)
master.lisp (file)
master.lisp (file)
(setf %stmx-impl/swtx/master-running) (setf expander)
master.lisp (file)
%stmx-impl/swtx/master-running (function)
master.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
adds a worker info and id to the master
master.lisp (file)
helper function that constructs the error
rmq-node.lisp (file)
master.lisp (file)
master.lisp (file)
takes two type count plists and combines the counts
master.lisp (file)
Sends a request failure message to the client with the supplied message.
master.lisp (file)
master.lisp (file)
rmq-node.lisp (file)
master.lisp (file)
Internal function that creates the rmq node top level forms.
dsl.lisp (file)
determines the best worker id for the recipe type
master.lisp (file)
finds the worker id with the smallest number of those nodes running
master.lisp (file)
drops unneeded frames and sends them to the client
master.lisp (file)
constructs the message frames and sends them on to the thread
master.lisp (file)
translates all workers into plists with node information
master.lisp (file)
translates a worker type count map object into an equivalent list of plists
master.lisp (file)
handles a specific message for the worker, return t if the worker should continue
worker.lisp (file)
handles a specific message for the master
master.lisp (file)
wraps the pull msg call in an appropriate handler
master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
master.lisp (file)
master.lisp (file)
(setf master-running) (setf expander)
master.lisp (file)
master-running (function)
creates the name that zmq uses for routing
worker.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
takes a master object and produces an fset map that links each recipe type to a plist with :running and :queued
master.lisp (file)
takes a worker-info and produces an fset map that links each recipe type to a plist with :running and :queued
master.lisp (file)
takes a worker-info and produces an fset map that links each recipe type to a plist with :queued
master.lisp (file)
takes a worker-info and produces an fset map that links each recipe type to a plist with :running
master.lisp (file)
mmop-master.lisp (file)
rmq-node.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
master.lisp (file)
responds to a worker-info message by pulling the info and turning it into json
master.lisp (file)
mmop-master.lisp (file)
starts up a handler thread listening for a router to send it messages
master.lisp (file)
Sends the start node request to a client with the supplied recipe
master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
mmop-worker.lisp (file)
mmop-master.lisp (file)
runs the thread router’s event loop in a new thread
master.lisp (file)
removes the record of the outstanding request and increments the type count for that client
master.lisp (file)
removes the record of the outstanding request
master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
mmop-control.lisp (file)
mmop-master.lisp (file)
calculates the total possible worker threads of that type
master.lisp (file)
attempts to translate the arg frames into MMOP/0 structs
mmop-worker.lisp (file)
attempts to translate the arg frames into MMOP/0 structs
mmop-master.lisp (file)
attempts to translate the arg frames into MMOP/0 adts
mmop-control.lisp (file)
master.lisp (file)
mmop-master.lisp (file)
mmop-master.lisp (file)
Next: Internal structures, Previous: Internal functions, Up: Internal definitions [Contents][Index]
node.lisp (file)
transactional condition that allows for safe shutdown
node.lisp (file)
the rmq connection, there should only be one per machine. Due to the library we are using, there will be one per node
rmq-node.lisp (file)
automatically generated reader method
rmq-node.lisp (file)
automatically generated reader method
rmq-node.lisp (file)
automatically generated reader method
rmq-node.lisp (file)
automatically generated reader method
rmq-node.lisp (file)
the zmq context used for the MMOP
worker.lisp (file)
the MMOP version the worker is using
worker.lisp (file)
a hash table of node names to nodes
worker.lisp (file)
the zmq dealer socket used for the MMOP
worker.lisp (file)
Previous: Internal generic functions, Up: Internal definitions [Contents][Index]
central structure passed through the node
rmq-node.lisp (file)
structure-object (structure)
rmq-message-body (function)
(setf rmq-message-body) (function)
rmq-message-delivery-tag (function)
(setf rmq-message-delivery-tag) (function)
the only info tracked by a worker is the number of worker nodes of each type and the number of outstanding requests of each type
master.lisp (file)
structure-object (structure)
(make-instance (quote stmx.util:thash-table) :test (quote equal))
worker-info-type-counts (function)
(setf worker-info-type-counts) (function)
(make-instance (quote stmx.util:thash-table) :test (quote equal))
worker-info-outstanding-request-counts (function)
(setf worker-info-outstanding-request-counts) (function)
Previous: Definitions, Up: Top [Contents][Index]
• Concept index | ||
• Function index | ||
• Variable index | ||
• Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
Jump to: | F L M |
---|
Jump to: | F L M |
---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
Jump to: | %
(
A B C D F G H J M N P R S T W |
---|
Jump to: | %
(
A B C D F G H J M N P R S T W |
---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
Jump to: | %
*
B C D E F H I M N O P R S T U W |
---|
Jump to: | %
*
B C D E F H I M N O P R S T U W |
---|
Previous: Variable index, Up: Indexes [Contents][Index]
Jump to: | C J M N P R S W |
---|
Jump to: | C J M N P R S W |
---|