The cl-simple-concurrent-jobs Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The cl-simple-concurrent-jobs Reference Manual

This is the cl-simple-concurrent-jobs Reference Manual, version 1.0.0, generated automatically by Declt version 2.3 "Robert April" on Wed Mar 14 03:32:52 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

cl-simple-concurrent-jobs

A simple API for running concurrent jobs in Common Lisp. cl-scj for short.

Dependencies

bordeaux-threads and chanl - for pain-free installation use quicklisp and something like (ql:quickload '(bordeaux-threads chanl))

Documentation

This is it. There's currently no other documentation. The focus here is 'simple' so a README should be enough to describe it. By example, no less.

Create a job executor

> (setf *je* (cl-scj:create-job-executor :num-threads 4))
#<CL-SIMPLE-CONCURENT-JOBS::JOBEXECUTOR {25842F19}>

At this point the threads have been created and are blocking, waiting for jobs. First let's create a way to create jobs easily

> (defun make-job (id)
    (let ((out *standard-output*))
      (lambda ()
        (format out "Starting very important job number ~d~%" id)
        (sleep (random 5))
        (format out "Finished very important job number ~d~%" id)
        (format out "Job ~d returning result ~d~%" id (/ id 100))
        (/ id 100))))
MAKE-JOB

We can test it

> (funcall (make-job 1000))
Starting very important job number 1000
Finished very important job number 1000
Job 1000 returning result 10
10

Add one to the executor

> (cl-scj:add-job *je* (make-job 0))
Starting very important job number 0
1
Finished very important job number 0
Job 0 returning result 0

That returned value of 1 is the number of total jobs the executor think it's supposed to have run. Now that it's completed its job, we can get the results:

> (cl-scj:join-results *je*)
(0)

Excellent. What about these 4 threads though?

> (setf *je* (cl-scj:create-job-executor :num-threads 4))
> (dotimes (x 20) (cl-scj:add-job *je* (make-job (+ 1 x))))
Starting very important job number 1
Starting very important job number 2
Starting very important job number 3
Starting very important job number 4
NIL
Finished very important job number Finished very important job number 21

Job Job 21 returning result  returning result 11//15000

Starting very important job number 5
Starting very important job number 6
Finished very important job number 4
Job 4Finished very important job number 3
 returning result Job 31 returning result /25
3/100
...
<snip>

That might take a while to finish. You should notice that only four jobs began concurrently (because we specified :threads 4). Afterwards, the ~~random sleep~~ important job time variation jumbled them up.

If you're quick, you can call join-results and see it block

> (cl-scj:join-results *je*)
(1/5 19/100 17/100 9/50 3/20 4/25 13/100 7/50 11/100 3/25 9/100 1/10 7/100 2/25 3/50 1/20 3/100 1/25 1/50 1/100 0)

The last thing to consider is how to stop the jobs (if you want to). First kick off an irritating number of jobs

> (dotimes (x 1000) (cl-scj:add-job *je* (make-job x)))
Starting very important job number Starting very important job number 01

Starting very important job number 2
Starting very important job number 3
NIL
...
<snip>

And after you have had enough of looking at the mangled *standard-output*

> (cl-scj:stop)
Finished very important job number 43
Job 43 returning result 43/100
Finished very important job number 40
Job 40 returning result 2/5
...
<snip>

The currently executing jobs will complete and push their results onto the return value of (cl-scj:join-results *je*) but that call will return immediately with whichever results it already has.

Finally, if you want to check if the results are available in a non-blocking-way you can try (cl-scj:has-all-results *je*) but note that if you call stop this will probably never be T unless you stopped it while all remaining jobs were executing.

Some miscellaneous notes:

If you're still confused please let me know why and I'll try to improve the documentation. Fixes, suggestions, requests, criticisms, enhancements and other contributions are all very welcome. See the contact section below if you're keen.

Why should I use it?

I'm not sure. I wrote it because I couldn't find something with such a simple API to do exactly what I wanted (dynamically execute jobs concurrently then give me all the results). If you know of a pre-existing or better system that does this with a simple API please let me know.

Is it safe? Will it work for me?

Maybe. It's working for me so far.

The Latest Version

To the extent that I'm using versioning, this is version 1.0.0. If I ever need to create another version, I promise to follow Semantic Versioning 2.0.0.

Installation

I just use (asdf:load-system :cl-simple-concurrent-jobs) - YMMV.

Licensing

Please see the LICENSE file

Contact

Github issues and pull-requests are the preferred method. If you live in Brighton and want to buy me a beer go ahead, you'll get a special mention here (remind me that it's a contribution to cl-simple-concurrent-jobs):

Beer contributors: '()


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 cl-simple-concurrent-jobs

Author

Jordan Rhys Powell

License

BSD 2-Clause

Description

A simple API for running concurrent jobs and collecting the results

Version

1.0.0

Dependencies
Source

cl-simple-concurrent-jobs.asd (file)

Component

src (module)


Next: , Previous: , Up: Top   [Contents][Index]

3 Modules

Modules are listed depth-first from the system components tree.


Previous: , Up: Modules   [Contents][Index]

3.1 cl-simple-concurrent-jobs/src

Parent

cl-simple-concurrent-jobs (system)

Location

src/

Components

Next: , Previous: , Up: Top   [Contents][Index]

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.1 cl-simple-concurrent-jobs.asd

Location

cl-simple-concurrent-jobs.asd

Systems

cl-simple-concurrent-jobs (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.2 cl-simple-concurrent-jobs/src/packages.lisp

Parent

src (module)

Location

src/packages.lisp

Packages

cl-simple-concurrent-jobs


Previous: , Up: Lisp files   [Contents][Index]

4.1.3 cl-simple-concurrent-jobs/src/scj.lisp

Dependency

packages.lisp (file)

Parent

src (module)

Location

src/scj.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

5.1 cl-simple-concurrent-jobs

A simple API for running concurrent jobs

Source

packages.lisp (file)

Nickname

cl-scj

Use List

common-lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Exported definitions


Previous: , Up: Exported definitions   [Contents][Index]

6.1.1 Functions

Function: add-job JE CALLABLE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: create-job-executor &key NUM-THREADS
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: has-all-results JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: join-results JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: stop JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)


Previous: , Up: Definitions   [Contents][Index]

6.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.1 Macros

Macro: with-job-executor-lock (WORKER) &body BODY
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.2 Functions

Function: append-result JE RESULT
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: cleanup-threads JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: create-job-worker JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Function: finish-and-return-results JE
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

6.2.3 Generic functions

Generic Function: chanl-tasks OBJECT
Generic Function: (setf chanl-tasks) NEW-VALUE OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: chanl-tasks (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Method: (setf chanl-tasks) NEW-VALUE (JOBEXECUTOR jobexecutor)

automatically generated writer method

Source

scj.lisp (file)

Generic Function: channel OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: channel (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Generic Function: completion-condition OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: completion-condition (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Generic Function: je-lock OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: je-lock (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Generic Function: num-results-expected OBJECT
Generic Function: (setf num-results-expected) NEW-VALUE OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: num-results-expected (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Method: (setf num-results-expected) NEW-VALUE (JOBEXECUTOR jobexecutor)

automatically generated writer method

Source

scj.lisp (file)

Generic Function: num-results-got OBJECT
Generic Function: (setf num-results-got) NEW-VALUE OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: num-results-got (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Method: (setf num-results-got) NEW-VALUE (JOBEXECUTOR jobexecutor)

automatically generated writer method

Source

scj.lisp (file)

Generic Function: num-threads OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: num-threads (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Generic Function: results OBJECT
Generic Function: (setf results) NEW-VALUE OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: results (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Method: (setf results) NEW-VALUE (JOBEXECUTOR jobexecutor)

automatically generated writer method

Source

scj.lisp (file)

Generic Function: should-run OBJECT
Generic Function: (setf should-run) NEW-VALUE OBJECT
Package

cl-simple-concurrent-jobs

Methods
Method: should-run (JOBEXECUTOR jobexecutor)

automatically generated reader method

Source

scj.lisp (file)

Method: (setf should-run) NEW-VALUE (JOBEXECUTOR jobexecutor)

automatically generated writer method

Source

scj.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

6.2.4 Classes

Class: jobexecutor ()
Package

cl-simple-concurrent-jobs

Source

scj.lisp (file)

Direct superclasses

standard-object (class)

Direct methods
Direct slots
Slot: num-threads
Initargs

:num-threads

Readers

num-threads (generic function)

Slot: chanl-tasks
Readers

chanl-tasks (generic function)

Writers

(setf chanl-tasks) (generic function)

Slot: results
Readers

results (generic function)

Writers

(setf results) (generic function)

Slot: num-results-expected
Initform

0

Readers

num-results-expected (generic function)

Writers

(setf num-results-expected) (generic function)

Slot: num-results-got
Initform

0

Readers

num-results-got (generic function)

Writers

(setf num-results-got) (generic function)

Slot: completion-condition
Initform

(bordeaux-threads:make-condition-variable)

Readers

completion-condition (generic function)

Slot: should-run
Initform

t

Readers

should-run (generic function)

Writers

(setf should-run) (generic function)

Slot: je-lock
Initform

(bordeaux-threads:make-recursive-lock)

Readers

je-lock (generic function)

Slot: channel
Initform

(make-instance (quote chanl:unbounded-channel))

Readers

channel (generic function)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   C   F   L   M  
Index Entry  Section

C
cl-simple-concurrent-jobs.asd: The cl-simple-concurrent-jobs<dot>asd file
cl-simple-concurrent-jobs/src: The cl-simple-concurrent-jobs/src module
cl-simple-concurrent-jobs/src/packages.lisp: The cl-simple-concurrent-jobs/src/packages<dot>lisp file
cl-simple-concurrent-jobs/src/scj.lisp: The cl-simple-concurrent-jobs/src/scj<dot>lisp file

F
File, Lisp, cl-simple-concurrent-jobs.asd: The cl-simple-concurrent-jobs<dot>asd file
File, Lisp, cl-simple-concurrent-jobs/src/packages.lisp: The cl-simple-concurrent-jobs/src/packages<dot>lisp file
File, Lisp, cl-simple-concurrent-jobs/src/scj.lisp: The cl-simple-concurrent-jobs/src/scj<dot>lisp file

L
Lisp File, cl-simple-concurrent-jobs.asd: The cl-simple-concurrent-jobs<dot>asd file
Lisp File, cl-simple-concurrent-jobs/src/packages.lisp: The cl-simple-concurrent-jobs/src/packages<dot>lisp file
Lisp File, cl-simple-concurrent-jobs/src/scj.lisp: The cl-simple-concurrent-jobs/src/scj<dot>lisp file

M
Module, cl-simple-concurrent-jobs/src: The cl-simple-concurrent-jobs/src module

Jump to:   C   F   L   M  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
A   C   F   G   H   J   M   N   R   S   W  
Index Entry  Section

(
(setf chanl-tasks): Internal generic functions
(setf chanl-tasks): Internal generic functions
(setf num-results-expected): Internal generic functions
(setf num-results-expected): Internal generic functions
(setf num-results-got): Internal generic functions
(setf num-results-got): Internal generic functions
(setf results): Internal generic functions
(setf results): Internal generic functions
(setf should-run): Internal generic functions
(setf should-run): Internal generic functions

A
add-job: Exported functions
append-result: Internal functions

C
chanl-tasks: Internal generic functions
chanl-tasks: Internal generic functions
channel: Internal generic functions
channel: Internal generic functions
cleanup-threads: Internal functions
completion-condition: Internal generic functions
completion-condition: Internal generic functions
create-job-executor: Exported functions
create-job-worker: Internal functions

F
finish-and-return-results: Internal functions
Function, add-job: Exported functions
Function, append-result: Internal functions
Function, cleanup-threads: Internal functions
Function, create-job-executor: Exported functions
Function, create-job-worker: Internal functions
Function, finish-and-return-results: Internal functions
Function, has-all-results: Exported functions
Function, join-results: Exported functions
Function, stop: Exported functions

G
Generic Function, (setf chanl-tasks): Internal generic functions
Generic Function, (setf num-results-expected): Internal generic functions
Generic Function, (setf num-results-got): Internal generic functions
Generic Function, (setf results): Internal generic functions
Generic Function, (setf should-run): Internal generic functions
Generic Function, chanl-tasks: Internal generic functions
Generic Function, channel: Internal generic functions
Generic Function, completion-condition: Internal generic functions
Generic Function, je-lock: Internal generic functions
Generic Function, num-results-expected: Internal generic functions
Generic Function, num-results-got: Internal generic functions
Generic Function, num-threads: Internal generic functions
Generic Function, results: Internal generic functions
Generic Function, should-run: Internal generic functions

H
has-all-results: Exported functions

J
je-lock: Internal generic functions
je-lock: Internal generic functions
join-results: Exported functions

M
Macro, with-job-executor-lock: Internal macros
Method, (setf chanl-tasks): Internal generic functions
Method, (setf num-results-expected): Internal generic functions
Method, (setf num-results-got): Internal generic functions
Method, (setf results): Internal generic functions
Method, (setf should-run): Internal generic functions
Method, chanl-tasks: Internal generic functions
Method, channel: Internal generic functions
Method, completion-condition: Internal generic functions
Method, je-lock: Internal generic functions
Method, num-results-expected: Internal generic functions
Method, num-results-got: Internal generic functions
Method, num-threads: Internal generic functions
Method, results: Internal generic functions
Method, should-run: Internal generic functions

N
num-results-expected: Internal generic functions
num-results-expected: Internal generic functions
num-results-got: Internal generic functions
num-results-got: Internal generic functions
num-threads: Internal generic functions
num-threads: Internal generic functions

R
results: Internal generic functions
results: Internal generic functions

S
should-run: Internal generic functions
should-run: Internal generic functions
stop: Exported functions

W
with-job-executor-lock: Internal macros

Jump to:   (  
A   C   F   G   H   J   M   N   R   S   W  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   C   J   N   R   S  
Index Entry  Section

C
chanl-tasks: Internal classes
channel: Internal classes
completion-condition: Internal classes

J
je-lock: Internal classes

N
num-results-expected: Internal classes
num-results-got: Internal classes
num-threads: Internal classes

R
results: Internal classes

S
should-run: Internal classes
Slot, chanl-tasks: Internal classes
Slot, channel: Internal classes
Slot, completion-condition: Internal classes
Slot, je-lock: Internal classes
Slot, num-results-expected: Internal classes
Slot, num-results-got: Internal classes
Slot, num-threads: Internal classes
Slot, results: Internal classes
Slot, should-run: Internal classes

Jump to:   C   J   N   R   S  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   J   P   S  
Index Entry  Section

C
cl-simple-concurrent-jobs: The cl-simple-concurrent-jobs system
cl-simple-concurrent-jobs: The cl-simple-concurrent-jobs package
Class, jobexecutor: Internal classes

J
jobexecutor: Internal classes

P
Package, cl-simple-concurrent-jobs: The cl-simple-concurrent-jobs package

S
System, cl-simple-concurrent-jobs: The cl-simple-concurrent-jobs system

Jump to:   C   J   P   S