Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the eazy-process Reference Manual, version 0.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Apr 19 15:57:58 2021 GMT+0.
• Introduction | What eazy-process is all about | |
• Systems | The systems documentation | |
• Modules | The modules documentation | |
• Files | The files documentation | |
• Packages | The packages documentation | |
• Definitions | The symbols documentation | |
• Indexes | Concepts, functions, variables and data types |
#+startup: showall + Simplified API/implementation, achieved by *REMOVING* some features in run-program + Declarative process handling + Subprocesses spawned by POSIX fork&exec, no impl-specific interface + Memory / CPU-time management of the child processes + [-] Compatibility layer to the existing State-of-the-Art libraries + [X] trivial-shell + [ ] inferior-shell + [ ] sb-ext:run-program * Usage #+BEGIN_SRC lisp (defvar pipe (pipe)) (defvar in "t/test-input") (defvar out "t/test-output") (defvar p1 (shell '("cat") `(,in ,pipe))) ; fd 0:pathname, 1:pipe (defvar p2 (shell '("cat") `(,pipe ,out))) ; fd 0:pipe, 1:pathname (wait p1) (wait p2) #+END_SRC Processes are async by default. Even if you forget waiting the process, it is handled by trivial-garbage --- when a process object (here, p1 and p2) is GC-ed, it is automatically finalized -- killed and waited. To ensure the process is finalized, =with-process= might help you. It ensures the subprocess is terminated and no zombie remains. However, it does not synchronize with the subprocess by itself --- synchronization should be done manually. #+BEGIN_SRC lisp (with-process (p1 '("sleep" "3")) (print :x)) ;; p1 is killed right after the execution of (print :x), ;; which means that the subprocess does not actually run for 3 seconds. (with-process (p1 '("sleep" "3")) (print :x) (wait p1)) ;; synchronization. Waits for 3 sec #+END_SRC Note that if you leave too many processes alive, the total number of file descriptors might hit the system-provided limit (around 1000 by default). ** Implicit Piping If you do not specify pipes nor pathnames, a new pipe is created for each fd, and the /other/ end of the pipe is accessible with =(fd process fdnum)=. When =nil= is specified, it means /dev/null. #+BEGIN_SRC lisp (let* ((in "t/test-input") (p1 (shell '("cat") `(,in :output))) ; fd 0,1 where 1 is an implicit pipe (p2 (shell '("cat") `(,(fd p1 1) :out :o :input :in :i nil)))) ; fd 0-6 (wait p1) (wait p2)) #+END_SRC ** Reading the output The output of the subprocess is *NOT directly accessible with EAZY-PROCESS*. Instead, open the pathname returned by =fd-as-pathname=, which is =/dev/fd/[fd]=. #+BEGIN_SRC lisp (test read (let ((p1 (shell `("hostname")))) (with-open-file (s (fd-as-pathname p1 1)) (is (string= (machine-instance) (read-line s)))))) #+END_SRC This has greatly simplified the API of eazy-process. ** Resource management Macro =with-rlimit= dynamically binds the current rlimit resource limitation. As noted in =*rlimit-resources*= docstring, this does not affect the lisp process itself. --- It only affect the new child processes spawned by =shell=. Example below shows the usecase where =*spendtime*= contains a path to a simple C program that busy-waits for 10 seconds. The execution is terminated in 3 seconds. TERMSIG is set to 24 because the program is killed by SIGXCPU. #+BEGIN_SRC lisp (with-rlimit ((+rlimit-cpu-time+ 3)) ; 3 sec (let ((p (shell `(,(namestring *spendtime*))))) (multiple-value-bind (exited exitstatus ifsignalled termsig ...) (wait p) (is-false exited) (is-false exitstatus) (is-true ifsignalled) (is (= 24 termsig))))) #+END_SRC This macro can be nested, and the new subprocess reflects the inntermost limitation. #+BEGIN_SRC lisp (with-rlimit ((+rlimit-cpu-time+ 3)) (shell ...) ; 3 sec (with-rlimit ((+rlimit-cpu-time+ 5) (+rlimit-as+ 500000)) (shell ...))) ; 5 sec, 500 MB #+END_SRC * Somewhat Longer Description In `run-program` interface in the popular implementations, piping between subprocesses are hard. It requires either reading the entire output stream and packing the contents as a new string-input-stream, or using some other implementation-specific functions. Also, compatibility libraries e.g. trivial-shell or inferior-shell, often depend on these functions, implying the same problem. Iolib also has `run-program` that allows easy piping, but it is restricted to 3 fds: `input,output,error`. Eazy-process provides a clean, declarative and thin layer for the processes. It depends on the concept of "everything is a file" and do not provide interfaces to streams. * Tested Impl This library is at least tested on implementation listed below: + SBCL 1.2.1 on X86-64 Linux 3.13.0-39-generic (author's environment) + SBCL 1.1.14 on X86 Linux 3.13.0-44-generic (author's environment) + CCL 1.10 on linux currently has a problem reading =/dev/fd/[fd]=, which is actually a symlink to =/proc/[pid]/fd/[fd]=, and the test does not pass. Do not use =(fd-as-pathname process fd)= and use temorary files instead. + ECL opens =/dev/fd/[fd]= correctly, but it fails to load CFFI... + ABCL has more problems than CCL. It fails to open =/proc/[pid]/fd/[fd]= and also have problems with CFFI. Test reports on other OS'es/impls are greatly appreciated. Run =./simple-build-test.sh=, assuming it already loads quicklisp in your init files. * Dependencies It depends on the latest libfixposix available at https://github.com/sionescu/libfixposix . Also, it depends on the following libraries: + iterate by *Jonathan Amsterdam* : Jonathan Amsterdam's iterator/gatherer/accumulator facility + Alexandria by ** : Alexandria is a collection of portable public domain utilities. + cffi by *James Bielman* : The Common Foreign Function Interface + trivia by *Masataro Asai* + iolib + trivial-garbage + cl-rlimit * Syntax #+BEGIN_SRC lisp (defun shell (argv &optional (fdspecs '(:input :output :output)) (environments nil env-p) (search t)) ...) #+END_SRC When =search= is nil, it disables the pathname resolving using PATH. ** Fdspecs #+BEGIN_SRC fdspecs := {fdspec}* fdspec := output | input | fd | path-or-pipe | openspec output := :output | :out | :o input := :input | :in | :i fd := openspec := (path-or-pipe &key direction if-exists if-does-not-exist) path-or-pipe := | direction := :input | :output | :io | :probe if-exists := :overwrite | :supersede | :append | :error if-does-not-exist := :create | :error #+END_SRC + =output= form and =input= form implicitly create a new pipe. + The fixnum =fd= should be a value of function =(fd process fdnum)=. + Openfilespec is almost identical to the argument list of =OPEN= in ANSI spec, however =:rename=, =:rename-and-delete=, =:new-version= are not supported and signals an error. + Function =pipe= generates a new pipe object that can be used in an fdspec. + If a = = or a = = are given without options, it uses a default direction, which is =:input= for fd 0 and =:output= for fd 1 and fd 2. For fd > 2, missing direction signals an error. + Be careful when you open a fifo, the process will be blocked. ** Environments : environments := {environment}* : environment := env-pair | env-string : env-pair := (name . value) : env-string := "name=value" : name, value -- string If we omit the second argument =environments=, the subprocess inherits the environment of the parent lisp process. =unset= -ting the environment value is not available. * Compatibility Layers There are compatibility layers for /trivial-shell/ and /inferior-shell/. For documentation, see [[compat/README.org]] . ** /run-program/ compatibility *Abandoned* , since the design of =run-program= is not thoroughly delibelated, and any form of its descendants is not acceptable. It is a mistake to combine processes with streams in a tightly coupled manner. * Author & Copyright Copyright (c) 2014 Masataro Asai (guicho2.71828@gmail.com)
Next: Modules, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
• The eazy-process system |
Masataro Asai
MIT
Yet Another Portable Library for Process Handling / Subshell Invokation
In ‘run-program‘ interface in the popular
implementations, piping between subprocesses are hard. It requires either
reading the entire output stream and packing the contents as a new
string-input-stream, or using some other implementation-specific
functions. Also, compatibility libraries e.g. trivial-shell or
inferior-shell, often depend on these functions, implying the same
problem. Iolib also has ‘run-program‘ that allows easy piping, but it is
restricted to 3 fds: ‘input,output,error‘.
Eazy-process provides a clean, declarative and thin layer to the processes. It depends on the concept of ’everything is a file’ and do not provide interfaces to streams.
0.1
eazy-process.asd (file)
Modules are listed depth-first from the system components tree.
• The eazy-process/src module | ||
• The eazy-process/compat module |
Next: The eazy-process/compat module, Previous: Modules, Up: Modules [Contents][Index]
eazy-process (system)
src/
Previous: The eazy-process/src module, Up: Modules [Contents][Index]
src (module)
eazy-process (system)
compat/
trivial-shell.lisp (file)
Files are sorted by type and then listed depth-first from the systems components trees.
• Lisp files |
Next: The eazy-process/src/package․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
eazy-process.asd
eazy-process (system)
Next: The eazy-process/src/mktempfifo․lisp file, Previous: The eazy-process․asd file, Up: Lisp files [Contents][Index]
src (module)
src/package.lisp
Next: The eazy-process/src/pipe․lisp file, Previous: The eazy-process/src/package․lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
src (module)
src/mktempfifo.lisp
mktempfifo (function)
Next: The eazy-process/src/environ․lisp file, Previous: The eazy-process/src/mktempfifo․lisp file, Up: Lisp files [Contents][Index]
mktempfifo.lisp (file)
src (module)
src/pipe.lisp
Next: The eazy-process/src/specials․lisp file, Previous: The eazy-process/src/pipe․lisp file, Up: Lisp files [Contents][Index]
pipe.lisp (file)
src (module)
src/environ.lisp
environment (function)
Next: The eazy-process/src/fdspec․lisp file, Previous: The eazy-process/src/environ․lisp file, Up: Lisp files [Contents][Index]
environ.lisp (file)
src (module)
src/specials.lisp
+max-rlimit-constant+ (special variable)
Next: The eazy-process/src/shell․lisp file, Previous: The eazy-process/src/specials․lisp file, Up: Lisp files [Contents][Index]
specials.lisp (file)
src (module)
src/fdspec.lisp
Next: The eazy-process/src/process․lisp file, Previous: The eazy-process/src/fdspec․lisp file, Up: Lisp files [Contents][Index]
fdspec.lisp (file)
src (module)
src/shell.lisp
shell (function)
Next: The eazy-process/src/macros․lisp file, Previous: The eazy-process/src/shell․lisp file, Up: Lisp files [Contents][Index]
shell.lisp (file)
src (module)
src/process.lisp
Next: The eazy-process/src/resources․lisp file, Previous: The eazy-process/src/process․lisp file, Up: Lisp files [Contents][Index]
process.lisp (file)
src (module)
src/macros.lisp
with-process (macro)
Next: The eazy-process/compat/trivial-shell․lisp file, Previous: The eazy-process/src/macros․lisp file, Up: Lisp files [Contents][Index]
macros.lisp (file)
src (module)
src/resources.lisp
with-rlimit (macro)
Previous: The eazy-process/src/resources․lisp file, Up: Lisp files [Contents][Index]
compat (module)
compat/trivial-shell.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
• The eazy-process-asd package | ||
• The eazy-process package |
Next: The eazy-process package, Previous: Packages, Up: Packages [Contents][Index]
eazy-process.asd
Previous: The eazy-process-asd package, Up: Packages [Contents][Index]
package.lisp (file)
Definitions are sorted by export status, category, package, and then by lexicographic order.
• Exported definitions | ||
• Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
• Exported special variables | ||
• Exported symbol macros | ||
• Exported macros | ||
• Exported functions | ||
• Exported generic functions | ||
• Exported structures | ||
• Exported classes |
Next: Exported symbol macros, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
Command line string which is invoked in order to run the subshell. Default value is "sh -c".
The value is then followed by the command specified in shell-command, e.g. ,
when command = ’ls -la’, the running command is "sh -c ’ls -la’".
It provides the ability to call the interpreters like perl/AWK easily.
The name/path of the interpreter must be separated by spaces from the options like ’-c’.
Do not use complex things like — LANG=C sh . The first word is always considered the pathname.
If you want to do complicated stuff, then do it inside the interpreter.
The process is forked, then the child process calls execvp with the name of
the interpreter. (This is defined in function ‘eazy-process:shell’.) The
actual pathname of the intepreter can be resolved using PATH environment
variable.
trivial-shell.lisp (file)
A vector of rlimit resource value, where the index suggests +rlimit-XXX+, which is an integer (usually below 16)
This value does not affect the lisp process itself; The effect is applied to the child process only.
Rather than modifying this variable directly, use ‘with-rlimit’ instead.
specials.lisp (file)
specials.lisp (file)
Next: Exported macros, Previous: Exported special variables, Up: Exported definitions [Contents][Index]
trivial-shell.lisp (file)
eazy-process:*interpreter*
Next: Exported functions, Previous: Exported symbol macros, Up: Exported definitions [Contents][Index]
Spawn a process, bind the process to PROCESS,
execute the body and finalize (= kill and wait w/ nohang) the process.
BODY works as an implicit progn, and the return value is that of the progn.
It does not synchronously wait the process. Therefore, it is the user’s
responsibility to inhibit the early termination of the process.
The aim of finalization is to release the resources of the process e.g. file descriptor and pipes.
Example:
(with-open-process (p ("sleep" "10"))
(print :waiting)
(wait p))
macros.lisp (file)
Append/overwrite the current rlimit resouce limitation.
This does not affect the lisp process itself.
Only the spawned child process will be affected.
resources.lisp (file)
Next: Exported generic functions, Previous: Exported macros, Up: Exported definitions [Contents][Index]
pipe.lisp (file)
environ.lisp (file)
Returns the file descriptor of the lisp process.
The returned fd is connected to the n-th fd of the child process through a pipe.
Example:
(fd P 1) ; –> 5
This means that the 5’th fd of the lisp process is connected to the 1st fd of the process P.
process.lisp (file)
Return the pathname for each file descriptor.
Lisp can read/write to each fd by opening this file.
Since the buffer size of file-stream is implementation-dependent, and the call to cl:read might cache a considerable amount of characters from the file, we are not able to ensure that the same file can be opened more than once.
Even if you just cl:open the file and just cl:read-char the stream,
it might read the entire characters of the file into the buffer as a side-effect.
We just advise you not to open the file more than once.
process.lisp (file)
Returns an fresh array of fds available from lisp process.
process.lisp (file)
Waitpid the process. If the process is alive, kill it with SIG first, then with SIGKILL.
process.lisp (file)
pipe.lisp (file)
Asynchronously execute ‘argv’ using fork(2) and ‘execve’
family of system calls, returns a process structure object.
ARGV is a sequence of strings. Each string is converted to null-terminated
C strings char* ARGV and passed to execvp. The first element is also
passed to execvp as the pathname of the executable.
FDSPECS is a sequence specifying how the child process should handle its output.
For the documentation of fd-specifier see ‘+fdspecs-default+’.
Example: (:in :out :out) , (5 :out 8)
Each element in FDSPECS corresponds to each file descriptor of the child process.
When an i-th element of FDSPECS is a symbol, then a new pipe(2) is created for each i.
The i-th fd of the child process p1 is associated with the correct end of the new pipe,
the other end of the pipe is readable/writable from lisp from the process interface.
(see the documentation of class PROCESS.)
When an i-th element of FDSPECS is an integer j,
then it should be the lisp end of a pipe connected to some other child process p2.
(not the normal fd of the lisp process, like 0,1,2)
The i-th fd of p1 is connected to this pipe,
then p1 and p2 can exchange the data each other.
When this happens, lisp process should not read from this pipe
(or the data will not be sent to the destination of the pipe).
For that purpose, add a layer like ‘tee’.
When ENVIRONMENT is specified, it is passed to execve/execvpe.
It should be a list of strings ("NAME1=VALUE1" "NAME2=VALUE2")
or an alist (("NAME1" . "VALUE1") ...).
When SEARCH is t, it uses execvp/execvpe. (==executable is searched through PATH)
On error during system call in the parent process, iolib.syscalls:syscall-error is signalled.
In the child process, the system call failure result in error status 203.
FIXME: this might not be good.
shell.lisp (file)
simple interface compatible to trivial-shell @
https://github.com/gwkkwg/trivial-shell.git.
returns (values output error-output exit-status).
The input is read from the :input key argument.
Additional functionality:
:external-format — specifies the decoding of the output.
trivial-shell.lisp (file)
option is one of :nohang, :untraced, :continued.
Returns a value of the following signature:
(list (boolean ifexited)
(integer exitstatus)
(boolean ifsignalled)
(integer termsig)
(boolean coredump)
(boolean ifstopped)
(integer stopsig)
(boolean ifcontinued)
(integer status)).
When the value is inappropriate as defined in man wait(2),
some integer values may return NIL.
When :nohang is specified but no child has changed its state,
then it returns NIL instead.
‘wait(0)’, i.e. wait for any children, is not available.
process.lisp (file)
Next: Exported structures, Previous: Exported functions, Up: Exported definitions [Contents][Index]
automatically generated reader method
process.lisp (file)
Next: Exported classes, Previous: Exported generic functions, Up: Exported definitions [Contents][Index]
pipe.lisp (file)
structure-object (structure)
fixnum
0
pipe-read (function)
(setf pipe-read) (function)
fixnum
1
pipe-write (function)
(setf pipe-write) (function)
Previous: Exported structures, Up: Exported definitions [Contents][Index]
A class representing a process.
process.lisp (file)
standard-object (class)
:pid
pid (generic function)
:fds
%fds (generic function)
Previous: Exported definitions, Up: Definitions [Contents][Index]
• Internal special variables | ||
• Internal macros | ||
• Internal functions | ||
• Internal generic functions |
Next: Internal macros, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
specials.lisp (file)
Next: Internal functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
trivial-shell.lisp (file)
Next: Internal generic functions, Previous: Internal macros, Up: Internal definitions [Contents][Index]
process.lisp (file)
shell.lisp (file)
shell.lisp (file)
shell.lisp (file)
True finalizer of a process object. However,
This function should not contain reference to the process itself
because it prevents process object from GC-ing.
process.lisp (file)
if the child process died of errno, the exit value is that value
shell.lisp (file)
shell.lisp (file)
process.lisp (file)
ANSI CL standard:
if-does-not-exist—one of :error, :create, or nil.
The default is
:error if direction is :input or if-exists is :overwrite or :append
:create if direction is :output or :io, and if-exists is neither :overwrite nor :append;
or nil when direction is :probe.
fdspec.lisp (file)
fdspec.lisp (file)
shell.lisp (file)
Take an fd-specifier and return a cons of (parent-fn . child-fn) where
both parent-fn and child-fn are closures of 1 arg, the fd on the child end.
Each function runs a job that should be done in the context of
parent/child process.
if FD is specified and the direction is not specified, use the default
direction of that fd by default. For FD > 2, there is no default direction
and it signals an error.
Parent-fn should return the fd of the parent-end.
fdspec.lisp (file)
trivial-shell.lisp (file)
trivial-shell.lisp (file)
pipe.lisp (file)
Duplicate the old fd, set it to the new fd, then close the old fd
fdspec.lisp (file)
trivial-shell.lisp (file)
shell.lisp (file)
pipe.lisp (file)
mktempfifo.lisp (file)
pipe.lisp (file)
pipe.lisp (file)
pipe.lisp (file)
trivial-shell.lisp (file)
Previous: Internal functions, Up: Internal definitions [Contents][Index]
automatically generated reader method
process.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
• Concept index | ||
• Function index | ||
• Variable index | ||
• Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
Jump to: | E F L M |
---|
Jump to: | E F L M |
---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
Jump to: | %
(
C D E F G I M P S T W |
---|
Jump to: | %
(
C D E F G I M P S T W |
---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
Jump to: | *
+
F P R S W |
---|
Jump to: | *
+
F P R S W |
---|
Previous: Variable index, Up: Indexes [Contents][Index]
Jump to: | C E P S |
---|
Jump to: | C E P S |
---|