The shlex Reference Manual

Next: , Previous: , Up: (dir)   [Contents][Index]

The shlex Reference Manual

This is the shlex Reference Manual, generated automatically by Declt version 4.0 beta 2 "William Riker" on Thu Sep 15 04:10:06 2022 GMT+0.

Table of Contents


1 Introduction

A lexer for syntaxes that use shell-like rules for quoting and commenting. It is a port of the shlex module from Python’s standard library.

The point of entry is split:

(shlex:split "a 'b c' \"d e\" \"f ' g\"")
=> '("a" "b c" "d e" "f ' g")

For a parse that is closer to a shell, you can pass the :punctuation-chars keyword:

;; Not what you want.
(shlex:split "a && b; c && d || e; f >'abc'; (def \"ghi\")")
=> '("a" "&&" "b;" "c" "&&" "d" "||" "e;" "f" ">abc;" "(def" "ghi)")

(shlex:split "a && b; c && d || e; f >'abc'; (def \"ghi\")" :punctuation-chars t)
=> ("a" "&&" "b" ";" "c" "&&" "d" "||" "e" ";" "f" ">" "abc" ";" "(" "def" "ghi" ")")

If you would rather iterate over each token in turn, you can use shlex:map-tokens or shlex:do-tokens instead.

;; Print each token in STRING.
(shlex:map-tokens #'print string :punctuation-chars t)

;; Do the same thing with a macro.
(shlex:do-tokens (token (string :punctuation-chars t))
  (print token)

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 shlex

Lexical analyzer for simple shell-like syntax.

Author

Paul M. Rodriguez <pmr@ruricolist.com>

License

MIT

Dependencies
  • alexandria (system).
  • serapeum (system).
  • cl-ppcre (system).
  • cl-unicode (system).
Source

shlex.asd.

Child Components

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp   [Contents][Index]

3.1.1 shlex/shlex.asd

Source

shlex.asd.

Parent Component

shlex (system).

ASDF Systems

shlex.


Next: , Previous: , Up: Lisp   [Contents][Index]

3.1.2 shlex/package.lisp

Source

shlex.asd.

Parent Component

shlex (system).

Packages

shlex.


Previous: , Up: Lisp   [Contents][Index]

3.1.3 shlex/shlex.lisp

Dependency

package.lisp (file).

Source

shlex.asd.

Parent Component

shlex (system).

Public Interface
Internals

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 shlex

Source

package.lisp.

Use List
  • alexandria.
  • common-lisp.
  • serapeum.
Public Interface
Internals

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Public Interface


5.1.1 Macros

Macro: do-tokens ((token (source &rest args &key &allow-other-keys) &optional return) &body body)

Iterate over the tokens of SOURCE, using shell-like rules.

Package

shlex.

Source

shlex.lisp.


5.1.2 Ordinary functions

Function: quote (s)

Return a shell-escaped version of string S.

Package

shlex.

Source

shlex.lisp.

Function: split (source &rest args &key &allow-other-keys)

Split SOURCE, a string or stream, into tokens using shell-like rules.

Pass ‘:posix‘ (true by default) to obey POSIX parsing rules.

Pass ‘:punctuation-chars‘ (nil by default) if you want runs of punctuation characters (semicolons, greater and less than signs, ampersands, etc.) to be parsed as separate tokens.

Package

shlex.

Source

shlex.lisp.


5.1.3 Generic functions

Generic Function: map-tokens (fn source &key comments posix debug-input punctuation-chars whitespace-split &allow-other-keys)

Map FN over the tokens of SOURCE, a string or stream, using shell-like rules. See the documentation of ‘split’ for a discussion of possible options.

Package

shlex.

Source

shlex.lisp.

Methods
Method: map-tokens (fn (in stream) &rest args &key comments posix debug-input punctuation-chars whitespace-split &allow-other-keys)
Method: map-tokens (fn (s string) &rest args &key &allow-other-keys)

5.1.4 Standalone methods

Method: initialize-instance :after ((self shlex) &key punctuation-chars posix)
Source

shlex.lisp.


5.2 Internals


Next: , Previous: , Up: Internals   [Contents][Index]

5.2.1 Symbol macros

Symbol Macro: default-commenters
Package

shlex.

Source

shlex.lisp.

Symbol Macro: default-whitespace
Package

shlex.

Source

shlex.lisp.

Symbol Macro: non-posix-word-chars
Package

shlex.

Source

shlex.lisp.

Symbol Macro: posix-word-chars
Package

shlex.

Source

shlex.lisp.

Symbol Macro: safe-char-map
Package

shlex.

Source

shlex.lisp.


5.2.2 Compiler macros

Compiler Macro: make-lexer (s &rest args)
Package

shlex.

Source

shlex.lisp.


5.2.3 Ordinary functions

Function: ensure-string-stream (s)
Package

shlex.

Source

shlex.lisp.

Function: find-sorted (char string)

Find CHAR in STRING, a sorted string, by bisection.

Package

shlex.

Source

shlex.lisp.

Function: lexer-tokens (lexer)
Package

shlex.

Source

shlex.lisp.

Function: make-lexer (s &rest args &key &allow-other-keys)
Package

shlex.

Source

shlex.lisp.

Function: map-lexer-tokens (fn lexer)
Package

shlex.

Source

shlex.lisp.

Function: safe-char? (char)
Package

shlex.

Source

shlex.lisp.

Function: string-conc (string x)

Add X, a character or string, to the end of STRING, an adjustable string with a fill pointer.

Package

shlex.

Source

shlex.lisp.


Next: , Previous: , Up: Internals   [Contents][Index]

5.2.4 Generic functions

Generic Reader: base-message (condition)
Generic Writer: (setf base-message) (condition)
Package

shlex.

Methods
Reader Method: base-message ((condition shlex-error))
Writer Method: (setf base-message) ((condition shlex-error))
Source

shlex.lisp.

Target Slot

base-message.

Generic Function: get-token (lexer)

Return the next token.

Package

shlex.

Source

shlex.lisp.

Methods
Method: get-token ((self shlex))
Generic Reader: lexer-input (object)
Package

shlex.

Methods
Reader Method: lexer-input ((shlex shlex))

automatically generated reader method

Source

shlex.lisp.

Target Slot

debug-input.

Generic Reader: lexer-line-number (object)
Package

shlex.

Methods
Reader Method: lexer-line-number ((shlex shlex))

Source line number.

Source

shlex.lisp.

Target Slot

line-number.

Generic Function: read-token (lexer)

Read a raw token.

Package

shlex.

Source

shlex.lisp.

Methods
Method: read-token ((self shlex))

Read a raw token.

Generic Function: token (self)
Package

shlex.

Methods
Method: token ((self shlex))
Source

shlex.lisp.

Generic Function: (setf token) (self)
Package

shlex.

Methods
Method: (setf token) ((self shlex))
Source

shlex.lisp.


Next: , Previous: , Up: Internals   [Contents][Index]

5.2.5 Conditions

Condition: no-closing-quotation
Package

shlex.

Source

shlex.lisp.

Direct superclasses

shlex-error.

Direct slots
Slot: base-message
Initform

(quote "missing closing quotation in string")

Condition: no-escaped-character
Package

shlex.

Source

shlex.lisp.

Direct superclasses

shlex-error.

Direct slots
Slot: base-message
Initform

(quote "no escaped character in string")

Condition: shlex-error
Package

shlex.

Source

shlex.lisp.

Direct superclasses

error.

Direct subclasses
Direct methods
Direct slots
Slot: lexer
Initargs

:lexer

Slot: base-message
Readers

base-message.

Writers

(setf base-message).


Previous: , Up: Internals   [Contents][Index]

5.2.6 Classes

Class: shlex
Package

shlex.

Source

shlex.lisp.

Direct methods
Direct Default Initargs
InitargValue
:posixnil
:c-stringsnil
:whitespacedefault-whitespace
:whitespace-splitnil
:commentersdefault-commenters
:quotes'"
:escape\
:escaped-quotes"
:debug-inputunknown
Direct slots
Slot: pushback
Slot: pushback-chars
Initform

(serapeum:queue)

Slot: punctuation-chars

Characters that will be considered punctuation. Runs of punctuation are returned as a single token.

Type

string

Slot: line-number

Source line number.

Initform

1

Readers

lexer-line-number.

Writers

This slot is read-only.

Slot: instream
Type

stream

Initargs

:instream

Slot: state
Type

(or null character)

Initform

#\

Slot: posix

If true, use POSIX-esque parsing rules.

Type

boolean

Initargs

:posix

Slot: c-strings

If true, interpret ANSI-C quoting.

Type

boolean

Initargs

:c-strings

Slot: whitespace-split

If true, only split tokens on whitespaces.

Type

boolean

Initargs

:whitespace-split

Slot: whitespace

Characters that bound tokens.

Type

string

Initargs

:whitespace

Slot: commenters

String of characters to recognize as comment beginnners.

Type

string

Initargs

:commenters

Slot: escape

Characters to be considered an escape.

Package

serapeum.

Type

string

Initargs

:escape

Slot: word-chars

String of characters to accumulate into tokens.

Type

string

Slot: quotes

Characters that will be considered string quotes.

Type

string

Initargs

:quotes

Slot: escaped-quotes

The subset of QUOTES that allow escapes.

Type

string

Initargs

:escaped-quotes

Slot: eof

The token that determines EOF.

Slot: debug-input
Type

string

Initargs

:debug-input

Readers

lexer-input.

Writers

This slot is read-only.

Slot: token-buffer

Token buffer.

Initform

(make-array 10 :adjustable t :fill-pointer 0 :element-type (quote character))


Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts


Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
B   C   D   E   F   G   I   L   M   Q   R   S   T  
Index Entry  Section

(
(setf base-message): Private generic functions
(setf base-message): Private generic functions
(setf token): Private generic functions
(setf token): Private generic functions

B
base-message: Private generic functions
base-message: Private generic functions

C
Compiler Macro, make-lexer: Private compiler macros

D
do-tokens: Public macros

E
ensure-string-stream: Private ordinary functions

F
find-sorted: Private ordinary functions
Function, ensure-string-stream: Private ordinary functions
Function, find-sorted: Private ordinary functions
Function, lexer-tokens: Private ordinary functions
Function, make-lexer: Private ordinary functions
Function, map-lexer-tokens: Private ordinary functions
Function, quote: Public ordinary functions
Function, safe-char?: Private ordinary functions
Function, split: Public ordinary functions
Function, string-conc: Private ordinary functions

G
Generic Function, (setf base-message): Private generic functions
Generic Function, (setf token): Private generic functions
Generic Function, base-message: Private generic functions
Generic Function, get-token: Private generic functions
Generic Function, lexer-input: Private generic functions
Generic Function, lexer-line-number: Private generic functions
Generic Function, map-tokens: Public generic functions
Generic Function, read-token: Private generic functions
Generic Function, token: Private generic functions
get-token: Private generic functions
get-token: Private generic functions

I
initialize-instance: Public standalone methods

L
lexer-input: Private generic functions
lexer-input: Private generic functions
lexer-line-number: Private generic functions
lexer-line-number: Private generic functions
lexer-tokens: Private ordinary functions

M
Macro, do-tokens: Public macros
make-lexer: Private compiler macros
make-lexer: Private ordinary functions
map-lexer-tokens: Private ordinary functions
map-tokens: Public generic functions
map-tokens: Public generic functions
map-tokens: Public generic functions
Method, (setf base-message): Private generic functions
Method, (setf token): Private generic functions
Method, base-message: Private generic functions
Method, get-token: Private generic functions
Method, initialize-instance: Public standalone methods
Method, lexer-input: Private generic functions
Method, lexer-line-number: Private generic functions
Method, map-tokens: Public generic functions
Method, map-tokens: Public generic functions
Method, read-token: Private generic functions
Method, token: Private generic functions

Q
quote: Public ordinary functions

R
read-token: Private generic functions
read-token: Private generic functions

S
safe-char?: Private ordinary functions
split: Public ordinary functions
string-conc: Private ordinary functions

T
token: Private generic functions
token: Private generic functions

Jump to:   (  
B   C   D   E   F   G   I   L   M   Q   R   S   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   B   C   D   E   I   L   N   P   Q   S   T   W  
Index Entry  Section

B
base-message: Private conditions
base-message: Private conditions
base-message: Private conditions

C
c-strings: Private classes
commenters: Private classes

D
debug-input: Private classes
default-commenters: Private symbol macros
default-whitespace: Private symbol macros

E
eof: Private classes
escape: Private classes
escaped-quotes: Private classes

I
instream: Private classes

L
lexer: Private conditions
line-number: Private classes

N
non-posix-word-chars: Private symbol macros

P
posix: Private classes
posix-word-chars: Private symbol macros
punctuation-chars: Private classes
pushback: Private classes
pushback-chars: Private classes

Q
quotes: Private classes

S
safe-char-map: Private symbol macros
Slot, base-message: Private conditions
Slot, base-message: Private conditions
Slot, base-message: Private conditions
Slot, c-strings: Private classes
Slot, commenters: Private classes
Slot, debug-input: Private classes
Slot, eof: Private classes
Slot, escape: Private classes
Slot, escaped-quotes: Private classes
Slot, instream: Private classes
Slot, lexer: Private conditions
Slot, line-number: Private classes
Slot, posix: Private classes
Slot, punctuation-chars: Private classes
Slot, pushback: Private classes
Slot, pushback-chars: Private classes
Slot, quotes: Private classes
Slot, state: Private classes
Slot, token-buffer: Private classes
Slot, whitespace: Private classes
Slot, whitespace-split: Private classes
Slot, word-chars: Private classes
state: Private classes
Symbol Macro, default-commenters: Private symbol macros
Symbol Macro, default-whitespace: Private symbol macros
Symbol Macro, non-posix-word-chars: Private symbol macros
Symbol Macro, posix-word-chars: Private symbol macros
Symbol Macro, safe-char-map: Private symbol macros

T
token-buffer: Private classes

W
whitespace: Private classes
whitespace-split: Private classes
word-chars: Private classes

Jump to:   B   C   D   E   I   L   N   P   Q   S   T   W