The shlex Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The shlex Reference Manual

This is the shlex Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Dec 02 09:47:42 2019 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

A lexer for syntaxes that use shell-like rules for quoting and commenting. It is a port of the shlex module from Python’s standard library.

The point of entry is split:

(shlex:split "a 'b c' \"d e\" \"f ' g\"")
=> '("a" "b c" "d e" "f ' g")

For a parse that is closer to a shell, you can pass the :punctuation-chars keyword:

;; Not what you want.
(shlex:split "a && b; c && d || e; f >'abc'; (def \"ghi\")")
=> '("a" "&&" "b;" "c" "&&" "d" "||" "e;" "f" ">abc;" "(def" "ghi)")

(shlex:split "a && b; c && d || e; f >'abc'; (def \"ghi\")" :punctuation-chars t)
=> ("a" "&&" "b" ";" "c" "&&" "d" "||" "e" ";" "f" ">" "abc" ";" "(" "def" "ghi" ")")

If you would rather iterate over each token in turn, you can use shlex:map-tokens or shlex:do-tokens instead.

;; Print each token in STRING.
(shlex:map-tokens #'print string :punctuation-chars t)

;; Do the same thing with a macro.
(shlex:do-tokens (token (string :punctuation-chars t))
  (print token)

Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 shlex

Author

Paul M. Rodriguez <pmr@ruricolist.com>

License

MIT

Description

Lexical analyzer for simple shell-like syntax.

Dependencies
Source

shlex.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 shlex.asd

Location

shlex.asd

Systems

shlex (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 shlex/package.lisp

Parent

shlex (system)

Location

package.lisp

Packages

shlex


Previous: , Up: Lisp files   [Contents][Index]

3.1.3 shlex/shlex.lisp

Dependency

package.lisp (file)

Parent

shlex (system)

Location

shlex.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 shlex

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Macros

Macro: do-tokens (TOKEN (SOURCE &rest ARGS &key &allow-other-keys) &optional RETURN) &body BODY

Iterate over the tokens of SOURCE, using shell-like rules.

Package

shlex

Source

shlex.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Functions

Function: quote S

Return a shell-escaped version of string S.

Package

shlex

Source

shlex.lisp (file)

Function: split SOURCE &rest ARGS &key &allow-other-keys

Split SOURCE, a string or stream, into tokens using shell-like rules.

Pass ‘:posix‘ (true by default) to obey POSIX parsing rules.

Pass ‘:punctuation-chars‘ (nil by default) if you want runs of punctuation characters (semicolons, greater and less than signs, ampersands, etc.) to be parsed as separate tokens.

Package

shlex

Source

shlex.lisp (file)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.3 Generic functions

Generic Function: map-tokens FN SOURCE &key COMMENTS POSIX DEBUG-INPUT PUNCTUATION-CHARS WHITESPACE-SPLIT &allow-other-keys

Map FN over the tokens of SOURCE, a string or stream, using shell-like rules. See the documentation of ‘split’ for a discussion of possible options.

Package

shlex

Source

shlex.lisp (file)

Methods
Method: map-tokens FN (IN stream) &rest ARGS &key COMMENTS POSIX DEBUG-INPUT PUNCTUATION-CHARS WHITESPACE-SPLIT &allow-other-keys
Method: map-tokens FN (S string) &rest ARGS &key &allow-other-keys

Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Constants

Constant: +storage-for-deflex-var-safe-char-map+

Bitmap for a safe char.

Package

shlex

Source

shlex.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Symbol macros

Symbol Macro: default-commenters
Package

shlex

Source

shlex.lisp (file)

Expansion

shlex::*storage-for-deflex-var-default-commenters*

Symbol Macro: default-whitespace
Package

shlex

Source

shlex.lisp (file)

Expansion

shlex::*storage-for-deflex-var-default-whitespace*

Symbol Macro: non-posix-word-chars
Package

shlex

Source

shlex.lisp (file)

Expansion

shlex::*storage-for-deflex-var-non-posix-word-chars*

Symbol Macro: posix-word-chars
Package

shlex

Source

shlex.lisp (file)

Expansion

shlex::*storage-for-deflex-var-posix-word-chars*

Symbol Macro: safe-char-map
Package

shlex

Source

shlex.lisp (file)

Expansion

shlex::+storage-for-deflex-var-safe-char-map+


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Compiler macros

Compiler Macro: make-lexer S &rest ARGS
Package

shlex

Source

shlex.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.4 Functions

Function: bisect-left STR CHAR

Return the index in STR to insert CHAR and keep STR sorted.

Package

shlex

Source

shlex.lisp (file)

Function: ensure-string-stream S
Package

shlex

Source

shlex.lisp (file)

Function: find-sorted CHAR STRING

Find CHAR in STRING, a sorted string, by bisection.

Package

shlex

Source

shlex.lisp (file)

Function: lexer-tokens LEXER
Package

shlex

Source

shlex.lisp (file)

Function: make-lexer S &rest ARGS &key &allow-other-keys
Package

shlex

Source

shlex.lisp (file)

Function: map-lexer-tokens FN LEXER
Package

shlex

Source

shlex.lisp (file)

Function: safe-char? CHAR
Package

shlex

Source

shlex.lisp (file)

Function: string-conc STRING X

Add X, a character or string, to the end of STRING, an adjustable string with a fill pointer.

Package

shlex

Source

shlex.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.5 Generic functions

Generic Function: get-token LEXER

Return the next token.

Package

shlex

Source

shlex.lisp (file)

Methods
Method: get-token (SELF shlex)
Generic Function: lexer-input OBJECT
Package

shlex

Methods
Method: lexer-input (SHLEX shlex)

automatically generated reader method

Source

shlex.lisp (file)

Generic Function: lexer-line-number OBJECT
Package

shlex

Methods
Method: lexer-line-number (SHLEX shlex)

Source line number.

Source

shlex.lisp (file)

Generic Function: read-token LEXER

Read a raw token.

Package

shlex

Source

shlex.lisp (file)

Methods
Method: read-token (SELF shlex)

Read a raw token.

Generic Function: token SELF
Generic Function: (setf token) VALUE SELF
Package

shlex

Methods
Method: token (SELF shlex)
Source

shlex.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.6 Conditions

Condition: no-closing-quotation ()
Package

shlex

Source

shlex.lisp (file)

Direct superclasses

shlex-error (condition)

Condition: no-escaped-character ()
Package

shlex

Source

shlex.lisp (file)

Direct superclasses

shlex-error (condition)

Condition: shlex-error ()
Package

shlex

Source

shlex.lisp (file)

Direct superclasses

error (condition)

Direct subclasses
Direct slots
Slot: lexer
Initargs

:lexer


Previous: , Up: Internal definitions   [Contents][Index]

5.2.7 Classes

Class: shlex ()
Package

shlex

Source

shlex.lisp (file)

Direct superclasses

standard-object (class)

Direct methods
Direct slots
Slot: pushback
Slot: pushback-chars
Initform

(serapeum:queue)

Slot: punctuation-chars

Characters that will be considered punctuation. Runs of punctuation are returned as a single token.

Type

string

Slot: line-number

Source line number.

Initform

1

Readers

lexer-line-number (generic function)

Slot: instream
Type

stream

Initargs

:instream

Slot: state
Type

(or null character)

Initform

#\

Slot: posix

If true, use POSIX-esque parsing rules.

Type

boolean

Initargs

:posix

Slot: c-strings

If true, interpret ANSI-C quoting.

Type

boolean

Initargs

:c-strings

Slot: whitespace-split

If true, only split tokens on whitespaces.

Type

boolean

Initargs

:whitespace-split

Slot: whitespace

Characters that bound tokens.

Type

string

Initargs

:whitespace

Slot: commenters

String of characters to recognize as comment beginnners.

Type

string

Initargs

:commenters

Slot: escape

Characters to be considered an escape.

Type

string

Initargs

:escape

Slot: word-chars

String of characters to accumulate into tokens.

Type

string

Slot: quotes

Characters that will be considered string quotes.

Type

string

Initargs

:quotes

Slot: escaped-quotes

The subset of QUOTES that allow escapes.

Type

string

Initargs

:escaped-quotes

Slot: eof

The token that determines EOF.

Slot: debug-input
Type

string

Initargs

:debug-input

Readers

lexer-input (generic function)

Slot: token-buffer

Token buffer.

Initform

(make-array 10 :adjustable t :fill-pointer 0 :element-type (quote character))

Direct Default Initargs
InitargValue
:posixnil
:c-stringsnil
:whitespaceshlex::default-whitespace
:whitespace-splitnil
:commentersshlex::default-commenters
:quotes"'\""
:escape"\\"
:escaped-quotes"\""
:debug-input"unknown"

Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   F   L   S  
Index Entry  Section

F
File, Lisp, shlex.asd: The shlex․asd file
File, Lisp, shlex/package.lisp: The shlex/package․lisp file
File, Lisp, shlex/shlex.lisp: The shlex/shlex․lisp file

L
Lisp File, shlex.asd: The shlex․asd file
Lisp File, shlex/package.lisp: The shlex/package․lisp file
Lisp File, shlex/shlex.lisp: The shlex/shlex․lisp file

S
shlex.asd: The shlex․asd file
shlex/package.lisp: The shlex/package․lisp file
shlex/shlex.lisp: The shlex/shlex․lisp file

Jump to:   F   L   S  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
B   C   D   E   F   G   L   M   Q   R   S   T  
Index Entry  Section

(
(setf token): Internal generic functions

B
bisect-left: Internal functions

C
Compiler Macro, make-lexer: Internal compiler macros

D
do-tokens: Exported macros

E
ensure-string-stream: Internal functions

F
find-sorted: Internal functions
Function, bisect-left: Internal functions
Function, ensure-string-stream: Internal functions
Function, find-sorted: Internal functions
Function, lexer-tokens: Internal functions
Function, make-lexer: Internal functions
Function, map-lexer-tokens: Internal functions
Function, quote: Exported functions
Function, safe-char?: Internal functions
Function, split: Exported functions
Function, string-conc: Internal functions

G
Generic Function, (setf token): Internal generic functions
Generic Function, get-token: Internal generic functions
Generic Function, lexer-input: Internal generic functions
Generic Function, lexer-line-number: Internal generic functions
Generic Function, map-tokens: Exported generic functions
Generic Function, read-token: Internal generic functions
Generic Function, token: Internal generic functions
get-token: Internal generic functions
get-token: Internal generic functions

L
lexer-input: Internal generic functions
lexer-input: Internal generic functions
lexer-line-number: Internal generic functions
lexer-line-number: Internal generic functions
lexer-tokens: Internal functions

M
Macro, do-tokens: Exported macros
make-lexer: Internal compiler macros
make-lexer: Internal functions
map-lexer-tokens: Internal functions
map-tokens: Exported generic functions
map-tokens: Exported generic functions
map-tokens: Exported generic functions
Method, get-token: Internal generic functions
Method, lexer-input: Internal generic functions
Method, lexer-line-number: Internal generic functions
Method, map-tokens: Exported generic functions
Method, map-tokens: Exported generic functions
Method, read-token: Internal generic functions
Method, token: Internal generic functions

Q
quote: Exported functions

R
read-token: Internal generic functions
read-token: Internal generic functions

S
safe-char?: Internal functions
split: Exported functions
string-conc: Internal functions

T
token: Internal generic functions
token: Internal generic functions

Jump to:   (  
B   C   D   E   F   G   L   M   Q   R   S   T  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   +  
C   D   E   I   L   N   P   Q   S   T   W  
Index Entry  Section

+
+storage-for-deflex-var-safe-char-map+: Internal constants

C
c-strings: Internal classes
commenters: Internal classes
Constant, +storage-for-deflex-var-safe-char-map+: Internal constants

D
debug-input: Internal classes
default-commenters: Internal symbol macros
default-whitespace: Internal symbol macros

E
eof: Internal classes
escape: Internal classes
escaped-quotes: Internal classes

I
instream: Internal classes

L
lexer: Internal conditions
line-number: Internal classes

N
non-posix-word-chars: Internal symbol macros

P
posix: Internal classes
posix-word-chars: Internal symbol macros
punctuation-chars: Internal classes
pushback: Internal classes
pushback-chars: Internal classes

Q
quotes: Internal classes

S
safe-char-map: Internal symbol macros
Slot, c-strings: Internal classes
Slot, commenters: Internal classes
Slot, debug-input: Internal classes
Slot, eof: Internal classes
Slot, escape: Internal classes
Slot, escaped-quotes: Internal classes
Slot, instream: Internal classes
Slot, lexer: Internal conditions
Slot, line-number: Internal classes
Slot, posix: Internal classes
Slot, punctuation-chars: Internal classes
Slot, pushback: Internal classes
Slot, pushback-chars: Internal classes
Slot, quotes: Internal classes
Slot, state: Internal classes
Slot, token-buffer: Internal classes
Slot, whitespace: Internal classes
Slot, whitespace-split: Internal classes
Slot, word-chars: Internal classes
state: Internal classes
Symbol Macro, default-commenters: Internal symbol macros
Symbol Macro, default-whitespace: Internal symbol macros
Symbol Macro, non-posix-word-chars: Internal symbol macros
Symbol Macro, posix-word-chars: Internal symbol macros
Symbol Macro, safe-char-map: Internal symbol macros

T
token-buffer: Internal classes

W
whitespace: Internal classes
whitespace-split: Internal classes
word-chars: Internal classes

Jump to:   +  
C   D   E   I   L   N   P   Q   S   T   W  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   N   P   S  
Index Entry  Section

C
Class, shlex: Internal classes
Condition, no-closing-quotation: Internal conditions
Condition, no-escaped-character: Internal conditions
Condition, shlex-error: Internal conditions

N
no-closing-quotation: Internal conditions
no-escaped-character: Internal conditions

P
Package, shlex: The shlex package

S
shlex: The shlex system
shlex: The shlex package
shlex: Internal classes
shlex-error: Internal conditions
System, shlex: The shlex system

Jump to:   C   N   P   S