The stepster Reference Manual

This is the stepster Reference Manual, version 0.1.5, generated automatically by Declt version 4.0 beta 2 "William Riker" on Mon Feb 26 17:57:54 2024 GMT+0.

Table of Contents


1 Introduction


2 Systems

The main system appears first, followed by any subsystem dependency.


2.1 stepster

Web scraping library

Author

Walpurgisnatch

License

MIT

Version

0.1.5

Dependencies
  • dexador (system).
  • plump (system).
  • clss (system).
  • cl-ppcre (system).
  • babel (system).
  • jonathan (system).
  • cl-reexport (system).
Source

stepster.asd.

Child Component

src (module).


3 Modules

Modules are listed depth-first from the system components tree.


3.1 stepster/src

Source

stepster.asd.

Parent Component

stepster (system).

Child Components

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


4.1 Lisp


4.1.1 stepster/stepster.asd

Source

stepster.asd.

Parent Component

stepster (system).

ASDF Systems

stepster.


4.1.2 stepster/src/utils.lisp

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.utils.

Public Interface
Internals

4.1.3 stepster/src/urlworks.lisp

Dependency

utils.lisp (file).

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.urlworks.

Public Interface
Internals

4.1.4 stepster/src/json-works.lisp

Dependency

urlworks.lisp (file).

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.json-works.

Public Interface
Internals

4.1.5 stepster/src/parser.lisp

Dependency

json-works.lisp (file).

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.parser.

Public Interface
Internals

4.1.6 stepster/src/extensions.lisp

Dependency

parser.lisp (file).

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.extensions.

Public Interface
Internals

for-js (macro).


4.1.7 stepster/src/stepster.lisp

Dependency

extensions.lisp (file).

Source

stepster.asd.

Parent Component

src (module).

Packages

stepster.


5 Packages

Packages are listed by definition order.


5.1 stepster.json-works

Source

json-works.lisp.

Use List

common-lisp.

Public Interface
Internals

5.2 stepster.parser

Source

parser.lisp.

Use List

common-lisp.

Used By List

stepster.extensions.

Public Interface
Internals

5.3 stepster.extensions

Source

extensions.lisp.

Use List
Public Interface
Internals

for-js (macro).


5.4 stepster.urlworks

Source

urlworks.lisp.

Use List

common-lisp.

Public Interface
Internals

5.5 stepster.utils

Source

utils.lisp.

Use List

common-lisp.

Public Interface
Internals

5.6 stepster

Source

stepster.lisp.

Nickname

ss

Use List

common-lisp.


6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


6.1 Public Interface


6.1.1 Ordinary functions

Function: arguments-keys (list)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: arguments-values (list)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: attribute (node attr)

Return attribute from node.

Package

stepster.parser.

Source

parser.lisp.

Function: carlast (x)
Package

stepster.utils.

Source

utils.lisp.

Function: check-attr (attribute)
Package

stepster.parser.

Source

parser.lisp.

Function: clist (x &rest args)
Package

stepster.utils.

Source

utils.lisp.

Function: collect-from (parent-node selectors &key attr test test-args)

Return list of nodes or attributes from parrent node.

Package

stepster.parser.

Source

parser.lisp.

Function: collect-json (json keys)
Package

stepster.json-works.

Source

json-works.lisp.

Function: download-all-images (url dir)
Package

stepster.extensions.

Source

extensions.lisp.

Function: download-file (url filename)
Package

stepster.parser.

Source

parser.lisp.

Function: download-page (url filename)
Package

stepster.parser.

Source

parser.lisp.

Function: equal-getf (plist indicator)
Package

stepster.utils.

Source

utils.lisp.

Function: extract-forms (page &optional name)

Return list of forms from page or single form if page has only one form or specified form-name.

Package

stepster.parser.

Source

parser.lisp.

Function: extract-input-names (page)
Package

stepster.extensions.

Source

extensions.lisp.

Function: extract-js-src (page)
Package

stepster.extensions.

Source

extensions.lisp.

Function: extract-urls (page &optional test arg)
Package

stepster.extensions.

Source

extensions.lisp.

Function: fill-form (form data)

Return list of pairs (input-name value).

Package

stepster.parser.

Source

parser.lisp.

Function: find-by-text (parent-node text &key attr)
Package

stepster.parser.

Source

parser.lisp.

Function: flatten (x)
Package

stepster.utils.

Source

utils.lisp.

Function: get-arguments (url)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: get-json (url)
Package

stepster.parser.

Source

parser.lisp.

Function: get-last (url)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: get-status-code (url)
Package

stepster.parser.

Source

parser.lisp.

Function: get-values (list keys)
Package

stepster.json-works.

Source

json-works.lisp.

Function: getj (list key)
Package

stepster.json-works.

Source

json-works.lisp.

Function: internks (item)
Package

stepster.utils.

Source

utils.lisp.

Function: jfinder (json key)
Package

stepster.json-works.

Source

json-works.lisp.

Function: join-with-main (main path)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: last-char (x)
Package

stepster.utils.

Source

utils.lisp.

Function: make-query-string (url data)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: member-list (x y)
Package

stepster.utils.

Source

utils.lisp.

Function: mklist (el)
Package

stepster.utils.

Source

utils.lisp.

Function: node-with-attr (parent-node selector attr val)
Package

stepster.parser.

Source

parser.lisp.

Function: pack-to-json (keys values)
Package

stepster.json-works.

Source

json-works.lisp.

Function: pack-with (keys values)
Package

stepster.json-works.

Source

json-works.lisp.

Function: page-text (page)

Return string of text from all of the children nodes.

Package

stepster.parser.

Source

parser.lisp.

Function: parse (page &key headers)

Return plump root node for given url

Package

stepster.parser.

Source

parser.lisp.

Function: parse-regex (url regex)
Package

stepster.parser.

Source

extensions.lisp.

Function: parse-text (page selectors)
Package

stepster.parser.

Source

parser.lisp.

Function: post-json (url data)
Package

stepster.parser.

Source

parser.lisp.

Function: prepare-url (url &optional main)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: print-error (e)
Package

stepster.utils.

Source

utils.lisp.

Function: query-string (data)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: replace-argument (url arg value)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: reverse-group (source n)

Split list into groups by n and reverse them.

Package

stepster.utils.

Source

utils.lisp.

Function: safe-get (url &key headers)
Package

stepster.parser.

Source

parser.lisp.

Function: safe-post (url data &key headers)
Package

stepster.parser.

Source

parser.lisp.

Function: same-domain (url domain)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: string-starts-with (string x)
Package

stepster.utils.

Source

utils.lisp.

Function: submit-form (url &key form data)
Package

stepster.parser.

Source

parser.lisp.

Function: substp (regex string)
Package

stepster.utils.

Source

utils.lisp.


6.2 Internals


6.2.1 Special variables

Special Variable: *cookie-jar*
Package

stepster.parser.

Source

parser.lisp.

Special Variable: *user-agent-header*
Package

stepster.parser.

Source

parser.lisp.


6.2.2 Macros

Macro: for-js (page &body body)
Package

stepster.extensions.

Source

extensions.lisp.


6.2.3 Ordinary functions

Function: component-present-p (value)
Package

stepster.utils.

Source

utils.lisp.

Function: directory-pathname-p (p)
Package

stepster.utils.

Source

utils.lisp.

Function: fj (json keys &optional acc list)
Package

stepster.json-works.

Source

json-works.lisp.

Function: get-main (url)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: http-join (url &key http-only)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: intern-list (list)
Package

stepster.json-works.

Source

json-works.lisp.

Function: internk (item)
Package

stepster.utils.

Source

utils.lisp.

Function: nodes-to-string (list)

Return selector string from list of symbols.

Package

stepster.parser.

Source

parser.lisp.

Function: pathname-as-directory (name)
Package

stepster.utils.

Source

utils.lisp.

Function: relative (url)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: replace-arguments-with-value (url value)
Package

stepster.urlworks.

Source

urlworks.lisp.

Function: setf-assoc (field key value)
Package

stepster.utils.

Source

utils.lisp.

Function: split-url (url)
Package

stepster.urlworks.

Source

urlworks.lisp.


Appendix A Indexes


A.1 Concepts


A.2 Functions

Jump to:   A   C   D   E   F   G   H   I   J   L   M   N   P   Q   R   S  
Index Entry  Section

A
arguments-keys: Public ordinary functions
arguments-values: Public ordinary functions
attribute: Public ordinary functions

C
carlast: Public ordinary functions
check-attr: Public ordinary functions
clist: Public ordinary functions
collect-from: Public ordinary functions
collect-json: Public ordinary functions
component-present-p: Private ordinary functions

D
directory-pathname-p: Private ordinary functions
download-all-images: Public ordinary functions
download-file: Public ordinary functions
download-page: Public ordinary functions

E
equal-getf: Public ordinary functions
extract-forms: Public ordinary functions
extract-input-names: Public ordinary functions
extract-js-src: Public ordinary functions
extract-urls: Public ordinary functions

F
fill-form: Public ordinary functions
find-by-text: Public ordinary functions
fj: Private ordinary functions
flatten: Public ordinary functions
for-js: Private macros
Function, arguments-keys: Public ordinary functions
Function, arguments-values: Public ordinary functions
Function, attribute: Public ordinary functions
Function, carlast: Public ordinary functions
Function, check-attr: Public ordinary functions
Function, clist: Public ordinary functions
Function, collect-from: Public ordinary functions
Function, collect-json: Public ordinary functions
Function, component-present-p: Private ordinary functions
Function, directory-pathname-p: Private ordinary functions
Function, download-all-images: Public ordinary functions
Function, download-file: Public ordinary functions
Function, download-page: Public ordinary functions
Function, equal-getf: Public ordinary functions
Function, extract-forms: Public ordinary functions
Function, extract-input-names: Public ordinary functions
Function, extract-js-src: Public ordinary functions
Function, extract-urls: Public ordinary functions
Function, fill-form: Public ordinary functions
Function, find-by-text: Public ordinary functions
Function, fj: Private ordinary functions
Function, flatten: Public ordinary functions
Function, get-arguments: Public ordinary functions
Function, get-json: Public ordinary functions
Function, get-last: Public ordinary functions
Function, get-main: Private ordinary functions
Function, get-status-code: Public ordinary functions
Function, get-values: Public ordinary functions
Function, getj: Public ordinary functions
Function, http-join: Private ordinary functions
Function, intern-list: Private ordinary functions
Function, internk: Private ordinary functions
Function, internks: Public ordinary functions
Function, jfinder: Public ordinary functions
Function, join-with-main: Public ordinary functions
Function, last-char: Public ordinary functions
Function, make-query-string: Public ordinary functions
Function, member-list: Public ordinary functions
Function, mklist: Public ordinary functions
Function, node-with-attr: Public ordinary functions
Function, nodes-to-string: Private ordinary functions
Function, pack-to-json: Public ordinary functions
Function, pack-with: Public ordinary functions
Function, page-text: Public ordinary functions
Function, parse: Public ordinary functions
Function, parse-regex: Public ordinary functions
Function, parse-text: Public ordinary functions
Function, pathname-as-directory: Private ordinary functions
Function, post-json: Public ordinary functions
Function, prepare-url: Public ordinary functions
Function, print-error: Public ordinary functions
Function, query-string: Public ordinary functions
Function, relative: Private ordinary functions
Function, replace-argument: Public ordinary functions
Function, replace-arguments-with-value: Private ordinary functions
Function, reverse-group: Public ordinary functions
Function, safe-get: Public ordinary functions
Function, safe-post: Public ordinary functions
Function, same-domain: Public ordinary functions
Function, setf-assoc: Private ordinary functions
Function, split-url: Private ordinary functions
Function, string-starts-with: Public ordinary functions
Function, submit-form: Public ordinary functions
Function, substp: Public ordinary functions

G
get-arguments: Public ordinary functions
get-json: Public ordinary functions
get-last: Public ordinary functions
get-main: Private ordinary functions
get-status-code: Public ordinary functions
get-values: Public ordinary functions
getj: Public ordinary functions

H
http-join: Private ordinary functions

I
intern-list: Private ordinary functions
internk: Private ordinary functions
internks: Public ordinary functions

J
jfinder: Public ordinary functions
join-with-main: Public ordinary functions

L
last-char: Public ordinary functions

M
Macro, for-js: Private macros
make-query-string: Public ordinary functions
member-list: Public ordinary functions
mklist: Public ordinary functions

N
node-with-attr: Public ordinary functions
nodes-to-string: Private ordinary functions

P
pack-to-json: Public ordinary functions
pack-with: Public ordinary functions
page-text: Public ordinary functions
parse: Public ordinary functions
parse-regex: Public ordinary functions
parse-text: Public ordinary functions
pathname-as-directory: Private ordinary functions
post-json: Public ordinary functions
prepare-url: Public ordinary functions
print-error: Public ordinary functions

Q
query-string: Public ordinary functions

R
relative: Private ordinary functions
replace-argument: Public ordinary functions
replace-arguments-with-value: Private ordinary functions
reverse-group: Public ordinary functions

S
safe-get: Public ordinary functions
safe-post: Public ordinary functions
same-domain: Public ordinary functions
setf-assoc: Private ordinary functions
split-url: Private ordinary functions
string-starts-with: Public ordinary functions
submit-form: Public ordinary functions
substp: Public ordinary functions


A.4 Data types

Jump to:   E   F   J   M   P   S   U  
Index Entry  Section

E
extensions.lisp: The stepster/src/extensions․lisp file

F
File, extensions.lisp: The stepster/src/extensions․lisp file
File, json-works.lisp: The stepster/src/json-works․lisp file
File, parser.lisp: The stepster/src/parser․lisp file
File, stepster.asd: The stepster/stepster․asd file
File, stepster.lisp: The stepster/src/stepster․lisp file
File, urlworks.lisp: The stepster/src/urlworks․lisp file
File, utils.lisp: The stepster/src/utils․lisp file

J
json-works.lisp: The stepster/src/json-works․lisp file

M
Module, src: The stepster/src module

P
Package, stepster: The stepster package
Package, stepster.extensions: The stepster․extensions package
Package, stepster.json-works: The stepster․json-works package
Package, stepster.parser: The stepster․parser package
Package, stepster.urlworks: The stepster․urlworks package
Package, stepster.utils: The stepster․utils package
parser.lisp: The stepster/src/parser․lisp file

S
src: The stepster/src module
stepster: The stepster system
stepster: The stepster package
stepster.asd: The stepster/stepster․asd file
stepster.extensions: The stepster․extensions package
stepster.json-works: The stepster․json-works package
stepster.lisp: The stepster/src/stepster․lisp file
stepster.parser: The stepster․parser package
stepster.urlworks: The stepster․urlworks package
stepster.utils: The stepster․utils package
System, stepster: The stepster system

U
urlworks.lisp: The stepster/src/urlworks․lisp file
utils.lisp: The stepster/src/utils․lisp file