Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the http-parse Reference Manual, version 0.1.10, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 13:45:07 2020 GMT+0.
| • Introduction | What http-parse is all about | |
| • Systems | The systems documentation | |
| • Files | The files documentation | |
| • Packages | The packages documentation | |
| • Definitions | The symbols documentation | |
| • Indexes | Concepts, functions, variables and data types |
http-parse had a good, long life and served many HTTP requests, but it's now time for it to stand aside and let libraries better than itself take its place. fast-http is an incredible library (and is now the core parser used in Wookie).
Use fast-http instead of http-parse. This library is retired.
A pure-lisp library for parsing HTTP requests/responses. Right now the focus for this library is making it useful and easy to use. With time, slower parts of http-parse will be replaced to make it screaming fast.
The purpose of this library is to be able to easily parse incoming HTTP data either synchronously or asynchronously (but it was mainly built for streaming HTTP data asynchronously).
This is a base class extended by http-request and http-response. It holds values that both requests/responses deal with (HTTP version, headers, body).
This class, and those that extend it, are meant to be instiantiated with no parameters and passed into the make-parser function, which fills in all the details which can be read out later.
Accessor for the parsed HTTP version out of the http object.
Accessor for the headers parsed from the HTTP request/response
store-body specifies whether the HTTP body should be stored (in its entirety)
in the http-body accessor. If this is set after initializing a
parser, it should be done so no later than the header-callback
being fired, or else pieces of the body may be missing.
force-stream lets the parser know that you want every TCP packet that comes in
to be passed into a body callback as if it was sent via an HTTP chunk. This is
an advanced option, but can be very useful in some cases. For instance, if you
have a server that supports file uploads and a client doesn't know how to chunk
an upload (like every browser ever), your server is going to spin its CPU
and waste memory buffering the entire file and passing it around as a huge array
instead of dealing with it packet by packet. This is a great way to fake HTTP
chunking in your server/client.
Accessor for the full HTTP body from the request/response (although storing of the
body in the http object must be explicitely asked for by passing :store-body t
into make-parser.
extends http
Holds values specific to an HTTP request (method, resource)
Accessor for the parsed HTTP request method as a keyword: :GET, :POST, etc.
Accessor for the resource in the request. Parse with puri.
extends http
Holds values specific to an HTTP response (status, status text).
Accessor for the HTTP response status (integer).
Accessor for the HTTP response status string: OK, Insufficient Privileges,
etc.
(defun make-parser (http &key header-callback body-callback multipart-callback finish-callback store-body)
=> closure
This is what you've all been waiting for, folks. This function initializes an HTTP parser (a closure) which can be fed binary data in sequence and will parse an HTTP request/response as it comes in.
It accepts a class of http-request or http-response
as its only required argument. It returns a closure which only has one argument:
a byte array that contains pieces of an HTTP request/response. The pieces must
be in order, but other than that, there is no restriction on how many times the
parser can be called with new data until it is finished. In some cases (older
HTTP versions), the end of an HTTP payload is marked by an EOF on the socket. If
this occurs, you can pass :eof into the parser instead of a byte array to
signal that it should finish up.
The parser closure returns three values: the http object passed in, a boolean indicating if the headers are finished parsing, and a boolean indicating if the HTTP body has been fully parsed.
make-parser accepts these callbacks: :header-callback, :body-callback,
:multipart-callback, and :finish-callback.
:content-length being present in
the headers) or piece by piece as it is sent in (when the body is chunked).The :store-body keyword specifies that the parser should store the body (as a
byte array) into the given http object as it is parsed. Otherwise, the
best way to get the body data is via the body-callback.
;; example. anything under my-app is not included.
(let ((http (make-instance 'http-response))
(parser (make-parser http
:header-callback (lambda (headers)
(my-app:got-headers!!! headers))
:body-callback (lambda (bytes)
(my-app:got-body-piece bytes)))))
(loop for http-data = (my-app:get-http-data-from-request-i-sent-out-earlier) do
(multiple-value-bind (http headers-finished-p body-finished-p)
(funcall parser http-data)
(when body-finished-p
(my-app:close-http-stream))
...)))
(lambda (byte-array) ...)
=> http, headers-finished-p, body-finished-p
As noted, if an EOF happens on the socket the HTTP data is coming in on, you may
indicate this to the parser by sending in :eof instead of the byte array.
(lambda (header-plist) ...)
Headers are in the form '(:host "musio.com" :content-type "text/html" ...).
Headers are not reversed, they are passed in the order they occur in the
HTTP payload.
(lambda (byte-array last-chunk-p) ...)
Byte-array is not cumulative, it is just the new data that has been parsed
from the payload. If multiple chunks are parsed at once, their body data is sent
in as one call to the body-callback. Incomplete chunks are not sent in until
they are completed.
last-chunk-p is true if the entire body has been processed (if a Content-Length
was specified and all bytes accounted for, or if the body is chunked and the
0-byte chunk has been encountered).
See multipart parser callback definition.
(lambda () ...)
This callback is fired when the HTTP parser is finished parsing the request/response.
(defun make-multipart-parser (headers callback))
=> closure
Returns a parser closure to deal with multipart form data. Data is fed to the
parser in as many chunks as needed (or all at once) and the given callback
will be fired at least once for each form field present in the multipart form
data. If data for a field is spread over multiple chunks, the callback is fired
for each of the chunks, along with a second argument indicating whether the
current chunk is that last for that field.
headers are all the headers from a parsed HTTP payload, in plist form.
NOTE: If make-multipart-parser detects that the data being decoded is not in
multipart format (determined by reading the headers), it returns nil instead
of a closure.
(lambda (field-name field-headers field-meta body-bytes body-complete-p) ...)
This callback is fired for each form field encountered in a multipart request.
The field-name arg is a string indicating the name of the form field. The
field-headers arg is a plist containing the headers for that field (generally
this is Content-Disposition and sometimes Content-Type for uploads). The
field-meta arg is a plist of key/value pairs found in the Content-Disposition
header for the field (this is where the field-name arg comes from, and is also
used to specify the filename of uploaded files). body-bytes is a byte array
containing all or a chunk of that field's data, and body-complete-p indicates
whether or not the body-bytes being sent into the callback is the last bit of
data for that field.
Generally, this callback will be a closure that is able to track the current
field it's operating on and be able to handle the case where body-bytes is
spread over multiple calls if body-complete-p is nil.
Tests are under the http-parse-test package:
(ql:quickload :http-parse-test)
(http-parse-test:run-tests)
Please report any bugs you find or failing tests to the issues list.
MIT Licensed. Enjoy.
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
| • The http-parse system |
Andrew Danger Lyon <orthecreedence@gmail.com>
MIT
A library for parsing HTTP requests/responses (synchronous or asynchronous).
0.1.10
http-parse.asd (file)
Files are sorted by type and then listed depth-first from the systems components trees.
| • Lisp files |
| • The http-parse.asd file | ||
| • The http-parse/package.lisp file | ||
| • The http-parse/util.lisp file | ||
| • The http-parse/parse.lisp file | ||
| • The http-parse/multipart-parse.lisp file |
Next: The http-parse/package․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
http-parse.asd
http-parse (system)
Next: The http-parse/util․lisp file, Previous: The http-parse․asd file, Up: Lisp files [Contents][Index]
http-parse (system)
package.lisp
Next: The http-parse/parse․lisp file, Previous: The http-parse/package․lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
http-parse (system)
util.lisp
Next: The http-parse/multipart-parse․lisp file, Previous: The http-parse/util․lisp file, Up: Lisp files [Contents][Index]
util.lisp (file)
http-parse (system)
parse.lisp
Previous: The http-parse/parse․lisp file, Up: Lisp files [Contents][Index]
util.lisp (file)
http-parse (system)
multipart-parse.lisp
make-multipart-parser (function)
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
| • The http-parse package |
package.lisp (file)
common-lisp
Definitions are sorted by export status, category, package, and then by lexicographic order.
| • Exported definitions | ||
| • Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
| • Exported functions | ||
| • Exported generic functions | ||
| • Exported classes |
Next: Exported generic functions, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
Make a multipart parser. Returns a closure that accepts a byte array of data from an HTTP body. Can be sent in in chunks. Callback will be called whenever a complete data chunk is sent in (called for each part in the data chunk) or as a continuation of a chunk that complete headers were sent in for but didn’t finish sending in its body.
multipart-parse.lisp (file)
Return a closure that parses an HTTP request/response by calling it with
the bytes received as its only argument. The closure returns three values:
the http object passed in, a boolean representing whether the headers have
been fully parsed, and a boolean representing whether the request/response
is finished (blank body, all body bytes accounted for, or 0-length chunk
received).
During the parsing, the closure will call (if specified) the ‘header-callback‘
with all the headers as a plist once they are fully parsed, and the
‘body-callback‘ with the body once either it finishes parsing (if we have
Content-Length) or once for each set of completed chunks sent, which allows
streaming the body as it comes in. If a multipart callback is given, it will
be called at least once for each form field present in the multipart form
data.
The ‘:finish-callback‘ will be called when the HTTP payload is fully
processed.
The :store-body keyword indicates to the parser that we wish to keep the
body (in its entirety) in the http object passed in (accessible via the
http-body accessor). Otherwise, the body will be discarded as it’s parsed
(but remember, will still be sent to the body-callback as it comes in).
Parsing can be forced to completion by passing :EOF into the data arg. It
is recommended to do this if the client/server closes the connection before
you do.
parse.lisp (file)
Next: Exported classes, Previous: Exported functions, Up: Exported definitions [Contents][Index]
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
automatically generated reader method
parse.lisp (file)
automatically generated writer method
parse.lisp (file)
Previous: Exported generic functions, Up: Exported definitions [Contents][Index]
Base HTTP class, holds data common to both requests and responses.
parse.lisp (file)
standard-object (class)
:version
1
http-version (generic function)
(setf http-version) (generic function)
:headers
http-headers (generic function)
(setf http-headers) (generic function)
:store-body
http-store-body (generic function)
(setf http-store-body) (generic function)
:force-stream
http-force-stream (generic function)
(setf http-force-stream) (generic function)
:body
(make-array 0 :element-type (quote (unsigned-byte 8)))
http-body (generic function)
(setf http-body) (generic function)
HTTP request class, extends ‘http‘ and holds all request-specific data.
parse.lisp (file)
http (class)
:method
http-method (generic function)
(setf http-method) (generic function)
:resource
"/"
http-resource (generic function)
(setf http-resource) (generic function)
HTTP response class, extends ‘http‘ and holds all response-specific data.
parse.lisp (file)
http (class)
:status
0
http-status (generic function)
(setf http-status) (generic function)
:status-text
""
http-status-text (generic function)
(setf http-status-text) (generic function)
Previous: Exported definitions, Up: Definitions [Contents][Index]
| • Internal special variables | ||
| • Internal functions | ||
| • Internal generic functions |
Next: Internal functions, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
Grabs the key/value pairs from a Content-Disposition header.
multipart-parse.lisp (file)
Create a scanner to find the first header in a string.
parse.lisp (file)
Create a regex scanner for splitting header kv pairs up.
parse.lisp (file)
Create a regex scanner for splitting header lines up.
parse.lisp (file)
Create a regex scanner that detects if a string can be converted to a numver.
parse.lisp (file)
Next: Internal generic functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
Create an array, made up of arr1 followed by arr2.
util.lisp (file)
Pull out headers in a plist from a string.
parse.lisp (file)
Find the position of the first non-whitespace character in a sequence.
util.lisp (file)
Given a chunk (octet vector) of HTTP data, return only the data from the *complete* chunks in the data and return the position in the byte vector of the start of the next chunk, and lastly return whether the last chunk (the 0-byte chunk) has been parsed (ie, HTTP body complete).
parse.lisp (file)
Given the bytes of an HTTP request/response, pull out only the headers and optionally the line above the start of the headers. Returns the headers as a string.
parse.lisp (file)
Given a content-disposition header value, pull out the key/value pairs in it.
multipart-parse.lisp (file)
Previous: Internal functions, Up: Internal definitions [Contents][Index]
Given a slew of HTTP bytes, parse the headers out of them along with any other useful information lurking in there (status code, resource, etc). Returns the HTTP object passed in.
parse.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
| • Concept index | ||
| • Function index | ||
| • Variable index | ||
| • Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
| Jump to: | F H L |
|---|
| Jump to: | F H L |
|---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
| Jump to: | (
A C F G H M P |
|---|
| Jump to: | (
A C F G H M P |
|---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
| Jump to: | *
B F H M R S V |
|---|
| Jump to: | *
B F H M R S V |
|---|
Previous: Variable index, Up: Indexes [Contents][Index]
| Jump to: | C H P S |
|---|
| Jump to: | C H P S |
|---|