Next: Introduction, Previous: (dir), Up: (dir) [Contents][Index]
This is the 3bz Reference Manual, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Dec 02 08:57:36 2019 GMT+0.
| • Introduction | What 3bz is all about | |
| • Systems | The systems documentation | |
| • Files | The files documentation | |
| • Packages | The packages documentation | |
| • Definitions | The symbols documentation | |
| • Indexes | Concepts, functions, variables and data types |
Yet another CL impementation of rfc1951 deflate decompression (optionally with rfc1950 zlib or rfc1952 gzip wrappers), with support for reading from foreign pointers (for use with mmap and similar, etc), and from CL octet vectors and streams.
Performance for vectors/pointers is somewhere between FFI to libz and chipz, still needs some low-level optimization of copy routines and checksums. Stream API is very slow, and may be replaced at some point.
API isn't completely stable yet, needs some actual use to figure out the details.
decompress-vector (compressed &key (format :zlib) output
;; pass a (simple-array (unsigned-byte 8) (*))
(3bz:decompress-vector (alexandria:read-file-into-byte-vector "foo.gz")
:format :gzip) ;; accepts :deflate, :zlib, :gzip
;; ->
#(....)
1234
;; get back decompressed data and size as 2 values
If decompressed size is known, you can save some copies by passing in
a pre-allocated (simple-array (unsigned-byte 8) (*)) vector with
:OUTPUT.
Allows input and output in multiple pieces, as well as input from vectors, FFI pointers, or streams (streams are currently very slow though).
step 1: make a decompression state:
make-deflate-state, make-zlib-state, or make-gzip-state.
Optionally with :output-buffer octet-vector to provide initial
output buffer.
step 2: make an input context
make-octet-vector-context, make-octet-stream-context, or with-octet-pointer+make-octet-pointer-context.
pass source of coresponding type, + optional :start,:end,:offset
to specify valid region within source. For FFI pointers, use
(with-octet-pointer (octet-pointer ffi-pointer size) ...) to wrap a
raw pointer + size into octet-pointer to pass to
make-octet-pointer-context. (If you need indefinite scope pointers,
file an issue so support that can be added to API.)
step 3: decompress
(decompress context state) returns current offset into output buffer.
step 4: check decompression state
if (finished state), you are done.
if (input-underrun state), you need to supply more input by creating a new input context, and call decompress again with new context.
if (output-overflow state), you need to provide a new output buffer with (replace-output-buffer state new-buffer) and call decompress again.
Streams API is currently very slow, and will probably be rewritten at some point.
Output in small pieces is slow:
deflate needs to maintain a 32kb history window, so every time
output buffer overflows, it copies 32kb of history. If output buffer
is smaller than 32kb, that means more history is copied than total
output, so avoid small output buffers.
Ideally, if output size is known in advance and output buffer never overflows, there is no extra copies for history window.
If output size isn't known, if you start with something like (min input-size 32768) and double (or at least scale by 1.x) on each
overflow, there will only be O(N) bytes copied to combine output
buffers, and O(lgN) byytes copied for history.
Next: Files, Previous: Introduction, Up: Top [Contents][Index]
The main system appears first, followed by any subsystem dependency.
| • The 3bz system |
Bart Botta <00003b at gmail.com>
MIT
deflate decompressor
3bz.asd (file)
Files are sorted by type and then listed depth-first from the systems components trees.
| • Lisp files |
Next: The 3bz/package․lisp file, Previous: Lisp files, Up: Lisp files [Contents][Index]
3bz.asd
3bz (system)
Next: The 3bz/util․lisp file, Previous: The 3bz․asd file, Up: Lisp files [Contents][Index]
Next: The 3bz/constants․lisp file, Previous: The 3bz/package․lisp file, Up: Lisp files [Contents][Index]
package.lisp (file)
3bz (system)
util.lisp
Next: The 3bz/types․lisp file, Previous: The 3bz/util․lisp file, Up: Lisp files [Contents][Index]
util.lisp (file)
3bz (system)
constants.lisp
Next: The 3bz/huffman-tree․lisp file, Previous: The 3bz/constants․lisp file, Up: Lisp files [Contents][Index]
constants.lisp (file)
3bz (system)
types.lisp
Next: The 3bz/io․lisp file, Previous: The 3bz/types․lisp file, Up: Lisp files [Contents][Index]
types.lisp (file)
3bz (system)
huffman-tree.lisp
Next: The 3bz/deflate․lisp file, Previous: The 3bz/huffman-tree․lisp file, Up: Lisp files [Contents][Index]
huffman-tree.lisp (file)
3bz (system)
io.lisp
Next: The 3bz/checksums․lisp file, Previous: The 3bz/io․lisp file, Up: Lisp files [Contents][Index]
io.lisp (file)
3bz (system)
deflate.lisp
make-deflate-state (function)
Next: The 3bz/zlib․lisp file, Previous: The 3bz/deflate․lisp file, Up: Lisp files [Contents][Index]
deflate.lisp (file)
3bz (system)
checksums.lisp
Next: The 3bz/gzip․lisp file, Previous: The 3bz/checksums․lisp file, Up: Lisp files [Contents][Index]
checksums.lisp (file)
3bz (system)
zlib.lisp
make-zlib-state (function)
Next: The 3bz/api․lisp file, Previous: The 3bz/zlib․lisp file, Up: Lisp files [Contents][Index]
zlib.lisp (file)
3bz (system)
gzip.lisp
make-gzip-state (function)
Previous: The 3bz/gzip․lisp file, Up: Lisp files [Contents][Index]
gzip.lisp (file)
3bz (system)
api.lisp
Next: Definitions, Previous: Files, Up: Top [Contents][Index]
Packages are listed by definition order.
| • The 3bz package |
package.lisp (file)
common-lisp
Definitions are sorted by export status, category, package, and then by lexicographic order.
| • Exported definitions | ||
| • Internal definitions |
Next: Internal definitions, Previous: Definitions, Up: Definitions [Contents][Index]
| • Exported macros | ||
| • Exported functions | ||
| • Exported generic functions |
Next: Exported functions, Previous: Exported definitions, Up: Exported definitions [Contents][Index]
Next: Exported generic functions, Previous: Exported macros, Up: Exported definitions [Contents][Index]
decompress octet-vector COMPRESSED using
FORMAT (:deflate,:zlib,:gzip). If output is supplied, it should be an
octet-vector large enough to hold entire uncompressed output.
Returns buffer containing decompressed data (OUTPUT if supplied) and # of octets decompressed.
deflate.lisp (file)
Previous: Exported functions, Up: Exported definitions [Contents][Index]
Previous: Exported definitions, Up: Definitions [Contents][Index]
| • Internal constants | ||
| • Internal special variables | ||
| • Internal macros | ||
| • Internal functions | ||
| • Internal generic functions | ||
| • Internal structures | ||
| • Internal classes | ||
| • Internal types |
Next: Internal special variables, Previous: Internal definitions, Up: Internal definitions [Contents][Index]
constants.lisp (file)
checksums.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
constants.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
Next: Internal macros, Previous: Internal constants, Up: Internal definitions [Contents][Index]
huffman-tree.lisp (file)
huffman-tree.lisp (file)
deflate.lisp (file)
Next: Internal functions, Previous: Internal special variables, Up: Internal definitions [Contents][Index]
deflate.lisp (file)
Next: Internal generic functions, Previous: Internal macros, Up: Internal definitions [Contents][Index]
checksums.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
deflate.lisp (file)
huffman-tree.lisp (file)
checksums.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
deflate.lisp (file)
checksums.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
huffman-tree.lisp (file)
Next: Internal structures, Previous: Internal functions, Up: Internal definitions [Contents][Index]
automatically generated reader method
io.lisp (file)
automatically generated reader method
io.lisp (file)
automatically generated reader method
io.lisp (file)
Next: Internal classes, Previous: Internal generic functions, Up: Internal definitions [Contents][Index]
io.lisp (file)
structure-object (structure)
3bz::size-t
0
cb-start (function)
(setf cb-start) (function)
3bz::size-t
0
cb-end (function)
(setf cb-end) (function)
3bz::size-t
0
cb-offset (function)
(setf cb-offset) (function)
deflate.lisp (file)
structure-object (structure)
:start-of-block
ds-current-state (function)
(setf ds-current-state) (function)
(or nil t)
ds-last-block-flag (function)
(setf ds-last-block-flag) (function)
(cons 3bz::huffman-tree 3bz::huffman-tree)
(cons (3bz::make-huffman-tree) (3bz::make-huffman-tree))
ds-dynamic-huffman-tree (function)
(setf ds-dynamic-huffman-tree) (function)
(cons 3bz::huffman-tree 3bz::huffman-tree)
3bz::+static-huffman-trees+
ds-current-huffman-tree (function)
(setf ds-current-huffman-tree) (function)
(unsigned-byte 10)
0
ds-dht-hlit (function)
(setf ds-dht-hlit) (function)
(unsigned-byte 10)
0
ds-dht-hlit+hdist (function)
(setf ds-dht-hlit+hdist) (function)
(unsigned-byte 4)
0
ds-dht-hclen (function)
(setf ds-dht-hclen) (function)
(simple-array (unsigned-byte 4) (19))
(make-array 19 :element-type (quote (unsigned-byte 4)) :initial-element 0)
ds-dht-len-codes (function)
(setf ds-dht-len-codes) (function)
(3bz::make-huffman-tree)
ds-dht-len-tree (function)
(setf ds-dht-len-tree) (function)
3bz::code-table-type
(make-array (+ 288 32) :element-type (quote (unsigned-byte 4)) :initial-element 0)
ds-dht-lit/len/dist (function)
(setf ds-dht-lit/len/dist) (function)
(mod 320)
0
ds-dht-lit/len/dist-index (function)
(setf ds-dht-lit/len/dist-index) (function)
3bz::octet
255
ds-dht-last-len (function)
(setf ds-dht-last-len) (function)
(unsigned-byte 16)
0
ds-bytes-to-copy (function)
(setf ds-bytes-to-copy) (function)
(unsigned-byte 16)
0
ds-copy-offset (function)
(setf ds-copy-offset) (function)
(unsigned-byte 64)
0
ds-partial-bits (function)
(setf ds-partial-bits) (function)
(unsigned-byte 7)
0
ds-bits-remaining (function)
(setf ds-bits-remaining) (function)
fixnum
0
ds-output-offset (function)
(setf ds-output-offset) (function)
3bz::octet-vector
(make-array 0 :element-type (quote 3bz::octet))
ds-output-buffer (function)
(setf ds-output-buffer) (function)
(or null 3bz::octet-vector)
ds-window (function)
(setf ds-window) (function)
(or nil t)
ds-finished (function)
(setf ds-finished) (function)
(or nil t)
ds-output-overflow (function)
(setf ds-output-overflow) (function)
(or nil t)
ds-input-underrun (function)
(setf ds-input-underrun) (function)
gzip.lisp (file)
deflate-state (structure)
:header
gs-gzip-state (function)
(setf gs-gzip-state) (function)
gs-compression-method (function)
(setf gs-compression-method) (function)
gs-flags (function)
(setf gs-flags) (function)
gs-extra (function)
(setf gs-extra) (function)
gs-name (function)
(setf gs-name) (function)
gs-comment (function)
(setf gs-comment) (function)
gs-operating-system (function)
(setf gs-operating-system) (function)
gs-mtime/unix (function)
(setf gs-mtime/unix) (function)
gs-mtime/universal (function)
(setf gs-mtime/universal) (function)
:default
gs-compression-level (function)
(setf gs-compression-level) (function)
(make-array 16 :adjustable t :fill-pointer 0)
gs-header-bytes (function)
(setf gs-header-bytes) (function)
(unsigned-byte 32)
0
gs-crc32 (function)
(setf gs-crc32) (function)
huffman-tree.lisp (file)
structure-object (structure)
3bz::ht-bit-count-type
0
ht-start-bits (function)
(setf ht-start-bits) (function)
(mod 29)
0
ht-max-bits (function)
(setf ht-max-bits) (function)
3bz::ht-node-array-type
(make-array 3bz::+max-tree-size+ :element-type (quote 3bz::ht-node-type) :initial-element (3bz::ht-invalid-node))
ht-nodes (function)
(setf ht-nodes) (function)
zlib.lisp (file)
deflate-state (structure)
:header
zs-zlib-state (function)
(setf zs-zlib-state) (function)
zs-compression-method (function)
(setf zs-compression-method) (function)
0
zs-window-size (function)
(setf zs-window-size) (function)
zs-dict-id (function)
(setf zs-dict-id) (function)
:default
zs-compression-level (function)
(setf zs-compression-level) (function)
(unsigned-byte 16)
1
zs-s1 (function)
(setf zs-s1) (function)
(unsigned-byte 16)
0
zs-s2 (function)
(setf zs-s2) (function)
Next: Internal types, Previous: Internal structures, Up: Internal definitions [Contents][Index]
io.lisp (file)
standard-object (class)
:octet-stream
octet-stream (generic function)
:boxes
boxes (generic function)
io.lisp (file)
standard-object (class)
:octet-vector
octet-vector (generic function)
:boxes
boxes (generic function)
Previous: Internal classes, Up: Internal definitions [Contents][Index]
types.lisp (file)
types.lisp (file)
types.lisp (file)
types.lisp (file)
types.lisp (file)
Previous: Definitions, Up: Top [Contents][Index]
| • Concept index | ||
| • Function index | ||
| • Variable index | ||
| • Data type index |
Next: Function index, Previous: Indexes, Up: Indexes [Contents][Index]
| Jump to: | 3
F L |
|---|
| Jump to: | 3
F L |
|---|
Next: Variable index, Previous: Concept index, Up: Indexes [Contents][Index]
| Jump to: | %
(
A B C D F G H I M O R S V W Z |
|---|
| Jump to: | %
(
A B C D F G H I M O R S V W Z |
|---|
Next: Data type index, Previous: Function index, Up: Indexes [Contents][Index]
| Jump to: | *
+
B C D E F G H I L M N O P S W Z |
|---|
| Jump to: | *
+
B C D E F G H I L M N O P S W Z |
|---|
Previous: Variable index, Up: Indexes [Contents][Index]
| Jump to: | 3
C D G H O P S T Z |
|---|
| Jump to: | 3
C D G H O P S T Z |
|---|