The dataloader Reference Manual

Table of Contents

The dataloader Reference Manual

This is the dataloader Reference Manual, version 0.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Sun May 15 04:36:15 2022 GMT+0.

1 Introduction

Dataloader - A universal loader library for various data formats for images/audio

This is what you might have needed for a long time. It loads a file in an arbitrary data format into a numcl-compatible array. File types are automatically detected by the mime type information obtained by libmagic.

Supported files are currently png, jpg, tiff, csv, tsv, wav, npy. We plan to include support for bmp, gif, npz. This library relies on existing libraries to load the files, such as cl-png, retrospectiff, cl-csv, but provides a consistent and natural array-based interface.

An image file is loaded into a NUMCL-compatible array with corresponding width, height and channels.

A sound file is loaded into a 1D array (monoral), or a 2D array with 2 channels (stereo). It provides a wrapper over cl-wav and correctly returns a (unsigned-byte 16) arrays for a 16-bit PCM file, which is not performed by cl-wav. The save method also automatically generates RIFF headers, which is not performed by cl-wav.

A numpy file is loaded with the corresponding width and height using numpy-file-format library.

Load

DATALOADER:LOAD
  [symbol]

LOAD names a compiled function:
  Lambda-list: (FILE &REST ARGS &KEY
                (MIME (MAGICFFI:PATHNAME-MIME-TYPE FILE))
                
                ;; jpeg options
                (DECODE-FRAME T) CACHED-SOURCE-P (COLORSPACE-CONVERSION T) BUFFER
                
                ;; csv options
                ESCAPE-MODE NEWLINE
                TRIM-OUTER-WHITESPACE QUOTED-EMPTY-STRING-IS-NIL
                UNQUOTED-EMPTY-STRING-IS-NIL
                (ESCAPE CL-CSV:*QUOTE-ESCAPE*) 'CL-CSV:*QUOTE*
                (SEPARATOR CL-CSV:*SEPARATOR*) SKIP-FIRST-P SAMPLE
                DATA-MAP-FN MAP-FN ROW-FN CSV-READER TYPE)

This function loads a file and returns a numcl-compatible array. The file format can be specified manually in the mime string, but by default detected by libmagic.

Save

DATALOADER:SAVE
  [symbol]

SAVE names a compiled function:
  Lambda-list: (ARRAY FILE &REST ARGS &KEY (MIME (PATHNAME-TYPE FILE))
  
                ;; wav options
                (BITS-PER-SECOND 44100))

This function saves the array into a file, using the file name extension or the additional mime argument.

Installation

Available from quicklisp.

Dependencies

This library is at least tested on implementation listed below:

• SBCL 1.4.12 on X86-64 Linux 4.4.0-141-generic (author's environment)

Also, it depends on the following libraries:

• cl-wav :

• cl-png :

• cl-jpeg :

• retrospectiff :

• iterate by ** : Jonathan Amsterdam's iterator/gatherer/accumulator facility

• alexandria by Nikodemus Siivola nikodemus@sb-studio.net, and others. : Alexandria is a collection of portable public domain utilities.

• trivia by Masataro Asai : NON-optimized pattern matcher compatible with OPTIMA, with extensible optimizer interface and clean codebase

Author

• Masataro Asai (guicho2.71828@gmail.com)

Copyright

Copyright (c) 2019 Masataro Asai (guicho2.71828@gmail.com)

License

Licensed under the LLGPL License.

2 Systems

The main system appears first, followed by any subsystem dependency.

2.1 dataloader

Author

Masataro Asai

Contact

guicho2.71828@gmail.com

License

LLGPL

Description

A universal loader library for various data formats for images/audio

Version

0.1

Dependencies

• magicffi
• cl-wav
• png
• cl-jpeg
• retrospectiff
• cl-csv
• numpy-file-format
• iterate
• alexandria
• trivia
• numcl

Source

dataloader.asd (file)

Components

• 0package.lisp (file)
• 1generic.lisp (file)
• 2load.lisp (file)
• 2save.lisp (file)
• 3api.lisp (file)

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.

3.1 Lisp

3.1.1 dataloader.asd

Location

/home/quickref/quicklisp/dists/quicklisp/software/dataloader-20210531-git/dataloader.asd

Systems

dataloader (system)

3.1.2 dataloader/0package.lisp

Parent

dataloader (system)

Location

0package.lisp

Packages

• dataloader
• dataloader.mime

3.1.3 dataloader/1generic.lisp

Dependency

0package.lisp (file)

Parent

dataloader (system)

Location

1generic.lisp

Exported Definitions

• define-load-method (macro)
• define-save-method (macro)
• perform-load (generic function)
• perform-load (method)
• perform-load (method)
• perform-save (generic function)
• perform-save (method)
• perform-save (method)

Internal Definitions

• *load-keyword-arg-list* (special variable)
• *save-keyword-arg-list* (special variable)
• add-to-load-keyword-arg-list (function)
• add-to-save-keyword-arg-list (function)
• external-format (function)
• intern-mime (function)

3.1.4 dataloader/2load.lisp

Dependency

1generic.lisp (file)

Parent

dataloader (system)

Location

2load.lisp

Exported Definitions

• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)

Internal Definitions

find-chunk (function)

3.1.5 dataloader/2save.lisp

Dependency

2load.lisp (file)

Parent

dataloader (system)

Location

2save.lisp

Exported Definitions

• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)

3.1.6 dataloader/3api.lisp

Dependency

2save.lisp (file)

Parent

dataloader (system)

Location

3api.lisp

Exported Definitions

• load (function)
• save (function)

4 Packages

Packages are listed by definition order.

4.1 dataloader

The main package for dataloader.

Source

0package.lisp (file)

Use List

• alexandria
• trivia.level2
• iterate
• common-lisp

Exported Definitions

• define-load-method (macro)
• define-save-method (macro)
• load (function)
• perform-load (generic function)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-load (method)
• perform-save (generic function)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• perform-save (method)
• save (function)

Internal Definitions

• *load-keyword-arg-list* (special variable)
• *save-keyword-arg-list* (special variable)
• add-to-load-keyword-arg-list (function)
• add-to-save-keyword-arg-list (function)
• external-format (function)
• find-chunk (function)
• intern-mime (function)

4.2 dataloader.mime

A package containing symbols that are supported by dataloader.
Each symbol has a name read from a mime type string, e.g., "image/png" -> ‘IMAGE/PNG‘ .
The symbols are used as an ‘eql‘ specializer for a generic function ‘dataloader:perform-load‘ and ‘dataloader:perform-save‘.

Source

0package.lisp (file)

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.

5.1 Exported definitions

5.1.1 Macros

Macro: define-load-method (FILEVAR MIME-STRING-OR-STRINGS &rest LAMBDA-LIST) &body BODY

A wrapper over ‘defmethod‘. Made primarily because defmethod does not support ‘(or (eql ...))‘ or ‘member‘ as a specializer. ‘mime-string-or-strings‘ is a string or a list of strings, where each string specifies a mime type e.g. "image/png". ‘lambda-list‘ contains a keyword argument portion of the congruent lambda list of ‘defgeneric‘.
In order to have a compatible lambda-list with the generic function,
it adds ‘&rest‘ argument if not present,
and ‘&allow-other-keys‘ when ‘&key‘ is present and ‘&allow-other-keys‘ is missing.

Package

dataloader

Source

1generic.lisp (file)

Macro: define-save-method (ARRAYVAR FILEVAR MIME-STRING-OR-STRINGS &rest LAMBDA-LIST) &body BODY

A wrapper over ‘defmethod‘. Made primarily because defmethod does not support ‘(or (eql ...))‘ or ‘member‘ as a specializer. ‘mime-string-or-strings‘ is a string or a list of strings, where each string specifies a mime type e.g. "image/png". ‘lambda-list‘ contains a keyword argument portion of the congruent lambda list of ‘defgeneric‘.
In order to have a compatible lambda-list with the generic function,
it adds ‘&rest‘ argument if not present,
and ‘&allow-other-keys‘ when ‘&key‘ is present and ‘&allow-other-keys‘ is missing.

Package

dataloader

Source

1generic.lisp (file)

5.1.2 Functions

Function: load FILE &rest ARGS &key MIME DECODE-FRAME CACHED-SOURCE-P COLORSPACE-CONVERSION BUFFER EXTERNAL-FORMAT ESCAPE-MODE NEWLINE TRIM-OUTER-WHITESPACE QUOTED-EMPTY-STRING-IS-NIL UNQUOTED-EMPTY-STRING-IS-NIL ESCAPE QUOTE SEPARATOR SKIP-FIRST-P SAMPLE DATA-MAP-FN MAP-FN ROW-FN CSV-READER TYPE

Loads a file and returns a numcl-compatible array. The file format is detected by libmagic, but it can optionally be specified.

Package

dataloader

Source

3api.lisp (file)

Function: save ARRAY FILE &rest ARGS &key MIME BITS-PER-SECOND

This function saves the array into a file, using the format specified by the file name extension or by the mime type.

Package

dataloader

Source

3api.lisp (file)

5.1.3 Generic functions

Generic Function: perform-load FILE MIME &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT BUFFER COLORSPACE-CONVERSION CACHED-SOURCE-P DECODE-FRAME &allow-other-keys

A generic function that is defined for each mime type symbol in ‘DATALOADER.MIME‘ package.

Package

dataloader

Source

1generic.lisp (file)

Methods

Method: perform-load FILE (MIME (eql application/octet-stream)) &rest ARGS

fallback method for application/octet-stream mime type.
Since not much can be inferred from this mime type, it calls itself with pathname extension as the second argument.

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql npy)) &rest ARGS0

libmagic can detect numpy file, but does not return a meaningful mime type

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql audio/vnd.wave)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql audio/x-wave)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql audio/wave)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql audio/x-wav)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql audio/wav)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql wav)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/plain)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/tab-separated-values)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/x-comma-separated-values)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/x-csv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/comma-separated-values)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql text/csv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql application/vnd.ms-excel)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql application/x-csv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql application/csv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql tsv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql csv)) &rest ARGS &key TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE EXTERNAL-FORMAT &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql image/tiff)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql tiff)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql image/jpeg)) &rest ARGS &key BUFFER COLORSPACE-CONVERSION CACHED-SOURCE-P DECODE-FRAME &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql jpeg)) &rest ARGS &key BUFFER COLORSPACE-CONVERSION CACHED-SOURCE-P DECODE-FRAME &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql jpg)) &rest ARGS &key BUFFER COLORSPACE-CONVERSION CACHED-SOURCE-P DECODE-FRAME &allow-other-keys

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql image/png)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME (eql png)) &rest ARGS0

Source

2load.lisp (file)

Method: perform-load FILE (MIME string) &rest ARGS &key &allow-other-keys

Convert a mime string into a symbol

Method: perform-load (FILE string) MIME &rest ARGS &key &allow-other-keys

Convert a namestring into a pathanme

Generic Function: perform-save ARRAY FILE MIME &rest ARGS &key BITS-PER-SECOND &allow-other-keys

A generic function that is defined for each mime type symbol in ‘DATALOADER.MIME‘ package.

Package

dataloader

Source

1generic.lisp (file)

Methods

Method: perform-save DATA FILE (MIME (eql npy)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql audio/vnd.wave)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql audio/x-wave)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql audio/wave)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql audio/x-wav)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql audio/wav)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql wav)) &rest ARGS0 &key BITS-PER-SECOND &allow-other-keys

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql text/tab-separated-values)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql tsv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql text/x-comma-separated-values)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql text/x-csv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql text/comma-separated-values)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql text/csv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql application/x-csv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql application/csv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql csv)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql image/tiff)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql tiff)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql image/jpeg)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql jpeg)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql jpg)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql image/png)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save DATA FILE (MIME (eql png)) &rest ARGS0

Source

2save.lisp (file)

Method: perform-save ARRAY FILE (MIME string) &rest ARGS &key &allow-other-keys

Convert a mime string into a symbol

Method: perform-save ARRAY (FILE string) MIME &rest ARGS &key &allow-other-keys

Convert a namestring into a pathanme

5.2 Internal definitions

5.2.1 Special variables

Special Variable: *load-keyword-arg-list*

Package

dataloader

Source

1generic.lisp (file)

Special Variable: *save-keyword-arg-list*

Package

dataloader

Source

1generic.lisp (file)

5.2.2 Functions

Function: add-to-load-keyword-arg-list KEYWORD-ARG-LIST

Package

dataloader

Source

1generic.lisp (file)

Function: add-to-save-keyword-arg-list KEYWORD-ARG-LIST

Package

dataloader

Source

1generic.lisp (file)

Function: external-format FILE

Package

dataloader

Source

1generic.lisp (file)

Function: find-chunk ID RIFF

Package

dataloader

Source

2load.lisp (file)

Function: intern-mime MIME

Package

dataloader

Source

1generic.lisp (file)

Appendix A Indexes

A.1 Concepts

A.2 Functions

A.3 Variables

A.4 Data types