The dataloader Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The dataloader Reference Manual

This is the dataloader Reference Manual, version 0.1, generated automatically by Declt version 3.0 "Montgomery Scott" on Tue Dec 22 13:16:24 2020 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

Dataloader - A universal loader library for various data formats for images/audio

This is what you might have needed for a long time. It loads a file in an arbitrary data format into a numcl-compatible array.

Usage

load (file &rest args
         &key (mime (magicffi:pathname-mime-type file))
         ;; asarray
           type
           ;; csv args
           csv-reader row-fn map-fn data-map-fn sample skip-first-p 
           (separator                    cl-csv:*separator*)
           (quote                        cl-csv:*quote*)
           (escape                       cl-csv:*quote-escape*)
           unquoted-empty-string-is-nil
           quoted-empty-string-is-nil
           trim-outer-whitespace
           newline
           escape-mode
         &allow-other-keys)

Load an arbitrary file using the mime type information obtained by libmagic.

Supported files are currently png, jpg, tiff, bmp, csv, tsv. We plan to include support for wav, gif, npy, npz.

This library merely provides a bridge to various libraries. It uses cl-png, retrospectiff, cl-csv and so on.

save (array file &key (mime (pathname-type file)))

This function saves the array into a file, using the file name extension or the additional mime argument.

Installation

You currently need a patched version of magicffi in https://github.com/guicho271828/magicffi . It also depends on NUMCL.

If you have roswell, run follows:

ros install numcl/constantfold numcl/specialized-function numcl/gtype numcl/numcl guicho271828/magicffi guicho271828/dataloader

Dependencies

This library is at least tested on implementation listed below:

Also, it depends on the following libraries:

Author

Copyright

Copyright (c) 2019 Masataro Asai (guicho2.71828@gmail.com)

License

Licensed under the LLGPL License.


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 dataloader

Author

Masataro Asai

Contact

guicho2.71828@gmail.com

License

LLGPL

Description

A universal loader library for various data formats for images/audio

Version

0.1

Dependencies
Source

dataloader.asd (file)

Component

package.lisp (file)


Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 dataloader.asd

Location

/home/quickref/quicklisp/dists/quicklisp/software/dataloader-20201220-git/dataloader.asd

Systems

dataloader (system)


Previous: , Up: Lisp files   [Contents][Index]

3.1.2 dataloader/package.lisp

Parent

dataloader (system)

Location

package.lisp

Packages

dataloader

Exported Definitions
Internal Definitions

external-format (function)


Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Previous: , Up: Packages   [Contents][Index]

4.1 dataloader

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

external-format (function)


Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Functions

Function: load FILE &rest ARGS &key MIME TYPE CSV-READER ROW-FN MAP-FN DATA-MAP-FN SAMPLE SKIP-FIRST-P SEPARATOR QUOTE ESCAPE UNQUOTED-EMPTY-STRING-IS-NIL QUOTED-EMPTY-STRING-IS-NIL TRIM-OUTER-WHITESPACE NEWLINE ESCAPE-MODE CHUNK-DATA-READER &allow-other-keys
Package

dataloader

Source

package.lisp (file)

Function: save ARRAY FILE &key MIME
Package

dataloader

Source

package.lisp (file)


Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Functions

Function: external-format FILE
Package

dataloader

Source

package.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   D   F   L  
Index Entry  Section

D
dataloader.asd: The dataloader․asd file
dataloader/package.lisp: The dataloader/package․lisp file

F
File, Lisp, dataloader.asd: The dataloader․asd file
File, Lisp, dataloader/package.lisp: The dataloader/package․lisp file

L
Lisp File, dataloader.asd: The dataloader․asd file
Lisp File, dataloader/package.lisp: The dataloader/package․lisp file

Jump to:   D   F   L  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   E   F   L   S  
Index Entry  Section

E
external-format: Internal functions

F
Function, external-format: Internal functions
Function, load: Exported functions
Function, save: Exported functions

L
load: Exported functions

S
save: Exported functions

Jump to:   E   F   L   S  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables


Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   D   P   S  
Index Entry  Section

D
dataloader: The dataloader system
dataloader: The dataloader package

P
Package, dataloader: The dataloader package

S
System, dataloader: The dataloader system

Jump to:   D   P   S