The dfio Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The dfio Reference Manual

This is the dfio Reference Manual, version 1.3, generated automatically by Declt version 3.0 "Montgomery Scott" on Mon Apr 19 15:54:27 2021 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

Contributors Forks Stargazers Issues MS-PL License LinkedIn

About the Project

A data frame isn't much use if you have to type all the data in by hand. This system provides I/O functions to read data sets from delimited (CSV, TSV, etc.) files. File may be either on local disk or accessible from network locations.

Built With

Getting Started

To get a local copy up and running follow these steps:

Prerequisites

An ANSI Common Lisp implementation. Developed and tested with SBCL and CCL.

Installation

  1. Clone the repository
    cd ~/quicklisp/local-projects &&
    git clone https://github.com/Lisp-Stat/dfio.git
    
  2. Reset the ASDF source-registry to find the new system (from the REPL)
    (asdf:clear-source-registry)
    
  3. Load the system
    (ql:quickload :dfio)
    

Usage

Create a data frame from a file named "computers.csv" on the local disk:

(defparameter *df* (dfio:csv-to-data-frame
		      (uiop:read-file-string "computers.csv")))

For more examples, please refer to the Documentation.

Roadmap

See the open issues for a list of proposed features (and known issues).

Resources

This system is part of the Lisp-Stat project; that should be your first stop for information. Also see the community page for more information.

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated. Please see CONTRIBUTING for details on the code of conduct, and the process for submitting pull requests.

License

Distributed under the MS-PL License. See LICENSE for more information.

Contact

Project Link: https://github.com/lisp-stat/dfio


Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 dfio

Maintainer

Steve Nunez <steve@symbolics.tech>

Author

Tamas Papp <tkpapp@gmail.com>

License

MS-PL

Description

Common Lisp library for reading data from text files (eg CSV).

Version

1.3

Dependencies
Source

dfio.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

3.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.1 dfio.asd

Location

dfio.asd

Systems

dfio (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.2 dfio/decimal.lisp

Parent

dfio (system)

Location

decimal.lisp

Packages

dfio.decimal

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.3 dfio/string-table.lisp

Dependency

decimal.lisp (file)

Parent

dfio (system)

Location

string-table.lisp

Packages

dfio.string-table

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

3.1.4 dfio/data-column.lisp

Dependency

string-table.lisp (file)

Parent

dfio (system)

Location

data-column.lisp

Packages

dfio.data-column

Exported Definitions
Internal Definitions

non-negative-integer (type)


Previous: , Up: Lisp files   [Contents][Index]

3.1.5 dfio/dfio.lisp

Dependency

data-column.lisp (file)

Parent

dfio (system)

Location

dfio.lisp

Packages

dfio

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

4 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

4.1 dfio.decimal

Source

decimal.lisp (file)

Use List
Used By List

dfio.data-column

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

4.2 dfio.string-table

Source

string-table.lisp (file)

Use List
Used By List

dfio.data-column

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

4.3 dfio.data-column

Source

data-column.lisp (file)

Use List
Used By List

dfio

Exported Definitions
Internal Definitions

non-negative-integer (type)


Previous: , Up: Packages   [Contents][Index]

4.4 dfio

Source

dfio.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

5.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.1 Functions

Function: csv-to-data-frame STREAM-OR-STRING &key SKIP-FIRST-ROW? COLUMN-KEYS-OR-FUNCTION

Read a CSV file (or stream, or string) into a DATA-FRAME, which is returned.

When SKIP-FIRST-ROW?, the first row is read separately and COLUMN-KEYS-OR-FUNCTION is used to form column keys.

When COLUMN-KEYS-OR-FUNCTION is a sequence, it is used for column keys, regardless of the value of SKIP-FIRST-ROW?.

Package

dfio

Source

dfio.lisp (file)

Function: data-column &key MAP-ALIST DEFAULT-FLOAT-FORMAT
Package

dfio.data-column

Source

data-column.lisp (file)

Function: data-column-add DATA-COLUMN STRING
Package

dfio.data-column

Source

data-column.lisp (file)

Function: data-column-counts DATA-COLUMN

Return the counts.

Package

dfio.data-column

Source

data-column.lisp (file)

Function: data-column-vector DATA-COLUMN

Return the collected elements as a vector.

Package

dfio.data-column

Source

data-column.lisp (file)

Function: data-frame-to-csv DF &key STREAM ADD-FIRST-ROW (SEPARATOR *SEPARATOR*) (QUOTE *QUOTE*) (ESCAPE *QUOTE-ESCAPE*) (NEWLINE *WRITE-NEWLINE*) (ALWAYS-QUOTE *ALWAYS-QUOTE*)

Write a data-frame to a stream.

Keywords:
stream: stream to write to. Default: nil.
nil - writes the rows to a string and returns it
an open stream
a pathname (overwrites if the file exists)
quote: quoting character. Defaults to *quote*
escape: escaping character. Defaults to *quote-escape*
newline: newline character. Defaults to *write-newline*
always-quote: Defaults to *always-quote*
add-first-row: Add column names as the first

Notes:
The :newline keyword requires a sequence, so use :newline ’(#newline) or use cl-interpol

Package

dfio

Source

dfio.lisp (file)

Function: parse-rational STRING &key START END EXPONENT-CHARS

Parse a decimal rational in (subseq string start end) of the form [sign][whole][.[fraction]][exponent] where

sign ::= + | - | empty
whole ::= digit*
fraction ::= digit*
exponent ::= exponent-char[sign]digit+

with the restriction that WHOLE and FRACTION cannot be empty at the same time. EXPONENT-CHAR is a string and contains the valid exponent chars.

Whitespace is NOT trimmed, and leads to an error. In case of a parsing failure, PARSE-RATIONAL-ERROR is used.

Return (values NUMBER DECIMAL-DOT? EXPONENT-CHAR). NUMBER is a RATIONAL, DECIMAL-DOT? is T when a decimal dot is present, otherwise NIL, EXPONENT-CHAR contains the exponent character, NIL if not present.

Numbers of the form .112 and 112. are valid syntax, representing 0.112 and 112.0, respectively.

Examples:

(parse-rational "7") => (values 7 NIL NIL)
(parse-rational "7.") => (values 7 T NIL)
(parse-rational "0.7") => (values 7/10 T NIL)
(parse-rational ".7") => (values 7/10 T NIL)
(parse-rational "7.e2") => (values 700 T #e)
(parse-rational ".7d1") => (values 7 T #d)

Package

dfio.decimal

Source

decimal.lisp (file)

Function: parse-real STRING &key START END S-FLOAT F-FLOAT D-FLOAT L-FLOAT E-FLOAT

Wrapper for PARSE-RATIONAL, converting non-integers to floats. The float type is determined by the -float arguments for each exponent character. Integers are not converted to floats. Return a single value, type of (or integer float).

See PARSE-RATIONAL for accepted formats, errors, etc.

Package

dfio.decimal

Source

decimal.lisp (file)

Function: string-table &key (TABLE TABLE)
Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-add STRING-TABLE STRING &optional VALUE

Add STRING mapped to VALUE to STRING-TABLE, raising STRING-TABLE-DUPLICATE if STRING is already in the table. Return VALUE.

Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-count STRING-TABLE

Number of distinct strings in the table.

Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-intern STRING-TABLE STRING &optional NEW-VALUE

If STRING is already in STRING-TABLE, return its value, otherwise add it and return NEW-VALUE. When used with the default argument for NEW-VALUE, EQUAL strings are always mapped to values that are EQ.

Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-lookup STRING-TABLE STRING

Return the value corresponding to STRING in STRING-TABLE, or raise the STRING-TABLE-NOT-FOUND error.

Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-strings STRING-TABLE

List of strings in STRING-TABLE.

Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-to-keyword STRING

Map string to a keyword.

This is the default for constructing column keys for CSV files.

The current implementation replaces #. and #space with a #-, and upcases all other characters.

Package

dfio

Source

dfio.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.2 Conditions

Condition: parse-rational-error ()

Error used by parse-rational and parse-real.

Package

dfio.decimal

Source

decimal.lisp (file)

Direct superclasses

error (condition)

Direct slots
Slot: string
Initform

(quote :string)

Slot: message
Initform

(quote :message)

Condition: string-table-duplicate ()

String is already in the table.

Package

dfio.string-table

Source

string-table.lisp (file)

Direct superclasses

error (condition)

Condition: string-table-not-found ()

String not found in table.

Package

dfio.string-table

Source

string-table.lisp (file)

Direct superclasses

error (condition)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

5.1.3 Structures

Structure: string-table ()

A table of distinct strings, optionally mapping each one to a value.

Package

dfio.string-table

Source

string-table.lisp (file)

Direct superclasses

structure-object (structure)

Direct methods

print-object (method)

Direct slots
Slot: table
Type

hash-table

Initform

(make-hash-table :test (function equalp))

Readers

string-table-table (function)

Writers

(setf string-table-table) (function)


Previous: , Up: Exported definitions   [Contents][Index]

5.1.4 Classes

Class: data-column ()
Package

dfio.data-column

Source

data-column.lisp (file)

Direct superclasses

standard-object (class)

Direct slots
Slot: reverse-elements
Type

list

Slot: default-float-format
Type

symbol

Initargs

:default-float-format

Slot: float-count
Type

dfio.data-column::non-negative-integer

Initform

0

Slot: integer-count
Type

dfio.data-column::non-negative-integer

Initform

0

Slot: integer-min
Type

integer

Initform

0

Slot: integer-max
Type

integer

Initform

0

Slot: map-count
Type

dfio.data-column::non-negative-integer

Initform

0

Slot: map-table
Type

dfio.string-table:string-table

Initargs

:map-table

Slot: string-count
Type

dfio.data-column::non-negative-integer

Initform

0

Slot: string-table
Type

dfio.string-table:string-table

Initform

(dfio.string-table:string-table)


Previous: , Up: Definitions   [Contents][Index]

5.2 Internal definitions


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.1 Special variables

Special Variable: +exponent-chars+

Default exponent characters.

Package

dfio.decimal

Source

decimal.lisp (file)


Next: , Previous: , Up: Internal definitions   [Contents][Index]

5.2.2 Functions

Function: 2d-array-to-list ARRAY

Helper for CSV writing.

Package

dfio

Source

dfio.lisp (file)

Function: copy-string-table INSTANCE
Package

dfio.string-table

Source

string-table.lisp (file)

Function: csv-to-data-columns STREAM-OR-STRING SKIP-FIRST-ROW?

Read a CSV file (or stream, or string), accumulate the values in DATA-COLUMNs, return a list of these. Rows are checked to have the same number of elements.

When SKIP-FIRST-ROW?, the first row is read separately and returned as the second value (list of strings), otherwise it is considered data like all other rows.

Package

dfio

Source

dfio.lisp (file)

Function: gobble-positive-integer STRING START END

If (SUBSEQ STRING START END) starts with a nonnegative integer (ie a sequence of digits 0-9), return the integer and position at which it ends as two values.

Otherwise, return NIL and 0.

START < END has to hold, END cannot be NIL. Consequences are undefined when START >= END.

Package

dfio.decimal

Source

decimal.lisp (file)

Function: gobble-sign STRING START

Return (values SIGNUM INDEX), where SIGNUM is -1 or 1 depending on whether (CHAR STRING START) was a sign, and INDEX is the index of the subsequent character (START or START+1).

Package

dfio.decimal

Source

decimal.lisp (file)

Function: string-table-get STRING-TABLE STRING

Synonym for GETHASH, used internally.

Package

dfio.string-table

Source

string-table.lisp (file)

Writer

(setf string-table-get) (function)

Function: (setf string-table-get) VALUE STRING-TABLE STRING

Synonym for (SETF GETHASH), used internally, checks that STRING is a string.

Package

dfio.string-table

Source

string-table.lisp (file)

Reader

string-table-get (function)

Function: string-table-p OBJECT
Package

dfio.string-table

Source

string-table.lisp (file)

Function: string-table-table INSTANCE
Function: (setf string-table-table) VALUE INSTANCE
Package

dfio.string-table

Source

string-table.lisp (file)


Previous: , Up: Internal definitions   [Contents][Index]

5.2.3 Types

Type: non-negative-integer ()
Package

dfio.data-column

Source

data-column.lisp (file)


Previous: , Up: Top   [Contents][Index]

Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts

Jump to:   D   F   L  
Index Entry  Section

D
dfio.asd: The dfio․asd file
dfio/data-column.lisp: The dfio/data-column․lisp file
dfio/decimal.lisp: The dfio/decimal․lisp file
dfio/dfio.lisp: The dfio/dfio․lisp file
dfio/string-table.lisp: The dfio/string-table․lisp file

F
File, Lisp, dfio.asd: The dfio․asd file
File, Lisp, dfio/data-column.lisp: The dfio/data-column․lisp file
File, Lisp, dfio/decimal.lisp: The dfio/decimal․lisp file
File, Lisp, dfio/dfio.lisp: The dfio/dfio․lisp file
File, Lisp, dfio/string-table.lisp: The dfio/string-table․lisp file

L
Lisp File, dfio.asd: The dfio․asd file
Lisp File, dfio/data-column.lisp: The dfio/data-column․lisp file
Lisp File, dfio/decimal.lisp: The dfio/decimal․lisp file
Lisp File, dfio/dfio.lisp: The dfio/dfio․lisp file
Lisp File, dfio/string-table.lisp: The dfio/string-table․lisp file

Jump to:   D   F   L  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (   2  
C   D   F   G   P   S  
Index Entry  Section

(
(setf string-table-get): Internal functions
(setf string-table-table): Internal functions

2
2d-array-to-list: Internal functions

C
copy-string-table: Internal functions
csv-to-data-columns: Internal functions
csv-to-data-frame: Exported functions

D
data-column: Exported functions
data-column-add: Exported functions
data-column-counts: Exported functions
data-column-vector: Exported functions
data-frame-to-csv: Exported functions

F
Function, (setf string-table-get): Internal functions
Function, (setf string-table-table): Internal functions
Function, 2d-array-to-list: Internal functions
Function, copy-string-table: Internal functions
Function, csv-to-data-columns: Internal functions
Function, csv-to-data-frame: Exported functions
Function, data-column: Exported functions
Function, data-column-add: Exported functions
Function, data-column-counts: Exported functions
Function, data-column-vector: Exported functions
Function, data-frame-to-csv: Exported functions
Function, gobble-positive-integer: Internal functions
Function, gobble-sign: Internal functions
Function, parse-rational: Exported functions
Function, parse-real: Exported functions
Function, string-table: Exported functions
Function, string-table-add: Exported functions
Function, string-table-count: Exported functions
Function, string-table-get: Internal functions
Function, string-table-intern: Exported functions
Function, string-table-lookup: Exported functions
Function, string-table-p: Internal functions
Function, string-table-strings: Exported functions
Function, string-table-table: Internal functions
Function, string-to-keyword: Exported functions

G
gobble-positive-integer: Internal functions
gobble-sign: Internal functions

P
parse-rational: Exported functions
parse-real: Exported functions

S
string-table: Exported functions
string-table-add: Exported functions
string-table-count: Exported functions
string-table-get: Internal functions
string-table-intern: Exported functions
string-table-lookup: Exported functions
string-table-p: Internal functions
string-table-strings: Exported functions
string-table-table: Internal functions
string-to-keyword: Exported functions

Jump to:   (   2  
C   D   F   G   P   S  

Next: , Previous: , Up: Indexes   [Contents][Index]

A.3 Variables

Jump to:   +  
D   F   I   M   R   S   T  
Index Entry  Section

+
+exponent-chars+: Internal special variables

D
default-float-format: Exported classes

F
float-count: Exported classes

I
integer-count: Exported classes
integer-max: Exported classes
integer-min: Exported classes

M
map-count: Exported classes
map-table: Exported classes
message: Exported conditions

R
reverse-elements: Exported classes

S
Slot, default-float-format: Exported classes
Slot, float-count: Exported classes
Slot, integer-count: Exported classes
Slot, integer-max: Exported classes
Slot, integer-min: Exported classes
Slot, map-count: Exported classes
Slot, map-table: Exported classes
Slot, message: Exported conditions
Slot, reverse-elements: Exported classes
Slot, string: Exported conditions
Slot, string-count: Exported classes
Slot, string-table: Exported classes
Slot, table: Exported structures
Special Variable, +exponent-chars+: Internal special variables
string: Exported conditions
string-count: Exported classes
string-table: Exported classes

T
table: Exported structures

Jump to:   +  
D   F   I   M   R   S   T  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   C   D   N   P   S   T  
Index Entry  Section

C
Class, data-column: Exported classes
Condition, parse-rational-error: Exported conditions
Condition, string-table-duplicate: Exported conditions
Condition, string-table-not-found: Exported conditions

D
data-column: Exported classes
dfio: The dfio system
dfio: The dfio package
dfio.data-column: The dfio․data-column package
dfio.decimal: The dfio․decimal package
dfio.string-table: The dfio․string-table package

N
non-negative-integer: Internal types

P
Package, dfio: The dfio package
Package, dfio.data-column: The dfio․data-column package
Package, dfio.decimal: The dfio․decimal package
Package, dfio.string-table: The dfio․string-table package
parse-rational-error: Exported conditions

S
string-table: Exported structures
string-table-duplicate: Exported conditions
string-table-not-found: Exported conditions
Structure, string-table: Exported structures
System, dfio: The dfio system

T
Type, non-negative-integer: Internal types

Jump to:   C   D   N   P   S   T