The pgloader Reference Manual

Table of Contents

Next: , Previous: , Up: (dir)   [Contents][Index]

The pgloader Reference Manual

This is the pgloader Reference Manual, generated automatically by Declt version 2.4 "Will Decker" on Wed Jun 20 12:23:39 2018 GMT+0.


Next: , Previous: , Up: Top   [Contents][Index]

1 Introduction

PGLoader

Build Status

Join the chat at https://gitter.im/dimitri/pgloader

pgloader is a data loading tool for PostgreSQL, using the COPY command.

Its main advantage over just using COPY or \copy, and over using a Foreign Data Wrapper, is its transaction behaviour, where pgloader will keep a separate file of rejected data, but continue trying to copy good data in your database.

The default PostgreSQL behaviour is transactional, which means that any erroneous line in the input data (file or remote database) will stop the entire bulk load for the table.

pgloader also implements data reformatting, a typical example of that being the transformation of MySQL datestamps 0000-00-00 and 0000-00-00 00:00:00 to PostgreSQL NULL value (because our calendar never had a year zero).

Versioning

pgloader version 1.x is quite old and was developed in TCL. When faced with maintaining that code, the new emerging development team (hi!) picked python instead because that made sense at the time. So pgloader version 2.x was written in python.

The current version of pgloader is the 3.x series, which is written in Common Lisp for better development flexibility, runtime performance, and support of real threading.

The versioning is now following the Emacs model, where any X.0 release number means you're using a development version (alpha, beta, or release candidate). The next stable versions are going to be 3.1 then 3.2 etc.

When using a development snapshot rather than a released version the version number includes the git hash (in its abbreviated form):

LICENCE

pgloader is available under The PostgreSQL Licence.

INSTALL

You can install pgloader directly from apt.postgresql.org and from official debian repositories, see packages.debian.org/pgloader.

$ apt-get install pgloader

You can also use a docker image for pgloader at https://hub.docker.com/r/dimitri/pgloader/:

$ docker pull dimitri/pgloader
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --version
$ docker run --rm --name pgloader dimitri/pgloader:latest pgloader --help

Build from sources

pgloader is now a Common Lisp program, tested using the SBCL (>= 1.2.5) and Clozure CL implementations with Quicklisp.

When building from sources, you should always build from the current git HEAD as it's basically the only source that is managed in a way to ensure it builds aginst current set of dependencies versions.

Building from sources on debian

$ apt-get install sbcl unzip libsqlite3-dev make curl gawk freetds-dev libzip-dev
$ cd /path/to/pgloader
$ make pgloader
$ ./build/bin/pgloader --help

Building from sources on MacOSX

When using brew, it should be a simple brew install --HEAD pgloader.

When using macports, then we have a situation to deal with with shared objects pgloader depends on, as reported in issue #161 at https://github.com/dimitri/pgloader/issues/161#issuecomment-201162647:

I was able to get a clean build without having to disable compression after symlinking /usr/local/lib to /opt/local/lib. Note that I did not have anything installed to /usr/local/lib so I didn't lose anything here.

Building from sources on Windows

Building pgloader on Windows is supported, thanks to Common Lisp implementations being available on that platform, and to the Common Lisp Standard for making it easy to write actually portable code.

It is recommended to have a look at the issues labelled with Windows support if you run into trouble when building pgloader:

https://github.com/dimitri/pgloader/issues?utf8=✓&q=label%3A%22Windows%20support%22%20

More options when building from source

The Makefile target pgloader knows how to produce a Self Contained Binary file for pgloader, found at ./build/bin/pgloader:

$ make pgloader

By default, the Makefile uses SBCL to compile your binary image, though it's possible to build using CCL.

$ make CL=ccl pgloader

If using SBCL and it supports core compression, the make process will use it to generate a smaller binary. To force disabling core compression, you may use:

$ make COMPRESS_CORE=no pgloader

The --compress-core is unique to SBCL, so not used when CC is different from the sbcl value.

You can also tweak the default amount of memory that the pgloader image will allow itself using when running through your data (don't ask for more than your current RAM tho):

$ make DYNSIZE=8192 pgloader

The make pgloader command when successful outputs a ./build/bin/pgloader file for you to use.

Usage

You can either give a command file to pgloader or run it all from the command line, see the pgloader quick start on http://pgloader.io for more details.

$ ./build/bin/pgloader --help
$ ./build/bin/pgloader <file.load>

For example, for a full migration from SQLite:

$ createdb newdb
$ pgloader ./test/sqlite/sqlite.db postgresql:///newdb

Or for a full migration from MySQL, including schema definition (tables, indexes, foreign keys, comments) and parallel loading of the corrected data:

$ createdb pagila
$ pgloader mysql://user@localhost/sakila postgresql:///pagila

See the documentation file pgloader.1.md for details. You can compile that file into a manual page or an HTML page thanks to the ronn application:

$ apt-get install ruby-ronn
$ make doc

Next: , Previous: , Up: Top   [Contents][Index]

2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 pgloader

Author

Dimitri Fontaine <dimitri@2ndQuadrant.fr>

License

The PostgreSQL Licence

Description

Load data into PostgreSQL

Dependencies
Source

pgloader.asd (file)

Components

Next: , Previous: , Up: Top   [Contents][Index]

3 Modules

Modules are listed depth-first from the system components tree.


Next: , Previous: , Up: Modules   [Contents][Index]

3.1 pgloader/src

Parent

pgloader (system)

Location

src/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.2 pgloader/src/monkey

Parent

src (module)

Location

src/monkey/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.3 pgloader/src/utils

Dependencies
Parent

src (module)

Location

src/utils/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.4 pgloader/src/pgsql

Dependencies
Parent

src (module)

Location

src/pgsql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.5 pgloader/src/parsers

Dependencies
Parent

src (module)

Location

src/parsers/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.6 pgloader/src/sources

Dependencies
Parent

src (module)

Location

src/sources/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.7 pgloader/src/sources/common

Parent

sources (module)

Location

src/sources/common/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.8 pgloader/src/sources/csv

Dependency

common (module)

Parent

sources (module)

Location

src/sources/csv/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.9 pgloader/src/sources/db3

Dependencies
Parent

sources (module)

Location

src/sources/db3/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.10 pgloader/src/sources/ixf

Dependency

common (module)

Parent

sources (module)

Location

src/sources/ixf/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.11 pgloader/src/sources/sqlite

Dependency

common (module)

Parent

sources (module)

Location

src/sources/sqlite/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.12 pgloader/src/sources/mssql

Dependency

common (module)

Parent

sources (module)

Location

src/sources/mssql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.13 pgloader/src/sources/mysql

Dependency

common (module)

Parent

sources (module)

Location

src/sources/mysql/

Components

Next: , Previous: , Up: Modules   [Contents][Index]

3.14 pgloader/src/regress

Dependencies
Parent

src (module)

Location

src/regress/

Component

regress.lisp (file)


Next: , Previous: , Up: Modules   [Contents][Index]

3.15 pgloader/docs

Dependency

src (module)

Parent

pgloader (system)

Location

docs/

Component

src (module)


Previous: , Up: Modules   [Contents][Index]

3.16 pgloader/docs/src

Parent

docs (module)

Location

docs/src/

Component

docs.lisp (file)


Next: , Previous: , Up: Top   [Contents][Index]

4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.1 pgloader.asd

Location

pgloader.asd

Systems

pgloader (system)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.2 pgloader/src/params.lisp

Parent

src (module)

Location

src/params.lisp

Packages

pgloader.params

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.3 pgloader/src/package.lisp

Dependency

params.lisp (file)

Parent

src (module)

Location

src/package.lisp

Packages

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.4 pgloader/src/monkey/bind.lisp

Parent

monkey (module)

Location

src/monkey/bind.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.5 pgloader/src/monkey/mssql.lisp

Parent

monkey (module)

Location

src/monkey/mssql.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.6 pgloader/src/utils/charsets.lisp

Parent

utils (module)

Location

src/utils/charsets.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.7 pgloader/src/utils/batch.lisp

Parent

utils (module)

Location

src/utils/batch.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.8 pgloader/src/utils/threads.lisp

Parent

utils (module)

Location

src/utils/threads.lisp

Exported Definitions

make-kernel (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.9 pgloader/src/utils/logs.lisp

Parent

utils (module)

Location

src/utils/logs.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.10 pgloader/src/utils/utils.lisp

Parent

utils (module)

Location

src/utils/utils.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.11 pgloader/src/utils/state.lisp

Parent

utils (module)

Location

src/utils/state.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.12 pgloader/src/utils/transforms.lisp

Parent

utils (module)

Location

src/utils/transforms.lisp

Packages

pgloader.user-symbols

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.13 pgloader/src/utils/read-sql-files.lisp

Parent

utils (module)

Location

src/utils/read-sql-files.lisp

Exported Definitions

read-queries (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.14 pgloader/src/utils/queries.lisp

Parent

utils (module)

Location

src/utils/queries.lisp

Exported Definitions

sql (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.15 pgloader/src/utils/quoting.lisp

Parent

utils (module)

Location

src/utils/quoting.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.16 pgloader/src/utils/catalog.lisp

Dependency

quoting.lisp (file)

Parent

utils (module)

Location

src/utils/catalog.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.17 pgloader/src/utils/alter-table.lisp

Dependency

catalog.lisp (file)

Parent

utils (module)

Location

src/utils/alter-table.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.18 pgloader/src/utils/reject.lisp

Dependency

state.lisp (file)

Parent

utils (module)

Location

src/utils/reject.lisp

Internal Definitions

%process-bad-row (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.19 pgloader/src/utils/report.lisp

Dependencies
Parent

utils (module)

Location

src/utils/report.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.20 pgloader/src/utils/monitor.lisp

Dependencies
Parent

utils (module)

Location

src/utils/monitor.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.21 pgloader/src/utils/archive.lisp

Dependency

logs.lisp (file)

Parent

utils (module)

Location

src/utils/archive.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.22 pgloader/src/utils/connection.lisp

Dependency

archive.lisp (file)

Parent

utils (module)

Location

src/utils/connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.23 pgloader/src/pgsql/copy-format.lisp

Parent

pgsql (module)

Location

src/pgsql/copy-format.lisp

Exported Definitions

format-vector-row (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.24 pgloader/src/pgsql/connection.lisp

Dependency

copy-format.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/connection.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.25 pgloader/src/pgsql/pgsql-ddl.lisp

Dependency

connection.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-ddl.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.26 pgloader/src/pgsql/pgsql-schema.lisp

Dependency

pgsql-ddl.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.27 pgloader/src/pgsql/merge-catalogs.lisp

Dependencies
Parent

pgsql (module)

Location

src/pgsql/merge-catalogs.lisp

Exported Definitions

merge-catalogs (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.28 pgloader/src/pgsql/pgsql-trigger.lisp

Dependency

merge-catalogs.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-trigger.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.29 pgloader/src/pgsql/pgsql-index-filter.lisp

Dependency

pgsql-trigger.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/pgsql-index-filter.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.30 pgloader/src/pgsql/pgsql-create-schema.lisp

Dependencies
Parent

pgsql (module)

Location

src/pgsql/pgsql-create-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.31 pgloader/src/pgsql/retry-batch.lisp

Dependency

pgsql-create-schema.lisp (file)

Parent

pgsql (module)

Location

src/pgsql/retry-batch.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.32 pgloader/src/pgsql/copy-from-queue.lisp

Dependencies
Parent

pgsql (module)

Location

src/pgsql/copy-from-queue.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.33 pgloader/src/parsers/parse-ini.lisp

Parent

parsers (module)

Location

src/parsers/parse-ini.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.34 pgloader/src/parsers/command-utils.lisp

Dependency

parse-ini.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-utils.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.35 pgloader/src/parsers/command-keywords.lisp

Dependency

command-utils.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-keywords.lisp

Internal Definitions

def-keyword-rule (macro)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.36 pgloader/src/parsers/command-regexp.lisp

Dependency

command-keywords.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-regexp.lisp

Internal Definitions

process-quoted-regex (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.37 pgloader/src/parsers/command-db-uri.lisp

Dependency

command-regexp.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-db-uri.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.38 pgloader/src/parsers/command-source.lisp

Dependency

command-db-uri.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-source.lisp

Internal Definitions

filename-character-p (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.39 pgloader/src/parsers/command-options.lisp

Dependency

command-source.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-options.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.40 pgloader/src/parsers/command-sql-block.lisp

Dependency

command-options.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sql-block.lisp

Internal Definitions

sql-code-block (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.41 pgloader/src/parsers/command-sexp.lisp

Dependency

command-sql-block.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sexp.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.42 pgloader/src/parsers/command-csv.lisp

Dependency

command-sexp.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-csv.lisp

Exported Definitions

lisp-code-for-loading-from-csv (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.43 pgloader/src/parsers/command-ixf.lisp

Dependency

command-csv.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-ixf.lisp

Exported Definitions

lisp-code-for-loading-from-ixf (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.44 pgloader/src/parsers/command-fixed.lisp

Dependency

command-ixf.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-fixed.lisp

Exported Definitions

lisp-code-for-loading-from-fixed (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.45 pgloader/src/parsers/command-copy.lisp

Dependency

command-fixed.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-copy.lisp

Exported Definitions

lisp-code-for-loading-from-copy (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.46 pgloader/src/parsers/command-dbf.lisp

Dependency

command-copy.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-dbf.lisp

Exported Definitions

lisp-code-for-loading-from-dbf (function)

Internal Definitions

lisp-code-for-dbf-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.47 pgloader/src/parsers/command-cast-rules.lisp

Dependency

command-dbf.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-cast-rules.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.48 pgloader/src/parsers/command-materialize-views.lisp

Dependency

command-cast-rules.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-materialize-views.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.49 pgloader/src/parsers/command-alter-table.lisp

Dependency

command-materialize-views.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-alter-table.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.50 pgloader/src/parsers/command-mysql.lisp

Dependency

command-alter-table.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-mysql.lisp

Exported Definitions

lisp-code-for-loading-from-mysql (function)

Internal Definitions

lisp-code-for-mysql-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.51 pgloader/src/parsers/command-including-like.lisp

Dependency

command-mysql.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-including-like.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.52 pgloader/src/parsers/command-mssql.lisp

Dependency

command-including-like.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-mssql.lisp

Exported Definitions

lisp-code-for-loading-from-mssql (function)

Internal Definitions

lisp-code-for-mssql-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.53 pgloader/src/parsers/command-sqlite.lisp

Dependency

command-mssql.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-sqlite.lisp

Exported Definitions

lisp-code-for-loading-from-sqlite (function)

Internal Definitions

lisp-code-for-sqlite-dry-run (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.54 pgloader/src/parsers/command-archive.lisp

Dependency

command-sqlite.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-archive.lisp


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.55 pgloader/src/parsers/command-parser.lisp

Dependency

command-archive.lisp (file)

Parent

parsers (module)

Location

src/parsers/command-parser.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.56 pgloader/src/parsers/date-format.lisp

Dependency

command-parser.lisp (file)

Parent

parsers (module)

Location

src/parsers/date-format.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.57 pgloader/src/sources/common/api.lisp

Parent

common (module)

Location

src/sources/common/api.lisp

Exported Definitions
Internal Definitions

proprocess-row (generic function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.58 pgloader/src/sources/common/methods.lisp

Dependency

api.lisp (file)

Parent

common (module)

Location

src/sources/common/methods.lisp

Exported Definitions
Internal Definitions

preprocess-row (method)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.59 pgloader/src/sources/common/md-methods.lisp

Dependency

api.lisp (file)

Parent

common (module)

Location

src/sources/common/md-methods.lisp

Exported Definitions
Internal Definitions

preprocess-row (method)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.60 pgloader/src/sources/common/db-methods.lisp

Dependency

api.lisp (file)

Parent

common (module)

Location

src/sources/common/db-methods.lisp

Exported Definitions
Internal Definitions

process-catalog (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.61 pgloader/src/sources/common/casting-rules.lisp

Parent

common (module)

Location

src/sources/common/casting-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.62 pgloader/src/sources/common/files-and-pathnames.lisp

Parent

common (module)

Location

src/sources/common/files-and-pathnames.lisp

Exported Definitions
Internal Definitions

filter-directory (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.63 pgloader/src/sources/common/project-fields.lisp

Parent

common (module)

Location

src/sources/common/project-fields.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.64 pgloader/src/sources/csv/csv-guess.lisp

Parent

csv (module)

Location

src/sources/csv/csv-guess.lisp

Exported Definitions

guess-csv-params (function)

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.65 pgloader/src/sources/csv/csv.lisp

Parent

csv (module)

Location

src/sources/csv/csv.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.66 pgloader/src/sources/fixed.lisp

Dependencies
Parent

sources (module)

Location

src/sources/fixed.lisp

Exported Definitions
Internal Definitions

parse-row (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.67 pgloader/src/sources/copy.lisp

Dependencies
Parent

sources (module)

Location

src/sources/copy.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.68 pgloader/src/sources/db3/db3-schema.lisp

Parent

db3 (module)

Location

src/sources/db3/db3-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.69 pgloader/src/sources/db3/db3.lisp

Dependency

db3-schema.lisp (file)

Parent

db3 (module)

Location

src/sources/db3/db3.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.70 pgloader/src/sources/ixf/ixf-schema.lisp

Parent

ixf (module)

Location

src/sources/ixf/ixf-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.71 pgloader/src/sources/ixf/ixf.lisp

Dependency

ixf-schema.lisp (file)

Parent

ixf (module)

Location

src/sources/ixf/ixf.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.72 pgloader/src/sources/sqlite/sqlite-cast-rules.lisp

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.73 pgloader/src/sources/sqlite/sqlite-schema.lisp

Dependency

sqlite-cast-rules.lisp (file)

Parent

sqlite (module)

Location

src/sources/sqlite/sqlite-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.74 pgloader/src/sources/sqlite/sqlite.lisp

Dependencies
Parent

sqlite (module)

Location

src/sources/sqlite/sqlite.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.75 pgloader/src/sources/mssql/mssql-cast-rules.lisp

Parent

mssql (module)

Location

src/sources/mssql/mssql-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.76 pgloader/src/sources/mssql/mssql-schema.lisp

Dependency

mssql-cast-rules.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.77 pgloader/src/sources/mssql/mssql.lisp

Dependencies
Parent

mssql (module)

Location

src/sources/mssql/mssql.lisp

Exported Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.78 pgloader/src/sources/mssql/mssql-index-filters.lisp

Dependency

mssql.lisp (file)

Parent

mssql (module)

Location

src/sources/mssql/mssql-index-filters.lisp

Exported Definitions

translate-index-filter (method)

Internal Definitions

parse-index-filter-clause (function)


Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.79 pgloader/src/sources/mysql/mysql-cast-rules.lisp

Parent

mysql (module)

Location

src/sources/mysql/mysql-cast-rules.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.80 pgloader/src/sources/mysql/mysql-schema.lisp

Dependency

mysql-cast-rules.lisp (file)

Parent

mysql (module)

Location

src/sources/mysql/mysql-schema.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.81 pgloader/src/sources/mysql/mysql.lisp

Dependencies
Parent

mysql (module)

Location

src/sources/mysql/mysql.lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.82 pgloader/src/regress/regress.lisp

Parent

regress (module)

Location

src/regress/regress.lisp

Internal Definitions

Next: , Previous: , Up: Lisp files   [Contents][Index]

4.1.83 pgloader/src/main.lisp

Dependencies
Parent

src (module)

Location

src/main.lisp

Exported Definitions
Internal Definitions

Previous: , Up: Lisp files   [Contents][Index]

4.1.84 pgloader/docs/src/docs.lisp

Parent

src (module)

Location

docs/src/docs.lisp

Packages

pgloader.docs

Exported Definitions

build-docs (function)

Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

5 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

5.1 pgloader.params

Source

params.lisp (file)

Use List

common-lisp

Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.2 pgloader

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.3 pgloader.parser

Source

package.lisp (file)

Use List
Used By List

pgloader

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.4 pgloader.syslog

Source

package.lisp (file)

Use List

Next: , Previous: , Up: Packages   [Contents][Index]

5.5 pgloader.mssql.index-filter

Source

package.lisp (file)

Use List
Internal Definitions

parse-index-filter-clause (function)


Next: , Previous: , Up: Packages   [Contents][Index]

5.6 pgloader.mssql

Source

package.lisp (file)

Use List
Used By List

pgloader.mssql.index-filter

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.7 pgloader.sqlite

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.8 pgloader.mysql

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.9 pgloader.db3

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.10 pgloader.ixf

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.11 pgloader.copy

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.12 pgloader.fixed

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

parse-row (function)


Next: , Previous: , Up: Packages   [Contents][Index]

5.13 pgloader.csv

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.14 pgloader.sql

Source

package.lisp (file)

Use List

common-lisp

Used By List

pgloader.parser

Exported Definitions

read-queries (function)

Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.15 pgloader.ini

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.16 pgloader.sources

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.17 pgloader.pgsql

Source

package.lisp (file)

Use List
Used By List

pgloader.sources

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.18 pgloader.connection

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.19 pgloader.parse-date

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.20 pgloader.archive

Source

package.lisp (file)

Use List
Used By List

pgloader.connection

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.21 pgloader.utils

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.22 pgloader.queries

Source

package.lisp (file)

Use List
Used By List

pgloader.utils

Exported Definitions

sql (function)

Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.23 pgloader.batch

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.24 pgloader.monitor

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.25 pgloader.state

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.26 pgloader.catalog

Source

package.lisp (file)

Use List
Used By List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.27 pgloader.quoting

Source

package.lisp (file)

Use List
Used By List
Exported Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.28 pgloader.logs

Source

package.lisp (file)

Use List
Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.29 pgloader.transforms

Source

package.lisp (file)

Use List

common-lisp

Exported Definitions
Internal Definitions

Next: , Previous: , Up: Packages   [Contents][Index]

5.30 pgloader.user-symbols

Source

transforms.lisp (file)


Previous: , Up: Packages   [Contents][Index]

5.31 pgloader.docs

Source

docs.lisp (file)

Use List
Exported Definitions

build-docs (function)

Internal Definitions

Next: , Previous: , Up: Top   [Contents][Index]

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Exported definitions


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.1 Special variables

Special Variable: *cast-rules*

Specific casting rules added in the command.

Package

pgloader.sources

Source

casting-rules.lisp (file)

Special Variable: *client-min-messages*
Package

pgloader.params

Source

params.lisp (file)

Special Variable: *copy-batch-rows*

How many rows to batch per COPY transaction.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *copy-batch-size*

Maximum memory size allowed for a single batch.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *decoding-as*

Special per-table encoding/decoding overloading rules for MySQL.

Package

pgloader.mysql

Source

mysql.lisp (file)

Special Variable: *default-cast-rules*

Default casting rules.

Package

pgloader.sources

Source

casting-rules.lisp (file)

Special Variable: *default-tmpdir*

Place where to fetch and expand archives on-disk.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *dry-run*

Set to non-nil to only run checks about the load setup.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *fd-path-root*

Where to load files from, when loading from an archive or expanding regexps.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *identifier-case*

Dealing with source databases casing rules.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *log-filename*

Main pgloader log file

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *log-messengers*

Currently active log-messengers

Package

pgloader.logs

Source

logs.lisp (file)

Special Variable: *log-min-messages*
Package

pgloader.params

Source

params.lisp (file)

Special Variable: *monitoring-kernel*

Internal lparallel kernel to manage the separate monitor thread.

Package

pgloader.monitor

Source

monitor.lisp (file)

Special Variable: *monitoring-queue*

Internal lparallel queue where to send and receive messages from.

Package

pgloader.monitor

Source

monitor.lisp (file)

Special Variable: *mssql-default-cast-rules*

Data Type Casting to migrate from MSSQL to PostgreSQL

Package

pgloader.mssql

Source

mssql-cast-rules.lisp (file)

Special Variable: *mysql-default-cast-rules*

Data Type Casting rules to migrate from MySQL to PostgreSQL

Package

pgloader.mysql

Source

mysql-cast-rules.lisp (file)

Special Variable: *mysql-settings*

An alist of GUC names and values.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *on-error-stop*

Set to non-nil to for pgloader to refrain from handling errors, quitting instead.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *pg-settings*

An alist of GUC names and values.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *pgsql-reserved-keywords*

We need to always quote PostgreSQL reserved keywords

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *prefetch-rows*

How many rows do read in advance in the reader queue.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *preserve-index-names*

Dealing with source databases index naming.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *report-stream*

Stream where to format the output stream.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *root-dir*

Top directory where to store all data logs and reject files.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *rows-per-range*

How many rows to read in each reader’s thread, per SQL query.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *self-upgrade-immutable-systems*

Used for –self-upgrade.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *sqlite-default-cast-rules*

Data Type Casting to migrate from SQLite to PostgreSQL

Package

pgloader.sqlite

Source

sqlite-cast-rules.lisp (file)

Special Variable: *summary-pathname*

Pathname where to output the summary.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: *version-string*

pgloader version strings, following Emacs versionning model.

Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-bad-source+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-regress+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-error-usage+
Package

pgloader.params

Source

params.lisp (file)

Special Variable: +os-code-success+
Package

pgloader.params

Source

params.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.2 Macros

Macro: push-to-end ITEM PLACE
Package

pgloader.catalog

Source

catalog.lisp (file)

Macro: timing &body FORMS

return both how much real time was spend in body and its result

Package

pgloader.monitor

Source

monitor.lisp (file)

Macro: with-connection (VAR CONNECTION &rest ARGS &key &allow-other-keys) &body FORMS

Connect to DB-CONNECTION and handle any condition when doing so, and when connected execute FORMS in a protected way so that we always disconnect at the end.

Package

pgloader.connection

Source

connection.lisp (file)

Macro: with-monitor (&key START-LOGGER) &body BODY

Start and stop the monitor around BODY code. The monitor is responsible for processing logs into a central logfile

Package

pgloader.monitor

Source

monitor.lisp (file)

Macro: with-pgsql-connection (PGCONN) &body FORMS

Run FROMS within a PostgreSQL connection to DBNAME. To get the connection spec from the DBNAME, use ‘get-connection-spec’.

Package

pgloader.pgsql

Source

connection.lisp (file)

Macro: with-pgsql-transaction (&key PGCONN DATABASE) &body FORMS

Run FORMS within a PostgreSQL transaction to DBNAME, reusing DATABASE if given.

Package

pgloader.pgsql

Source

connection.lisp (file)

Macro: with-schema (VAR TABLE-NAME) &body BODY

When table-name is a CONS, SET search_path TO its CAR and return its CDR, otherwise just return the TABLE-NAME. A PostgreSQL connection must be established when calling this function.

Package

pgloader.catalog

Source

catalog.lisp (file)

Macro: with-stats-collection (TABLE-NAME &key SECTION DBNAME USE-RESULT-AS-READ USE-RESULT-AS-ROWS) &body FORMS

Measure time spent in running BODY into STATE, accounting the seconds to given DBNAME and TABLE-NAME

Package

pgloader.monitor

Source

monitor.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.3 Functions

Function: apply-casting-rules TABLE-NAME COLUMN-NAME DTYPE CTYPE DEFAULT NULLABLE EXTRA &key RULES

Apply the given RULES to the MySQL SOURCE type definition

Package

pgloader.sources

Source

casting-rules.lisp (file)

Function: apply-identifier-case IDENTIFIER

Return given IDENTIFIER with CASE handled to be PostgreSQL compatible.

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: archivep ARCHIVE-FILE

Return non-nil when the ARCHIVE-FILE is something we know how to expand.

Package

pgloader.archive

Source

archive.lisp (file)

Function: batch-bytes INSTANCE
Function: (setf batch-bytes) VALUE INSTANCE
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-count INSTANCE
Function: (setf batch-count) VALUE INSTANCE
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-data INSTANCE
Function: (setf batch-data) VALUE INSTANCE
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-full-p BATCH
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-max-count INSTANCE
Function: (setf batch-max-count) VALUE INSTANCE
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-oversized-p BATCH

Return a generalized boolean that is true only when BATCH is considered over-sized when its size in BYTES is compared *copy-batch-size*.

Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-p OBJECT
Package

pgloader.batch

Source

batch.lisp (file)

Function: batch-start INSTANCE
Function: (setf batch-start) VALUE INSTANCE
Package

pgloader.batch

Source

batch.lisp (file)

Function: build-docs ()

Build the HTML files from the Markdown ones.

Package

pgloader.docs

Source

docs.lisp (file)

Function: build-identifier SEP &rest PARTS

Concatenante PARTS into a PostgreSQL identifier, with SEP in between
parts. That’s useful for creating an index name from a table’s oid and name.

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: camelcase-to-colname STRING

Transform input STRING into a suitable column name. lahmanID lahman_id
playerID player_id
birthYear birth_year

Package

pgloader.utils

Source

utils.lisp (file)

Function: catalog-name INSTANCE
Function: (setf catalog-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: catalog-schema-list INSTANCE
Function: (setf catalog-schema-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: catalog-types-without-btree INSTANCE
Function: (setf catalog-types-without-btree) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-comment INSTANCE
Function: (setf column-comment) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-default INSTANCE
Function: (setf column-default) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-extra INSTANCE
Function: (setf column-extra) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-name INSTANCE
Function: (setf column-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-nullable INSTANCE
Function: (setf column-nullable) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-transform INSTANCE
Function: (setf column-transform) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-type-mod INSTANCE
Function: (setf column-type-mod) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: column-type-name INSTANCE
Function: (setf column-type-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: comment-on-tables-and-columns CATALOG &key LABEL SECTION

Install comments on tables and columns from CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: convert-ini-into-commands FILENAME

Read the INI file at FILENAME and convert each section of it to a command in the new pgloader format.

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: convert-ini-into-files FILENAME TARGET-DIRECTORY &key WITH-DATA-INLINE INCLUDE-SQL-FILE

Reads the INI file at FILENAME and creates files names <section>.load for each section in the INI file, in TARGET-DIRECTORY.

When WITH-DATA-INLINE is true, read the CSV file listed as the section’s filename and insert its content in the command itself, as inline data.

When INCLUDE-SQL-FILE is :if-exists, try to find a sibling file to the data file, with the same name and with the "sql" type, and use its content in a BEFORE LOAD DO clause.

When INCLUDE-SQL-FILE is t, not finding the SQL file is an error.

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: create-indexes-again TARGET CATALOG &key MAX-PARALLEL-CREATE-INDEX SECTION DROP-INDEXES

Create the indexes that we dropped previously.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-indexes-in-kernel PGCONN TABLE KERNEL CHANNEL &key LABEL

Create indexes for given table in dbname, using given lparallel KERNEL and CHANNEL so that the index build happen in concurrently with the data copying.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-pgsql-fkeys CATALOG &key SECTION LABEL

Actually create the Foreign Key References that where declared in the MySQL database

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-schemas CATALOG &key INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all schemas from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-sqltypes CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create the needed data types for given CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-table MAYBE-QUALIFIED-NAME

Create a table instance from the db-uri component, either a string or a cons of two strings: (schema . table).

Package

pgloader.catalog

Source

catalog.lisp (file)

Function: create-tables CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all tables from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-triggers CATALOG &key LABEL SECTION CLIENT-MIN-MESSAGES

Create the catalog objects that come after the data has been loaded.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: create-views CATALOG &key IF-NOT-EXISTS INCLUDE-DROP CLIENT-MIN-MESSAGES

Create all tables from the given database CATALOG.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: distribute LIST-OF-RANGES COUNT

Split a list of ranges into COUNT sublists.

Package

pgloader.utils

Source

utils.lisp (file)

Function: drop-indexes TABLE-OR-CATALOG &key CASCADE

Drop indexes in PGSQL-INDEX-LIST. A PostgreSQL connection must already be active when calling that function.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: drop-pgsql-fkeys CATALOG &key CASCADE

Drop all Foreign Key Definitions given, to prepare for a clean run.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: elapsed-time-since START &optional END

Return how many seconds ticked between START and now

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: ensure-unquoted IDENTIFIER
Package

pgloader.quoting

Source

quoting.lisp (file)

Function: expand-archive ARCHIVE-FILE &key TMPDIR

Expand given ARCHIVE-FILE in TMPDIR/(pathname-name ARCHIVE-FILE). Return the pathname where we did expand the archive file.

Package

pgloader.archive

Source

archive.lisp (file)

Function: expand-user-homedir-pathname NAMESTRING

Expand NAMESTRING replacing leading ~ with (user-homedir-pathname)

Package

pgloader.utils

Source

utils.lisp (file)

Function: fetch-pgsql-catalog DBNAME &key TABLE SOURCE-CATALOG INCLUDING EXCLUDING

Fetch PostgreSQL catalogs for the target database. A PostgreSQL connection must be opened.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: fkey-columns INSTANCE
Function: (setf fkey-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-condef INSTANCE
Function: (setf fkey-condef) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-deferrable INSTANCE
Function: (setf fkey-deferrable) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-delete-rule INSTANCE
Function: (setf fkey-delete-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-foreign-columns INSTANCE
Function: (setf fkey-foreign-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-foreign-table INSTANCE
Function: (setf fkey-foreign-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-initially-deferred INSTANCE
Function: (setf fkey-initially-deferred) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-match-rule INSTANCE
Function: (setf fkey-match-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-name INSTANCE
Function: (setf fkey-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-oid INSTANCE
Function: (setf fkey-oid) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-table INSTANCE
Function: (setf fkey-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: fkey-update-rule INSTANCE
Function: (setf fkey-update-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: flush-summary &key RESET
Package

pgloader.monitor

Source

monitor.lisp (file)

Function: format-vector-row ROW &optional TRANSFORMS PRE-FORMATTED

Add a ROW in the STREAM, formating ROW in PostgreSQL COPY TEXT format.

See http://www.postgresql.org/docs/9.2/static/sql-copy.html#AEN66609 for details about the format, and format specs.

Package

pgloader.pgsql

Source

copy-format.lisp (file)

Function: get-matching-filenames DIRECTORY REGEX

Apply given REGEXP to the DIRECTORY contents and return the list of matching files.

Package

pgloader.archive

Source

archive.lisp (file)

Function: get-pathname DBNAME TABLE-NAME &key FD-PATH-ROOT

Return a pathname where to read or write the file data

Package

pgloader.sources

Source

files-and-pathnames.lisp (file)

Function: getenv-default NAME &optional DEFAULT

Return the value of the NAME variable as found in the environment, or DEFAULT if that variable isn’t set

Package

pgloader.params

Source

params.lisp (file)

Function: guess-csv-params FILENAME COLS &key SAMPLE-SIZE

Try a bunch of field separators with LINES and return the first one that returns COLS number of columns

Package

pgloader.csv

Source

csv-guess.lisp (file)

Function: http-fetch-file URL &key TMPDIR

Download a file from URL into TMPDIR.

Package

pgloader.archive

Source

archive.lisp (file)

Function: index-columns INSTANCE
Function: (setf index-columns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-condef INSTANCE
Function: (setf index-condef) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-conname INSTANCE
Function: (setf index-conname) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-filter INSTANCE
Function: (setf index-filter) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-fk-deps INSTANCE
Function: (setf index-fk-deps) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-name INSTANCE
Function: (setf index-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-oid INSTANCE
Function: (setf index-oid) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-primary INSTANCE
Function: (setf index-primary) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-schema INSTANCE
Function: (setf index-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-sql INSTANCE
Function: (setf index-sql) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-table INSTANCE
Function: (setf index-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: index-unique INSTANCE
Function: (setf index-unique) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: init-params-from-environment ()

Some of our parameters get their default value from the env. Do that at runtime when using a compiled binary.

Package

pgloader.params

Source

params.lisp (file)

Function: intern-symbol SYMBOL-NAME &optional OVERRIDES

Return a symbol in either PGLOADER.TRANSFORMS if it exists there
already (it’s a user provided function) or a PGLOADER.USER-SYMBOLS package.

OVERRIDES is an alist of symbol . value, allowing called to force certain values: the classic example is how to parse the "nil" symbol-name. Given OVERRIDES as ’((nil . nil)) the returned symbol will be cl:nil rather than pgloader.user-symbols::nil.

Package

pgloader.transforms

Source

transforms.lisp (file)

Function: lisp-code-for-loading-from-copy COPY-CONN FIELDS PG-DB-CONN &key ENCODING COLUMNS GUCS BEFORE AFTER OPTIONS &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-copy.lisp (file)

Function: lisp-code-for-loading-from-csv CSV-CONN FIELDS PG-DB-CONN &key ENCODING COLUMNS GUCS BEFORE AFTER OPTIONS &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-csv.lisp (file)

Function: lisp-code-for-loading-from-dbf DBF-DB-CONN PG-DB-CONN &key ENCODING GUCS BEFORE AFTER OPTIONS
Package

pgloader.parser

Source

command-dbf.lisp (file)

Function: lisp-code-for-loading-from-fixed FIXED-CONN FIELDS PG-DB-CONN &key ENCODING COLUMNS GUCS BEFORE AFTER OPTIONS &aux WORKER-COUNT CONCURRENCY
Package

pgloader.parser

Source

command-fixed.lisp (file)

Function: lisp-code-for-loading-from-ixf IXF-DB-CONN PG-DB-CONN &key GUCS BEFORE AFTER OPTIONS
Package

pgloader.parser

Source

command-ixf.lisp (file)

Function: lisp-code-for-loading-from-mssql MS-DB-CONN PG-DB-CONN &key GUCS CASTS BEFORE AFTER OPTIONS ALTER-SCHEMA ALTER-TABLE INCLUDING EXCLUDING
Package

pgloader.parser

Source

command-mssql.lisp (file)

Function: lisp-code-for-loading-from-mysql MY-DB-CONN PG-DB-CONN &key GUCS MYSQL-GUCS CASTS VIEWS BEFORE AFTER OPTIONS ALTER-TABLE ALTER-SCHEMA (INCLUDING INCL) (EXCLUDING EXCL) (DECODING DECODING-AS)
Package

pgloader.parser

Source

command-mysql.lisp (file)

Function: lisp-code-for-loading-from-sqlite SQLITE-DB-CONN PG-DB-CONN &key GUCS CASTS BEFORE AFTER OPTIONS ALTER-TABLE ALTER-SCHEMA (INCLUDING INCL) (EXCLUDING EXCL)
Package

pgloader.parser

Source

command-sqlite.lisp (file)

Function: list-all-columns CATALOG &key TABLE-TYPE INCLUDING EXCLUDING &aux TABLE-TYPE-NAME

Get the list of PostgreSQL column names per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-all-fkeys CATALOG &key INCLUDING EXCLUDING

Get the list of PostgreSQL index definitions per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-all-indexes CATALOG &key INCLUDING EXCLUDING

Get the list of PostgreSQL index definitions per table.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-encodings-and-aliases ()

Return an alist of encoding names supported by the current implementation, associated with a list of encoding name aliases for each of them.

Package

pgloader.utils

Source

charsets.lisp (file)

Function: list-missing-fk-deps CATALOG

Add in the CATALOG the foreign keys we don’t have to deal with directly but that the primary keys we are going to DROP then CREATE again depend on: we need to take care of those first.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-reserved-keywords PGCONN

Connect to PostgreSQL DBNAME and fetch reserved keywords.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: list-schemas ()

Return the list of PostgreSQL schemas in the already established PostgreSQL connection.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-table-oids TABLE-NAMES

Return an hash table mapping TABLE-NAME to its OID for all table in the TABLE-NAMES list. A PostgreSQL connection must be established already.

Package

pgloader.pgsql

Source

pgsql-schema.lisp (file)

Function: list-typenames-without-btree-support ()

Fetch PostgresQL data types without btree support, so that it’s possible to later CREATE INDEX ... ON ... USING gist(...), or even something else than gist.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: load-data &key (FROM SOURCE) (INTO TARGET) ENCODING FIELDS OPTIONS GUCS CASTS BEFORE AFTER START-LOGGER FLUSH-SUMMARY

Load data from SOURCE into TARGET.

Package

pgloader

Source

main.lisp (file)

Function: log-message CATEGORY DESCRIPTION &rest ARGUMENTS

Send given message into our monitoring queue for processing.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: make-batch &key MAX-COUNT &aux DATA
Package

pgloader.batch

Source

batch.lisp (file)

Function: make-catalog &key (NAME NAME) (SCHEMA-LIST SCHEMA-LIST) (TYPES-WITHOUT-BTREE TYPES-WITHOUT-BTREE)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-column &key (NAME NAME) (TYPE-NAME TYPE-NAME) (TYPE-MOD TYPE-MOD) (NULLABLE NULLABLE) (DEFAULT DEFAULT) (COMMENT COMMENT) (TRANSFORM TRANSFORM) (EXTRA EXTRA)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-external-format NAME

Return an object suitable as an external format in the current implementation.

Package

pgloader.utils

Source

charsets.lisp (file)

Function: make-fkey &key (NAME NAME) (OID OID) (TABLE TABLE) (COLUMNS COLUMNS) (FOREIGN-TABLE FOREIGN-TABLE) (FOREIGN-COLUMNS FOREIGN-COLUMNS) (CONDEF CONDEF) (UPDATE-RULE UPDATE-RULE) (DELETE-RULE DELETE-RULE) (MATCH-RULE MATCH-RULE) (DEFERRABLE DEFERRABLE) (INITIALLY-DEFERRED INITIALLY-DEFERRED)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-index &key (NAME NAME) (OID OID) (SCHEMA SCHEMA) (TABLE TABLE) (PRIMARY PRIMARY) (UNIQUE UNIQUE) (COLUMNS COLUMNS) (SQL SQL) (CONNAME CONNAME) (CONDEF CONDEF) (FILTER FILTER) (FK-DEPS FK-DEPS)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-kernel WORKER-COUNT &key BINDINGS

Wrapper around lparallel:make-kernel that sets our usual bindings.

Package

pgloader.utils

Source

threads.lisp (file)

Function: make-match-rule &key (RULE RULE) (SCHEMA SCHEMA) (ACTION ACTION) (ARGS ARGS)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-pgstate &key (TABLES TABLES) (TABNAMES TABNAMES) (READ READ) (ROWS ROWS) (ERRS ERRS) (SECS SECS) (RS RS) (WS WS)
Package

pgloader.state

Source

state.lisp (file)

Function: make-procedure &key (NAME NAME) (RETURNS RETURNS) (LANGUAGE LANGUAGE) (BODY BODY)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-regex-match-rule &key (TARGET TARGET)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-schema &key (SOURCE-NAME SOURCE-NAME) (NAME NAME) (CATALOG CATALOG) (TABLE-LIST TABLE-LIST) (VIEW-LIST VIEW-LIST)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-sqltype &key (NAME NAME) (TYPE TYPE) (SOURCE-DEF SOURCE-DEF) (EXTRA EXTRA)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-string-match-rule &key (TARGET TARGET)
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: make-table &key (SOURCE-NAME SOURCE-NAME) (NAME NAME) (SCHEMA SCHEMA) (OID OID) (COMMENT COMMENT) (STORAGE-PARAMETER-LIST STORAGE-PARAMETER-LIST) (FIELD-LIST FIELD-LIST) (COLUMN-LIST COLUMN-LIST) (INDEX-LIST INDEX-LIST) (FKEY-LIST FKEY-LIST) (TRIGGER-LIST TRIGGER-LIST)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: make-trigger &key (NAME NAME) (TABLE TABLE) (ACTION ACTION) (PROCEDURE-NAME PROCEDURE-NAME) (PROCEDURE PROCEDURE)
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: match-rule-action INSTANCE
Function: (setf match-rule-action) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-args INSTANCE
Function: (setf match-rule-args) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-rule INSTANCE
Function: (setf match-rule-rule) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: match-rule-schema INSTANCE
Function: (setf match-rule-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: maybe-drop-indexes CATALOG &key DROP-INDEXES

Drop the indexes for TABLE-NAME on TARGET PostgreSQL connection, and returns a list of indexes to create again. A PostgreSQL connection must already be active when calling that function.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: merge-catalogs SOURCE-CATALOG TARGET-CATALOG

In order for the data loading to be as fast as possible, we DROP the constraints and indexes on the target table. Once the data is loaded we want to install the same constraints as found pre-existing in the TARGET-CATALOG rather than the one we casted from the SOURCE-CATALOG.

Also, we want to recheck the cast situation and the selected transformation functions of each column.

Package

pgloader.pgsql

Source

merge-catalogs.lisp (file)

Function: new-label SECTION LABEL &optional DBNAME

Send an event to create a new LABEL for registering a shared state under SECTION.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: parse-cli-casts CASTS

Parse additional CAST rules when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-encoding ENCODING

Parse the –encoding option

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-fields TYPE FIELDS

Parse the –fields option.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-gucs GUCS

Parse PostgreSQL GUCs as per the SET clause when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-options TYPE OPTIONS

Parse options as per the WITH clause when we get them from the CLI.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-cli-type TYPE

Parse the –type option

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-commands COMMANDS

Parse a command and return a LAMBDA form that takes no parameter.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-commands-from-file MAYBE-RELATIVE-FILENAME &aux FILENAME

The command could be using from :inline, in which case we want to parse as much as possible then use the command against an already opened stream where we moved at the beginning of the data.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-date-format FORMAT-STRING

Parse a given format string and return a format specification for parse-date-string

Package

pgloader.parse-date

Source

date-format.lisp (file)

Function: parse-date-string DATE-STRING &optional FORMAT

Apply this function when input date in like ’20041002152952’

Package

pgloader.parse-date

Source

date-format.lisp (file)

Function: parse-ini-file FILENAME

Parse an old-style INI file into a list of PARAMS structures

Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: parse-source-string SOURCE-STRING
Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-source-string-for-type TYPE SOURCE-STRING

use the parse rules as per xxx-source rules

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-sql-file FILENAME

Parse FILENAME for SQL statements

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-target-pg-db-uri COMMAND-FILE

Partially parse COMMAND-FILE and return its target connection string.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: parse-target-string TARGET-STRING
Package

pgloader.parser

Source

command-parser.lisp (file)

Function: pgsql-connect-and-execute-with-timing PGCONN SECTION LABEL SQL

Run pgsql-execute-with-timing within a newly establised connection.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgsql-execute SQL &key CLIENT-MIN-MESSAGES ON-ERROR-STOP

Execute given SQL list of statements in current transaction.

When ON-ERROR-STOP is non-nil (the default), we stop at the first sql statement that fails. That’s because this facility is meant for DDL. With ON_ERROR_STOP nil, log the problem and continue thanks to PostgreSQL savepoints.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgsql-execute-with-timing SECTION LABEL SQL-LIST &key ON-ERROR-STOP CLIENT-MIN-MESSAGES

Execute given SQL and resgister its timing into STATE.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: pgstate-decf PGSTATE NAME &key READ ROWS ERRS SECS RS WS
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-errs INSTANCE
Function: (setf pgstate-errs) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-get-label PGSTATE NAME
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-incf PGSTATE NAME &key READ ROWS ERRS SECS RS WS
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-new-label PGSTATE LABEL

Instanciate a new pgtable structure to hold our stats, and return it.

Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-read INSTANCE
Function: (setf pgstate-read) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-rows INSTANCE
Function: (setf pgstate-rows) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-secs INSTANCE
Function: (setf pgstate-secs) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-setf PGSTATE NAME &key READ ROWS ERRS SECS RS WS
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-tables INSTANCE
Function: (setf pgstate-tables) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgstate-tabnames INSTANCE
Function: (setf pgstate-tabnames) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgtable-reject-data INSTANCE
Function: (setf pgtable-reject-data) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgtable-reject-logs INSTANCE
Function: (setf pgtable-reject-logs) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgtable-secs INSTANCE
Function: (setf pgtable-secs) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgtable-start INSTANCE
Function: (setf pgtable-start) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pgtable-stop INSTANCE
Function: (setf pgtable-stop) VALUE INSTANCE
Package

pgloader.state

Source

state.lisp (file)

Function: pretty-print-bytes BYTES &key UNIT

Return a string to reprensent bytes in human readable format, with units

Package

pgloader.utils

Source

utils.lisp (file)

Function: procedure-body INSTANCE
Function: (setf procedure-body) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: procedure-language INSTANCE
Function: (setf procedure-language) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: procedure-name INSTANCE
Function: (setf procedure-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: procedure-returns INSTANCE
Function: (setf procedure-returns) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: process-bad-row TABLE CONDITION DATA

Send an event to log the bad row DATA in the reject and log files for given TABLE-NAME (a label in section :data), for reason found in CONDITION.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: process-relative-pathnames FILENAME COMMAND

Walk the COMMAND to replace relative pathname with absolute ones, merging them within the directory where we found the command FILENAME.

Package

pgloader.parser

Source

command-parser.lisp (file)

Function: project-fields &key FIELDS COLUMNS COMPILE

The simplest projection happens when both FIELDS and COLS are nil: in this case the projection is an identity, we simply return what we got.

Other forms of projections consist of forming columns with the result of applying a transformation function. In that case a cols entry is a list of ’(colname type expression), the expression being the (already compiled) function to use here.

Package

pgloader.sources

Source

project-fields.lisp (file)

Function: push-row BATCH ROW &optional ROW-BYTES
Package

pgloader.batch

Source

batch.lisp (file)

Function: quoted-p S

Return true if s is a double-quoted string

Package

pgloader.quoting

Source

quoting.lisp (file)

Function: read-ini-file FILENAME
Package

pgloader.ini

Source

parse-ini.lisp (file)

Function: read-queries FILENAME

read SQL queries in given file and split them, returns a list

Package

pgloader.sql

Source

read-sql-files.lisp (file)

Function: reformat-then-process &key FIELDS COLUMNS TARGET

Return a lambda form to apply to each row we read.

The lambda closes over the READ paramater, which is a counter of how many lines we did read in the file.

Package

pgloader.sources

Source

project-fields.lisp (file)

Function: regex-match-rule-target INSTANCE
Function: (setf regex-match-rule-target) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: report-footer LEGEND READ ROWS ERRORS SECONDS &optional RS WS
Package

pgloader.state

Source

report.lisp (file)

Function: report-full-summary LEGEND SECTIONS TOTAL-SECS

Report the full story when given three different sections of reporting.

Package

pgloader.state

Source

report.lisp (file)

Function: report-header ()
Package

pgloader.state

Source

report.lisp (file)

Function: report-pgstate-stats PGSTATE LEGEND
Package

pgloader.state

Source

report.lisp (file)

Function: report-pgtable-stats PGSTATE NAME
Package

pgloader.state

Source

report.lisp (file)

Function: report-results READ ROWS ERRORS SECONDS RS WS &optional EOL
Package

pgloader.state

Source

report.lisp (file)

Function: report-summary PGSTATE &key HEADER FOOTER

Report a whole summary.

Package

pgloader.state

Source

report.lisp (file)

Function: report-table-name TABLE-NAME
Package

pgloader.state

Source

report.lisp (file)

Function: reset-sequences TARGET CATALOG &key SECTION

Reset all sequences created during this MySQL migration.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: run-commands SOURCE &key START-LOGGER FLUSH-SUMMARY (SUMMARY *SUMMARY-PATHNAME*) (LOG-FILENAME *LOG-FILENAME*) (LOG-MIN-MESSAGES *LOG-MIN-MESSAGES*) (CLIENT-MIN-MESSAGES *CLIENT-MIN-MESSAGES*)

SOURCE can be a function, which is run, a list, which is compiled as CL code then run, a pathname containing one or more commands that are parsed then run, or a commands string that is then parsed and each command run.

Package

pgloader

Source

main.lisp (file)

Function: sanitize-user-gucs GUCS

Forbid certain actions such as setting a client_encoding different from utf8.

Package

pgloader.pgsql

Source

connection.lisp (file)

Function: schema-catalog INSTANCE
Function: (setf schema-catalog) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: schema-name INSTANCE
Function: (setf schema-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: schema-source-name INSTANCE
Function: (setf schema-source-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: schema-table-list INSTANCE
Function: (setf schema-table-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: schema-view-list INSTANCE
Function: (setf schema-view-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: send-event EVENT

Add a new event to be processed by the monitor.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: set-table-oids CATALOG

MySQL allows using the same index name against separate tables, which PostgreSQL forbids. To get unicity in index names without running out of characters (we are allowed only 63), we use the table OID instead.

This function grabs the table OIDs in the PostgreSQL database and update the definitions with them.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: show-encodings ()

List known encodings names and aliases from charsets::*lisp-encodings*.

Package

pgloader.utils

Source

charsets.lisp (file)

Function: split-range MIN MAX &optional COUNT

Split the range from MIN to MAX into sub-ranges of COUNT elements.

Package

pgloader.utils

Source

utils.lisp (file)

Function: sql URL

Abstract the hash-table based implementation of our SQL file system.

Package

pgloader.queries

Source

queries.lisp (file)

Function: sqltype-extra INSTANCE
Function: (setf sqltype-extra) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: sqltype-name INSTANCE
Function: (setf sqltype-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: sqltype-source-def INSTANCE
Function: (setf sqltype-source-def) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: sqltype-type INSTANCE
Function: (setf sqltype-type) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: start-logger &key LOG-FILENAME (LOG-MIN-MESSAGES *LOG-MIN-MESSAGES*) (CLIENT-MIN-MESSAGES *CLIENT-MIN-MESSAGES*)

Start the pgloader log manager and messenger.

Package

pgloader.logs

Source

logs.lisp (file)

Function: start-monitor &key START-LOGGER (QUEUE *MONITORING-QUEUE*) (LOG-FILENAME *LOG-FILENAME*) (LOG-MIN-MESSAGES *LOG-MIN-MESSAGES*) (CLIENT-MIN-MESSAGES *CLIENT-MIN-MESSAGES*)

Start the monitor and its logger.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: stop-logger ()

Stop the pgloader manager and messengers.

Package

pgloader.logs

Source

logs.lisp (file)

Function: stop-monitor &key CHANNEL STOP-LOGGER

Stop the current monitor task.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: string-match-rule-target INSTANCE
Function: (setf string-match-rule-target) VALUE INSTANCE
Package

pgloader.catalog

Source

alter-table.lisp (file)

Function: table-column-list INSTANCE
Function: (setf table-column-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-comment INSTANCE
Function: (setf table-comment) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-field-list INSTANCE
Function: (setf table-field-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-fkey-list INSTANCE
Function: (setf table-fkey-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-index-list INSTANCE
Function: (setf table-index-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-name INSTANCE
Function: (setf table-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-oid INSTANCE
Function: (setf table-oid) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-schema INSTANCE
Function: (setf table-schema) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-source-name INSTANCE
Function: (setf table-source-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-storage-parameter-list INSTANCE
Function: (setf table-storage-parameter-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: table-trigger-list INSTANCE
Function: (setf table-trigger-list) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: trigger-action INSTANCE
Function: (setf trigger-action) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: trigger-name INSTANCE
Function: (setf trigger-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: trigger-procedure INSTANCE
Function: (setf trigger-procedure) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: trigger-procedure-name INSTANCE
Function: (setf trigger-procedure-name) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: trigger-table INSTANCE
Function: (setf trigger-table) VALUE INSTANCE
Package

pgloader.catalog

Source

catalog.lisp (file)

Function: truncate-tables CATALOG-OR-TABLE

Truncate given TABLE-NAME in database DBNAME. A PostgreSQL connection must already be active when calling that function.

Package

pgloader.pgsql

Source

pgsql-create-schema.lisp (file)

Function: typemod-expr-to-function EXPR

Transform given EXPR into a callable function object.

Package

pgloader.transforms

Source

transforms.lisp (file)

Function: unquote STRING &optional QUOTE ESCAPE

Given ’0’, returns 0.

Package

pgloader.utils

Source

utils.lisp (file)

Function: update-stats SECTION LABEL &key READ ROWS ERRS SECS RS WS START STOP

Send an event to update stats for given SECTION and LABEL.

Package

pgloader.monitor

Source

monitor.lisp (file)

Function: write-command-to-string CONFIG SECTION &key WITH-DATA-INLINE END-COMMAND

Return the new syntax for the command found in SECTION.

When WITH-DATA-INLINE is true, instead of using the SECTION’s filename option, use the constant INLINE in the command.

Package

pgloader.ini

Source

parse-ini.lisp (file)


Next: , Previous: , Up: Exported definitions   [Contents][Index]

6.1.4 Generic functions

Generic Function: add-column OBJECT COLUMN &key
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-column (INDEX index) COLUMN &key

Add COLUMN name to INDEX and return the INDEX.

Method: add-column (TABLE table) COLUMN &key

Add COLUMN to TABLE and return the TABLE.

Generic Function: add-field TABLE FIELD &key
Package

pgloader.catalog

Methods
Method: add-field (TABLE table) FIELD &key

Add COLUMN to TABLE and return the TABLE.

Source

catalog.lisp (file)

Generic Function: add-fkey OBJECT FKEY &key
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-fkey (TABLE table) FKEY &key

Add FKEY to TABLE and return the TABLE.

Generic Function: add-index OBJECT INDEX &key
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-index (TABLE table) INDEX &key

Add INDEX to TABLE and return the TABLE.

Generic Function: add-schema OBJECT SCHEMA-NAME &key
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-schema (CATALOG catalog) SCHEMA-NAME &key

Add SCHEMA-NAME to CATALOG and return the new schema instance.

Generic Function: add-table OBJECT TABLE-NAME &key COMMENT OID
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-table (SCHEMA schema) TABLE-NAME &key COMMENT OID

Add TABLE-NAME to SCHEMA and return the new table instance.

Generic Function: add-view OBJECT VIEW-NAME &key COMMENT
Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: add-view (SCHEMA schema) VIEW-NAME &key COMMENT

Add TABLE-NAME to SCHEMA and return the new table instance.

Generic Function: alter-schema OBJECT ALTER-SCHEMA-RULE-LIST
Package

pgloader.catalog

Source

alter-table.lisp (file)

Methods
Method: alter-schema (SCHEMA schema) ALTER-SCHEMA-RULE-LIST

Apply ALTER-SCHEMA-RULE-LIST to SCHEMA.

Method: alter-schema (CATALOG catalog) ALTER-SCHEMA-RULE-LIST

Apply ALTER-SCHEMA-RULE-LIST to all schema of CATALOG.

Generic Function: alter-table OBJECT ALTER-TABLE-RULE-LIST
Package

pgloader.catalog

Source

alter-table.lisp (file)

Methods
Method: alter-table (TABLE table) ALTER-TABLE-RULE-LIST

Apply ALTER-TABLE-RULE-LIST to TABLE.

Method: alter-table (SCHEMA schema) ALTER-TABLE-RULE-LIST

Apply ALTER-TABLE-RULE-LIST to all tables and views of SCHEMA.

Method: alter-table (CATALOG catalog) ALTER-TABLE-RULE-LIST

Apply ALTER-TABLE-RULE-LIST to all schema of CATALOG.

Generic Function: cast OBJECT

Cast a FIELD definition from a source database into a PostgreSQL COLUMN definition.

Package

pgloader.catalog

Source

catalog.lisp (file)

Methods
Method: cast (COL mysql-column)

Return the PostgreSQL type definition from given MySQL column definition.

Source

mysql-cast-rules.lisp (file)

Method: cast (FIELD mssql-column)

Return the PostgreSQL type definition from given MS SQL column definition.

Source

mssql-cast-rules.lisp (file)

Method: cast (COL coldef)

Return the PostgreSQL type definition from given SQLite column definition.

Source

sqlite-cast-rules.lisp (file)

Method: cast (COL ixf-column)

Return the PostgreSQL type definition from given IXF column definition.

Source

ixf-schema.lisp (file)

Method: cast (FIELD db3-field)

Return the PostgreSQL type definition given the DB3 one.

Source

db3-schema.lisp (file)

Method: cast (CATALOG catalog)

Cast all fields of all tables in all schemas in CATALOG into columns.

Method: cast (SCHEMA schema)

Cast all fields of all tables in SCHEMA into columns.

Method: cast (TABLE table)

Cast all fields in table into columns.

Generic Function: check-connection CONNECTION

Check that we can actually connect.

Package

pgloader.connection

Source

connection.lisp (file)

Methods
Method: check-connection (C db-connection)

Check that it is possible to connect to db-connection C.

Method: check-connection (FD fd-connection)

Check that it is possible to connect to db-connection C.

Generic Function: cleanup DB-COPY CATALOG &key MATERIALIZE-VIEWS

Clean-up after prepare-pgsql-database failure.

Package

pgloader.sources

Source

api.lisp (file)

Methods
Method: cleanup (MYSQL copy-mysql) (CATALOG catalog) &key MATERIALIZE-VIEWS

When there is a PostgreSQL error at prepare-pgsql-database step, we might need to clean-up any view created in the MySQL connection for the migration purpose.

Source

mysql.lisp (file)

Method: cleanup (COPY db-copy) (CATALOG catalog) &key MATERIALIZE-VIEWS

In case anything wrong happens at ‘prepare-pgsql-database’ step, this function will be called to clean-up the mess left behind, if any.

Source

db-methods.lisp (file)

Generic Function: clone-connection CONNECTION

Instanciate a new connection object with similar properties.

Package

pgloader.connection

Source

connection.lisp (file)

Methods
Method: clone-connection (C mysql-connection)
Source

mysql-schema.lisp (file)

Method: clone-connection (C mssql-connection)
Source

mssql-schema.lisp (file)

Method: clone-connection (SLCONN sqlite-connection)
Source

sqlite-schema.lisp (file)

Method: clone-connection (C ixf-connection)
Source

ixf-schema.lisp (file)

Method: clone-connection (C dbf-connection)
Source

db3-schema.lisp (file)

Method: clone-connection (C pgsql-connection)
Source

connection.lisp (file)

Method: clone-connection (C db-connection)
Method: clone-connection (FD fd-connection)
Generic Function: clone-copy-for MD-COPY PATH-SPEC

Create a new instance for copying PATH-SPEC data.

Package

pgloader.sources

Source

api.lisp (file)

Methods
Method: clone-copy-for (COPY copy-copy) PATH-SPEC

Create a copy of FIXED for loading data from PATH-SPEC.

Source

copy.lisp (file)

Method: clone-copy-for (FIXED copy-fixed) PATH-SPEC

Create a copy of FIXED for loading data from PATH-SPEC.

Source

fixed.lisp (file)

Method: clone-copy-for (CSV copy-csv) PATH-SPEC

Create a copy of CSV for loading data from PATH-SPEC.

Source

csv.lisp (file)

Method: clone-copy-for (COPY md-copy) PATH-SPEC

Create a copy of CSV for loading data from PATH-SPEC.

Source

md-methods.lisp (file)

Generic Function: close-connection CONNECTION

Close a connection to the data source.

Package

pgloader.connection

Source

connection.lisp (file)

Methods
Method: close-connection (MYCONN mysql-connection)
Source

mysql-schema.lisp (file)

Method: close-connection (MSCONN mssql-connection)
Source

mssql-schema.lisp (file)

Method: close-connection (SLCONN sqlite-connection)
Source

sqlite-schema.lisp (file)

Method: close-connection (IXFCONN ixf-connection)
Source

ixf-schema.lisp (file)

Method: close-connection (DBFCONN dbf-connection)
Source

db3-schema.lisp (file)

Method: close-connection (MD md-connection)

Reset.

Source

files-and-pathnames.lisp (file)

Method: close-connection (PGCONN pgsql-connection)

Close a PostgreSQL connection.

Source

connection.lisp (file)

Generic Function: columns OBJECT
Generic Function: (setf columns) NEW-VALUE OBJECT
Package

pgloader.sources

Methods
Method: columns (COPY copy)

automatically generated reader method

Source

api.lisp (file)

Method: (setf columns) NEW-VALUE (COPY copy)

automatically generated writer method

Source

api.lisp (file)

Generic Function: complete-pgsql-database DB-COPY CATALOG PKEYS &key FOREIGN-KEYS CREATE-INDEXES CREATE-TRIGGERS RESET-SEQUENCES

Alter load duties for database sources copy support.

Package

pgloader.sources

Source

api.lisp (file)

Methods
Method: complete-pgsql-database (COPY db-copy) (CATALOG catalog) PKEYS &key FOREIGN-KEYS CREATE-INDEXES CREATE-TRIGGERS RESET-SEQUENCES

After loading the data into PostgreSQL, we can now reset the sequences and declare foreign keys.

Source

db-methods.lisp (file)

Generic Function: concurrency-support COPY CONCURRENCY

Returns nil when no concurrency is supported, or a list of copy ojbects prepared to run concurrently.

Package

pgloader.sources

Source

api.lisp (file)

Methods
Method: concurrency-support (MYSQL copy-mysql) CONCURRENCY

Splits the read work thanks WHERE clauses when possible and relevant, return nil if we decide to read all in a single thread, and a list of as many copy-mysql instances as CONCURRENCY otherwise. Each copy-mysql instance in the returned list embeds specifications about how to read only its partition of the source data.

Source

mysql.lisp (file)

Generic Function: conn-handle OBJECT
Generic Function: (setf conn-handle) NEW-VALUE OBJECT
Package

pgloader.connection

Methods
Method: conn-handle (CONNECTION connection)

automatically generated reader method

Source

connection.lisp (file)

Method: (setf conn-handle) NEW-VALUE (CONNECTION connection)

automatically generated writer method

Source

connection.lisp (file)

Generic Function: conn-type OBJECT
Generic Function: (setf conn-type) NEW-VALUE OBJECT
Package

pgloader.connection

Methods
Method: conn-type (CONNECTION connection)

automatically generated reader method

Source

connection.lisp (file)

Method: (setf conn-type) NEW-VALUE (CONNECTION connection)

automatically generated writer method

Source

connection.lisp (file)

Generic Function: copy-column-list SOURCE

Return the list of column names for the data sent in the queue.

Package

pgloader.sources

Source

api.lisp (file)

Methods
Method: copy-column-list (MYSQL copy-mysql)

We are sending the data in the MySQL columns ordering here.

Source

mysql.lisp (file)

Method: copy-column-list (SQLITE copy-sqlite)

Send the data in the SQLite column ordering.

Source

sqlite.lisp (file)

Method: copy-column-list (COPY md-copy)

We did reformat-then-process the column list, so we now send them in the COPY buffer as found in (columns fixed).

Source