The cl-migratum Reference Manual

Next: , Previous: , Up: (dir)   [Contents][Index]

The cl-migratum Reference Manual

This is the cl-migratum Reference Manual, version 0.6.0, generated automatically by Declt version 4.0 beta 2 "William Riker" on Tue Nov 15 04:55:51 2022 GMT+0.

Table of Contents


1 Introduction

Database Schema Migration System for Common Lisp

cl-migratum is a Common Lisp system, which provides facilities for performing database schema migrations.

migratum-demo

Requirements

Systems

The following systems are available as part of this repo.

| System | Description | |---------------------------------------|-----------------------------------------------------| | cl-migratum | Core system | | cl-migratum.provider.local-path | Provider which discovers migrations from local-path | | cl-migratum.driver.dbi | cl-dbi database driver | | cl-migratum.driver.rmdbs-postgresql | PostgreSQL driver based on hu.dwim.rdbms | | cl-migratum.driver.mixins | Various mixin classes used by drivers | | cl-migratum.test | Test suite for cl-migratum | | cl-migratum.cli | CLI application of migratum |

Installation

Latest development version can be installed from the Git repo.

Clone the cl-migratum repo in your Quicklisp local-projects directory.

git clone https://github.com/dnaeon/cl-migratum.git

CLI

You can install the CLI application of migratum by executing the following command.

make cli

The migratum binary will be built into the bin/migratum directory. You can now install the binary somewhere in your PATH.

Once the migratum binary is built you can also generate the Zsh completions by executing the following command.

migratum zsh-completions > ~/.zsh-completions/_migratum

Make sure that ~/.zsh-completions is part of your Zsh FPATH.

In order to generate the Markdown documentation of the CLI app use the cli-doc target:

make cli-doc

Or you can build a Docker image instead.

docker build -t migratum.cli:latest -f Dockerfile.cli .

Concepts

The cl-migratum system uses the following concepts to describe the process of discovering and applying schema migrations, so it is important that you get familiar with them first.

Migration

A migration represents a resource that provides information about a schema change, e.g. it provides the unique id of the change, the required scripts that can be used to upgrade and/or downgrade the database.

Migration resources are discovered via providers and are being used by drivers during the upgrade or downgrade process.

Provider

The provider is responsible for discovering migration resources.

For example a provider can discover migrations from a local path by scanning for files that match a given pattern or fetch migrations from a remote endpoint (e.g. HTTP service).

The provider is also responsible for creating new migration sequences.

The following providers are supported by cl-migratum.

| Name | Description | System | |--------------|------------------------------------------------------------------|-----------------------------------| | local-path | Provider that can discover migration resources from a local path | cl-migratum.provider.local-path |

Driver

The driver carries out the communication with the database against which schema changes will be applied.

It is responsible for applying schema changes, registering successfully applied migrations and unregistering them when reverting back to a previous state.

A driver internally uses a provider in order to discover migrations, which can be applied against the database it is connected to.

The following drivers are currently supported by cl-migratum.

| Name | Description | System | Migration Registration | |--------------------|---------------------------------------------------------------------------------------------------------------------------|---------------------------------------|------------------------| | dbi | Driver for performing schema migrations against a SQL database using CL-DBI | cl-migratum.driver.dbi | In table migration | | rdbms-postgresql | Driver for performing schema migrations against a SQL database using hu.dwim.rdbms | cl-migratum.driver.rdbms-postgresql | In table migration |

Usage

The following section contains some examples to get you started.

In order to use cl-migratum you will need to load the core system, along with any provider and driver systems.

Load the core system.

CL-USER> (ql:quickload :cl-migratum)

Create Provider

First, we will create a new provider that can discover migration files from a local path. In order to create a local-path provider we need to load the respective system.

CL-USER> (ql:quickload :cl-migratum.provider.local-path)

Once you load the system we can create a local-path provider. The local-path provider can discover migrations from multiple paths.

Typical example where having multiple paths with migration resources might be useful is when you have a set of base migrations you want to apply to all environments, and then have a separate path for each environment with their own migrations. For example you might want to run specific migrations only in your dev environment, but don't want them in your production environment.

In that case it makes sense to separate the migration resources in multiple paths, for each environment respectively.

CL-USER> (defparameter *provider*
           (migratum.provider.local-path:make-provider (list #P"/path/to/cl-migratum/t/migrations/")))
*PROVIDER*

The LOCAL-PATH-PROVIDER discovers migration files which match the following pattern.

| Pattern | Description | |----------------------------------|------------------| | <id>-<description>.up.<kind> | Upgrade script | | <id>-<description>.down.<kind> | Downgrade script |

In above pattern <id> is a monotonically increasing integer, which represents the timestamp the migration has been created. The format of <id> is YYYYMMDDHHMMSS.

The supported migration kinds by LOCAL-PATH-PROVIDER are these.

| Extension | Kind | Description | |-----------|---------|--------------------------------------------------| | .sql | :sql | SQL migration resource | | .lisp | :lisp | Migration resource which invokes a Lisp function |

A provider can optionally be initialized, which can be done using the MIGRATUM:PROVIDER-INIT generic function. Not all providers would require initialization, but some will and therefore it is good that you always initialize them first.

In order to list the migrations provided by a provider you can use the MIGRATUM:PROVIDER-LIST-MIGRATIONS function, e.g.

CL-USER> (migratum:provider-list-migrations *provider*)
(#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {1004713F73}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603BE3}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603B03}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {10046039B3}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603733}>)

The following generic functions can be used to interact with discovered migrations.

| Method | Description | |----------------------------------|---------------------------------------------------------------| | MIGRATUM:MIGRATION-ID | Returns the unique migration id | | MIGRATUM:MIGRATION-DESCRIPTION | Returns the description of the migration | | MIGRATUM:MIGRATION-APPLIED | Returns the timestamp of when the migration was applied | | MIGRATUM:MIGRATION-LOAD | Loads the :up or :down migration | | MIGRATUM:MIGRATION-KIND | Returns the kind of the migration, e.g. :sql, :lisp, etc. |

For example in order to collect the unique IDs of all migration resources you can evaluate the following while at the REPL.

CL-USER> (mapcar #'migratum:migration-id
                 (migratum:provider-list-migrations *provider*))
(20200421180337 20200421173908 20200421173657 ...)

Create Driver

Once we have a provider for discovering migration resources we need to create a driver, which can be used to communicate with the database we want to apply migrations on.

DBI Driver

Here is how we can create a dbi driver. First, lets load the system, which provides that driver.

CL-USER> (ql:quickload :cl-migratum.driver.dbi)

The dbi driver uses CL-DBI interface to communicate with the database, so we will need to create a database connection in order to initialize it.

CL-USER> (defparameter *conn*
           (dbi:connect :sqlite3 :database-name "/path/to/migratum.db"))
*CONN*

And now we can instantiate our SQL driver using the previously created provider and connection.

CL-USER> (defparameter *driver*
           (migratum.driver.dbi:make-driver *provider* *conn*))
*DRIVER*

RDBMS PostgreSQL Driver

The process to create an rdbms-postgresql driver is analogous. The driver needs a connection specification to create a DB connection. In this example we use the same environment variables which are used by the PostgreSQL client psql to override the defaults:

(ql:quickload :cl-migratum.driver.rdbms-postgresql)
(defparameter *driver*
  (migratum.driver.rdbms-postgresql:make-driver
   *provider*
   `(:host ,(or (uiop:getenv "PGHOST") "localhost")
     :database ,(or (uiop:getenv "PGDATABASE") "migratum")
     :user-name ,(or (uiop:getenv "PGUSER") "migratum")
     :password ,(or (uiop:getenv "PGPASSWORD") "tiger"))))

Initialization

A driver and provider may require some initialization steps to be executed before being able to apply migrations, so make sure that you initialize them.

In order to initialize your provider use the MIGRATUM:PROVIDER-INIT function.

CL-USER> (migratum:provider-init *provider*)

An example requirement for a driver might be to create some required database schema used to track which migrations have been applied already, so lets initialize our driver first.

CL-USER> (migratum:driver-init *driver*)

Pending Migrations

In order to get the list of pending (not applied yet) migrations we can use the MIGRATUM:LIST-PENDING function, e.g.

CL-USER> (migratum:list-pending *driver*)
(#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363DFC3}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E0A3}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E183}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E263}>
 #<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100363E343}>)

Or, we can display a table of the pending migrations using the MIGRATUM:DISPLAY-PENDING function instead.

CL-USER> (migratum:display-pending *driver*)
.---------------------------------------------.
|             PENDING MIGRATIONS              |
+----------------+---------------------+------+
| ID             | DESCRIPTION         | KIND |
+----------------+---------------------+------+
| 20200421173657 | create_table_foo    | SQL  |
| 20200421173908 | create_table_bar    | SQL  |
| 20200421180337 | create_table_qux    | SQL  |
| 20200605144633 | multiple_statements | SQL  |
| 20220327224455 | lisp_code_migration | LISP |
+----------------+---------------------+------+
|                | TOTAL               |    5 |
+----------------+---------------------+------+
NIL

The migrations will be sorted in the order they need to be applied.

Applying Migrations

The following functions are used for applying pending migrations.

This is how we can apply all pending migrations for example.

CL-USER> (migratum:apply-pending *driver*)
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-pending base-driver) -
  Found 5 pending migration(s) to be applied
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421173657 - create_table_foo (SQL)
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421173908 - create_table_bar (SQL)
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421180337 - create_table_qux (SQL)
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200605144633 - multiple_statements (SQL)
 <INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20220327224455 - lisp_code_migration (LISP)
NIL

Get Latest Migration

You can use the MIGRATUM:LATEST-MIGRATION function to get the latest applied migration, e.g.

CL-USER> (migratum:migration-id (migratum:latest-migration *driver*))
20220327224455 (45 bits, #x1263E96BE487)

The MIGRATUM:CONTAINS-APPLIED-MIGRATIONS-P predicate can be used to query whether there are any migrations applied, e.g.

CL-USER> (migratum:contains-applied-migrations-p *driver*)
T

Displaying Applied Migrations

The following functions can be used to get and display the list of applied database migrations.

| Function | Description | |--------------------------------|----------------------------------------| | MIGRATUM:DRIVER-LIST-APPLIED | Returns the list of applied migrations | | MIGRATUM:DISPLAY-APPLIED | Display a table of applied migrations |

This is how we can get the list of applied migrations.

CL-USER> (migratum:driver-list-applied *driver*)
(#<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175993}>
 #<CL-MIGRATUM.CORE:BASE-MIGRATION {10021759D3}>
 #<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A13}>
 #<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A53}>
 #<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A93}>)

Or we can display a nice table of the applied migrations instead.

CL-USER> (migratum:display-applied *driver*)
.-------------------------------------------------------------------.
|                        APPLIED MIGRATIONS                         |
+----------------+---------------------+---------------------+------+
| ID             | DESCRIPTION         | APPLIED             | KIND |
+----------------+---------------------+---------------------+------+
| 20220327224455 | lisp_code_migration | 2022-03-29 18:20:53 | LISP |
| 20200605144633 | multiple_statements | 2022-03-29 18:20:51 | SQL  |
| 20200421180337 | create_table_qux    | 2022-03-29 18:20:51 | SQL  |
| 20200421173908 | create_table_bar    | 2022-03-29 18:20:51 | SQL  |
| 20200421173657 | create_table_foo    | 2022-03-29 18:20:51 | SQL  |
+----------------+---------------------+---------------------+------+
|                |                     | TOTAL               |    5 |
+----------------+---------------------+---------------------+------+
NIL

The output of these functions will be the applied migrations in descending order by their id, first one being the most recent one.

The dbi and rdbms-postgresql drivers by default will fetch the last 100 applied migrations. You can control this behaviour by using the :offset and :limit keyword parameters, which allows you to fetch applied migrations in pages.

For example, if you are interested only in the last ten applied migrations you can evaluate the following expression.

CL-USER> (migratum:display-applied *driver* :limit 10)

Or if you want to skip the first ten migrations, you can evaluate this expression instead.

CL-USER> (migratum:display-applied *driver* :offset 10 :limit 10)

Stepping through migrations

Using the following functions you can step through migrations.

| Function | Description | |------------------------|--------------------------------------| | MIGRATUM:APPLY-NEXT | Apply the next pending migration(s) | | MIGRATUM:REVERT-LAST | Revert the last applied migration(s) |

This is useful in situations when you don't want to apply all migrations at once, but rather do it one at a time. Both of these functions accept a keyword parameter :count which specifies the number of migrations to apply or revert.

Consider the following pending migrations.

CL-USER> (migratum:display-pending *driver*)
.---------------------------------------------.
|             PENDING MIGRATIONS              |
+----------------+---------------------+------+
| ID             | DESCRIPTION         | KIND |
+----------------+---------------------+------+
| 20200421173657 | create_table_foo    | SQL  |
| 20200421173908 | create_table_bar    | SQL  |
| 20200421180337 | create_table_qux    | SQL  |
| 20200605144633 | multiple_statements | SQL  |
| 20220327224455 | lisp_code_migration | LISP |
+----------------+---------------------+------+
|                | TOTAL               |    5 |
+----------------+---------------------+------+
NIL

We can apply them one by one and verify them as we go.

CL-USER> (migratum:apply-next *driver*)
 <INFO> [21:25:45] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421173657 - create_table_foo (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
 <INFO> [21:25:47] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421173908 - create_table_bar (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
 <INFO> [21:25:48] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200421180337 - create_table_qux (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
 <INFO> [21:25:49] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20200605144633 - multiple_statements (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
 <INFO> [21:25:50] cl-migratum.core core.lisp (apply-and-register base-driver) -
  Applying migration 20220327224455 - lisp_code_migration (LISP)
NIL

If we want to revert the last three migrations we can use the MIGRATUM:REVERT-LAST function, e.g.

CL-USER> (migratum:revert-last *driver* :count 3)
 <INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
  Reverting migration 20220327224455 - lisp_code_migration (LISP)
 <INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
  Reverting migration 20200605144633 - multiple_statements (SQL)
 <INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
  Reverting migration 20200421180337 - create_table_qux (SQL)
NIL

Now, we should have 3 pending migrations again.

CL-USER> (migratum:display-pending *driver*)
.---------------------------------------------.
|             PENDING MIGRATIONS              |
+----------------+---------------------+------+
| ID             | DESCRIPTION         | KIND |
+----------------+---------------------+------+
| 20200421180337 | create_table_qux    | SQL  |
| 20200605144633 | multiple_statements | SQL  |
| 20220327224455 | lisp_code_migration | LISP |
+----------------+---------------------+------+
|                | TOTAL               |    3 |
+----------------+---------------------+------+
NIL

Creating New Migrations

The MIGRATUM:PROVIDER-CREATE-MIGRATION generic function creates a new migration sequence. The arguments it expects are the direction (e.g. :up or :down), the migration kind (e.g. :sql, :lisp, etc.), an instance of a provider, id, description and content.

Knowing migratum:provider-create-migration by the documentation below, you can also use the wrapper migratum.provider.local-path:touch-migration to create a pair of up-and-down empty migrations with a LOCAL-PATH provider.

SQL migrations

Here's an example of using the LOCAL-PATH provider for creating a new SQL migration.

CL-USER> (defparameter *migration-id* (migratum:make-migration-id))
*MIGRATION-ID*
CL-USER>
CL-USER> (migratum:provider-create-migration
          :up            ;; <- direction
          :sql           ;; <- kind
          *provider*     ;; <- provider
          *migration-id* ;; <- the migration id
          "fubar"        ;; <- description
          "CREATE ...")  ;; <- contents
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1009570AD3}>

The downgrade migration is created in a similar way. Simply use the :down direction and the appropriate contents for it. Also, make sure that the description and id are the same when creating a pair of upgrade/downgrade migrations.

Lisp migrations

The :lisp migration kind allows invoking a handler, which performs the upgrade and downgrade.

The handler is a regular Lisp function, which accepts a single argument that is an instance of a driver.

In order to create a :lisp migration you need to create a spec, which specifies the Common Lisp system, package, upgrade and downgrade handler.

Here's an example of a spec, which is a regular property list.

(:system :my-system :package :my-system.package :handler :upgrade-handler)

And here's a full example of creating an :up and :down migration.

CL-USER> (let ((id (migratum:make-migration-id))
               (desc "my lisp migration")
               (up-spec '(:system :my-system
                          :package :my-system.package
                          :handler :upgrade-handler))
               (down-spec '(:system :my-system
                            :package :my-system.package
                            :handler :downgrade-handler)))
           (values
            (migratum:provider-create-migration :up :lisp *provider* id desc up-spec)
            (migratum:provider-create-migration :down :lisp *provider* id desc down-spec)))
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100A39C083}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100A39D6C3}>

Multiple SQL Statements

If you need to run multiple SQL statements when using the dbi or rdbms-postgresql driver you can separate each statement in the migration using the --;; separator.

The following example migration would create two tables as part of a single transaction.

CREATE TABLE foo (
    id INTEGER PRIMARY KEY
);
--;;
CREATE TABLE bar (
    id INTEGER PRIMARY KEY
);

Debug logging

cl-migratum uses log4cl, so you can enable debug logging, if needed.

CL-USER> (log:config :debug)

Shutdown / Teardown

Once done with the migrations you should call the cleanup functions of the provider and driver.

CL-USER> (migratum:provider-shutdown *provider*)
CL-USER> (migratum:driver-shutdown *driver*)

Implemeting new migration resources

Generally new migration resources will be implemented along with a provider, which discovers them.

You can implement a new migration resource by inheriting from the MIGRATUM:BASE-MIGRATION class.

The following generic functions should be implemented on the newly defined class.

| Method | Description | |---------------------------|-------------------------------------| | MIGRATUM:MIGRATION-LOAD | Loads the :up and :down scripts | | MIGRATUM:MIGRATION-KIND | Returns the kind of the migration |

The following generic functions can be overriden, if needed.

| Method | Description | |----------------------------------|-----------------------------------------------------| | MIGRATUM:MIGRATION-ID | Returns the unique migration id | | MIGRATUM:MIGRATION-DESCRIPTION | Returns description of the migration | | MIGRATUM:MIGRATION-APPLIED | Returns timestamp of when the migration was applied |

Example implementation that loads migration resources from a remote HTTP server might look like this. The following code uses Dexador as the HTTP client.

(defun http-get (url)
  "HTTP GETs the given URL"
  (let ((contents (dex:get url :force-string t)))
    contents))

(defclass http-migration (base-migration)
  ((up-script-url
    :initarg :up-script-url
    :initform (error "Must specify URL to upgrade script")
    :accessor http-migration-up-script-url
    :documentation "URL to the upgrade script")
   (down-script-url
    :initarg :down-script-url
    :initform (error "Must specify URL to downgrade script")
    :accessor http-migration-down-script-url
    :documentation "URL to the downgrade script"))
  (:documentation "HTTP migration resource"))

(defmethod migration-load ((direction (eql :up)) (migration http-migration))
  (http-get (http-migration-up-script-url migration)))

(defmethod migration-load ((direction (eql :down)) (migration http-migration))
  (http-get (http-migration-down-script-url migration)))

A provider should simply MAKE-INSTANCE of the HTTP-MIGRATION class by providing values for the required slots while discovering migrations via the PROVIDER-LIST-MIGRATIONS function.

Implementing new providers

You can implement custom providers, which can discover migration resources from various sources, e.g. local path, remote HTTP endpoints, etc.

Each provider determines the rules, which identify a resource as a valid migration, so custom logic for discovering them can be implemented by using your own provider. For example the local-path provider considers files to be valid migrations, if they match a given regex pattern.

In order to create a new provider you should inherit from the MIGRATUM:BASE-PROVIDER class and implement the following generic functions on your newly defined class.

| Method | Description | |--------------------------------------|--------------------------------------------| | MIGRATUM:PROVIDER-LIST-MIGRATIONS | Responsible for discovering migrations | | MIGRATUM:PROVIDER-CREATE-MIGRATION | Creates a new migration using the provider |

The following methods can be overriden, if needed.

| Method | Description | |---------------------------------|--------------------------------------------------------------| | MIGRATUM:PROVIDER-NAME | Returns a human-friendly name of the provider | | MIGRATUM:PROVIDER-INIT | Initializes the provider, if needed | | MIGRATUM:PROVIDER-INITIALIZED | Returns T if provider is initialized, NIL otherwise | | MIGRATUM:PROVIDER-SHUTDOWN | Shutdowns the provider and cleans up any allocated resources |

You can also check the local-path provider implementation for some example code.

Implementing new drivers

A driver is responsible for communicating with the database we are migrating and executes the upgrade and downgrade scripts.

The driver also takes care of registering applied migrations after applying an upgrade script and also unregistering them during downgrade, thus it is the drivers' decision how to implement registering and unregistering. For example the dbi builtin driver registers applied migrations in the same database it is migrating, but a custom driver could choose a different stategy instead, e.g. use a key/value store, local files, or some remote endpoint instead.

New drivers can be implemented by inheriting from the MIGRATUM:BASE-DRIVER class. The following methods should be implemented on new drivers.

| Method | Description | |--------------------------------------|--------------------------------------------------------| | MIGRATUM:DRIVER-LIST-APPLIED | Returns the list of applied migrations | | MIGRATUM:DRIVER-REGISTER-MIGRATION | Registers/unregister a successfully applied migration | | MIGRATUM:DRIVER-APPLY-MIGRATION | Applies a migration according to the given direction |

The following methods can be overriden, if needed.

| Method | Description | |-------------------------------|------------------------------------------------------------| | MIGRATUM:DRIVER-INIT | Initializes the driver, if needed | | MIGRATUM:DRIVER-NAME | Returns the human-friendly name of the driver | | MIGRATUM:DRIVER-PROVIDER | Returns the provider used by the driver | | MIGRATUM:DRIVER-INITIALIZED | Returns T if driver is initialized, NIL otherwise | | MIGRATUM:DRIVER-SHUTDOWN | Shutdowns the driver and cleans up any allocated resources |

You can check the dbi driver implementation for some example code.

Additional methods can also be overriden, if needed. For example if you need to have a different representation for the pending migrations you can override the MIGRATUM:DISPLAY-PENDING method.

Tests

Tests are provided as part of the cl-migratum.test system.

In order to run the tests you can execute.

make test

Or you can run the tests in a Docker container instead.

Contributing

cl-migratum is hosted on Github. Please contribute by reporting issues, suggesting features or by sending patches using pull requests.

Authors

License

This project is Open Source and licensed under the BSD License.


2 Systems

The main system appears first, followed by any subsystem dependency.


Previous: , Up: Systems   [Contents][Index]

2.1 cl-migratum

Database schema migration system for Common Lisp

Long Name

cl-migratum

Maintainer

Marin Atanasov Nikolov <dnaeon@gmail.com>

Author

Marin Atanasov Nikolov <dnaeon@gmail.com>

Home Page

https://github.com/dnaeon/cl-migratum

Source Control

https://github.com/dnaeon/cl-migratum

Bug Tracker

https://github.com/dnaeon/cl-migratum

License

BSD 2-Clause

Long Description

# Database Schema Migration System for Common Lisp

‘cl-migratum‘ is a Common Lisp system, which provides facilities for
performing [database schema
migrations](https://en.wikipedia.org/wiki/Schema_migration).

![migratum-demo](./images/migratum-demo.gif)

## Requirements

* [Quicklisp](https://www.quicklisp.org/beta/)

## Systems

The following systems are available as part of this repo.

| System | Description |
|—————————————|—————————————————–|
| ‘cl-migratum‘ | Core system |
| ‘cl-migratum.provider.local-path‘ | Provider which discovers migrations from local-path |
| ‘cl-migratum.driver.dbi‘ | ‘cl-dbi‘ database driver |
| ‘cl-migratum.driver.rmdbs-postgresql‘ | PostgreSQL driver based on ‘hu.dwim.rdbms‘ |
| ‘cl-migratum.driver.mixins‘ | Various mixin classes used by drivers |
| ‘cl-migratum.test‘ | Test suite for ‘cl-migratum‘ |
| ‘cl-migratum.cli‘ | CLI application of ‘migratum‘ |

## Installation

Latest development version can be installed from the Git repo.

Clone the [cl-migratum](https://github.com/dnaeon/cl-migratum) repo in your
[Quicklisp local-projects directory](https://www.quicklisp.org/beta/faq.html).

“‘ shell
git clone https://github.com/dnaeon/cl-migratum.git
“‘

## CLI

You can install the CLI application of ‘migratum‘ by executing the
following command.

“‘ shell
make cli
“‘

The ‘migratum‘ binary will be built into the ‘bin/migratum‘
directory. You can now install the binary somewhere in your ‘PATH‘.

Once the ‘migratum‘ binary is built you can also generate the Zsh
completions by executing the following command.

“‘ shell
migratum zsh-completions > ~/.zsh-completions/_migratum
“‘

Make sure that ‘~/.zsh-completions‘ is part of your Zsh ‘FPATH‘.

In order to generate the Markdown documentation of the CLI app use the
‘cli-doc‘ target:

“‘ shell
make cli-doc
“‘

Or you can build a Docker image instead.

“‘ shell
docker build -t migratum.cli:latest -f Dockerfile.cli .
“‘

## Concepts

The ‘cl-migratum‘ system uses the following concepts to describe
the process of discovering and applying schema migrations, so it is
important that you get familiar with them first.

### Migration

A ‘migration‘ represents a resource that provides information about a
schema change, e.g. it provides the unique id of the change, the
required scripts that can be used to upgrade and/or downgrade the
database.

Migration resources are discovered via ‘providers‘ and are being
used by ‘drivers‘ during the upgrade or downgrade process.

### Provider

The ‘provider‘ is responsible for discovering migration resources.

For example a provider can discover migrations from a local path by
scanning for files that match a given pattern or fetch migrations from
a remote endpoint (e.g. HTTP service).

The ‘provider‘ is also responsible for creating new migration
sequences.

The following providers are supported by ‘cl-migratum‘.

| Name | Description | System | |————–|——————————————————————|———————————–|
| ‘local-path‘ | Provider that can discover migration resources from a local path | ‘cl-migratum.provider.local-path‘ |

### Driver

The ‘driver‘ carries out the communication with the database against
which schema changes will be applied.

It is responsible for applying schema changes, registering
successfully applied migrations and unregistering them when reverting
back to a previous state.

A ‘driver‘ internally uses a ‘provider‘ in order to discover
‘migrations‘, which can be applied against the database it is
connected to.

The following drivers are currently supported by ‘cl-migratum‘.

| Name | Description | System | Migration Registration | |——————–|—————————————————————————————————————————|—————————————|————————| | ‘dbi‘ | Driver for performing schema migrations against a SQL database using [CL-DBI](https://github.com/fukamachi/cl-dbi) | ‘cl-migratum.driver.dbi‘ | In table ‘migration‘ | | ‘rdbms-postgresql‘ | Driver for performing schema migrations against a SQL database using [hu.dwim.rdbms](http://dwim.hu/darcs/hu.dwim.rdbms/) | ‘cl-migratum.driver.rdbms-postgresql‘ | In table ‘migration‘ |

## Usage

The following section contains some examples to get you started.

In order to use ‘cl-migratum‘ you will need to load the core system,
along with any ‘provider‘ and ‘driver‘ systems.

Load the core system.

“‘ shell
CL-USER> (ql:quickload :cl-migratum)
“‘

### Create Provider

First, we will create a new ‘provider‘ that can discover migration
files from a local path. In order to create a ‘local-path‘ provider we
need to load the respective system.

“‘ common-lisp
CL-USER> (ql:quickload :cl-migratum.provider.local-path)
“‘

Once you load the system we can create a ‘local-path‘ provider. The
‘local-path‘ provider can discover migrations from multiple paths.

Typical example where having multiple paths with migration resources
might be useful is when you have a set of base migrations you want to
apply to all environments, and then have a separate path for each
environment with their own migrations. For example you might want to
run specific migrations only in your ‘dev‘ environment, but don’t want
them in your ‘production‘ environment.

In that case it makes sense to separate the migration resources in
multiple paths, for each environment respectively.

“‘ common-lisp
CL-USER> (defparameter *provider*
(migratum.provider.local-path:make-provider (list #P"/path/to/cl-migratum/t/migrations/")))
*PROVIDER*
“‘

The ‘LOCAL-PATH-PROVIDER‘ discovers migration files which match the
following pattern.

| Pattern | Description |
|———————————-|——————|
| ‘<id>-<description>.up.<kind>‘ | Upgrade script |
| ‘<id>-<description>.down.<kind>‘ | Downgrade script |

In above pattern ‘<id>‘ is a monotonically increasing integer, which
represents the timestamp the migration has been created. The format of
‘<id>‘ is ‘YYYYMMDDHHMMSS‘.

The supported migration kinds by ‘LOCAL-PATH-PROVIDER‘ are these.

| Extension | Kind | Description |
|———–|———|————————————————–|
| ‘.sql‘ | ‘:sql‘ | SQL migration resource |
| ‘.lisp‘ | ‘:lisp‘ | Migration resource which invokes a Lisp function |

A provider can optionally be initialized, which can be done using the
‘MIGRATUM:PROVIDER-INIT‘ generic function. Not all providers would
require initialization, but some will and therefore it is good that
you always initialize them first.

In order to list the migrations provided by a ‘provider‘ you can use
the ‘MIGRATUM:PROVIDER-LIST-MIGRATIONS‘ function, e.g.

“‘ common-lisp
CL-USER> (migratum:provider-list-migrations *provider*)
(#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {1004713F73}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603BE3}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603B03}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {10046039B3}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1004603733}>)
“‘

The following generic functions can be used to interact with
discovered migrations.

| Method | Description |
|———————————-|—————————————————————|
| ‘MIGRATUM:MIGRATION-ID‘ | Returns the unique migration id |
| ‘MIGRATUM:MIGRATION-DESCRIPTION‘ | Returns the description of the migration |
| ‘MIGRATUM:MIGRATION-APPLIED‘ | Returns the timestamp of when the migration was applied |
| ‘MIGRATUM:MIGRATION-LOAD‘ | Loads the ‘:up‘ or ‘:down‘ migration |
| ‘MIGRATUM:MIGRATION-KIND‘ | Returns the kind of the migration, e.g. ‘:sql‘, ‘:lisp‘, etc. |

For example in order to collect the unique IDs of all migration
resources you can evaluate the following while at the REPL.

“‘ common-lisp
CL-USER> (mapcar #’migratum:migration-id
(migratum:provider-list-migrations *provider*))
(20200421180337 20200421173908 20200421173657 ...)
“‘

### Create Driver

Once we have a provider for discovering migration resources we need to
create a driver, which can be used to communicate with the database
we want to apply migrations on.

#### DBI Driver

Here is how we can create a ‘dbi‘ driver. First, lets load the system,
which provides that driver.

“‘ common-lisp
CL-USER> (ql:quickload :cl-migratum.driver.dbi)
“‘

The ‘dbi‘ driver uses [CL-DBI](https://github.com/fukamachi/cl-dbi)
interface to communicate with the database, so we will need to create
a database connection in order to initialize it.

“‘ common-lisp
CL-USER> (defparameter *conn*
(dbi:connect :sqlite3 :database-name "/path/to/migratum.db"))
*CONN*
“‘

And now we can instantiate our SQL driver using the previously created
provider and connection.

“‘ common-lisp
CL-USER> (defparameter *driver*
(migratum.driver.dbi:make-driver *provider* *conn*))
*DRIVER*
“‘

#### RDBMS PostgreSQL Driver

The process to create an ‘rdbms-postgresql‘ driver is analogous. The
driver needs a connection specification to create a DB connection. In
this example we use the same environment variables which are used by
the PostgreSQL client ‘psql‘ to override the defaults:

“‘ common-lisp
(ql:quickload :cl-migratum.driver.rdbms-postgresql)
(defparameter *driver*
(migratum.driver.rdbms-postgresql:make-driver
*provider*
‘(:host ,(or (uiop:getenv "PGHOST") "localhost")
:database ,(or (uiop:getenv "PGDATABASE") "migratum")
:user-name ,(or (uiop:getenv "PGUSER") "migratum")
:password ,(or (uiop:getenv "PGPASSWORD") "tiger"))))
“‘

### Initialization

A ‘driver‘ and ‘provider‘ may require some initialization steps to be
executed before being able to apply migrations, so make sure that you
initialize them.

In order to initialize your ‘provider‘ use the
‘MIGRATUM:PROVIDER-INIT‘ function.

“‘ common-lisp
CL-USER> (migratum:provider-init *provider*)
“‘

An example requirement for a driver might be to create some required
database schema used to track which migrations have been applied
already, so lets initialize our driver first.

“‘ common-lisp
CL-USER> (migratum:driver-init *driver*)
“‘

### Pending Migrations

In order to get the list of pending (not applied yet) migrations we can
use the ‘MIGRATUM:LIST-PENDING‘ function, e.g.

“‘ common-lisp
CL-USER> (migratum:list-pending *driver*)
(#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363DFC3}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E0A3}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E183}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {100363E263}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100363E343}>)
“‘

Or, we can display a table of the pending migrations using the
‘MIGRATUM:DISPLAY-PENDING‘ function instead.

“‘ common-lisp
CL-USER> (migratum:display-pending *driver*)
.———————————————.
| PENDING MIGRATIONS |
+—————-+———————+——+
| ID | DESCRIPTION | KIND |
+—————-+———————+——+
| 20200421173657 | create_table_foo | SQL |
| 20200421173908 | create_table_bar | SQL |
| 20200421180337 | create_table_qux | SQL |
| 20200605144633 | multiple_statements | SQL |
| 20220327224455 | lisp_code_migration | LISP |
+—————-+———————+——+
| | TOTAL | 5 |
+—————-+———————+——+
NIL
“‘

The migrations will be sorted in the order they need to be applied.

### Applying Migrations

The following functions are used for applying pending migrations.

* ‘MIGRATUM:APPLY-PENDING‘ - applies all pending migrations
* ‘MIGRATUM:APPLY-NEXT‘ - applies the next pending migration

This is how we can apply all pending migrations for example.

“‘ common-lisp
CL-USER> (migratum:apply-pending *driver*)
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-pending base-driver) -
Found 5 pending migration(s) to be applied
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421173657 - create_table_foo (SQL)
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421173908 - create_table_bar (SQL)
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421180337 - create_table_qux (SQL)
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200605144633 - multiple_statements (SQL)
<INFO> [21:20:51] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20220327224455 - lisp_code_migration (LISP)
NIL
“‘

### Get Latest Migration

You can use the ‘MIGRATUM:LATEST-MIGRATION‘ function to get the latest
applied migration, e.g.

“‘ common-lisp
CL-USER> (migratum:migration-id (migratum:latest-migration *driver*))
20220327224455 (45 bits, #x1263E96BE487)
“‘

The ‘MIGRATUM:CONTAINS-APPLIED-MIGRATIONS-P‘ predicate can be used to
query whether there are any migrations applied, e.g.

“‘ common-lisp
CL-USER> (migratum:contains-applied-migrations-p *driver*)
T
“‘

### Displaying Applied Migrations

The following functions can be used to get and display the
list of applied database migrations.

| Function | Description |
|——————————–|—————————————-|
| ‘MIGRATUM:DRIVER-LIST-APPLIED‘ | Returns the list of applied migrations |
| ‘MIGRATUM:DISPLAY-APPLIED‘ | Display a table of applied migrations |

This is how we can get the list of applied migrations.

“‘ common-lisp
CL-USER> (migratum:driver-list-applied *driver*)
(#<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175993}>
#<CL-MIGRATUM.CORE:BASE-MIGRATION {10021759D3}>
#<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A13}>
#<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A53}>
#<CL-MIGRATUM.CORE:BASE-MIGRATION {1002175A93}>)
“‘

Or we can display a nice table of the applied migrations instead.

“‘ common-lisp
CL-USER> (migratum:display-applied *driver*)
.——————————————————————-.
| APPLIED MIGRATIONS |
+—————-+———————+———————+——+
| ID | DESCRIPTION | APPLIED | KIND |
+—————-+———————+———————+——+
| 20220327224455 | lisp_code_migration | 2022-03-29 18:20:53 | LISP |
| 20200605144633 | multiple_statements | 2022-03-29 18:20:51 | SQL |
| 20200421180337 | create_table_qux | 2022-03-29 18:20:51 | SQL |
| 20200421173908 | create_table_bar | 2022-03-29 18:20:51 | SQL |
| 20200421173657 | create_table_foo | 2022-03-29 18:20:51 | SQL |
+—————-+———————+———————+——+
| | | TOTAL | 5 |
+—————-+———————+———————+——+
NIL
“‘

The output of these functions will be the applied migrations in
descending order by their id, first one being the most recent one.

The ‘dbi‘ and ‘rdbms-postgresql‘ drivers by default will fetch the
last 100 applied migrations. You can control this behaviour by using
the ‘:offset‘ and ‘:limit‘ keyword parameters, which allows you to
fetch applied migrations in pages.

For example, if you are interested only in the last ten applied
migrations you can evaluate the following expression.

“‘ common-lisp
CL-USER> (migratum:display-applied *driver* :limit 10)
“‘

Or if you want to skip the first ten migrations, you can evaluate
this expression instead.

“‘ common-lisp
CL-USER> (migratum:display-applied *driver* :offset 10 :limit 10)
“‘

### Stepping through migrations

Using the following functions you can step through migrations.

| Function | Description |
|————————|————————————–|
| ‘MIGRATUM:APPLY-NEXT‘ | Apply the next pending migration(s) |
| ‘MIGRATUM:REVERT-LAST‘ | Revert the last applied migration(s) |

This is useful in situations when you don’t want to apply
all migrations at once, but rather do it one at a time. Both of these
functions accept a keyword parameter ‘:count‘ which specifies the
number of migrations to apply or revert.

Consider the following pending migrations.

“‘ common-lisp
CL-USER> (migratum:display-pending *driver*)
.———————————————.
| PENDING MIGRATIONS |
+—————-+———————+——+
| ID | DESCRIPTION | KIND |
+—————-+———————+——+
| 20200421173657 | create_table_foo | SQL |
| 20200421173908 | create_table_bar | SQL |
| 20200421180337 | create_table_qux | SQL |
| 20200605144633 | multiple_statements | SQL |
| 20220327224455 | lisp_code_migration | LISP |
+—————-+———————+——+
| | TOTAL | 5 |
+—————-+———————+——+
NIL
“‘

We can apply them one by one and verify them as we go.

“‘ common-lisp
CL-USER> (migratum:apply-next *driver*)
<INFO> [21:25:45] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421173657 - create_table_foo (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
<INFO> [21:25:47] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421173908 - create_table_bar (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
<INFO> [21:25:48] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200421180337 - create_table_qux (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
<INFO> [21:25:49] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20200605144633 - multiple_statements (SQL)
NIL
CL-USER> (migratum:apply-next *driver*)
<INFO> [21:25:50] cl-migratum.core core.lisp (apply-and-register base-driver) -
Applying migration 20220327224455 - lisp_code_migration (LISP)
NIL
“‘

If we want to revert the last three migrations we can use the
‘MIGRATUM:REVERT-LAST‘ function, e.g.

“‘ common-lisp
CL-USER> (migratum:revert-last *driver* :count 3)
<INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
Reverting migration 20220327224455 - lisp_code_migration (LISP)
<INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
Reverting migration 20200605144633 - multiple_statements (SQL)
<INFO> [21:26:14] cl-migratum.core core.lisp (revert-and-unregister base-driver) -
Reverting migration 20200421180337 - create_table_qux (SQL)
NIL
“‘

Now, we should have 3 pending migrations again.

“‘ common-lisp
CL-USER> (migratum:display-pending *driver*)
.———————————————.
| PENDING MIGRATIONS |
+—————-+———————+——+
| ID | DESCRIPTION | KIND |
+—————-+———————+——+
| 20200421180337 | create_table_qux | SQL |
| 20200605144633 | multiple_statements | SQL |
| 20220327224455 | lisp_code_migration | LISP |
+—————-+———————+——+
| | TOTAL | 3 |
+—————-+———————+——+
NIL
“‘

### Creating New Migrations

The ‘MIGRATUM:PROVIDER-CREATE-MIGRATION‘ generic function creates a
new migration sequence. The arguments it expects are the ‘direction‘
(e.g. ‘:up‘ or ‘:down‘), the migration ‘kind‘ (e.g. ‘:sql‘, ‘:lisp‘,
etc.), an instance of a ‘provider‘, ‘id‘, ‘description‘ and ‘content‘.

Knowing ‘migratum:provider-create-migration‘ by the documentation
below, you can also use the wrapper
‘migratum.provider.local-path:touch-migration‘ to create a pair of
up-and-down empty migrations with a ‘LOCAL-PATH‘ provider.

#### SQL migrations

Here’s an example of using the ‘LOCAL-PATH‘ provider for creating a
new SQL migration.

“‘ common-lisp
CL-USER> (defparameter *migration-id* (migratum:make-migration-id))
*MIGRATION-ID*
CL-USER>
CL-USER> (migratum:provider-create-migration
:up ;; <- direction
:sql ;; <- kind
*provider* ;; <- provider
*migration-id* ;; <- the migration id
"fubar" ;; <- description
"CREATE ...") ;; <- contents
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:SQL-MIGRATION {1009570AD3}>
“‘

The downgrade migration is created in a similar way. Simply use the
‘:down‘ direction and the appropriate contents for it. Also, make sure
that the description and id are the same when creating a pair of
upgrade/downgrade migrations.

#### Lisp migrations

The ‘:lisp‘ migration kind allows invoking a handler, which performs
the upgrade and downgrade.

The handler is a regular Lisp function, which accepts a single
argument that is an instance of a ‘driver‘.

In order to create a ‘:lisp‘ migration you need to create a spec,
which specifies the Common Lisp ‘system‘, ‘package‘, ‘upgrade‘ and
‘downgrade‘ handler.

Here’s an example of a spec, which is a regular property list.

“‘ common-lisp
(:system :my-system :package :my-system.package :handler :upgrade-handler)
“‘

And here’s a full example of creating an ‘:up‘ and ‘:down‘ migration.

“‘ common-lisp
CL-USER> (let ((id (migratum:make-migration-id))
(desc "my lisp migration")
(up-spec ’(:system :my-system
:package :my-system.package
:handler :upgrade-handler))
(down-spec ’(:system :my-system
:package :my-system.package
:handler :downgrade-handler)))
(values
(migratum:provider-create-migration :up :lisp *provider* id desc up-spec)
(migratum:provider-create-migration :down :lisp *provider* id desc down-spec)))
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100A39C083}>
#<CL-MIGRATUM.PROVIDER.LOCAL-PATH:LISP-MIGRATION {100A39D6C3}>
“‘

### Multiple SQL Statements

If you need to run multiple SQL statements when using the ‘dbi‘ or
‘rdbms-postgresql‘ driver you can separate each statement in the
migration using the ‘–;;‘ separator.

The following example migration would create two tables as part of a
single transaction.

“‘ sql
CREATE TABLE foo (
id INTEGER PRIMARY KEY
);
–;;
CREATE TABLE bar (
id INTEGER PRIMARY KEY
);
“‘

### Debug logging

‘cl-migratum‘ uses [log4cl](https://github.com/sharplispers/log4cl),
so you can enable debug logging, if needed.

“‘ common-lisp
CL-USER> (log:config :debug)
“‘

### Shutdown / Teardown

Once done with the migrations you should call the cleanup functions of
the ‘provider‘ and ‘driver‘.

“‘ common-lisp
CL-USER> (migratum:provider-shutdown *provider*)
CL-USER> (migratum:driver-shutdown *driver*)
“‘

## Implemeting new migration resources

Generally new migration resources will be implemented along with a
‘provider‘, which discovers them.

You can implement a new migration resource by inheriting from the
‘MIGRATUM:BASE-MIGRATION‘ class.

The following generic functions should be implemented on the newly
defined class.

| Method | Description |
|—————————|————————————-|
| ‘MIGRATUM:MIGRATION-LOAD‘ | Loads the ‘:up‘ and ‘:down‘ scripts |
| ‘MIGRATUM:MIGRATION-KIND‘ | Returns the kind of the migration |

The following generic functions can be overriden, if needed.

| Method | Description |
|———————————-|—————————————————–|
| ‘MIGRATUM:MIGRATION-ID‘ | Returns the unique migration id |
| ‘MIGRATUM:MIGRATION-DESCRIPTION‘ | Returns description of the migration |
| ‘MIGRATUM:MIGRATION-APPLIED‘ | Returns timestamp of when the migration was applied |

Example implementation that loads migration resources from a remote
HTTP server might look like this. The following code uses
[Dexador](https://github.com/fukamachi/dexador) as the HTTP client.

“‘ common-lisp
(defun http-get (url)
"HTTP GETs the given URL"
(let ((contents (dex:get url :force-string t)))
contents))

(defclass http-migration (base-migration)
((up-script-url
:initarg :up-script-url
:initform (error "Must specify URL to upgrade script")
:accessor http-migration-up-script-url
:documentation "URL to the upgrade script")
(down-script-url
:initarg :down-script-url
:initform (error "Must specify URL to downgrade script")
:accessor http-migration-down-script-url
:documentation "URL to the downgrade script"))
(:documentation "HTTP migration resource"))

(defmethod migration-load ((direction (eql :up)) (migration http-migration))
(http-get (http-migration-up-script-url migration)))

(defmethod migration-load ((direction (eql :down)) (migration http-migration))
(http-get (http-migration-down-script-url migration)))
“‘

A provider should simply ‘MAKE-INSTANCE‘ of the ‘HTTP-MIGRATION‘ class by
providing values for the required slots while discovering migrations via
the ‘PROVIDER-LIST-MIGRATIONS‘ function.

## Implementing new providers

You can implement custom ‘providers‘, which can discover migration
resources from various sources, e.g. local path, remote HTTP endpoints,
etc.

Each ‘provider‘ determines the rules, which identify a resource as a
valid migration, so custom logic for discovering them can be
implemented by using your own provider. For example the ‘local-path‘
provider considers files to be valid migrations, if they match a given
regex pattern.

In order to create a new provider you should inherit from the
‘MIGRATUM:BASE-PROVIDER‘ class and implement the following generic
functions on your newly defined class.

| Method | Description |
|————————————–|——————————————–|
| ‘MIGRATUM:PROVIDER-LIST-MIGRATIONS‘ | Responsible for discovering migrations |
| ‘MIGRATUM:PROVIDER-CREATE-MIGRATION‘ | Creates a new migration using the provider |

The following methods can be overriden, if needed.

| Method | Description |
|———————————|————————————————————–|
| ‘MIGRATUM:PROVIDER-NAME‘ | Returns a human-friendly name of the provider |
| ‘MIGRATUM:PROVIDER-INIT‘ | Initializes the provider, if needed |
| ‘MIGRATUM:PROVIDER-INITIALIZED‘ | Returns ‘T‘ if provider is initialized, ‘NIL‘ otherwise |
| ‘MIGRATUM:PROVIDER-SHUTDOWN‘ | Shutdowns the provider and cleans up any allocated resources |

You can also check the [local-path
provider](./src/provider/local-path.lisp) implementation for some
example code.

## Implementing new drivers

A ‘driver‘ is responsible for communicating with the database we are
migrating and executes the upgrade and downgrade scripts.

The driver also takes care of registering applied migrations after
applying an upgrade script and also unregistering them during
downgrade, thus it is the drivers’ decision how to implement
registering and unregistering. For example the ‘dbi‘ builtin driver
registers applied migrations in the same database it is migrating, but
a custom driver could choose a different stategy instead, e.g. use a
key/value store, local files, or some remote endpoint instead.

New drivers can be implemented by inheriting from the
‘MIGRATUM:BASE-DRIVER‘ class. The following methods should be
implemented on new drivers.

| Method | Description |
|————————————–|——————————————————–|
| ‘MIGRATUM:DRIVER-LIST-APPLIED‘ | Returns the list of applied migrations |
| ‘MIGRATUM:DRIVER-REGISTER-MIGRATION‘ | Registers/unregister a successfully applied migration |
| ‘MIGRATUM:DRIVER-APPLY-MIGRATION‘ | Applies a migration according to the given ‘direction‘ |

The following methods can be overriden, if needed.

| Method | Description |
|——————————-|————————————————————|
| ‘MIGRATUM:DRIVER-INIT‘ | Initializes the driver, if needed |
| ‘MIGRATUM:DRIVER-NAME‘ | Returns the human-friendly name of the driver |
| ‘MIGRATUM:DRIVER-PROVIDER‘ | Returns the ‘provider‘ used by the ‘driver‘ |
| ‘MIGRATUM:DRIVER-INITIALIZED‘ | Returns ‘T‘ if driver is initialized, ‘NIL‘ otherwise |
| ‘MIGRATUM:DRIVER-SHUTDOWN‘ | Shutdowns the driver and cleans up any allocated resources |

You can check the [dbi driver](./src/driver/dbi.lisp) implementation
for some example code.

Additional methods can also be overriden, if needed. For example if
you need to have a different representation for the pending migrations
you can override the ‘MIGRATUM:DISPLAY-PENDING‘ method.

## Tests

Tests are provided as part of the ‘cl-migratum.test‘ system.

In order to run the tests you can execute.

“‘ common-lisp
make test
“‘

Or you can run the tests in a Docker container instead.

## Contributing

‘cl-migratum‘ is hosted on
[Github](https://github.com/dnaeon/cl-migratum). Please contribute by
reporting issues, suggesting features or by sending patches using pull
requests.

## Authors

* Marin Atanasov Nikolov (dnaeon@gmail.com)

## License

This project is Open Source and licensed under the [BSD
License](http://opensource.org/licenses/BSD-2-Clause).

Version

0.6.0

Dependencies
  • local-time (system).
  • cl-ascii-table (system).
  • cl-reexport (system).
  • log4cl (system).
Source

cl-migratum.asd.

Child Components

3 Modules

Modules are listed depth-first from the system components tree.


Next: , Previous: , Up: Modules   [Contents][Index]

3.1 cl-migratum/util

Source

cl-migratum.asd.

Parent Component

cl-migratum (system).

Child Component

util.lisp (file).


3.2 cl-migratum/core

Dependency

util (module).

Source

cl-migratum.asd.

Parent Component

cl-migratum (system).

Child Component

core.lisp (file).


Previous: , Up: Modules   [Contents][Index]

3.3 cl-migratum/client-package

Dependencies
Source

cl-migratum.asd.

Parent Component

cl-migratum (system).

Child Component

package.lisp (file).


4 Files

Files are sorted by type and then listed depth-first from the systems components trees.


Previous: , Up: Files   [Contents][Index]

4.1 Lisp


Next: , Previous: , Up: Lisp   [Contents][Index]

4.1.1 cl-migratum/cl-migratum.asd

Source

cl-migratum.asd.

Parent Component

cl-migratum (system).

ASDF Systems

cl-migratum.

Packages

cl-migratum-system.


4.1.2 cl-migratum/util/util.lisp

Source

cl-migratum.asd.

Parent Component

util (module).

Packages

cl-migratum.util.

Public Interface

4.1.3 cl-migratum/core/core.lisp

Source

cl-migratum.asd.

Parent Component

core (module).

Packages

cl-migratum.core.

Public Interface
Internals

4.1.4 cl-migratum/client-package/package.lisp

Source

cl-migratum.asd.

Parent Component

client-package (module).

Packages

cl-migratum.


5 Packages

Packages are listed by definition order.


Next: , Previous: , Up: Packages   [Contents][Index]

5.1 cl-migratum.core

Source

core.lisp.

Nickname

migratum.core

Use List

common-lisp.

Public Interface
Internals

Next: , Previous: , Up: Packages   [Contents][Index]

5.2 cl-migratum-system

Source

cl-migratum.asd.

Use List
  • asdf/interface.
  • common-lisp.

5.3 cl-migratum

Source

package.lisp.

Nickname

migratum

Use List

common-lisp.


Previous: , Up: Packages   [Contents][Index]

5.4 cl-migratum.util

Source

util.lisp.

Nickname

migratum.util

Use List

common-lisp.

Public Interface

6 Definitions

Definitions are sorted by export status, category, package, and then by lexicographic order.


Next: , Previous: , Up: Definitions   [Contents][Index]

6.1 Public Interface


6.1.1 Ordinary functions

Function: make-migration-id (&optional timezone)

Creates a new migration id from current timestamp

Package

cl-migratum.util.

Source

util.lisp.

Function: take (n sequence)

Take N items from the given sequence

Package

cl-migratum.util.

Source

util.lisp.


6.1.2 Generic functions

Generic Function: apply-next (driver &key count)

Apply the next COUNT of pending migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: apply-next ((driver base-driver) &key count)
Generic Function: apply-pending (driver)

Applies all pending migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: apply-pending ((driver base-driver))
Generic Function: contains-applied-migrations-p (driver)

Predicate for testing whether there are any applied migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: contains-applied-migrations-p ((driver base-driver))
Generic Function: display-applied (driver &key offset limit)

Displays the applied migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: display-applied ((driver base-driver) &key offset limit)
Generic Function: display-pending (driver)

Displays the pending migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: display-pending ((driver base-driver))
Generic Function: driver-apply-migration (direction kind driver migration &key)

Applies the migration script using the given driver and direction

Package

cl-migratum.core.

Source

core.lisp.

Generic Function: driver-init (driver &key)

Initializes the driver, e.g. creates required schema

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: driver-init ((driver base-driver) &key)
Generic Reader: driver-initialized (object)
Generic Writer: (setf driver-initialized) (object)
Package

cl-migratum.core.

Methods
Reader Method: driver-initialized ((base-driver base-driver))
Writer Method: (setf driver-initialized) ((base-driver base-driver))

Returns T if driver is initialized, NIL otherwise

Source

core.lisp.

Target Slot

initialized.

Generic Function: driver-list-applied (driver &key offset limit)

Returns a list of the applied migrations in descending order

Package

cl-migratum.core.

Source

core.lisp.

Generic Reader: driver-name (object)
Package

cl-migratum.core.

Methods
Reader Method: driver-name ((base-driver base-driver))

Name of the driver

Source

core.lisp.

Target Slot

name.

Generic Reader: driver-provider (object)
Package

cl-migratum.core.

Methods
Reader Method: driver-provider ((base-driver base-driver))

Provider used by the driver

Source

core.lisp.

Target Slot

provider.

Generic Function: driver-register-migration (direction driver migration &key)

Register or unregister a migration depending on the given direction

Package

cl-migratum.core.

Source

core.lisp.

Generic Function: driver-shutdown (driver &key)

Shutdowns the driver and cleans up any allocated resources

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: driver-shutdown ((driver base-driver) &key)
Generic Function: find-migration-by-id (provider id)

Returns the migration with the given id from the provider

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: find-migration-by-id ((provider base-provider) id)
Generic Function: latest-migration (driver)

Returns the latest applied migration

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: latest-migration ((driver base-driver))
Generic Function: list-pending (driver)

Returns the list of migrations that have not been applied yet

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: list-pending ((driver base-driver))
Generic Reader: migration-applied (migration)

Returns a timestamp describing when the migration was applied or NIL if not applied

Package

cl-migratum.core.

Source

core.lisp.

Methods
Reader Method: migration-applied ((base-migration base-migration))

Timestamp when the migration was applied or NIL if it is not applied

Target Slot

applied.

Generic Reader: migration-description (migration)

Returns a string describing the migration resource

Package

cl-migratum.core.

Source

core.lisp.

Methods
Reader Method: migration-description ((base-migration base-migration))

Description of the migration

Target Slot

description.

Generic Reader: migration-id (migration)

Returns the ID of the migration. The ID must be a positive integer

Package

cl-migratum.core.

Source

core.lisp.

Methods
Reader Method: migration-id ((base-migration base-migration))

Unique migration id

Target Slot

id.

Generic Reader: migration-kind (migration)

Returns the kind of the migration resource as a keyword, e.g. :sql, :lisp, etc

Package

cl-migratum.core.

Source

core.lisp.

Methods
Reader Method: migration-kind ((base-migration base-migration))

A keyword describing the migration kind

Target Slot

kind.

Generic Function: migration-load (direction migration)

Loads the given migration. Direction is :UP or
:DOWN which specifies whether to load the upgrade or downgrade migration respectively.

Package

cl-migratum.core.

Source

core.lisp.

Generic Function: provider-create-migration (direction kind provider id description content &key)

Creates a new migration resource using the given provider

Package

cl-migratum.core.

Source

core.lisp.

Generic Function: provider-init (provider &key)

Initializes the driver, if needed

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: provider-init ((provider base-provider) &key)
Generic Reader: provider-initialized (object)
Generic Writer: (setf provider-initialized) (object)
Package

cl-migratum.core.

Methods
Reader Method: provider-initialized ((base-provider base-provider))
Writer Method: (setf provider-initialized) ((base-provider base-provider))

Returns T if provider is initialized, NIL otherwise

Source

core.lisp.

Target Slot

initialized.

Generic Function: provider-list-migrations (provider &key)

Returns the list of migration resources discovered by the provider

Package

cl-migratum.core.

Source

core.lisp.

Generic Reader: provider-name (object)
Package

cl-migratum.core.

Methods
Reader Method: provider-name ((base-provider base-provider))

Name of the provider

Source

core.lisp.

Target Slot

name.

Generic Function: provider-shutdown (provider &key)

Shutdowns the provider and cleans up any allocated resources

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: provider-shutdown ((provider base-provider) &key)
Generic Function: reset (driver)

Resets the database by reverting all migrations and re-applying them again

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: reset ((driver base-driver))
Generic Function: revert-last (driver &key count)

Reverts the last COUNT applied migrations

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: revert-last ((driver base-driver) &key count)

6.1.3 Classes

Class: base-driver

Base class for migration drivers

Package

cl-migratum.core.

Source

core.lisp.

Direct methods
Direct slots
Slot: name

Name of the driver

Type

string

Initform

(error "must specify driver name")

Initargs

:name

Readers

driver-name.

Writers

This slot is read-only.

Slot: provider

Provider used by the driver

Initform

(error "must specify migrations provider")

Initargs

:provider

Readers

driver-provider.

Writers

This slot is read-only.

Slot: initialized

Returns T if driver is initialized, NIL otherwise

Initargs

:initialized

Readers

driver-initialized.

Writers

(setf driver-initialized).

Class: base-migration

Base class for migration resources

Package

cl-migratum.core.

Source

core.lisp.

Direct methods
Direct slots
Slot: id

Unique migration id

Type

integer

Initform

(error "must specify migration id")

Initargs

:id

Readers

migration-id.

Writers

This slot is read-only.

Slot: description

Description of the migration

Type

string

Initform

(error "must specify migration description")

Initargs

:description

Readers

migration-description.

Writers

This slot is read-only.

Slot: applied

Timestamp when the migration was applied or NIL if it is not applied

Initargs

:applied

Readers

migration-applied.

Writers

This slot is read-only.

Slot: kind

A keyword describing the migration kind

Initform

(error "must specify migration kind")

Initargs

:kind

Readers

migration-kind.

Writers

This slot is read-only.

Class: base-provider

Base class for migration providers

Package

cl-migratum.core.

Source

core.lisp.

Direct methods
Direct slots
Slot: name

Name of the provider

Type

string

Initform

(error "must specify provider name")

Initargs

:name

Readers

provider-name.

Writers

This slot is read-only.

Slot: initialized

Returns T if provider is initialized, NIL otherwise

Initargs

:initialized

Readers

provider-initialized.

Writers

(setf provider-initialized).


6.2 Internals


Previous: , Up: Internals   [Contents][Index]

6.2.1 Generic functions

Generic Function: apply-and-register (driver migration)

Applies the migration and registers it

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: apply-and-register ((driver base-driver) migration)
Generic Function: revert-and-unregister (driver migration)

Reverts and unregisters a given migration.

Package

cl-migratum.core.

Source

core.lisp.

Methods
Method: revert-and-unregister ((driver base-driver) migration)

The migration to be reverted is first loaded via the driver provider, in order to ensure we can load the downgrade script.


Appendix A Indexes


Next: , Previous: , Up: Indexes   [Contents][Index]

A.1 Concepts


Next: , Previous: , Up: Indexes   [Contents][Index]

A.2 Functions

Jump to:   (  
A   C   D   F   G   L   M   P   R   T  
Index Entry  Section

(
(setf driver-initialized): Public generic functions
(setf driver-initialized): Public generic functions
(setf provider-initialized): Public generic functions
(setf provider-initialized): Public generic functions

A
apply-and-register: Private generic functions
apply-and-register: Private generic functions
apply-next: Public generic functions
apply-next: Public generic functions
apply-pending: Public generic functions
apply-pending: Public generic functions

C
contains-applied-migrations-p: Public generic functions
contains-applied-migrations-p: Public generic functions

D
display-applied: Public generic functions
display-applied: Public generic functions
display-pending: Public generic functions
display-pending: Public generic functions
driver-apply-migration: Public generic functions
driver-init: Public generic functions
driver-init: Public generic functions
driver-initialized: Public generic functions
driver-initialized: Public generic functions
driver-list-applied: Public generic functions
driver-name: Public generic functions
driver-name: Public generic functions
driver-provider: Public generic functions
driver-provider: Public generic functions
driver-register-migration: Public generic functions
driver-shutdown: Public generic functions
driver-shutdown: Public generic functions

F
find-migration-by-id: Public generic functions
find-migration-by-id: Public generic functions
Function, make-migration-id: Public ordinary functions
Function, take: Public ordinary functions

G
Generic Function, (setf driver-initialized): Public generic functions
Generic Function, (setf provider-initialized): Public generic functions
Generic Function, apply-and-register: Private generic functions
Generic Function, apply-next: Public generic functions
Generic Function, apply-pending: Public generic functions
Generic Function, contains-applied-migrations-p: Public generic functions
Generic Function, display-applied: Public generic functions
Generic Function, display-pending: Public generic functions
Generic Function, driver-apply-migration: Public generic functions
Generic Function, driver-init: Public generic functions
Generic Function, driver-initialized: Public generic functions
Generic Function, driver-list-applied: Public generic functions
Generic Function, driver-name: Public generic functions
Generic Function, driver-provider: Public generic functions
Generic Function, driver-register-migration: Public generic functions
Generic Function, driver-shutdown: Public generic functions
Generic Function, find-migration-by-id: Public generic functions
Generic Function, latest-migration: Public generic functions
Generic Function, list-pending: Public generic functions
Generic Function, migration-applied: Public generic functions
Generic Function, migration-description: Public generic functions
Generic Function, migration-id: Public generic functions
Generic Function, migration-kind: Public generic functions
Generic Function, migration-load: Public generic functions
Generic Function, provider-create-migration: Public generic functions
Generic Function, provider-init: Public generic functions
Generic Function, provider-initialized: Public generic functions
Generic Function, provider-list-migrations: Public generic functions
Generic Function, provider-name: Public generic functions
Generic Function, provider-shutdown: Public generic functions
Generic Function, reset: Public generic functions
Generic Function, revert-and-unregister: Private generic functions
Generic Function, revert-last: Public generic functions

L
latest-migration: Public generic functions
latest-migration: Public generic functions
list-pending: Public generic functions
list-pending: Public generic functions

M
make-migration-id: Public ordinary functions
Method, (setf driver-initialized): Public generic functions
Method, (setf provider-initialized): Public generic functions
Method, apply-and-register: Private generic functions
Method, apply-next: Public generic functions
Method, apply-pending: Public generic functions
Method, contains-applied-migrations-p: Public generic functions
Method, display-applied: Public generic functions
Method, display-pending: Public generic functions
Method, driver-init: Public generic functions
Method, driver-initialized: Public generic functions
Method, driver-name: Public generic functions
Method, driver-provider: Public generic functions
Method, driver-shutdown: Public generic functions
Method, find-migration-by-id: Public generic functions
Method, latest-migration: Public generic functions
Method, list-pending: Public generic functions
Method, migration-applied: Public generic functions
Method, migration-description: Public generic functions
Method, migration-id: Public generic functions
Method, migration-kind: Public generic functions
Method, provider-init: Public generic functions
Method, provider-initialized: Public generic functions
Method, provider-name: Public generic functions
Method, provider-shutdown: Public generic functions
Method, reset: Public generic functions
Method, revert-and-unregister: Private generic functions
Method, revert-last: Public generic functions
migration-applied: Public generic functions
migration-applied: Public generic functions
migration-description: Public generic functions
migration-description: Public generic functions
migration-id: Public generic functions
migration-id: Public generic functions
migration-kind: Public generic functions
migration-kind: Public generic functions
migration-load: Public generic functions

P
provider-create-migration: Public generic functions
provider-init: Public generic functions
provider-init: Public generic functions
provider-initialized: Public generic functions
provider-initialized: Public generic functions
provider-list-migrations: Public generic functions
provider-name: Public generic functions
provider-name: Public generic functions
provider-shutdown: Public generic functions
provider-shutdown: Public generic functions

R
reset: Public generic functions
reset: Public generic functions
revert-and-unregister: Private generic functions
revert-and-unregister: Private generic functions
revert-last: Public generic functions
revert-last: Public generic functions

T
take: Public ordinary functions

Jump to:   (  
A   C   D   F   G   L   M   P   R   T  

Previous: , Up: Indexes   [Contents][Index]

A.4 Data types

Jump to:   B   C   F   M   P   S   U  
Index Entry  Section

B
base-driver: Public classes
base-migration: Public classes
base-provider: Public classes

C
cl-migratum: The cl-migratum system
cl-migratum: The cl-migratum package
cl-migratum-system: The cl-migratum-system package
cl-migratum.asd: The cl-migratum/cl-migratum․asd file
cl-migratum.core: The cl-migratum․core package
cl-migratum.util: The cl-migratum․util package
Class, base-driver: Public classes
Class, base-migration: Public classes
Class, base-provider: Public classes
client-package: The cl-migratum/client-package module
core: The cl-migratum/core module
core.lisp: The cl-migratum/core/core․lisp file

F
File, cl-migratum.asd: The cl-migratum/cl-migratum․asd file
File, core.lisp: The cl-migratum/core/core․lisp file
File, package.lisp: The cl-migratum/client-package/package․lisp file
File, util.lisp: The cl-migratum/util/util․lisp file

M
Module, client-package: The cl-migratum/client-package module
Module, core: The cl-migratum/core module
Module, util: The cl-migratum/util module

P
Package, cl-migratum: The cl-migratum package
Package, cl-migratum-system: The cl-migratum-system package
Package, cl-migratum.core: The cl-migratum․core package
Package, cl-migratum.util: The cl-migratum․util package
package.lisp: The cl-migratum/client-package/package․lisp file

S
System, cl-migratum: The cl-migratum system

U
util: The cl-migratum/util module
util.lisp: The cl-migratum/util/util․lisp file

Jump to:   B   C   F   M   P   S   U