How to migrate dbunit data sets?

I often use DBUnit when it comes to integration tests related to the database. DBUnit is a lovely tool to setup a database for integration tests, but it also has a downside: “As more tests you write, more files need to be changed in case of a database schema change“. Wouldn’t it be nice if there would be a tool support, a tool that just scans your project, collects all the data sets it finds and migrates them using your Flyway scripts?

Background

The best for tests is that you have one data set for one test and maybe even data sets for the database state assertions, but this would increase the files you have to mange. So a lot of projects try to minimize the data set files they have to edit less files when it comes to database schema changes, but this couples tests. It will become harder to add new data to one file that serves a lot of tests, because you can easily break other tests.

So I thought about database evolution and the tools we use there. One tool that a lot of developers use is Flyway. But wait when we use Flyway to migrate our data from one database version to another, why can’t we use Flywayto migrate DBUnit data sets that are only a desription of data. If we could use Flyway to migrate DBUnit data sets, we could also test that our Flyway scripts work well. If we could automate this process, our flyway migration tests could run on a CI servers or a Build Pipeline whenever we change code. Well we are programmers so we are good in automating things, so I did.

The overall migration process is quite simple:

1. Start a clean database.
2. Use Flyway to migrate the database to the version the data set files are based on.
3. Load a data set from a DBUnit file into the database.
4. Let Flyway migrate the database to a target version.
5. Use DBUnit to extract the database to a DBunit data set file.
6. Repeat for all data set files you want to migrate.

A DBUnit Migration Maven Plugin

Even the process to migrate data sets is simple, it is time consuming to do it manually. I guess it would be easier with a tool. I was looking for a tool that automates that process. Since I didn’t find one, I wrote one – the dbunit-migration-maven-plugin. This maven plugin is based on the dbunit-extensions library that I also wrote. Actually the maven plugin is just an adaption of the lis-dbunit-migration library to the maven api. The maven plugin eases the configuration of the library classes since it can extract a lot of information out of the maven project. But you can do anything the maven plugin does with the library itself. So feel free to implement a gradle plugin or an own app. You can also send me a pull request on github. To setup a clean database I use testcontainers.

The minimal plugin configuration you need looks like this:

<build>
    <plugins>
        <plugin>
            <groupId>com.link-intersystems.dbunit.maven</groupId>
            <artifactId>dbunit-migration-maven-plugin</artifactId>
            <version>RELEASE</version> <!-- check for the actual version -->
            <configuration>
                <flyway>
                    <sourceVersion>1</sourceVersion>
                </flyway>
                <testcontainers>
                    <!-- ls-dbunit-testcontainers ships with support for postgres and mysql -->
                    <!-- If you need to use another container please read the Complete Plugin Configuration section. -->
                    <image>postgres:latest</image>
                </testcontainers>
            </configuration>
        </plugin>
    </plugins>
</build>

If you only use the configuration above the plugin will pick up all data sets in ${project.basedir}/src/test/resources that are based on flyway version 1 and migrate them to the ${project.build.directory}. The source directory structure will be maintained. So if you want to use the migrated dataset files in your tests you can simple copy and overwrite your test resources and commit them.

You can run the plugin with mvn com.link-intersystems.dbunit.maven:dbunit-migration-maven-plugin:1.0.0:flyway-migrate. When you execute it you will see an output like this:

[info] Detected 4 data set resources to migrate
[info] ♻︎ Start migration '...\src\test\resources\flat\tiny-sakila-flat-column-sensing.xml'
[info] ✔︎ Migrated '...\target\flat\tiny-sakila-flat-column-sensing.xml'
[info] ♻︎ Start migration '...\src\test\resources\flat\tiny-sakila-flat.xml'
[info] ✔︎ Migrated '...\target\flat\tiny-sakila-flat.xml'
[info] ♻︎ Start migration '...\src\test\resources\tiny-sakila-csv'
[info] ✔︎ Migrated '...\target\tiny-sakila-csv'
[info] ♻︎ Start migration '...\src\test\resources\xml\tiny-sakila.xml'
[info] ✔︎ Migrated '...\target\xml\tiny-sakila.xml'
[info] Migrated 4 data set resources 

For a complete list of all configuration options take a look at the dbunit-migration-maven-plugin page.

Please let me know if you like the plugin, e.g. by giving a star to the github repository. You can also fork it and help me improve it, leave a comment on github or create an issue. Any help is welcome.