Migration update for Drupal 8.1

For those of you using the migration system under Drupal 8.0.x, with Drupal 8.1 scheduled to release tomorrow, let’s take a look at where the migration ecosystem now stands. We’ll discuss the biggest core API change, then how moving to 8.1 affects various use cases.

Migrations are now plugins

You may recall that the migration support in core is experimental - that is, it is still open to backwards-compatibility-breaking changes. There is a big one in 8.1 - migrations themselves are now plugins rather than configuration entities. This reduces technical debt, in the form of migration-specific infrastructure that’s now being handled by standard core features. Specifically, the plugins themselves serve the function that migration templates did in 8.0.x - they provide default configuration for migrations which require additional configuration (in particular, a database connection) to become runnable. In addition, the migration system’s home-grown builder system (used to generate, say, bundle-specific migrations from one generic node template) is now replaced by standard plugin derivatives. The lack of configuration entities, however, presents challenges for general-purpose migration tools. One is that there is now no direct core support for persisting configuration changes for the plugins - any configuration (such as a database connection) injected into a migration plugin is only applied at runtime, and without a standard method of persisting that configuration a general-purpose tool (such as the drush commands in migrate_tools) has no way of knowing what to inject or how. Likewise, we have the goal of developing tools for creating migrations from scratch through a UI, and the core migration approach of plugin configuration provided by YAML files within a module isn’t dynamic enough for this - again, we need a dynamic way to persist the configuration. Another issue is discovery - a tool like drush migrate-status wants to show you all “relevant” migrations (migrations which have been fully configured and are runnable), but with core full of migration plugins which aren’t relevant in most cases (and there’s virtually no case where they would all be relevant) simply discovering all the plugins and instantiating them doesn’t work. The irrelevant plugins throw a variety of exceptions, some during the initial discovery (making them difficult to handle) - perhaps worse, some give no indication at discovery or instantiation time so cannot be filtered out without hardcoded knowledge. At this point the cleanest answer I could come up with to the discovery problem was a registration process - an explicit way to mark a migration as eligible to be managed by the tools. And one means of “registering” a migration is to create a configuration entity for it, which (compared to a pure registration implementation like migration in Drupal 7 had) has the advantage of addressing the configuration persistence issues as well. So, that’s what I’ve done. The migrate_plus module now defines a Migration configuration entity akin to that which was in core in 8.0.x - but cleaner. You’ll find that unlike 8.0.x Migration entities, which contained a huge amount of functionality, the migrate_plus 8.1.x Migration entity class consists of pretty much nothing but annotation and schema - a much cleaner use of the configuration entity system. The migration functionality is contained within the core Migration plugin class. So, how does the entity relate to the plugin? Let’s look first at how the core migration plugins are defined. The migrate module in core implements MigrationPluginManager (provided as the plugin.manager.migration service). This plugin manager uses the YamlDirectoryDiscovery class to provide the plugin configuration - basically, it tells the discovery class to look in each enabled modules’ migrations directory (preferred) or migration_templates directory (for backwards compatibility) to find YAML files defining migration plugins. Thus, if you ask that service for all plugins it manages, you get a list corresponding to each enabled module’s migrations/*.yml and migration_templates/*.yml files. In migrate_plus, I’ve introduced an alternative MigrationConfigEntityPluginManager (provided as the plugin.manager.config_entity_migration service). This plugin manager uses a custom ConfigEntityDiscovery class, which will find all configuration entities of the specified entity type (this class is, by the way, fully general) and create migration plugins using that configuration. [Edit: This plugin manager no longer exists, we switched from using a separate plugin manager to using the standard one with a deriver to wrap "standard" migration plugins in config entities.] These configuration entities may originate at a module’s installation time (from the config/install directory), or be generated by a UI or drush tool. So, it sounds like we’ve got two different kinds of migration plugins now, right? No, not really - what we have is two distinct means of providing migration plugins with their configuration. For comparison’s sake:

MigrationPluginManager "migrations" directory

1. Configuration comes from YAML files in a module’s migrations or migration_templates directory.
2. May be incomplete (requiring runtime injection of required configuration such as a database connection).
3. Cannot be managed by migrate_tools - needs a specialized runner (e.g., migrate_drupal_ui for the core upgrade process) to inject required configuration. Edit: No longer true, migrate_tools can handle these now.
4. Appropriate for module-defined migrations which require further configuration to be run, or which are completely hard-coded (no need to edit the configuration with UI tools, for example).

MigrationConfigEntityPluginManager "config/install" directory

1. Configuration comes from configuration entities in the site’s active configuration (which may originate from a module’s config/install, or be generated dynamically).
2. Is assumed to be complete - the plugin should be able to be instantiated and run without any runtime injection.
3. Can be managed by migrate_tools.
4. Appropriate for migrations generated by tools, or module-provided migrations which you want to have editable.

Straight Drupal upgrades

If you’re looking to simply upgrade your Drupal 6 or Drupal 7 site to Drupal 8 as-is, the big news here is that the upgrade UI (formerly part of the migrate_upgrade contrib module) is now in core as the migrate_drupal_ui module. It works the same as it did previously - provide your database credentials and source of your public files and go. It should be noted that the previous support for rolling back or performing incremental migrations after an initial upgrade has been removed - the premise is that rather than doing a rollback, you should expect to reinstall Drupal 8 and rerun the upgrade from scratch. The contrib migrate_upgrade module still provides the migrate-upgrade drush command (as well as migrate-upgrade-rollback).

Custom Drupal migrations

While the plan is to eventually provide UI tools to make it easy to customize Drupal-to-Drupal migration paths (omit a content type here, map an old field to a differently-named new one there, etc.), in the meantime people have to use drush migrate-upgrade --configure-only to tweak their upgrade paths. More details on this workflow will follow - for the moment, the main thing to be aware of is that (after a period of absence) --configure-only is back and can be used to generated configuration entities which you can then edit to your specific needs.

Implementing upgrade paths for contrib modules

Because the new migration plugin manager reads the migration_templates directory for backward compatibility (and because the source/process/destination plugins have not changed), in most cases contrib modules implementing upgrade paths should not need to make any changes. The main issue would be in the rare case where you were using builder plugins, which are now gone and should be replaced by derivers instead. See the change record for more details.

General custom migrations

If you are implementing other migration scenarios, you’ll need to make the following changes to your migration modules for 8.1:

1. If in 8.0.x you were implementing migrations by providing .yml files in config/install, you’ll need to rename those files from migrate.migration.* to migrate_plus.migration.* (because it is now the contrib migrate_plus module rather than the core migrate module which implements migration configuration entities).
2. If you were previously using migration groups, please note that they are now first-class properties of the migration entity. So, you can replace third_party_settings:   migrate_plus:     migration_group: beer with migration_group: beer
3. Previously you could omit migration_dependencies in migrations with no dependencies, but the core plugin is now a stickler for seeing an empty array. So, in this case you’ll need to add migration_dependencies: {}

Going forward

There’s still a fair amount of work being done to fill in the gaps in the core migration support (particularly for upgrades from D6/D7). Hopefully we can remove the experimental label from the migration system for Drupal 8.2. In the contrib space, migrate_upgrade 8.x-2.0.beta1 (or 8.x-2.x-dev) is the release to use with Drupal 8.1. I will be releasing beta versions of migrate_plus and migrate_tools shortly. Once those are out, the main foci of my community time will be:

• Core issues (I haven’t been much involved on that side lately as I’ve been bringing the contrib side up-to-date with 8.1.x).
• Finally getting started on the D8 version of migrate_d2d, so we have proper tools for general Drupal-to-Drupal migrations.
• Consolidated XML/JSON (and other file-based) source plugins. I got these basically working a few months ago, but there’s still a fair amount of fleshing out to do.

Acknowledgements

My recent work in the migration space has been sponsored by Drupalize.Me. In particular, Will Hetherington has given a lot of great feedback.