Configuring migrations via a form

Frequently, there may be parts of a migration configuration which shouldn’t be hard-coded into your YAML file - some configuration may need to be changed periodically, some may vary according to environment (for example, a dev environment may access a dev or test API endpoint, while prod needs to access a production endpoint), or you may need a password or other credentials to access a secure endpoint (or for a database source which you can’t put into settings.php). You may also need to upload a data file for input into your migration. If you are implementing your migrations as configuration entities (a feature provided by the migrate_plus module), all this is fairly straightforward - migration configuration entities may easily be loaded, modified, and saved based on form input, implemented in a standard form class.

Uploading data files

For this project, while other CSV source files were static enough to go into the migration module itself, we needed to periodically update the blog data during the development and launch process. A file upload field is set up in the normal way:

$form['acme_blog_file'] = [
 '#type' => 'file',
 '#title' => $this->t('Blog data export file (CSV)'),
 '#description' => $this->t('Select an exported CSV file of blog data. Maximum file size is @size.',
   ['@size' => format_size(file_upload_max_size())]),
];

And saved to the public file directory in the normal way:

$all_files = $this->getRequest()->files->get('files', []);
if (!empty($all_files['acme_blog_file'])) {
 $validators = ['file_validate_extensions' => ['csv']];
 if ($file = file_save_upload('acme_blog_file', $validators, 'public://', 0)) {

So, once we’ve got the file in place, we need to point the migration at it. We load the blog migration, retrieve its source configuration, set the path to the uploaded file, and save it back to active configuration storage.

   $blog_migration = Migration::load('blog');
   $source = $blog_migration->get('source');
   $source['path'] = $file->getFileUri();
   $blog_migration->set('source', $source);
   $blog_migration->save();
   drupal_set_message($this->t('File uploaded as @uri.', ['@uri' => $file->getFileUri()]));
   }
 else {
   drupal_set_message($this->t('File upload failed.'));
 }
}

It’s important to understand that get() and set() only operate directly on top-level configuration keys - we can’t simply do something like $blog_migration->set(‘source.path’, $file->getFileUri()), so we need to retrieve the whole source configuration array, and set the whole array back on the entity.

Endpoints and credentials

The endpoint and credentials for our event service are configurable through the same webform. Note that we obtain the current values from the event migration configuration entity to prepopulate the form:

$event_migration = Migration::load('event');
$source = $event_migration->get('source');
if (!empty($source['urls'])) {
 if (is_array($source['urls'])) {
   $default_value = reset($source['urls']);
 }
 else {
   $default_value = $source['urls'];
 }
}
else {
 $default_value = 'http://services.example.com/CFService.asmx?wsdl';
}

$form['acme_event'] = [
 '#type' => 'details',
 '#title' => $this->t('Event migration'),
 '#open' => TRUE,
];

$form['acme_event']['event_endpoint'] = [
 '#type' => 'textfield',
 '#title' => $this->t('CF service endpoint for retrieving event data'),
 '#default_value' => $default_value,
];

$form['acme_event']['event_clientid'] = [
 '#type' => 'textfield',
 '#title' => $this->t('Client ID for the CF service'),
 '#default_value' => @$source['parameters']['clientId'] ?: 1234,
];

$form['acme_event']['event_password'] = [
 '#type' => 'password',
 '#title' => $this->t('Password for the CF service'),
 '#default_value' => @$source['parameters']['clientCredential']['Password'] ?: '',
];

In submitForm(), we again load the migration configuration, insert the form values, and save:

$event_migration = Migration::load('event');
$source = $event_migration->get('source');
$source['urls'] = $form_state->getValue('event_endpoint');
$source['parameters'] = [
 'clientId' => $form_state->getValue('event_clientid'),
 'clientCredential' => [
   'ClientID' => $form_state->getValue('event_clientid'),
   'Password' => $form_state->getValue('event_password'),
 ],
 'startDate' => date('m-d-Y'),
];

$event_migration->set('source', $source);
$event_migration->save();
drupal_set_message($this->t('Event migration configuration saved.'));

Note that we also reset the startDate value while we’re at it (see the previous SOAP blog post).