PHP

Soong 0.7.0 released

mikeryan — Fri, 05 Jul 2019 21:05:10 +0000

Soong 0.7.0 released mikeryan Friday, July 5, 2019 - 04:05pm

The 0.7.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.6.0 include:

The property abstractions (PropertyInterface, PropertyFactory, and implementations) have been removed - it seems like the abstractions will only get in the way of leveraging PHP 7.x's improved native type-checking.
The Record Transformer concept has been introduced. While our experience with Drupal emphasized the mapping of properties within each record, the logical purpose of the transformation segment of an ETL pipeline is to transform a record at a time (stipulating that, yes, 90+% of the time you're individually transforming each property within the record)...
...thus one of the two RecordTransformer implementations provided out-of-the-box is the PropertyMapper. The field mapping configuration we previously provided within the transform key is now the PropertyMapper's configuration, and the classes are now implementations of PropertyTransformer rather than Transformer.
The other provided RecordTransformer is Copy, for bulk-copying properties directly from the source record to the destination record. In many instances, most if not all properties being migrated are retaining exactly the same name and content, so it can help clarify the "interesting" property transformations if you don't have to individual express each and every trivial property copy.

This release was meant to be more ambitious (including refactoring of task pipelines) and sooner, but I've been busy working a couple of contracts since May. So, for the forseeable future, progress will be slower (unless/until other folks start contributing, of course!). For now, the emphasis for 0.8.0 is task refactoring, starting with seeing if we can leverage other libraries like league/pipeline.

Tags

Migration

Soong

PHP

Use the Twitter thread below to comment on this post:

A new release of Soong ETL is out! https://t.co/9CgXWl5vHx
— Virtuoso Performance (@VirtPerformance) July 5, 2019

Soong 0.6.0 released

mikeryan — Wed, 01 May 2019 20:40:18 +0000

Soong 0.6.0 released mikeryan Wednesday, May 1, 2019 - 03:40pm

The 0.6.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.5.3 include:

The Filter component has been introduced, which examines a Record and "approves" it for further processing, along with a filters configuration option added to extractors to limit what gets extracted. The Select filter and migrate command option --select have been added - extractor results may thus be filtered either in the base configuration for the extractor (representing the canonical set of data to be processed) or at runtime (for testing). An example of the latter, restricting input to records that have both a low id and a high foo value:

bin/soong migrate arraytosql --select='id<8' --select='foo>g'
Dependency injection has been improved - where previously class names were passed through to be instantiated by the components that needed them, responsibility for constructing all (almost, see below) components now belongs to the application (for now, the Symfony console commands being the single example) which will inject the class instances.
Since components do need to dynamically generate Property and Record instances during migration, in those cases a PropertyFactory or RecordFactory instance is injected.
The DataProperty and DataRecord classes have been renamed to Property and Record respectably.
Error handling has been beefed up, with several exception classes added.
We're now integrated with Scrutinizer for coverage and quality analysis. Unfortunately, Scrutinizer does not yet allow public access to analysis for Gitlab-based projects, so you can't look for yourself - but here's where we are now (having built out the tests further):

At the moment, the main emphases for 0.7.0 are:

Build out the console commands to be more-or-less feature-complete.
Refactor the transformation pipeline.
Refactor our approach to tasks, in particular looking at using or gaining inspiration from other tools such as Robo or PortPHP.

Obligatory self-promotion - I'm available for data migration projects.

Thanks!

Tags

Migration

Soong

PHP

Use the Twitter thread below to comment on this post:

https://t.co/Lk9zVdhwQM
— Virtuoso Performance (@VirtPerformance) May 1, 2019

Update on Soong ETL

mikeryan — Tue, 26 Feb 2019 17:26:01 +0000

Update on Soong ETL mikeryan Tuesday, February 26, 2019 - 11:26am

It's been over a month now since I made the Soong ETL library publicly available - about time for some updates.

One focus has been fleshing out areas that will aid in contribution. These include:

(Too) early I had split things out into a myriad of repositories, a case of premature optimization. I merged things back to a monorepo for now - as APIs are still fluid, it's much easier to make API changes in one repo and keep all the components in sync. Once the APIs are reasonably stable (at least, say, at "beta" level) at the very least specialized integrations like Csv and DBAL will move to their own repos. Although I initially imagined soong/soong including only the interfaces (and maybe some base classes), I'm now thinking it should also hold basic implementations of at least the Data, KeyMap, and Task interfaces.
Adding tests for existing code (still in progress), in particular adding base classes corresponding to the component interfaces to ease testing that implementations of those interfaces behave consistently and in accordance with the contracts. No new functionality will be added without tests.
Putting documentation up on Read the Docs - in particular, fleshing out the code documentation and generating it with Doxygen, and providing more information on contributing.

Priorities now are:

Seeking more participation from other developers (hi out there!). And, by the way, I'll be at Midwest PHP next week, my first non-Drupal PHP conference - any Midwesterners interested in data migration, hit me up and we'll talk!
Looking at other ETL systems for ideas - we have looked at PortPHP so far, which has some ideas we might borrow (and perhaps we can even integrate its readers).
Addressing any proposed changes that would break the existing API. Note that in the current 0.x.x stream, minor versions (e.g., 0.4.0) are API breakers, so be sure to pin any applications using Soong to the minor version ("~0.4.0" constraint in composer.json).

Coalescing those three priorities - if you have any interest in data migration in PHP, please stop by and offer your thoughts on the architecture!

Oh, by the way - I'm currently between projects, so if you need a data migration process implemented please contact me. I will (for now) take a reduced rate for a project using Soong, as there's nothing like a real-world application to take a general-purpose library to the next level.

Thanks!

Tags

PHP

Migration

Soong

Use the Twitter thread below to comment on this post:

https://t.co/zNhHXmbO8P
— Virtuoso Performance (@VirtPerformance) February 26, 2019

Announcing the Soong project - developing a general-purpose ETL framework

mikeryan — Tue, 22 Jan 2019 20:10:33 +0000

Announcing the Soong project - developing a general-purpose ETL framework mikeryan Tuesday, January 22, 2019 - 02:10pm

I'd like to invite members of the open-source community, particularly (but not exclusively) those involved with PHP, to join in designing and developing a general-purpose ETL framework for data migration. The vendor name for packaging components of this project is soong, and git repos for existing components are under the GitLab account "soongetl".

Note: Finally having finished composition of this lengthy monologue, it's clear to me that it's very ambitious (some might say arrogant) of me to write with the expectation that this will grow into a large and robust open-source ecosystem. Very well - it is ambitious, and the effort may very well fall flat on its face. C'est la vie...

Who am I?

I'm Mike Ryan - a lot of people in the Drupal community know me, but not so much the wider open-source community. Almost eleven years ago at a Drupal meetup in Boston, amongst general agreement that everyone hates to do data migration, Moshe Weitzman looked across the table at me and said "there's an opportunity here." Since then data migration into Drupal has been the primary focus of my professional life, first in partnership with Moshe, then as an Acquia employee and finally as a solo consultant. Over the years I've created several migration-related contrib modules for Drupal, was part of the team integrating migration into Drupal core for D8, and have been involved in dozens of real-world migration projects.

Why am I doing this?

I think we can do better

Just within Drupal, the migration framework can be improved:

Each step of Drupal migration support has been a port of the previous - from the hook-based Drupal 6 version, to the inheritence-and-composition model in Drupal 7, to the plugin-based system in Drupal 8, technical debt has accumulated. I've wanted for a while to start over with a clean slate - given my experience (and others), what would we do differently starting from scratch? Can we step back and re-examine the assumptions we've been carrying forward?
At a specific technical level, the biggest itch I've wanted to scratch is decoupling the components. Within the migration system as it is in D8 today, pretty much every component knows everything about every other component. At one point we had a destination plugin which was using some of the migration's source plugin configuration - that one made my eye twitch!
There's also the coupling of the migration system with Drupal - in particular, migration classes *are* Drupal plugins (i.e., their interfaces extend PluginInspectionInterface) rather than being *managed by* Drupal plugins. I would like to see migration classes be all about migration, rather than worry about being plugins as well. And once the basic migration classes are no longer Drupal plugins, then it's a small step to them being entirely independent of Drupal...

The larger PHP community

With Drupal 8, we’ve often talked about “getting off the island” in terms of benefiting from much fine PHP work done outside of the Drupal community. We haven’t talked so much about going in the opposite direction - making our own fine work available for use beyond Drupal. To my knowledge, the only published example of this so far is Kris Vanderwater (EclipseGC) with the plugin library.

Likewise, we Drupal developers don’t have a monopoly on good migration ideas - by moving the general-purpose aspects of migration into a separate open source project, we have the opportunity to benefit from new ideas and new talent.

The community we build

The major key to success for any large open-source project like this is a thriving community. After seeing open-source projects like Drupal grow organically - and face growing pains as they find themselves dealing with community problems reactively rather than proactively - if a community does form around this project, I would like to establish a supportive and welcoming tone from the beginning.

Diversity in particular remains an issue in the tech industry in general, and open-source especially - and a lack of diversity is difficult to correct after the fact. In building a community around this framework, my hope is that we draw a diverse set of developers in the beginning, in the hopes that seeding the garden well will be, if not self-sustaining, at least more sustainable. How to do that, I'm not certain - a concerted outreach effort could easily end up looking like Pokemon Go, searching for unique creatures to collect. Apart from starting with a good Code of Conduct, I'm open to suggestions!

Another aspect of community-building is providing opportunities for relative novices (whether new to open-source development, new to PHP, or new to migration). The proposed architecture involves myriad small, well-focused packages - an extractor here, a set of related transformers there, integrations for specific frameworks and APIs... Individual transformers, in particular, will generally be very simple. This ecosystem thus will provide ample opportunities for novices to gain experience with mentorship and also establish an online presence.

Now, all that being said, what about The Ethics of Unpaid Labor and the OSS Community (also see the recent Twitter discussion in the Drupal community)? In reaching out to underrepresented groups and to novices, we are reaching out to the people who have the least ability to work on open source for free. One way to ameliorate this effect may be to explicitly try to draw in students - whether in formal programs or teaching themselves software development - who will benefit from some free practical education and mentorship. Down the road, if this framework does start being adopted in real-world applications, we can look at ways to get sponsorships for people who maintain projects within the ecosystem. At any rate, as the community here grows I expect this will be an ongoing conversation.

Selfishness

Yes, I'm willing to cop to selfish reasons to pursue this.

Simple ego: I'm proud of the work I've done on migration in Drupal, and think it can be useful on a larger stage. Being old enough to see retirement on the horizon, I admit I'm thinking of this as my magnum opus - the last major contribution I make to open source. I would love to leave behind a significant piece of quality software with a vital community behind it.
Money: I've done fine as a Drupal data migration specialist. I hope to do better by expanding my market beyond Drupal, working on a wider variety of migration projects. Yes, retirement is on the horizon but, given earlier attempts at consulting which went less well than my "migration period" has, my funds put that horizon farther out than I'd like...

What's done so far?

Early last year I started playing around with a proof-of-concept in a single repo, getting a single basic ETL migration scenario running with a decoupled class structure based on the basic architecture of the Drupal migration system. Much of the work after getting the initial POC running was figuring out appropriate boundaries between components, and gradually introducing features beyond the most basic ones I started with. And then breaking pieces out into separate source repos, and figuring out those boundaries.

My role

This will certainly change according to the number and skills of contributors who join into this effort (assuming there are some!), but what I'm aiming for in terms of my own role:

Primary architect of version 1 of Soong. This would mean being the primary maintainer of architecture documentation and the repository of central interfaces/base classes. Per "selfishness" above - I have an architectural vision I want to see brought to fruition. Others may take it in different directions after that, but V1 is mine! tl;dr - I don't want to be BDFL; I do want to be BDF1.
Community leader. Per "community" above, I have a vision for building a diverse and vibrant open-source community from the ground up. Unlike the technical architecture, however, this plays less to my strengths, so I will be happy to defer as better-suited people show leadership in the community.
Mentorship. I'd like to help people up their development skills, their open-source involvement, and their understanding of the pits and perils of data migration.

Why did it take me so long?

After having it in the back of my head for a few years, I finally started creating repos and putting my thoughts into actual interfaces and classes several months ago. Why did I wait until now to share my work with the larger community? I certainly felt seen when I read this:

https://twitter.com/jessfraz/status/1063425181509652481

Frankly, there's an element of imposter syndrome here - I wanted to be sure I wasn't exposing any dumb ideas! Well, enough of that - instead, I now stipulate that you will find dumb things I did here, and ask that you help smartify them.

The architecture itself

There's a ways to go documenting the architecture as it is currently implemented in soong/soong, but right now it broadly looks like this:

A Task accepts configuration defining a migration process, and implements operations - most notably migrate, but it may also support other operations like rollback, status, analyze, … The following steps describe the migrate operation.
The task constructs the configured Extractor, which obtains data from a source such as a SQL query, a CSV file, an XML/JSON API, etc.
Iterating over the extractor returns one DataRecord (collection of named DataProperty instances) at a time containing source data. The task creates an empty DataRecord representing the destination data.
The task configuration defines a transform pipeline keyed by destination property names. For each of these properties, a sequence of one or more Transformer classes with corresponding configuration is invoked to determine the destination property value - usually, the first one will be configured to accept one or more source property names, and the results will be fed to subsequent transformers, with the final result assigned to the named property in the destination DataRecord.
The destination DataRecord is passed to the configured Loader to be loaded into the destination store - a SQL database, a CSV file, etc.
If an optional KeyMap is configured within the task, it is used to store the mapping from the source record's unique key to the destination record's unique key. This enables keyed relationships to be maintained even if keys change when migrating, as well as enabling rollback.

To try out a couple of working demos, git clone git@gitlab.com:soongetl/soong.git and follow the README.

Initial technical priorities

One of those infamous hard problems in computer science is naming things. Before we go too far, let's figure out how best to name things - I think Extractor/Transformer/Loader are pretty solid, but let's discuss whether other components (like Task) could use better names. Also, let's decide what naming conventions for implementations should look like - e.g., should CSV extractor and loader classes both be named CSV (or for that matter, Csv) with namespaces alone distinguishing them, or should they be CSVExtractor and CSVLoader?
The initial architecture, as I've said before, comes from my narrow experience in Drupal. I'm sure there are plenty of other good migration ideas out there - maybe there's even a package I've missed that's good enough that this effort would better be directed towards improving it rather than starting from scratch. I did do some research last year and did not find any PHP ETL packages that appeared to have wide adoption or as much flexibility, but with more eyes on it (eyes that have seen more beyond Drupal than I have) let's see if we can do a thorough review of prior art and see if there are some good ideas which may influence this effort. And let's look beyond PHP as well - are there ETL frameworks written in other object-oriented languages which may provide some architectural inspiration?
Review the boilerplate for Soong code repos (based on https://github.com/thephpleague/skeleton) - let's go over what we've got there (especially the code of conduct and contributing guidelines).
Test all the things! Before adding new stuff, we need to add tests for the existing components, and set up automated testing on Gitlab.

Technical goals

For V1, require PHP 7.1 and leverage strict type checking. I expect future versions to require PHP 7.4 and leverage typed object properties.
The central interface package soong/soong ideally should not depend on anything other than PSR interfaces. It should be approached as if it were a PSR itself - a completely general interface for ETL functionality not dependent on any non-standard interfaces.

Conclusion

Again, I know I am getting way ahead of myself here by imagining an active open-source community will quickly spring up here. I have talked to Drupal people about my ideas on occasion, and I expect there will be some interest there, but I very much hope other open-source developers can join this effort and provide different perspectives. I do believe strongly that a standard ETL library with a core of simple standard interfaces (making a simple move-my-stuff-from-here-to-there application a breeze) plus the flexibility to build complex systems to handle many types of data will be extremely valuable across many domains.

If I may try your patience a bit longer - I've spent a substantial portion of my time since my last contract pulling these thoughts together, and I am now in need of paid work (contact me if you need some data migration done!). I may fantasize about being sponsored to work fulltime on Soong, or be hopeful there's someone with a project that they think will benefit from Soong and thus I can make progress here in the course of solving their migration problem. Realistically, my next contract (or employment) most likely will not involve Soong development, so once I'm working I won't have as much time to manage this project - let's hope plenty of people join in to pick up my slack!

If you've made it this far, thank you for your time and I look forward to your merge requests!

Tags

Drupal

Planet Drupal

PHP

Migration

Soong

Use the Twitter thread below to comment on this post:

https://t.co/0uVJ13t8md
— Virtuoso Performance (@VirtPerformance) January 22, 2019

Drupal file migrations: The s3fs module

mikeryan — Wed, 09 Jan 2019 19:56:19 +0000

Drupal file migrations: The s3fs module mikeryan Wednesday, January 9, 2019 - 01:56pm

A recent project gave me the opportunity to familiarize myself with the Drupal 8 version of the S3 File System (s3fs) module (having used the D7 version briefly in the distant past). This module provides an s3:// stream wrapper for files stored in an S3 bucket, allowing them to be used as seamlessly as locally stored public and private files. First we present the migrations and some of the plugins implemented to support import of files stored on S3 - below we will go into some of the challenges we faced.

Our client was already storing video files in an S3 bucket, and it was decided that for the Drupal site we would also store image files there. The client handled bulk uploading of images to an "image" folder within the bucket, using the same (relative) paths as those stored for the images in the legacy database. Thus, for migration we did not need to physically copy files around (the bane of many a media migration!) - we "merely" needed to create the appropriate entities in Drupal pointing at the S3 location of the files.

The following examples are modified from the committed code - to obfuscate the client/project, and to simplify so we focus on the subject at hand.

Image migrations

Gallery images

In the legacy database all gallery images were stored in a table named asset_metadata, which is structured very much like Drupal's file_managed table, with the file paths in an asset_path column. The file migration looked like this:

id: acme_image
source:
  plugin: acme
process:
  filename:
    plugin: callback
    callable: basename
    source: asset_path
  uri:
    # Construct the S3 URI - see implementation below.
    plugin: acme_s3_uri
    source: asset_path
  # Source data created/last_modified fields are YYYY-MM-DD HH:MM:SS - convert
  # them to the classic UNIX timestamps Drupal loves. Oh, and they're optional,
  # so when empty leave them empty and let Drupal set them to the current time.
  created:
    -
      plugin: skip_on_empty
      source: created
      method: process
    -
      plugin: callback
      callable: strtotime
  changed:
    -
      plugin: skip_on_empty
      source: last_modified
      method: process
    -
      plugin: callback
      callable: strtotime
destination:
  plugin: entity:file

Because we also needed to construct the S3 uris in places besides the acme_s3_uri process plugin, we implemented the construction in a trait which cleans up some inconsistencies and prepends the image location:

trait AcmeMakeS3Uri {
  /**
   * Turn a legacy image path into an S3 URI.
   *
   * @param string $value
   *
   * @return string
   */
  protected function makeS3Uri($value) {
     // Some have leading tabs.
     $value = trim($value);
     // Path fields are inconsistent about leading slashes.
     $value = ltrim($value, '/');
     // Sometimes they contain doubled-up slashes.
     $value = str_replace('//', '/', $value);
     return 's3://image/' . $value;
  }
}

So, the process plugin in the image migration above uses the trait to construct the URI, and verifies that the file is actually in S3 - if not, we skip it. See the Challenges and Contributions section below for more on the s3fs_file table.

/**
 * Turn a legacy image path into an S3 URI.
 *
 * @MigrateProcessPlugin(
 *   id = "acme_s3_uri"
 * )
 */
class AcmeS3Uri extends ProcessPluginBase {
  use AcmeMakeS3Uri;
 
  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $uri = $this->makeS3Uri($value);
  // For now, skip any images not cached by s3fs.
  $s3_uri = \Drupal::database()->select('s3fs_file', 's3')
    ->fields('s3', ['uri'])
    ->condition('uri', $uri)
    ->execute()
    ->fetchField();
    if (!$s3_uri) {
      throw new MigrateSkipRowException("$uri missing from s3fs_file table");
    }
    return $uri;
  }
}

The above creates the file entities - next, we need to create the media entities that reference the files above via entity reference fields (and add other fields). These media entities are then referenced from content entities.

id: acme_image_media
source:
  plugin: acme
process:
  # For the media "name" property - displayed at /admin/content/media - our
  # first choice is the image caption, followed by the "event_name" field in
  # our source table. If necessary, we fall back to the original image path.
  name:
    -
      # Produces an array containing only the non-empty values.
      plugin: callback
      callable: array_filter
      source:
        - caption
        - event_name
        - asset_path
    -
      # From the array, pass on the first value as a scalar.
      plugin: callback
      callable: current
    -
      # Some captions are longer than the name property length.
      plugin: substr
      length: 255
  # Entity reference to the image - convert the source ID to Drupal's file ID.
  field_media_image/target_id:
    plugin: migration_lookup
    migration: acme_image
    source: id
  # Use the name we computed above as the alt text.
  field_media_image/alt: '@name'
  # We need to explicitly set the image dimensions in the field's width/height
  # subfields (more on this below under Challenges and Contributions). Note that in
  # the process pipeline you can effectively create temporary fields which can be
  # used later in the pipeline - just be sure they won't conflict with
  # anything that might be used within the Drupal entity.
  _uri:
    plugin: acme_s3_uri
    source: asset_path
  _image_dimensions:
    plugin: acme_image_dimensions
    source: '@_uri'
  field_media_image/width: '@_image_dimensions/width'
  field_media_image/height: '@_image_dimensions/height'
  caption: caption
destination:
  plugin: entity:media
  default_bundle: image
migration_dependencies:
  required:
    - acme_image

Other images

The gallery images have their own metadata table - but, there are many other images which are simply stored as paths in content tables (in some cases, there are multiple such path fields in a single table). One might be tempted to deal with these in process plugins in the content migrations - creating the file and media entities on the fly - but that would be, well, ugly. Instead we implemented a drush command, run before our migration tasks, to canonicalize and gather those paths into a single table, which then feeds the acme_image_consolidated and acme_image_media_consolidated migrations (which end up being simpler versions of acme_image and acme_image_media, since "path" is the only available source field).

function drush_acme_migrate_gather_images() {
  // Key is legacy table name, value is list of image path columns to migrate.
  $table_fields = [
    'person' => [
      'profile_picture_path',
      'left_standing_path',
      'right_standing_path',
    ],
    'event' => [
      'feature_image',
      'secondary_feature_image',
    ],
    'subevent' => [
      'generated_medium_thumbnail',
    ],
    'news_article' => [
      'thumbnail',
    ]
  ];
  $legacy_db = Database::getConnection('default', 'migrate');
  // Create the table if necessary.
  if (!$legacy_db->schema()->tableExists('consolidated_image_paths')) {
    $table = [
      'fields' => [
         'path' => [
          'type' => 'varchar',
          'length' => 191,    // Longest known path is 170.
          'not null' => TRUE,
         ]
      ],
      'primary key' => ['path'],
    ];
    $legacy_db->schema()->createTable('consolidated_image_paths', $table);
    drush_print('Created consolidated_image_paths table');
  }
  $max = 0;
  foreach ($table_fields as $table => $field_list) {
    drush_print("Gathering paths from $table");
    $count = 0;
    $query = $legacy_db->select($table, 't')
      ->fields('t', $field_list);
    foreach ($query->execute() as $row) {
      // Iterate the image path columns returned in the row.
      foreach ($row as $path) {
        if ($path) {
          $len = strlen($path);
          if ($len > $max) $max = $len;
          $path = str_replace('//', '/', $path);
          $count++;
          $legacy_db->merge('consolidated_image_paths')
            ->key('path', $path)
            ->execute();
        }
      }
    }
    // Note we will end up with far fewer rows in the table due to duplication.
    drush_print("$count paths added from $table");
  }
  drush_print("Maximum path length is $max");
}

Video migrations

The legacy database contained a media table referencing videos tagged with three different types - internal, external, and embedded. "Internal" videos were those stored in S3 with a relative path in the internal_url column; "external" videos (most on client-specific domains, but with some Youtube domains as well) had a full URL in the external_url column; and "embedded" videos were with a very few exceptions Youtube videos with the Youtube ID in the embedded_id column. It was decided that we would migrate the internal and Youtube videos, ignoring the rest of the external/embedded videos. Here we focus on the internal (S3-based) videos.

id: acme_video
source:
  plugin: acme_internal_video
  constants:
    s3_prefix: s3://
process:
  _trimmed_url:
    # Since the callback process plugin only permits a single source value to be
    # passed to the specified PHP function, we have a custom plugin which enables us
    # to pass a character list to be trimmed.
    plugin: acme_trim
    source: internal_url
    trim_type: left
    charlist: /
  uri:
    -
      plugin: concat
      source:
        - constants/s3_prefix
        - '@_trimmed_url'
    -
      # Make sure the referenced file actually exists in S3 (does a simple query on
      # the s3fs_file table, throwing MigrateSkipRowException if missing).
      plugin: acme_skip_missing_file
  fid:
    # This operates much like migrate_plus's entity_lookup, to return an existing
    # entity ID based on arbitrary properties. The purpose here is if the file URI
    # is already in file_managed, point the migrate map table to the existing file
    # entity - otherwise, a new file entity will be created.
    plugin: acme_load_by_properties
    entity_type: file
    properties: uri
    source: '@uri'
    default_value: NULL
  filename:
    plugin: callback
    callable: basename
    source: '@uri'
destination:
  plugin: entity:file

The media entity migration is pretty straightforward:

id: acme_video_media
source:
  plugin: acme_internal_video
  constants:
    true: 1
process:
  status: published
  name: title
  caption: caption
  # The source column media_date is YYYY-MM-DD HH:DD:SS format - the Drupal field is
  # configured as date-only, so the source value must be truncated to YYYY-MM-DD.
  date:
    -
      plugin: skip_on_empty
      source: media_date
      method: process
    -
      plugin: substr
      length: 10
  field_media_video/0/target_id:
    -
      plugin: migration_lookup
      migration: acme_video
      source: id
      no_stub: true
    -
      # If we haven't migrated a file entity, skip this media entity.
      plugin: skip_on_empty
      method: row
  field_media_video/0/display: constants/true
  field_media_video/0/description: caption
destination:
  plugin: entity:media
  default_bundle: video
migration_dependencies:
  required:
    - acme_video

Did I mention that we needed to create a node for each video, linking to related content of other types? Here we go:

id: acme_video_node
source:
  plugin: acme_internal_video
  constants:
    text_format: formatted
    url_prefix: http://www.acme.com/media/
    s3_prefix: s3://image/
process:
  title: title
  status: published
  teaser/value: caption
  teaser/format: constants/text_format
  length:
    # Converts HH:MM:SS to integer seconds. Left as an exercise to the reader.
    plugin: acme_video_length
    source: duration
  video:
    plugin: migration_lookup
    migration: acme_video_media
    source: id
    no_stub: true
  # Field to preserve the original URL.
  old_url:
    plugin: concat
    source:
      - constants/url_prefix
      - url_name
  _trimmed_thumbnail:
    plugin: acme_trim
    trim_type: left
    charlist: '/'
    source: thumbnail
  teaser_image:
    -
      plugin: skip_on_empty
      source: '@_trimmed_thumbnail'
      method: process
    -
      # Form the URI as stored in file_managed.
      plugin: concat
      source:
        - constants/s3_prefix
        - '@_trimmed_thumbnail'
    -
      # Look up the fid.
      plugin: acme_load_by_properties
      entity_type: file
      properties: uri
    -
      # Find the media entity referencing that fid.
      plugin: acme_load_by_properties
      entity_type: media
      properties: field_media_image
  # Note that for each of these entity reference fields, we skipped some content,
  # so need to make sure stubs aren't created for the missing content. Also note
  # that the source fields here are populated in a PREPARE_ROW event.
  related_people:
    plugin: migration_lookup
    migration: acme_people
    source: related_people
    no_stub: true
  related_events:
    plugin: migration_lookup
    migration: acme_event
    source: related_events
    no_stub: true
  tag_keyword:
    plugin: migration_lookup
    migration: acme_keyword
    source: keyword_ids
    no_stub: true
destination:
  plugin: entity:node
  default_bundle: video
migration_dependencies:
  required:
    - acme_image_media
    - acme_video_media
    - acme_people
    - acme_event
    - acme_keyword

Auditing missing files

A useful thing to know (particularly with the client incrementally populating the S3 bucket with image files) is what files are referenced in the legacy tables but not actually in the bucket. Below is a drush command we threw together to answer that question - it will query each legacy image or video path field we're using, construct the s3:// version of the path, and look it up in the s3fs_file table to see if it exists in S3.

/**
 * Find files missing from S3.
 */
function drush_acme_migrate_missing_files() {
  $legacy_db = Database::getConnection('default', 'migrate');
  $drupal_db = Database::getConnection();
  $table_fields = [
    [
      'table_name' => 'asset_metadata',
      'url_column' => 'asset_path',
      'date_column' => 'created',
    ],
    [
      'table_name' => 'media',
      'url_column' => 'internal_url',
      'date_column' => 'media_date',
    ],
    [
      'table_name' => 'person',
      'url_column' => 'profile_picture_path',
      'date_column' => 'created',
    ],
    // … on to 9 more columns among three more tables...
  ];
 
  $header = 'uri,legacy_table,legacy_column,date';
  drush_print($header);
  foreach ($table_fields as $table_info) {
    $missing_count = 0;
    $total_count = 0;
    $table_name = $table_info['table_name'];
    $url_column = $table_info['url_column'];
    $date_column = $table_info['date_column'];
    $query = $legacy_db->select($table_name, 't')
      ->fields('t', [$url_column])
      ->isNotNull($url_column)
      ->condition($url_column, '', '<>');
    if ($table_name == 'media') {
      $query->condition('type', 'INTERNALVIDEO');
    }
    if ($table_name == 'people') {
      // This table functions much like Drupal's node table.
      $query->innerJoin('publishable_entity', 'pe', 't.id=pe.id');
      $query->fields('pe', [$date_column]);
    }
    else {
      $query->fields('t', [$date_column]);
    }
    $query->distinct();
    foreach ($query->execute() as $row) {
      $path = trim($row->$url_column);
      if ($path) {
        $total_count++;
        // Paths are inconsistent about leading slashes.
        $path = ltrim($path, '/');
        // Sometimes they have doubled-up slashes.
        $path = str_replace('//', '/', $path);
        if ($table_name == 'media') {
          $s3_path = 's3://' . $path;
        }
        else {
          $s3_path = 's3://image/' . $path;
        }
        $s3 = $drupal_db->select('s3fs_file', 's3')
          ->fields('s3', ['uri'])
          ->condition('uri', $s3_path)
          ->execute()
            ->fetchField();
        if (!$s3) {
          $output_row = "$s3_path,$table_name,$url_column,{$row->$date_column}";
          drush_print($output_row);
          $missing_count++;
        }
      }
    }
    drush_log("$missing_count of $total_count files missing in $table_name column $url_column", 'ok');
  }
}

Challenges and contributions

The s3fs module's primary use case is where the configured S3 bucket is used only by the Drupal site, and populated directly by file uploads through Drupal - our project was an outlier in terms of having all files in the S3 bucket first, and in sheer volume. A critical piece of the implementation is the s3fs_file table, which caches metadata for all files in the bucket so Drupal rarely needs to access the bucket itself other than on file upload (since file URIs are converted to direct S3 URLs when rendering, web clients go directly to S3 to fetch files, not through Drupal). In our case, the client had an existing S3 bucket which contained all the video files (and more) used by their legacy site, and to which they bulk uploaded image files directly so we did not need to do this during migration. The module does have an s3fs-refresh-cache command to populate the s3fs_file table from the current bucket contents, but we did have to deal with some issues around the cache table.

Restriction on URI lengths

As soon as we started trying to use drush s3fs-refresh-cache, we ran into the existing issue Getting Exception 'PDOException'SQLSTATE[22001] When Running drush s3fs-refresh-cache - URIs in the bucket longer than the 255-character length of s3fs_file's uri column. The exception aborted the refresh entirely, and because the refresh operation generates a temporary version of the table from scratch, then swaps it for the "live" table, the exception prevented any file metadata from being refreshed if there was one overflowing URI. I submitted a patch implementing the simplest workaround - just generating a message and ignoring overly-long URIs. Discussion continues around an alternate approach, but we used my patch in our project.

Lost primary key

So, once we got the cache refresh to work, we found serious performance problems. We had stumbled on an existing issue, "s3fs_file" table has no primary key. I tracked down the cause - because the uri column is 255 characters long, with InnoDB it cannot be indexed when using a multibyte collation such as utf8_general_ci. And Drupal core has a bug, DatabaseSchema_mysql::createTableSql() can't set table collation, preventing the setting of the utf8_bin collation directly in the table schema. The s3fs module works around that bug when creating the s3fs_file table at install time by altering the collation after table creation - but the cache refresh created a new cache table using only the schema definition and did not pick up the altered collation. Thus, only people like us who used cache refresh would lose the index, and those with more modest bucket sizes might never even notice. My patch to apply the collation (later refined by jansete) was committed to the s3fs module.

Scalability of cache refresh

As the client loaded more and more images into the bucket, drush s3fs-refresh-cache started running out of memory. Our bucket was quite large (1.7 million files at last count), and the refresh function gathered all file metadata in memory before writing it to the database. I submitted a patch to chunk the metadata to the db within the loop, which has been committed to the module.

Image dimensions

Once there were lots of images in S3 to migrate, the image media migrations were running excruciatingly slowly. I quickly guessed and confirmed that they were accessing the files directly from S3, and then (less quickly) stepped through the debugger to find the reason - the image fields needed the image width and height, and since this data wasn't available from the source database to be directly mapped in the migration, it went out and fetched the S3 image to get the dimensions itself. This was, of course, necessary - but given that migrations were being repeatedly run for testing on various environments, there was no reason to do it repeatedly. Thus, we introduced an image dimension cache table to capture the width and height the first time we imported an image, and any subsequent imports of that image only needed to get the cached dimensions.

In the acme_image_media migration above, we use this process plugin which takes the image URI and returns an array with width and height keys populated with the cached values if present, and NULL if the dimensions are not yet cached:

/**
 * Fetch cached dimensions for an image path (purportedly) in S3.
 *
 * @MigrateProcessPlugin(
 *   id = "acme_image_dimensions"
 * )
 */
class AcmeImageDimensions extends ProcessPluginBase {
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    $dimensions = Database::getConnection('default', 'migrate')
      ->select('s3fs_image_cache', 's3')
      ->fields('s3', ['width', 'height'])
      ->condition('uri', $value)
      ->execute()
      ->fetchAssoc();
    if (empty($dimensions)) {
      return ['width' => NULL, 'height' => NULL];
    }
    return $dimensions;
  }
}

If the dimensions were empty, when the media entity was saved Drupal core fetched the image from S3 and the width and height were saved to the image field table. We then caught the migration POST_ROW_SAVE event to cache the dimensions:

class AcmeMigrateSubscriber implements EventSubscriberInterface {
  public static function getSubscribedEvents() {
    $events[MigrateEvents::POST_ROW_SAVE] = 'import';
    return $events;
  }
 
  public function import(MigratePostRowSaveEvent $event) {
    $row = $event->getRow();
    // For image media, if width/height have been freshly obtained, cache them.
    if (strpos($event->getMigration()->id(), 'image_media') > 0) {
      // Note that this "temporary variable" was populated in the migration as a
      // width/height array, using the acme_image_dimensions process plugin.
      $original_dimensions = $row->getDestinationProperty('_image_dimensions');
      // If the dimensions are populated, everything's find and all of this is skipped.
      if (empty($original_dimensions['width'])) {
        // Find the media entity ID.
        $destination_id_values = $event->getDestinationIdValues();
        if (is_array($destination_id_values)) {
          $destination_id = reset($destination_id_values);
          // For performance, cheat and look directly at the table instead of doing
          // an entity query.
          $dimensions = Database::getConnection()
            ->select('media__field_media_image', 'msi')
            ->fields('msi', ['field_media_image_width', 'field_media_image_height'])
            ->condition('entity_id', $destination_id)
            ->execute()
            ->fetchAssoc();
          // If we have dimensions, cache them.
          if ($dimensions && !empty($dimensions['field_media_image_width'])) {
            $uri = $row->getDestinationProperty('_uri');
            Database::getConnection('default', 'migrate')
              ->merge('s3fs_image_cache')
              ->key('uri', $uri)
              ->fields([
                'width' => $dimensions['field_media_image_width'],
                'height' => $dimensions['field_media_image_height'],
              ])
              ->execute();
          }
        }
      }
    }
  }
}

Safely testing with the bucket

Another problem with the size of our bucket was that it was too large to economically make and maintain a separate copy to use for development and testing. So, we needed to use the single bucket - but of course, the videos in it were being used in the live site, so it was critical not to mess with them. We decided to use the live bucket with credentials allowing us to read and add files to the bucket, but not delete them - this would permit us to test uploading files through the admin interface, and most importantly from a migration standpoint access the files, but not do any damage. Worst-case scenario would be the inability to clean out test files, but writing a cleanup tool after the fact to clear any extra files out would be simple enough. Between this, and the fact that images were in a separate folder in the bucket (and we weren't doing any uploads of videos, simply migrating references to them), the risk of using the live bucket was felt to be acceptable. At first, though, the client was having trouble finding credentials that worked as we needed. As a short-term workaround, I implemented a configuration option for the s3fs module to disable deletion in the stream wrapper.

Investigating the permissions issues with my own test bucket, trying to add the bare minimum permissions needed for reading and writing objects, I arrived at a point where migration worked as desired, and deletion was prevented - but uploading files to the bucket through Drupal silently failed. There was an existing issue in the s3fs queue but it had not been diagnosed. I finally figured out the cause (Slack comment - "God, the layers of middleware I had to step through to find the precise point of death…") - by default, objects are private when uploaded to S3, and you need to explicitly set public-read in the ACL. Which the s3fs module does - but, to do this requires the PutObjectAcl policy, which I had not set (I've suggested the s3fs validator could detect and warn of this situation). Adding that policy enabled everything to work; once the client applied the necessary policies we were in business…

… for a while. The use of a single bucket became a problem once front-end developers began actively testing with image styles, and we were close enough to launch to enable deletion so image styles could be flushed when changed. The derivatives for S3 images are themselves stored in S3 - and with people generating derivatives in different environments, the s3fs_file table in any given environment (in particular the "live" environment on Pantheon, where the eventual production site was taking shape) became out of sync with the actual contents of S3. In particular, if styles were generated in the live environment then flushed in another environment, the live cache table would still contain entries for the derived styles (thus the site would generate URLs to them) even though they didn't actually exist in S3 - thus, no derived images would render. To address this, we had each environment set the s3fs root_folder option so they would each have their own sandbox - developers could then work on image styles at least with files they uploaded locally for testing, although their environments would not then see the "real" files in the bucket.

We discussed more permanent alternatives and Sean Blommaert put forth some suggestions in the s3fs issue queue - ultimately (after site launch) we found there is an existing (if minimally maintained) module extending stage_file_proxy. I will most certainly work with this module on any future projects using s3fs.

The tl;dr - lessons learned

To summarize the things to keep in mind if planning on using s3fs in your Drupal project:

Install the s3fs_file_proxy_to_s3 module first thing, and make sure all environments have it enabled and configured.
Make sure the credentials you use for your S3 bucket have the PutObjectAcl permission - this is non-obvious but essential if you are to publicly serve files from S3.
Watch your URI lengths - if the s3://… form of the URI is > 255 characters, it won't work (Drupal's file_managed table has a 255-character limit). When using image styles, the effective limit is significantly lower due to folders added to the path.
With image fields which reference images stored in S3, if you don't have width and height to set on the field at entity creation time, you'll want to implement a caching solution similar to the above.

Acknowledgements

Apart from the image style issues, most of the direct development detailed above was mine, but as on any project thoughts were bounced off the team, project managers handled communication with the client, testers provided feedback, etc. Thanks to the whole team, particularly Sean Blommaert (image styles, post feedback), Kevin Thompson (client communications), and Karoly Negyesi (post feedback).

Tags

Migration

Drupal

Planet Drupal

PHP

Use the Twitter thread below to comment on this post:

https://t.co/erY3Gvhd97
— Virtuoso Performance (@VirtPerformance) January 9, 2019

Migrating from an OAuth2 authenticated JSON feed

mikeryan — Mon, 04 Jun 2018 15:24:26 +0000

Migrating from an OAuth2 authenticated JSON feed mikeryan Monday, June 4, 2018 - 10:24am

Continuing with techniques from the “Acme” project, another ongoing feed I implemented was import from a JSON feed protected by OAuth2 authentication into “doctor” nodes. Let’s look first at the community contributions we needed to implement this.

Community contributions

Provide authentication plugins to HTTP fetcher - Moshe Weitzman had already suggested (and provided a patch for) adding basic and digest authentication to the HTTP fetcher plugin. I broadened the scope to add an Authentication plugin type, and implemented an OAuth2 authentication plugin.

Implement xpath-like selectors for the JSON parser - The JSON parser, from Karen Stevenson’s original JSON source plugin, used a numeric depth to retrieve data elements. The JSON feed we had here did not work with that approach, because at the top level in addition to the array containing our data was another array (and the depth approach would draw from both arrays). Implementing a means to select fields with a /-separated syntax made this much more flexible.

Project implementation

So, let’s look at the source plugin implementation:

source:
  plugin: url
  # We want to reimport any doctors whose source data has changed.
  track_changes: true
  # Counting the available records requires fetching the whole feed - cache the
  # counts to minimize overhead.
  cache_counts: true
  # Until https://www.drupal.org/project/drupal/issues/2751829 is fixed, this
  # should be used in conjunction with cache_counts in most cases. It was not
  # strictly necessary in this project because this was the only cached ‘url’
  # source plugin.
  cache_key: doctor
  data_fetcher_plugin: http
  data_parser_plugin: json
  item_selector: /providers
  # Note that the source .yml file does not contain the urls, or half the
  # authentication configuration - these are merged in using the configuration
  # UI (see http://virtuoso-performance.com/blog/mikeryan/configuring-migrations-form).
  # We present sample values here so you can see what the complete configuration
  # looks like.
  # The endpoint from which the data itself is fetched.
  urls: https://kservice.example2.com/providers
  # The http fetcher plugin calls the authentication plugin (if present),
  # which accepts plugin-specific configuration and returns the appropriate
  # authentication headers to add to the HTTP request.
  authentication:
    # migrate_plus also has ‘basic’ and ‘digest’ authentication plugins.
    plugin: oauth2
    # The grant type used by the feed (other grant types supported in theory,
    # but untested, are authorization_code, password, refresh_token, and
    # urn:ietf:params:oauth:grant-type:jwt-bearer.
    grant_type: client_credentials
    # The base URI for retrieving the token (provided through the UI).
    base_uri: https://kservice.example2.com
    # The relative URL for retrieving the token.
    token_url: /oauth2/token
    # The client ID for the service (provided through the UI).
    client_id: default_client_id
    # The client secret for the service (provided through the UI).
    client_secret: abcdef12345678

The ids and fields configuration operate as they do with other JSON and XML feeds I’ve blogged about.

Tags

Planet Drupal

Drupal

Migration

PHP

Use the Twitter thread below to comment on this post:

https://t.co/KJzBIauuVG
— Virtuoso Performance (@VirtPerformance) June 4, 2018

Disabling functionality temporarily during migration

mikeryan — Thu, 31 May 2018 15:25:45 +0000

Disabling functionality temporarily during migration mikeryan Thursday, May 31, 2018 - 10:25am

Continuing with techniques from the “Acme” project, the location content type had an address field and a geofield, with field_geofield configured to automatically determine latitude and longitude from the associated field_address - a fact I was initially unaware of. Our source data contained latitude and longitude already, which I mapped directly in the migration:

 field_geofield:
   plugin: geofield_latlon
   source:
     - latitude
     - longitude

However, testing location migrations by repeatedly running the import, I soon started getting messages from Google Maps API that my daily quota had been exceeded, and quickly tracked down the integration with field_address. Clearly, the calls out to Google Maps were both unnecessary and hazardous - how to prevent them? Fortunately, the migration system provides events which fire before and after each migration is executed. So, we subscribe to MigrateEvents::PRE_IMPORT to save the current settings and disable the external call:

public function onMigrationPreImport(MigrateImportEvent $event) {
   if ($event->getMigration()->id() == 'location') {
     $fields = \Drupal::entityTypeManager()->getStorage('field_config')->loadByProperties(['field_name' => 'field_geofield']);
     if ($fields) {
        /** @var \Drupal\field\Entity\FieldConfig $field */
        if ($field = $fields['node.location.field_geofield']) {
         $this->originalSettings = $field->getThirdPartySettings('geocoder_field');
         $field->setThirdPartySetting('geocoder_field', 'method', 'none');
         $field->save();
        }
     }
   }
 }

And we subscribe to MigrateEvents::POST_IMPORT to restore the original settings:

 public function onMigrationPostImport(MigrateImportEvent $event) {
   if ($event->getMigration()->id() == 'location') {
     $fields = \Drupal::entityTypeManager()->getStorage('field_config')->loadByProperties(['field_name' => 'field_geofield']);
     if ($fields) {
       /** @var \Drupal\field\Entity\FieldConfig $field */
        if ($field = $fields['node.location.field_geofield']) {
         foreach ($this->originalSettings as $key => $value) {
           $field->setThirdPartySetting('geocoder_field', $key, $value);
         }
         $field->save();
        }
     }
   }
 }

The thoughtful reader may note a risk here - what if someone were adding or editing a location node while this were running? The geofield would not be populated from the address field. In this case, this is not a problem - this is a one-time bulk migration (and no one should be making changes on a production website at such a time). In cases involving an ongoing feed where the feed data is used as-is on the Drupal site, it would also not be a problem, although if there were a practice of manually editing imported content there would be some risk.

Tags

Drupal

Planet Drupal

Migration

PHP

Use the Twitter thread below to comment on this post:

https://t.co/TRqRsWJPlA
— Virtuoso Performance (@VirtPerformance) May 31, 2018

Configuring migrations via a form

mikeryan — Wed, 23 May 2018 02:29:19 +0000

Configuring migrations via a form mikeryan Tuesday, May 22, 2018 - 09:29pm

Frequently, there may be parts of a migration configuration which shouldn’t be hard-coded into your YAML file - some configuration may need to be changed periodically, some may vary according to environment (for example, a dev environment may access a dev or test API endpoint, while prod needs to access a production endpoint), or you may need a password or other credentials to access a secure endpoint (or for a database source which you can’t put into settings.php). You may also need to upload a data file for input into your migration. If you are implementing your migrations as configuration entities (a feature provided by the migrate_plus module), all this is fairly straightforward - migration configuration entities may easily be loaded, modified, and saved based on form input, implemented in a standard form class.

Uploading data files

For this project, while other CSV source files were static enough to go into the migration module itself, we needed to periodically update the blog data during the development and launch process. A file upload field is set up in the normal way:

$form['acme_blog_file'] = [
 '#type' => 'file',
 '#title' => $this->t('Blog data export file (CSV)'),
 '#description' => $this->t('Select an exported CSV file of blog data. Maximum file size is @size.',
   ['@size' => format_size(file_upload_max_size())]),
];

And saved to the public file directory in the normal way:

$all_files = $this->getRequest()->files->get('files', []);
if (!empty($all_files['acme_blog_file'])) {
 $validators = ['file_validate_extensions' => ['csv']];
 if ($file = file_save_upload('acme_blog_file', $validators, 'public://', 0)) {

So, once we’ve got the file in place, we need to point the migration at it. We load the blog migration, retrieve its source configuration, set the path to the uploaded file, and save it back to active configuration storage.

   $blog_migration = Migration::load('blog');
   $source = $blog_migration->get('source');
   $source['path'] = $file->getFileUri();
   $blog_migration->set('source', $source);
   $blog_migration->save();
   drupal_set_message($this->t('File uploaded as @uri.', ['@uri' => $file->getFileUri()]));
   }
 else {
   drupal_set_message($this->t('File upload failed.'));
 }
}

It’s important to understand that get() and set() only operate directly on top-level configuration keys - we can’t simply do something like $blog_migration->set(‘source.path’, $file->getFileUri()), so we need to retrieve the whole source configuration array, and set the whole array back on the entity.

Endpoints and credentials

The endpoint and credentials for our event service are configurable through the same webform. Note that we obtain the current values from the event migration configuration entity to prepopulate the form:

$event_migration = Migration::load('event');
$source = $event_migration->get('source');
if (!empty($source['urls'])) {
 if (is_array($source['urls'])) {
   $default_value = reset($source['urls']);
 }
 else {
   $default_value = $source['urls'];
 }
}
else {
 $default_value = 'http://services.example.com/CFService.asmx?wsdl';
}

$form['acme_event'] = [
 '#type' => 'details',
 '#title' => $this->t('Event migration'),
 '#open' => TRUE,
];

$form['acme_event']['event_endpoint'] = [
 '#type' => 'textfield',
 '#title' => $this->t('CF service endpoint for retrieving event data'),
 '#default_value' => $default_value,
];

$form['acme_event']['event_clientid'] = [
 '#type' => 'textfield',
 '#title' => $this->t('Client ID for the CF service'),
 '#default_value' => @$source['parameters']['clientId'] ?: 1234,
];

$form['acme_event']['event_password'] = [
 '#type' => 'password',
 '#title' => $this->t('Password for the CF service'),
 '#default_value' => @$source['parameters']['clientCredential']['Password'] ?: '',
];

In submitForm(), we again load the migration configuration, insert the form values, and save:

$event_migration = Migration::load('event');
$source = $event_migration->get('source');
$source['urls'] = $form_state->getValue('event_endpoint');
$source['parameters'] = [
 'clientId' => $form_state->getValue('event_clientid'),
 'clientCredential' => [
   'ClientID' => $form_state->getValue('event_clientid'),
   'Password' => $form_state->getValue('event_password'),
 ],
 'startDate' => date('m-d-Y'),
];

$event_migration->set('source', $source);
$event_migration->save();
drupal_set_message($this->t('Event migration configuration saved.'));

Note that we also reset the startDate value while we’re at it (see the previous SOAP blog post).

Tags

Drupal

Planet Drupal

Migration

PHP

Use the Twitter thread below to comment on this post:

Configuring migrations via a form https://t.co/EZTiUKBazX
— Virtuoso Performance (@VirtPerformance) May 22, 2018

Importing specific fields with overwrite_properties

mikeryan — Tue, 15 May 2018 15:50:15 +0000

Importing specific fields with overwrite_properties mikeryan Tuesday, May 15, 2018 - 10:50am

While I had planned to stretch out my posts related to the "Acme" project, there are currently some people with questions about using overwrite_properties - so, I've moved this post forward.

By default, migration treats the source data as the system of record - that is, when reimporting previously-imported records, the expectation is to completely replace the destination side with fresh source data, discarding any interim changes which might have been made on the destination side. However, sometimes, when updating you may want to only pull specific fields from the source, leaving others (potentially manually-edited) intact. We had this situation with the event feed - in particular, the titles received from the feed may need to be edited for the public site. To achieve that, we used the overwrite_properties property on the destination plugin:

destination:
  plugin: 'entity:node'
  overwrite_properties:
    - 'field_address/address_line1'
    - 'field_address/address_line2'
    - 'field_address/locality'
    - 'field_address/administrative_area'
    - 'field_address/postal_code'
    - field_start_date
    - field_end_date
    - field_instructor
    - field_location_name
    - field_registration_price
    - field_remaining_spots
    - field_synchronized_title

When overwrite_properties is present, nothing changes when importing a new entity - but, if the destination entity already exists, the existing entity is loaded, and only the fields and properties enumerated in overwrite_properties will be, well, overwritten. In our example, note in particular field_synchronized_title - on initial import, both the regular node title and this field are populated from ClassName, but on updates only field_synchronized_title receives any changes in ClassName. This prevents any unexpected changes to the public title, but does make the canonical title from the feed available should an editor care to review and decide whether to modify the public title to reflect any changes.

Now, in this case we are creating the entities initially through this migration, and thus we know via the map table when a previously-migrated entity is being updated and thus overwrite_properties should be applied. Another use case is when the entire purpose of your migration is to update specific fields on pre-existing entities (i.e., not created by this migration). In this case, you need to map the IDs of the entities that are to be updated, otherwise the migration will simply create new entities. So, if you had a "nid_to_update" property in your source data, you would include

process:
  nid: nid_to_update

in your migration configuration. The destination plugin will then load that existing node, and only alter the specified overwrite_properties in it.

Tags

Drupal

Planet Drupal

Migration

PHP

Use the Twitter thread below to comment on this post:

Importing specific fields with overwrite_properties https://t.co/0H3W1Ll0ts
— Virtuoso Performance (@VirtPerformance) May 15, 2018

Drupal 8 migration from a SOAP API

mikeryan — Tue, 15 May 2018 15:12:00 +0000

Drupal 8 migration from a SOAP API mikeryan Tuesday, May 15, 2018 - 10:12am

Returning from my sabbatical, as promised I’m catching up on blogging about previous projects. For one such project, I was contracted by Acquia to provide migration assistance to a client of theirs [redacted, but let’s call them Acme]. This project involved some straightforward node migrations from CSV files, but more interestingly required implementing two ongoing feeds to synchronize external data periodically - one a SOAP feed, and the other a JSON feed protected by OAuth-based authentication. There were a number of other interesting techniques employed on this project which I think may be broadly useful and haven’t previously blogged about - all-in-all, there was enough to write about on this project that rather than compose one big epic post, I’m going to break things down in a series of posts, spread out over several days so as not to spam Planet Drupal. In this first post of the sequence, I’ll cover migration from SOAP. The full custom migration module for this project is on Gitlab.

A key requirement of the Acme project was to implement an ongoing feed, representing classes (the kind people attend in person, not the PHP kind), from a SOAP API to “event” nodes in Drupal. The first step, of course, was to develop (in migrate_plus) a parser plugin to handle SOAP feeds, based on PHP’s SoapClient class. This class exposes functions of the web service as class methods which may be directly invoked. In WSDL mode (the default, and the only mode this plugin currently supports), it can also report the signatures of the methods it supports (via __getFunctions()) and the data structures passed as parameters and returned as results (via __getTypes()). WSDL allows our plugin to do introspection and saves the need for some explicit configuration (in particular, it can automatically determine the property to be returned from within the response).

migrate_example_advanced (a submodule of migrate_plus) demonstrates a simple example of how to use the SOAP parser plugin - the .yml is well-documented, so please review that for a general introduction to the configuration. Here’s the basic source configuration for this specific project:

source:
  plugin: url
  # To remigrate any changed events.
  track_changes: true 
  data_fetcher_plugin: http # Ignored - SoapClient does the fetching itself.
  data_parser_plugin: soap
  # The method to invoke via the SOAP API.
  function: GetClientSessionsByClientId
  # Within the response, the object property containing the list of events.
  item_selector: SessionBOLExternal
  # Indicates that the response will be in the form of a PHP object.
  response_type: object
  # You won’t find ‘urls’ and ‘parameters’ in the source .yml file (they are inserted
  # by a web UI - the subject of a future post), but for demonstration purposes
  # this is what they might look like.
  urls: http://services.example.com/CFService.asmx?wsdl
  parameters:
    clientId: 1234
    clientCredential:
      ClientID: 1234
      Password: service_password
    startDate: 08-31-2016
  # Unique identifier for each event (section) to be imported, composed of 3 columns.
  ids:
    ClassID:
      type: integer
    SessionID:
      type: integer
    SectionID:
      type: integer
  fields:
    -
      name: ClientSessionID
      label: Session ID for the client
      selector: ClientSessionID
    ...

Of particular note is the three-part source ID defined here. The way this data is structured, a “class” contains multiple “sessions”, which each have multiple “sections” - the sections are the instances that have specific dates and times, which we need to import into event nodes, and we need all three IDs to uniquely identify each unique section.

Not all of the data we need for our event nodes is in the session feed, unfortunately - we want to capture some of the class-level data as well. So, while, the base migration uses the SOAP parser plugin to get the session rows to migrate, we need to fetch the related data at run time by making direct SOAP calls ourselves. We do this in our subscriber to the PREPARE_ROW event - this event is dispatched after the source plugin has obtained the basic data per its configuration, and gives us an opportunity to retrieve further data to add to the canonical source row before it enters the processing pipeline. I won’t go into detail on how that data is retrieved since it isn’t relevant to general migration principles, but the idea is since all the class data is not prohibitively large, and multiple sessions may reference the same class data, we fetch it all on the first source row processed and cache it for reference by subsequent rows.

Community contributions

SOAP Source plugin - Despite the title (from the original feature request), it was implemented as a parser plugin.

Altering migration configuration at import time - the PRE_IMPORT event

Our event feed permits filtering by the event start date - by passing a ‘startDate’ parameter in the format 12-31-2016 to the SOAP method, the feed will only return events starting on or after that date. At any given point in time we are only interested in future events, and don’t want to waste time retrieving and processing past events. To optimize this, we want the startDate parameter in our source configuration to be today’s date each time we run the migration. We can do this by subscribing to the PRE_IMPORT event.

In acme_migrate.services.yml:

services:
 ...
 acme_migrate.update_event_filter:
   class: Drupal\acme_migrate\EventSubscriber\UpdateEventFilter
   tags:
     - { name: event_subscriber }

In UpdateEventFilter.php:

class UpdateEventFilter implements EventSubscriberInterface {

 /**
  * {@inheritdoc}
  */
 public static function getSubscribedEvents() {
    $events[MigrateEvents::PRE_IMPORT] = 'onMigrationPreImport';
    return $events;
 }

The migration system dispatches the PRE_IMPORT event before the actual import begins executing. At that point, we can insert the desired date filter into the migration configuration entity and save it:

 /**
  * Set the event start date filter to today.
  *
  * @param \Drupal\migrate\Event\MigrateImportEvent $event
  * The import event.
  */
 public function onMigrationPreImport(MigrateImportEvent $event) {
   // $event->getMigration() returns the migration *plugin*.
   if ($event->getMigration()->id() == 'event') {
     // Migration::load() returns the migration *entity*.
     $event_migration = Migration::load('event');
     $source = $event_migration->get('source');
     $source['parameters']['startDate'] = date('m-d-Y');
     $event_migration->set('source', $source);
     $event_migration->save();
   }
 }

Note that the entity get() and set() functions only operate directly on top-level configuration properties - we can’t get and set, for example ‘source.parameters.startDate’ directly. We need to retrieve the entire source configuration, modify our one value within it, and set the entire source configuration back on the migration.

Tags

Drupal

Planet Drupal

Migration

PHP

Use the Twitter thread below to comment on this post:

Drupal 8 migration from a SOAP API https://t.co/hf8LGiATsh
— Virtuoso Performance (@VirtPerformance) May 15, 2018