I'd like to invite members of the open-source community, particularly (but not exclusively) those involved with PHP, to join in designing and developing a general-purpose ETL framework for data migration. The vendor name for packaging components of this project is soong, and git repos for existing components are under the GitLab account "soongetl".
Continuing with techniques from the “Acme” project, another ongoing feed I implemented was import from a JSON feed protected by OAuth2 authentication into “doctor” nodes. Let’s look first at the community contributions we needed to implement this.
Continuing with techniques from the “Acme” project, the location content type had an address field and a geofield, with field_geofield configured to automatically determine latitude and longitude from the associated field_address - a fact I was initially unaware of. Our source data contained latitude and longitude already, which I mapped directly in the migration:
Frequently, there may be parts of a migration configuration which shouldn’t be hard-coded into your YAML file - some configuration may need to be changed periodically, some may vary according to environment (for example, a dev environment may access a dev or test API endpoint, while prod needs to access a production endpoint), or you may need a password or other credentials to access a secure endpoint (or for a database source which you can’t put into settings.php). You may also need to upload a data file for input into your migration.
Return to action
Those of you with an interest in Drupal migration may have noticed my absence in the last several months. A confluence of things led me to take a break from the Drupal community: a bout of physical exhaustion (initially diagnosed as Lyme disease, and then ¯\_(ツ)_/¯); professional exhaustion managing the D8 migration core initiative and maintaining a few contrib modules on top of paid contracts; and emotional exhaustion from the community drama of early last year.
I did make one comment on Dries’ blog in the immediate aftermath of learning about the situation which is roiling the Drupal community, but since then have taken some time to listen and ponder. The community is in deep pain now, and many of us are reacting to that pain with anger. Trust is in short supply. Healing seems nearly impossible.