Soong 0.7.0 released
The 0.7.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.6.0 include:
Soong 0.6.0 released
The 0.6.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.5.3 include:
Soong 0.5.2 released
The 0.5.2 release of the Soong ETL library is now available on Packagist. tl;dr:
- It's still too early to use in real-world applications - APIs continue to evolve.
- It's an excellent time to get involved in development, and help determine the direction we take!
Key changes include:
Update on Soong ETL
It's been over a month now since I made the Soong ETL library publicly available - about time for some updates.
One focus has been fleshing out areas that will aid in contribution. These include:
Announcing the Soong project - developing a general-purpose ETL framework
I'd like to invite members of the open-source community, particularly (but not exclusively) those involved with PHP, to join in designing and developing a general-purpose ETL framework for data migration. The vendor name for packaging components of this project is soong, and git repos for existing components are under the GitLab account "soongetl".
Drupal file migrations: The s3fs module
A recent project gave me the opportunity to familiarize myself with the Drupal 8 version of the S3 File System (s3fs) module (having used the D7 version briefly in the distant past). This module provides an s3:// stream wrapper for files stored in an S3 bucket, allowing them to be used as seamlessly as locally stored public and private files. First we present the migrations and some of the plugins implemented to support import of files stored on S3 - below we will go into some of the challenges we faced.
Migrating from an OAuth2 authenticated JSON feed
Continuing with techniques from the “Acme” project, another ongoing feed I implemented was import from a JSON feed protected by OAuth2 authentication into “doctor” nodes. Let’s look first at the community contributions we needed to implement this.
Disabling functionality temporarily during migration
Continuing with techniques from the “Acme” project, the location content type had an address field and a geofield, with field_geofield configured to automatically determine latitude and longitude from the associated field_address - a fact I was initially unaware of. Our source data contained latitude and longitude already, which I mapped directly in the migration:
Configuring migrations via a form
Frequently, there may be parts of a migration configuration which shouldn’t be hard-coded into your YAML file - some configuration may need to be changed periodically, some may vary according to environment (for example, a dev environment may access a dev or test API endpoint, while prod needs to access a production endpoint), or you may need a password or other credentials to access a secure endpoint (or for a database source which you can’t put into settings.php). You may also need to upload a data file for input into your migration.
Importing specific fields with overwrite_properties
While I had planned to stretch out my posts related to the "Acme" project, there are currently some people with questions about using overwrite_properties - so, I've moved this post forward.