Update on Soong ETL

It's been over a month now since I made the Soong ETL library publicly available - about time for some updates.

One focus has been fleshing out areas that will aid in contribution. These include:

  1. (Too) early I had split things out into a myriad of repositories, a case of premature optimization. I merged things back to a monorepo for now - as APIs are still fluid, it's much easier to make API changes in one repo and keep all the components in sync. Once the APIs are reasonably stable (at least, say, at "beta" level) at the very least specialized integrations like Csv and DBAL will move to their own repos. Although I initially imagined soong/soong including only the interfaces (and maybe some base classes), I'm now thinking it should also hold basic implementations of at least the Data, KeyMap, and Task interfaces.

  2. Adding tests for existing code (still in progress), in particular adding base classes corresponding to the component interfaces to ease testing that implementations of those interfaces behave consistently and in accordance with the contracts. No new functionality will be added without tests.

  3. Putting documentation up on Read the Docs - in particular, fleshing out the code documentation and generating it with Doxygen, and providing more information on contributing.

Priorities now are:

  1. Seeking more participation from other developers (hi out there!). And, by the way, I'll be at Midwest PHP next week, my first non-Drupal PHP conference - any Midwesterners interested in data migration, hit me up and we'll talk!

  2. Looking at other ETL systems for ideas - we have looked at PortPHP so far, which has some ideas we might borrow (and perhaps we can even integrate its readers).

  3. Addressing any proposed changes that would break the existing API. Note that in the current 0.x.x stream, minor versions (e.g., 0.4.0) are API breakers, so be sure to pin any applications using Soong to the minor version ("~0.4.0" constraint in composer.json).

Coalescing those three priorities - if you have any interest in data migration in PHP, please stop by and offer your thoughts on the architecture!

Oh, by the way - I'm currently between projects, so if you need a data migration process implemented please contact me. I will (for now) take a reduced rate for a project using Soong, as there's nothing like a real-world application to take a general-purpose library to the next level.


Use the Twitter thread below to comment on this post: