Soong

Soong 0.7.0 released

mikeryan — Fri, 05 Jul 2019 21:05:10 +0000

Soong 0.7.0 released mikeryan Friday, July 5, 2019 - 04:05pm

The 0.7.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.6.0 include:

The property abstractions (PropertyInterface, PropertyFactory, and implementations) have been removed - it seems like the abstractions will only get in the way of leveraging PHP 7.x's improved native type-checking.
The Record Transformer concept has been introduced. While our experience with Drupal emphasized the mapping of properties within each record, the logical purpose of the transformation segment of an ETL pipeline is to transform a record at a time (stipulating that, yes, 90+% of the time you're individually transforming each property within the record)...
...thus one of the two RecordTransformer implementations provided out-of-the-box is the PropertyMapper. The field mapping configuration we previously provided within the transform key is now the PropertyMapper's configuration, and the classes are now implementations of PropertyTransformer rather than Transformer.
The other provided RecordTransformer is Copy, for bulk-copying properties directly from the source record to the destination record. In many instances, most if not all properties being migrated are retaining exactly the same name and content, so it can help clarify the "interesting" property transformations if you don't have to individual express each and every trivial property copy.

This release was meant to be more ambitious (including refactoring of task pipelines) and sooner, but I've been busy working a couple of contracts since May. So, for the forseeable future, progress will be slower (unless/until other folks start contributing, of course!). For now, the emphasis for 0.8.0 is task refactoring, starting with seeing if we can leverage other libraries like league/pipeline.

Tags

Migration

Soong

PHP

Use the Twitter thread below to comment on this post:

A new release of Soong ETL is out! https://t.co/9CgXWl5vHx
— Virtuoso Performance (@VirtPerformance) July 5, 2019

Soong 0.6.0 released

mikeryan — Wed, 01 May 2019 20:40:18 +0000

Soong 0.6.0 released mikeryan Wednesday, May 1, 2019 - 03:40pm

The 0.6.0 release of the Soong ETL library is now available on Packagist. Key changes since 0.5.3 include:

The Filter component has been introduced, which examines a Record and "approves" it for further processing, along with a filters configuration option added to extractors to limit what gets extracted. The Select filter and migrate command option --select have been added - extractor results may thus be filtered either in the base configuration for the extractor (representing the canonical set of data to be processed) or at runtime (for testing). An example of the latter, restricting input to records that have both a low id and a high foo value:

bin/soong migrate arraytosql --select='id<8' --select='foo>g'
Dependency injection has been improved - where previously class names were passed through to be instantiated by the components that needed them, responsibility for constructing all (almost, see below) components now belongs to the application (for now, the Symfony console commands being the single example) which will inject the class instances.
Since components do need to dynamically generate Property and Record instances during migration, in those cases a PropertyFactory or RecordFactory instance is injected.
The DataProperty and DataRecord classes have been renamed to Property and Record respectably.
Error handling has been beefed up, with several exception classes added.
We're now integrated with Scrutinizer for coverage and quality analysis. Unfortunately, Scrutinizer does not yet allow public access to analysis for Gitlab-based projects, so you can't look for yourself - but here's where we are now (having built out the tests further):

At the moment, the main emphases for 0.7.0 are:

Build out the console commands to be more-or-less feature-complete.
Refactor the transformation pipeline.
Refactor our approach to tasks, in particular looking at using or gaining inspiration from other tools such as Robo or PortPHP.

Obligatory self-promotion - I'm available for data migration projects.

Thanks!

Tags

Migration

Soong

PHP

Use the Twitter thread below to comment on this post:

https://t.co/Lk9zVdhwQM
— Virtuoso Performance (@VirtPerformance) May 1, 2019

Update on Soong ETL

mikeryan — Tue, 26 Feb 2019 17:26:01 +0000

Update on Soong ETL mikeryan Tuesday, February 26, 2019 - 11:26am

It's been over a month now since I made the Soong ETL library publicly available - about time for some updates.

One focus has been fleshing out areas that will aid in contribution. These include:

(Too) early I had split things out into a myriad of repositories, a case of premature optimization. I merged things back to a monorepo for now - as APIs are still fluid, it's much easier to make API changes in one repo and keep all the components in sync. Once the APIs are reasonably stable (at least, say, at "beta" level) at the very least specialized integrations like Csv and DBAL will move to their own repos. Although I initially imagined soong/soong including only the interfaces (and maybe some base classes), I'm now thinking it should also hold basic implementations of at least the Data, KeyMap, and Task interfaces.
Adding tests for existing code (still in progress), in particular adding base classes corresponding to the component interfaces to ease testing that implementations of those interfaces behave consistently and in accordance with the contracts. No new functionality will be added without tests.
Putting documentation up on Read the Docs - in particular, fleshing out the code documentation and generating it with Doxygen, and providing more information on contributing.

Priorities now are:

Seeking more participation from other developers (hi out there!). And, by the way, I'll be at Midwest PHP next week, my first non-Drupal PHP conference - any Midwesterners interested in data migration, hit me up and we'll talk!
Looking at other ETL systems for ideas - we have looked at PortPHP so far, which has some ideas we might borrow (and perhaps we can even integrate its readers).
Addressing any proposed changes that would break the existing API. Note that in the current 0.x.x stream, minor versions (e.g., 0.4.0) are API breakers, so be sure to pin any applications using Soong to the minor version ("~0.4.0" constraint in composer.json).

Coalescing those three priorities - if you have any interest in data migration in PHP, please stop by and offer your thoughts on the architecture!

Oh, by the way - I'm currently between projects, so if you need a data migration process implemented please contact me. I will (for now) take a reduced rate for a project using Soong, as there's nothing like a real-world application to take a general-purpose library to the next level.

Thanks!

Tags

PHP

Migration

Soong

Use the Twitter thread below to comment on this post:

https://t.co/zNhHXmbO8P
— Virtuoso Performance (@VirtPerformance) February 26, 2019

Announcing the Soong project - developing a general-purpose ETL framework

mikeryan — Tue, 22 Jan 2019 20:10:33 +0000

Announcing the Soong project - developing a general-purpose ETL framework mikeryan Tuesday, January 22, 2019 - 02:10pm

I'd like to invite members of the open-source community, particularly (but not exclusively) those involved with PHP, to join in designing and developing a general-purpose ETL framework for data migration. The vendor name for packaging components of this project is soong, and git repos for existing components are under the GitLab account "soongetl".

Note: Finally having finished composition of this lengthy monologue, it's clear to me that it's very ambitious (some might say arrogant) of me to write with the expectation that this will grow into a large and robust open-source ecosystem. Very well - it is ambitious, and the effort may very well fall flat on its face. C'est la vie...

Who am I?

I'm Mike Ryan - a lot of people in the Drupal community know me, but not so much the wider open-source community. Almost eleven years ago at a Drupal meetup in Boston, amongst general agreement that everyone hates to do data migration, Moshe Weitzman looked across the table at me and said "there's an opportunity here." Since then data migration into Drupal has been the primary focus of my professional life, first in partnership with Moshe, then as an Acquia employee and finally as a solo consultant. Over the years I've created several migration-related contrib modules for Drupal, was part of the team integrating migration into Drupal core for D8, and have been involved in dozens of real-world migration projects.

Why am I doing this?

I think we can do better

Just within Drupal, the migration framework can be improved:

Each step of Drupal migration support has been a port of the previous - from the hook-based Drupal 6 version, to the inheritence-and-composition model in Drupal 7, to the plugin-based system in Drupal 8, technical debt has accumulated. I've wanted for a while to start over with a clean slate - given my experience (and others), what would we do differently starting from scratch? Can we step back and re-examine the assumptions we've been carrying forward?
At a specific technical level, the biggest itch I've wanted to scratch is decoupling the components. Within the migration system as it is in D8 today, pretty much every component knows everything about every other component. At one point we had a destination plugin which was using some of the migration's source plugin configuration - that one made my eye twitch!
There's also the coupling of the migration system with Drupal - in particular, migration classes *are* Drupal plugins (i.e., their interfaces extend PluginInspectionInterface) rather than being *managed by* Drupal plugins. I would like to see migration classes be all about migration, rather than worry about being plugins as well. And once the basic migration classes are no longer Drupal plugins, then it's a small step to them being entirely independent of Drupal...

The larger PHP community

With Drupal 8, we’ve often talked about “getting off the island” in terms of benefiting from much fine PHP work done outside of the Drupal community. We haven’t talked so much about going in the opposite direction - making our own fine work available for use beyond Drupal. To my knowledge, the only published example of this so far is Kris Vanderwater (EclipseGC) with the plugin library.

Likewise, we Drupal developers don’t have a monopoly on good migration ideas - by moving the general-purpose aspects of migration into a separate open source project, we have the opportunity to benefit from new ideas and new talent.

The community we build

The major key to success for any large open-source project like this is a thriving community. After seeing open-source projects like Drupal grow organically - and face growing pains as they find themselves dealing with community problems reactively rather than proactively - if a community does form around this project, I would like to establish a supportive and welcoming tone from the beginning.

Diversity in particular remains an issue in the tech industry in general, and open-source especially - and a lack of diversity is difficult to correct after the fact. In building a community around this framework, my hope is that we draw a diverse set of developers in the beginning, in the hopes that seeding the garden well will be, if not self-sustaining, at least more sustainable. How to do that, I'm not certain - a concerted outreach effort could easily end up looking like Pokemon Go, searching for unique creatures to collect. Apart from starting with a good Code of Conduct, I'm open to suggestions!

Another aspect of community-building is providing opportunities for relative novices (whether new to open-source development, new to PHP, or new to migration). The proposed architecture involves myriad small, well-focused packages - an extractor here, a set of related transformers there, integrations for specific frameworks and APIs... Individual transformers, in particular, will generally be very simple. This ecosystem thus will provide ample opportunities for novices to gain experience with mentorship and also establish an online presence.

Now, all that being said, what about The Ethics of Unpaid Labor and the OSS Community (also see the recent Twitter discussion in the Drupal community)? In reaching out to underrepresented groups and to novices, we are reaching out to the people who have the least ability to work on open source for free. One way to ameliorate this effect may be to explicitly try to draw in students - whether in formal programs or teaching themselves software development - who will benefit from some free practical education and mentorship. Down the road, if this framework does start being adopted in real-world applications, we can look at ways to get sponsorships for people who maintain projects within the ecosystem. At any rate, as the community here grows I expect this will be an ongoing conversation.

Selfishness

Yes, I'm willing to cop to selfish reasons to pursue this.

Simple ego: I'm proud of the work I've done on migration in Drupal, and think it can be useful on a larger stage. Being old enough to see retirement on the horizon, I admit I'm thinking of this as my magnum opus - the last major contribution I make to open source. I would love to leave behind a significant piece of quality software with a vital community behind it.
Money: I've done fine as a Drupal data migration specialist. I hope to do better by expanding my market beyond Drupal, working on a wider variety of migration projects. Yes, retirement is on the horizon but, given earlier attempts at consulting which went less well than my "migration period" has, my funds put that horizon farther out than I'd like...

What's done so far?

Early last year I started playing around with a proof-of-concept in a single repo, getting a single basic ETL migration scenario running with a decoupled class structure based on the basic architecture of the Drupal migration system. Much of the work after getting the initial POC running was figuring out appropriate boundaries between components, and gradually introducing features beyond the most basic ones I started with. And then breaking pieces out into separate source repos, and figuring out those boundaries.

My role

This will certainly change according to the number and skills of contributors who join into this effort (assuming there are some!), but what I'm aiming for in terms of my own role:

Primary architect of version 1 of Soong. This would mean being the primary maintainer of architecture documentation and the repository of central interfaces/base classes. Per "selfishness" above - I have an architectural vision I want to see brought to fruition. Others may take it in different directions after that, but V1 is mine! tl;dr - I don't want to be BDFL; I do want to be BDF1.
Community leader. Per "community" above, I have a vision for building a diverse and vibrant open-source community from the ground up. Unlike the technical architecture, however, this plays less to my strengths, so I will be happy to defer as better-suited people show leadership in the community.
Mentorship. I'd like to help people up their development skills, their open-source involvement, and their understanding of the pits and perils of data migration.

Why did it take me so long?

After having it in the back of my head for a few years, I finally started creating repos and putting my thoughts into actual interfaces and classes several months ago. Why did I wait until now to share my work with the larger community? I certainly felt seen when I read this:

https://twitter.com/jessfraz/status/1063425181509652481

Frankly, there's an element of imposter syndrome here - I wanted to be sure I wasn't exposing any dumb ideas! Well, enough of that - instead, I now stipulate that you will find dumb things I did here, and ask that you help smartify them.

The architecture itself

There's a ways to go documenting the architecture as it is currently implemented in soong/soong, but right now it broadly looks like this:

A Task accepts configuration defining a migration process, and implements operations - most notably migrate, but it may also support other operations like rollback, status, analyze, … The following steps describe the migrate operation.
The task constructs the configured Extractor, which obtains data from a source such as a SQL query, a CSV file, an XML/JSON API, etc.
Iterating over the extractor returns one DataRecord (collection of named DataProperty instances) at a time containing source data. The task creates an empty DataRecord representing the destination data.
The task configuration defines a transform pipeline keyed by destination property names. For each of these properties, a sequence of one or more Transformer classes with corresponding configuration is invoked to determine the destination property value - usually, the first one will be configured to accept one or more source property names, and the results will be fed to subsequent transformers, with the final result assigned to the named property in the destination DataRecord.
The destination DataRecord is passed to the configured Loader to be loaded into the destination store - a SQL database, a CSV file, etc.
If an optional KeyMap is configured within the task, it is used to store the mapping from the source record's unique key to the destination record's unique key. This enables keyed relationships to be maintained even if keys change when migrating, as well as enabling rollback.

To try out a couple of working demos, git clone git@gitlab.com:soongetl/soong.git and follow the README.

Initial technical priorities

One of those infamous hard problems in computer science is naming things. Before we go too far, let's figure out how best to name things - I think Extractor/Transformer/Loader are pretty solid, but let's discuss whether other components (like Task) could use better names. Also, let's decide what naming conventions for implementations should look like - e.g., should CSV extractor and loader classes both be named CSV (or for that matter, Csv) with namespaces alone distinguishing them, or should they be CSVExtractor and CSVLoader?
The initial architecture, as I've said before, comes from my narrow experience in Drupal. I'm sure there are plenty of other good migration ideas out there - maybe there's even a package I've missed that's good enough that this effort would better be directed towards improving it rather than starting from scratch. I did do some research last year and did not find any PHP ETL packages that appeared to have wide adoption or as much flexibility, but with more eyes on it (eyes that have seen more beyond Drupal than I have) let's see if we can do a thorough review of prior art and see if there are some good ideas which may influence this effort. And let's look beyond PHP as well - are there ETL frameworks written in other object-oriented languages which may provide some architectural inspiration?
Review the boilerplate for Soong code repos (based on https://github.com/thephpleague/skeleton) - let's go over what we've got there (especially the code of conduct and contributing guidelines).
Test all the things! Before adding new stuff, we need to add tests for the existing components, and set up automated testing on Gitlab.

Technical goals

For V1, require PHP 7.1 and leverage strict type checking. I expect future versions to require PHP 7.4 and leverage typed object properties.
The central interface package soong/soong ideally should not depend on anything other than PSR interfaces. It should be approached as if it were a PSR itself - a completely general interface for ETL functionality not dependent on any non-standard interfaces.

Conclusion

Again, I know I am getting way ahead of myself here by imagining an active open-source community will quickly spring up here. I have talked to Drupal people about my ideas on occasion, and I expect there will be some interest there, but I very much hope other open-source developers can join this effort and provide different perspectives. I do believe strongly that a standard ETL library with a core of simple standard interfaces (making a simple move-my-stuff-from-here-to-there application a breeze) plus the flexibility to build complex systems to handle many types of data will be extremely valuable across many domains.

If I may try your patience a bit longer - I've spent a substantial portion of my time since my last contract pulling these thoughts together, and I am now in need of paid work (contact me if you need some data migration done!). I may fantasize about being sponsored to work fulltime on Soong, or be hopeful there's someone with a project that they think will benefit from Soong and thus I can make progress here in the course of solving their migration problem. Realistically, my next contract (or employment) most likely will not involve Soong development, so once I'm working I won't have as much time to manage this project - let's hope plenty of people join in to pick up my slack!

If you've made it this far, thank you for your time and I look forward to your merge requests!

Tags

Drupal

Planet Drupal

PHP

Migration

Soong

Use the Twitter thread below to comment on this post:

https://t.co/0uVJ13t8md
— Virtuoso Performance (@VirtPerformance) January 22, 2019