DrupalCon NOLA migration sprint summary

DrupalCon NOLA is over - perhaps too soon for those of us who love this great city, or perhaps just soon enough for those who have given Drupal and the city of New Orleans every last ounce of energy they had this past week.

We had a very productive week sprinting for migration - it may not be reflected in the number of commits made by the end of the extended sprints, but we'll see the fruits over the coming weeks. Our focus was on triaging the issue queue, and on diving deeply into some of the more complex outstanding issues, rather than picking the low-hanging fruit.

Triage

I previously wrote about the upfront triage the first Sunday - I prepped a Google sheet with various breakdowns of outstanding core migration system issues, and with xjm and alexpott leading the way we got about halfway through the "Migrate critical" issues. We picked up with the other half on Friday, with a few more people attending - Ryan Weal, ultimike, penyaskito, vasi, and a few more gathered around the table. Because we were working through from the oldest created issues to newest, the first round caught most of the hardest issues and the Friday session went quicker. As of Sunday morning, we were down to 18 open migrate-criticals, of which 14 were touched in the past week (11 in the previous 48 hours), so we've got momentum going.

Specific issues addressed at NOLA

I had meant to cover everything worked on this week - but I had to cut it off, for space and time. This represents fewer than half the migration issues that saw work during DrupalCon.

Migrate D6 i18n nodes

This is the biggest outstanding issue at the moment. The problem here is quite complex - the representation of translated content has completely changed in Drupal 8. In particular, translations of a node are no longer separate nodes - all translations of a specific piece of content are represented by a single node, so merging the revision histories of multiple nodes into one is... challenging. vasi has come up with a clever solution here - my current project needs this, so I had real-world data to test against. This week:

vasi got the tests green
I rerolled the patch against 8.1.x (the issue is targeting 8.2.x, but my client is on 8.1.x at the moment).
bwinett performed a manual test - migrating from D6 to 8.1.1 (using my reroll) using the core upgrade UI - and successfully validated that the right translations and revision history were in the right place.
I performed a manual test with my client's data, with less successful results. Our scenario is a custom migration into a site containing manually-entered data, so we cannot preserve node and revision IDs. It turns out that the patch is assuming source revision IDs match the destination, when it attempts to consolidate multiple translations into a single revision.
For the patch to do that, it needs to perform a lookup in the migration map table given just a revision ID. The revision map table has a two-column key - revision ID and language - and the API (MigrateIdMapInterface) doesn't currently support partial lookups.
So, vasi opened an issue to extend the API and implemented a solution there, which is has now been committed.

At this writing, vasi is working on the next version of the node i18n patch using the idmap extension, and I'll be testing it against my client's data.

Migrate content type language settings from Drupal 6 & 7

Speaking of i18n, penyaskito, quietone (remotely), and vasi have moved this issue forward, rerolling the previous 8.0.x patch for 8.2.x and continuing to refine it. Next step (per penyaskito): Add an assertion that language_alterable is TRUE when there's no i18n_lock_node_TYPE.

Invalid passwords after D7 to D8 migration

alexpott took this one on and produced a patch which quietone has now manually verified - next step is complete code review.

Random failure in \Drupal\migrate_drupal_ui\Tests\d7\MigrateUpgrade7Test

A major annoyance with testing on drupal.org, this test has started randomly failing. This tests the upgrade UI, which is batched, and what happens per batch is non-deterministic (affected by timing/load), so it's difficult to track down. We had some discussion about this at the closing weekend Saturday extended sprint, and alexpott has been actively pursuing it.

Refactor EntityFile and use process plugins instead

I created a patch for this, which vasi and quietone have helped refine, and benjifisher applied CodeSniffer. Next step is for me to respond to benjy's review.

Destination bundle set in destination plugin, not in process

I've created a first draft of this patch, which the testbot tells me still needs work.

Reduce/remove tight coupling of migration destination plugins

quietone provided a suggested draft for the change record, and I tweaked the patch to deprecate rather than entirely remove md_entity:field_storage_config.

Cckfield Plugins must distinguish core versions

jmuzz has continued refining this patch.

Mentored sprint

The mentored core sprint (with mentors including heddn, vegantriathlete, and willwh) on Friday also contributed.

Fixes to docs and code style for migrate/src/*.php files

A few sprinters participated in rerolling the existing patch for 8.1.x (with some guidance from hussainweb): jeffrey.vargas, kurthill4, nJim, and The Sean.

Rename MigrationCreationTrait as it no longer creates migrations - it configures them

Participants in this patch include barbarae, hussainweb, douggreen, and hanoii.

Value should be null when is produced skip process

rakesh.gectcr submitted a reroll of this patch, cmanalansan also worked on it. This one is committed!

D6->D8: Migrating links without leading slash leads to fatal error

ancapaaron and bsnodgrass did manual testing on D6 and D7 respectively - looks like this is something we only need to worry about on D6.

Non-NOLA progress

Not everyone gets to go to every DrupalCon - while we fortunate ones were sprinting on-site, work progressed around the world...

Sql::getRowBySource doesn't adhere to MigrateIdMapInterface::getRowBySource

On the BC-breakers list but the NOLA sprinters didn't get to this - quietone working remotely rerolled the 7-month-old patch that was there, it's now ready for review.

Migrated custom block body field is hidden on form and display

rocketerrbkw identified this problem and has provided a patch - ready for review.

Last imported timestamps not set in map tables

milesw found a nasty "oops!" in a recent patch - looks good, just needs tests.

Variable to config: statistics.settings [d7]

vprocessor rerolled the existing patch and quietone reviewed.

Next steps

A lot of the work done at DrupalCon (not to mention a lot of work done previous to DrupalCon as well) needs review - let's get in there and see what we can move forward to RTBC.

I feel like we have a lot of momentum going now - but in my experience that DrupalCon momentum can dissipate pretty quickly. Let's keep it going - I'd like to propose ongoing migration sprints on a regular basis. Specifically:

Each sprint lasts a calendar day - during that day (according to your local calendar), those interested in pushing D8 migration forward try to find at least one chunk of time to work on migration issues, coordinating on IRC in #drupal-migrate.
For now, the theme will be Migrate-critical issues. Once we trim that list down to just a couple, we can pick different themes.
Since not every day of the week is equally open for everyone, let's rotate it - I suggest doing it every 8 days. So, Monday May 23 (8 days after the last DrupalCon extended sprint), Tuesday May 31, Wednesday June 8, etc...
I'll try to put out a blog post the day before each sprint day going over the status of key migrate-criticals.

Obligatory travelogue

I just couldn't keep up the daily blogging pace from mid-week on... Most of the meals have faded from my memory (and frankly I was finding better food earlier in the week than I was later), but I will leave you with my musical memories.

Wednesday night, sleep-deprived, I was heading back to my hotel on Esplanade when I ran into a couple of former Acquia colleagues on Royal. With very little arm-twisting, they convinced me to lead them to Frenchmen Street, where we picked a club pretty much at random (30°/-90°). It was shortly after 9pm and the band started up right after we got there, so I figured I'd stay for one set and still get a good night's sleep. It turned out they only played one set - and it lasted almost to 1am. And that band (Soul Company, with a somewhat different lineup than you see there) was simply awesome - high-energy jazz-funk (or funk-jazz?), very tight as they ran through covers of Herbie Hancock, Stevie Wonder, CCR/Ike & Tina, and many more. They closed with a medley of Kiss and Purple Rain, with a guest singer whose falsetto would have made Prince (R.I.P.) himself jealous. The late night pretty much wiped out my Thursday, but it was well worth it.

Saturday night I closed out the week with an actual plan - local legend John Boutté, (yes, I only knew him from Treme) was at d.b.a., so I made sure to get there good and early. This show was, unfortunately, a mixed bag - people gabbing loudly in the back of the room made it difficult to enjoy the performance, especially on the quieter tunes.

<rant>Who are these idiots who pay a cover for a show and then pay absolutely no attention to the artist!? WHY ARE YOU EVEN HERE!!!?</rant>

Sigh... Anyway, Boutté and his band (3+ piece horn section, guitar, and bass) did a fine job despite the distractions. Highlights were gentle takes on Time After Time and Hallelujah, with the emotional center being a combination of a nearly-a-cappella You've Got to Be Carefully Taught followed by a funky Southern Man.

And so, as my train departed Sunday, I left satisfied - a great week in and out of the convention center, can't wait until DrupalCon comes back here (hint, hint...)!