Migrating from an OAuth2 authenticated JSON feed

Continuing with techniques from the “Acme” project, another ongoing feed I implemented was import from a JSON feed protected by OAuth2 authentication into “doctor” nodes. Let’s look first at the community contributions we needed to implement this.

Community contributions

Provide authentication plugins to HTTP fetcher - Moshe Weitzman had already suggested (and provided a patch for) adding basic and digest authentication to the HTTP fetcher plugin. I broadened the scope to add an Authentication plugin type, and implemented an OAuth2 authentication plugin.

Implement xpath-like selectors for the JSON parser - The JSON parser, from Karen Stevenson’s original JSON source plugin, used a numeric depth to retrieve data elements. The JSON feed we had here did not work with that approach, because at the top level in addition to the array containing our data was another array (and the depth approach would draw from both arrays). Implementing a means to select fields with a /-separated syntax made this much more flexible.

Project implementation

So, let’s look at the source plugin implementation:

source:
  plugin: url
  # We want to reimport any doctors whose source data has changed.
  track_changes: true
  # Counting the available records requires fetching the whole feed - cache the
  # counts to minimize overhead.
  cache_counts: true
  # Until https://www.drupal.org/project/drupal/issues/2751829 is fixed, this
  # should be used in conjunction with cache_counts in most cases. It was not
  # strictly necessary in this project because this was the only cached ‘url’
  # source plugin.
  cache_key: doctor
  data_fetcher_plugin: http
  data_parser_plugin: json
  item_selector: /providers
  # Note that the source .yml file does not contain the urls, or half the
  # authentication configuration - these are merged in using the configuration
  # UI (see http://virtuoso-performance.com/blog/mikeryan/configuring-migrations-form).
  # We present sample values here so you can see what the complete configuration
  # looks like.
  # The endpoint from which the data itself is fetched.
  urls: https://kservice.example2.com/providers
  # The http fetcher plugin calls the authentication plugin (if present),
  # which accepts plugin-specific configuration and returns the appropriate
  # authentication headers to add to the HTTP request.
  authentication:
    # migrate_plus also has ‘basic’ and ‘digest’ authentication plugins.
    plugin: oauth2
    # The grant type used by the feed (other grant types supported in theory,
    # but untested, are authorization_code, password, refresh_token, and
    # urn:ietf:params:oauth:grant-type:jwt-bearer.
    grant_type: client_credentials
    # The base URI for retrieving the token (provided through the UI).
    base_uri: https://kservice.example2.com
    # The relative URL for retrieving the token.
    token_url: /oauth2/token
    # The client ID for the service (provided through the UI).
    client_id: default_client_id
    # The client secret for the service (provided through the UI).
    client_secret: abcdef12345678

The ids and fields configuration operate as they do with other JSON and XML feeds I’ve blogged about.