10 KiB
Using process plugins for data transformation in Drupal migrations
In the previous chapter, we wrote our first Drupal migration. In that example, we copied verbatim values from the source to the destination. More often than not, the data needs to be transformed in some way or another to match the format expected by the destination system or to meet business requirements. Now we will learn more about process plugins and how they work as part of the Drupal migration pipeline.
Syntactic sugar
The Migrate API offers a lot of syntactic sugar to make it easier to write migration plugins. Field mappings in the process section are an example of this. Each of them requires a process plugin to be defined. If none is manually set, then the get plugin is assumed. The following two code snippets are equivalent in functionality.
process:
title: creative_title
process:
title:
plugin: get
source: creative_title
The get process plugin copies a value from the source to the destination without making any changes. Because this is a common operation, get is considered the default. There are many process plugins provided by Drupal core and contributed modules. Their configuration can be generalized as follows:
process:
destination_field:
plugin: plugin_name
config_1: value_1
config_2: value_2
config_3: value_3
The process plugin is configured within an extra level of indentation under the destination field. The plugin key is required and determines which plugin to use. Then, a list of configuration options follows. Refer to the documentation of each plugin to know what options are available. Some configuration options will be required while others will be optional. For example, the concat plugin requires a source, but the delimiter is optional. An example of its use appears later in this chapter.
Providing default values
Sometimes the destination requires a property or field to be set, but that information is not present in the source. Imagine you are migrating nodes. It is recommended to write one migration per content type. If you have a migration to nodes of type Basic page, it would be redundant to have a column in the source with the same value for every row. The data might not be needed. Or it might not exist. In any case, the default_value plugin can be used to provide a value when the data is not available in the source.
source: ...
process:
type:
plugin: default_value
default_value: page
destination:
plugin: entity:node
The above example sets the type property for all nodes in this migration to page, which is the machine name of the Basic page content type. Do not confuse the name of the plugin with the name of its configuration property as they happen to be the same: default_value. Also note that because type is manually set in the process section, the default_bundle key in the destination section is no longer required. You can see the latter being used in the example of the chapter 2. If type is defined in the process section and default_bundle in the destination section, the former takes precedence.
Concatenating values
Consider the following migration request: you have a source listing people with first and last name in separate columns. Both are capitalized. The two values need to be put together (concatenated) and used as the title of nodes of type Basic page. The character casing needs to be changed so that only the first letter of each word is capitalized. If there is a need to display them in all caps, CSS can be used for presentation. For example: FELIX DELATTRE would be transformed to Felix Delattre.
Tip: Question business requirements when they might produce undesired results. For instance, if you were to implement this feature as requested DAMIEN MCKENNA would be transformed to Damien Mckenna. That is not the correct capitalization for the last name McKenna. If automatic transformation is not possible or feasible for all variations of the source data, take note and perform manual updates after the initial migration. Evaluate as many use cases as possible and bring them to the client's attention.
To implement this feature, let's create a new module process_example. Inside its migrations folder, and write a migration plugin called process_example.yml. Download the sample module from https://www.drupal.org/project/migrate_examples For this example, we assume a Drupal installation using the standard installation profile which comes with the Basic Page content type. Let's see how to handle the concatenation of first and last name.
id: process_example
label: Process Plugins Example
source:
plugin: embedded_data
data_rows:
- unique_id: 1
first_name: FELIX
last_name: DELATTRE
- unique_id: 2
first_name: BENJAMIN
last_name: MELANÇON
- unique_id: 3
first_name: STEFAN
last_name: FREUDENBERG
ids:
unique_id:
type: integer
process:
type:
plugin: default_value
default_value: page
title:
plugin: concat
source:
- first_name
- last_name
delimiter: ' '
destination:
plugin: entity:node
The concat plugin can be used to glue together an arbitrary number of strings. Its source property contains an array of all the values that you want put together. The delimiter is an optional parameter that defines a string to add between the elements as they are concatenated. If not set, there will be no separation between the elements in the concatenated result. This plugin has an important limitation. You cannot use strings literals as part of what you want to concatenate. For example, joining the string Hello with the value of the first_name column. All the values to concatenate need to provided by the source plugin or fields already available in the process pipeline. It is possible to leverage a feature called source constants to concatenate string literals. We will talk about this in chapter !!!.
To execute the above migration, you need to enable the process_example module. Open a terminal, switch directories to your Drupal's webroot, and execute the following command: drush migrate:import process_example. If the migration fails, refer to the end of the chapter 2 for debugging information. If it works, you will see three basic pages whose title contains the names of some of my Drupal mentors. #DrupalThanks
Chaining process plugins
Good progress so far, but the feature has not been fully implemented. You still need to change the capitalization so that only the first letter of each word in the resulting title is uppercase. Thankfully, the Migrate API allows chaining of process plugins. This works similarly to unix pipelines in that the output of one process plugin becomes the input of the next one in the chain. When the last plugin in the chain completes its transformation, the return value is assigned to the destination field. Let's see this in action:
id: process_example
label: Process Plugins Example
source: ...
process:
type: ...
title:
- plugin: concat
source:
- first_name
- last_name
delimiter: ' '
- plugin: callback
callable: mb_strtolower
- plugin: callback
callable: ucwords
destination: ...
The callback process plugin passes a value to a PHP function and returns its result. The function to call is specified in the callable configuration option. The source option holds a value provided by the source plugin or one from process pipeline. That value is sent as the first argument to the function. Because we are using the callback plugin as part of a chain, the source is assumed to be the output of the previous plugin. Hence, there is no need to define a source. The example concatenates the first and last names, make them all lowercase, and then capitalize each word.
Relying on direct PHP function calls should be a last resort. Better alternatives include writing your own process plugins which encapsulate your business logic. The callback plugin is versatile. It can public methods in a PHP class. If the source is an array, it can be expanded to be so each element is passed as a different argument to the callable. Refer to the plugin documentation for various examples.
Tip: You should have a good understanding of your source and destination formats. In this example, one of the values to transform is MELANÇON. Because of the cedilla (ç) using strtolower is not adequate in this case. It would leave that character uppercase (melanÇon). Multibyte string functions (mb_*) are required for proper transformation. ucwords is not one of them and would present similar issues if the first letter of a word is special characters. Also, attention should be given to the character encoding of the tables in your destination database.
Technical note: mb_strtolower is a function provided by the mbstring PHP extension. It does not come enabled by default or you might not have it installed altogether. In those cases, the function would not be available when Drupal tries to call it. The following error is produced when trying to call a function that is not available: The "callable" must be a valid function or method. For this example, the error would never be triggered even if the extension is missing. That is because Drupal core depends on some Symfony packages which in turn depend on the symfony/polyfill-mbstring package. The latter provides a polyfill for mb_* functions that has been leveraged since version 8.6.x of Drupal.