Fix list spacing

This commit is contained in:
Mauricio Dinarte 2020-10-04 13:40:27 -06:00
parent c15d9ada31
commit 787c936caa
19 changed files with 202 additions and 202 deletions

10
11.txt
View file

@ -29,11 +29,11 @@ The `created`, *entity property* stores a [UNIX timestamp](https://en.wikipedia.
Back to the migration, you need to transform the provided date from `Month day, year` format to a UNIX timestamp. To do this, you use the [format_date](https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Plugin%21migrate%21process%21FormatDate.php/class/FormatDate) plugin. The `from_format` is set to `F j, Y` which means your source date consists of:
- The full textual representation of a month: `April`.
- Followed by a space character.
- Followed by the day of the month without leading zeros: `4`.
- Followed by a comma and another space character.
- Followed by the full numeric representation of a year using four digits: `2014`.
- The full textual representation of a month: `April`.
- Followed by a space character.
- Followed by the day of the month without leading zeros: `4`.
- Followed by a comma and another space character.
- Followed by the full numeric representation of a year using four digits: `2014`.
If the value of `from_format` does not make sense, you are not alone. It is actually assembled from format characters of the [date](https://www.php.net/manual/en/function.date.php) PHP function. When you need to specify the `from` and `to` formats, you basically need to look at the [documentation](https://www.php.net/manual/en/function.date.php#refsect1-function.date-parameters) and assemble a string that matches the desired date format. You need to pay close attention because upper and lowercase letters represent different things like `Y` and `y` for the year with four-digits versus two-digits respectively. Some date components have subtle variations like `d` and `j` for the day with or without leading zeros respectively. Also, take into account white spaces and date component separators. To finish the plugin configuration, you need to set the `to_format` configuration to something that produces a UNIX timestamp. If you look again at the documentation, you will see that `U` does the job.

8
12.txt
View file

@ -98,9 +98,9 @@ If you need the timezone to be dynamic, things get a bit harder. The 'from_timez
Date migrations can be tricky because they can be affected by things outside of the Migrate API. Here is a non-exhaustive list of things to consider:
- For *date and time* fields, the transformation might be affected by your server's timezone if you do not manually set the `from_timezone` configuration.
- People might see the date and time according to the preferences in their user profile. That is, two users might see a different value for the same migrated field if their preferred timezones are not the same.
- For *date only* fields, the user might see a time depending on the format used to display them. A list of available formats can be found at `/admin/config/regional/date-time`.
- A field can be configured to be presented in a specific timezone always. This would override the site's timezone and the user's preferred timezone.
- For *date and time* fields, the transformation might be affected by your server's timezone if you do not manually set the `from_timezone` configuration.
- People might see the date and time according to the preferences in their user profile. That is, two users might see a different value for the same migrated field if their preferred timezones are not the same.
- For *date only* fields, the user might see a time depending on the format used to display them. A list of available formats can be found at `/admin/config/regional/date-time`.
- A field can be configured to be presented in a specific timezone always. This would override the site's timezone and the user's preferred timezone.
What did you learn in today's blog post? Did you know that entity properties and date fields expect different destination formats? Did you know how to do timezone conversions? What challenges have you found when migrating dates and times? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

12
13.txt
View file

@ -94,9 +94,9 @@ field_ud_address/country_code: country
The mapping is relatively simple. You specify a value for each subfield. The tricky part is to know the *name of the subfield* and the *value to store in it*. The format for an address component can change among countries. The easiest way to see what components are expected for each country is to create a node for a content type that has an address field. With this example, you can go to `/node/add/ud_address` and try it yourself. For simplicity sake, let's consider only 3 countries:
- For USA, *city*, *state*, and *ZIP code* are all required. And for state, you have a specific list form which you need to select from.
- For Germany, the *company* is [moved above](https://github.com/google/libaddressinput/issues/83) *first and last name*. The ZIP code label changes to Postal code and it is required. The *city* is also required. It is not possible to set a *state*.
- For Nicaragua, the *Postal code* is optional. The *State* label changes to *Department*. It is required and offers a predefined list to choose from. The *city* is also required.
- For USA, *city*, *state*, and *ZIP code* are all required. And for state, you have a specific list form which you need to select from.
- For Germany, the *company* is [moved above](https://github.com/google/libaddressinput/issues/83) *first and last name*. The ZIP code label changes to Postal code and it is required. The *city* is also required. It is not possible to set a *state*.
- For Nicaragua, the *Postal code* is optional. The *State* label changes to *Department*. It is required and offers a predefined list to choose from. The *city* is also required.
Pay very close attention. The *available subfields* will depend on the *country*. Also, the form labels change per country or language settings. They do not necessarily match the subfield names. Moreover, the values that you see on the screen might not match what is stored in the database. For example, a Nicaraguan address will store the full *department* name like `Managua`. On the other hand, a USA address will only store a two-letter code for the *state* like `MA` for `Massachusetts`.
@ -108,9 +108,9 @@ Something else that is not apparent even from the user interface is data validat
Values for the same subfield can vary per country. How can you find out which value to use? There are a few ways, but they all require varying levels of technical knowledge or access to resources:
- You can inspect the source code of the address field widget. When the *country* and *state* components are rendered as select input fields (dropdowns), you can have a look at the `value` attribute for the `option` that you want to select. This will contain the two-letter code for *countries*, the two-letter abbreviations for USA *states*, and the fully spelled string for Nicaraguan *departments*.
- You can use the [Devel module](https://www.drupal.org/project/devel). Create a node containing an address. Then use the `devel` tab of the node to inspect how the values are stored. It is not recommended to have the `devel` module in a production site. In fact, do not deploy the code even if the module is not enabled. This approach should only be used in a local development environment. Make sure no module or configuration is committed to the repo nor deployed.
- You can inspect the database. Look for the records in a table named `node__field_[field_machine_name]`, if migrating nodes. First create some example nodes via the user interface and then query the table. You will see how Drupal stores the values in the database.
- You can inspect the source code of the address field widget. When the *country* and *state* components are rendered as select input fields (dropdowns), you can have a look at the `value` attribute for the `option` that you want to select. This will contain the two-letter code for *countries*, the two-letter abbreviations for USA *states*, and the fully spelled string for Nicaraguan *departments*.
- You can use the [Devel module](https://www.drupal.org/project/devel). Create a node containing an address. Then use the `devel` tab of the node to inspect how the values are stored. It is not recommended to have the `devel` module in a production site. In fact, do not deploy the code even if the module is not enabled. This approach should only be used in a local development environment. Make sure no module or configuration is committed to the repo nor deployed.
- You can inspect the database. Look for the records in a table named `node__field_[field_machine_name]`, if migrating nodes. First create some example nodes via the user interface and then query the table. You will see how Drupal stores the values in the database.
If you know a better way, please share it in the comments.

16
14.txt
View file

@ -51,10 +51,10 @@ The other configuration that you can optionally set in the *destination* section
You can execute the paragraph migration with this command: `drush migrate:import
ud_migrations_paragraph_intro_paragraph`. After running the migration, there is not much you can do to *verify that it worked*. Contrary to other entities, there is no user interface, available out of the box, that lists all paragraphs in the system. One way to verify if the migration worked is to manually create a [View](https://understanddrupal.com/articles/what-view-drupal-how-do-they-work) that shows paragraphs. Another way is to query the database directly. You can inspect the tables that store the paragraph fields' data. In this example, the tables would be:
- `paragraph__field_ud_book_paragraph_author` for the current author.
- `paragraph__field_ud_book_paragraph_title` for the current title.
- `paragraph_r__8c3a9563ac` for all the author revisions.
- `paragraph_r__3fa7e9863a` for all the title revisions.
- `paragraph__field_ud_book_paragraph_author` for the current author.
- `paragraph__field_ud_book_paragraph_title` for the current title.
- `paragraph_r__8c3a9563ac` for all the author revisions.
- `paragraph_r__3fa7e9863a` for all the title revisions.
Each of those tables contains information about the *bundle* (paragraph type), the *entity id*, the *revision id*, and the migrated *field value*. Table names are derived from the *machine names* of the fields. If they are too long, the field name will be hashed to produce a shorter table name. Having to query the database is not ideal. Unfortunately, the options available to check if a paragraph migration worked are limited at the moment.
@ -162,10 +162,10 @@ $ drush migrate:import --tag='UD Paragraphs Intro'
And that is *one way* to map paragraph reference fields. In the end, all you have to do is set the `target_id` and `target_revision_id` subfields. The process pipeline that gets you to that point can vary depending on how your paragraphs are configured. The following is a non-exhaustive list of things to consider when migrating paragraphs:
- How many paragraphs types can be referenced?
- How many paragraphs instances are being migrated? Is this a multivalue field?
- Do paragraphs have translations?
- Do paragraphs have revisions?
- How many paragraphs types can be referenced?
- How many paragraphs instances are being migrated? Is this a multivalue field?
- Do paragraphs have translations?
- Do paragraphs have revisions?
## Do migrated paragraphs disappear upon node rollback?

34
15.txt
View file

@ -26,10 +26,10 @@ In any migration project, understanding the source is very important. For CSV mi
This file will be used in the *node* migration. The four columns are used as follows:
- `unique_id` is the unique identifier for each record in this CSV file.
- `name` is the name of a person. This will be used as the node title.
- `photo_file` is the unique identifier of an image that was created in a separate migration.
- `book_ref` is the unique identifier of a book paragraph that was created in a separate migration.
- `unique_id` is the unique identifier for each record in this CSV file.
- `name` is the name of a person. This will be used as the node title.
- `photo_file` is the unique identifier of an image that was created in a separate migration.
- `book_ref` is the unique identifier of a book paragraph that was created in a separate migration.
The following snippet shows the configuration of the CSV *source* plugin for the *node* migration:
@ -132,27 +132,27 @@ process:
When setting the `path` configuration you have three options to indicate the CSV file location:
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/csv_files/example.csv`.
- Use an *absolute path* pointing to the CSV location in the file system. The path should start with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/csv_files/example.csv`.
- Use a [stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x). This feature was introduced in the 8.x-3.x branch of the module. Previous versions cannot make use of them.
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/csv_files/example.csv`.
- Use an *absolute path* pointing to the CSV location in the file system. The path should start with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/csv_files/example.csv`.
- Use a [stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x). This feature was introduced in the 8.x-3.x branch of the module. Previous versions cannot make use of them.
Being able to use stream wrappers gives you many options for setting the location to the CSV file. For instance:
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leverages functionality already available in Drupal core. For example: `public://csv_files/example.csv`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/csv_files/example.csv`.
- Files located in remote servers including RSS feeds. You can use the [Remote stream wrapper module](https://www.drupal.org/project/remote_stream_wrapper) to get this functionality. For example, `https://understanddrupal.com/csv-files/example.csv`.
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leverages functionality already available in Drupal core. For example: `public://csv_files/example.csv`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/csv_files/example.csv`.
- Files located in remote servers including RSS feeds. You can use the [Remote stream wrapper module](https://www.drupal.org/project/remote_stream_wrapper) to get this functionality. For example, `https://understanddrupal.com/csv-files/example.csv`.
## CSV source plugin configuration
The configuration options for the CSV source plugin are very well documented in the [source code](https://git.drupalcode.org/project/migrate_source_csv/blob/8.x-3.x/src/Plugin/migrate/source/CSV.php#L14). They are included here for quick reference:
- `path` is required. It contains the path to the CSV file. Starting with the 8.x-3.x branch, stream wrappers are supported.
- `ids` is required. It contains an array of column names that uniquely identify each record.
- `header_offset` is optional. The index of record to be used as the CSV header and the thereby each record's field name. It defaults to zero (`0`) because the index is zero-based. For CSV files with no header row the value should be set to `null`.
- `fields` is optional. It contains a nested array of names and labels to use instead of a header row. If set, it will overwrite the column names obtained from `header_offset`.
- `delimiter` is optional. It contains *one character* used as column delimiter. It defaults to a *comma* (**,**). For example, if your file uses *tabs* as delimiter, you set this configuration to `\t`.
- `enclosure` is optional. It contains *one character* used to enclose the column values. Defaults to *double quotation marks* (**"**).
- `escape` is optional. It contains *one character* used for character escaping in the column values. It defaults to a *backslash* (**\**).
- `path` is required. It contains the path to the CSV file. Starting with the 8.x-3.x branch, stream wrappers are supported.
- `ids` is required. It contains an array of column names that uniquely identify each record.
- `header_offset` is optional. The index of record to be used as the CSV header and the thereby each record's field name. It defaults to zero (`0`) because the index is zero-based. For CSV files with no header row the value should be set to `null`.
- `fields` is optional. It contains a nested array of names and labels to use instead of a header row. If set, it will overwrite the column names obtained from `header_offset`.
- `delimiter` is optional. It contains *one character* used as column delimiter. It defaults to a *comma* (**,**). For example, if your file uses *tabs* as delimiter, you set this configuration to `\t`.
- `enclosure` is optional. It contains *one character* used to enclose the column values. Defaults to *double quotation marks* (**"**).
- `escape` is optional. It contains *one character* used for character escaping in the column values. It defaults to a *backslash* (**\**).
**Important**: The configuration options changed significantly between the 8.x-3.x and 8.x-2.x branches. Refer to this [change record](https://www.drupal.org/node/3060246) for a reference of how to configure the plugin for the 8.x-2.x.

28
16.txt
View file

@ -76,10 +76,10 @@ In any migration project, understanding the *source* is very important. For JSON
The array of records containing node data lies two levels deep in the hierarchy. Starting with `data` at the root and then descending one level to `udm_people`. Each element of this array is an object with four properties:
- `unique_id` is the *unique identifier* for each record **within** the `/data/udm_people` hierarchy.
- `name` is the name of a person. This will be used in the node title.
- `photo_file` is the *unique identifier* of an image that was created in a separate migration.
- `book_ref` is the *unique identifier* of a book paragraph that was created in a separate migration.
- `unique_id` is the *unique identifier* for each record **within** the `/data/udm_people` hierarchy.
- `name` is the name of a person. This will be used in the node title.
- `photo_file` is the *unique identifier* of an image that was created in a separate migration.
- `book_ref` is the *unique identifier* of a book paragraph that was created in a separate migration.
The following snippet shows the configuration to read a *local* JSON file for the *node* migration:
@ -115,9 +115,9 @@ The `item_selector` configuration indicates where in the JSON file lies the *arr
`fields` has to be set to an *array*. Each element represents a field that will be made available to the migration. The following options can be set:
- `name` is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contains spaces, you need to put *double quotation marks* (**"**) around it when referring to it in the migration.
- `label` is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the [Migrate Tools module](https://www.drupal.org/project/migrate_tools). When defined, you **do not use** the *label* to refer to the field. Keep using the *name*.
- `selector` is required. This is another XPath-like string to find the field to import. The value must be relative to the location specified by the `item_selector` configuration. In the example, the fields are direct children of the records to migrate. Therefore, only the property name is specified (e.g., `unique_id`). If you had nested objects or arrays, you would use a *slash* (**/**) character to go deeper in the hierarchy. This will be demonstrated in the *image* and *paragraph* migrations.
- `name` is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contains spaces, you need to put *double quotation marks* (**"**) around it when referring to it in the migration.
- `label` is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the [Migrate Tools module](https://www.drupal.org/project/migrate_tools). When defined, you **do not use** the *label* to refer to the field. Keep using the *name*.
- `selector` is required. This is another XPath-like string to find the field to import. The value must be relative to the location specified by the `item_selector` configuration. In the example, the fields are direct children of the records to migrate. Therefore, only the property name is specified (e.g., `unique_id`). If you had nested objects or arrays, you would use a *slash* (**/**) character to go deeper in the hierarchy. This will be demonstrated in the *image* and *paragraph* migrations.
Finally, you specify an `ids` *array* of field *names* that would uniquely identify each record. As already stated, the `src_unique_id` field serves that purpose. The following snippet shows part of the *process*, *destination*, and *dependencies* configuration of the node migration:
@ -257,16 +257,16 @@ process:
When using the `file` data fetcher plugin, you have three options to indicate the location to the JSON files in the `urls` configuration:
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/json_files/example.json`.
- Use an *absolute path* pointing to the JSON location in the file system. The path *should start* with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/json_files/example.json`.
- Use a *fully-qualified URL* to any [built-in wrapper](https://www.php.net/manual/en/wrappers.php) like `http`, `https`, `ftp`, `ftps`, etc. For example, `https://understanddrupal.com/json-files/example.json`.
- Use a [custom stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x).
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/json_files/example.json`.
- Use an *absolute path* pointing to the JSON location in the file system. The path *should start* with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/json_files/example.json`.
- Use a *fully-qualified URL* to any [built-in wrapper](https://www.php.net/manual/en/wrappers.php) like `http`, `https`, `ftp`, `ftps`, etc. For example, `https://understanddrupal.com/json-files/example.json`.
- Use a [custom stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x).
Being able to use stream wrappers gives you many more options. For instance:
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leveragers functionality already available in Drupal core. For example: `public://json_files/example.json`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/json_files/example.json`.
- Files located in [AWS Amazon S3](https://aws.amazon.com/s3/). You can use the [S3 File System module](https://www.drupal.org/project/s3fs) along with the [S3FS File Proxy to S3 module](https://www.drupal.org/project/s3fs_file_proxy_to_s3) to get this functionality.
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leveragers functionality already available in Drupal core. For example: `public://json_files/example.json`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/json_files/example.json`.
- Files located in [AWS Amazon S3](https://aws.amazon.com/s3/). You can use the [S3 File System module](https://www.drupal.org/project/s3fs) along with the [S3FS File Proxy to S3 module](https://www.drupal.org/project/s3fs_file_proxy_to_s3) to get this functionality.
## Migrating remote JSON files

28
17.txt
View file

@ -85,10 +85,10 @@ In any migration project, understanding the source is very important. For XML mi
The *set of elements* containing node data lies two levels deep in the hierarchy. Starting with `data` at the root and then descending one level to `udm_people`. Each element of this array is an object with four properties:
- `unique_id` is the *unique identifier* for each element **returned by** the `data/udm_people` hierarchy.
- `name` is the name of a person. This will be used in the node title.
- `photo_file` is the *unique identifier* of an image that was created in a separate migration.
- `book_ref` is the *unique identifier* of a book paragraph that was created in a separate migration.
- `unique_id` is the *unique identifier* for each element **returned by** the `data/udm_people` hierarchy.
- `name` is the name of a person. This will be used in the node title.
- `photo_file` is the *unique identifier* of an image that was created in a separate migration.
- `book_ref` is the *unique identifier* of a book paragraph that was created in a separate migration.
The following snippet shows the configuration to read a *local* XML file for the *node* migration:
@ -131,9 +131,9 @@ The `item_selector` configuration indicates where in the XML file lies the *set
`fields` has to be set to an *array*. Each element represents a field that will be made available to the migration. The following options can be set:
- `name` is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contained spaces, you need to put *double quotation marks* (**"**) around it when referring to it in the migration.
- `label` is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the [Migrate Tools module](https://www.drupal.org/project/migrate_tools). When defined, you **do not use** the *label* to refer to the field. Keep using the *name*.
- `selector` is required. This is another XPath-like string to find the field to import. The value must be relative to the subtree specified by the `item_selector` configuration. In the example, the fields are direct children of the elements to migrate. Therefore, the XPath expression only includes the element name (e.g., `unique_id`). If you had nested elements, you could use a *slash* (**/**) character to go deeper in the hierarchy. This will be demonstrated in the *image* and *paragraph* migrations.
- `name` is required. This is how the field is going to be referenced in the migration. The name itself can be arbitrary. If it contained spaces, you need to put *double quotation marks* (**"**) around it when referring to it in the migration.
- `label` is optional. This is a description used when presenting details about the migration. For example, in the user interface provided by the [Migrate Tools module](https://www.drupal.org/project/migrate_tools). When defined, you **do not use** the *label* to refer to the field. Keep using the *name*.
- `selector` is required. This is another XPath-like string to find the field to import. The value must be relative to the subtree specified by the `item_selector` configuration. In the example, the fields are direct children of the elements to migrate. Therefore, the XPath expression only includes the element name (e.g., `unique_id`). If you had nested elements, you could use a *slash* (**/**) character to go deeper in the hierarchy. This will be demonstrated in the *image* and *paragraph* migrations.
Finally, you specify an `ids` *array* of field *names* that would uniquely identify each record. As already stated, the `unique_id` field servers that purpose. The following snippet shows part of the *process*, *destination*, and *dependencies* configuration of the node migration:
@ -289,16 +289,16 @@ process:
When using the `file` data fetcher plugin, you have three options to indicate the location to the XML files in the `urls` configuration:
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/xml_files/example.xml`.
- Use an *absolute path* pointing to the XML location in the file system. The path *should start* with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/xml_files/example.xml`.
- Use a *fully-qualified URL* to any [built-in wrapper](https://www.php.net/manual/en/wrappers.php) like `http`, `https`, `ftp`, `ftps`, etc. For example, `https://understanddrupal.com/xml-files/example.xml`.
- Use a [custom stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x).
- Use a *relative path* from the **Drupal root**. The path *should not start* with a *slash* (**/**). This is the approach used in this demo. For example, `modules/custom/my_module/xml_files/example.xml`.
- Use an *absolute path* pointing to the XML location in the file system. The path *should start* with a *slash* (**/**). For example, `/var/www/drupal/modules/custom/my_module/xml_files/example.xml`.
- Use a *fully-qualified URL* to any [built-in wrapper](https://www.php.net/manual/en/wrappers.php) like `http`, `https`, `ftp`, `ftps`, etc. For example, `https://understanddrupal.com/xml-files/example.xml`.
- Use a [custom stream wrapper](https://api.drupal.org/api/drupal/namespace/Drupal!Core!StreamWrapper/8.8.x).
Being able to use stream wrappers gives you many more options. For instance:
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leveragers functionality already available in Drupal core. For example: `public://xml_files/example.xml`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/xml_files/example.xml`.
- Files located in [AWS Amazon S3](https://aws.amazon.com/s3/). You can use the [S3 File System module](https://www.drupal.org/project/s3fs) along with the [S3FS File Proxy to S3 module](https://www.drupal.org/project/s3fs_file_proxy_to_s3) to get this functionality.
- Files located in the [public](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PublicStream.php/class/PublicStream/8.8.x), [private](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21PrivateStream.php/class/PrivateStream/8.8.x), and [temporary](https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21StreamWrapper%21TemporaryStream.php/class/TemporaryStream/8.8.x) file systems managed by Drupal. This leveragers functionality already available in Drupal core. For example: `public://xml_files/example.xml`.
- Files located in profiles, modules, and themes. You can use the [System stream wrapper module](https://www.drupal.org/project/system_stream_wrapper) or [apply](https://www.drupal.org/patch/apply) this [core patch](https://www.drupal.org/project/drupal/issues/1308152) to get this functionality. For example, `module://my_module/xml_files/example.xml`.
- Files located in [AWS Amazon S3](https://aws.amazon.com/s3/). You can use the [S3 File System module](https://www.drupal.org/project/s3fs) along with the [S3FS File Proxy to S3 module](https://www.drupal.org/project/s3fs_file_proxy_to_s3) to get this functionality.
## Migrating remote XML files

14
18.txt
View file

@ -20,10 +20,10 @@ The second fetcher plugin is `http`. Under the hood, it uses the [Guzzle HTTP Cl
*Data parsers* are responsible for processing the files considering their type: JSON, XML, or SOAP. These plugins let you select a subtree *within* the file hierarchy that contains the elements to be imported. Each record might contain more data than what you need for the migration. So, you make a second selection to manually indicate which elements will be made available to the migration. Migrate plus provides four data parses, but only two use the data fetcher plugins. Here is a summary:
- `json` can use any of the data fetchers. Offers an extra configuration option called `include_raw_data`. When set to true, in addition to all the `fields` manually defined, a new one is attached to the source with the name `raw`. This contains a copy of the full object currently being processed.
- `simple_xml` can use any data fetcher. It uses the [SimpleXML](https://www.php.net/manual/en/book.simplexml.php) class.
- `xml` does not use any of the data fetchers. It uses the [XMLReader](https://www.php.net/manual/en/class.xmlreader.php) class to directly fetch the file. Therefore, it is not possible to set HTTP headers or authentication.
- `soap` does not use any data fetcher. It uses the [SoapClient](https://www.php.net/manual/en/class.soapclient.php) class to directly fetch the file. Therefore, it is not possible to set HTTP headers or authentication.
- `json` can use any of the data fetchers. Offers an extra configuration option called `include_raw_data`. When set to true, in addition to all the `fields` manually defined, a new one is attached to the source with the name `raw`. This contains a copy of the full object currently being processed.
- `simple_xml` can use any data fetcher. It uses the [SimpleXML](https://www.php.net/manual/en/book.simplexml.php) class.
- `xml` does not use any of the data fetchers. It uses the [XMLReader](https://www.php.net/manual/en/class.xmlreader.php) class to directly fetch the file. Therefore, it is not possible to set HTTP headers or authentication.
- `soap` does not use any data fetcher. It uses the [SoapClient](https://www.php.net/manual/en/class.soapclient.php) class to directly fetch the file. Therefore, it is not possible to set HTTP headers or authentication.
The difference between `xml` and `simple_xml` were presented in the [previous article.](https://understanddrupal.com/articles/migrating-xml-files-drupal)
@ -31,9 +31,9 @@ The difference between `xml` and `simple_xml` were presented in the [previous ar
These plugins add *authentication* headers to the request. If correct, you could fetch data from protected resources. They work exclusively with the `http` data fetcher. Therefore, you can use them only with `json` and `simple_xml` data parsers. To do that, you set an `authentication` configuration whose value can be one of the following:
- `basic` for HTTP Basic authentication.
- `digest` for HTTP Digest authentication.
- `oauth2` for OAuth2 authentication over HTTP.
- `basic` for HTTP Basic authentication.
- `digest` for HTTP Digest authentication.
- `oauth2` for OAuth2 authentication over HTTP.
Below are examples for JSON and XML imports with HTTP headers and authentication configured. The code snippets do not contain real migrations. You can also find them in the `ud_migrations_http_headers_authentication` directory of the demo repository <https://github.com/dinarcon/ud_migrations>.

8
19.txt
View file

@ -67,10 +67,10 @@ Take a moment to read the JSON export and try to understand its structure. It co
The following is a list of headers as they appear in the Google Sheet compared to how they appear in the JSON export:
- `unique_id` appears like `gsx$uniqueid`.
- `name` appears like `gsx$name`.
- `photo-file` appears like `gsx$photo-file`.
- `Book Ref` appears like `gsx$bookref`.
- `unique_id` appears like `gsx$uniqueid`.
- `name` appears like `gsx$name`.
- `photo-file` appears like `gsx$photo-file`.
- `Book Ref` appears like `gsx$bookref`.
So, the header name from Google Sheet gets transformed in the JSON export. They get a prefix of `gsx$` and the header name is transformed to all lowercase letters with spaces and most special characters removed. On top of this, the actual cell value, that you will eventually import, is in a `$t` property one level under the header name. Now, you should create a list of fields to migrate using XPath expressions as selectors. For example, for the `Book Ref` header, the selector would be `gsx$bookref/$t`. But that is not the way to configure the Google Sheets data parser. The module makes some assumptions to make the selector clearer. So, the `gsx$` prefix and `/$t` hierarchy are assumed. For the selector, you only need to use the *transformed name*. In this case: `uniqueid`, `name`, `photo-file`, and `bookref`.

14
20.txt
View file

@ -20,13 +20,13 @@ In any migration project, understanding the source is very important. For Micros
The `spreadsheet` source plugin exposes seven configuration options. The values to use might change depending on the presence of a header row, but all of them apply for both types of document. Here is a summary of the available configurations:
- `file` is required. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or [stream wrappers](https://understanddrupal.com/articles/migrating-csv-files-drupal#file-location).
- `worksheet` is required. It contains the name of the one worksheet to process.
- `header_row` is optional. This number indicates which row contains the headers. Contrary to [CSV migrations](https://understanddrupal.com/articles/migrating-csv-files-drupal), the row number is not zero-based. So, set this value to `1` if headers are on the first row, `2` if they are on the second, and so on.
- `origin` is optional and defaults to `A2`. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.
- `columns` is optional. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns: `A`, `B`, `C`, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.
- `row_index_column` is optional. This is a special column that contains the row number for each record. This can be used as a *unique identifier* for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in the `columns` configuration. Important: this is an autogenerated column, not any of the columns that comes with your dataset.
- `keys` is optional and, if not set, it defaults to the value of `row_index_column`. It contains an array of column names that *uniquely identify* each record. For files with a header row, you can use the values set in the `columns` configuration. Otherwise, use default column titles like `A`, `B`, `C`, etc. In both cases, you can use the `row_index_column` column if it was set. Each value in the array will contain database storage details for the column.
- `file` is required. It stores the path to the document to process. You can use a relative path from the Drupal root, an absolute path, or [stream wrappers](https://understanddrupal.com/articles/migrating-csv-files-drupal#file-location).
- `worksheet` is required. It contains the name of the one worksheet to process.
- `header_row` is optional. This number indicates which row contains the headers. Contrary to [CSV migrations](https://understanddrupal.com/articles/migrating-csv-files-drupal), the row number is not zero-based. So, set this value to `1` if headers are on the first row, `2` if they are on the second, and so on.
- `origin` is optional and defaults to `A2`. It indicates which non-header cell contains the first value you want to import. It assumes a grid layout and you only need to indicate the position of the top-left cell value.
- `columns` is optional. It is the list of columns you want to make available for the migration. In case of files with a header row, use those header values in this list. Otherwise, use the default title for columns: `A`, `B`, `C`, etc. If this setting is missing, the plugin will return all columns. This is not ideal, especially for very large files containing more columns than needed for the migration.
- `row_index_column` is optional. This is a special column that contains the row number for each record. This can be used as a *unique identifier* for the records in case your dataset does not provide a suitable value. Exposing this special column in the migration is up to you. If so, you can come up with any name as long as it does not conflict with header row names set in the `columns` configuration. Important: this is an autogenerated column, not any of the columns that comes with your dataset.
- `keys` is optional and, if not set, it defaults to the value of `row_index_column`. It contains an array of column names that *uniquely identify* each record. For files with a header row, you can use the values set in the `columns` configuration. Otherwise, use default column titles like `A`, `B`, `C`, etc. In both cases, you can use the `row_index_column` column if it was set. Each value in the array will contain database storage details for the column.
Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.

16
22.txt
View file

@ -10,10 +10,10 @@ The [configuration management system](https://www.drupal.org/docs/8/configuratio
Here are a few use cases of what is possible:
- When migrations are managed in code, you need file system access to make any changes. Using configuration entities allows site administrators to customize or change the migration via the user interface. This is not about rewriting all the migrations. That should happen during development and never on production environments. But it is possible to tweak certain options. For example, administrators could change the location to the file that is going to be migrated, be it a local file or on remote server.
- When writing migrations, it is very likely that you will work on a subset of the data that will eventually be used to get content into the production environment.  Having migrations as configuration allow you to override part of the migration definition per environment. You could use the [Configuration Split module](https://www.drupal.org/project/config_split) to configure different source files or paths per environment. For example, you could link to a small sample of the data in *development*, a larger sample in *staging*, and the complete dataset in *production*.
- It would be possible to provide extra configuration options via the user interface. In the article about [adding HTTP authentication to fetch remote JSON and XML files](https://understanddrupal.com/articles/adding-http-request-headers-and-authentication-remote-json-and-xml-drupal-migrations), the credentials were hardcoded in the migration definition file. That is less than ideal and exposes sensitive information. An alternative would be to provide a configuration form in the administration interface for the credentials to be added. Then, the submitted values could be injected into the configuration for the migration. Again, you could make use of contrib modules like Configuration Split to make sure those credentials are never exported with the rest of your site's configuration.
- You could provide a user interface to upload migration source files. In fact, the [Migrate source UI module](https://www.drupal.org/project/migrate_source_ui) does exactly this. It exposes an administration interface where you have a file field to upload a CSV file. In the same interface, you get a list of supported migrations in the system. This allows a site administrator to manually upload a file to run the migration against. *Note*: The module is supposed to work with JSON and XML migrations. It did not work during my tests. I opened [this issue](https://www.drupal.org/project/migrate_source_ui/issues/3076725) to follow up on this.
- When migrations are managed in code, you need file system access to make any changes. Using configuration entities allows site administrators to customize or change the migration via the user interface. This is not about rewriting all the migrations. That should happen during development and never on production environments. But it is possible to tweak certain options. For example, administrators could change the location to the file that is going to be migrated, be it a local file or on remote server.
- When writing migrations, it is very likely that you will work on a subset of the data that will eventually be used to get content into the production environment.  Having migrations as configuration allow you to override part of the migration definition per environment. You could use the [Configuration Split module](https://www.drupal.org/project/config_split) to configure different source files or paths per environment. For example, you could link to a small sample of the data in *development*, a larger sample in *staging*, and the complete dataset in *production*.
- It would be possible to provide extra configuration options via the user interface. In the article about [adding HTTP authentication to fetch remote JSON and XML files](https://understanddrupal.com/articles/adding-http-request-headers-and-authentication-remote-json-and-xml-drupal-migrations), the credentials were hardcoded in the migration definition file. That is less than ideal and exposes sensitive information. An alternative would be to provide a configuration form in the administration interface for the credentials to be added. Then, the submitted values could be injected into the configuration for the migration. Again, you could make use of contrib modules like Configuration Split to make sure those credentials are never exported with the rest of your site's configuration.
- You could provide a user interface to upload migration source files. In fact, the [Migrate source UI module](https://www.drupal.org/project/migrate_source_ui) does exactly this. It exposes an administration interface where you have a file field to upload a CSV file. In the same interface, you get a list of supported migrations in the system. This allows a site administrator to manually upload a file to run the migration against. *Note*: The module is supposed to work with JSON and XML migrations. It did not work during my tests. I opened [this issue](https://www.drupal.org/project/migrate_source_ui/issues/3076725) to follow up on this.
These are some examples, but many more possibilities are available. The point is that you have the whole configuration management ecosystem at your disposal. Do you have another example? Please share it in the comments.
@ -21,10 +21,10 @@ These are some examples, but many more possibilities are available. The point is
Managing configuration as configuration adds an extra layer of abstraction in the migration process. This adds a bit of complexity. For example:
- Now you have to keep the `uuid` and `id` keys in sync. This might not seem like a big issue, but it is something to pay attention to.
- When you work with migrations groups (explained in the next article), your migration definition could live in more file.
- The configuration management system has its own restrictions and workflows that you need to follow, particularly for updates.
- You need to be extra careful with your YAML syntax, specially if syncing configuration via the user interface. It is possible to import invalid configuration without getting an error. It is until the migration fails that you realize something is wrong.
- Now you have to keep the `uuid` and `id` keys in sync. This might not seem like a big issue, but it is something to pay attention to.
- When you work with migrations groups (explained in the next article), your migration definition could live in more file.
- The configuration management system has its own restrictions and workflows that you need to follow, particularly for updates.
- You need to be extra careful with your YAML syntax, specially if syncing configuration via the user interface. It is possible to import invalid configuration without getting an error. It is until the migration fails that you realize something is wrong.
Using configuration entities to define migrations certainly offers lots of benefits. But it requires being extra careful managing them.

22
24.txt
View file

@ -6,20 +6,20 @@ In the [previous post](https://understanddrupal.com/articles/using-migration-gro
In the article on [declaring migration dependencies](https://understanddrupal.com/articles/introduction-migration-dependencies-drupal) we briefly touched on the topic of tags. Here is a summary of migration tags:
- They are a feature provided by the core Migrate API.
- Multiple tags can be assigned to a single migration.
- They are defined in the migration definition file alone and do not require creating a separate file.
- Both Migrate Tools and Migrate Run provide a flag to execute operations by tag.
- They do not allow you to share configuration among migrations tagged with the same value.
- They are a feature provided by the core Migrate API.
- Multiple tags can be assigned to a single migration.
- They are defined in the migration definition file alone and do not require creating a separate file.
- Both Migrate Tools and Migrate Run provide a flag to execute operations by tag.
- They do not allow you to share configuration among migrations tagged with the same value.
Here is a summary of migration groups:
- You need to install the Migrate Plus module to take advantage of them.
- Only one group can be assigned to any migration.
- You need to create a separate file to declare group. This affects the readability of migrations as their configuration will be spread over two files.
- Only the Migrate Tools provides a flag to execute operations by group.
- They offer the possibility to share configuration among members of the same group.
- Any shared configuration could be overridden in the individual migration definition files.
- You need to install the Migrate Plus module to take advantage of them.
- Only one group can be assigned to any migration.
- You need to create a separate file to declare group. This affects the readability of migrations as their configuration will be spread over two files.
- Only the Migrate Tools provides a flag to execute operations by group.
- They offer the possibility to share configuration among members of the same group.
- Any shared configuration could be overridden in the individual migration definition files.
## What do migration tags and groups have in common?

14
25.txt
View file

@ -57,13 +57,13 @@ This example includes a paragraph migrations. As explained in [this article](htt
Although you can import and execute migrations from the user interface, this workflow comes with many limitations.The following is a list of some of the things that you should consider:
- Updating configuration in production environments is not recommended. This can be enforced using the [Configuration Read-only mode module](https://www.drupal.org/project/config_readonly).
- If the imported configuration entity did not contain a `uuid`, you need to export that configuration to get the auto generated value. This should be used in subsequent configuration import operations if updates to the YAML definition files are needed. Otherwise, you will get an error like "An entity with this machine name already exists but the import did not specify a UUID."
- The following operations to not provide any user interface feedback if they succeeded: "Rollback", "Stop", and "Reset". See [this issue](https://www.drupal.org/node/3012731) for more context.
- As of this writing, most operations for CSV sources fail if using the 8.x-3.x branch of the [Migrate Source CSV module](https://www.drupal.org/project/migrate_source_csv). See [this issue](https://www.drupal.org/node/3068017) for more context.
- As of this writing, the user interface for renaming columns in CSV sources produces a fatal error if using the 8.x-3.x branch of the Migrate Source CSV module. See [this issue](https://www.drupal.org/node/3077558) for more context.
- As of this writing, it is not possible to execute all migrations in a group from the user interface in one operation. See [this issue](https://www.drupal.org/node/2996610) for more context.
- The [Migrate source UI module](https://www.drupal.org/project/migrate_source_ui) can be used to upload a file to be used as source in CSV migrations. As of this writing, a similar feature for JSON and XML files is not working. See [this issue](https://www.drupal.org/node/3076725) for more context.
- Updating configuration in production environments is not recommended. This can be enforced using the [Configuration Read-only mode module](https://www.drupal.org/project/config_readonly).
- If the imported configuration entity did not contain a `uuid`, you need to export that configuration to get the auto generated value. This should be used in subsequent configuration import operations if updates to the YAML definition files are needed. Otherwise, you will get an error like "An entity with this machine name already exists but the import did not specify a UUID."
- The following operations to not provide any user interface feedback if they succeeded: "Rollback", "Stop", and "Reset". See [this issue](https://www.drupal.org/node/3012731) for more context.
- As of this writing, most operations for CSV sources fail if using the 8.x-3.x branch of the [Migrate Source CSV module](https://www.drupal.org/project/migrate_source_csv). See [this issue](https://www.drupal.org/node/3068017) for more context.
- As of this writing, the user interface for renaming columns in CSV sources produces a fatal error if using the 8.x-3.x branch of the Migrate Source CSV module. See [this issue](https://www.drupal.org/node/3077558) for more context.
- As of this writing, it is not possible to execute all migrations in a group from the user interface in one operation. See [this issue](https://www.drupal.org/node/2996610) for more context.
- The [Migrate source UI module](https://www.drupal.org/project/migrate_source_ui) can be used to upload a file to be used as source in CSV migrations. As of this writing, a similar feature for JSON and XML files is not working. See [this issue](https://www.drupal.org/node/3076725) for more context.
To reiterate, it is not recommended to use the user interface to add configuration related to migrations and execute them. The extra layer of abstractions can make it harder to debug problems with migrations if they arise. If possible, execute your migrations using the commands provided by Migrate Tools. Finally, it is recommended to read [this article](https://understanddrupal.com/articles/defining-drupal-migrations-configuration-entities-migrate-plus-module) to learn more about the difference between managing migrations as code or configuration.

8
26.txt
View file

@ -81,10 +81,10 @@ field_tags:
The terms will be assigned to the `field_tags` field using a process pipeline of four plugins:
- `skip_on_empty` will skip the processing of this field if the record does not have a `src_fruit_list` column.
- `explode` will break the string of comma separated files into individual elements.
- `callback` will use the `trim` PHP function to remove any whitespace from the start or end of the taxonomy term name.
- `entity_generate` takes care of finding the taxonomy terms in the system and creating the ones that do not exist.
- `skip_on_empty` will skip the processing of this field if the record does not have a `src_fruit_list` column.
- `explode` will break the string of comma separated files into individual elements.
- `callback` will use the `trim` PHP function to remove any whitespace from the start or end of the taxonomy term name.
- `entity_generate` takes care of finding the taxonomy terms in the system and creating the ones that do not exist.
For a detailed explanation of the `skip_on_empty` and `explode` plugins see [this article](https://understanddrupal.com/articles/migrating-taxonomy-terms-and-multivalue-fields-drupal). For the `callback` plugin see [this article](https://understanddrupal.com/articles/using-process-plugins-data-transformation-drupal-migrations). Let's focus on the `entity_generate` plugin for now. The `field_tags` field expects an array of taxonomy terms IDs (`tid`). The *source* data contains term names so we need to query the database to get the corresponding term IDs. The taxonomy terms that will be referenced were not imported using the Migrate API. And they might exist in the system yet. If that is the case, they should be created on the fly. Therefore, `migration_lookup` cannot be used, but `entity_generate` can.

6
27.txt
View file

@ -56,9 +56,9 @@ The error produced in the console does not say much. Let's see if any messages w
In this case, the migration messages are good enough t o let us know what is wrong. The required `delimiter` configuration option was not set. When an error occurs, usually you need to perform at least three steps:
- Rollback the migration. This will also clear the messages.
- Make changes to definition file and make they are applied. This will depend on whether you are managing the migrations as *code* or *configuration*.
- Import the migration again.
- Rollback the migration. This will also clear the messages.
- Make changes to definition file and make they are applied. This will depend on whether you are managing the migrations as *code* or *configuration*.
- Import the migration again.
Let's say we performed these steps, but we got an error again. The following snippet shows the updated plugin configuration and the messages that were logged:

30
28.txt
View file

@ -115,9 +115,9 @@ Called from +56 /var/www/drupalvm/drupal/web/modules/contrib/migrate_devel/src/E
The Migrate Devel module also provides a new process plugin called [debug](https://git.drupalcode.org/project/migrate_devel/blob/8.x-1.x/src/Plugin/migrate/process/Debug.php). The plugin works by printing the value it receives to the terminal. As [Benji Fisher](https://www.drupal.org/u/benjifisher) explains in [this issue](https://www.drupal.org/node/3021648), the `debug` plugin offers the following advantages over the `log` plugin provided by the core Migrate API:
- The use of [print_r](https://www.php.net/manual/en/function.print-r.php) handles both arrays and scalar values gracefully.
- It is easy to differentiate debugging code that should be removed from logging plugin configuration that should stay.
- It saves time as there is no need to run the `migrate:messages` command to read the logged values.
- The use of [print_r](https://www.php.net/manual/en/function.print-r.php) handles both arrays and scalar values gracefully.
- It is easy to differentiate debugging code that should be removed from logging plugin configuration that should stay.
- It saves time as there is no need to run the `migrate:messages` command to read the logged values.
In short, you can use the `debug` plugin in place of `log`. There is a particular case where using `debug` is really useful. If used in between of a process plugin chain, you can see how elements are being transformed in each step. The following snippet shows an example of this setup and the output it produces:
@ -173,8 +173,8 @@ Step 4: Generated taxonomy term IDs Array
The process pipeline is part of the *node* migration from the [entity_generate plugin example](https://understanddrupal.com/articles/understanding-entitylookup-and-entitygenerate-process-plugins-migrate-tools). In the code snippet, a `debug` step is added after each plugin in the chain. That way, you can verify that the transformations are happening as expected. In the last step you get an array of the taxonomy term IDs (`tid`) that will be associated to the `field_tags` field. Note that this plugin accepts two optional parameters:
- `label` is a string to print before the debug output. It can be used to give context of what is being printed.
- `multiple` is a boolean that when set to `true` signals the next plugin in the pipeline to process each element of an array individually. The functionality is similar to the [multiple_values](https://cgit.drupalcode.org/migrate_plus/tree/src/Plugin/migrate/process/MultipleValues.php) plugin provided by Migrate Plus.
- `label` is a string to print before the debug output. It can be used to give context of what is being printed.
- `multiple` is a boolean that when set to `true` signals the next plugin in the pipeline to process each element of an array individually. The functionality is similar to the [multiple_values](https://cgit.drupalcode.org/migrate_plus/tree/src/Plugin/migrate/process/MultipleValues.php) plugin provided by Migrate Plus.
## Using the right tool for the job: a debugger
@ -182,9 +182,9 @@ Many migration issues can be solved by following the recommendations from the pr
In the next article we will explain how to configure XDebug to work with PHPStorm and [DrupalVM](https://www.drupalvm.com/). For now, let's consider where are good places to add breakpoints. In [this article](https://www.mtech-llc.com/blog/lucas-hedding/troubleshooting-drupal-8-migration), [Lucas Hedding](https://www.drupal.org/u/heddn) recommends adding them in:
- The `import` method of the [MigrateExecutable](https://git.drupalcode.org/project/drupal/blob/8.8.x/core/modules/migrate/src/MigrateExecutable.php) class.
- The `processRow` method of the MigrateExecutable class.
- The process plugin if you know which one might be causing an issue. The `transform` method is a good place to set the breakpoint.
- The `import` method of the [MigrateExecutable](https://git.drupalcode.org/project/drupal/blob/8.8.x/core/modules/migrate/src/MigrateExecutable.php) class.
- The `processRow` method of the MigrateExecutable class.
- The process plugin if you know which one might be causing an issue. The `transform` method is a good place to set the breakpoint.
The use of a debugger is no guarantee that you will find the solution to your issue. It will depend on many factors including your familiarity with the system and how deep lies the problem. Previous debugging experience, even if not directly related to migrations, will help a lot. Do not get discouraged if it takes you too much time to discover what is causing the problem or if you cannot find it at all. Each time you will get a better understanding of the system.
@ -196,13 +196,13 @@ One of the best ways to reduce the time you spend debugging an issue is having e
Throughout [the series](https://understanddrupal.com/migrations), we have created [many examples](https://github.com/dinarcon/ud_migrations). We have made our best effort to explain how each example work. But we were not able to document every detail in the articles. In part to keep them within a reasonable length. But also, because we do not fully comprehend the system. In any case, we highly encourage you to take the examples and break them in every imaginable way. Making one change at a time, see how the migration behaves and what errors are produced. These are some things to try:
- Do not leave a space after a colon (**:**) when setting a configuration option. Example: `id:this_is_going_to_be_fun`.
- Change the indentation of plugin definitions.
- Try to use a plugin provided by a contributed module that is not enabled.
- Do not set a required plugin configuration option.
- Leave out a full section like source, process, or destination.
- Mix the upper and lowercase letters in configuration options, variables, pseudofields, etc.
- Try to convert a migration managed as code to configuration; and vice versa.
- Do not leave a space after a colon (**:**) when setting a configuration option. Example: `id:this_is_going_to_be_fun`.
- Change the indentation of plugin definitions.
- Try to use a plugin provided by a contributed module that is not enabled.
- Do not set a required plugin configuration option.
- Leave out a full section like source, process, or destination.
- Mix the upper and lowercase letters in configuration options, variables, pseudofields, etc.
- Try to convert a migration managed as code to configuration; and vice versa.
## The migrate:fields-source Drush command

6
29.txt
View file

@ -60,9 +60,9 @@ XDEBUG_CONFIG="idekey=PHPSTORM" /var/www/drupalvm/drupal/vendor/bin/drush migrat
For the breakpoint to be triggered, the following needs to happen:
- *You must execute Drush from the `vendor` directory*. DrupalVM has a globally available Drush binary located at `/usr/local/bin/drush`. That is **not** the one to use. For debugging purposes, **always** execute Drush from the `vendor` directory.
- The command needs to have `XDEBUG_CONFIG` environment variable set to "idekey=PHPSTORM". There are many ways to accomplish this, but prepending the variable as shown in the example is a valid way to do it.
- Because the breakpoint was set in the import method, we need to execute an import command to stop at the breakpoint. The migration in the example Drush command is part of the example module that was enabled earlier.
- *You must execute Drush from the `vendor` directory*. DrupalVM has a globally available Drush binary located at `/usr/local/bin/drush`. That is **not** the one to use. For debugging purposes, **always** execute Drush from the `vendor` directory.
- The command needs to have `XDEBUG_CONFIG` environment variable set to "idekey=PHPSTORM". There are many ways to accomplish this, but prepending the variable as shown in the example is a valid way to do it.
- Because the breakpoint was set in the import method, we need to execute an import command to stop at the breakpoint. The migration in the example Drush command is part of the example module that was enabled earlier.
When the command is executed, a dialog will appear in PHPStorm. In it, you will be asked to select a project or a file to debug. Accept what is selected by default for now.  By accepting the prompt a new server will be configured using the proper IP of the virtual machine.  After doing so, go to "Files > Settings > Language and Frameworks > PHP > Servers". You should see one already created. Make sure the "Use path mappings" option is selected. Then, look for the direct child of "Project files". It should be the directory in your host computer where the VM files are located. In that row, set the "Absolute path on the server" column to  `/var/www/drupalvm`. You can delete any other path mapping. There should only be one from the previous prompt. Now, click "OK" in the dialog to accept the changes.

88
30.txt
View file

@ -6,81 +6,81 @@ When one starts working with migrations, it is easy to be overwhelmed by so many
At the time of this writing, Drupal core ships with four migration modules:
- **Migrate**: provides the base API for migrating data.
- **Migrate Drupal**: offers functionality to migrate from other Drupal installations. It serves as the foundation for upgrades from Drupal 6 and 7\. It also supports reading configuration entities from Drupal 8 sites.
- **Drupal Migrate UI**: provides a [user interface for upgrading](https://www.drupal.org/docs/8/upgrade/upgrade-using-web-browser) a Drupal 6 or 7 site to Drupal 8.
- **Migrate Drupal Multilingual**: is an experimental module required by multilingual translations. When they become stable, the module will be removed from Drupal core. See [this article](https://www.drupal.org/node/2959712) for more information.
- **Migrate**: provides the base API for migrating data.
- **Migrate Drupal**: offers functionality to migrate from other Drupal installations. It serves as the foundation for upgrades from Drupal 6 and 7\. It also supports reading configuration entities from Drupal 8 sites.
- **Drupal Migrate UI**: provides a [user interface for upgrading](https://www.drupal.org/docs/8/upgrade/upgrade-using-web-browser) a Drupal 6 or 7 site to Drupal 8.
- **Migrate Drupal Multilingual**: is an experimental module required by multilingual translations. When they become stable, the module will be removed from Drupal core. See [this article](https://www.drupal.org/node/2959712) for more information.
## Migration runners
Once the migration definition files have been created, there are many options to execute them:
- [Migrate Tools](https://www.drupal.org/project/migrate_tools): provides [Drush](https://www.drush.org/) commands to [run migrations from the command line](https://understanddrupal.com/articles/tips-writing-drupal-migrations-and-understanding-their-workflow). It also exposes a [user interface to run migrations](https://understanddrupal.com/articles/executing-drupal-migrations-user-interface-migrate-tools) created as [configuration entities](https://understanddrupal.com/articles/defining-drupal-migrations-configuration-entities-migrate-plus-module). It offers support for [migration groups](https://understanddrupal.com/articles/using-migration-groups-share-configuration-among-drupal-migrations) and [tags](https://understanddrupal.com/articles/what-difference-between-migration-tags-and-migration-groups-drupal). The module *depends* on [Migrate Plus](https://www.drupal.org/project/migrate_plus).
- [Migrate Run](https://www.drupal.org/project/migrate_run): provides Drush commands to run migrations from the command line. It *does not* offer support for migration groups, but tags are supported.  The module *does not* depend on Migrate Plus.
- [Migrate Upgrade](https://www.drupal.org/project/migrate_upgrade): provides [Drush support for upgrading](https://www.drupal.org/docs/8/upgrade/upgrade-using-drush) a Drupal 6 or 7 site to Drupal 8.
- [Migrate Manifest](https://www.drupal.org/project/migrate_manifest): provides a Drush command for running migrations using a manifest file. See [this article](https://www.drupal.org/node/2350651#s--running-specific-migrations-using-migrate-manifest) for an example of using this module for Drupal upgrades.
- [Migrate Scheduler](https://www.drupal.org/project/migrate_scheduler): integrates with Drupal Core's [Cron API](https://api.drupal.org/api/drupal/core!core.api.php/function/hook_cron/8.8.x) to execute migrations on predefined schedules.
- [Migrate Cron](https://www.drupal.org/project/migrate_cron): exposes a user interface to execute migrations when Cron is triggered. At the time of this writing, the module does not execute dependent migrations. Follow [this issue](https://www.drupal.org/project/migrate_cron/issues/3051619) for updates.
- [Migrate source UI](https://www.drupal.org/project/migrate_source_ui): provides a form for uploading files to use as source for already defined [CSV](https://understanddrupal.com/articles/migrating-csv-files-drupal), [JSON](https://understanddrupal.com/articles/migrating-json-files-drupal), and [XML](https://understanddrupal.com/articles/migrating-xml-files-drupal) migrations. At the time of this writing, it seems that JSON and XML migrations are not being detected. Follow [this issue](https://www.drupal.org/node/3076725) for updates.
- [Migrate Tools](https://www.drupal.org/project/migrate_tools): provides [Drush](https://www.drush.org/) commands to [run migrations from the command line](https://understanddrupal.com/articles/tips-writing-drupal-migrations-and-understanding-their-workflow). It also exposes a [user interface to run migrations](https://understanddrupal.com/articles/executing-drupal-migrations-user-interface-migrate-tools) created as [configuration entities](https://understanddrupal.com/articles/defining-drupal-migrations-configuration-entities-migrate-plus-module). It offers support for [migration groups](https://understanddrupal.com/articles/using-migration-groups-share-configuration-among-drupal-migrations) and [tags](https://understanddrupal.com/articles/what-difference-between-migration-tags-and-migration-groups-drupal). The module *depends* on [Migrate Plus](https://www.drupal.org/project/migrate_plus).
- [Migrate Run](https://www.drupal.org/project/migrate_run): provides Drush commands to run migrations from the command line. It *does not* offer support for migration groups, but tags are supported.  The module *does not* depend on Migrate Plus.
- [Migrate Upgrade](https://www.drupal.org/project/migrate_upgrade): provides [Drush support for upgrading](https://www.drupal.org/docs/8/upgrade/upgrade-using-drush) a Drupal 6 or 7 site to Drupal 8.
- [Migrate Manifest](https://www.drupal.org/project/migrate_manifest): provides a Drush command for running migrations using a manifest file. See [this article](https://www.drupal.org/node/2350651#s--running-specific-migrations-using-migrate-manifest) for an example of using this module for Drupal upgrades.
- [Migrate Scheduler](https://www.drupal.org/project/migrate_scheduler): integrates with Drupal Core's [Cron API](https://api.drupal.org/api/drupal/core!core.api.php/function/hook_cron/8.8.x) to execute migrations on predefined schedules.
- [Migrate Cron](https://www.drupal.org/project/migrate_cron): exposes a user interface to execute migrations when Cron is triggered. At the time of this writing, the module does not execute dependent migrations. Follow [this issue](https://www.drupal.org/project/migrate_cron/issues/3051619) for updates.
- [Migrate source UI](https://www.drupal.org/project/migrate_source_ui): provides a form for uploading files to use as source for already defined [CSV](https://understanddrupal.com/articles/migrating-csv-files-drupal), [JSON](https://understanddrupal.com/articles/migrating-json-files-drupal), and [XML](https://understanddrupal.com/articles/migrating-xml-files-drupal) migrations. At the time of this writing, it seems that JSON and XML migrations are not being detected. Follow [this issue](https://www.drupal.org/node/3076725) for updates.
## Source plugins
The Migrate API offers many options to fetch data from:
- **Migrate** (core module): provides the `SqlBase` abstract class to help with fetching data from a database connection. See [this article](https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins/migrating-data-from-a-sql-source) for an example. It also exposes the `embedded_data` plugin which allows the source data to be defined inside the migration definition file. It was used extensively in the [example migrations](https://github.com/dinarcon/ud_migrations) of [this series](https://understanddrupal.com/migrations). It also offers the `empty` plugin which returns a row based on provided constants. It is used in multilingual migrations for entity references.
- [Migrate Plus](https://www.drupal.org/project/migrate_plus): combining various plugins, it allows fetching data in [JSON](https://understanddrupal.com/articles/migrating-json-files-drupal), [XML](https://understanddrupal.com/articles/migrating-xml-files-drupal), and SOAP formats. It also provides various plugins for parsing HTML. See [this article](https://isovera.com/blog/handling-html-with-drupals-migrate-api/) by [Benji Fisher](https://www.drupal.org/u/benjifisher) for an example. There is also a [patch to add support for PDF parsing](https://www.drupal.org/project/migrate_plus/issues/3019758).
- [Migrate Source CSV](https://www.drupal.org/project/migrate_source_csv): allows fetching data from [CSV](https://understanddrupal.com/articles/migrating-csv-files-drupal) files.
- [Migrate Google Sheets](https://www.drupal.org/project/migrate_google_sheets): leverages Migrate Plus functionality to allow fetching data from [Google Sheets](https://understanddrupal.com/articles/migrating-google-sheets-drupal).
- [Migrate Spreadsheet](https://www.drupal.org/project/migrate_spreadsheet): allows fetching data from [Microsoft Excel and LibreOffice Calc](https://understanddrupal.com/articles/migrating-microsoft-excel-and-libreoffice-calc-files-drupal) files.
- [Migrate Source YAML](https://www.drupal.org/project/migrate_source_yaml): allows fetching data from YAML files.
- [WP Migrate](https://www.drupal.org/project/wp_migrate): allows fetching data from a WordPress database.
- **Migrate** (core module): provides the `SqlBase` abstract class to help with fetching data from a database connection. See [this article](https://www.drupal.org/docs/8/api/migrate-api/migrate-source-plugins/migrating-data-from-a-sql-source) for an example. It also exposes the `embedded_data` plugin which allows the source data to be defined inside the migration definition file. It was used extensively in the [example migrations](https://github.com/dinarcon/ud_migrations) of [this series](https://understanddrupal.com/migrations). It also offers the `empty` plugin which returns a row based on provided constants. It is used in multilingual migrations for entity references.
- [Migrate Plus](https://www.drupal.org/project/migrate_plus): combining various plugins, it allows fetching data in [JSON](https://understanddrupal.com/articles/migrating-json-files-drupal), [XML](https://understanddrupal.com/articles/migrating-xml-files-drupal), and SOAP formats. It also provides various plugins for parsing HTML. See [this article](https://isovera.com/blog/handling-html-with-drupals-migrate-api/) by [Benji Fisher](https://www.drupal.org/u/benjifisher) for an example. There is also a [patch to add support for PDF parsing](https://www.drupal.org/project/migrate_plus/issues/3019758).
- [Migrate Source CSV](https://www.drupal.org/project/migrate_source_csv): allows fetching data from [CSV](https://understanddrupal.com/articles/migrating-csv-files-drupal) files.
- [Migrate Google Sheets](https://www.drupal.org/project/migrate_google_sheets): leverages Migrate Plus functionality to allow fetching data from [Google Sheets](https://understanddrupal.com/articles/migrating-google-sheets-drupal).
- [Migrate Spreadsheet](https://www.drupal.org/project/migrate_spreadsheet): allows fetching data from [Microsoft Excel and LibreOffice Calc](https://understanddrupal.com/articles/migrating-microsoft-excel-and-libreoffice-calc-files-drupal) files.
- [Migrate Source YAML](https://www.drupal.org/project/migrate_source_yaml): allows fetching data from YAML files.
- [WP Migrate](https://www.drupal.org/project/wp_migrate): allows fetching data from a WordPress database.
## Destination plugins
The Migrate API is mostly used to move data into Drupal, but it is possible to [write to other destinations](https://understanddrupal.com/articles/drupal-migrations-understanding-etl-process):
- **Migrate** (core): provides classes for creating content and configuration entities. It also offers the `null` plugin which in itself does not write to anything. It is used in multilingual migrations for entity references.
- [Migrate Plus](https://www.drupal.org/project/migrate_plus): provides the `table` plugin for migrating into tables not registered with Drupal Schema API.
- [CSV file](https://github.com/jonathanfranks/d8migrate/tree/master/web/modules/custom/migrate_destination_csv): example destination plugin implementation to write CSV files. The module was created by [J Franks](https://www.drupal.org/u/franksj) for a [DrupalCamp presentation](https://2018.tcdrupal.org/session/drupal-8-migrate-its-not-rocket-science). Check out the [repository](https://github.com/jonathanfranks/d8migrate/tree/master/web/modules/custom/migrate_destination_csv) and [video recording](https://2018.tcdrupal.org/session/drupal-8-migrate-its-not-rocket-science).
- **Migrate** (core): provides classes for creating content and configuration entities. It also offers the `null` plugin which in itself does not write to anything. It is used in multilingual migrations for entity references.
- [Migrate Plus](https://www.drupal.org/project/migrate_plus): provides the `table` plugin for migrating into tables not registered with Drupal Schema API.
- [CSV file](https://github.com/jonathanfranks/d8migrate/tree/master/web/modules/custom/migrate_destination_csv): example destination plugin implementation to write CSV files. The module was created by [J Franks](https://www.drupal.org/u/franksj) for a [DrupalCamp presentation](https://2018.tcdrupal.org/session/drupal-8-migrate-its-not-rocket-science). Check out the [repository](https://github.com/jonathanfranks/d8migrate/tree/master/web/modules/custom/migrate_destination_csv) and [video recording](https://2018.tcdrupal.org/session/drupal-8-migrate-its-not-rocket-science).
## Development related
These modules can help with [writing Drupal migrations](https://understanddrupal.com/articles/writing-your-first-drupal-migration):
- **Migrate** (core): provides the [log](https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Plugin%21migrate%21process%21Log.php/class/Log) process plugin. See [this article](https://understanddrupal.com/articles/how-debug-drupal-migrations-part-1) for an example of its use.
- [Migrate Devel](https://www.drupal.org/project/migrate_devel): offers Drush options for printing debug information when executing migrations. It also provides the [debug](https://git.drupalcode.org/project/migrate_devel/blob/8.x-1.x/src/Plugin/migrate/process/Debug.php) process plugin. See [this article](https://understanddrupal.com/articles/how-debug-drupal-migrations-part-2) for an example of its use.
- [Migrate Process Vardump](https://www.drupal.org/project/migrate_process_vardump): provides the [vardump](https://git.drupalcode.org/project/migrate_process_vardump/blob/8.x-1.x/src/Plugin/migrate/process/Vardump.php) plugin. It works like the `debug` plugin.
- **Migrate** (core): provides the [log](https://api.drupal.org/api/drupal/core%21modules%21migrate%21src%21Plugin%21migrate%21process%21Log.php/class/Log) process plugin. See [this article](https://understanddrupal.com/articles/how-debug-drupal-migrations-part-1) for an example of its use.
- [Migrate Devel](https://www.drupal.org/project/migrate_devel): offers Drush options for printing debug information when executing migrations. It also provides the [debug](https://git.drupalcode.org/project/migrate_devel/blob/8.x-1.x/src/Plugin/migrate/process/Debug.php) process plugin. See [this article](https://understanddrupal.com/articles/how-debug-drupal-migrations-part-2) for an example of its use.
- [Migrate Process Vardump](https://www.drupal.org/project/migrate_process_vardump): provides the [vardump](https://git.drupalcode.org/project/migrate_process_vardump/blob/8.x-1.x/src/Plugin/migrate/process/Vardump.php) plugin. It works like the `debug` plugin.
## Field and module related
- [Migrate Media Handler](https://www.drupal.org/project/migrate_media_handler): provides migration process plugins to facilitate the migration into Drupal 8 media entities. The source can be Drupal 7 file or image fields. It also supports inline file embeds in rich text. It leverages the DOM parsing plugins provided by Migrate Plus.
- [Media Migration](https://www.drupal.org/project/media_migration): provides an upgrade path from Drupal 7 to Drupal 8 media entities. The source can be image fields and fields attached to media and file entities.
- [Migrate File Entities to Media Entities](https://www.drupal.org/project/migrate_file_to_media): migrates Drupal 8.0 file entities to Drupal 8.5 media entities.
- [Migrate Files](https://www.drupal.org/project/migrate_file): provides process plugins for [migrating files and images](https://understanddrupal.com/articles/migrating-files-and-images-drupal).
- [Webform Migrate](https://www.drupal.org/project/webform_migrate): provides plugin to help migrating from the Drupal 6 and 7 versions of the Webform module.
- [Migrate HTML to Paragraphs](https://www.drupal.org/project/migrate_html_to_paragraphs): turns HTML markup into paragraph entities.
- [Commerce Migrate](https://www.drupal.org/project/commerce_migrate): offers a general purpose migration framework for bringing store information into Drupal Commerce.
- [Address](https://www.drupal.org/project/address): offers a process plugin to migrate data into fields of type address. It also provides an upgrade path from Drupal 7's [Address Field module](https://www.drupal.org/project/addressfield). See [this article](https://understanddrupal.com/articles/migrating-addresses-drupal) for an example.
- [Geofield](https://www.drupal.org/project/geofield): offers a process plugin to migrate data into fields of type geofield. See [this article](https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/contrib-process-plugin-geofield_latlon) for an example.
- [Office Hours](https://www.drupal.org/project/office_hours): offers a process plugin to migrate data into fields of type office hours.
- [Workbench Moderation to Content Moderation](https://www.drupal.org/project/wbm2cm): migrates configuration from one module to the other.
- [Migrate Media Handler](https://www.drupal.org/project/migrate_media_handler): provides migration process plugins to facilitate the migration into Drupal 8 media entities. The source can be Drupal 7 file or image fields. It also supports inline file embeds in rich text. It leverages the DOM parsing plugins provided by Migrate Plus.
- [Media Migration](https://www.drupal.org/project/media_migration): provides an upgrade path from Drupal 7 to Drupal 8 media entities. The source can be image fields and fields attached to media and file entities.
- [Migrate File Entities to Media Entities](https://www.drupal.org/project/migrate_file_to_media): migrates Drupal 8.0 file entities to Drupal 8.5 media entities.
- [Migrate Files](https://www.drupal.org/project/migrate_file): provides process plugins for [migrating files and images](https://understanddrupal.com/articles/migrating-files-and-images-drupal).
- [Webform Migrate](https://www.drupal.org/project/webform_migrate): provides plugin to help migrating from the Drupal 6 and 7 versions of the Webform module.
- [Migrate HTML to Paragraphs](https://www.drupal.org/project/migrate_html_to_paragraphs): turns HTML markup into paragraph entities.
- [Commerce Migrate](https://www.drupal.org/project/commerce_migrate): offers a general purpose migration framework for bringing store information into Drupal Commerce.
- [Address](https://www.drupal.org/project/address): offers a process plugin to migrate data into fields of type address. It also provides an upgrade path from Drupal 7's [Address Field module](https://www.drupal.org/project/addressfield). See [this article](https://understanddrupal.com/articles/migrating-addresses-drupal) for an example.
- [Geofield](https://www.drupal.org/project/geofield): offers a process plugin to migrate data into fields of type geofield. See [this article](https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins/contrib-process-plugin-geofield_latlon) for an example.
- [Office Hours](https://www.drupal.org/project/office_hours): offers a process plugin to migrate data into fields of type office hours.
- [Workbench Moderation to Content Moderation](https://www.drupal.org/project/wbm2cm): migrates configuration from one module to the other.
## Modules created by Tess Flynn (socketwench)
While doing the research for this article, we found many useful modules created by [Tess Flynn (socketwench)](https://www.drupal.org/u/socketwench). She is a [fantastic presenter](http://drupal.tv/all-videos?search_api_fulltext=socketwench) who also has written about Drupal migrations, testing, and [much more](https://deninet.com/topic/drupal). Here are some of her modules:
- [Migrate Directory](https://www.drupal.org/project/migrate_directory): imports files from a directory into Drupal as managed files.
- [Migrate Process S3](https://www.drupal.org/project/migrate_process_s3): downloads objects from an S3 bucket into Drupal.
- [Migrate Process URL](https://www.drupal.org/project/migrate_process_url): provides a process plugin to make it easier to migrate into link fields.
- [Migrate Process Vardump](https://www.drupal.org/project/migrate_process_vardump): helps with [debugging migrations](https://understanddrupal.com/articles/how-configure-xdebug-phpstorm-drupalvm-debug-drupal-migrations-drush-commands-browser).
- Many process plugins that wrap PHP functions. For example: [Migrate Process Array](https://www.drupal.org/project/migrate_process_array), [Migrate Process Trim](https://www.drupal.org/project/migrate_process_trim), [Migrate Process Regex](https://www.drupal.org/project/migrate_process_regex), [Migrate Process Skip](https://www.drupal.org/project/migrate_process_skip), and [Migrate Process XML](https://www.drupal.org/project/migrate_process_xml).
- [Migrate Directory](https://www.drupal.org/project/migrate_directory): imports files from a directory into Drupal as managed files.
- [Migrate Process S3](https://www.drupal.org/project/migrate_process_s3): downloads objects from an S3 bucket into Drupal.
- [Migrate Process URL](https://www.drupal.org/project/migrate_process_url): provides a process plugin to make it easier to migrate into link fields.
- [Migrate Process Vardump](https://www.drupal.org/project/migrate_process_vardump): helps with [debugging migrations](https://understanddrupal.com/articles/how-configure-xdebug-phpstorm-drupalvm-debug-drupal-migrations-drush-commands-browser).
- Many process plugins that wrap PHP functions. For example: [Migrate Process Array](https://www.drupal.org/project/migrate_process_array), [Migrate Process Trim](https://www.drupal.org/project/migrate_process_trim), [Migrate Process Regex](https://www.drupal.org/project/migrate_process_regex), [Migrate Process Skip](https://www.drupal.org/project/migrate_process_skip), and [Migrate Process XML](https://www.drupal.org/project/migrate_process_xml).
## Miscellaneous
- [Feeds Migrate](https://www.drupal.org/project/feeds_migrate): it aims to provide a user interface similar to the one from Drupal 7's [Feeds module](https://www.drupal.org/project/feeds), but working on top of Drupal 8's Migrate API.
- [Migrate Override](https://www.drupal.org/project/migrate_override): allows flagging fields in a content entity so they can be manually changed by side editors without being overridden in a subsequent migration.
- [Migrate Status](https://www.drupal.org/project/migrate_status): checks if migrations are currently running.
- [Migrate QA](https://www.drupal.org/project/migrate_qa): provides tools for validating content migrations. See [this presentation](https://events.drupal.org/seattle2019/sessions/introducing-the-migrate-qa-module) for more details.
- [Feeds Migrate](https://www.drupal.org/project/feeds_migrate): it aims to provide a user interface similar to the one from Drupal 7's [Feeds module](https://www.drupal.org/project/feeds), but working on top of Drupal 8's Migrate API.
- [Migrate Override](https://www.drupal.org/project/migrate_override): allows flagging fields in a content entity so they can be manually changed by side editors without being overridden in a subsequent migration.
- [Migrate Status](https://www.drupal.org/project/migrate_status): checks if migrations are currently running.
- [Migrate QA](https://www.drupal.org/project/migrate_qa): provides tools for validating content migrations. See [this presentation](https://events.drupal.org/seattle2019/sessions/introducing-the-migrate-qa-module) for more details.
And [many, many more modules](https://www.drupal.org/project/project_module?f[3]=drupal_core%3A7234&f[4]=sm_field_project_type%3Afull&text=migrate&solrsort=score+desc)!

42
31.txt
View file

@ -8,34 +8,34 @@ Throughout the series, we explored many migration topics. We started with an [ov
The information we presented in the series is generic enough that it applies to many types of Drupal migrations. There is one particular use case that stands out from the rest: **Drupal upgrades**. An *upgrade* is the process of taking your existing Drupal site and copy its *configuration* and *content* over to a new major version of Drupal. For example, going from Drupal 6 or 7 to Drupal 8\. The following is an oversimplification of the workflow to perform the upgrade process:
- Install a fresh Drupal 8 site.
- Add credentials so that the new site can connect to Drupal 7's database.
- Use the Migrate API to generate migration definition files. They will copy over Drupal 7's configuration and content. This step is only about generating the YAML files.
- Execute those migrations to bring the configuration and content over to Drupal 8.
- Install a fresh Drupal 8 site.
- Add credentials so that the new site can connect to Drupal 7's database.
- Use the Migrate API to generate migration definition files. They will copy over Drupal 7's configuration and content. This step is only about generating the YAML files.
- Execute those migrations to bring the configuration and content over to Drupal 8.
## Preparing your migration
Any migration project requires a good plan of action, but this is particularly important for Drupal upgrades. You need to have a general sense of how the upgrade process works, what assumptions are made by the system, and what limitations exist. Read [this article](https://www.drupal.org/docs/8/upgrade/preparing-a-site-for-upgrade-to-drupal-8) for more details on how to prepare a site for upgrading it to Drupal 8\. Some highlights include:
- Both sites need to be in the latest stable version of their corresponding branch. That means, the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
- Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden and there is no need for it anyways. Part of the process includes recreating the old site's configuration: content types, fields, taxonomy vocabularies, etc.
- Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
- For the system to detect a module's configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., [link](https://www.drupal.org/project/link)) that were moved to core in Drupal 8\. Also to Drupal 7 modules (e.g. [address field](https://www.drupal.org/project/addressfield)) that were superseded by a different one in Drupal 8 (e.g. [address](https://www.drupal.org/project/address)). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
- Some modules do not offer automatic upgrade paths. The primary example is the [Views module](https://www.drupal.org/project/views). This means that any view create in Drupal 7 needs to be manually recreated in Drupal 8.
- The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal's database, you can use the Migrate API to move it over to the new site.
- Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced [Twig](https://www.drupal.org/docs/8/theming/twig) which is significantly different to the [PHPTemplate engine](https://api.drupal.org/api/drupal/themes%21engines%21phptemplate%21phptemplate.engine/7.x) used by Drupal 7.
- Both sites need to be in the latest stable version of their corresponding branch. That means, the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
- Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden and there is no need for it anyways. Part of the process includes recreating the old site's configuration: content types, fields, taxonomy vocabularies, etc.
- Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
- For the system to detect a module's configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., [link](https://www.drupal.org/project/link)) that were moved to core in Drupal 8\. Also to Drupal 7 modules (e.g. [address field](https://www.drupal.org/project/addressfield)) that were superseded by a different one in Drupal 8 (e.g. [address](https://www.drupal.org/project/address)). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
- Some modules do not offer automatic upgrade paths. The primary example is the [Views module](https://www.drupal.org/project/views). This means that any view create in Drupal 7 needs to be manually recreated in Drupal 8.
- The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal's database, you can use the Migrate API to move it over to the new site.
- Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced [Twig](https://www.drupal.org/docs/8/theming/twig) which is significantly different to the [PHPTemplate engine](https://api.drupal.org/api/drupal/themes%21engines%21phptemplate%21phptemplate.engine/7.x) used by Drupal 7.
## Customizing your migration
Note that the *creation* and *execution* of the migration files are separate steps. Upgrading to a major version of Drupal is often a good opportunity to introduce changes to the website. For example, you might want to change the content modeling, navigation, user permissions, etc. To accomplish that, you can modify the generated migration files to account for any scenario where the new site's configuration diverts from the old one. And only when you are done with the customizations, you *execute* the migrations. Examples of things that could change include:
- Combining or breaking apart content types.
- Moving data about people from node entities to user entities, or vice versa.
- Renaming content types, fields, taxonomy vocabularies and terms, etc.
- Changing field types. For example, going from [Address Field module](https://www.drupal.org/project/addressfield) in Drupal 7 to [Address module](https://www.drupal.org/project/address) in Drupal 8.
- Merging multiple taxonomy vocabularies into one.
- Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
- Changing how your multimedia files are stored. For example, going from image fields to media entities.
- Combining or breaking apart content types.
- Moving data about people from node entities to user entities, or vice versa.
- Renaming content types, fields, taxonomy vocabularies and terms, etc.
- Changing field types. For example, going from [Address Field module](https://www.drupal.org/project/addressfield) in Drupal 7 to [Address module](https://www.drupal.org/project/address) in Drupal 8.
- Merging multiple taxonomy vocabularies into one.
- Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
- Changing how your multimedia files are stored. For example, going from image fields to media entities.
## Performing the upgrade
@ -63,8 +63,8 @@ Drupal upgrades tend to be fun, challenging projects. The more you know about th
In March 2017, project lead [Dries Buytaert](https://www.drupal.org/u/dries) announced a plan to make [Drupal upgrades easier forever](https://dri.es/making-drupal-upgrades-easy-forever). This was reinforced during his [keynote at DrupalCon Seattle 2019](https://dri.es/state-of-drupal-presentation-april-2019). You can watch the video recording in [this link](https://www.youtube.com/watch?v=Nf_aD3dTloY). In short, [Drupal 9.0](https://www.drupal.org/docs/9) will be the latest point release of Drupal 8 minus deprecated APIs. This has very important implications:
- When Drupal 9 is released, the Migrate API should be mostly the same of Drupal 8\. Therefore, anything that you learn today will be useful for Drupal 9 as well.
- As long as your code does not use deprecated APIs, [upgrading](https://www.drupal.org/docs/8/upgrade) from Drupal 8 to Drupal 9 will be as easy as [updating](https://www.drupal.org/docs/8/update) from Drupal 8.7 to 8.8.
- Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.
- When Drupal 9 is released, the Migrate API should be mostly the same of Drupal 8\. Therefore, anything that you learn today will be useful for Drupal 9 as well.
- As long as your code does not use deprecated APIs, [upgrading](https://www.drupal.org/docs/8/upgrade) from Drupal 8 to Drupal 9 will be as easy as [updating](https://www.drupal.org/docs/8/update) from Drupal 8.7 to 8.8.
- Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.
What did you learn in today's blog post? Did you know the upgrade process is able to copy content and configuration? Did you know that you can execute the upgrade procedure either from the user interface or the command line? Share your answers in the comments. Also, we would be grateful if you shared this blog post with others.