agaric/31dom
10
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update articles

Browse source
This commit is contained in:
Mauricio Dinarte 2023-08-15 06:07:17 -06:00
parent 0a46f584fb
commit 378703cbd2
1 changed files with 34 additions and 24 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

58
06.md
View file

@ -1,32 +1,30 @@
# Tips for writing Drupal migrations and understanding their workflow
We have presented several examples so far. They started very simple and have been increasing in complexity. Until now, we have been rather optimistic. Get the sample code, install any module dependency, enable the module that defines the migration, and execute it assuming everything works on the first try. But Drupal migrations often involve a bit of trial and error. At the very least, it is an iterative process. In this chapter we are going to see what happens after **import** and **rollback** operations, how to **recover from a failed migration**, and some **tips for writing definition files**.
We have presented several examples so far. They started very simple and have been increasing in complexity. Until now, we have been rather optimistic. Get the sample code, install any module dependency, enable the module that defines the migration, and execute it assuming everything works on the first try. But Drupal migrations often involve a bit of trial and error. At the very least, it is an iterative process. In this chapter we are going to see what happens after **import** and **rollback** operations, how to **recover from a failed migration**, and some **tips for writing the migration plugin files**.
## Importing and rolling back migrations
When working on a migration project, it is common to write many migration definition files. Even if you were to have only one, it is very likely that your destination will require many field mappings. Running an _import_ operation to get the data into Drupal is the first step. With so many moving parts, it is easy not to get the expected results on the first try. When that happens, you can run a _rollback_ operation. This instructs the system to revert anything that was introduced when then migration was initially imported. After rolling back, you can make changes to the migration definition file and rebuild Drupal's cache for the system to pick up your changes. Finally, you can do another _import_ operation. Repeat this process until you get the results you expect. The following code snippet shows a basic Drupal migration workflow:
When working on a migration project, it is common to write many migration plugin files. Even if you were to have only one, it is very likely that your destination will require many field mappings. Running an _import_ operation to get the data into Drupal is the first step. With so many moving parts, it is easy not to get the expected results on the first try. When that happens, you can run a _rollback_ operation. This instructs the system to revert anything that was introduced when then migration was initially imported. After rolling back, you can make changes to the migration plugin file and rebuild Drupal's cache for the system to pick up your changes. Finally, you can do another _import_ operation. Repeat this process until you get the results you expect. The following code snippet shows a basic Drupal migration workflow:
```console
# 1) Run the migration.
$ drush migrate:import udm_subfields
$ drush migrate:import first_example
# 2) Rollback migration because the expected results were not obtained.
$ drush migrate:rollback udm_subfields
$ drush migrate:rollback first_example
# 3) Change the migration definition file.
# 3) Change the migration plugin file.
# 4) Rebuild caches for changes to be picked up.
# 4) Rebuild caches for changes to take effect.
$ drush cache:rebuild
# 5) Run the migration again.
$ drush migrate:import udm_subfields
$ drush migrate:import first_example
```
The example above assumes you are using Drush to run the migration commands. Specifically, the commands provided by Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided for two modules are the same. If you were to have both enabled, they will conflict with each other and fail. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands. Finally, `udm_subfields` is the `id` of the migration to run. You can find the full code in the Migrating data into Drupal subfields chapter.
In all cases, `first_example` is the `id` of the migration to run. You can find the full code in Chapter 2!!!. Anytime you modify the plugin files, you need to rebuild Drupal's caches for the changes to take effect. This is the procedure to follow when creating the YAML files using Migrate API core features and placing them under the `migrations` directory.
_Tip_: You can use Drush command aliases to write shorter commands. Type `drush [command-name] --help` for a list of the available aliases.
_Technical note_: To pick up changes to the definition file you need to rebuild Drupal's caches. This is the procedure to follow when creating the YAML files using Migrate API core features and placing them under the `migrations` directory. It is also possible to define migrations as configuration entities using the Migrate Plus module. In those cases, the YAML files follow a different naming convention and are placed under the `config/install` directory. For picking up changes in this case, you need to sync the YAML definition using [configuration management](https://www.drupal.org/docs/configuration-management/managing-your-sites-configuration) workflows. This will be covered in a future chapter.
_Note_: It is also possible to define migrations as configuration entities using the Migrate Plus module. In those cases, the YAML files follow a different naming convention and are placed under the `config/install` directory. For picking up changes in this case, you need to sync the YAML definition using [configuration management](https://www.drupal.org/docs/configuration-management/managing-your-sites-configuration) workflows. This will be covered in chapter !!!.
## Stopping and resetting migrations
@ -36,30 +34,42 @@ You can check the state of any migration by running the `drush migrate:status` c
```console
# 1) Run the migration.
$ drush migrate:import udm_process_intro
$ drush migrate:import first_example
# 2) Some non recoverable error occurs. Check the status of the migration.
$ drush migrate:status udm_process_intro
$ drush migrate:status first_example
# 3) Stop the migration.
$ drush migrate:stop udm_process_intro
$ drush migrate:stop first_example
# 4) Reset the status to idle.
$ drush migrate:reset-status udm_process_intro
$ drush migrate:reset-status first_example
# 5) Rollback migration because the expected results were not obtained.
$ drush migrate:rollback udm_process_intro
$ drush migrate:rollback first_example
# 6) Change the migration definition file.
# 6) Change the migration plugin file.
# 7) Rebuild caches for changes to be picked up.
# 7) Rebuild caches for changes to take effect.
$ drush cache:rebuild
# 8) Run the migration again.
$ drush migrate:import udm_process_intro
$ drush migrate:import first_example
```
_Tip_: The errors thrown by the Migrate API might not provide enough information to determine what went wrong. An excellent way to familiarize yourselves with the possible errors is by intentionally braking working migrations. In the example repository for this book there are many migrations you can modify. Try anything that comes to mind: not leaving a space after a _colon_ (**:**) in a key-value assignment; not using proper indentation; using wrong subfield names; using invalid values in property assignments; etc. You might be surprised by how Migrate API deals with such errors. Also note that many other Drupal APIs are involved. For example, you might get a YAML file parse error or an [Entity API](https://www.drupal.org/docs/8/api/entity-api) save error. When you have seen an error before, it is usually faster to identify the cause and fix it in the future.
All the commands above are provided by Drush itself. The Migrate Tools module is a migration runner that offers additional commands and provides extra features to some of the ones listed above. It adds the `migrate:tree` command to show a tree of migration dependencies. For `migrate:import`, it add additional flags like `--group` and `--continue-on-failure`. You can use the following to know which commands are available in your installation and get details on usage:
```console
# List of all migration related commands.
$ drush list --filter=migrate
# Get information on usage and available flags using --help
$ drush [command-name] --help
```
_Tip_: You can use Drush command aliases to write shorter commands. For example, `drush mim first_example` to start an import operation. When using `drush list` that aliases appear in parenthesis next to the command name.
The errors thrown by the Migrate API might not provide enough information to determine what went wrong. An excellent way to familiarize yourselves with possible errors is by intentionally braking working migrations. In the example repository for this book there are many migrations you can modify. Try anything that comes to mind: not leaving a space after a _colon_ (**:**) in a key-value assignment; not using proper indentation; using wrong subfield names; using invalid values in property assignments; etc. You might be surprised by how Migrate API deals with such errors. Also note that many other Drupal APIs are involved. For example, you might get a YAML file parse error or an [Entity API](https://www.drupal.org/docs/8/api/entity-api) save exception. When you have seen an error before, it is usually faster to identify the cause and fix it in the future. In chapters !!! we will talk about debugging migrations.
## What happens when you rollback a Drupal migration?
@ -69,21 +79,21 @@ For example, when using the `file_import` or `image_import` plugins provided by
In the next chapter we are going to start talking about migration dependencies. What happens with dependent migrations (e.g. files and paragraphs) when the migration for host entity (e.g. node) is rolled back? In this case, the Migrate API will perform an entity delete operation on the node. When this happens, referenced files are kept in the system, but paragraphs are automatically deleted. For the curious, this behavior for paragraphs is actually determined by its module dependency: [Entity Reference Revisions](https://www.drupal.org/project/entity_reference_revisions). We will talk more about paragraphs migrations in future chapters.
The moral of the story is that the behavior migration system might be affected by other Drupal APIs. And in the case of _rollback_ operations, make sure to read the documentation or test manually to find out when migrations clean after themselves and when they do not.
The moral of the story is that the resulting state of the system might be affected by other Drupal APIs. And in the case of _rollback_ operations, make sure to read the documentation or test manually to find out when migrations clean after themselves and when they do not.
_Note_: The focus of this section was [content entity](https://www.drupal.org/docs/8/api/entity-api/content-entity) migrations. The general idea can be applied to [configuration entities](https://www.drupal.org/docs/8/api/migrate-api/migrate-destination-plugins-examples/migrating-configuration) or any custom target of the ETL process.
## Re-import or update migrations
We just mentioned that Migrate API issues an entity delete action when rolling back a migration. This has another important side effect. Entity IDs (nid, uid, tid, fid, etc.) are going to change every time you _rollback_ an _import_ again. Depending on auto generated IDs is generally not a good idea. But keep it in mind in case your workflow might be affected. For example, if you are running migrations in a content staging environment, references to the migrated entities can break if their IDs change. Also, if you were to manually update the migrated entities to clean up edge cases, those changes would be lost if you _rollback_ and _import_ again. Finally, keep in mind test data might remain in the system, as described in the previous section, which could find its way to production environments.
We just mentioned that Migrate API triggers an entity delete action when rolling back a migration. This has another important side effect. Entity IDs (`nid`, `uid`, `tid`, `fid`, `mid`, etc.) are going to change every time you _rollback_ and _import_ again. Depending on auto generated IDs is generally not a good idea. But keep it in mind in case your workflow might rely on them. For example, if you are running migrations in a content staging environment, references to the migrated entities can break if their IDs change. Also, if you were to manually update the migrated entities to clean up edge cases, those changes would be lost if you _rollback_ and _import_ again. As described in the previous section, test data might remain in the system after a rollback so make sure to clean things up when deploying to production environments.
An alternative to rolling back a migration is to not execute this operation at all. Instead, you run an _import_ operation again using the `update` flag. This tells the system that in addition to migrating unprocessed items from the source, you also want to update items that were previously imported using their current values. To do this, the Migrate API relies on _source identifiers_ and _map tables_. You might want to consider this option when your source changes overtime, when you have a large number of records to import, or when you want to execute the same migration many times on a schedule.
An alternative to rolling back a migration is to not execute this operation at all. Instead, you run an _import_ operation again using the `--update` flag. This tells the system that in addition to migrating unprocessed items from the source, you also want to update items that were previously imported using their current values. To do this, the Migrate API relies on _source identifiers_ and _map tables_. You might want to consider this option when your source changes overtime, when you have a large number of records to import, or when you want to execute the same migration many times on a schedule.
_Note_: On import operations, the Migrate API issues an entity save action.
## Tips for writing Drupal migrations
When working on migration projects, you might end up with many migration definition files. They can set dependencies on each other. Each file might contain a significant number of field mappings. There are many things you can do to make Drupal migrations more straightforward. For example, practicing with different migration scenarios and studying working examples. As a reference to help you in the process of migrating into Drupal, consider these tips:
When working on migration projects, you might end up with many migration plugin files. They can set dependencies on each other. Each file might contain a significant number of field mappings. There are many things you can do to make Drupal migrations more straightforward. For example, practicing with different migration scenarios and studying working examples. As a reference to help you in the process of migrating into Drupal, consider these tips:
- Start from an existing migration. Look for an example online that does something close to what you need and modify it to your requirements.
- Pay close attention to the syntax of the YAML file. An extraneous space or wrong indentation level can break the whole migration.