geo/geo-coop
geo/geo-coop
7
1
Fork 0
Code Issues 54 Pull requests Projects Releases Packages Wiki Activity Actions

Migrate content #5

New issue
Closed
opened 2019-07-17 16:16:10 +00:00 by cedewey · 17 comments
cedewey commented 2019-07-17 16:16:10 +00:00 (Migrated from gitlab.com)
Copy link

Work Required

## Work Required * [x] Finalize migration plan - https://ethercalc.org/geo-migration-plan-13sdfsdfq34 * [x] Map fields - https://ethercalc.org/geo-field-mapping-123sdfs4 * [x] Write migration scripts * [x] Run migration
mlncn commented 2019-07-27 11:24:51 +00:00 (Migrated from gitlab.com)
Copy link

marked the checklist item Finalize migration plan - https://ethercalc.org/geo-migration-plan-13sdfsdfq34 as completed

marked the checklist item **Finalize migration plan - https://ethercalc.org/geo-migration-plan-13sdfsdfq34** as completed
mlncn commented 2019-07-27 11:24:54 +00:00 (Migrated from gitlab.com)
Copy link

marked the checklist item Map fields - https://ethercalc.org/geo-field-mapping-123sdfs4 as completed

marked the checklist item **Map fields - https://ethercalc.org/geo-field-mapping-123sdfs4** as completed
mlncn commented 2019-07-27 11:24:56 +00:00 (Migrated from gitlab.com)
Copy link

marked the checklist item Write migration scripts as completed

marked the checklist item **Write migration scripts** as completed
mlncn commented 2019-07-27 11:25:18 +00:00 (Migrated from gitlab.com)
Copy link

assigned to @gnuget and unassigned @cedewey

assigned to @gnuget and unassigned @cedewey
mlncn commented 2019-07-27 11:33:10 +00:00 (Migrated from gitlab.com)
Copy link

web/modules/custom/geo_upgrade/migrations/upgrade_d7_file.yml:12: source_base_path: '/var/www/html/d7/'

That's not going to work except on local, right? Can't it be defined relative to the web root?

Moreover... can the old files just be moved to web/sites/default/files? I expect there'll be a lot of "unmanaged" files that are in there that we'll still want.

`web/modules/custom/geo_upgrade/migrations/upgrade_d7_file.yml:12: source_base_path: '/var/www/html/d7/'` That's not going to work *except* on local, right? Can't it be defined relative to the web root? Moreover... can the old files just be moved to `web/sites/default/files`? I expect there'll be a lot of "unmanaged" files that are in there that we'll still want.
mlncn commented 2019-07-27 12:17:02 +00:00 (Migrated from gitlab.com)
Copy link

And fascinatingly:

[notice] Processed 4104 items (4101 created, 0 updated, 3 failed, 0 ignored) - done with 'upgrade_d7_file'

In MigrateToolsCommands.php line 770:

upgrade_d7_file Migration - 3 failed.

and it bails on the migration, but re-running the migration group picks up and continues fine.

And fascinatingly: > [notice] Processed 4104 items (4101 created, 0 updated, 3 failed, 0 ignored) - done with 'upgrade_d7_file' > > In MigrateToolsCommands.php line 770: > > upgrade_d7_file Migration - 3 failed. and it bails on the migration, but re-running the migration group picks up and continues fine.
mlncn commented 2019-07-27 12:49:08 +00:00 (Migrated from gitlab.com)
Copy link

@gnuget There are many aliases, but they are all the ones created on the new site.

There are no redirects: https://geo.ddev.site/admin/config/search/redirect

We definitely need the 4,853 old path aliases — /about, /story/charter-social-solidarity-economy, /content/brewery-co-op-brings-everyone-table-loomio to name a few representative ones — to be brought over to the new site as redirects.

Or maybe because you made the node IDs the same we can export the aliases from the old site and create a big .htaccess-style redirect of them all to their associated node/ids, and then the new site can take it from there?

If doing old paths as redirects in a migration is difficult, let us know and we (@wolcen) can pursue other approaches.

Overall the migration looks great!!

@gnuget There are many aliases, but they are all the ones created on the new site. There are no redirects: https://geo.ddev.site/admin/config/search/redirect We definitely need the 4,853 old path aliases — `/about`, `/story/charter-social-solidarity-economy`, `/content/brewery-co-op-brings-everyone-table-loomio` to name a few representative ones — to be brought over to the new site as redirects. Or maybe because you made the node IDs the same we can export the aliases from the old site and create a big .htaccess-style redirect of them all to their associated node/ids, and then the new site can take it from there? If doing old paths as redirects in a migration is difficult, let us know and we (@wolcen) can pursue other approaches. Overall the migration looks great!!
mlncn commented 2019-07-27 18:57:37 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #9

marked this issue as related to #9
mlncn commented 2019-07-27 18:57:38 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #10

marked this issue as related to #10
mlncn commented 2019-07-27 18:57:38 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #8

marked this issue as related to #8
mlncn commented 2019-07-27 21:13:03 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #15

marked this issue as related to #15
mlncn commented 2019-07-27 21:43:10 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #16

marked this issue as related to #16
wolcen commented 2019-07-27 23:11:59 +00:00 (Migrated from gitlab.com)
Copy link

@gnuget I ran into some things on the actual run of the migration on the server instance.

First, I was unable to find the source files. This turned out just to be the hardcoded path in the config.yml for the migration. See this line in upgrade_d7_file.yml.

Once this was adjusted, the file section ran, but stopped afterwards:

...
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/reas-biobox.jpeg' could not be copied to 'public://reas-biobox.jpeg'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/jim_johnson_interview_2019.png' could not be copied to 'public://jim_johnson_interview_2019.png'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/500kv_3-phase_transmission_lines.png' could not be copied to 'public://500kv_3-phase_transmission_lines.png'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/beaver_creek_housing_coop_slider_crop.png' could not be copied to 'public://beaver_creek_housing_coop_slider_crop.png'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/rmfu-mfw-picture-e1563811462956.jpg' could not be copied to 'public://rmfu-mfw-picture-e1563811462956.jpg'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/quadratic-equation.svg_-1024x1022.png' could not be copied to 'public://quadratic-equation.svg_-1024x1022.png'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/eva_cover2-slider-crop.png' could not be copied to 'public://eva_cover2-slider-crop.png'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/rockaway-youth-task-force-1024x500.jpeg' could not be copied to 'public://rockaway-youth-task-force-1024x500.jpeg'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/sevananda-coop-front-store-605x289.jpg' could not be copied to 'public://sevananda-coop-front-store-605x289.jpg'. 
 [error]  The specified file '/home/geo_live/site/d7//sites/default/files/co-op_serving_beer.png' could not be copied to 'public://co-op_serving_beer.png'. 
 [notice] Processed 4104 items (2387 created, 0 updated, 1717 failed, 0 ignored) - done with 'upgrade_d7_file'

Despite the above complaints, and stopping after this, files were being populated into the files folder, and we now have >3k in there.

I simply reran the import:

geo_live@elizabeth:~$ drush mim --group="migrate_drupal_7_geo"                                                                                                                            
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_file'                                                                                                               
 [notice] Processed 6 items (6 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role'                                                                                                          
 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 121 items (121 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'
 [notice] Processed 1057 items (1057 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_story_body_paragraph'
 [notice] Processed 17 items (6 created, 0 updated, 0 failed, 11 ignored) - done with 'upgrade_d7_taxonomy_vocabulary'
 [notice] Processed 33 items (33 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_1'
 [notice] Processed 9 items (9 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_2'
 [notice] Processed 16 items (16 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_10'
 [notice] Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_12'
 [notice] Processed 12 items (12 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_13'
 [notice] Processed 28 items (28 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_7'
 [warning] Missing file with ID 3805. ImageItem.php:329
 [warning] Missing file with ID 2850. ImageItem.php:329
 [warning] Missing file with ID 2847. ImageItem.php:329
 [warning] Missing file with ID 2848. ImageItem.php:329
 [warning] Missing file with ID 2849. ImageItem.php:329
 [warning] Missing file with ID 2727. ImageItem.php:329
...
...these continue for quite a while, until:
 [warning] Missing file with ID 4600. ImageItem.php:329                                                                                                                                                            
 [notice] Processed 390 items (390 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_blog'                                                                                      
...
...then a whole bunch of overlong complaints:
 [warning] An overlong word (more than 50 characters) was encountered while indexing: productionfinancedistributionconsumptioneducationcommunication.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article.
 [warning] An overlong word (more than 50 characters) was encountered while indexing: workersvsinvestorsfamouswindowsfactorydangerliquidation.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article. 
 [warning] An overlong word (more than 50 characters) was encountered while indexing: 2080pachamamaandprogressconflictingvisionsforlatinamericasfuture.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article.

So - after this, it appears there is some content!

It'd be great to see this project use a containerized deploy, eh? (where this type of setting is pulled into a known file system layout).

@gnuget I ran into some things on the actual run of the migration on the server instance. First, I was unable to find the source files. This turned out just to be the hardcoded path in the config.yml for the migration. See [this line in upgrade_d7_file.yml](https://gitlab.com/agaric/sites/geo/blob/master/web/modules/custom/geo_upgrade/migrations/upgrade_d7_file.yml#L12). Once this was adjusted, the file section ran, but stopped afterwards: ``` ... [error] The specified file '/home/geo_live/site/d7//sites/default/files/reas-biobox.jpeg' could not be copied to 'public://reas-biobox.jpeg'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/jim_johnson_interview_2019.png' could not be copied to 'public://jim_johnson_interview_2019.png'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/500kv_3-phase_transmission_lines.png' could not be copied to 'public://500kv_3-phase_transmission_lines.png'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/beaver_creek_housing_coop_slider_crop.png' could not be copied to 'public://beaver_creek_housing_coop_slider_crop.png'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/rmfu-mfw-picture-e1563811462956.jpg' could not be copied to 'public://rmfu-mfw-picture-e1563811462956.jpg'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/quadratic-equation.svg_-1024x1022.png' could not be copied to 'public://quadratic-equation.svg_-1024x1022.png'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/eva_cover2-slider-crop.png' could not be copied to 'public://eva_cover2-slider-crop.png'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/rockaway-youth-task-force-1024x500.jpeg' could not be copied to 'public://rockaway-youth-task-force-1024x500.jpeg'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/sevananda-coop-front-store-605x289.jpg' could not be copied to 'public://sevananda-coop-front-store-605x289.jpg'. [error] The specified file '/home/geo_live/site/d7//sites/default/files/co-op_serving_beer.png' could not be copied to 'public://co-op_serving_beer.png'. [notice] Processed 4104 items (2387 created, 0 updated, 1717 failed, 0 ignored) - done with 'upgrade_d7_file' ``` Despite the above complaints, and stopping after this, files were being populated into the files folder, and we now have >3k in there. I simply reran the import: ``` geo_live@elizabeth:~$ drush mim --group="migrate_drupal_7_geo" [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_file' [notice] Processed 6 items (6 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role' [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance [notice] Processed 121 items (121 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user' [notice] Processed 1057 items (1057 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_story_body_paragraph' [notice] Processed 17 items (6 created, 0 updated, 0 failed, 11 ignored) - done with 'upgrade_d7_taxonomy_vocabulary' [notice] Processed 33 items (33 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_1' [notice] Processed 9 items (9 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_2' [notice] Processed 16 items (16 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_10' [notice] Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_12' [notice] Processed 12 items (12 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_13' [notice] Processed 28 items (28 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_vocabulary_7' [warning] Missing file with ID 3805. ImageItem.php:329 [warning] Missing file with ID 2850. ImageItem.php:329 [warning] Missing file with ID 2847. ImageItem.php:329 [warning] Missing file with ID 2848. ImageItem.php:329 [warning] Missing file with ID 2849. ImageItem.php:329 [warning] Missing file with ID 2727. ImageItem.php:329 ... ...these continue for quite a while, until: [warning] Missing file with ID 4600. ImageItem.php:329 [notice] Processed 390 items (390 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_blog' ... ...then a whole bunch of overlong complaints: [warning] An overlong word (more than 50 characters) was encountered while indexing: productionfinancedistributionconsumptioneducationcommunication.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article. [warning] An overlong word (more than 50 characters) was encountered while indexing: workersvsinvestorsfamouswindowsfactorydangerliquidation.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article. [warning] An overlong word (more than 50 characters) was encountered while indexing: 2080pachamamaandprogressconflictingvisionsforlatinamericasfuture.<br />Since database search servers currently cannot index words of more than 50 characters, the word was truncated for indexing. If this should not be a single word, please make sure the "Tokenizer" processor is enabled and configured correctly for index Article. ``` So - after this, it appears there is some content! It'd be great to see this project use a containerized deploy, eh? (where this type of setting is pulled into a known file system layout).
mlncn commented 2019-07-30 15:41:47 +00:00 (Migrated from gitlab.com)
Copy link

marked this issue as related to #20

marked this issue as related to #20
mlncn commented 2019-08-01 20:19:14 +00:00 (Migrated from gitlab.com)
Copy link

OK, the aliases are being handed in, and we can deal with them afterward. Closing this issue as we have enough related ones following up on the weird parts of the migration (and David's doing and did tons of work we never even wrote down first, also).

OK, the aliases are being handed in, and we can deal with them afterward. Closing this issue as we have enough related ones following up on the weird parts of the migration (and David's doing and did tons of work we never even wrote down first, also).
mlncn commented 2019-08-01 20:19:15 +00:00 (Migrated from gitlab.com)
Copy link

closed

closed
mlncn commented 2019-10-10 06:46:42 +00:00 (Migrated from gitlab.com)
Copy link

marked the checklist item Run migration as completed

marked the checklist item **Run migration** as completed
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
drupal10-rebased
drupal10
master
12-comments
old-aug-left-on-local
No results found.
Labels
Clear labels   
A11y

Accessibility-related issues

  
Automated Testing

   
Contributable

Issues with work that can (potentially) be contributed back to Drupal or other FLOSS projects.

  
Contributed

Tag issues where we have contributed work back to FLOSS projects (replace Contributable tag; the Contributed tag can be removed once the contribution is recorded in a blog or something.

  
Decision

An issue that requires a decision be made.

  
Design

Issues that should include a UX/design work or review.

  
Development

   
Drutopia

Issues related to Drutopia features or infrastructure, or are potential Drutopia enhancements (with tag Contributable).

  
IndieWeb

Anything related to interlinking web sites in a broadly IndieWeb fashion, see https://indieweb.org/

  
Infrastructure

   
Launch Critical

Issue that must be done in time for launch.

  
Marketing

   
Needs documentation

When an issue has uncovered something that should be documented in a user guide or the project wiki or anywhere else outside the issue queue.

  
Post-Launch

An issue that needs to be completed, but can happen after launch.

  
status
Abandoned
An issue that has been abandoned and will not be completed.

  
status
Blocked
An issue that is blocked for a reason outside Agaric's control.

  
status
Deploy
An issue that has been tested and is ready to be deployed.

  
status
Doing

  
status
Done

  
status
Duplicate
Same as another issue that is prior, already in progress, or more completely discussed or described.

  
status
In Review
An issue that has been completed and is ready for a code and/or design review.

  
status
Needs Clarification

  
status
Test
An issue that has been completed and is ready to be tested.

  
status
To Do
An issue that is an agreed priority and has all information needed to be completed.

  
type
Bug

  
type
Task

  
type
User Story

No labels
A11y
Automated Testing
Contributable
Contributed
Decision
Design
Development
Drutopia
IndieWeb
Infrastructure
Launch Critical
Marketing
Needs documentation
Post-Launch
status
Abandoned
status
Blocked
status
Deploy
status
Doing
status
Done
status
Duplicate
status
In Review
status
Needs Clarification
status
Test
status
To Do
type
Bug
type
Task
type
User Story
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: geo/geo-coop#5
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?