Relocate to MayFirst standard hosting infrastructure #142

New issue

Open

opened 2025-08-13 18:20:53 +00:00 by wolcen · 3 comments

wolcen commented

2025-08-13 18:20:53 +00:00

Owner

The process is relatively involved, but has multiple safety nets and easy reversion points should anything not function properly during the transition. Additionally, there is an existing test instance that can be moved independently to test response of the new infrastructure and the move procedure itself.

Expected benefits of this move:

Better caching support directly from MayFirst's NGINX inbound reverse-proxy server
Leverages over-arching support from MayFirst for DDOS protection, and other shared security monitoring, where incidents/issues are visible across a larger install base and response protects all users.
Improved rate-limiting support: it is generally simpler to rate-limit requests with NGINX vs Apache. MayFirst already has sensible limiters in place for common bot scanning, etc.
[For others on Elizabeth/the current Drutopia server] improved reliability
Dedicated SQL instance: MFMT has been running a dedicated MySQL instance which has been performant across all users. The hardware dedicated to this instance is specific to running MySQL, and in our experience, it's reliability, performance, and maintenance has been solid. Use of this primary/dedicated resource gives MayFirst the ability to better manage resources, maintaining performance.

Instructions from MayFirst:

For starters, I just assigned the private IP address 10.9.67.105, so now you will be able to reach our internal, network MySQL servers from elizabeth.

Next, let's take geo.coop as an example (maybe things are different with other sites, but it seems like a good one to start with).

The first step is for me or Jaime to move the existing geo.coop hosting order to the new infrastructure. I see that there is a web configuration and that https is not enabled. I assume that web directory is either empty or can be purged since the actual web site is on elizabeth. Therefore, when we complete this move, it should have no impact on the live web site on elizabeth. This is the only step that requires Jaime or my intervention.

Next, Chris should create a mysql user and database, via the control panel, on a network MySQL server (that's the default for new databases). To make things simpler, use the same database name, user name and even password that are currently in use with the live site.

Next step: Chris should copy the database from elizabeth to the network mysql server. Once complete, change your drupal MySQL configuration from "localhost" to "mysql006.mayfirst.cx" (or whatever server you are using). Note the ".cx" domain - this is intercepted by our DNS server and resolved to the proper local 10.0.0.0/24 address of the MySQL server.

At this point, your site should be working properly, but now using our network database server. Double check to be sure!

Next, copy all the web site files to the new infrastructure. You'll need to go via @shell.mayfirst.org, do the control panel login dance, but it should all work properly.

Next, modify your settings file, .user.ini file, .htaccess file and any other file that might have the old path in it. You'll have to update to the new path. AND - be sure to change the database host from mysql004.mayfirst.cx to localhost.

Next, modify your apache configuration file on elizabeth. Add the following to the http section:

ProxyPass "/.well-known/acme-challenge/" "http://key.mayfirst.org/.well-known/acme-challenge/"
ProxyPassReverse "/.well-known/acme-challenge/" "http://key.mayfirst.org/.well-known/acme-challenge/"

That ensures that Lets Encrypt requests will be properly directed to our dedicated key server. There is a possible race condition here if, by chance, elizabeth tries to renew the certificate during the transfer, but renewals are done 1 month ahead of time, so even if it fails, you should still have plenty of time to complete the transfer.

Once this is done, you can now edit the Web Configuration record in the control panel and change it from http to https. This will generate a proper https certificate on our new infrastructure (while the web site is still being served on elizabeth).

The next step is optional - it allows you to test before committing. Edit the Apache configuration and add these lines in the https configuration:

SSLProxyEngine On
ProxyPreserveHost On
ProxyPass "/" https://weborigin005.mayfirst.org/ # UPDATE TO THE RIGHT WEBORIGIN!
ProxyPassReverse "/" https://weborigin005.mayfirst.org/

Now you can restart apache and ensure everything is working properly. You can comment this out if there is a big failure to go back to elizabeth.

Once you are satisfied that it's working, you can optionally rsync the files again in case something was uploaded (but be sure to use --update to you don't overwrite your config file changes).

Lastly, you can update the DNS record to be an ALIAS record pointing to: a.webproxy.mayfirst.org

Ok - that's a lot! How does that sound? Anything I might be getting wrong or missing?

Given the success of MayFirsts latest hosting infrastructure, and also the amount of traffic generated on GEO/the performance costs on the existing aging server, we would like to move to MayFirst's standard hosting infrastructure. The process is relatively involved, but has multiple safety nets and easy reversion points should anything not function properly during the transition. Additionally, there is an existing test instance that can be moved independently to test response of the new infrastructure and the move procedure itself. Expected benefits of this move: - Better caching support directly from MayFirst's NGINX inbound reverse-proxy server - Leverages over-arching support from MayFirst for DDOS protection, and other shared security monitoring, where incidents/issues are visible across a larger install base and response protects all users. - Improved rate-limiting support: it is generally simpler to rate-limit requests with NGINX vs Apache. MayFirst already has sensible limiters in place for common bot scanning, etc. - [For others on Elizabeth/the current Drutopia server] improved reliability - Dedicated SQL instance: MFMT has been running a dedicated MySQL instance which has been performant across all users. The hardware dedicated to this instance is specific to running MySQL, and in our experience, it's reliability, performance, and maintenance has been solid. Use of this primary/dedicated resource gives MayFirst the ability to better manage resources, maintaining performance. Instructions from MayFirst: >For starters, I just assigned the private IP address 10.9.67.105, so now you will be able to reach our internal, network MySQL servers from elizabeth. > > Next, let's take geo.coop as an example (maybe things are different with other sites, but it seems like a good one to start with). > > The first step is for me or Jaime to move the existing geo.coop hosting order to the new infrastructure. I see that there is a web configuration and that https is not enabled. I assume that web directory is either empty or can be purged since the actual web site is on elizabeth. Therefore, when we complete this move, it should have no impact on the live web site on elizabeth. This is the only step that requires Jaime or my intervention. > > Next, Chris should create a mysql user and database, via the control panel, on a network MySQL server (that's the default for new databases). To make things simpler, use the same database name, user name and even password that are currently in use with the live site. > > Next step: Chris should copy the database from elizabeth to the network mysql server. Once complete, change your drupal MySQL configuration from "localhost" to "mysql006.mayfirst.cx" (or whatever server you are using). Note the ".cx" domain - this is intercepted by our DNS server and resolved to the proper local 10.0.0.0/24 address of the MySQL server. > > At this point, your site should be working properly, but now using our network database server. Double check to be sure! > > Next, copy all the web site files to the new infrastructure. You'll need to go via @shell.mayfirst.org, do the control panel login dance, but it should all work properly. > > Next, modify your settings file, .user.ini file, .htaccess file and any other file that might have the old path in it. You'll have to update to the new path. AND - be sure to change the database host from mysql004.mayfirst.cx to localhost. > > Next, modify your apache configuration file on elizabeth. Add the following to the http section: > > ProxyPass "/.well-known/acme-challenge/" "http://key.mayfirst.org/.well-known/acme-challenge/" > ProxyPassReverse "/.well-known/acme-challenge/" "http://key.mayfirst.org/.well-known/acme-challenge/" > > That ensures that Lets Encrypt requests will be properly directed to our dedicated key server. There is a possible race condition here if, by chance, elizabeth tries to renew the certificate during the transfer, but renewals are done 1 month ahead of time, so even if it fails, you should still have plenty of time to complete the transfer. > > Once this is done, you can now edit the Web Configuration record in the control panel and change it from http to https. This will generate a proper https certificate on our new infrastructure (while the web site is still being served on elizabeth). > > The next step is optional - it allows you to test before committing. Edit the Apache configuration and add these lines in the https configuration: > > SSLProxyEngine On > ProxyPreserveHost On > ProxyPass "/" https://weborigin005.mayfirst.org/ # UPDATE TO THE RIGHT WEBORIGIN! > ProxyPassReverse "/" https://weborigin005.mayfirst.org/ > > Now you can restart apache and ensure everything is working properly. You can comment this out if there is a big failure to go back to elizabeth. > > Once you are satisfied that it's working, you can optionally rsync the files again in case something was uploaded (but be sure to use --update to you don't overwrite your config file changes). > > Lastly, you can update the DNS record to be an ALIAS record pointing to: a.webproxy.mayfirst.org > > Ok - that's a lot! How does that sound? Anything I might be getting wrong or missing? >

wolcen commented

2025-08-13 22:27:22 +00:00

Author

Owner

@joshua_davis just pinging you to ensure you are aware of this. Mayfirst will reach out w/confirmation for the action they'd have to do.

wolcen commented

2025-08-13 23:05:40 +00:00

Author

Owner

https://members.mayfirst.org/cp/index.php?area=member&member_id=1719&service_id=13

Which would grant me access to do that for you.

@joshua_davis additionally, in order to create a new hosting order (for the testing domain) under your account, I'd either need you to create one for us (and add me to that), or else add "wolcen" to the list in the Mayfirst member control panel at: https://members.mayfirst.org/cp/index.php?area=member&member_id=1719&service_id=13 Which would grant me access to do that for you.