Replacing a Clustered AtScale Machine
This document describes how to replace a single machine in an AtScale cluster.
Before you begin
- These instructions are for CentOS/RedHat operating systems. You may have to adapt some commands to your supported operating system.
- Unless otherwise specified, run commands as
atscale
user.
About this task
Machines required
Running Clustered AtScale requires a customer-provided external load balancer and three AtScale machines: one that will run AtScale Coordinator (a stripped-down version of AtScale), and two that will run the full AtScale application both in clustered mode. In addition, these instructions refer to a Replacement Host that will replace an existing cluster host.
These AtScale machines will be referred to as follows:
- Application Host 1 - runs full AtScale
- Application Host 2 - runs full AtScale
- Coordinator Host - runs the AtScale Coordinator
- Replacement Host - new Host that will replace one of the other hosts
Replacing an AtScale Host with a Host that has an Identical DNS Fully Qualified Domain Name (FQDN)
Replacing a single clustered AtScale machine using the identical DNS FQDN is a simpler process than replacing an AtScale cluster host with a different FQDN. With in the context of the CONVERT STAND-ALONE ATSCALE TO CLUSTERED ATSCALE Instructions, the replacement host should be thought of as a "standby" host since a "master" is already running. Follow these steps:
- If the host you want to replace is still running then shut it down.
- Copy the installation package to the replacement host. As root
Install the AtScale installer package, but do not run the
confgurator.sh
script. If you are not root, but have sudo privileges install the rpm with thesudo -E
option to preserve the ATSCALE_USER environment variable in the subshell.
rpm -i installer_package_filename.rpm
or if using Debian packages:
dpkg --install installer_package_filename.deb
-
Copy the
/opt/atscale/conf/atscale.yaml
from the running application host to/opt/atscale/conf/atscale.yaml
on the replacement host. -
Activate the replacement host:
- On the replacement host, as the ATSCALE_USER user (
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--activate
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the replacement host, as the ATSCALE_USER user (
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --activate
1. You will be asked to confirm that you want to apply this
configuration. Answer `y` to start the activation process, which
will configure and run AtScale.
Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.
Replacing an AtScale Host with a Host that has a Different DNS Fully Qualified Domain Name (FQDN)
Replacing an AtScale host using a different DNS FQDN requires that AtScale be installed and activated on the new machine and reconfigured on the two machines still running AtScale.
1. On the new machine, install, but do not activate, the AtScale installer package.
Follow the Stand-Alone installation instructions, but do not run the Activate AtScale step.
2. Identify the AtScale Application Host that's the current Database Leader.
Identify the current Database Leader member by running the patronictl
utility on one of the AtScale Application Hosts.
\$ /opt/atscale/current/bin/database/patronictl list
+--------------------------+------------+------------+--------+---------+-----------+
\| Cluster \| Member \| Host \| Role \| State \| Lag in MB \|
+--------------------------+------------+------------+--------+---------+-----------+
\| atscale_postgres_cluster \| atscale-01 \| atscale-01 \| Leader \|
running \| \|
\| atscale_postgres_cluster \| atscale-02 \| atscale-02 \| \| running \|
\|
+--------------------------+------------+------------+--------+---------+-----------+
3. Stop AtScale on each of the machines where it's still running.
Stop AtScale on each of the machines where it's still running, in the following order. If AtScale is currently not running on a given machine, you can proceed to the next one in the list.
- AtScale Application Host that is not the current Database Leader
- AtScale Application Host that is the current Database Leader
- AtScale Coordinator
Stop AtScale:
$ /opt/atscale/bin/atscale_stop
4. Edit /opt/atscale/conf/atscale.yaml.
On the AtScale Application Host that is the current Database Leader,
edit /opt/atscale/conf/atscale.yaml
, changing the information for the
host that's being replaced to that of the host that's being newly
introduced.
Copy the modified /opt/atscale/conf/atscale.yaml
from the AtScale
Application Host that is the current Database Leader to
/opt/atscale/conf/atscale.yaml
on both of the other hosts.
5. Activate or reconfigure AtScale Coordinator.
If the AtScale Coordinator host is the one being replaced, it will need to be activated. Otherwise it can just have the updated configuration applied.
-
Activate the "Coordinator" host
- On the "Coordinator" host, as the ATSCALE_USER user (
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--activate
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the "Coordinator" host, as the ATSCALE_USER user (
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --activate
1. You will be asked to confirm:
2. That you want to apply this configuration
3. Answer `y` to start the activation process, which will configure and
run AtScale.
Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.
-
Apply config to the "Coordinator" host
- On the "Coordinator" host, as the ATSCALE_USER user (
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--apply
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the "Coordinator" host, as the ATSCALE_USER user (
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --apply
1. You will be asked to confirm:
2. That you want to apply this configuration
3. Answer `y` to start the config apply process, which will configure
and run AtScale.
6. Reconfigure the AtScale Application Host that's the Database Leader.
The Database Leader host will never be the one that's being replaced and will therefore only need to be reconfigured.
-
Apply config to the "Database Leader" host
- On the "Database Leader" host, as the ATSCALE_USER user
(
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--apply
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the "Database Leader" host, as the ATSCALE_USER user
(
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --apply
1. You will be asked to confirm:
2. That you want to apply this configuration
3. Answer `y` to start the config apply process, which will configure
and run AtScale.
7. Activate or reconfigure remaining AtScale Application Host.
If the remaining AtScale Application host is the one being replaced, it will need to be activated. Otherwise it can just have the updated configuration applied.
-
Activate the remaining AtScale Application host
- On the remaining AtScale Application host, as the ATSCALE_USER
user (
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--activate
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the remaining AtScale Application host, as the ATSCALE_USER
user (
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --activate
1. You will be asked to confirm:
2. That you want to apply this configuration
3. Answer `y` to start the activation process, which will configure and
run AtScale.
Note: If an error occurs, you must perform a full reinstallation. You cannot simply rerun the installer. See Uninstall AtScale.
-
Apply config to the remaining AtScale Application host
- On the remaining AtScale Application host, as the ATSCALE_USER
user (
atscale
by default), run the installed AtScale'sconfigurator.sh
script in--apply
mode. If running AtScale under a different account, alter the switch user command to switch to the desired user (e.g.su - myatscaleuser
):
- On the remaining AtScale Application host, as the ATSCALE_USER
user (
su - atscale
cd /opt/atscale/versions/\
./bin/configurator.sh --apply
1. You will be asked to confirm:
2. That you want to apply this configuration
3. Answer `y` to start the config apply process, which will configure
and run AtScale.
8. Update the External Load Balancer Configuration.
- If replacing an application host, update your external load balancer configuration to reference the replacement host name rather than the host that was removed from the cluster. This step is not necessary if you are replacing a Coordinator host.
What to do next
- You may now use the AtScale cluster.