Upgrading to Flow Enterprise Server 2021.3.1 – Pluralsight Help Center

If you’re upgrading from a previous Flow version to 2021.3.1, you must reinstall Flow from the command line. You cannot upgrade through the KOTS admin console. You must export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow. Follow the steps outlined below to upgrade to Flow 2021.3.1.

In this article

Before you begin
Export your configuration
Uninstall Flow Enterprise
Reinstall Flow Enterprise

Before you begin

Before upgrading to Flow Enterprise 2021.3.1, read through this information carefully.

Check out the Flow Enterprise Server 2021.3.1 release notes for an overview of updates included in this version.

Important: To upgrade to 2021.3.1, you must reinstall Flow from the command line. This is due to required upgrades needed for the KOTS platform, which bring additional bug fixes, stability, and feature upgrades. You cannot upgrade through the KOTS admin console. This article explains the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.

New system requirements for 2021.3.1

Read the Flow 2021.3.1 system requirements carefully before proceeding.

All new installations require a raw block device for Ceph storage usage on every node in the cluster. This is optional for upgrades in the 2021.3.1 release from an existing Flow installation. Future versions of Flow Enterprise Server will require raw block devices for upgrades.

If you don’t have a raw block device, you will see a warning message for a missing raw block device, but you will be able to proceed. If you currently have a raw block device used by Flow, continue to use it for this and future upgrades.

When you add a raw block device, you must do a full reinstallation, including exporting your configurations and uninstalling Flow. This is true whether you add the raw block device when upgrading to 2021.3.1 or if you add it later.

Information to gather before you begin

Your Flow license file.
Your TLS/SSL certificates.
The latest version of the flow-enterprise-tools package. Request this from Pluralsight Support. Uninstall any older versions of this package from all nodes, prior to installing the latest version of the flow-enterprise-tools package.
Important: You must have the latest version of flow-enterprise-tools to successfully upgrade. Version 2.1.x or later is required.
Pluralsight recommends backing up your database before you begin the upgrade process.
If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated . A password is required. If you can't access Replicated, contact Support for assistance.
Tip: Depending on how you install Flow, you need to download a few different packages. flow-enterprise-tools contains the tools for installation and maintenance of the Kubernetes framework. Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided by flow-enterprise-tools.

To install the flow-enterprise-tools package:

For the host server, copy flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz to the home directory of the user account used for the installation on the host server.
Extract the tools file using tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.
Note: You can run any tool from the bin directory by running cd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with the install-enterprise-tools.sh script cd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is /usr/local/share/flow-enterprise-tools.

If your repository_cache directory is preserved when uninstalling Flow, your [app_directory]/repository_cache sub-directory may be owned by root. You must change the owner to 37355:37355 and permission to 0755 to upgrade Flow.

Upgrade the the ownership and permission of [app_directory] (default: /opt/flow) to the following:

chown -R 37355:37355 [app_directory]/repository_cache

chmod -R 0755 [app_directory]/repository_cache

Export your Flow configuration

The process of uninstalling and reinstalling Flow automatically exports your Flow configuration and saves the output into the current directory.

flow-enterprise-tools also has a script to manually back up the configuration. This does not back up your external database or Ceph raw volumes. Ceph raw volumes do not require backups.

To manually backup your configuration, run sudo flow-tools export --preserve-tls.

Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.

admin-user@primary-node:~$ sudo flow-tools export --preserve-tls

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq

• Connecting to cluster ✓

The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise

After editing these files, you can upload a new version using

kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise

[INFO] Exporting config bundle: kots-config-2021-04-07-18-19-56.tar.gz

[INFO] Cleaning up...

Validate the exported configuration file has all the required files in it. Extract the tar.gz file just created using the tar tvf <configuration file> command.

admin-user@primary-node:~$ tar tvf kots-config-2021-04-07-18-27-33.tar.gz

drwxr-xr-x root/root 0 2021-04-07 18:27 migration/

-rw-r--r-- root/root 1704 2021-04-07 18:27 migration/server.key

-rw-r--r-- root/root 12352 2021-04-07 18:27 migration/kots.yaml

-rw-r--r-- root/root 3460 2021-04-07 18:27 migration/server.pem

-rw-r--r-- root/root 5443 2021-04-07 18:27 migration/license.yaml

Note: If you have an external database connected to Flow, it will not be impacted or removed by default. If you used an embedded database for testing, etc., please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.

Uninstall Flow Enterprise

Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes. The uninstall script will perform two reboots.

Tip: Check the help screen by running flow-uninstall -h to see all available options for the uninstall command.

Run kubectl get nodes on the primary node to verify the current state.

admin-user@primary-node:~$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

primary-node Ready master 22h v1.19.13

worker-node Ready <none> 21h v1.19.13

Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue the removal.

Tip: To bypass all prompts and perform any reboots automatically, pass the --quiet flag.

root@primary-node bin]$ sudo ./flow-uninstall --quiet

[WARN] ======================= WARNING ===========================

[WARN] Running this script will remove Kubernetes and all

[WARN] related components, Flow application from this server.

[WARN] Worker nodes will be drained and deleted from cluster.

[WARN] Following directory contents will be removed:

[WARN] var/lib/kubelet/var/lib/rook

[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow

[WARN] /opt/replicated/var/lib/weave

[WARN] ===========================================================

[INFO] Running on Primary node

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Primary node: OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /home/root/flow-enterprise-tools/packages/yq

• Connecting to cluster ✓

The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsR4K/flow-enterprise

After editing these files, you can upload a new version using

kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsR4K/flow-enterprise

[INFO] Processing map: postgres_dbhost=frontend_postgres_dbhost

[INFO] postgres_dbhost does not exist

[INFO] Processing map: postgres_dbport=frontend_postgres_dbport

[INFO] postgres_dbport does not exist

[INFO] Processing map: postgres_dbpass=frontend_postgres_dbpass

[INFO] postgres_dbpass does not exist

[INFO] Processing map: postgres_dbuser=frontend_postgres_dbuser

[INFO] postgres_dbuser does not exist

[INFO] Processing map: postgres_dbname=frontend_postgres_dbname

[INFO] postgres_dbname does not exist

[INFO] Processing map: database_config_hash=frontend_database_config_hash

[INFO] database_config_hash does not exist

[INFO] Processing Java options values ..

[INFO] Exporting config bundle: kots-config-2021-08-31-17-25-51.tar.gz

[INFO] Cleaning up...

[INFO] Scaling down flow ..

deployment.apps/flow-helm-repo scaled

deployment.apps/flow-integrations-aod-worker scaled

deployment.apps/flow-integrations-bp-worker scaled

deployment.apps/flow-integrations-inc-repo-worker scaled

deployment.apps/flow-integrations-misc-queue-worker scaled

deployment.apps/flow-integrations-new-repo-worker scaled

deployment.apps/flow-integrations-pr-worker scaled

deployment.apps/flow-integrations-schedule-worker scaled

deployment.apps/flow-integrations-ticket-worker scaled

deployment.apps/flow-prj-wkr scaled

deployment.apps/flow-scheduler scaled

deployment.apps/flow-syslog scaled

deployment.apps/flow-web scaled

deployment.apps/flow-web-proxy scaled

deployment.apps/flow-work-log-enterprise scaled

Pods left = 0 [/]

[INFO] Deleting Embedded DB pods ..

statefulset.apps "flow-db" deleted

statefulset.apps "flow-redis-master" deleted

NAME READY AGE

kotsadm-postgres 1/1 23h

[INFO] Disabling Services

[INFO] Stopping kubelet service ..

Removed /etc/systemd/system/multi-user.target.wants/kubelet.service.

[INFO] Stopping and disabling Docker ..

Failed to stop docker.service: Unit docker.service not loaded.

[INFO] Stopping containerd service ..

Removed /etc/systemd/system/multi-user.target.wants/containerd.service.

[INFO] Issuing a hard reset/reboot. Re-run this script after server comes back up to continue.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

... server reboots here ...

Log in to the server again and rerun the flow-uninstall script. There is no need to provide any flags to the uninstall command as the tool preserves the initial flags that were passed to it.

[root@primary-node ~]$ cd flow-enterprise-tools/bin

[root@primary-node bin]$ sudo ./flow-uninstall

[INFO] Continuing uninstall of Flow ..

[INFO] Services stopped, continuing cleanup ..

[INFO] Removing K8s packages ..

[INFO] Removing kubectl-1.19.13-0.x86_64kubectl-1.19.13-0.x86_64 removed

[INFO] Removing kubelet-1.19.13-0.x86_64kubelet-1.19.13-0.x86_64 removed

[INFO] Removing kubernetes-cni-0.8.7-0.x86_64kubernetes-cni-0.8.7-0.x86_64 removed

[INFO] Removing container runtime packages ..

[INFO] Removing containerd.io-1.4.4-3.1.el8.x86_64containerd.io-1.4.4-3.1.el8.x86_64 removed

[INFO] Cleaning up mounts ..

[INFO] Cleaning up folders ..

[INFO] Removing kubernetes components ..

[INFO] Removing kubernetes config dir ..

[INFO] Removing rook lib dir ..

[INFO] Removing kubelet lib dir ..

[INFO] Removing replicated dir ..

[INFO] Removing weave lib dir ..

[INFO] Removing etcd lib dir ..

[INFO] Removing docker components ..

[INFO] Removing pod logs ..

[INFO] Removing pods logs dir ..

[INFO] Removing containers logs dir ..

[INFO] Removing containerd components ..

[INFO] Removing containerd opt dir ..

[INFO] Removing containerd lib dir ..

[INFO] Removing CNI components ..

[INFO] Removing cni lib dir ..

[INFO] Removing cni config dir ..

[INFO] Removing cni opt dir ..

[INFO] Removing Flow components ..

[INFO] Repository_cache directory at: /opt/flow/repository_cache

[INFO] Removing repository cache ..

[INFO] Removing /opt/flow/repository_cache..

[INFO] Removing Flow logs dir ..

[INFO] Removing kubeconfig ..

[INFO] Removing root's .kube ..

[INFO] Removing kurl cache ..

[INFO] Removing kurl lib dir ..

[INFO] Removing kubeadm binary ..

[INFO] Syncing filesystem ..

[INFO] Resetting IPTABLES ..

[INFO] ceph raw disks detected..

[INFO] Removing /dev/mapper links of raw disk /dev/mapper/ceph--4551d108--8e9c--44f5--bcb3--918e3cf97a40-osd--data--ca0de9e1--9606--4e5f--8805--47cecd9a49eb

[INFO] Resetting MBR of raw disk /dev/xvdb

Creating new GPT entries.

GPT data structures destroyed! You may now partition the disk using fdisk or

other utilities.

[INFO] Removing uninstallation files ..

[INFO] Flow uninstall has been completed.

[INFO] Executing a graceful reboot.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

... Server reboots here a second time ...

Note that the existing configuration was saved in this example to a file named kots-config-2021-08-31-17-25-51.tar.gz.

This completes the process of uninstalling Flow on the primary node. If you did not have any raw disks on your systems, the output above may differ slightly

For each worker node, follow the exact same steps as described above, including rebooting the server.

Reinstall Flow Enterprise

Now we are ready to reinstall Flow Enterprise 2021.3.1.

When you ran flow-uninstall, a configuration export file was created. Above, the file was called kots-config-2021-08-31-17-25-51.tar.gz. Pass this file to the flow-tools install command to automate the upgrade processes. The command may look like sudo ./flow-tools install -C kots-config-2021-08-31-17-25-51.tar.gz -d xvdb. In this example, a raw device with the name xvdb is being used. If you are performing an upgrade from an existing Flow installation without using a raw block device, this parameter is not required.

Note: Use other parameters like --tls-cert, --tls-key, --license-file, --config-values to override parts of your configuration bundle if you need to change them when you upgrade. This is useful if you have any expired items from an export of an older version.

[root@primary-node bin]$ sudo ./flow-tools install -C kots-config-2021-08-31-17-25-51.tar.gz -d xvdb

[INFO] Extracting kots config bundle...

[INFO] Processing map: postgres_dbhost=frontend_postgres_dbhost

[INFO] postgres_dbhost does not exist

[INFO] Processing map: postgres_dbport=frontend_postgres_dbport

[INFO] postgres_dbport does not exist

[INFO] Processing map: postgres_dbpass=frontend_postgres_dbpass

[INFO] postgres_dbpass does not exist

[INFO] Processing map: postgres_dbuser=frontend_postgres_dbuser

[INFO] postgres_dbuser does not exist

[INFO] Processing map: postgres_dbname=frontend_postgres_dbname

[INFO] postgres_dbname does not exist

[INFO] Processing map: database_config_hash=frontend_database_config_hash

[INFO] database_config_hash does not exist

[INFO] Processing Java options values ..

[INFO] Verifying installation environment...

[INFO] HTTP command (curl): OK

[INFO] Archive command (tar): OK

[INFO] Swarm does not exist: OK

[INFO] No existing ceph raw disks detected

[INFO] Installation type is : NEW

[INFO] Validating block storage device filter...

Device match: /dev/xvdb

Device size: 150G

Device status: valid

[INFO] Total valid block storage: 150G

[INFO] Block storage: OK

[INFO] Adding patch to use raw ceph block devices for installation

[INFO] Installing KOTS application

[INFO] Saving environment

[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/flow-enterprise-stable

[INFO] Fetching join script from: https://k8s.kurl.sh/flow-enterprise-stable/join.sh

...

Installation

Complete ✔

The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.

To access Grafana use the generated user:password of admin:ZE59u4y5L .

Kotsadm: http://192.168.1.10:8800

To access the cluster with kubectl, reload your shell:

bash -l

[INFO] Loading environment

[INFO] Existing ceph disk found : /dev/xvdb

Kubernetes connection credentials for worker node. Expires in 24 hours

Kubernetes Connection String : kubernetes-master-address=192.168.1.10:6443 kubeadm-token=nu06qf.on2mtmtjoaslr5mx kubeadm-token-ca-hash=sha256:60f3199213db1d45e25613e9464a33d7753db116929766eb11c17e109d4f4805 kubernetes-version=1.19.13 docker-registry-ip=10.96.2.32

You may add additional command line options to the flow-tools join command.

Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.

Node join command for this cluster is below:

sudo ./flow-tools join --channel stable kubernetes-master-address=172.31.40.69:6443 kubeadm-token=nu06qf.on2mtmtjoaslr5mx kubeadm-token-ca-hash=sha256:60f3199213db1d45e25613e9464a33d7753db116929766eb11c17e109d4f4805 kubernetes-version=1.19.13 docker-registry-ip=10.96.2.32

node/primary-node.mydomain.com labeled

[INFO] Primary node has been labelled with

gui=true

worker=true

If adding an additional node, please run the following,

after adding a worker node:

kubectl label nodes worker- --selector='node-role.kubernetes.io/master'

kubectl label nodes worker= --selector='!node-role.kubernetes.io/master'

[]

• Reset the admin console password for default

Enter a new password to be used for the Admin Console: ••••••••

• The admin console password has been reset

[INFO] Extracting any app and db backups found...

find: ‘/tmp/flow-toolsqn0/config’: No such file or directory

[INFO] Installing app...

/bin/kubectl kots install flow-enterprise/stable --namespace default --kotsadm-namespace default --name flow-enterprise --skip-preflights --wait-duration 5m --license-file /tmp/flow-tools1CO/config/migration/license.yaml --config-values /tmp/flow-tools1CO/config/migration/kots.yaml

• Deploying Admin Console

• Waiting for Admin Console to be ready ✓

• Done

[INFO] Configuring TLS for host: primary-node.mydomain.com

secret/kotsadm-tls annotated

pod "kurl-proxy-kotsadm-5f6bcb4dcd-xcn6x" deleted

[INFO] Setting up kubectl command for current user

[INFO] Processing home directory: /home/root

[INFO] Setting up kube-config for user: root

Now Flow is fully upgraded now on the primary node. For the other nodes in the cluster, follow the steps for joining a node to the cluster and add them to this cluster. Once the nodes are fully joined and all pods are running, log in to Flow Enterprise using the URL you used to access Flow before upgrading.