Upgrading to Flow Enterprise Server 2022.3.1

Tags: Flow Enterprise Server

When upgrading your installation from 2022.1.1 or newer to 2022.3.1, perform all upgrade steps in the KOTS admin console.

If you're upgrading from a Flow version earlier than 2022.1.1 to 2022.3.1, reinstall Flow from the command line. You cannot upgrade through the KOTS admin console. Export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow.

Follow the steps outlined below to upgrade to Flow 2022.3.1.

In this article

Before you begin
Upgrading to 2022.3.1 in the KOTS admin console
Export your Flow configuration for Flow versions before 2022.1.1
Uninstall Flow Enterprise for Flow versions before 2022.1.1
Reinstall Flow Enterprise for Flow versions before 2022.1.1

Before you begin

Before upgrading to Flow Enterprise 2022.3.1, read through this information carefully.

Check out the Flow Enterprise Server 2022.3.1 release notes for an overview of updates included in this version.

Important: To upgrade to 2022.3.1 from a version before 2022.1.1, you must reinstall Flow from the command line. This is due to required upgrades for the KOTS platform, which bring additional bug fixes, stability, and feature upgrades. You cannot upgrade through the KOTS admin console unless you’re doing an upgrade from 2022.1.1 or newer. This article explains the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.

New system requirements for 2022.3.1

Read the Flow Enterprise Server 2022.3.1 system requirements carefully before proceeding.

All installations require a raw block device for Ceph storage usage on every node in the cluster. This is required for upgrades in the 2022.3.1 release.

Information to gather before you begin

Your Flow license file.
Your TLS/SSL certificates.
The latest version of the flow-enterprise-tools package. Request this from Pluralsight Support. Uninstall any older versions of this package from all nodes, prior to installing the latest version of the flow-enterprise-tools package.

Important: You must have the latest version of flow-enterprise-tools to successfully upgrade. Version 2.1.x or later is required.

Pluralsight recommends backing up your database before you begin the upgrade process.
If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated. A password is required. If you can't access Replicated, contact Support for assistance.

Tip: Depending on how you install Flow, you need to download a few different packages. flow-enterprise-tools contains the tools for installation and maintenance of the Kubernetes framework. Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided by flow-enterprise-tools.

To install the flow-enterprise-tools package:

For the host server, copy flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz to the home directory of the user account used for the installation on the host server.
Extract the tools file using tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.

Note: You can run any tool from the bin directory by running cd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with the install-enterprise-tools.sh script cd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is /usr/local/share/flow-enterprise-tools.

If your repository_cache directory is preserved when uninstalling Flow, your [app_directory]/repository_cache sub-directory may be owned byroot. You must change the owner to 37355:37355 and permission to 0755 to upgrade Flow.

Upgrade the the ownership and permission of [app_directory] (default: /opt/flow) to the following:

chown -R 37355:37355 [app_directory]/repository_cache

chmod -R 0755 [app_directory]/repository_cache

Upgrading to 2022.3.1 in the KOTS admin console

Important: These steps only work when performing an upgrade from Flow Enterprise Server 2022.1.1 or newer to 2022.3.1. If you’re upgrading from an older version, go to the section on Exporting your Flow configuration for the steps to uninstall and reinstall Flow.

To upgrade from 2022.1.1 or newer to 2022.3.1:

Log in to your KOTS admin console.
Click Version history at the top of the page.
If the page doesn’t already say there’s a New version available, click Check for updates. This refreshes the page.
Now you will see a blue Deploy button next to a 2022.3.1 Sequence NN version of Flow Enterprise Server. Click Deploy. This process takes a few minutes, depending on your internet speed.

Once you’ve upgraded Flow Enterprise Server, complete the additional configurations after upgrading Flow Enterprise server.

Export your Flow configuration for Flow versions before 2022.1.1

Important: If you’re upgrading from a version earlier than 2022.1.1, follow these steps to upgrade Flow Enterprise Server.

The process of uninstalling and reinstalling Flow automatically exports your Flow configuration and saves the output into the current directory.

flow-enterprise-tools also has a script to manually back up the configuration. This does not back up your external database or Ceph raw volumes. Ceph raw volumes do not require backups.

To manually backup your configuration, run sudo flow-tools export --preserve-tls.

Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.

admin-user@primary-node:~$ sudo flow-tools export --preserve-tls

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq

• Connecting to cluster ✓

The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise

After editing these files, you can upload a new version using

kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise

[INFO] Exporting config bundle: kots-config-2022-07-01-16-46-09.tar.gz

[INFO] Cleaning up...

Verify that the exported configuration file has all the required files in it. Extract the tar.gz file just created using the tar tvf <configuration file> command.

admin-user@primary-node:~$ tar tvf kots-config-2022-07-01-16-46-09.tar.gz

drwxr-xr-x root/root 0 2021-04-07 18:27 migration/

-rw-r--r-- root/root 1704 2021-04-07 18:27 migration/server.key

-rw-r--r-- root/root 12352 2021-04-07 18:27 migration/kots.yaml

-rw-r--r-- root/root 3460 2021-04-07 18:27 migration/server.pem

-rw-r--r-- root/root 5443 2021-04-07 18:27 migration/license.yaml

Note: If you have an external database connected to Flow, it will not be impacted or removed when exporting Flow. If you used an embedded database for testing, please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.

Uninstall Flow Enterprise for Flow versions before 2022.1.1

Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes. The uninstall script performs two reboots.

Tip: Check the help screen by running flow-uninstall -h to see all available options for the uninstall command.

Run kubectl get nodes on the primary node to verify the current state.

admin-user@primary-node:~$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

primary-node Ready master 22h v1.19.13

worker-node Ready <none> 21h v1.19.13

Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue the removal.

Tip: To bypass all prompts and perform any reboots automatically, pass the --quiet flag.

root@primary-node bin]$ sudo ./flow-uninstall -q

[WARN] ======================= WARNING ===========================

[WARN] Running this script will remove Kubernetes and all

[WARN] related components, Flow application from this server.

[WARN] Worker nodes will be drained and deleted from cluster.

[WARN] Following directory contents will be removed:

[WARN] var/lib/kubelet/var/lib/rook

[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow

[WARN] /opt/replicated/var/lib/weave

[WARN] ===========================================================

[INFO] Primary node detected with 1 worker nodes

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Primary node: OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /home/sazzad_khandakar/flow-enterprise-tools/packages/yq

• Connecting to cluster ✓

The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsZ8d/flow-enterprise

After editing these files, you can upload a new version using

kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsZ8d/flow-enterprise

[INFO] Scrubbing config

[INFO] Processing 6 configuration maps ..

[INFO] Processed configuration maps.

[INFO] Processing Java options values ..

[INFO] Processed Java options values.

[INFO] Processing 8 obsolete settings ..

[INFO] Processed obsolete settings.

[INFO] Processing settings with defaults ..

[INFO] Found 205 keys with a default setting .. removing defaults

[INFO] Removed 205 default settings.

[INFO] Processing empty settings ..

[INFO] Removed 225 empty settings.

[INFO] Exporting config bundle: kots-config-2022-07-01-16-46-09.tar.gz

[INFO] Cleaning up...

[INFO] Draining worker node ..

[INFO] Draining node = worker-node

node/worker-node cordoned

WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-rhjdm, kube-system/weave-net-xz47z, monitoring/prometheus-node-exporter-2tdc4, projectcontour/envoy-xxnkf, rook-ceph/rook-ceph-agent-lffkm, rook-ceph/rook-discover-zpb9c

evicting pod default/run-unified-deduper-sync-1642780800-tpm44

evicting pod default/flow-integrations-bp-worker-7575d9d9d6-vcbc4

evicting pod default/flow-integrations-pr-worker-787b6b97-gxrfg

evicting pod default/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w

evicting pod default/flow-integrations-schedule-worker-76d766b6c8-fncdp

evicting pod default/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl

evicting pod default/flow-integrations-integration-lp-worker-c6f986556-gpbng

evicting pod default/flow-integrations-ticket-worker-6dd89c477c-hth7g

evicting pod default/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

evicting pod default/flow-integrations-new-repo-worker-797979d547-t98rb

evicting pod default/flow-work-log-enterprise-7bd9c8789d-k9gfv

evicting pod default/flow-integrations-aod-worker-5d9fd95c9d-fc49x

evicting pod default/run-unified-deduper-sync-1642773600-xbq7q

evicting pod default/flow-commit-reprocess-1642783500-q52wd

pod/flow-commit-reprocess-1642783500-q52wd evicted

pod/run-unified-deduper-sync-1642780800-tpm44 evicted

pod/run-unified-deduper-sync-1642773600-xbq7q evicted

I0121 16:46:11.645116 26071 request.go:645] Throttling request took 1.051446785s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-new-repo-worker-797979d547-t98rb

pod/flow-work-log-enterprise-7bd9c8789d-k9gfv evicted

I0121 16:46:21.845078 26071 request.go:645] Throttling request took 1.265863421s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg

I0121 16:46:31.845197 26071 request.go:645] Throttling request took 1.265877369s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg

I0121 16:46:42.045150 26071 request.go:645] Throttling request took 1.453749262s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

pod/flow-integrations-new-repo-worker-797979d547-t98rb evicted

pod/flow-integrations-schedule-worker-76d766b6c8-fncdp evicted

pod/flow-integrations-bp-worker-7575d9d9d6-vcbc4 evicted

pod/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl evicted

pod/flow-integrations-aod-worker-5d9fd95c9d-fc49x evicted

pod/flow-integrations-pr-worker-787b6b97-gxrfg evicted

I0121 16:46:52.045082 26071 request.go:645] Throttling request took 1.453770122s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

pod/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5 evicted

pod/flow-integrations-integration-lp-worker-c6f986556-gpbng evicted

pod/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w evicted

pod/flow-integrations-ticket-worker-6dd89c477c-hth7g evicted

node/worker-node evicted

Nodes left = 0

[INFO] Deleting worker node ..

node "worker-node" deleted

[INFO] Scaling down flow ..

deployment.apps/flow-frontend scaled

deployment.apps/flow-integrations-aod-worker scaled

deployment.apps/flow-integrations-bp-worker scaled

deployment.apps/flow-integrations-inc-repo-worker scaled

deployment.apps/flow-integrations-integration-lp-worker scaled

deployment.apps/flow-integrations-misc-queue-worker scaled

deployment.apps/flow-integrations-new-repo-worker scaled

deployment.apps/flow-integrations-pr-worker scaled

deployment.apps/flow-integrations-reprocess-worker scaled

deployment.apps/flow-integrations-schedule-worker scaled

deployment.apps/flow-integrations-ticket-worker scaled

deployment.apps/flow-prj-wkr scaled

deployment.apps/flow-proxy scaled

deployment.apps/flow-scheduler scaled

deployment.apps/flow-syslog scaled

deployment.apps/flow-work-log-enterprise scaled

statefulset.apps/flow-db scaled

statefulset.apps/flow-redis-master scaled

cronjob.batch "flow-commit-reprocess" deleted

Pods left = 0 [|]

[INFO] Disabling Services

[INFO] Stopping kubelet service ..

Removed /etc/systemd/system/multi-user.target.wants/kubelet.service.

[INFO] Stopping and disabling Docker ..

Failed to stop docker.service: Unit docker.service not loaded.

[INFO] Stopping containerd service ..

Removed /etc/systemd/system/multi-user.target.wants/containerd.service.

[INFO] Issuing a hard reset/reboot. Re-run this script after server comes back up to continue.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

... server reboots here ...

Log in to the server again and rerun the flow-uninstall script. There is no need to provide any flags to the uninstall command because the tool preserves the initial flags that were passed to it.

[root@primary-node ~]$ cd flow-enterprise-tools/bin

[root@primary-node bin]$ sudo ./flow-uninstall

[INFO] Continuing uninstall of Flow ..

[INFO] Services stopped, continuing cleanup ..

[INFO] Removing K8s packages ..

[INFO] Removing package kubeadm .. not found

[INFO] Removing package kubectl .. removed

[INFO] Removing package kubelet .. removed

[INFO] Removing package kubernetes-cni .. removed

[INFO] Removing container runtime packages ..

[INFO] Removing package docker-engine .. not installed

[INFO] Removing package docker .. not installed

[INFO] Removing package docker.io .. not installed

[INFO] Removing package docker-ce .. not installed

[INFO] Removing package docker-ce-cli .. not installed

[INFO] Removing package containerd .. not installed

[INFO] Removing package containerd.io .. removed

[INFO] Cleaning up mounts ..

[INFO] Cleaning up folders ..

[INFO] Removing kubernetes components ..

[INFO] Removing kubernetes config dir ..

[INFO] Removing rook lib dir ..

[INFO] Removing kubelet lib dir ..

[INFO] Removing replicated dir ..

[INFO] Removing weave lib dir ..

[INFO] Removing etcd lib dir ..

[INFO] Removing docker components ..

[INFO] Removing pod logs ..

[INFO] Removing pods logs dir ..

[INFO] Removing containers logs dir ..

[INFO] Removing containerd components ..

[INFO] Removing containerd opt dir ..

[INFO] Removing containerd lib dir ..

[INFO] Removing CNI components ..

[INFO] Removing cni lib dir ..

[INFO] Removing cni config dir ..

[INFO] Removing cni opt dir ..

[INFO] Removing Flow components ..

[INFO] Removing Flow logs dir ..

[INFO] Removing kubeconfig ..

[INFO] Removing root's .kube ..

[INFO] Removing kurl cache ..

[INFO] Removing kurl lib dir ..

[INFO] Removing kubeadm binary ..

[INFO] Syncing filesystem ..

[INFO] Resetting IPTABLES ..

[INFO] ceph raw disks detected..

[INFO] Removing /dev/mapper links of raw disk /dev/mapper/ceph--7a982bbc--8adc--4119--a87a--e436b68da377-osd--data--1a0a0ea6--5ec2--4920--8682--719eb7cab8ce

[INFO] Resetting MBR of raw disk /dev/nvme1n1

Creating new GPT entries.

GPT data structures destroyed! You may now partition the disk using fdisk or other utilities.

[INFO] Removing uninstallation files ..

[INFO] Flow uninstall has been completed.

[INFO] Executing a graceful reboot.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

... Server reboots here a second time ...

Note that the existing configuration was saved in this example to a file named kots-config-2022-07-01-16-46-09.tar.gz.

This completes the process of uninstalling Flow on the primary node.

For each worker node, follow the exact same steps as described above, including rebooting the server.

Reinstall Flow Enterprise for Flow versions before 2022.1.1

Now we are ready to reinstall Flow Enterprise 2022.3.1.

When you ran flow-uninstall, a configuration export file was created. In the above example, the file was called kots-config-2022-07-01-16-46-09.tar.gz. Pass this file to the flow-tools install command to automate the upgrade processes. The command may look like sudo ./flow-tools install -C kots-config-2022-07-01-16-46-09.tar.gz -d xvdb. In this example, a raw device with the name xvdb is being used. For airgapped installations, the command will look like sudo ./flow-tools install -a --airgap-app-bundle flow-enterprise-2022.3.1.airgap -C kots-config-2022-07-01-16-46-09.tar.gz -d xvdb.

Note: Use other parameters like --tls-cert, --tls-key, --license-file, --config-values to override parts of your configuration bundle if you need to change them when you upgrade. This is useful if you have any expired items from an export of an older version.

[root@primary-node bin]$ sudo ./flow-tools install -C kots-config-2022-07-01-16-46-09.tar.gz -d xvdb

[INFO] Extracting kots config bundle...

[INFO] Processing map: postgres_dbhost=frontend_postgres_dbhost

[INFO] postgres_dbhost does not exist

[INFO] Processing map: postgres_dbport=frontend_postgres_dbport

[INFO] postgres_dbport does not exist

[INFO] Processing map: postgres_dbpass=frontend_postgres_dbpass

[INFO] postgres_dbpass does not exist

[INFO] Processing map: postgres_dbuser=frontend_postgres_dbuser

[INFO] postgres_dbuser does not exist

[INFO] Processing map: postgres_dbname=frontend_postgres_dbname

[INFO] postgres_dbname does not exist

[INFO] Processing map: database_config_hash=frontend_database_config_hash

[INFO] database_config_hash does not exist

[INFO] Processing Java options values ..

[INFO] Verifying installation environment...

[INFO] HTTP command (curl): OK

[INFO] Archive command (tar): OK

[INFO] Swarm does not exist: OK

[INFO] No existing ceph raw disks detected

[INFO] Installation type is : NEW

[INFO] Validating block storage device filter...

Device match: /dev/xvdb

Device size: 150G

Device status: valid

[INFO] Total valid block storage: 150G

[INFO] Block storage: OK

[INFO] Adding patch to use raw ceph block devices for installation

[INFO] Installing KOTS application

[INFO] Saving environment

[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/flow-enterprise-stable

[INFO] Fetching join script from: https://k8s.kurl.sh/flow-enterprise-stable/join.sh

...

Installation

Complete ✔

The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.

To access Grafana use the generated user:password of admin:ZE59u4y5L .

Kotsadm: http://192.168.1.10:8800

To access the cluster with kubectl, reload your shell:

bash -l

[INFO] Loading environment

[INFO] Existing ceph disk found : /dev/xvdb

Kubernetes connection credentials for worker node. Expires in 24 hours

Kubernetes Connection String : kubernetes-master-address=192.168.1.10:6443 kubeadm-token=nu06qf.on2mtmtjoaslr5mx kubeadm-token-ca-hash=sha256:60f3199213db1d45e25613e9464a33d7753db116929766eb11c17e109d4f4805 kubernetes-version=1.19.13 docker-registry-ip=10.96.2.32

You may add additional command line options to the flow-tools join command.

Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.

Node join command for this cluster is below:

sudo ./flow-tools join --channel stable kubernetes-master-address=172.31.40.69:6443 kubeadm-token=nu06qf.on2mtmtjoaslr5mx kubeadm-token-ca-hash=sha256:60f3199213db1d45e25613e9464a33d7753db116929766eb11c17e109d4f4805 kubernetes-version=1.19.13 docker-registry-ip=10.96.2.32

node/primary-node.mydomain.com labeled

[INFO] Primary node has been labelled with

gui=true

worker=true

If adding an additional node, please run the following,

after adding a worker node:

kubectl label nodes worker- --selector='node-role.kubernetes.io/master'

kubectl label nodes worker= --selector='!node-role.kubernetes.io/master'

[]

• Reset the admin console password for default

Enter a new password to be used for the Admin Console: ••••••••

• The admin console password has been reset

[INFO] Extracting any app and db backups found...

[INFO] Installing app...

/bin/kubectl kots install flow-enterprise/stable --namespace default --kotsadm-namespace default --name flow-enterprise --skip-preflights --wait-duration 5m --license-file /tmp/flow-tools1CO/config/migration/license.yaml --config-values /tmp/flow-tools1CO/config/migration/kots.yaml

• Deploying Admin Console

• Waiting for Admin Console to be ready ✓

• Done

[INFO] Configuring TLS for host: primary-node.mydomain.com

secret/kotsadm-tls annotated

pod "kurl-proxy-kotsadm-5f6bcb4dcd-xcn6x" deleted

[INFO] Setting up kubectl command for current user

[INFO] Processing home directory: /home/root

[INFO] Setting up kube-config for user: root

Now Flow is fully upgraded on the primary node. For the other nodes in the cluster, follow the steps for joining a node to the cluster and add them to this cluster. Once the nodes are fully joined and all pods are running, log in to Flow Enterprise using the URL you used to access Flow before upgrading.

Additional configurations after upgrading Flow

Note: If you previously used these settings before upgrading, they should still be enabled after your upgrade.

To have additional visibility into logs, turn on the built-in ElasticSearch Kibana stack. If desired, forward the logs to an external Splunk Enterprise instance.
To prevent your Flow Enterprise Server cluster from running into a forced eviction due to a lack of disk space, go to Disk Pressure Check Settings and click the checkbox next to Enable Disk Pressure Check and Scale Down.
Enable monitoring services
Automate certificate renewal

If you need help, please contact Pluralsight Support.