Skip to main content
Contact Support

Upgrading to Flow Enterprise Server 2023.3.1

Tags:

Last updated

When upgrading your installation from 2023.2.1 to 2023.3.1, perform all upgrade steps in the KOTS admin console.

If you're upgrading from a Flow version earlier than 2023.2.1 to 2023.3.1, reinstall Flow from the command line. You cannot upgrade through the KOTS admin console. Export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow.

Follow the steps outlined below to upgrade to Flow 2023.3.1.

Before you begin

Before upgrading to Flow Enterprise 2023.3.1, read through this information carefully.

Check out the Flow Enterprise Server 2023.3.1 release notes for an overview of updates included in this version. Also refer to the 2023.3.1 system requirements to make sure you meet all requirements.

Important: To upgrade to 2023.3.1 from a version before 2023.2.1, you must reinstall Flow from the command line. This is due to required upgrades for the KOTS platform, which bring additional bug fixes, stability, and feature upgrades. You cannot upgrade through the KOTS admin console unless you’re doing an upgrade from 2023.2.1. This article explains the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.

Information to gather before you begin

  • Your Flow license file.

  • Your TLS/SSL certificates.

  • The latest version of the flow-enterprise-tools package. Request this from Pluralsight Support. Uninstall any older versions of this package from all nodes, prior to installing the latest version of the flow-enterprise-tools package.

Important: You must have the latest version of flow-enterprise-tools to successfully upgrade. Version 3.0.0.0 or later is required.

  • Pluralsight recommends backing up your database before you begin the upgrade process.

  • If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated. A password is required. If you can't access Replicated, contact Support for assistance.

Tip: Depending on how you install Flow, you need to download a few different packages.

  • flow-enterprise-tools contains the tools for installation and maintenance of the Kubernetes framework.

  • Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided by flow-enterprise-tools.

To install the flow-enterprise-tools package:

  • For the host server, copy flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz to the home directory of the user account used for the installation on the host server.

  • Extract the tools file using tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.

Note: You can run any tool from the bin directory by running cd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with the install-enterprise-tools.sh script cd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is /usr/local/share/flow-enterprise-tools.

If your repository_cache directory is preserved when uninstalling Flow, your [app_directory]/repository_cache sub-directory may be owned by root. You must change the owner to 37355:37355 and permission to 0755 to upgrade Flow.

Upgrade the ownership and permission of [app_directory] (default: /opt/flow) to the following:

chown -R 37355:37355 [app_directory]/repository_cache

chmod -R 0755 [app_directory]/repository_cache

back to top

Upgrading to 2023.3.1 in the KOTS admin console

Important: These steps only work when performing an upgrade from Flow Enterprise Server 2022.1.1 or newer to 2022.3.1. If you’re upgrading from an older version, go to the section on Exporting your Flow configuration for the steps to uninstall and reinstall Flow.

To upgrade from 2023.2.1 to 2022.3.1:

  1. Log in to your KOTS admin console.
  2. Click Version history at the top of the page.
  3. If the page doesn’t already say there’s a New version available, click Check for updates. This refreshes the page.
  4. Now you will see a blue Deploy button next to a 2023.3.1 Sequence NN version of Flow Enterprise Server. Click Deploy. This process takes a few minutes, depending on your internet speed.

Once you’ve upgraded Flow Enterprise Server, complete the additional configurations after upgrading Flow Enterprise server.

back to top

Export your Flow configuration for Flow versions before 2023.2.1

Important: If you’re upgrading from a version earlier than 2023.2.1, follow these steps to upgrade Flow Enterprise Server.

The process of uninstalling and reinstalling Flow automatically exports your Flow configuration and saves the output into the current directory.

flow-enterprise-tools also has a script to manually back up the configuration. This doesn't back up your external database or Ceph raw volumes. Ceph raw volumes do not require backups.

To manually backup your configuration, run sudo flow-tools export --preserve-tls.

Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.

admin-user@primary-node:~$ sudo flow-tools export --preserve-tls
[INFO] Checking environment...
[INFO] Archive command (tar): OK
[INFO] HTTP command (curl): OK
[INFO] YAML command (yq): OK
[INFO] KOTS CLI command (kubectl kots): OK
[INFO] Flow app installed: OK
[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq
  • Connecting to cluster ✓
    The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise
After editing these files, you can upload a new version using
      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise
[INFO] Exporting config bundle: kots-config-2023-09-26-16-46-09.tar.gz
[INFO] Cleaning up...

Verify that the exported configuration file has all the required files in it. Extract the tar.gz file just created using the tar tvf <configuration file> command.

admin-user@primary-node:~$ tar tvf kots-config-2023-09-26-16-46-09.tar.gz
drwxr-xr-x root/root         0 2021-04-07 18:27 migration/
-rw-r--r-- root/root      1704 2021-04-07 18:27 migration/server.key
-rw-r--r-- root/root     12352 2021-04-07 18:27 migration/kots.yaml
-rw-r--r-- root/root      3460 2021-04-07 18:27 migration/server.pem
-rw-r--r-- root/root      5443 2021-04-07 18:27 migration/license.yaml 

Note: If you have an external database connected to Flow, it will not be impacted or removed when exporting Flow. If you used an embedded database for testing, please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.

back to top

Uninstall Flow Enterprise for Flow versions before 2023.2.1

Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes. The uninstall script performs two reboots.

Tip: Check the help screen by running flow-uninstall -h to see all available options for the uninstall command.

Run kubectl get nodes on the primary node to verify the current state.


admin-user@primary-node:~$ kubectl get nodes
NAME                     STATUS   ROLES    AGE   VERSION
primary-node             Ready    master   22h   v1.19.13
worker-node              Ready    <none>   21h   v1.19.13

Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue the removal.

Tip: To bypass all prompts and perform any reboots automatically, pass the --quiet flag.

root@primary-node bin]$ sudo ./flow-uninstall -q
[WARN] ======================= WARNING ===========================
[WARN] Running this script will remove Kubernetes and all  
[WARN] related components, Flow application from this server.
[WARN] Worker nodes will be drained and deleted from cluster.
[WARN] Following directory contents will be removed:
[WARN] var/lib/kubelet/var/lib/rook
[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow
[WARN] /opt/replicated/var/lib/weave
[WARN] ===========================================================
[INFO] Primary node detected with 1 worker nodes
[INFO] Checking environment...
[INFO] Archive command (tar): OK
[INFO] HTTP command (curl): OK
[INFO] YAML command (yq): OK
[INFO] KOTS CLI command (kubectl kots): OK
[INFO] Primary node: OK
[INFO] Flow app installed: OK
[INFO] Using yq command: /home/sazzad_khandakar/flow-enterprise-tools/packages/yq
  • Connecting to cluster ✓  
 
    The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsZ8d/flow-enterprise
 
After editing these files, you can upload a new version using
 
      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsZ8d/flow-enterprise
 ...
[INFO] Exporting config bundle: kots-config-2023-09-26-16-46-09.tar.gz
[INFO] Cleaning up...
[INFO] Draining worker node ..
[INFO] Draining node = worker-node
node/worker-node cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-rhjdm, kube-system/weave-net-xz47z, monitoring/prometheus-node-exporter-2tdc4, projectcontour/envoy-xxnkf, rook-ceph/rook-ceph-agent-lffkm, rook-ceph/rook-discover-zpb9c
evicting pod default/run-unified-deduper-sync-1642780800-tpm44
evicting pod default/flow-integrations-bp-worker-7575d9d9d6-vcbc4
evicting pod default/flow-integrations-pr-worker-787b6b97-gxrfg
evicting pod default/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w
evicting pod default/flow-integrations-schedule-worker-76d766b6c8-fncdp
evicting pod default/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl
evicting pod default/flow-integrations-integration-lp-worker-c6f986556-gpbng
evicting pod default/flow-integrations-ticket-worker-6dd89c477c-hth7g
evicting pod default/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5
evicting pod default/flow-integrations-new-repo-worker-797979d547-t98rb
evicting pod default/flow-work-log-enterprise-7bd9c8789d-k9gfv
evicting pod default/flow-integrations-aod-worker-5d9fd95c9d-fc49x
evicting pod default/run-unified-deduper-sync-1642773600-xbq7q
evicting pod default/flow-commit-reprocess-1642783500-q52wd
pod/flow-commit-reprocess-1642783500-q52wd evicted
pod/run-unified-deduper-sync-1642780800-tpm44 evicted
pod/run-unified-deduper-sync-1642773600-xbq7q evicted
I0121 16:46:11.645116   26071 request.go:645] Throttling request took 1.051446785s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-new-repo-worker-797979d547-t98rb
pod/flow-work-log-enterprise-7bd9c8789d-k9gfv evicted
I0121 16:46:21.845078   26071 request.go:645] Throttling request took 1.265863421s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg
I0121 16:46:31.845197   26071 request.go:645] Throttling request took 1.265877369s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg
I0121 16:46:42.045150   26071 request.go:645] Throttling request took 1.453749262s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5
pod/flow-integrations-new-repo-worker-797979d547-t98rb evicted
pod/flow-integrations-schedule-worker-76d766b6c8-fncdp evicted
pod/flow-integrations-bp-worker-7575d9d9d6-vcbc4 evicted
pod/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl evicted
pod/flow-integrations-aod-worker-5d9fd95c9d-fc49x evicted
pod/flow-integrations-pr-worker-787b6b97-gxrfg evicted
I0121 16:46:52.045082   26071 request.go:645] Throttling request took 1.453770122s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5
pod/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5 evicted
pod/flow-integrations-integration-lp-worker-c6f986556-gpbng evicted
pod/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w evicted
pod/flow-integrations-ticket-worker-6dd89c477c-hth7g evicted
node/worker-node evicted
Nodes left = 0
[INFO] Deleting worker node ..
node "worker-node" deleted
[INFO] Scaling down flow ..
deployment.apps/flow-frontend scaled
deployment.apps/flow-integrations-aod-worker scaled
deployment.apps/flow-integrations-bp-worker scaled
deployment.apps/flow-integrations-inc-repo-worker scaled
deployment.apps/flow-integrations-integration-lp-worker scaled
deployment.apps/flow-integrations-misc-queue-worker scaled
deployment.apps/flow-integrations-new-repo-worker scaled
deployment.apps/flow-integrations-pr-worker scaled
deployment.apps/flow-integrations-reprocess-worker scaled
deployment.apps/flow-integrations-schedule-worker scaled
deployment.apps/flow-integrations-ticket-worker scaled
deployment.apps/flow-prj-wkr scaled
deployment.apps/flow-proxy scaled
deployment.apps/flow-scheduler scaled
deployment.apps/flow-syslog scaled
deployment.apps/flow-work-log-enterprise scaled
statefulset.apps/flow-db scaled
statefulset.apps/flow-redis-master scaled
cronjob.batch "flow-commit-reprocess" deleted
Pods left = 0 [|]  
[INFO] Disabling Services
[INFO] Stopping kubelet service ..
Removed /etc/systemd/system/multi-user.target.wants/kubelet.service.
[INFO] Stopping and disabling Docker ..
Failed to stop docker.service: Unit docker.service not loaded.
[INFO] Stopping containerd service ..
Removed /etc/systemd/system/multi-user.target.wants/containerd.service.
[INFO] Issuing a hard reset/reboot. Re-run this script after server comes back up to continue.
Connection to primary-node.mydomain.com closed by remote host.
Connection to primary-node.mydomain.com closed.
 
 
...  server reboots here ... 

Log in to the server again and rerun the flow-uninstall script. There is no need to provide any flags to the uninstall command because  the tool preserves the initial flags that were passed to it.

[root@primary-node ~]$ cd flow-enterprise-tools/bin
[root@primary-node bin]$ sudo ./flow-uninstall
[INFO] Continuing uninstall of Flow ..
[INFO] Services stopped, continuing cleanup ..
[INFO] Removing K8s packages ..
[INFO] Removing package kubeadm .. not found
[INFO] Removing package kubectl .. removed
[INFO] Removing package kubelet .. removed
[INFO] Removing package kubernetes-cni .. removed
[INFO] Removing container runtime packages ..
[INFO] Removing package docker-engine .. not installed
[INFO] Removing package docker .. not installed
[INFO] Removing package docker.io .. not installed
[INFO] Removing package docker-ce .. not installed
[INFO] Removing package docker-ce-cli .. not installed
[INFO] Removing package containerd .. not installed
[INFO] Removing package containerd.io .. removed
[INFO] Cleaning up mounts ..
[INFO] Cleaning up folders ..
[INFO] Removing kubernetes components ..
[INFO] Removing kubernetes config dir ..
[INFO] Removing rook lib dir ..
[INFO] Removing kubelet lib dir ..
[INFO] Removing replicated dir ..
[INFO] Removing weave lib dir ..
[INFO] Removing etcd lib dir ..
[INFO] Removing docker components ..
[INFO] Removing pod logs ..
[INFO] Removing pods logs dir ..
[INFO] Removing containers logs dir ..
[INFO] Removing containerd components ..
[INFO] Removing containerd opt dir ..
[INFO] Removing containerd lib dir ..
[INFO] Removing CNI components ..
[INFO] Removing cni lib dir ..
[INFO] Removing cni config dir ..
[INFO] Removing cni opt dir ..
[INFO] Removing Flow components ..
[INFO] Removing Flow logs dir ..
[INFO] Removing kubeconfig ..
[INFO] Removing root's .kube ..
[INFO] Removing kurl cache ..
[INFO] Removing kurl lib dir ..
[INFO] Removing kubeadm binary ..
[INFO] Syncing filesystem ..
[INFO] Resetting IPTABLES ..
[INFO] ceph raw disks detected..
[INFO] Removing /dev/mapper links of raw disk /dev/mapper/ceph--7a982bbc--8adc--4119--a87a--e436b68da377-osd--data--1a0a0ea6--5ec2--4920--8682--719eb7cab8ce
[INFO] Resetting MBR of raw disk /dev/nvme1n1
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or other utilities.
[INFO] Removing uninstallation files ..
[INFO] Flow uninstall has been completed.
[INFO] Executing a graceful reboot.
Connection to primary-node.mydomain.com closed by remote host.
Connection to primary-node.mydomain.com closed.

... Server reboots here a second time ...

Note that the existing configuration was saved in this example to a file named kots-config-2023-09-26-16-46-09.tar.gz.

This completes the process of uninstalling Flow on the primary node.

For each worker node, follow the exact same steps as described above, including rebooting the server.

back to top

Reinstall Flow Enterprise for Flow versions before 2023.2.1

Now we are ready to reinstall Flow Enterprise 2023.3.1.

When you ran flow-uninstall, a configuration export file was created. In the above example, the file was called kots-config-2023-09-26-16-46-09.tar.gz. Pass this file to the flow-tools install command to automate the upgrade processes. The command may look like sudo ./flow-tools install -C kots-config-2023-09-26-16-46-09.tar.gz -d xvdb. In this example, a raw device with the name xvdb is being used.  For airgapped installations, the command will look like sudo ./flow-tools install -a --airgap-app-bundle flow-enterprise-2023.3.1.airgap -C kots-config-2023-09-26-16-46-09.tar.gz -d xvdb.

Note: Use other parameters like --tls-cert, --tls-key, --license-file, --config-values to override parts of your configuration bundle if you need to change them when you upgrade. This is useful if you have any expired items from an export of an older version.

[admin_user@primary-node ~]$ sudo flow-tools install -C kots-config-2023-09-26-16-46-09.tar.gz -d 'xvdb'
[INFO] Running flow-tools with args : install -C kots-config-2023-09-26-16-46-09.tar.gz -d xvdb
[INFO] Extracting kots config bundle...
[INFO] Processing 6 configuration maps ..
[INFO] Processed configuration maps.
[INFO] Processing Java options values ..
[INFO] Processed Java options values.
[INFO] Processing 24 obsolete settings ..
[INFO] Processed obsolete settings.
[INFO] Processing settings with defaults ..
[INFO] Found 183 keys with a default setting .. removing defaults
[INFO] Removed 183 default settings.
[INFO] Processing empty settings ..
[INFO] Removed 214 empty settings.
[INFO] Verifying installation environment...
[INFO] HTTP command (curl): OK
[INFO] Archive command (tar): OK
[INFO] Swarm does not exist: OK
[INFO] Verifying system requirements...
[INFO] Checking networking...
[INFO] sysctl command : OK
[INFO] IPV6 Kernel module: LOADED
[INFO] IPV6 Check : OK
[INFO] IPv4 Forwarding: ENABLED
[INFO] Check IPtable Rules: OK
[INFO] Detecting proxy: NOT DETECTED
[INFO] https://replicated.app site check : OK
[INFO] Checking hardware...
[INFO] CPU: OK
[INFO] Memory: OK
[INFO] Space check in /var/lib/containerd: OK
[INFO] Space check in /var/lib/kubelet: OK
[INFO] Space check in /opt/replicated: OK
[INFO] Space check in /var/lib/kurl: OK
[INFO] Space check in /tmp: OK
[INFO] Space for Repo cache in /opt/flow: 199 GB
[INFO] Disk Space Check: OK
[INFO] Non SSD Disks: NOT DETECTED
[INFO] Checking filesystem and permissions...
[INFO] Login restrictions check: OK
[INFO] Selinux Status: enabled
[INFO] Selinux Current mode: permissive
[INFO] bash Umask setting: OK
[INFO] /etc/profile Umask setting: OK
[WARN] Checking PATH for /usr/local/bin: NOT DETECTED
[INFO] Checking distro...
[INFO] No existing ceph raw disks detected
[INFO] Installation type is : NEW
[INFO] Validating block storage device filter...
Device match:   /dev/xvdb
Device size:    150G
Device status:  valid
[INFO] Total valid block storage: 150G
[INFO] Block storage: OK
[INFO] Adding patch to use raw ceph block devices for installation
[INFO] Installing KOTS application
[INFO] Saving environment
[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/version/v2023.04.13-0/flow-enterprise-nick-dev
[INFO] Fetching join script from: https://k8s.kurl.sh/version/v2023.04.13-0/flow-enterprise-nick-dev/join.sh
 
...
...
...
 
                Installation
                  Complete ✔
The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.
To access Grafana use the generated user:password of admin:xxxxxxxxx .
Kotsadm: http://192.168.0.80:8800
Login with password (will not be shown again): xxxxxxxxx
This password has been set for you by default. It is recommended that you change this password; this can be done with the following command: kubectl kots reset-password default
To access the cluster with kubectl:
    bash -l
Kurl uses /etc/kubernetes/admin.conf, you might want to copy kubeconfig to your home directory:
    cp /etc/kubernetes/admin.conf ~/.kube/config
    chown -R 150028 ~/.kube
    echo unset KUBECONFIG >> ~/.bash_profile
    bash -l
You will likely need to use sudo to copy and chown /etc/kubernetes/admin.conf.
[INFO] Loading environment
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
[INFO] No existing ceph raw disks detected
Kubernetes connection credentials for worker node. Expires in 24 hours
Kubernetes Connection String :  kubernetes-master-address=192.168.0.80:6443 kubeadm-token=5geexa.b0k10q8w5lse854l kubeadm-token-ca-hash=sha256:7243fc4365fcc1c576cbb7b086d84a92d1a3a7aebb4009227cf48afe6652fa39 kubernetes-version=1.25.8 docker-registry-ip=10.96.3.47
You may add additional command line options to the flow-tools join command.
Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.
Node join command for this cluster is below:
sudo ./flow-tools join --channel nick-dev  kubernetes-master-address=192.168.0.80:6443 kubeadm-token=5geexa.b0k10q8w5lse854l kubeadm-token-ca-hash=sha256:7243fc4365fcc1c576cbb7b086d84a92d1a3a7aebb4009227cf48afe6652fa39 kubernetes-version=1.25.8 docker-registry-ip=10.96.3.47
node/primary-node labeled
[INFO] Primary node has been labelled with
        gui=true
        worker=true
        If adding an additional node, please run the following,
        after adding a worker node:
                kubectl label nodes worker- --selector='node-role.kubernetes.io/master'
                kubectl label nodes worker= --selector='!node-role.kubernetes.io/master'
[]
  • Reset the admin console password for default
Enter a new password for the admin console (6+ characters): ••••••••
  • The admin console password has been reset
[INFO] Extracting any app and db backups found...
[INFO] Installing app...
        /bin/kubectl kots install flow-enterprise/nick-dev --namespace default --kotsadm-namespace default --name flow-enterprise --skip-preflights --wait-duration 5m --license-file /tmp/flow-toolslZb/config/migration/license.yaml --config-values /tmp/flow-toolslZb/config/migration/kots.yaml
  • Deploying Admin Console
  • Waiting for Admin Console to be ready ✓
  • Waiting for installation to complete ✓
  • Done
[INFO] Configuring TLS for host: primary-node.mycompany.com
secret/kotsadm-tls annotated
pod "kurl-proxy-kotsadm-6b759788f8-wpqs7" deleted
[INFO] Setting up kubectl command for current user
[INFO] Processing home directory: /home/admin_user
[INFO] Setting up kube-config for user: admin_user 

Now Flow is fully upgraded on the primary node. For the other nodes in the cluster, follow the steps for joining a node to the cluster and add them to this cluster. Once the nodes are fully joined and all pods are running, log in to Flow Enterprise with the URL you used to access Flow before upgrading.

Additional configurations after upgrading Flow

Note: If you previously used these settings before upgrading, they should still be enabled after your upgrade.

  • To prevent your Flow Enterprise Server cluster from running into a forced eviction due to a lack of disk space, go to Disk Pressure Check Settings and click the checkbox next to Enable Disk Pressure Check and Scale Down.
  • Enable monitoring services
  • Automate certificate renewal