Skip to main content
Contact Support

Upgrading to Flow Enterprise Server 2023.2.1

Tags:

If you're upgrading from a previous Flow version to 2023.2.1 you must reinstall Flow from the command line. You can’t upgrade through the KOTS admin console. You must export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow. Follow the steps outlined below to upgrade to Flow 2023.2.1.

Before you begin

Before upgrading to Flow Enterprise 2023.2.1, read through this information carefully.

Check out the Flow Enterprise Server 2023.2.1 release notes for an overview of updates included in this version.

Important: To upgrade to 2023.2.1, you must reinstall Flow from the command line. This is due to required updates to the core infrastructure Flow runs on. You can’t upgrade through the KOTS admin console. This article explains the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.

Information to gather before you begin

  • Your Flow license file.
  • Your TLS/SSL certificates.
  • The latest version of the flow-enterprise-tools package. Request this from Pluralsight Support. Uninstall any older versions of this package from all nodes, prior to installing the latest version of the flow-enterprise-tools package.

Important: You must have the latest version of flow-enterprise-tools to successfully upgrade. Version 3.0.0.0 or later is required.

  • Pluralsight recommends backing up your database before you begin the upgrade process.
  • If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated. A password is required. If you can't access Replicated, contact Support for assistance.

Tip: Depending on how you install Flow, you need to download a few different packages. 

  • flow-enterprise-tools contains the tools for installation and maintenance of the Kubernetes framework. 
  • Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided by flow-enterprise-tools.

To install the flow-enterprise-tools package:

  • For the host server, copy flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz to the home directory of the user account used for the installation on the host server.
  • Extract the tools file using tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.

Note: You can run any tool from the bin directory by running cd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with the install-enterprise-tools.sh script cd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is /usr/local/share/flow-enterprise-tools.

If your repository_cache directory is preserved when uninstalling Flow, your [app_directory]/repository_cache sub-directory may be owned byroot. You must change the owner to 37355:37355 and permission to 0755 to upgrade Flow.

Upgrade the the ownership and permission of [app_directory] (default: /opt/flow) to the following:

chown -R 37355:37355 [app_directory]/repository_cache

chmod -R 0755 [app_directory]/repository_cache

back to top

Export your Flow configuration

The process of uninstalling and reinstalling Flow automatically exports your Flow configuration and saves the output into the current directory.

flow-enterprise-tools also has a script to manually back up the configuration. This doesn’t back up your external database or Ceph raw volumes. Ceph raw volumes don’t require backups.

To manually back up your configuration, run sudo flow-tools export --preserve-tls.

Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.

admin-user@primary-node:~$ sudo flow-tools export --preserve-tls

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq

  • Connecting to cluster ✓

    The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise

After editing these files, you can upload a new version using

      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise

[INFO] Exporting config bundle: kots-config-2023-04-21-16-46-09.tar.gz

[INFO] Cleaning up...

Validate that the exported configuration file has all the required files in it. Extract the tar.gz file just created using the tar tvf <configuration file> command.

admin-user@primary-node:~$ tar tvf kots-config-2023-04-21-16-46-09.tar.gz

drwxr-xr-x root/root         0 2021-04-07 18:27 migration/

-rw-r--r-- root/root      1704 2021-04-07 18:27 migration/server.key

-rw-r--r-- root/root     12352 2021-04-07 18:27 migration/kots.yaml

-rw-r--r-- root/root      3460 2021-04-07 18:27 migration/server.pem

-rw-r--r-- root/root      5443 2021-04-07 18:27 migration/license.yaml

Note: If you have an external database connected to Flow, by default it won’t be impacted or removed. If you used an embedded database for testing, please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.

back to top

Uninstall Flow Enterprise Server

Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes. The uninstall script will perform two reboots.

Tip: Check the help screen by running flow-uninstall -h to see all available options for the uninstall command.

Run kubectl get nodes on the primary node to verify the current state.

admin-user@primary-node:~$ kubectl get nodes

NAME                     STATUS   ROLES    AGE   VERSION

primary-node             Ready    master   22h   v1.19.13

worker-node              Ready    <none>   21h   v1.19.13

Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue the removal.

Tip: To bypass all prompts and perform any reboots automatically, pass the --quiet flag.

root@primary-node bin]$ sudo ./flow-uninstall -q

[WARN] ======================= WARNING ===========================

[WARN] Running this script will remove Kubernetes and all 

[WARN] related components, Flow application from this server.

[WARN] Worker nodes will be drained and deleted from cluster.

[WARN] Following directory contents will be removed:

[WARN] var/lib/kubelet/var/lib/rook

[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow

[WARN] /opt/replicated/var/lib/weave

[WARN] ===========================================================

[INFO] Primary node detected with 1 worker nodes

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Primary node: OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /home/sazzad_khandakar/flow-enterprise-tools/packages/yq

  • Connecting to cluster ✓  

 

    The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsZ8d/flow-enterprise

 

After editing these files, you can upload a new version using

 

      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsZ8d/flow-enterprise

 

 

 ...

[INFO] Exporting config bundle: kots-config-2023-04-21-16-46-09.tar.gz

[INFO] Cleaning up...

[INFO] Draining worker node ..

[INFO] Draining node = worker-node

node/worker-node cordoned

WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-rhjdm, kube-system/weave-net-xz47z, monitoring/prometheus-node-exporter-2tdc4, projectcontour/envoy-xxnkf, rook-ceph/rook-ceph-agent-lffkm, rook-ceph/rook-discover-zpb9c

evicting pod default/run-unified-deduper-sync-1642780800-tpm44

evicting pod default/flow-integrations-bp-worker-7575d9d9d6-vcbc4

evicting pod default/flow-integrations-pr-worker-787b6b97-gxrfg

evicting pod default/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w

evicting pod default/flow-integrations-schedule-worker-76d766b6c8-fncdp

evicting pod default/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl

evicting pod default/flow-integrations-integration-lp-worker-c6f986556-gpbng

evicting pod default/flow-integrations-ticket-worker-6dd89c477c-hth7g

evicting pod default/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

evicting pod default/flow-integrations-new-repo-worker-797979d547-t98rb

evicting pod default/flow-work-log-enterprise-7bd9c8789d-k9gfv

evicting pod default/flow-integrations-aod-worker-5d9fd95c9d-fc49x

evicting pod default/run-unified-deduper-sync-1642773600-xbq7q

evicting pod default/flow-commit-reprocess-1642783500-q52wd

pod/flow-commit-reprocess-1642783500-q52wd evicted

pod/run-unified-deduper-sync-1642780800-tpm44 evicted

pod/run-unified-deduper-sync-1642773600-xbq7q evicted

I0121 16:46:11.645116   26071 request.go:645] Throttling request took 1.051446785s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-new-repo-worker-797979d547-t98rb

pod/flow-work-log-enterprise-7bd9c8789d-k9gfv evicted

I0121 16:46:21.845078   26071 request.go:645] Throttling request took 1.265863421s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg

I0121 16:46:31.845197   26071 request.go:645] Throttling request took 1.265877369s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-pr-worker-787b6b97-gxrfg

I0121 16:46:42.045150   26071 request.go:645] Throttling request took 1.453749262s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

pod/flow-integrations-new-repo-worker-797979d547-t98rb evicted

pod/flow-integrations-schedule-worker-76d766b6c8-fncdp evicted

pod/flow-integrations-bp-worker-7575d9d9d6-vcbc4 evicted

pod/flow-integrations-reprocess-worker-55bdf6f8b9-vgkhl evicted

pod/flow-integrations-aod-worker-5d9fd95c9d-fc49x evicted

pod/flow-integrations-pr-worker-787b6b97-gxrfg evicted

I0121 16:46:52.045082   26071 request.go:645] Throttling request took 1.453770122s, request: GET:https://192.168.20.50:6443/api/v1/namespaces/default/pods/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5

pod/flow-integrations-misc-queue-worker-84c97b9cf5-rdxj5 evicted

pod/flow-integrations-integration-lp-worker-c6f986556-gpbng evicted

pod/flow-integrations-inc-repo-worker-5744cb64bf-mhz2w evicted

pod/flow-integrations-ticket-worker-6dd89c477c-hth7g evicted

node/worker-node evicted

Nodes left = 0

[INFO] Deleting worker node ..

node "worker-node" deleted

[INFO] Scaling down flow ..

deployment.apps/flow-frontend scaled

deployment.apps/flow-integrations-aod-worker scaled

deployment.apps/flow-integrations-bp-worker scaled

deployment.apps/flow-integrations-inc-repo-worker scaled

deployment.apps/flow-integrations-integration-lp-worker scaled

deployment.apps/flow-integrations-misc-queue-worker scaled

deployment.apps/flow-integrations-new-repo-worker scaled

deployment.apps/flow-integrations-pr-worker scaled

deployment.apps/flow-integrations-reprocess-worker scaled

deployment.apps/flow-integrations-schedule-worker scaled

deployment.apps/flow-integrations-ticket-worker scaled

deployment.apps/flow-prj-wkr scaled

deployment.apps/flow-proxy scaled

deployment.apps/flow-scheduler scaled

deployment.apps/flow-syslog scaled

deployment.apps/flow-work-log-enterprise scaled

statefulset.apps/flow-db scaled

statefulset.apps/flow-redis-master scaled

cronjob.batch "flow-commit-reprocess" deleted

Pods left = 0 [|]  

[INFO] Disabling Services

[INFO] Stopping kubelet service ..

Removed /etc/systemd/system/multi-user.target.wants/kubelet.service.

[INFO] Stopping and disabling Docker ..

Failed to stop docker.service: Unit docker.service not loaded.

[INFO] Stopping containerd service ..

Removed /etc/systemd/system/multi-user.target.wants/containerd.service.

[INFO] Issuing a hard reset/reboot. Re-run this script after server comes back up to continue.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

 

 

...  server reboots here ...

Log in to the server again and rerun the flow-uninstall script. There’s no need to provide any flags to the uninstall command because the tool preserves the initial flags that were passed to it.

[root@primary-node ~]$ cd flow-enterprise-tools/bin

[root@primary-node bin]$ sudo ./flow-uninstall 

[INFO] Continuing uninstall of Flow ..

[INFO] Services stopped, continuing cleanup ..

[INFO] Removing K8s packages ..

[INFO] Removing package kubeadm .. not found

[INFO] Removing package kubectl .. removed

[INFO] Removing package kubelet .. removed

[INFO] Removing package kubernetes-cni .. removed

[INFO] Removing container runtime packages ..

[INFO] Removing package docker-engine .. not installed 

[INFO] Removing package docker .. not installed 

[INFO] Removing package docker.io .. not installed 

[INFO] Removing package docker-ce .. not installed 

[INFO] Removing package docker-ce-cli .. not installed 

[INFO] Removing package containerd .. not installed 

[INFO] Removing package containerd.io .. removed

[INFO] Cleaning up mounts ..

[INFO] Cleaning up folders ..

[INFO] Removing kubernetes components ..

[INFO] Removing kubernetes config dir ..

[INFO] Removing rook lib dir ..

[INFO] Removing kubelet lib dir ..

[INFO] Removing replicated dir ..

[INFO] Removing weave lib dir ..

[INFO] Removing etcd lib dir ..

[INFO] Removing docker components ..

[INFO] Removing pod logs ..

[INFO] Removing pods logs dir ..

[INFO] Removing containers logs dir ..

[INFO] Removing containerd components ..

[INFO] Removing containerd opt dir ..

[INFO] Removing containerd lib dir ..

[INFO] Removing CNI components ..

[INFO] Removing cni lib dir ..

[INFO] Removing cni config dir ..

[INFO] Removing cni opt dir ..

[INFO] Removing Flow components ..

[INFO] Removing Flow logs dir ..

[INFO] Removing kubeconfig ..

[INFO] Removing root's .kube ..

[INFO] Removing kurl cache ..

[INFO] Removing kurl lib dir ..

[INFO] Removing kubeadm binary ..

[INFO] Syncing filesystem ..

[INFO] Resetting IPTABLES ..

[INFO] ceph raw disks detected..

[INFO] Removing /dev/mapper links of raw disk /dev/mapper/ceph--7a982bbc--8adc--4119--a87a--e436b68da377-osd--data--1a0a0ea6--5ec2--4920--8682--719eb7cab8ce

[INFO] Resetting MBR of raw disk /dev/nvme1n1

Creating new GPT entries.

GPT data structures destroyed! You may now partition the disk using fdisk or other utilities.

[INFO] Removing uninstallation files ..

[INFO] Flow uninstall has been completed.

[INFO] Executing a graceful reboot.

Connection to primary-node.mydomain.com closed by remote host.

Connection to primary-node.mydomain.com closed.

 

... Server reboots here a second time ...

Note that the existing configuration in this example was saved to a file named kots-config-2023-04-21-16-46-09.tar.gz.

This completes the process of uninstalling Flow on the primary node.

For each worker node, follow the exact same steps as described above, including rebooting the server.

back to top

Reinstall Flow Enterprise Server

Now we’re ready to reinstall Flow Enterprise 2023.2.1.

When you ran flow-uninstall, a configuration export file was created. In the above example, the file was called kots-config-2023-04-21-16-46-09.tar.gz. Pass this file to the flow-tools install command to automate the upgrade processes. The command may look like sudo ./flow-tools install -C kots-config-2023-04-21-16-46-09.tar.gz -d xvdb. In this example, a raw device with the name xvdb is being used.  For airgapped installations, the command will look like sudo ./flow-tools install -a --airgap-app-bundle flow-enterprise-2023.2.1.airgap -C kots-config-2024-04-21-17-25-51.tar.gz -d xvdb.

Note: Use other parameters like --tls-cert, --tls-key, --license-file, --config-values to override parts of your configuration bundle if you need to change them when you upgrade. This is useful if you have any expired items from an export of an older version.

[admin_user@primary-node ~]$ sudo flow-tools install -C kots-config-2023-04-21-16-46-09.tar.gz -d 'xvdb'

[INFO] Running flow-tools with args : install -C kots-config-2023-04-21-16-46-09.tar.gz -d xvdb

[INFO] Extracting kots config bundle...

[INFO] Processing 6 configuration maps ..

[INFO] Processed configuration maps.

[INFO] Processing Java options values ..

[INFO] Processed Java options values.

[INFO] Processing 24 obsolete settings ..

[INFO] Processed obsolete settings.

[INFO] Processing settings with defaults ..

[INFO] Found 183 keys with a default setting .. removing defaults

[INFO] Removed 183 default settings.

[INFO] Processing empty settings ..

[INFO] Removed 214 empty settings.

[INFO] Verifying installation environment...

[INFO] HTTP command (curl): OK

[INFO] Archive command (tar): OK

[INFO] Swarm does not exist: OK

[INFO] Verifying system requirements...

[INFO] Checking networking...

[INFO] sysctl command : OK

[INFO] IPV6 Kernel module: LOADED

[INFO] IPV6 Check : OK

[INFO] IPv4 Forwarding: ENABLED

[INFO] Check IPtable Rules: OK

[INFO] Detecting proxy: NOT DETECTED

[INFO] https://replicated.app site check : OK

[INFO] Checking hardware...

[INFO] CPU: OK

[INFO] Memory: OK

[INFO] Space check in /var/lib/containerd: OK

[INFO] Space check in /var/lib/kubelet: OK

[INFO] Space check in /opt/replicated: OK

[INFO] Space check in /var/lib/kurl: OK

[INFO] Space check in /tmp: OK

[INFO] Space for Repo cache in /opt/flow: 199 GB

[INFO] Disk Space Check: OK

[INFO] Non SSD Disks: NOT DETECTED

[INFO] Checking filesystem and permissions...

[INFO] Login restrictions check: OK

[INFO] Selinux Status: enabled

[INFO] Selinux Current mode: permissive

[INFO] bash Umask setting: OK

[INFO] /etc/profile Umask setting: OK

[WARN] Checking PATH for /usr/local/bin: NOT DETECTED

[INFO] Checking distro...

[INFO] No existing ceph raw disks detected

[INFO] Installation type is : NEW

[INFO] Validating block storage device filter...

 

Device match:   /dev/xvdb

Device size:    150G

Device status:  valid

 

[INFO] Total valid block storage: 150G

[INFO] Block storage: OK

[INFO] Adding patch to use raw ceph block devices for installation

[INFO] Installing KOTS application

[INFO] Saving environment

[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/version/v2023.04.13-0/flow-enterprise-nick-dev

[INFO] Fetching join script from: https://k8s.kurl.sh/version/v2023.04.13-0/flow-enterprise-nick-dev/join.sh

 

...

...

...

 

                Installation

                  Complete ✔

 

The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.

 

To access Grafana use the generated user:password of admin:xxxxxxxxx .

 

Kotsadm: http://192.168.0.80:8800

Login with password (will not be shown again): xxxxxxxxx

This password has been set for you by default. It is recommended that you change this password; this can be done with the following command: kubectl kots reset-password default

 

To access the cluster with kubectl:

 

    bash -l

 

Kurl uses /etc/kubernetes/admin.conf, you might want to copy kubeconfig to your home directory:

 

    cp /etc/kubernetes/admin.conf ~/.kube/config

    chown -R 150028 ~/.kube

    echo unset KUBECONFIG >> ~/.bash_profile

    bash -l

 

You will likely need to use sudo to copy and chown /etc/kubernetes/admin.conf.

 

[INFO] Loading environment

Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.

[INFO] No existing ceph raw disks detected

Kubernetes connection credentials for worker node. Expires in 24 hours

Kubernetes Connection String :  kubernetes-master-address=192.168.0.80:6443 kubeadm-token=5geexa.b0k10q8w5lse854l kubeadm-token-ca-hash=sha256:7243fc4365fcc1c576cbb7b086d84a92d1a3a7aebb4009227cf48afe6652fa39 kubernetes-version=1.25.8 docker-registry-ip=10.96.3.47

 

You may add additional command line options to the flow-tools join command.

Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.

 

 

Node join command for this cluster is below:

sudo ./flow-tools join --channel nick-dev  kubernetes-master-address=192.168.0.80:6443 kubeadm-token=5geexa.b0k10q8w5lse854l kubeadm-token-ca-hash=sha256:7243fc4365fcc1c576cbb7b086d84a92d1a3a7aebb4009227cf48afe6652fa39 kubernetes-version=1.25.8 docker-registry-ip=10.96.3.47

 

node/primary-node labeled

[INFO] Primary node has been labelled with

 

        gui=true

        worker=true

 

 

        If adding an additional node, please run the following,

        after adding a worker node:

 

 

                kubectl label nodes worker- --selector='node-role.kubernetes.io/master'

                kubectl label nodes worker= --selector='!node-role.kubernetes.io/master'

 

[]

 

 

  • Reset the admin console password for default

Enter a new password for the admin console (6+ characters): ••••••••

  • The admin console password has been reset

[INFO] Extracting any app and db backups found...

[INFO] Installing app...

 

        /bin/kubectl kots install flow-enterprise/nick-dev --namespace default --kotsadm-namespace default --name flow-enterprise --skip-preflights --wait-duration 5m --license-file /tmp/flow-toolslZb/config/migration/license.yaml --config-values /tmp/flow-toolslZb/config/migration/kots.yaml

 

 

  • Deploying Admin Console

  • Waiting for Admin Console to be ready ✓

  • Waiting for installation to complete ✓

 

  • Done

 

[INFO] Configuring TLS for host: primary-node.mycompany.com

secret/kotsadm-tls annotated

pod "kurl-proxy-kotsadm-6b759788f8-wpqs7" deleted

[INFO] Setting up kubectl command for current user

[INFO] Processing home directory: /home/admin_user

[INFO] Setting up kube-config for user: admin_user

Now Flow is fully upgraded on the primary node. For the other nodes in the cluster, follow the steps for joining a node to the cluster and add them to this cluster. Once the nodes are fully joined and all pods are running, log in to Flow Enterprise using the URL you used to access Flow before upgrading.

Additional configurations after upgrading Flow

Note: If you previously used these settings before upgrading, they should still be enabled after your upgrade.

  • To prevent your Flow Enterprise Server cluster from running into a forced eviction due to a lack of disk space, go to Disk Pressure Check Settings and click the checkbox next to Enable Disk Pressure Check and Scale Down.
  • Enable monitoring services.
  • Automate certificate renewal.

Related articles