Skip to main content
Contact Support

Upgrading to Flow Enterprise Server 2021.1.1

Tags:

If you’re upgrading from a previous Flow version to 2021.1.1, it will require a reinstallation of Flow from the command line. You cannot upgrade through the KOTS admin console. You must export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow. Follow the steps outlined below to upgrade to Flow 2021.1.1.

Before you begin

Before upgrading to Flow Enterprise 2021.1.1 there are a few things to know and be prepared for, Read through this information carefully before beginning the upgrade process.

Check out the Flow Enterprise Server 2021.1.1 release notes for an overview of updates included in this version.

Important: To upgrade to 2021.1.1, you must reinstall Flow from the command line. You cannot upgrade through the KOTS admin console. This article will go through the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.

New system requirements for 2021.1.1

  • Flow strongly recommends having a three node configuration to ensure proper disk quorum (opens in new tab) and system stability. The Flow software will show a warning message on the KOTS admin console if you have fewer than three nodes, but installation will proceed. Add additional nodes to your configuration at any point.
  • Flow Enterprise Server is moving to require a single, raw block device for Ceph storage usage on every node in the cluster. This is optional for upgrades in the 2021.1.1 release. Future versions of Flow Enterprise Server will require raw block devices for upgrades.

    When you add your raw block device, you must do a full reinstallation, including exporting your configurations and uninstalling Flow. This is true whether you add the raw block device when upgrading to 2021.1.1 or if you add it later.

  • The computing and storage requirements, including some directories, have been updated. Please read the Flow Enterprise Server 2021.1.1 system requirements to see the updated requirements and directories. Disk space requirements for nodes have increased.

Information to gather before you begin

  • Your Flow license file.
  • Your TLS/SSL certificates.
  • The latest version of the flow-enterprise-tools package. Request this from Pluralsight Support.
    Important: You must have the latest version of flow-enterprise-tools to successfully upgrade.
  • Pluralsight recommends backing up your database before you begin the upgrade process.
  • If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated. A password is required. If you can't access Replicated, contact Support for assistance.

    Tip: Depending on how you install Flow, you need to download a few different packages. flow-enterprise-tools contains the tools for installation and maintenance of the Kubernetes framework. Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided by flow-enterprise-tools.

To install the flow-enterprise-tools package:

  • For the host server, copy flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz to the home directory of the user account used for the installation on the host server.
  • Extract the tools file using tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.

    Note: You can run any tool from the bin directory by running cd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with the install-enterprise-tools.sh script cd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is /usr/local/share/flow-enterprise-tools.

back to top

Verify your current KOTS installation

On the primary node, run kubectl get installers flow-enterprise -o yaml to ensure you are running the latest version of Flow Enterprise with a KOTS version of 1.19.x and Kubernetes version 1.17.x.

Tip: Example text outputs are included in this article.

admin-user@primary-node:~$ kubectl get installers flow-enterprise -o yaml

apiVersion: cluster.kurl.sh/v1beta1

kind: Installer

metadata:

  annotations:

    kubectl.kubernetes.io/last-applied-configuration: |

      {"apiVersion":"cluster.kurl.sh/v1beta1","kind":"Installer","metadata":{"annotations":{},"name":"flow-enterprise","namespace":"default"},"spec":{"contour":{"version":"1.7.0"},"docker":{"version":"19.03.10"},"kotsadm":{"applicationSlug":"flow-enterprise","version":"1.19.6"},"kubernetes":{"version":"1.17.7"},"prometheus":{"version":"0.33.0"},"registry":{"version":"2.7.1"},"rook":{"version":"1.0.4"},"weave":{"version":"2.6.5"}}}

  creationTimestamp: "2021-04-06T19:50:30Z"

  generation: 1

  labels:

    velero.io/exclude-from-backup: "true"

  name: flow-enterprise

  namespace: default

  resourceVersion: "390"

  selfLink: /apis/cluster.kurl.sh/v1beta1/namespaces/default/installers/flow-enterprise

  uid: 78c208b3-bfb9-43c2-ac6f-c7ffad9bc592

spec:

  contour:

    version: 1.7.0

  docker:

    version: 19.03.10

  kotsadm:

    applicationSlug: flow-enterprise

    version: 1.19.6

  kubernetes:

    version: 1.17.7

  prometheus:

    version: 0.33.0

  registry:

    version: 2.7.1

  rook:

    version: 1.0.4

  weave:

    version: 2.6.5

back to top

Export your Flow configuration

Next, export the current KOTS configuration using flow-enterprise-tools with the command sudo flow-tools export --preserve-tls. Make sure to download the latest version of flow-enterprise-tools from Pluralsight. Version 2.0.x is required.

Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.

admin-user@primary-node:~$ sudo flow-tools export --preserve-tls

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Flow app installed: OK

[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq

  • Connecting to cluster ✓

    The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise

After editing these files, you can upload a new version using

      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise

 

[INFO] Exporting config bundle: kots-config-2021-04-07-18-19-56.tar.gz

[INFO] Cleaning up...

Validate that the exported configuration file has all the required files in it as shown below. Extract the tar.gz file just created using the tar tvf <configuration file> command.

admin-user@primary-node:~$ tar tvf kots-config-2021-04-07-18-27-33.tar.gz

drwxr-xr-x root/root         0 2021-04-07 18:27 migration/

-rw-r--r-- root/root      1704 2021-04-07 18:27 migration/server.key

-rw-r--r-- root/root     12352 2021-04-07 18:27 migration/kots.yaml

-rw-r--r-- root/root      3460 2021-04-07 18:27 migration/server.pem

-rw-r--r-- root/root      5443 2021-04-07 18:27 migration/license.yaml

Note: If you have an external database connected to Flow, it will not be impacted or removed by default. If you used an embedded database for testing, etc., please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.

back to top

Uninstall Flow Enterprise

Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes.

Run kubectl get nodes on the primary node to verify the current state.

admin-user@primary-node:~$ kubectl get nodes

NAME                     STATUS   ROLES    AGE   VERSION

primary-node             Ready    master   22h   v1.17.7

worker-node              Ready    <none>   21h   v1.17.7

Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue.

admin-user@primary-node:~$ sudo flow-uninstall

[WARN] ======================= WARNING ===========================

[WARN] Running this script will remove Kubernetes and all

[WARN] related components, Flow application from this server.

[WARN] Worker nodes will be drained and deleted from cluster.

[WARN] Following directory contents will be removed:

[WARN] var/lib/kubelet/var/lib/rook

[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow

[WARN] /opt/replicated/var/lib/weave

[WARN] ===========================================================

=== Warning: Removing all Kubernetes components ===

Enter y/n to continue or exit : y

 

[INFO] Checking environment...

[INFO] Cleaning up...

[INFO] Backing up database...

[INFO] Scaling down services

...

[INFO] Waiting for services to scale down...

[INFO] Waiting for services to scale down...

[INFO] Backing up embedded db

[INFO] Master node detected with 1 worker nodes

[INFO] Draining worker node ..

[INFO] Draining node = test-ubuntu-fossa-01-b

...

[INFO] Deleting worker node ..

...

[INFO] Scaling down flow ..

...

[INFO] Deleting Embedded DB pods ..

...

[INFO] Disabling Services

[INFO] Stopping kubelet service ..

... 

[INFO] Stopping and disabling Docker ..

... 

[INFO] Stopping containerd service ..

... 

[INFO] Please reboot the server and re-run this script to continue uninstall.

[INFO] Note: May need to do a force reboot/hard reset to clear mounted docker folders

[INFO] run: 'sudo sh -c 'echo b > /proc/sysrq-trigger''

Reboot the server as directed. Once the server is back up, rerun sudo flow-uninstall one more time.

admin-user@primary-node:~$ sudo flow-uninstall

[INFO] Continuing uninstall of Flow ..

[INFO] Docker Services stopped, continuing cleanup ..

[INFO] Removing K8s packages ..

... 

[INFO] Removing docker packages ..

... 

[INFO] Cleaning up mounts ..

[INFO] Cleaning up folders ..

... 

[INFO] Resetting IPTABLES ..

[INFO] No ceph raw disks detected..

[INFO] Flow uninstall has been completed.

This completes the process of uninstalling Flow on the primary node. Next, run flow-uninstall on each worker node. For each worker node, follow the exact same steps as described above, including rebooting the server.

back to top

Reinstall Flow Enterprise

Now we are ready to reinstall Flow Enterprise 2021.1.1. For an online installation, run sudo flow-tools install. For an airgapped installation, run sudo flow-tools install -a. This only needs to be done in the primary node.

Important: A raw block device for each node is required for new Flow installations. If you are upgrading to Flow 2021.1.1+ and an exported configuration is detected during installation, you will receive a warning about this requirement, but will be able to proceed without the raw block device requirement. This will become a requirement for all future upgrades.

The device filter is in golang regular expression format. Device names vary between distributions. You can add multiple filters if the devices are named differently in worker nodes. An example of such a case may look like sd[b-z]|nvme[1-3]|xvd[b-c].

admin-user@primary-node:~$ sudo flow-tools install

[INFO] Verifying installation environment...

[INFO] HTTP command (curl): OK

[INFO] Archive command (tar): OK

[INFO] Swarm does not exist: OK

[INFO] Verifying system requirements...

[INFO] Checking networking...

[INFO] sysctl command : OK

[INFO] IPV6 Kernel module: LOADED

[INFO] IPV6 Check : OK

[INFO] IPv4 Forwarding: ENABLED

[INFO] Check IPtable Rules: OK

[INFO] Checking ufw firewall: INACTIVE

[INFO] Detecting proxy: NOT DETECTED

[INFO] https://replicated.app site check : OK

[INFO] Checking hardware...

[INFO] CPU: OK

[INFO] Memory: OK

[INFO] Space check in /var/lib/containerd: OK

[INFO] Space check in /var/lib/kubelet: OK

[INFO] Space check in /opt/replicated: OK

[INFO] Space for Repo cache in /opt/flow: 149 GB

[INFO] Disk Space Check: OK

[INFO] Non SSD Disks: NOT DETECTED

[INFO] Checking filesystem and permissions...

[INFO] Login restrictions check: OK

[INFO] bash Umask setting: OK

[INFO] /etc/profile Umask setting: OK

[INFO] Checking PATH for /usr/local/bin: OK

[INFO] Checking distro...

[INFO] No existing ceph raw disks detected

[INFO] Installation type is : NEW

=== Discovered Block Devices ===

 

/dev/nvme1n1

 

Above is the list of block devices found during valid device discovery

Please provide pattern to match devices that should be used for K8s volume storage: nvme[1-9]n1

 

[INFO] Validating block storage device filter...

 

Device match:    /dev/nvme1n1

Device size:    500G

Device status:    valid

 

[INFO] Total valid block storage: 500G

[INFO] Block storage: OK

[INFO] Adding patch to use raw ceph block devices for installation

[INFO] Installing KOTS application

[INFO] Saving environment

[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/flow-enterprise

[INFO] Fetching join script from: https://k8s.kurl.sh/flow-enterprise/join.sh

... 

        Installation

          Complete ✔

 

 

The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.

 

To access Grafana use the generated user:password of admin:yJ8faebKRe .

 

 

 

 

Kotsadm: http://192.168.0.180:8800

Login with password (will not be shown again): 3fwbTdGgj

 

To access the cluster with kubectl, reload your shell:

 

    bash -l

 

[INFO] Loading environment

node/primary-node labeled

[INFO] Primary node has been labelled with

 

    gui=true

    worker=true

 

    If adding an additional node, please run the following,

    after adding a worker node:

 

        kubectl label nodes worker- --selector='node-role.kubernetes.io/master'

 

[INFO] Existing ceph disk found : /dev/nvme1n1

Kubernetes connection credentials for worker node. Expires in 24 hours

Kubernetes Connection String :  kubernetes-master-address=192.168.0.180:6443 kubeadm-token=e8sj6l.4h2oha16vr7zeths kubeadm-token-ca-hash=sha256:4373659bec1eafea7e91f13f75a91319dbdeeb6e52422e8af4caff4eab299f08 kubernetes-version=1.19.7 docker-registry-ip=10.96.1.73

 

You may add additional command line options to the flow-tools join command.

Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.

 

 

Node join command for this cluster is below:

sudo ./flow-tools join kubernetes-master-address=192.168.0.180:6443 kubeadm-token=e8sj6l.4h2oha16vr7zeths kubeadm-token-ca-hash=sha256:4373659bec1eafea7e91f13f75a91319dbdeeb6e52422e8af4caff4eab299f08 kubernetes-version=1.19.7 docker-registry-ip=10.96.1.73

 

[]

 

 

[INFO] To change the admin console password, run the following command:

    kubectl kots reset-password -n default

[INFO] Setting up kubectl command for current user

[INFO] Processing home directory: /home/admin-user

[INFO] Setting up kube-config for user: admin-user

On a single node cluster, the installation will label the primary node appropriately by default. However, in a multi-node installation, label the worker nodes as directed from the output of the installation command as seen above.

Reset the console password by running kubectl kots reset-password -n default if desired.

Next, install Flow Enterprise on each of the worker nodes using the join command output at the end of the installation of the primary node. Read more about joining a node to the cluster.

Then, open your browser and go to the URL provided at the end of the installation on the primary node. It will look like http://<ip address of server>:8800.

Click Continue to set up and configure the hostname and SSL certificates. Then log in to the KOTS admin console using the password created during the installation.

Then upload your license file.

If you have an airgapped installation, upload the airgap bundle you downloaded from Replicated before you began the installation.

Once the license is validated, you'll see the configuration screen. Click Save and continue to the next screen.

Then click Deploy

back to top

Import Flow configurations

Go back to the terminal session on the primary node. Run flow-tools import -i ./kots-config-YYYY-MM-DD-HH-mm-ss.tar.gz. This is the file created during the flow-tools export command.

admin-user@primary-node:~$ flow-tools import -i kots-config-2020-01-01-08-00-00.tar.gz

[INFO] Checking environment...

[INFO] Archive command (tar): OK

[INFO] HTTP command (curl): OK

[INFO] YAML command (yq): OK

[INFO] Kubernetes CLI command (kubectl): OK

[INFO] KOTS CLI command (kubectl kots): OK

[INFO] Input file (kots-config-2021-04-07-18-37-00.tar.gz): OK

[INFO] Extracting kots config bundle...

[INFO] Extracting any app and db backups found...

[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq

[INFO] Exporting KOTS config

  • Connecting to cluster ✓

 

    The application manifests have been downloaded and saved in /tmp/flow-import-to-kotsMDR/flow-enterprise

 

After editing these files, you can upload a new version using

 

      kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-import-to-kotsMDR/flow-enterprise

 

 

[INFO] Merging swarm config into KOTS config

[INFO] Merge result code: 0

[INFO] Updating KOTS with new config

  • Uploading local application to Admin Console ✓

Please make a minor config change in the admin console, and deploy to finalize the install!

[INFO] Cleaning up...

Once this completes, check the status of the pods. Run watch kubectl get pods and wait until all pods are in the running state, Once they are all in the running state, log in to the Flow Enterprise application using the URL you used to access Flow before upgrading.

Related articles