If you’re upgrading from a previous Flow version to 2021.1.1, it will require a reinstallation of Flow from the command line. You cannot upgrade through the KOTS admin console. You must export your Flow configuration, uninstall Flow from all nodes, then reinstall Flow. Follow the steps outlined below to upgrade to Flow 2021.1.1.
In this article
Before you begin
Before upgrading to Flow Enterprise 2021.1.1 there are a few things to know and be prepared for, Read through this information carefully before beginning the upgrade process.
Check out the Flow Enterprise Server 2021.1.1 release notes for an overview of updates included in this version.
Important: To upgrade to 2021.1.1, you must reinstall Flow from the command line. You cannot upgrade through the KOTS admin console. This article will go through the process of exporting your Flow configuration, uninstalling Flow, then reinstalling Flow.
New system requirements for 2021.1.1
- Flow strongly recommends having a three node configuration to ensure proper disk quorum (opens in new tab) and system stability. The Flow software will show a warning message on the KOTS admin console if you have fewer than three nodes, but installation will proceed. Add additional nodes to your configuration at any point.
- Flow Enterprise Server is moving to require a single, raw block device for Ceph storage usage on every node in the cluster. This is optional for upgrades in the 2021.1.1 release. Future versions of Flow Enterprise Server will require raw block devices for upgrades.
When you add your raw block device, you must do a full reinstallation, including exporting your configurations and uninstalling Flow. This is true whether you add the raw block device when upgrading to 2021.1.1 or if you add it later.
- The computing and storage requirements, including some directories, have been updated. Please read the Flow Enterprise Server 2021.1.1 system requirements to see the updated requirements and directories. Disk space requirements for nodes have increased.
Information to gather before you begin
- Your Flow license file.
- Your TLS/SSL certificates.
- The latest version of the
flow-enterprise-toolspackage. Request this from Pluralsight Support.Important: You must have the latest version offlow-enterprise-toolsto successfully upgrade. - Pluralsight recommends backing up your database before you begin the upgrade process.
- If your version of Flow Enterprise Server is airgapped, download the airgap bundle from Replicated. A password is required. If you can't access Replicated, contact Support for assistance.
Tip: Depending on how you install Flow, you need to download a few different packages.
flow-enterprise-toolscontains the tools for installation and maintenance of the Kubernetes framework. Download the airgap version for airgapped installations. The app airgap bundle, only for airgap installations, is downloaded from Replicated. It contains the Flow application files which are installed into the Kubernetes framework provided byflow-enterprise-tools.
To install the flow-enterprise-tools package:
- For the host server, copy
flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gzto the home directory of the user account used for the installation on the host server. - Extract the tools file using
tar xvf flow-enterprise-tools-<channel>[-airgap]-<version>.tar.gz.Note: You can run any tool from the
bindirectory by runningcd /path/to/flow-enterprise-tools/bin ./[tool name]. Install the tools package with theinstall-enterprise-tools.shscriptcd /path/to/flow-enterprise-tools ./install-enterprise-tools.sh. The script will ask where to install the components. The default is/usr/local/share/flow-enterprise-tools.
Verify your current KOTS installation
On the primary node, run kubectl get installers flow-enterprise -o yaml to ensure you are running the latest version of Flow Enterprise with a KOTS version of 1.19.x and Kubernetes version 1.17.x.
Tip: Example text outputs are included in this article.
admin-user@primary-node:~$ kubectl get installers flow-enterprise -o yaml
apiVersion: cluster.kurl.sh/v1beta1
kind: Installer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cluster.kurl.sh/v1beta1","kind":"Installer","metadata":{"annotations":{},"name":"flow-enterprise","namespace":"default"},"spec":{"contour":{"version":"1.7.0"},"docker":{"version":"19.03.10"},"kotsadm":{"applicationSlug":"flow-enterprise","version":"1.19.6"},"kubernetes":{"version":"1.17.7"},"prometheus":{"version":"0.33.0"},"registry":{"version":"2.7.1"},"rook":{"version":"1.0.4"},"weave":{"version":"2.6.5"}}}
creationTimestamp: "2021-04-06T19:50:30Z"
generation: 1
labels:
velero.io/exclude-from-backup: "true"
name: flow-enterprise
namespace: default
resourceVersion: "390"
selfLink: /apis/cluster.kurl.sh/v1beta1/namespaces/default/installers/flow-enterprise
uid: 78c208b3-bfb9-43c2-ac6f-c7ffad9bc592
spec:
contour:
version: 1.7.0
docker:
version: 19.03.10
kotsadm:
applicationSlug: flow-enterprise
version: 1.19.6
kubernetes:
version: 1.17.7
prometheus:
version: 0.33.0
registry:
version: 2.7.1
rook:
version: 1.0.4
weave:
version: 2.6.5
Export your Flow configuration
Next, export the current KOTS configuration using flow-enterprise-tools with the command sudo flow-tools export --preserve-tls. Make sure to download the latest version of flow-enterprise-tools from Pluralsight. Version 2.0.x is required.
Note: For the root user, /usr/local/bin must be in the PATH environment variable. The root user should be set up as a Flow user after the installation of Flow. This is critical if your OS is hardened.
admin-user@primary-node:~$ sudo flow-tools export --preserve-tls
[INFO] Checking environment...
[INFO] Archive command (tar): OK
[INFO] HTTP command (curl): OK
[INFO] YAML command (yq): OK
[INFO] KOTS CLI command (kubectl kots): OK
[INFO] Flow app installed: OK
[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq
• Connecting to cluster ✓
The application manifests have been downloaded and saved in /tmp/flow-export-to-kotsfVm/flow-enterprise
After editing these files, you can upload a new version using
kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-export-to-kotsfVm/flow-enterprise
[INFO] Exporting config bundle: kots-config-2021-04-07-18-19-56.tar.gz
[INFO] Cleaning up...
Validate that the exported configuration file has all the required files in it as shown below. Extract the tar.gz file just created using the tar tvf <configuration file> command.
admin-user@primary-node:~$ tar tvf kots-config-2021-04-07-18-27-33.tar.gz
drwxr-xr-x root/root 0 2021-04-07 18:27 migration/
-rw-r--r-- root/root 1704 2021-04-07 18:27 migration/server.key
-rw-r--r-- root/root 12352 2021-04-07 18:27 migration/kots.yaml
-rw-r--r-- root/root 3460 2021-04-07 18:27 migration/server.pem
-rw-r--r-- root/root 5443 2021-04-07 18:27 migration/license.yaml
Note: If you have an external database connected to Flow, it will not be impacted or removed by default. If you used an embedded database for testing, etc., please use flow-tools export --help for additional options to back up the application, repository cache, and database directories. Also, if the node is being migrated to a different external server, utilize additional flags to the flow-tools export command accordingly.
Uninstall Flow Enterprise
Next, uninstall Flow Enterprise from all nodes. Remove the primary node first, then all worker nodes.
Run kubectl get nodes on the primary node to verify the current state.
admin-user@primary-node:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
primary-node Ready master 22h v1.17.7
worker-node Ready <none> 21h v1.17.7
Run sudo flow-uninstall to remove Flow Enterprise from the primary node. Follow the prompts and press y to continue.
admin-user@primary-node:~$ sudo flow-uninstall
[WARN] ======================= WARNING ===========================
[WARN] Running this script will remove Kubernetes and all
[WARN] related components, Flow application from this server.
[WARN] Worker nodes will be drained and deleted from cluster.
[WARN] Following directory contents will be removed:
[WARN] var/lib/kubelet/var/lib/rook
[WARN] /var/lib/ceph~/.kube/var/lib/docker/opt/flow
[WARN] /opt/replicated/var/lib/weave
[WARN] ===========================================================
=== Warning: Removing all Kubernetes components ===
Enter y/n to continue or exit : y
[INFO] Checking environment...
[INFO] Cleaning up...
[INFO] Backing up database...
[INFO] Scaling down services
...
[INFO] Waiting for services to scale down...
[INFO] Waiting for services to scale down...
[INFO] Backing up embedded db
[INFO] Master node detected with 1 worker nodes
[INFO] Draining worker node ..
[INFO] Draining node = test-ubuntu-fossa-01-b
...
[INFO] Deleting worker node ..
...
[INFO] Scaling down flow ..
...
[INFO] Deleting Embedded DB pods ..
...
[INFO] Disabling Services
[INFO] Stopping kubelet service ..
...
[INFO] Stopping and disabling Docker ..
...
[INFO] Stopping containerd service ..
...
[INFO] Please reboot the server and re-run this script to continue uninstall.
[INFO] Note: May need to do a force reboot/hard reset to clear mounted docker folders
[INFO] run: 'sudo sh -c 'echo b > /proc/sysrq-trigger''
Reboot the server as directed. Once the server is back up, rerun sudo flow-uninstall one more time.
admin-user@primary-node:~$ sudo flow-uninstall
[INFO] Continuing uninstall of Flow ..
[INFO] Docker Services stopped, continuing cleanup ..
[INFO] Removing K8s packages ..
...
[INFO] Removing docker packages ..
...
[INFO] Cleaning up mounts ..
[INFO] Cleaning up folders ..
...
[INFO] Resetting IPTABLES ..
[INFO] No ceph raw disks detected..
[INFO] Flow uninstall has been completed.
This completes the process of uninstalling Flow on the primary node. Next, run flow-uninstall on each worker node. For each worker node, follow the exact same steps as described above, including rebooting the server.
Reinstall Flow Enterprise
Now we are ready to reinstall Flow Enterprise 2021.1.1. For an online installation, run sudo flow-tools install. For an airgapped installation, run sudo flow-tools install -a. This only needs to be done in the primary node.
Important: A raw block device for each node is required for new Flow installations. If you are upgrading to Flow 2021.1.1+ and an exported configuration is detected during installation, you will receive a warning about this requirement, but will be able to proceed without the raw block device requirement. This will become a requirement for all future upgrades.
The device filter is in golang regular expression format. Device names vary between distributions. You can add multiple filters if the devices are named differently in worker nodes. An example of such a case may look like sd[b-z]|nvme[1-3]|xvd[b-c].
admin-user@primary-node:~$ sudo flow-tools install
[INFO] Verifying installation environment...
[INFO] HTTP command (curl): OK
[INFO] Archive command (tar): OK
[INFO] Swarm does not exist: OK
[INFO] Verifying system requirements...
[INFO] Checking networking...
[INFO] sysctl command : OK
[INFO] IPV6 Kernel module: LOADED
[INFO] IPV6 Check : OK
[INFO] IPv4 Forwarding: ENABLED
[INFO] Check IPtable Rules: OK
[INFO] Checking ufw firewall: INACTIVE
[INFO] Detecting proxy: NOT DETECTED
[INFO] https://replicated.app site check : OK
[INFO] Checking hardware...
[INFO] CPU: OK
[INFO] Memory: OK
[INFO] Space check in /var/lib/containerd: OK
[INFO] Space check in /var/lib/kubelet: OK
[INFO] Space check in /opt/replicated: OK
[INFO] Space for Repo cache in /opt/flow: 149 GB
[INFO] Disk Space Check: OK
[INFO] Non SSD Disks: NOT DETECTED
[INFO] Checking filesystem and permissions...
[INFO] Login restrictions check: OK
[INFO] bash Umask setting: OK
[INFO] /etc/profile Umask setting: OK
[INFO] Checking PATH for /usr/local/bin: OK
[INFO] Checking distro...
[INFO] No existing ceph raw disks detected
[INFO] Installation type is : NEW
=== Discovered Block Devices ===
/dev/nvme1n1
Above is the list of block devices found during valid device discovery
Please provide pattern to match devices that should be used for K8s volume storage: nvme[1-9]n1
[INFO] Validating block storage device filter...
Device match: /dev/nvme1n1
Device size: 500G
Device status: valid
[INFO] Total valid block storage: 500G
[INFO] Block storage: OK
[INFO] Adding patch to use raw ceph block devices for installation
[INFO] Installing KOTS application
[INFO] Saving environment
[INFO] Fetching kurl.sh installation script from: https://k8s.kurl.sh/flow-enterprise
[INFO] Fetching join script from: https://k8s.kurl.sh/flow-enterprise/join.sh
...
Installation
Complete ✔
The UIs of Prometheus, Grafana and Alertmanager have been exposed on NodePorts 30900, 30902 and 30903 respectively.
To access Grafana use the generated user:password of admin:yJ8faebKRe .
Kotsadm: http://192.168.0.180:8800
Login with password (will not be shown again): 3fwbTdGgj
To access the cluster with kubectl, reload your shell:
bash -l
[INFO] Loading environment
node/primary-node labeled
[INFO] Primary node has been labelled with
gui=true
worker=true
If adding an additional node, please run the following,
after adding a worker node:
kubectl label nodes worker- --selector='node-role.kubernetes.io/master'
[INFO] Existing ceph disk found : /dev/nvme1n1
Kubernetes connection credentials for worker node. Expires in 24 hours
Kubernetes Connection String : kubernetes-master-address=192.168.0.180:6443 kubeadm-token=e8sj6l.4h2oha16vr7zeths kubeadm-token-ca-hash=sha256:4373659bec1eafea7e91f13f75a91319dbdeeb6e52422e8af4caff4eab299f08 kubernetes-version=1.19.7 docker-registry-ip=10.96.1.73
You may add additional command line options to the flow-tools join command.
Run ./flow-tools join --help for all available flags and options like [ -a|-f|-k|-n|--proxy ] etc.
Node join command for this cluster is below:
sudo ./flow-tools join kubernetes-master-address=192.168.0.180:6443 kubeadm-token=e8sj6l.4h2oha16vr7zeths kubeadm-token-ca-hash=sha256:4373659bec1eafea7e91f13f75a91319dbdeeb6e52422e8af4caff4eab299f08 kubernetes-version=1.19.7 docker-registry-ip=10.96.1.73
[]
[INFO] To change the admin console password, run the following command:
kubectl kots reset-password -n default
[INFO] Setting up kubectl command for current user
[INFO] Processing home directory: /home/admin-user
[INFO] Setting up kube-config for user: admin-user
On a single node cluster, the installation will label the primary node appropriately by default. However, in a multi-node installation, label the worker nodes as directed from the output of the installation command as seen above.
Reset the console password by running kubectl kots reset-password -n default if desired.
Next, install Flow Enterprise on each of the worker nodes using the join command output at the end of the installation of the primary node. Read more about joining a node to the cluster.
Then, open your browser and go to the URL provided at the end of the installation on the primary node. It will look like http://<ip address of server>:8800.
Click Continue to set up and configure the hostname and SSL certificates. Then log in to the KOTS admin console using the password created during the installation.
Then upload your license file.
If you have an airgapped installation, upload the airgap bundle you downloaded from Replicated before you began the installation.
Once the license is validated, you'll see the configuration screen. Click Save and continue to the next screen.
Then click Deploy
Import Flow configurations
Go back to the terminal session on the primary node. Run flow-tools import -i ./kots-config-YYYY-MM-DD-HH-mm-ss.tar.gz. This is the file created during the flow-tools export command.
admin-user@primary-node:~$ flow-tools import -i kots-config-2020-01-01-08-00-00.tar.gz
[INFO] Checking environment...
[INFO] Archive command (tar): OK
[INFO] HTTP command (curl): OK
[INFO] YAML command (yq): OK
[INFO] Kubernetes CLI command (kubectl): OK
[INFO] KOTS CLI command (kubectl kots): OK
[INFO] Input file (kots-config-2021-04-07-18-37-00.tar.gz): OK
[INFO] Extracting kots config bundle...
[INFO] Extracting any app and db backups found...
[INFO] Using yq command: /usr/local/share/flow-enterprise-tools/packages/yq
[INFO] Exporting KOTS config
• Connecting to cluster ✓
The application manifests have been downloaded and saved in /tmp/flow-import-to-kotsMDR/flow-enterprise
After editing these files, you can upload a new version using
kubectl kots upload --namespace default --slug flow-enterprise /tmp/flow-import-to-kotsMDR/flow-enterprise
[INFO] Merging swarm config into KOTS config
[INFO] Merge result code: 0
[INFO] Updating KOTS with new config
• Uploading local application to Admin Console ✓
Please make a minor config change in the admin console, and deploy to finalize the install!
[INFO] Cleaning up...
Once this completes, check the status of the pods. Run watch kubectl get pods and wait until all pods are in the running state, Once they are all in the running state, log in to the Flow Enterprise application using the URL you used to access Flow before upgrading.
If you need help, please contact Pluralsight Support.