RELEASE NOTES
Compatibility with the upcoming 6.0.2 minor release.
DOWNTIME
As the Cloud DB needs to be restarted, there might be up to 5 minutes downtime of the following services:
Connecting to systems
Cloud Portal
Email Notifications
Push Notifications.
Current connections will not be affected.
MAINTENANCE LOG
Jan 7th (PST):
17.20 - Preparation started
17.43 - Cloud Portal update started
17:43 - Cloud DB update started
17:45-17:49 - Cloud DB (downtime)
17.50 - Cloud Portal update completed
17:50 - Cloud DB update completed
17:51 - Testing started
18:08 - Testing ended
Overview
On 2024-12-08 00:20 AM PST, Amazon certificate renewal caused connectivity issues with the nxvms.com domain in various regions.
The issue was resolved by switching the certificate provider for nxvms.com domain to LetsEncrypt.
To avoid the same issue with other Cloud Portals, we have to perform the same procedure ASAP.
Release Notes
We will change TLS certificates provider for Cloud Portal to LetsEncrypt.
We will use certificates generated in K8s cluster.
DOWNTIME
None
MAINTENANCE LOG
19.20 - Preparation started
19.32 - Cloud Portal TLS Certificates update started
19.37 - Cloud Portal TLS Certificates update completed
19:37 - Testing started
19:50 - Testing successfully ended
Release Notes
This is another iteration of deploying the upcoming Enterprise functionality (Channel Partners).
DOWNTIME
None
MAINTENANCE LOG
Dec 19, 2024 (PST):
5:20 PM - Preparation
5:42 PM - Update started
5:54 PM - Update completed
5:55 PM - Testing started
6:13 PM - Testing successfully completed
UPDATE OVERVIEW
We are going to update Tools and Tools-EU. Those services are used to connect VMS to Cloud.
RELEASE NOTES / SCOPE OF WORK
Update OS to Ubuntu 24.04.
Unify all Servers’ settings (except domain names).
Firewall improvements.
SCHEDULE
Monday, Dec 16, 2024 updating Europe:
• 6:00 AM PST (14:00 UTC) - redeploy tools (Europe)
• 6:30 AM - 9:00 AM PST (14:30 - 17:00 UTC) - testing
Wednesday, Dec 19, 2024, 4:00 AM PST - updating US:
• 6:00 AM PST (14:00 UTC) - redeploy tools (Europe)
• 6:30 AM - 9:00 AM PST (14:30 - 17:00 UTC) - testing
DOWNTIME
None. Tools and Tools-EU are redundant: when one is being updated, the other one will be used.
MAINTENANCE LOG
Monday, Dec 16, 2024 - Updating Europe:
5:45 AM PST (13:45 UTC) - Preparation started.
6:00 AM PST (14:00 UTC) - Redeployment started.
6:20 AM PST (14:20 UTC) - Tools successfully redeployed in the Europe region.
6:20 AM PST (14:20 UTC) - Testing started.
7:30 AM PST (15:30 UTC) - Midway testing review completed.
9:00 AM PST (17:00 UTC) - Testing ended.
9:05 AM PST (17:05 UTC) - Final status logged and updates confirmed.
Wednesday, Dec 18, 2024 - Updating US:
5:45 AM PST (13:45 UTC) - Preparation started.
6:00 AM PST (14:00 UTC) - Redeployment started.
6:20 AM PST (14:20 UTC) - Tools successfully redeployed in the US region.
6:20 AM PST (14:20 UTC) - Testing started.
9:00 AM PST (17:00 UTC) - Network performance issues found during tests. Cloud users are not affected.
11:05 AM PST (19:05 UTC) - US instance has been upgraded.
12:15 PM PST (20:15 UTC) - Network performance improved.
2:38 PM PST (22:38 UTC) - Deployment of the updated tools started in the EU region.
2:48 PM PST (22:48 UTC) - Updated tools successfully redeployed in the EU region.
2:48 PM PST (22:48 UTC) - Testing in the EU region started.
3:10 PM PST (23:10 UTC) - Testing in the EU region successfully passed.
3:10 PM PST (23:10 UTC) - Deployment of the updated tools started in the US region.
3:18 PM PST (23:18 UTC) - Updated tools successfully redeployed in the US region.
3:18 PM PST (23:18 UTC) - Testing in the US region started.
3:25 PM PST (23:25 UTC) - Testing in the US region successfully passed.
Release Notes
BUG FIXES:
Archive from cameras streaming in H265 / MJPEG could not be played back on Cloud Portal. Fixed.
Email notifications from VMS could not be sent if routed through Cloud. Fixed.
Emails sent from Cloud Portal through Cloud SMTP Server could still contain incorrect links in signature. Additional fixes were applied.
KNOWN ISSUES:
The Desktop Client displays “Reconnecting” instead of "Your session has expired" window when disconnecting a system from Cloud.
DOWNTIME
There might be up to 4 minutes downtime of the following services:
Cloud Portal
Email Notifications
Push Notifications.
Сloud connectivity will not be affected.
MAINTENANCE LOG
Dec 10:
17.20 - Preparation started
17.55 - Cloud Portal update started
18:55 - 19:01 Cloud Portal unavailable (downtime)
19.17 - Cloud Portal update completed
19:17 - Testing started
20:29 - Testing ended
Summary
On 2024-12-08 00:20 AM PST, Amazon updated the certificates on the nxvms.com domain.
Even though they stated in this article that there should be no issues, we started getting complaints from customers. According to their post, only Windows machines should be involved, however we have been seeing complaints about other OS in various Reddit/Quora threads so they may be affected as well.
Customer Impact
On some Windows (and potentially other OS) machines, the certificates may not be automatically installed. Cloud connections to these servers will not be allowed causing systems to appear unreachable in the Desktop Client.
Required Actions
If you are experiencing problems accessing your systems, it will be necessary to update the certificates (recommended) or run the OS update on every single computer running VMS Server:
Download the following certificates from the Amazon Trust Services Repository:
Amazon Root CA 1
Amazon Root CA 2
Amazon Root CA 3
Amazon Root CA 4
Starfield Services Root Certificate Authority - G2.
Install the certificates on your OS (*.pem - for Linux, *.cer - for Windows).
[Linux only] Update the certificate store:
sudo update-ca-certificates.Restart VMS server.
If the issue persists, please contact Support.
We understand that the process of installing certificates on each computer may be time-consuming and painful, but we do not have other solutions at the moment.
We sincerely apologize for the inconvenience caused.
Further Actions
In the future, we will be monitoring similar announcements from Amazon and making sure they will not affect VMS Servers. Also we will be communicating such changes way in advance.
2024-12-08 8:00 PM PST Update
As of now, we have replaced the Amazon-issued SSL certificate (Amazon RSA) with a Let's Encrypt certificate for nxvms.com domain.
This change has resolved the connectivity issue.
As for now, no further actions are required on your end, even though we recommended this as a temporary solution in earlier communications. We shared that workaround as a known fix while we were working on finding and implementing a permanent solution.
Initial reports from the field indicate that this change has successfully restored functionality for affected systems. Please note, however, that this is part of an interim solution, and additional maintenance may be required in the future to ensure long-term stability.
We will follow up with the updates in this case.
We sincerely apologize for any inconvenience this issue has caused and greatly appreciate your patience and cooperation as we work to maintain the reliability and security of our services.
If you continue to experience any issues or have further questions, please don’t hesitate to reach out to our support team.