Category: Technical

A critique of Ubiquiti Dream Machine (UDM) Pro, etc.

Peter

Last updated February 22, 2023

My partner and I purchased a new home, and it came prewired with CAT5e, and outside rough-ins for security cameras.

With those requirements in mind, I settled on the following hardware from Ubiquiti for an end-to-end solution:

I decided it was time to get "serious" and have a proper pro-sumer setup at home for Internet access, and while we were at it: Security cameras! I didn't want to go with Hikvision or any of the other Chinese manufacturers of IP cameras.

  • UniFi Dream Machine (UDM) Pro (replaced with UNVR, and OpenWrt on a Protectli x86 machine)
  • UniFi Switch 16 POE-150W
  • UniFi Switch Flex 5 POE
  • 2x UniFi AP-Flex-HD

For security cameras (UniFI Protect), I went with the following:

  • 2x G4 Pro
  • 2x G3 Pro
  • UFP Viewport

These products, in my opinion, are not prosumer. They have a number of issues, especially for the price point (I would expect better.) In a lot of cases, pfsense would buy you better bang for your buck.

UDM Pro / Network

Dual WAN

  • Does not support load balancing, only fail over (despite option being available in the UI – disabled in 1.8.3 and above)
  • Port forwarding only works on the active WAN connection, despite both WAN connections being selected for the port forwards
  • No ability to set the primary WAN interface
    • Documentation claims to allow you to swap WAN interface port assignments around, unfortunately swapping interfaces around causes no interfaces to work
  • No ability to set independent upstream and downstream speeds per WAN interface (ala "Smart queues")
  • Static routes only work with the active WAN interface. My modem for Shaw uses 10.0.0.1 for management, to see levels, etc. I added a static interface route for 10.0.0.1/32 pointed at the interface for the modem, it does not work. No route is added, and nothing shows in iptables. If I add the route manually via SSH, it works.
  • The inactive WAN interface: DHCP will stop working, think it still has it's old IP address. Via the GUI it will say the interface is up, but via command line the interface will be shown as failed. To fix you need to restart the UDM.

Missing Features

  • No link aggregation (LACP or LAG) support
  • No policy routing (load balancing of Dual WAN doesn't work, so what's the point?)
  • Proper bandwidth limiting via QoS
  • No ability to see default blocked firewall rules

Incompatibilities

  • Unifi Controller does not support VLAN IDs above 4009.

Bugs

  • On 1.8.3 I experienced a condition where the UDM Pro stopped accepting traffic to Protect, Network and the web interface. I was not able to safely reboot the machine and had to power cycle it to restore service.
    • This issue has occurred on 1.9.3 as well. System ends up in an out of memory condition and will stop recording security cameras and you have to power cycle the device. It seems to occur when the Web UI is used.
  • DPI statistics do not accurately reflect traffic counts, and has apparently been broken for years. I have 900GB of traffic attributed to PayPal, and if you click for a breakdown about 100KB is used in total.
  • Unifi Network App on Android will drain battery if left to run in the background. Does not reliably provide notifications/alerts that are displayed in the web interface.
  • Adding new WiFi SSID causes all the Access Points to restart and service is unavailable for a few minutes.
  • Restarting the UDM Pro is a gamble, sometimes it will not boot at all and will display "UDM-Pro is having an issue booting" and stop there
  • Access Points will randomly stop working until PoE port is power cycled
  • L2 Isolation for Wireless Networks does not, in fact, isolate traffic between wireless clients
  • Restoring backups after running the setup wizard does not allow you to login, as the admin password is blank in the database
  • Restoring backups from Network does not work during the Setup Wizard for Unifi Controller, leaving the controller unusable

UNVR / Unifi Protect

I switched to a UNVR from UDM Pro, in hopes that the performance and stability issues would go away. I also increased the hard drive count to three. All the bugs and performance issues are still present.

Missing Features

  • No ability to backup camera footage to a remote endpoint
    • Videos stored in a proprietary format on disk
  • Only the G4 cameras support People and Car Detection
  • G4 Dome (UVC-G4-DOME) cameras do not support standalone mode, there is no mention of this in Ubiquiti documentation or specifications of the camera

Bugs

  • The service running on the UDM will crash randomly when retrieving videos
  • On Android, the video from an Event will not always load or take 3-5 seconds to load
  • Your authenticated session will be randomly logged out, and you will sometimes still receive notifications of movement, but not always. When you open the app, you will have to log in again.
  • Motion is detected for car headlights, even at very low motion sensitivity
  • Software updates despite automatic upgrades being disabled
  • Unifi Protect restore does not restore user accounts, on the UNVR
  • UI Account allows you to invite a user via the web interface, but there is no way to deactivate the user via the interface. This needs to be done from https://unifi.ui.com
  • In Protect 2.7.18 update, if you use Firefox to access the web view, the site returns "UniFi is having trouble with this direction"
  • The auto-generated Default live view is empty on certain installations (reported September 2022, not fixed)

UFP Viewport

Missing Features

  • HDMI-CEC to turn TV on and off at certain times

Bugs

  • In Protect 2.7.18 update, if >5 devices are enabled in a view: the device will restart every 30 minutes, and for the feeds to be delayed up to 3 minutes (reported to Ubiquiti Support in February 2023, not fixed)
  • Enabling more than 5 cameras (specifications say it supports 16 cameras) causes the FPS of the cameras to drop to 1-2 FPS, and 50+ second delay on the camera feeds (reported to Ubiquiti Support in March 2022, not fixed)
  • Views that change between cameras do not work on UFP Viewport despite being able to set them on the web interface
  • Takes many tries to adopt to Unifi Protect, as well as failed firmware update attempts
  • Sometimes the feed will freeze and you will have to restart the UFW Viewport for it to work again.

Other issues

Conclusion

Ubiquiti does not fixed reported, critical bugs in their products, and in some cases omit the capabilities of their products. Ubiquiti prodcuts are buggy, slow, unstable and do not follow industry standards. I would avoid installation Ubiquiti products in a new environment.

I have replaced the UDM with an OpenWrt install, but I don't know what to do with my existing cameras as it was not a low-cost or low-effort investment.

Ubiquiti may be "okay" if you're not doing anything complicated, but at that point you might as well buy a cheaper solution that "just works" and doesn't arbitrarily have features removed or just plain not working.

deCONZ VNC access without full blown Xorg install

Peter

deCONZ / Phoscon-GW recommends a full blown Xorg install on a Raspberry Pi to access their GUI. deCONZ is a utility to work with Zigbee controller provided by Raspbee / Conbee. deCONZ GUI provides more advanced control of Zigbee devices, and has functionality that is not available via web interface/REST API.

No need to install a full blown UI if you want to run headless. These instructions are for Raspbian.

Make sure to disable and stop existing deconz services if enabled:

systemctl disable deconz
systemctl stop deconz

You may also have deconz-gui service if you have followed their install guide, so make sure to disable and stop this as well if applicable.

Install TigerVNC which we will use to provide headless VNC access to the GUI:

apt-get install tigervnc-standalone-server

You will create a new systemd service definition, which is outlined below .

/etc/systemd/system/deconz-vnc.service

[Unit]
Description=deCONZ: ZigBee gateway -- GUI/REST API VNC
Wants=deconz-init.service deconz-update.service

[Service]
User=1000
Environment="DISPLAY=:0"
ExecStartPre=/usr/bin/tigervncserver -geometry 1024x768 -useold -SecurityTypes None -localhost yes -noxstartup "$DISPLAY"
ExecStart=/usr/bin/deCONZ --http-port=80
Restart=on-failure
StartLimitIntervalSec=0
RestartSec=30
AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_KILL CAP_SYS_BOOT CAP_SYS_TIME

[Install]
WantedBy=multi-user.target

Then reload systemd, enable and start the newly created service:

systemctl daemon-reload
systemctl enable deconz-vnc
systemctl start deconz-vnc

Once this is done you can setup a SSH port forward to localhost:5900 and access deCONZ GUI using your favourite VNC viewer.

Home Assistant sensors and CyberPower UPS via RMCARD205

Peter

RMCARD205 supplies metrics via SNMP for all supported CyberPower UPS. Who doesn't want to display power usage on their UPS via Home Assistant?

Lovelace card for CyberPower UPS

I created a "package" for CyberPower UPS RMCARD that exports data via SNMP. This assumes your default read-only community is named public.

packages/ups.yaml:

sensor:
  - platform: snmp
    scan_interval: 60
    name: ups_nompower
    host: x.x.x.x
    baseoid: 1.3.6.1.4.1.3808.1.1.1.4.2.5.0
    accept_errors: true
    unit_of_measurement: Watts
  - platform: snmp
    scan_interval: 60
    name: ups_nominv
    host: x.x.x.x
    baseoid: 1.3.6.1.4.1.3808.1.1.1.3.2.1.0
    accept_errors: true
    unit_of_measurement: Volts
    value_template: '{{((value | int) / 10) | int}}'
  - platform: snmp
    scan_interval: 60
    name: ups_itemp
    host: x.x.x.x
    baseoid: 1.3.6.1.4.1.3808.1.1.1.2.2.3.0
    accept_errors: true
    unit_of_measurement: "°C"
  - platform: snmp
    scan_interval: 60
    name: ups_timeleft
    host: x.x.x.x
    baseoid: 1.3.6.1.4.1.3808.1.1.1.2.2.4.0
    accept_errors: true
    unit_of_measurement: 'minutes'
    value_template: '{{((value | int) / 6000) | int}}'
  - platform: snmp
    scan_interval: 30
    name: ups_status
    host: x.x.x.x
    baseoid: 1.3.6.1.4.1.3808.1.1.1.4.1.1.0
    accept_errors: true
    value_template: >-
      {% set status = (value | int) %}
      {%- if status == 2 -%}
      Online
      {%- elif status ==  3 -%}
      On Battery
      {%- elif status ==  4 -%}
      On Boost
      {%- elif status ==  5 -%}
      On Sleep
      {%- elif status ==  6 -%}
      Off
      {%- elif status ==  7 -%}
      Rebooting
      {%- elif status ==  8 -%}
      On ECO
      {%- elif status ==  9 -%}
      On Bypass
      {%- elif status ==  10 -%}
      On Buck
      {%- elif status ==  11 -%}
      On Overload
      {%- else -%}
      Unknown
      {%- endif -%}
      
group:
  ups:
    name: UPS
    entities:
      - sensor.ups_status
      - sensor.ups_nompower
      - sensor.ups_nominv
      - sensor.ups_itemp
      - sensor.ups_timeleft

homeassistant:
  customize:
    sensor.ups_nompower:
      friendly_name: 'UPS Nominal Output Power'
      icon: mdi:flash
    sensor.ups_nominv:
      friendly_name: 'UPS Nominal Input Voltage'
      icon: mdi:flash
    sensor.ups_status:
      friendly_name: 'UPS Status'
      icon: mdi:information-outline
    sensor.ups_itemp:
      friendly_name: 'UPS Internal Temperature'
      icon: mdi:thermometer
    sensor.ups_timeleft:
      friendly_name: 'UPS Time Left'
      icon: mdi:clock-alert

NetGear GS108T, NetGear R7000 Nighthawk AP and Intel igb/e1000e on Linux: adapter reset

Peter

What's the issue?

At home I have a NetGear GS108T with a NetGear R7000 serving as an Access Point. My Ubuntu 16.04 machine which functions as a hypervisor (qemu/kvm and openvswitch) has an "adapter reset"/"transmit queue timed out" error on igb and e1000e Intel NICs and causes a 30-60 second outage when my NetGear R7000 Access Point restarts. Kernel logs show the following:

Jun 29 16:31:26 kvm kernel: [ 145.888552] ------------[ cut here ]------------
Jun 29 16:31:26 kvm kernel: [ 145.888557] NETDEV WATCHDOG: p6p1 (igb): transmit queue 0 timed out
Jun 29 16:31:26 kvm kernel: [ 145.888595] WARNING: CPU: 6 PID: 0 at /build/linux-hwe-0EwvTm/linux-hwe-4.15.0/net/sched/sch_generic.c:323 dev_watchdog+0x222/0x230
Jun 29 16:31:26 kvm kernel: [ 145.888597] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio ebtable_filter ebtables xt_multiport openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 binfmt_misc ip6t_REJECT nf_reject_ipv6 xt_hl ip6t_rt ppdev intel_rapl nf_conntrack_ipv6 nf_defrag_ipv6 x86_pkg_temp_thermal intel_powerclamp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ipt_REJECT nf_reject_ipv4 intel_cstate usblp xt_limit xt_tcpudp ipmi_si ipmi_devintf xt_addrtype intel_rapl_perf ipmi_msghandler lpc_ich parport_pc mei_me mei pcbc shpchp ie31200_edac mac_hid aesni_intel crypto_simd glue_helper cryptd aes_x86_64 dm_crypt vhost_net vhost tap kvm_intel kvm irqbypass nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ip6table_filter ip6_tables nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_ftp nf_nat nf_conntrack_ftp nf_conntrack
Jun 29 16:31:26 kvm kernel: [ 145.888689] libcrc32c iptable_filter ip_tables x_tables sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nct6775 hwmon_vid coretemp lp parport autofs4 btrfs xor zstd_compress raid6_pq raid1 raid10 ast ttm drm_kms_helper syscopyarea sysfillrect igb sysimgblt fb_sys_fops hid_generic dca i2c_algo_bit usbhid ahci ptp libahci drm mpt3sas pps_core hid raid_class scsi_transport_sas video
Jun 29 16:31:26 kvm kernel: [ 145.888761] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 4.15.0-54-generic #58~16.04.1-Ubuntu
Jun 29 16:31:26 kvm kernel: [ 145.888763] Hardware name: ASUSTeK COMPUTER INC. P9D-C Series/P9D-C Series, BIOS 1301 01/26/2015
Jun 29 16:31:26 kvm kernel: [ 145.888770] RIP: 0010:dev_watchdog+0x222/0x230
Jun 29 16:31:26 kvm kernel: [ 145.888773] RSP: 0018:ffff8a72efd83e68 EFLAGS: 00010282
Jun 29 16:31:26 kvm kernel: [ 145.888777] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
Jun 29 16:31:26 kvm kernel: [ 145.888779] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff8a72efd96490
Jun 29 16:31:26 kvm kernel: [ 145.888782] RBP: ffff8a72efd83e98 R08: 0000000000000001 R09: 000000000000049c
Jun 29 16:31:26 kvm kernel: [ 145.888785] R10: 0000000000000000 R11: 000000000000049c R12: 0000000000000008
Jun 29 16:31:26 kvm kernel: [ 145.888787] R13: ffff8a72bda80000 R14: ffff8a72bda80478 R15: ffff8a72bda79940
Jun 29 16:31:26 kvm kernel: [ 145.888791] FS: 0000000000000000(0000) GS:ffff8a72efd80000(0000) knlGS:0000000000000000
Jun 29 16:31:26 kvm kernel: [ 145.888794] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 29 16:31:26 kvm kernel: [ 145.888797] CR2: 00007fa1302dbf48 CR3: 00000001f120a002 CR4: 00000000001626e0
Jun 29 16:31:26 kvm kernel: [ 145.888800] Call Trace:
Jun 29 16:31:26 kvm kernel: [ 145.888803] <IRQ>
Jun 29 16:31:26 kvm kernel: [ 145.888812] ? dev_deactivate_queue.constprop.33+0x60/0x60
Jun 29 16:31:26 kvm kernel: [ 145.888819] call_timer_fn+0x32/0x140
Jun 29 16:31:26 kvm kernel: [ 145.888824] run_timer_softirq+0x1ed/0x440
Jun 29 16:31:26 kvm kernel: [ 145.888830] ? ktime_get+0x3e/0xa0
Jun 29 16:31:26 kvm kernel: [ 145.888838] ? lapic_next_deadline+0x26/0x30
Jun 29 16:31:26 kvm kernel: [ 145.888846] __do_softirq+0xf5/0x28f
Jun 29 16:31:26 kvm kernel: [ 145.888856] irq_exit+0xb8/0xc0
Jun 29 16:31:26 kvm kernel: [ 145.888861] smp_apic_timer_interrupt+0x79/0x140
Jun 29 16:31:26 kvm kernel: [ 145.888866] apic_timer_interrupt+0x84/0x90
Jun 29 16:31:26 kvm kernel: [ 145.888868] </IRQ>
Jun 29 16:31:26 kvm kernel: [ 145.888879] RIP: 0010:cpuidle_enter_state+0xa7/0x300
Jun 29 16:31:26 kvm kernel: [ 145.888881] RSP: 0018:ffffac58031cfe60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Jun 29 16:31:26 kvm kernel: [ 145.888886] RAX: ffff8a72efda2840 RBX: 0000000000000005 RCX: 000000000000001f
Jun 29 16:31:26 kvm kernel: [ 145.888888] RDX: 0000000000000000 RSI: 0000000025bbf79e RDI: 0000000000000000
Jun 29 16:31:26 kvm kernel: [ 145.888890] RBP: ffffac58031cfe98 R08: ffff8a72efda16a4 R09: 0000000000000018
Jun 29 16:31:26 kvm kernel: [ 145.888893] R10: ffffac58031cfe30 R11: 0000000000000391 R12: 0000000000000005
Jun 29 16:31:26 kvm kernel: [ 145.888895] R13: ffff8a72efdacf00 R14: ffffffff9f371e38 R15: 00000021f77f87f8
Jun 29 16:31:26 kvm kernel: [ 145.888905] cpuidle_enter+0x17/0x20
Jun 29 16:31:26 kvm kernel: [ 145.888914] call_cpuidle+0x23/0x40
Jun 29 16:31:26 kvm kernel: [ 145.888920] do_idle+0x197/0x200
Jun 29 16:31:26 kvm kernel: [ 145.888926] cpu_startup_entry+0x73/0x80
Jun 29 16:31:26 kvm kernel: [ 145.888931] start_secondary+0x1ab/0x200
Jun 29 16:31:26 kvm kernel: [ 145.888939] secondary_startup_64+0xa5/0xb0
Jun 29 16:31:26 kvm kernel: [ 145.888942] Code: 37 00 49 63 4e e8 eb 92 4c 89 ef c6 05 a8 63 d8 00 01 e8 52 33 fd ff 89 d9 48 89 c2 4c 89 ee 48 c7 c7 c0 1e fa 9e e8 3e 3c 80 ff <0f> 0b eb c0 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48
Jun 29 16:31:26 kvm kernel: [ 145.889026] ---[ end trace 87c230fe7d27f115 ]---
Jun 29 16:31:26 kvm kernel: [ 145.889079] igb 0000:0a:00.0 p6p1: Reset adapter
Jun 29 16:31:30 kvm kernel: [ 149.681151] igb 0000:0a:00.0 p6p1: igb: p6p1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jun 29 16:31:40 kvm kernel: [ 159.964671] igb 0000:0a:00.0 p6p1: Reset adapter
Jun 29 16:31:44 kvm kernel: [ 163.669265] igb 0000:0a:00.0 p6p1: igb: p6p1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

This only happened on my Linux hypervisor, none of the other devices on my network exhibited this behaviour.

What troubleshooting steps did I take?

  • Different network cables
  • Using a newer kernel (4.15, I was on 4.4 to start)
  • Building igb drivers from Intel's website via DKMS and loading them
  • Using a different network card in a PCI express slot (e1000e) vs. onboard (igb)
  • Trying all slots on the NIC
  • Updating the BIOS on the ASUS server mainboard (P9D-C) to the latest available
  • Turning off TSO/GSO offloading on the NIC via ethtool

None of these issues resolved the error I was encountering. If I restarted the R7000, it caused the adapter to time out every time.

How did I solve it?

Disabling RX/TX Flow control on the switch caused the issue to stop happening.

Since I did not want to disable RX/TX Flow control globally, I implemented the fix by disabling auto negotiate and setting RX/TX flow control off (aka. "PAUSE" Ethernet frames) using interface(8) and ethtool on Ubuntu:

pre-up /sbin/ethtool -s $IFACE autoneg off speed 1000 duplex full
pre-up /sbin/ethtool -A $IFACE autoneg off rx off

Now my R7000 can do whatever it likes, including randomly restarting and my network connectivity continues to function on my hypervisor.

prosody websocket behind nginx reverse proxy

Peter

Useful for kawai and other XMPP services behind the same URL (eg. for serving SSL traffic.)

WebSockets require HTTP/1.1, and prosody assumes traffic on TCP port 5280 is not secure (and trying to force it to starttls) thus requiring the configuration knob highlighted below.

prosody.cfg.lua

consider_websocket_secure = true

nginx.conf

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

server {
# ...
  location /xmpp-websocket {
    proxy_pass http://127.0.0.1:5280;
    proxy_buffering off;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
  }
}