Fix: UniFi Backup Fails In Docker - Restoration Bug Guide

by Pedro Alvarez 58 views

Introduction

This article addresses a bug encountered while trying to restore UniFi Network Application backups within a Docker environment. Specifically, the issue prevents successful restoration of backups, leading to a fresh installation instead of the expected restored state. This problem has been observed using the linuxserver.io Docker image for UniFi Network Application. This comprehensive guide dives deep into the problem, offering a detailed explanation, reproduction steps, and environment setup, ensuring that you, the reader, gain a thorough understanding of the issue and potential solutions. We aim to provide value by not just highlighting the bug, but also by offering insights into the debugging process and potential workarounds. This article is designed for network administrators, IT professionals, and home users who rely on Docker for their UniFi Network Application deployments. We'll walk you through the current behavior, expected behavior, and the exact steps to reproduce this frustrating issue. Understanding this bug is crucial for maintaining the integrity of your network configurations and ensuring a smooth recovery process in case of system failures or migrations. So, let's jump right in and explore this issue together!

Is there an existing issue for this?

  • [x] I have searched the existing issues

Current Behavior

The current behavior involves a frustrating scenario where restoring a UniFi Network Application backup in a Docker container fails. When you attempt to restore a backup, the UniFi interface indicates that the upload and restoration process are underway. The controller's HTTP site goes offline, and the Mongo logs show new collections and indexes being created. However, upon the container's restart, instead of the backed-up configuration, a new UniFi install is present. This means all your settings, devices, and configurations are lost, and you have to start from scratch. This issue is particularly concerning for users who rely on backups for disaster recovery or for migrating their UniFi controller to a new server. Imagine spending hours configuring your network, setting up guest networks, and fine-tuning your wireless settings, only to find that your backup is useless when you need it most. This inconsistency between the expected behavior (a successful restoration) and the actual outcome (a fresh install) can lead to significant downtime and data loss. To further validate this, the same backup files were successfully restored on a new UniFi controller installation on macOS, highlighting that the issue is specific to the Docker environment. This observation points towards potential environmental factors within the Docker setup that might be interfering with the restoration process. Possible culprits include file permissions, network configurations, or even the way the Docker image interacts with the underlying operating system. Understanding these nuances is key to finding a resolution and preventing future occurrences of this issue. So, let's delve deeper into the expected behavior and the steps to reproduce this bug to gain a clearer picture of what's going wrong.

Expected Behavior

The expected behavior when restoring a backup of the UniFi Network Application is straightforward: the application should revert to the state it was in when the backup was created. This includes all settings, configurations, network devices, user accounts, and historical data. In essence, restoring a backup should be a seamless process that allows users to quickly recover from system failures, migrate to new hardware, or revert unwanted changes. When a backup is restored successfully, users should expect to see their familiar dashboard, their devices connected and managed, and their network configurations intact. This ensures minimal disruption and a smooth transition back to normal operations. The UniFi controller should essentially pick up where it left off, without any loss of data or configuration. The restoration process should handle all the necessary database updates, file replacements, and service restarts automatically, without requiring manual intervention. This reliability is crucial for maintaining network uptime and ensuring that administrators can confidently rely on backups as a safety net. Furthermore, a successful restoration should not result in any errors or warnings in the logs, and the application should function exactly as it did before the backup was created. Any deviation from this expected behavior indicates a problem that needs to be addressed to maintain the integrity of the network management system. So, given this clear expectation, let's examine the specific steps that demonstrate the failure of the restoration process in the Docker environment.

Steps To Reproduce

To reproduce the issue, follow these steps meticulously. This process involves setting up a Docker Compose stack with UniFi and Mongo, making changes to the UniFi configuration, creating a backup, tearing down the environment, and then attempting to restore the backup in a new environment. This sequence accurately simulates a common scenario where backups are used for migration or disaster recovery. By following these steps, you can independently verify the bug and gain a deeper understanding of the problem. Let's break down each step to ensure clarity and accuracy:

  1. Bring up containers using docker-compose file: This step involves using a docker-compose.yml file (provided in the Environment section) to spin up the UniFi Network Application and MongoDB containers. Docker Compose simplifies the process of managing multi-container applications, making it easier to deploy and orchestrate the necessary services. This ensures that both the UniFi controller and its database are running in a consistent and isolated environment.
  2. Once the UniFi controller container is up, navigate to the controller interface, and proceed through the steps to create a new site: After the containers are running, access the UniFi controller's web interface. Typically, this is done by navigating to the specified port (usually 8443) on the host machine's IP address. Follow the initial setup wizard to create a new site. This step is crucial because it establishes a baseline configuration that we will later attempt to restore.
  3. Modify a couple of settings (such as Site Name, etc.), then take a backup of this site: To ensure the backup contains meaningful data, modify some settings within the UniFi controller. Changing the Site Name is a simple but effective way to verify that the restoration process is working correctly. After making these changes, create a backup of the site using the UniFi controller's built-in backup functionality. This backup file will be used in the subsequent restoration attempt.
  4. Bring down the compose stack, and ensure any volumes that were created are removed: To simulate a clean environment, bring down the Docker Compose stack using the docker-compose down command. It's essential to remove any volumes created by the containers to ensure that no residual data interferes with the restoration process. This step effectively resets the environment to a pre-existing state, mimicking a scenario where you might be restoring a backup to a new server or a clean installation.
  5. Re-create the compose stack: With the environment cleared, re-create the Docker Compose stack using the same docker-compose.yml file. This step sets up a fresh instance of the UniFi controller and MongoDB, ready for the backup restoration.
  6. Once the UniFi controller container is up, this time, choose Restore backup and select the backup file that was just created: After the containers are running again, access the UniFi controller's web interface. This time, instead of going through the setup wizard, choose the "Restore backup" option. Select the backup file that was created in step 3. This action initiates the restoration process, which should, according to the expected behavior, restore the UniFi controller to the state it was in when the backup was taken.
  7. Proceed through the pop-up confirmation dialog: The UniFi controller will typically display a confirmation dialog before starting the restoration process. Confirm the restoration to proceed.
  8. Wait for the container to restart: The restoration process usually involves a restart of the UniFi controller container. This allows the changes from the backup to be applied and the services to be initialized with the restored configuration.
  9. Observe that the backup has not been restored, and that a new install is present: After the container restarts, access the UniFi controller's web interface again. If the bug is present, you will observe that the UniFi controller has not been restored to the state it was in when the backup was created. Instead, you will be greeted with the initial setup wizard, indicating a fresh installation. This outcome confirms the failure of the backup restoration process.

By meticulously following these steps, you can reliably reproduce the bug and confirm that the UniFi Network Application backup restoration is failing in the Docker environment. This understanding is crucial for troubleshooting and finding a solution to the problem. Now, let's move on to the environment in which this issue was observed to understand the specific context and potential contributing factors.

Environment

Understanding the environment in which this bug occurs is crucial for identifying potential causes and finding effective solutions. This section details the operating system, Docker version, and the Docker Compose configuration used to reproduce the issue. By providing this information, we can help others replicate the bug and contribute to the troubleshooting process. The specific details of the environment, such as the OS version and Docker setup, can influence how applications behave within containers. For example, file permissions, network configurations, and even the underlying kernel version can play a role in the success or failure of a backup restoration process. Therefore, a thorough understanding of the environment is essential for diagnosing the root cause of the problem.

- OS: Docker 24.0.2 // Synology DSM 7.2.2-72806 Update 4 (latest)
- How docker service was installed: Synology package (Container Manager)

This indicates that the issue was observed on a Synology NAS device running DSM (DiskStation Manager) version 7.2.2-72806 Update 4. Docker was installed using the Synology package manager, which provides a convenient way to run containers on Synology devices. The Docker version in use is 24.0.2. This information is important because Synology's DSM environment has its own peculiarities, such as specific file permission handling and network configurations, which might interact with Docker containers in unexpected ways. The use of the Synology package for Docker installation also means that the Docker runtime might be configured differently compared to a standard Docker installation on a Linux server. These differences can potentially affect the behavior of the UniFi Network Application within the container. Therefore, when troubleshooting this bug, it's important to consider the Synology-specific aspects of the environment. Now, let's delve into the CPU architecture and the Docker Compose configuration to gain a more complete picture of the setup.

CPU architecture

The CPU architecture is x86-64, which is a common architecture for NAS devices and servers. This information is relevant because Docker images are often built for specific architectures. Using the correct image for the CPU architecture is essential for ensuring optimal performance and compatibility. In this case, the linuxserver.io image for UniFi Network Application is designed to run on x86-64 architecture, so this should not be a contributing factor to the bug. However, it's still important to verify the architecture to rule out any potential issues related to image compatibility. Now that we've established the CPU architecture, let's examine the Docker Compose configuration used to deploy the UniFi Network Application and MongoDB containers.

Docker creation

The Docker Compose configuration is a critical piece of information for understanding how the UniFi Network Application and MongoDB containers are deployed and configured. This docker-compose.yml file defines the services, networks, volumes, and environment variables used by the application. By examining this configuration, we can identify potential issues related to networking, storage, or resource allocation that might be contributing to the backup restoration failure. The configuration also provides insights into how the UniFi controller is connected to the MongoDB database, which is a crucial aspect of the restoration process. Any misconfiguration in this area could prevent the UniFi controller from correctly accessing and restoring the database backup.

services:
  unifi-network-application:
    image: lscr.io/linuxserver/unifi-network-application:9.3.45
    container_name: unifi-network-application
    restart: unless-stopped
    depends_on:
      mongo:
        condition: service_healthy
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=${TZ}
      - MONGO_USER=${MONGO_USER}
      - MONGO_PASS=${MONGO_PASS}
      - MONGO_HOST=${MONGO_HOST}
      - MONGO_PORT=${MONGO_PORT}
      - MONGO_DBNAME=${MONGO_DBNAME}
      - MONGO_AUTHSOURCE=${MONGO_AUTHSOURCE}
      - MEM_LIMIT=1024
      - MEM_STARTUP=1024
    volumes:
      - unifi_data:/config
    networks:
      externalnet:
        ipv4_address: ${IPV4_ADDRESS}
      caddy-unifi:
      unifi-internal:
    healthcheck:
      test: curl --fail -k https://localhost:8443 || exit 1
      interval: 60s
      retries: 5
      start_period: 20s
      timeout: 10s
  mongo:
    image: mongo:4.4
    container_name: unifi-db
    restart: unless-stopped
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD}
      - MONGO_USER=${MONGO_USER}
      - MONGO_PASS=${MONGO_PASS}
      - MONGO_DBNAME=${MONGO_DBNAME}
      - MONGO_AUTHSOURCE=${MONGO_AUTHSOURCE}
    volumes:
      - /volume1/docker/unifi/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh:ro
      - unifi_mongo_db:/data/db
    networks:
      unifi-internal:
    healthcheck:
      test: echo 'db.runCommand("ping").ok' | mongo localhost:27017/test --quiet
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 40s
volumes:
  unifi_data:
  unifi_mongo_db:
networks:
  externalnet:
    external: true
  caddy-unifi:
    external: true
  unifi-internal:

Let's break down this configuration:

  • Services:
    • unifi-network-application: This service uses the lscr.io/linuxserver/unifi-network-application:9.3.45 image. It's configured to restart unless explicitly stopped and depends on the mongo service, ensuring that the database is running before the UniFi controller starts. Environment variables are used to configure user and group IDs (PUID and PGID), timezone (TZ), and MongoDB connection details. Memory limits are set to 1024MB for both startup and runtime. A volume (unifi_data) is mounted to persist the UniFi configuration. The service is connected to three networks: externalnet, caddy-unifi, and unifi-internal. A health check is defined to ensure the UniFi controller is responsive.
    • mongo: This service uses the mongo:4.4 image. It's also configured to restart unless stopped. Environment variables are used to set the MongoDB root username and password, as well as the database user credentials. A volume (unifi_mongo_db) is mounted to persist the MongoDB data. An initialization script (/volume1/docker/unifi/init-mongo.sh) is mounted to initialize the database. The service is connected to the unifi-internal network. A health check is defined to ensure the MongoDB instance is running.
  • Volumes:
    • unifi_data: This volume is used to persist the UniFi Network Application configuration.
    • unifi_mongo_db: This volume is used to persist the MongoDB data.
  • Networks:
    • externalnet: This is an external network, likely used for external access to the UniFi controller.
    • caddy-unifi: This is another external network, possibly used for integration with a Caddy reverse proxy.
    • unifi-internal: This is an internal network used for communication between the UniFi controller and the MongoDB database.

From this configuration, we can see that the UniFi controller and MongoDB are set up to communicate over an internal network, and persistent volumes are used to store the configuration and database data. The environment variables ensure that the UniFi controller can connect to the MongoDB instance. The health checks provide a way to monitor the status of the services. However, there are a few potential areas of concern. The use of external networks and a reverse proxy (Caddy) might introduce complexity and potential points of failure. The file permissions on the mounted volumes could also be a factor. Additionally, the initialization script for MongoDB (/volume1/docker/unifi/init-mongo.sh) might be interfering with the restoration process. Now that we have a detailed understanding of the environment, let's examine the container logs to see if they provide any clues about the cause of the bug.

Container logs

Examining container logs is a critical step in troubleshooting any Docker-related issue. Logs can provide valuable insights into the application's behavior, error messages, and potential problems during the backup restoration process. By analyzing the logs, we can identify specific errors, warnings, or exceptions that might be causing the restoration to fail. The logs can also reveal whether the UniFi controller is successfully connecting to the MongoDB database, whether the backup file is being accessed correctly, and whether any file permission issues are occurring. In essence, the logs are a rich source of information that can help us pinpoint the root cause of the bug and develop effective solutions. So, let's dive into the provided logs and see what they reveal.

[migrations] started
[migrations] no migrations found
───────────────────────────────────────

      ██╗     ███████╗██╗ ██████╗
      ██║     ██╔════╝██║██╔═══██╗
      ██║     ███████╗██║██║   ██║
      ██║     ╚════██║██║██║   ██║
      ███████╗███████║██║╚██████╔╝
      ╚══════╝╚══════╝╚═╝ ╚═════╝

   Brought to you by linuxserver.io
───────────────────────────────────────

To support LSIO projects visit:
https://www.linuxserver.io/donate/

───────────────────────────────────────
GID/UID
───────────────────────────────────────

User UID:    1000
User GID:    1000
───────────────────────────────────────
Linuxserver.io version: 9.3.45-ls99
Build-date: 2025-07-31T15:03:35+00:00
───────────────────────────────────────
    
*** Waiting for MONGO_HOST unifi-db to be reachable. ***
Generating 4,096 bit RSA key pair and self-signed certificate (SHA384withRSA) with a validity of 3,650 days
	for: CN=unifi
[custom-init] No custom files found, skipping...
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called
Exception in thread "Thread-11" java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebApplicationContext@613c829c has been closed already
	at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1227)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1260)
	at com.ubnt.service.ooOO.Ö00000(Unknown Source)
	at com.ubnt.ace.Launcher.Ò00000(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called
Exception in thread "Thread-11" java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebApplicationContext@2f6f7113 has been closed already
	at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1227)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1260)
	at com.ubnt.service.ooOO.Ö00000(Unknown Source)
	at com.ubnt.ace.Launcher.Ò00000(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:840)
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called
Exception in thread "Thread-12" java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebApplicationContext@5ca01e01 has been closed already
	at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1227)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1260)
	at com.ubnt.service.ooOO.Ö00000(Unknown Source)
	at com.ubnt.ace.Launcher.Ò00000(Unknown Source)
	at java.base/java.lang.Thread.run(Thread.java:840)

These logs provide several key pieces of information:

  • Initial Startup: The logs show the standard startup sequence for the linuxserver.io UniFi Network Application Docker image. It includes information about the version, build date, and user/group IDs.
  • MongoDB Connection: The log message *** Waiting for MONGO_HOST unifi-db to be reachable. *** indicates that the UniFi controller is waiting for the MongoDB database to become available. This is a normal part of the startup process, as the UniFi controller depends on MongoDB to store its data.
  • Certificate Generation: The logs show that a new RSA key pair and self-signed certificate are being generated. This is also a standard part of the startup process, especially for a new installation or when the existing certificate is not found.
  • Spring Context Exceptions: The most concerning part of the logs is the java.lang.IllegalStateException: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebApplicationContext has been closed already error. This exception indicates that the Spring application context within the UniFi controller is being closed prematurely, which could disrupt the backup restoration process. The stack trace suggests that this exception is occurring in the context of background threads (Thread-11 and Thread-12).
  • UrlRewriteFilter Destroy: The org.tuckey.web.filters.urlrewrite.UrlRewriteFilter INFO: destroy called messages indicate that the URL rewrite filter is being destroyed, which is likely a part of the shutdown process. However, the fact that these messages appear in conjunction with the Spring context exceptions suggests that there might be an issue with the application's shutdown sequence.

Based on these logs, the java.lang.IllegalStateException related to the Spring application context is the most likely cause of the backup restoration failure. This exception suggests that the application is not shutting down cleanly, which could prevent the backup process from completing successfully. The fact that this exception occurs in background threads indicates that there might be a concurrency issue or a problem with the application's lifecycle management. To further investigate this issue, it would be helpful to examine the logs from the MongoDB container and to capture more detailed logs from the UniFi controller during the restoration process. Additionally, it might be necessary to debug the UniFi application code to understand why the Spring context is being closed prematurely. So, to wrap it up, the logs point towards a Spring context issue as the primary suspect in the backup restoration failure.

Conclusion

In conclusion, the issue of UniFi Network Application backup restoration failure within a Docker environment, specifically on Synology DSM, is a significant problem that can lead to data loss and network downtime. Through detailed steps to reproduce, environmental analysis, and log examination, we've pinpointed a potential cause: a java.lang.IllegalStateException related to the Spring application context. This exception suggests that the application is not shutting down cleanly, which could be disrupting the backup restoration process. While this analysis provides a strong lead, further investigation is needed to fully understand the root cause and develop a reliable solution. This might involve debugging the UniFi application code, examining the interaction between the UniFi controller and the MongoDB database during the restoration process, and considering the specific environment factors introduced by Synology DSM and Docker. Addressing this bug is crucial for ensuring the reliability of UniFi Network Application deployments in Docker environments, and we hope this comprehensive analysis serves as a valuable resource for the community and the developers working on this issue. By sharing this information, we aim to contribute to a more robust and dependable network management experience for all users.