Add settings retry attempts and interval retry attempts for recurring snapshots

[KreVet-Ka]

The required feature described as a wish

We are facing a problem with recurring snapshots for volumes. When a recurring snapshot job fails, ACS keeps making endless retry attempts.

The user does not receive any notifications when a recurring snapshot job fails. Only repeated retry attempts to create the snapshot are observed.

In our test case, we reproduced the issue with an error:

Error while creating entity for allocating snapshot

It would be good to have the ability to configure:

• the of retry attempts
• the interval between retries

To avoid an infinite snapshot creation loop that ends with an error. After the maximum number of retries is exhausted, a notification should be sent informing that the job has failed.

Related discussion: Recurring snapshots: How to limit retry attempts or configure interval in global settings? #13439

[DaanHoogland]

@KreVet-Ka , you say you reproduced. Did you force an error?

[KreVet-Ka]

@KreVet-Ka , you say you reproduced. Did you force an error?

I first encountered this issue on the production infrastructure, and then successfully reproduced it on the test environment.
Steps to reproduce:

1. Reduce the available free space on the secondary storage so that it becomes less than the space required to create a snapshot.
2. Create a recurring snapshot task for a disk.

In this case, each snapshot creation attempt fails with the following error:
“Error while creating entity for allocating snapshot”
However, after the failure, the system enters an infinite loop of snapshot creation attempts, with every subsequent attempt also failing with the same error.

[DaanHoogland]

@KreVet-Ka , any chance to review/test #13743?

[KreVet-Ka]

@KreVet-Ka , any chance to review/test #13743?

I’ll try, but I can’t promise it will work, since I’ve never built CloudStack myself.

[KreVet-Ka]

I tried to build and run this PR locally on Ubuntu 24.04 (all-in-one: management + agent on the same host).
Build and packaging succeeded, packages installed fine, but Management Server fails to start.

ERROR:

2026-08-03 08:37:45,678 INFO  [c.c.u.c.ComponentContext] (main:[]) (logid:) Setup Spring Application context
2026-08-03 08:37:45,681 DEBUG [c.c.u.d.DbProperties] (main:[]) (logid:) DB properties were already loaded
2026-08-03 08:37:45,683 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,755 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,795 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,817 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,836 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,848 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,859 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,872 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,879 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,915 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,923 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,949 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,964 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,977 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,985 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:45,997 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:46,471 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from DataCenterIpAddressDaoImpl
2026-08-03 08:37:47,069 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:47,081 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from RoleManagerImpl
2026-08-03 08:37:47,425 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HostTagsDaoImpl
2026-08-03 08:37:47,546 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServiceConfigurationImpl
2026-08-03 08:37:47,570 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from HostDaoImpl
2026-08-03 08:37:49,901 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:49,923 DEBUG [o.a.c.s.l.r.ExtensionRegistry] (main:[]) (logid:) Registering extension [VpcVirtualRouterElement] in [Site2 Site Vpn Service Providers Registry]
2026-08-03 08:37:49,924 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from Site2SiteVpnManagerImpl
2026-08-03 08:37:50,436 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:50,661 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ClvmPoolManager
2026-08-03 08:37:50,945 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:50,962 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:50,971 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:51,015 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VMSnapshotManagerImpl
2026-08-03 08:37:51,036 INFO  [c.c.s.GsonHelper] (main:[]) (logid:) Default Builder inited.
2026-08-03 08:37:51,446 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from IndirectAgentLBServiceImpl
2026-08-03 08:37:51,487 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from UserDataManagerImpl
2026-08-03 08:37:51,507 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from PremiumSecondaryStorageManagerImpl
2026-08-03 08:37:51,666 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:51,688 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ManagementServerImpl
2026-08-03 08:37:51,857 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:51,863 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from KMSManagerImpl
2026-08-03 08:37:51,969 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from VolumeApiServiceImpl
2026-08-03 08:37:52,020 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:52,026 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from BackupManagerImpl
2026-08-03 08:37:52,085 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from ExtensionRegistry
2026-08-03 08:37:52,116 DEBUG [o.a.c.f.c.i.ConfigDepotImpl] (main:[]) (logid:) Retrieving keys from SnapshotManagerImpl
2026-08-03 08:37:52,212 ERROR [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Failed to load module [core] due to: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ipDeployersRegistry' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.23.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-core-registry-core-context.xml]: Cannot resolve reference to bean 'VpcVirtualRouter' while setting bean property 'preRegistered' with key [0]; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VpcVirtualRouter': Unsatisfied dependency expressed through field '_networkMdl'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkModelImpl': Unsatisfied dependency expressed through field '_accountMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'accountManagerImpl': Unsatisfied dependency expressed through field '_templateDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VMTemplateDaoImpl': Unsatisfied dependency expressed through field '_templateDataStoreDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateDataStoreDaoImpl': Unsatisfied dependency expressed through field '_storeMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataStoreProviderManager': Unsatisfied dependency expressed through field 'primaryDataStoreProviderMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'primaryDataStoreProviderMgr': Unsatisfied dependency expressed through field 'storageMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'storageManagerImpl': Unsatisfied dependency expressed through field '_agentMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusteredAgentManagerImpl': Unsatisfied dependency expressed through field 'caService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'caManager': Unsatisfied dependency expressed through field 'networkOrchestrationService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkOrchestrator': Unsatisfied dependency expressed through field '_alertMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'alertManagerImpl': Unsatisfied dependency expressed through field '_capacityMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'capacityManagerImpl': Unsatisfied dependency expressed through field '_templatePoolDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vMTemplatePoolDaoImpl': Unsatisfied dependency expressed through field 'dataStoreManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dataStoreManagerImpl': Unsatisfied dependency expressed through field 'imageDataStoreMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'imageStoreProviderMgr': Unsatisfied dependency expressed through field '_statsCollector'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'statsCollector': Unsatisfied dependency expressed through field '_userVmMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userVmManagerImpl': Unsatisfied dependency expressed through field 'resourceScheduleManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ResourceScheduleManagerImpl': Unsatisfied dependency expressed through field 'workerList'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'VMScheduleWorker': Unsatisfied dependency expressed through field 'asyncJobManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'asyncJobManagerImpl': Unsatisfied dependency expressed through field '_jobDispatchers'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vmWorkJobDispatcher': Unsatisfied dependency expressed through field '_vmMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusteredVirtualMachineManagerImpl': Unsatisfied dependency expressed through field '_haMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'highAvailabilityManagerExtImpl': Unsatisfied dependency expressed through field 'consoleProxyManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'consoleProxyManagerImpl': Unsatisfied dependency expressed through field 'resourceManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceManagerImpl': Unsatisfied dependency expressed through field 'deploymentManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'DeploymentPlanningManager': Unsatisfied dependency expressed through field '_affinityGroupService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'AffinityGroupServiceImpl': Unsatisfied dependency expressed through field '_domainMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'domainManagerImpl': Unsatisfied dependency expressed through field '_projectMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'projectManagerImpl': Unsatisfied dependency expressed through field '_configMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'configurationManagerImpl': Unsatisfied dependency expressed through field '_networkSvc'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'networkServiceImpl': Unsatisfied dependency expressed through field '_rulesMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'rulesManagerImpl': Unsatisfied dependency expressed through field '_ipAddrMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ipAddressManagerImpl': Unsatisfied dependency expressed through field '_lbMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'loadBalancingRulesManagerImpl': Unsatisfied dependency expressed through field '_firewallMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'firewallManagerImpl': Unsatisfied dependency expressed through field '_vpcMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'vpcManagerImpl': Unsatisfied dependency expressed through field '_resourceLimitMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceLimitManagerImpl': Unsatisfied dependency expressed through field '_userVmJoinDao'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'userVmJoinDaoImpl': Unsatisfied dependency expressed through field 'extensionHelper'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ExtensionsManager': Unsatisfied dependency expressed through field 'clusterManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'clusterManagerImpl': Unsatisfied dependency expressed through field 'reconcileCommandService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'reconcileCommandServiceImpl': Unsatisfied dependency expressed through field 'volumeManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'volumeOrchestrator': Unsatisfied dependency expressed through field '_tmpltMgr'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateManagerImpl': Unsatisfied dependency expressed through field '_tmplFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'templateDataFactoryImpl': Unsatisfied dependency expressed through field 'directDownloadManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'directDownloadManager': Unsatisfied dependency expressed through field 'volService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'snapshotManagerImpl' defined in URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-4.23.0.0-SNAPSHOT.jar!/META-INF/cloudstack/core/spring-server-core-managers-context.xml]: Initialization of bean failed; nested exception is java.lang.NullPointerException: Cannot invoke "java.util.Set.add(Object)" because "currentConfigs" is null

Maybe I built it incorrectly?

[DaanHoogland]

@KreVet-Ka , usually these exceptions are DB schema related. check your logs to see if there were any upgrade errors. It looks like it is related to the configuration table.

If not, restore you DB and try again.

as for build related, It might be, for instance a spring configuration missing but unlikely.