Cloudstack error during template copy zone to zone

[arpanbht]

I am facing this error during copying a template from zone a to zone b

2026-09-30 09:18:31,366 DEBUG [o.a.c.s.i.TemplateServiceImpl] (RemoteHostEndPoint-6:[ctx-3086cf53]) (logid:d62e7587) Performing copy template cross zone callback after completion
2026-09-30 09:18:31,370 DEBUG [c.c.t.TemplateManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371, ctx-05350704]) (logid:1fb812b7) Copy Template failed for image store ImageStore {"id":3,"name":"secondary_dr","uuid":"bee4a48b-1a24-47f3-b4b3-0a126b89bda3"}: Connection refused
2026-09-30 09:18:31,372 WARN  [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-6:[ctx-3086cf53]) (logid:d62e7587) alertType=[28] dataCenterId=[3] podId=[null] clusterId=[null] message=[Failed to register template: 5ee83cac-d5a4-4515-a651-4e2d9d3ef505 with error: Connection refused].
2026-09-30 09:18:31,374 WARN  [c.c.a.AlertManagerImpl] (RemoteHostEndPoint-6:[ctx-3086cf53]) (logid:d62e7587) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Failed to register template: 5ee83cac-d5a4-4515-a651-4e2d9d3ef505 with error: Connection refused] and content [Failed to register template: 5ee83cac-d5a4-4515-a651-4e2d9d3ef505 with error: Connection refused].
2026-09-30 09:18:31,374 ERROR [o.a.c.s.i.BaseImageStoreDriverImpl] (RemoteHostEndPoint-6:[ctx-3086cf53]) (logid:d62e7587) Failed to register template: 5ee83cac-d5a4-4515-a651-4e2d9d3ef505 with error: Connection refused
2026-09-30 09:18:31,378 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Unexpected exception while executing org.apache.cloudstack.api.command.admin.template.CopyTemplateCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to copy Template
        at com.cloud.template.TemplateManagerImpl.copyTemplate(TemplateManagerImpl.java:1066)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy243.copyTemplate(Unknown Source)
        at org.apache.cloudstack.api.command.user.template.CopyTemplateCmd.execute(CopyTemplateCmd.java:176)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:686)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:634)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2026-09-30 09:18:31,379 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Complete async job-63371, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to copy Template"}
2026-09-30 09:18:31,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Publish async job-63371 complete on message bus
2026-09-30 09:18:31,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Wake up jobs related to job-63371
2026-09-30 09:18:31,380 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Update db status for job-63371
2026-09-30 09:18:31,381 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:[ctx-cde44d9d, job-63371]) (logid:1fb812b7) Wake up jobs joined with job-63371 and disjoin all subjobs created from job- 63371

We get this same kind of error during copying the snapshot but previously we have tried to copy the template and one of them worked but now it is failing....but before doing the copy we have synced both the secondary storage from zone a to zone b. So do we need to sync again from zone a to zone?

[Flpvoigt]

The Java stack trace is only wrapping the real failure. The useful line is Copy Template failed ... Connection refused. That is a TCP/network-service failure in the secondary-storage path, not evidence that the template files need to be manually synchronized again.

A cross-zone template copy is handled by the Secondary Storage VMs (SSVMs). I would check the following before touching the storage contents:

1. In Infrastructure → System VMs, verify that the SSVM in both zones is Running and its agent is Connected.
2. Follow the same log IDs (d62e7587 / 1fb812b7) in management-server.log to find the exact host and port that refused the connection.
3. From the affected SSVM, verify DNS/routing/firewall connectivity to the management server and the destination secondary store. SSVM agents must reach the management server (normally TCP 8250), and the destination SSVM must be able to mount/read/write its secondary storage.
4. Check that the destination image store secondary_dr is Up and that its NFS/object-store endpoint is listening. A TCP reject usually means the address is reachable but the service is stopped, bound to a different interface/port, or blocked with REJECT.
5. After fixing connectivity, restart only the affected SSVM if its agent/mount is stale, then retry the copy.

Do not manually rsync the two secondary-storage directory trees as the first fix. CloudStack tracks template state in its database and expects the SSVM to perform/register the copy; out-of-band file copies can leave the database and storage inconsistent.

The fact that one earlier copy succeeded and both template and snapshot operations now fail points especially to an SSVM, route/firewall, or destination-store service that changed after that successful run. The management-server lines immediately before the refusal, plus the affected SSVM log, should identify which endpoint is rejecting the connection.

[arpanbht]

@Flpvoigt

All the agents are in running state and agent state is up, here is the image for your reference.