I've been running in to an intermittent deadlock on shutdown of a Spring-DM powered integration test. About once every five runs, when the integration test finishes, I get the following output from Spring-DM:

ERROR [er.internal.util.concurrent.RunnableTimedExecution] - Closing runnable for context OsgiBundleXmlApplicationContext(bundle=****, config=osgibundle:/META-INF/spring/*.xml) did not finish in 10000ms; consider taking a snapshot and then shutdown the VM in case the thread still hangs

After getting this, the VM hangs until I force quit it. This is problematic when the tests are run as part of continuous integration because the build is always hung.

Below is a capture of the stack trace when the VM is hung. You can see there's a deadlock between Timer-1 and Thread-8 threads. Timer-1 thread first acquired the lock on OsgiServiceFactoryBean (id=3211) and is then trying to acquire the lock on the list of stateListeners (id=3266). Thread-8 acquires the lock on the stateListeners list first. This could be fixed by always acquiring the OsgiServiceFactoryBean lock before the stateListeners lock.


Java HotSpot(TM) 64-Bit Server VM[localhost:10000] (Suspended)
Thread Group [system]
Daemon System Thread [Reference Handler] (Suspended)
Daemon System Thread [Finalizer] (Suspended)
Daemon System Thread [Signal Dispatcher] (Suspended)
Thread Group [main]
Thread [main] (Suspended)
Daemon Thread [Framework Event Dispatcher] (Suspended)
Daemon Thread [Start Level Event Dispatcher] (Suspended)
Thread [Configuration Updater] (Suspended)
Daemon Thread [Timer-1] (Suspended)
owns: OsgiServiceFactoryBean (id=3211)
owns: LinkedHashMap<K,V> (id=3453)
owns: Object (id=3454)
owns: Object (id=3455)
waiting for: Collections$SynchronizedRandomAccessList<E> (id=3266)
Collections$SynchronizedRandomAccessList<E>(Collections$SynchronizedCollection<E>).remove(Object) line: 1580
OsgiServiceProxyFactoryBean$Executor.removeStateListener(ImporterStateListener) line: 78
ImporterController.removeStateListener(ImporterStateListener) line: 41
DefaultMandatoryDependencyManager.removeListener(Object, ImporterStateListener) line: 301
DefaultMandatoryDependencyManager.removeServiceExporter(Object, String) line: 268
MandatoryDependencyBeanPostProcessor.postProcessBeforeDestruction(Object, String) line: 77
DisposableBeanAdapter.destroy() line: 140
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroyBean(String, DisposableBean) line: 456
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroySingleton(String) line: 434
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroySingletons() line: 407
OsgiBundleXmlApplicationContext(AbstractApplicationContext).destroyBeans() line: 854
OsgiBundleXmlApplicationContext(AbstractOsgiBundleApplicationContext).destroyBeans() line: 211
OsgiBundleXmlApplicationContext(AbstractApplicationContext).doClose() line: 832
OsgiBundleXmlApplicationContext(AbstractOsgiBundleApplicationContext).doClose() line: 204
OsgiBundleXmlApplicationContext(AbstractDelegatedExecutionApplicationContext).normalClose() line: 169
DependencyWaiterApplicationContextExecutor.close() line: 354
OsgiBundleXmlApplicationContext(AbstractDelegatedExecutionApplicationContext).doClose() line: 181
OsgiBundleXmlApplicationContext(AbstractApplicationContext).close() line: 795
ContextLoaderListener$1.run() line: 539
RunnableTimedExecution$MonitoredRunnable.run() line: 60
DelegatingTimerTask.run() line: 66
TimerThread.mainLoop() line: 512 [local variables unavailable]
TimerThread.run() line: 462
Thread [Thread-8] (Suspended)
owns: Collections$SynchronizedRandomAccessList<E> (id=3266)
owns: LinkedHashMap<K,V> (id=3456)
waiting for: OsgiServiceFactoryBean (id=3211)
DefaultMandatoryDependencyManager$ImporterDependencyListener.importerUnsatisfied(Object, OsgiServiceDependency) line: 102
ServiceDynamicInterceptor$Listener.notifyUnsatisfiedStateListeners() line: 268
ServiceDynamicInterceptor$Listener.serviceChanged(ServiceEvent) line: 236
FilteredServiceListener.serviceChanged(ServiceEvent) line: 94
BundleContextImpl.dispatchEvent(Object, Object, int, Object) line: 1224
EventManager.dispatchEvent(EventListeners$ListElement[], EventDispatcher, int, Object) line: 195
ListenerQueue.dispatchEventSynchronous(int, Object) line: 141
Framework.publishServiceEventPrivileged(ServiceEvent) line: 1603
Framework.publishServiceEvent(int, ServiceReference) line: 1578
ServiceRegistrationImpl.unregister() line: 156
ServiceRegistrationDecorator.unregister() line: 63
OsgiServiceUtils.unregisterService(ServiceRegistration) line: 41
OsgiServiceFactoryBean.unregisterService(ServiceRegistration) line: 464
OsgiServiceFactoryBean.unregisterService() line: 454
OsgiServiceFactoryBean(AbstractOsgiServiceExporter).destroy() line: 83
DisposableBeanAdapter.destroy() line: 151
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroyBean(String, DisposableBean) line: 456
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroySingleton(String) line: 434
DefaultListableBeanFactory(DefaultSingletonBeanRegistry).destroySingletons() line: 407
OsgiBundleXmlApplicationContext(AbstractApplicationContext).destroyBeans() line: 854
OsgiBundleXmlApplicationContext(AbstractOsgiBundleApplicationContext).destroyBeans() line: 211
OsgiBundleXmlApplicationContext(AbstractApplicationContext).doClose() line: 832
OsgiBundleXmlApplicationContext(AbstractOsgiBundleApplicationContext).doClose() line: 204
OsgiBundleXmlApplicationContext(AbstractDelegatedExecutionApplicationContext).normalClose() line: 169
AbstractDelegatedExecutionApplicationContext$NoDependenciesWaitRefreshExecutor.close() line: 77
OsgiBundleXmlApplicationContext(AbstractDelegatedExecutionApplicationContext).doClose() line: 181
AbstractApplicationContext$1.run() line: 765
Thread [Thread-0] (Suspended)
Thread Group [spring-osgi-extender[2b80e8d1]-threads]
Daemon Thread [Timer-2] (Suspended)
Thread [pool-1-thread-1] (Suspended)