Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000

pchilumula
Hi,

We have recently upgraded to 4.3.0 & we have started seeing issues.The server responds good for 2 hrs & then suddenly it timseout & also with indexing.we see the below exceptions. Anyone facing same issue & have a resolution will be greatly appreciated.

c-1] [ERROR] (o.a.s.d.s.s.ArtifactoryHeartbeatServiceImpl:57) - Failed to update heartbeat
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.tomcat.jdbc.pool.PoolExhaustedException: [art-exec-1] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000].
          at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:245) ~[spring-jdbc-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:463) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) [spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at com.sun.proxy.$Proxy23.updateArtifactoryServerHeartbeat(Unknown Source) ~[na:na]
          at org.artifactory.storage.db.servers.service.ArtifactoryServersCommonServiceImpl.updateArtifactoryServerHeartbeat(ArtifactoryServersCommonServiceImpl.java:87) ~[artifactory-core-4.3.0.jar:na]
          at org.artifactory.storage.db.servers.service.ArtifactoryHeartbeatServiceImpl.updateHeartbeat(ArtifactoryHeartbeatServiceImpl.java:54) ~[artifactory-core-4.3.0.jar:na]
          at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source) ~[na:na]
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_66]
          at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_66]
          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) [spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) [spring-aop-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          at com.sun.proxy.$Proxy77.updateHeartbeat(Unknown Source) [na:na]
          at org.artifactory.storage.db.servers.service.ArtifactoryHeartbeatServiceImpl$HeartbeatJob.onExecute(ArtifactoryHeartbeatServiceImpl.java:66) [artifactory-core-4.3.0.jar:na]
          at org.artifactory.schedule.quartz.QuartzCommand.execute(QuartzCommand.java:50) [artifactory-storage-common-4.3.0.jar:na]
          at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:na]
          at org.artifactory.schedule.ArtifactoryConcurrentExecutor$RunnableWrapper.run(ArtifactoryConcurrentExecutor.java:102) [artifactory-storage-common-4.3.0.jar:na]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
          at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [art-exec-1] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000].
          at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:672) ~[tomcat-jdbc-8.0.22.jar:na]
          at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:186) ~[tomcat-jdbc-8.0.22.jar:na]
          at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127) ~[tomcat-jdbc-8.0.22.jar:na]
          at org.artifactory.storage.db.spring.ArtifactoryTomcatDataSource.getConnection(ArtifactoryTomcatDataSource.java:187) ~[artifactory-storage-db-4.3.0.jar:na]
          at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204) ~[spring-jdbc-4.1.5.RELEASE.jar:4.1.5.RELEASE]
          ... 22 common frames omitted
2015-12-02 00:16:13,942 [art-exec-9] [ERROR] (o.q.c.JobRunShell   :211) - Job artifactory.org.artifactory.search.archive.ArchiveIndexerImpl$ArchiveIndexJob#2321ef49-56f7-488c-8afe-8c8af626f2d3 threw an unhandled Exception:
org.artifactory.storage.StorageException: Failed to load tasks of type 'INDEX' : [art-exec-9] Timeout: Pool empty. Unable to fetch a connection in 120 seconds, none available[size:100; busy:100; idle:0; lastwait:120000].
          at org.artifactory.storage.db.fs.service.TasksServiceImpl.getRepoPathTasks(TasksServiceImpl.java:62) ~[artifactory-storage-db-4.3.0.jar:na]
          at org.artifactory.storage.db.fs.service.TasksServiceImpl.getIndexTasks(TasksServiceImpl.java:49) ~[artifactory-storage-db-4.3.0.jar:na]
          at org.artifactory.search.archive.ArchiveIndexerImpl.triggerQueueIndexing(ArchiveIndexerImpl.java:184) ~[artifactory-core-4.3.0.jar:na]
          at sun.reflect.GeneratedMethodAccessor149

Thanks