DBCP not recovering from db being temporarily unavailable

Description

qa4-us.sakaiproject.org had to be restarted because dbcp was constantly throwing errors like:

WARN: Sql.dbRead: sql: select XML from SAKAI_PREFERENCES where (PREFERENCES_ID = ?) (2007-10-07 15:53:19,159 TP-Processor3_org.sakaiproject.db.impl.BasicSqlService)
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:258)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:539)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:468)
at org.sakaiproject.util.BaseDbSingleStorage.getResource(BaseDbSingleStorage.java:266)
at org.sakaiproject.user.impl.DbPreferencesService$DbStorage.get(DbPreferencesService.java:165)
at org.sakaiproject.user.impl.BasePreferencesService.findPreferences(BasePreferencesService.java:456)
at org.sakaiproject.user.impl.BasePreferencesService.getPreferences(BasePreferencesService.java:270)
at org.sakaiproject.user.cover.PreferencesService.getPreferences(PreferencesService.java:79)
at org.sakaiproject.component.app.help.HelpManagerImpl.getSelectedLocale(HelpManagerImpl.java:952)
at org.sakaiproject.component.app.help.HelpManagerImpl.getToc(HelpManagerImpl.java:589)
at org.sakaiproject.component.app.help.HelpManagerImpl.getTableOfContents(HelpManagerImpl.java:360)
at sun.reflect.GeneratedMethodAccessor1825.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy24.getTableOfContents(Unknown Source)
at org.sakaiproject.tool.help.TableOfContentsTool.getTableOfContents(TableOfContentsTool.java:52)
at sun.reflect.GeneratedMethodAccessor1824.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.sun.faces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:79)
at com.sun.faces.el.impl.ArraySuffix.evaluate(ArraySuffix.java:167)
at com.sun.faces.el.impl.ComplexValue.evaluate(ComplexValue.java:151)
at com.sun.faces.el.impl.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:243)
at com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:173)
at com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:154)
at javax.faces.component.UIData.getValue(UIData.java:527)
at org.sakaiproject.jsf.help.ContextSensitiveTreeRender.encodeBegin(ContextSensitiveTreeRender.java:88)
at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:683)
at javax.faces.component.UIData.encodeBegin(UIData.java:681)
at javax.faces.webapp.UIComponentTag.encodeBegin(UIComponentTag.java:591)
at javax.faces.webapp.UIComponentTag.doStartTag(UIComponentTag.java:478)
at org.apache.jsp.TOCDisplay.main_jsp._jspx_meth_help_005fcsTree_005f0(main_jsp.java:128)
at org.apache.jsp.TOCDisplay.main_jsp._jspx_meth_f_005fview_005f0(main_jsp.java:100)
at org.apache.jsp.TOCDisplay.main_jsp._jspService(main_jsp.java:68)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:328)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:315)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:322)
at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:130)
at org.sakaiproject.jsf.app.SakaiViewHandler.renderView(SakaiViewHandler.java:137)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:87)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:117)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:198)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:555)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:403)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at org.sakaiproject.jsf.util.JsfTool.dispatch(JsfTool.java:221)
at org.sakaiproject.tool.help.HelpJsfTool.dispatch(HelpJsfTool.java:84)
at org.sakaiproject.jsf.util.JsfTool.doGet(JsfTool.java:241)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:469)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:459)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1340)
at org.sakaiproject.portal.charon.handlers.HelpHandler.doHelp(HelpHandler.java:105)
at org.sakaiproject.portal.charon.handlers.HelpHandler.doGet(HelpHandler.java:67)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:889)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:592)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:825)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
... 105 more

While it was throwing these errors I was able to connect to the db with another client and the errors didn't stop until Sakai was restarted. While the errors were being thrown no pages would load.

relates to

Activity

Show:

Peter Knoop October 14, 2007 at 10:03 AM

https://sakaiproject.atlassian.net/browse/SAK-11877#icft=SAK-11877 is tracking the possible swtich to c3p0 for Sakai 2.6.

Peter Knoop October 14, 2007 at 10:01 AM

We will not be switching to c3p0 for the 2.5 release. It is also not clear what the core problem is, as some folks, such as U-M have no problem using DBCP. Preliminary indications are that it might be something to do with performance related Course Management stuff.

Lance Speelmon October 9, 2007 at 7:46 AM

This will not be resolved until we upgrade 2.5 to the c3p0 connection pool.

Lance Speelmon October 9, 2007 at 7:42 AM

We really need to switch to c3p0. I no longer trust DBCP. L

Megan May October 9, 2007 at 6:18 AM

Assigning to DB lead (lance)

Won't Fix

Details

Priority

Affects versions

Assignee

Reporter

Created October 8, 2007 at 1:04 PM
Updated October 23, 2008 at 6:30 AM
Resolved October 9, 2007 at 7:46 AM

Flag notifications