ConcurrentModificationException in BaseSitePage with memory.use.legacy=false
Description
Attachments
relates to
Activity
Matthew Buckett July 31, 2014 at 9:16 AM
Attached is a patch that stops the BaseSitePage from being modified when a copy is created.
This should fix the original issue, my only concert is that if the page getting cached doesn't have it's tools loaded all the copies will never have their tools loaded.
Matthew Buckett July 31, 2014 at 8:59 AM
Introduced getters that don't go back to the database so that when copying a item from the cache you never modify the item you are copying.
Matthew Buckett July 31, 2014 at 8:27 AM
It looks like this patch works because the tool list is a synchronised collection (and so toArray() is safe to call in a multithreaded setup) and that new ArrayList(Collection) uses toArray() on the collection it's taking a copy of and doesn't internally create an iterator.
If this is the solution we're going for it would be good to make it explicit that this is required so any future refactoring doesn't re-introduce this bug.
Indentation on the patch looks wrong.
Aaron Zeckoski July 31, 2014 at 7:50 AM
Patch was applied in trunk that might fix this, waiting for testing to confirm
Aaron Zeckoski July 15, 2014 at 2:03 PM
"Why is it that the ehcache caching surfaces this problem whereas the legacy caching doesn't?"
I don't know. Both use Ehcache but in the new case there are no copies of the tool lists made (there are simply caches which hold an id->id mapping) and in the legacy case each set of tools, pages, and groups in each Site is stored in a separate ConcurrentHashMap. My guess is that it is somehow related to the direct use of the data in the Site in the new one and the use of the map copy in the legacy one but that's just a guess.
Legacy: kernel/kernel-impl/src/main/java/org/sakaiproject/site/impl/SiteCacheImpl.java
New: kernel/kernel-impl/src/main/java/org/sakaiproject/site/impl/SiteCacheSafe.java
If you have a way to test and verify the patch then that's great. Please let us know what happens.
Details
Details
Priority
CriticalAffects versions
Fix versions
Components
Assignee
Aaron ZeckoskiReporter
Stephen Marquard
Sample bug report. Several of these since upgrading to 10.x, all for /portal/tool/... URLs, identical stack trace.
bug-id: 0099fd21-de47-4d43-90c8-03ce2dd3567b
usage-session: c394373e-090a-480a-91ea-c2a11448f81d
stack-trace-digest: 24B1BFA125B4705F6845BDD3967F1623AB8091CC
sakai-version: 10
service-version: [r310714-r11772]
app-server: vula2a
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
browser-id: Mac-Mozilla
IP: 105.224.103.146
request-path: /portal/tool/eef91ed0-7cb7-4eda-9756-f27582c142bc/css/TableSorter.css
time: 12-Jul-2014 13:41:02
stack trace:
org.sakaiproject.portal.api.PortalHandlerException: java.util.ConcurrentModificationException
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:901)
caused by: java.util.ConcurrentModificationException
at java.util.Vector$Itr.checkForComodification(Vector.java:1156)
at java.util.Vector$Itr.next(Vector.java:1133)
at org.sakaiproject.site.impl.BaseSitePage.<init>(BaseSitePage.java:257)
at org.sakaiproject.site.impl.BaseSite.set(BaseSite.java:700)
at org.sakaiproject.site.impl.BaseSite.<init>(BaseSite.java:254)
at org.sakaiproject.site.impl.BaseSiteService.getCachedSite(BaseSiteService.java:609)
at org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:648)
at org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:740)
at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:818)
at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:151)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:169)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:98)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:901)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:455)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Tool Placement:
No Placement
Request:
: AuthType:null
: CharEncoding:UTF-8
: ContentLength:0
: ContentType:null
: ContextPath:/portal
: LocalAddress:vula.uct.ac.za
: LocalName:vula.uct.ac.za
: LocalPort:443
: Method:GET
: PathInfo:/tool/eef91ed0-7cb7-4eda-9756-f27582c142bc/css/TableSorter.css
: Protocol:HTTP/1.1
: QueryString:null
: RemoteAddress:105.224.103.146
: RemoteHost:105.224.103.146
: RemotePort:-1
: Requested URL:https://vula.uct.ac.za/portal/tool/eef91ed0-7cb7-4eda-9756-f27582c142bc/css/TableSorter.css
: Scheme:https
: ServerName:vula.uct.ac.za
: Headers:
: Header:host:vula.uct.ac.za
: Header:connection:keep-alive
: Header:accept:text/css,/;q=0.1
: Header:user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
: Header:referer:https://vula.uct.ac.za/portal/tool/eef91ed0-7cb7-4eda-9756-f27582c142bc/synMainLite?time=1
: Header:Accept-Encoding:gzip,deflate,sdch
: Header:Accept-Language:en-US,en;q=0.8,it;q=0.6
: Header:cookie:---censored---
: Header:content-length:0
: Parameters:
: Attributes:
: Attribute:sakai.character.encoding.done:sakai.character.encoding.done
: Attribute:sakai.filtered:sakai.filtered
: Attribute:sakai.session:MyS_smarquard{b9a0f17f-46a4-42f4-baee-fdc0fc390ba4, userId='14085e39-ce5d-4d36-8699-161ae4103612', at=15, ts=19, cs=2, Sat Jul 12 13:38:33 SAST 2014}
Session:
: Created:1405165113092
: LastAccess:1405165262555
: CreationDateAndTime:Saturday 12 July 2014
: LastAccessDateAndTime:Saturday 12 July 2014
: MaxInactive:3600
: Attributes:
: Attribute:portalskin:neoskin