Re: [controller-dev] Integration distribution failing Single Feature test in master

Home Messages Hashtags Subgroups

This group is locked. No changes can be made to the group while it is locked.

#536

Re: [controller-dev] Integration distribution failing Single Feature test in master

Lori Jakab <lojakab@...> #536 On 10/14/15 3:28 AM, Luis Gomez wrote: Thanks Tom for your analysis, lisp and ofjava people, would you mind taking a look at these comments? Hi Luis, all, We just pushed a patch to disable the timeout on the blueprint container, to allow more time for the services to come up. We've seen this exception in the past for example when a patch slowed down the config subsystem as a side effect. It means that in the 5 minutes default timeout the core mappingservice did not initialize, and didn't register an implementation of IMappingServiceShell with OSGi. I have no idea how the test environment handles the exception, so I hope disabling the timeout will help. From the other analysis on the thread it looks like lisp is only causing issues in terms of more heap needed after the additional features were added, but not in terms of functionality. -Lori BR/Luis On Oct 13, 2015, at 2:31 PM, Tom Pantelis <tompantelis@... <mailto:tompantelis@...>> wrote: Comparing to the last successful run on Oct 5th, there's a couple errors now appearing: 2015-10-13 15:41:51,778 \| ERROR \| bundle-tracker-0 \| ModuleInfoBundleTracker \| 131 - org.opendaylight.controller.config-manager - 0.4.0.SNAPSHOT \| Failed to process bundleentry://188.fwk532513438/META-INF/services/org.opendaylight.yangtools.yang.binding.YangModelBindingProvider for bundle org.opendaylight.openflowjava.openflow-protocol-api_0.7.0.SNAPSHOT [188] java.lang.IllegalStateException: Error while executing getModuleInfo on org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.protocol.rev130731.$YangModelBindingProvider@4a0236de ... Caused by: java.lang.IllegalStateException: Resource '/META-INF/yang/openflow-instruction.yang' is missing at org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.instruction.rev130731.$YangModuleInfoImpl.<init>($YangModuleInfoImpl.java:31)[188:org.opendaylight.openflowjava.openflow-protocol-api:0.7.0.SNAPSHOT] This one has been happening for a while but it seems to be benign. 2015-10-13 15:48:11,064 \| ERROR \| rint Extender: 3 \| BlueprintContainerImpl \| 15 - org.apache.aries.blueprint.core - 1.4.2 \| Unable to start blueprint container for bundle org.opendaylight.lispflowmapping.mappingservice.shell due to unresolved dependencies [(objectClass=org.opendaylight.lispflowmapping.interfaces.mappingservice.IMappingServiceShell)] java.util.concurrent.TimeoutException at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:336)[15:org.apache.aries.blueprint.core:1.4.2] at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)[15:org.apache.aries.blueprint.core:1.4.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_85] at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_85] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)[:1.7.0_85] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)[:1.7.0_85] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_85] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_85] at java.lang.Thread.run(Thread.java:745)[:1.7.0_85] This one appears to emanate from lispflowmapping. It's using blueprint and appears to import an OSGi service, IMappingServiceShell, that wasn't found. The default timeout for blueprint is 5 min but I don't know if the test blocks on this blueprint container and will fail as a result. Either way it seems this should be looked at by a lispflowmapping contributor. There may be other stuff going on. It would be useful to run it by hand and, when/if it appears stuck, use jstack to get a thread dump. Also it's hard to tell why the OOM errors are occurring - one of the tests indicates they started to occur after shutdown was started. For that it would be useful to get a heap dump via jmap or, better yet, run the test with the -XX:+HeapDumpOnOutOfMemoryError option enabled if possible. On Tue, Oct 13, 2015 at 4:43 PM, Luis Gomez <ecelgp@... <mailto:ecelgp@...>> wrote: Look in at more failing distribution jobs, there is always a timeout in the Single Feature, the distribution used to build in 8 mins while now it takes more than 1 hour. BR/Luis On Oct 13, 2015, at 12:13 PM, Luis Gomez <ecelgp@... <mailto:ecelgp@...>> wrote: Hi all, I just observed the distribution in master is failing since Oct 6th [1]. Last errors [2] show memory issues like below but I am not sure this is the root cause for this. Can anyone help identifying the problem here? Thanks/Luis [1] https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/ [2] https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/444/testReport/ Exception in thread "qtp1815616686-79" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.HashMap.newKeyIterator(HashMap.java:968) at java.util.HashMap$KeySet.iterator(HashMap.java:1002) at java.util.HashSet.iterator(HashSet.java:170) at sun.nio.ch.Util$2.iterator(Util.java:303) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space at java.io.BufferedReader.<init>(BufferedReader.java:98) at java.io.BufferedReader.<init>(BufferedReader.java:109) at java.io.LineNumberReader.<init>(LineNumberReader.java:72) at org.apache.felix.utils.properties.Properties$PropertiesReader.<init>(Properties.java:748) at org.apache.felix.utils.properties.Properties.loadLayout(Properties.java:352) at org.apache.felix.utils.properties.Properties.load(Properties.java:142) at org.apache.felix.utils.properties.Properties.load(Properties.java:138) at org.apache.felix.utils.properties.Properties.load(Properties.java:122) at org.apache.felix.utils.properties.Properties.<init>(Properties.java:107) at org.apache.felix.utils.properties.Properties.<init>(Properties.java:96) at org.apache.karaf.jaas.modules.properties.AutoEncryptionSupport$1.run(AutoEncryptionSupport.java:63) Exception in thread "INT-2,ISPN,rk-c7-merge-6c0-16483" at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) java.lang.OutOfMemoryError: Java heap space at org.jgroups.util.Util.readLongSequence(Util.java:2235) at org.jgroups.util.Digest.readFrom(Digest.java:166) at org.jgroups.util.Digest.readFrom(Digest.java:154) at org.jgroups.util.Util.readStreamable(Util.java:1105) at org.jgroups.util.Util.streamableFromBuffer(Util.java:773) at org.jgroups.protocols.pbcast.STABLE.readDigest(STABLE.java:695) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237) at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:448) at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:636) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147) at org.jgroups.protocols.FD.up(FD.java:255) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301) at org.jgroups.protocols.MERGE2.up(MERGE2.java:209) at org.jgroups.protocols.Discovery.up(Discovery.java:379) at org.jgroups.protocols.TP.passMessageUp(TP.java:1399) at org.jgroups.protocols.TP <http://org.jgroups.protocols.tp/>$4.run(TP.java:1327) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Exception in thread "qtp431119273-318" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "qtp431119273-85" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "RMI RenewClean-[10.30.11.239:44444 <http://10.30.11.239:44444/>]" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space _______________________________________________ controller-dev mailing list controller-dev@... <mailto:controller-dev@...> https://lists.opendaylight.org/mailman/listinfo/controller-dev _______________________________________________ controller-dev mailing list controller-dev@... https://lists.opendaylight.org/mailman/listinfo/controller-dev

View All 6 Messages In Topic

#536

Join z.archive.openflowjava-dev@lists.opendaylight.org to automatically receive all group messages.

Home Hashtags Subgroups Terms