On Oct 13, 2015, at 2:31 PM, Tom Pantelis <tompantelis@...> wrote:

Comparing to the last successful run on Oct 5th, there's a couple errors now appearing:

2015-10-13 15:41:51,778 | ERROR | bundle-tracker-0 | ModuleInfoBundleTracker          | 131 - org.opendaylight.controller.config-manager - 0.4.0.SNAPSHOT | Failed to process bundleentry://188.fwk532513438/META-INF/services/org.opendaylight.yangtools.yang.binding.YangModelBindingProvider for bundle org.opendaylight.openflowjava.openflow-protocol-api_0.7.0.SNAPSHOT [188]
java.lang.IllegalStateException: Error while executing getModuleInfo on org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.protocol.rev130731.$YangModelBindingProvider@4a0236de

...

Caused by: java.lang.IllegalStateException: Resource '/META-INF/yang/openflow-instruction.yang' is missing
	at org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.instruction.rev130731.$YangModuleInfoImpl.<init>($YangModuleInfoImpl.java:31)[188:org.opendaylight.openflowjava.openflow-protocol-api:0.7.0.SNAPSHOT]
	

This one has been happening for a while but it *seems* to be benign.

2015-10-13 15:48:11,064 | ERROR | rint Extender: 3 | BlueprintContainerImpl           | 15 - org.apache.aries.blueprint.core - 1.4.2 | Unable to start blueprint container for bundle org.opendaylight.lispflowmapping.mappingservice.shell due to unresolved dependencies [(objectClass=org.opendaylight.lispflowmapping.interfaces.mappingservice.IMappingServiceShell)]
java.util.concurrent.TimeoutException
	at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:336)[15:org.apache.aries.blueprint.core:1.4.2]
	at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)[15:org.apache.aries.blueprint.core:1.4.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_85]
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_85]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)[:1.7.0_85]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)[:1.7.0_85]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_85]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_85]
	at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]

This one appears to emanate from lispflowmapping. It's using blueprint and appears to import an OSGi service, IMappingServiceShell, that wasn't found. The default timeout for blueprint is 5 min but I don't know if the test blocks on this blueprint container and will fail as a result. Either way it seems this should be looked at by a lispflowmapping contributor.

There may be other stuff going on. It would be useful to run it by hand and, when/if it appears stuck, use jstack to get a thread dump. Also it's hard to tell why the OOM errors are occurring - one of the tests indicates they started to occur after shutdown was started. For that it would be useful to get a heap dump via jmap or, better yet, run the test with the -XX:+HeapDumpOnOutOfMemoryError option enabled if possible.

On Tue, Oct 13, 2015 at 4:43 PM, Luis Gomez <ecelgp@...> wrote:

Look in at more failing distribution jobs, there is always a timeout in the Single Feature, the distribution used to build in 8 mins while now it takes more than 1 hour.

BR/Luis

On Oct 13, 2015, at 12:13 PM, Luis Gomez <ecelgp@...> wrote:

Hi all,

I just observed the distribution in master is failing since Oct 6th [1]. Last errors [2] show memory issues like below but I am not sure this is the root cause for this. Can anyone help identifying the problem here?

Thanks/Luis 

[1] https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/

[2] https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/444/testReport/

Exception in thread "qtp1815616686-79" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.HashMap.newKeyIterator(HashMap.java:968)
	at java.util.HashMap$KeySet.iterator(HashMap.java:1002)
	at java.util.HashSet.iterator(HashSet.java:170)
	at sun.nio.ch.Util$2.iterator(Util.java:303)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)

Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
	at java.io.BufferedReader.<init>(BufferedReader.java:98)
	at java.io.BufferedReader.<init>(BufferedReader.java:109)
	at java.io.LineNumberReader.<init>(LineNumberReader.java:72)
	at org.apache.felix.utils.properties.Properties$PropertiesReader.<init>(Properties.java:748)
	at org.apache.felix.utils.properties.Properties.loadLayout(Properties.java:352)
	at org.apache.felix.utils.properties.Properties.load(Properties.java:142)
	at org.apache.felix.utils.properties.Properties.load(Properties.java:138)
	at org.apache.felix.utils.properties.Properties.load(Properties.java:122)
	at org.apache.felix.utils.properties.Properties.<init>(Properties.java:107)
	at org.apache.felix.utils.properties.Properties.<init>(Properties.java:96)
	at org.apache.karaf.jaas.modules.properties.AutoEncryptionSupport$1.run(AutoEncryptionSupport.java:63)
Exception in thread "INT-2,ISPN,rk-c7-merge-6c0-16483" 	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
java.lang.OutOfMemoryError: Java heap space
	at org.jgroups.util.Util.readLongSequence(Util.java:2235)
	at org.jgroups.util.Digest.readFrom(Digest.java:166)
	at org.jgroups.util.Digest.readFrom(Digest.java:154)
	at org.jgroups.util.Util.readStreamable(Util.java:1105)
	at org.jgroups.util.Util.streamableFromBuffer(Util.java:773)
	at org.jgroups.protocols.pbcast.STABLE.readDigest(STABLE.java:695)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:448)
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:636)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
	at org.jgroups.protocols.FD.up(FD.java:255)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
	at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
	at org.jgroups.protocols.Discovery.up(Discovery.java:379)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)
	at org.jgroups.protocols.TP$4.run(TP.java:1327)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Exception in thread "qtp431119273-318" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "qtp431119273-85" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "RMI RenewClean-[10.30.11.239:44444]" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space

_______________________________________________
controller-dev mailing list
controller-dev@...
https://lists.opendaylight.org/mailman/listinfo/controller-dev