On Oct 14, 2015, at 2:33 AM, Lori Jakab <lojakab@...> wrote:

On 10/14/15 3:28 AM, Luis Gomez wrote:

Thanks Tom for your analysis, lisp and ofjava people, would you mind
taking a look at these comments?

Hi Luis, all,

We just pushed a patch to disable the timeout on the blueprint
container, to allow more time for the services to come up.  We've seen
this exception in the past for example when a patch slowed down the
config subsystem as a side effect.  It means that in the 5 minutes
default timeout the core mappingservice did not initialize, and didn't
register an implementation of IMappingServiceShell with OSGi.

I have no idea how the test environment handles the exception, so I hope
disabling the timeout will help.

From the other analysis on the thread it looks like lisp is only causing
issues in terms of more heap needed after the additional features were
added, but not in terms of functionality.

-Lori

BR/Luis

On Oct 13, 2015, at 2:31 PM, Tom Pantelis <tompantelis@...
<mailto:tompantelis@...>> wrote:

Comparing to the last successful run on Oct 5th, there's a couple
errors now appearing:

2015-10-13 15:41:51,778 | ERROR | bundle-tracker-0 | ModuleInfoBundleTracker          | 131 - org.opendaylight.controller.config-manager - 0.4.0.SNAPSHOT | Failed to process bundleentry://188.fwk532513438/META-INF/services/org.opendaylight.yangtools.yang.binding.YangModelBindingProvider for bundle org.opendaylight.openflowjava.openflow-protocol-api_0.7.0.SNAPSHOT [188]
java.lang.IllegalStateException: Error while executing getModuleInfo on org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.protocol.rev130731.$YangModelBindingProvider@4a0236de
...
Caused by: java.lang.IllegalStateException: Resource '/META-INF/yang/openflow-instruction.yang' is missing
at org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.common.instruction.rev130731.$YangModuleInfoImpl.<init>($YangModuleInfoImpl.java:31)[188:org.opendaylight.openflowjava.openflow-protocol-api:0.7.0.SNAPSHOT]

This one has been happening for a while but it *seems* to be benign.

2015-10-13 15:48:11,064 | ERROR | rint Extender: 3 | BlueprintContainerImpl           | 15 - org.apache.aries.blueprint.core - 1.4.2 | Unable to start blueprint container for bundle org.opendaylight.lispflowmapping.mappingservice.shell due to unresolved dependencies [(objectClass=org.opendaylight.lispflowmapping.interfaces.mappingservice.IMappingServiceShell)]
java.util.concurrent.TimeoutException
at org.apache.aries.blueprint.container.BlueprintContainerImpl$1.run(BlueprintContainerImpl.java:336)[15:org.apache.aries.blueprint.core:1.4.2]
at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)[15:org.apache.aries.blueprint.core:1.4.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)[:1.7.0_85]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)[:1.7.0_85]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)[:1.7.0_85]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)[:1.7.0_85]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_85]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_85]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_85]
This one appears to emanate from lispflowmapping. It's using
blueprint and appears to import an OSGi
service, IMappingServiceShell, that wasn't found. The default timeout
for blueprint is 5 min but I don't know if the test blocks on this
blueprint container and will fail as a result. Either way it seems
this should be looked at by a lispflowmapping contributor.
There may be other stuff going on. It would be useful to run it by
hand and, when/if it appears stuck, use jstack to get a thread dump.
Also it's hard to tell why the OOM errors are occurring - one of the
tests indicates they started to occur after shutdown was started. For
that it would be useful to get a heap dump via jmap or, better yet,
run the test with the -XX:+HeapDumpOnOutOfMemoryError option enabled
if possible.

On Tue, Oct 13, 2015 at 4:43 PM, Luis Gomez <ecelgp@...
<mailto:ecelgp@...>> wrote:

   Look in at more failing distribution jobs, there is always a
   timeout in the Single Feature, the distribution used to build in
   8 mins while now it takes more than 1 hour.

   BR/Luis

   On Oct 13, 2015, at 12:13 PM, Luis Gomez <ecelgp@...
   <mailto:ecelgp@...>> wrote:

   Hi all,

   I just observed the distribution in master is failing since Oct
   6th [1]. Last errors [2] show memory issues like below but I am
   not sure this is the root cause for this. Can anyone help
   identifying the problem here?

   Thanks/Luis

   [1] https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/
   [2]
   https://jenkins.opendaylight.org/releng/view/yangtools/job/yangtools-distribution-beryllium/444/testReport/

   Exception in thread "qtp1815616686-79" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.HashMap.newKeyIterator(HashMap.java:968)
    at java.util.HashMap$KeySet.iterator(HashMap.java:1002)
    at java.util.HashSet.iterator(HashSet.java:170)
    at sun.nio.ch.Util$2.iterator(Util.java:303)
    at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
    at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)

   Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
    at java.io.BufferedReader.<init>(BufferedReader.java:98)
    at java.io.BufferedReader.<init>(BufferedReader.java:109)
    at java.io.LineNumberReader.<init>(LineNumberReader.java:72)
    at org.apache.felix.utils.properties.Properties$PropertiesReader.<init>(Properties.java:748)
    at org.apache.felix.utils.properties.Properties.loadLayout(Properties.java:352)
    at org.apache.felix.utils.properties.Properties.load(Properties.java:142)
    at org.apache.felix.utils.properties.Properties.load(Properties.java:138)
    at org.apache.felix.utils.properties.Properties.load(Properties.java:122)
    at org.apache.felix.utils.properties.Properties.<init>(Properties.java:107)
    at org.apache.felix.utils.properties.Properties.<init>(Properties.java:96)
    at org.apache.karaf.jaas.modules.properties.AutoEncryptionSupport$1.run(AutoEncryptionSupport.java:63)
   Exception in thread "INT-2,ISPN,rk-c7-merge-6c0-16483" at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
   java.lang.OutOfMemoryError: Java heap space
    at org.jgroups.util.Util.readLongSequence(Util.java:2235)
    at org.jgroups.util.Digest.readFrom(Digest.java:166)
    at org.jgroups.util.Digest.readFrom(Digest.java:154)
    at org.jgroups.util.Util.readStreamable(Util.java:1105)
    at org.jgroups.util.Util.streamableFromBuffer(Util.java:773)
    at org.jgroups.protocols.pbcast.STABLE.readDigest(STABLE.java:695)
    at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:237)
    at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:448)
    at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:636)
    at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
    at org.jgroups.protocols.FD.up(FD.java:255)
    at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
    at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)
    at org.jgroups.protocols.Discovery.up(Discovery.java:379)
    at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)
    at org.jgroups.protocols.TP <http://org.jgroups.protocols.tp/>$4.run(TP.java:1327)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
   Exception in thread "qtp431119273-318" java.lang.OutOfMemoryError: GC overhead limit exceeded
   Exception in thread "qtp431119273-85" java.lang.OutOfMemoryError: GC overhead limit exceeded
   Exception in thread "RMI RenewClean-[10.30.11.239:44444 <http://10.30.11.239:44444/>]" java.lang.OutOfMemoryError: GC overhead limit exceeded
   Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space

   _______________________________________________
   controller-dev mailing list
   controller-dev@...
   <mailto:controller-dev@...>
   https://lists.opendaylight.org/mailman/listinfo/controller-dev

_______________________________________________
controller-dev mailing list
controller-dev@...
https://lists.opendaylight.org/mailman/listinfo/controller-dev