Re: [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT


Jamo Luhrsen <jluhrsen@...>
 

On 07/03/2018 12:31 PM, Michael Vorburger wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:
Tom,____
   I couldn’t follow your response below completely.____
    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the
shiro error.____
   When I updated the CSIT to retry on failure, it works on subsequent retries.____
    Below is the exception stack trace :
I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.
even if that is a better thing for CSIT to do, it would just be a workaround
to whatever problem is really there.

JamO
____
2018-07-02T02:47:54,463 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id 3u5gk3qqbpby1c13f3hpivnge____
2018-07-02T02:47:54,470 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id yev7cqw2kyjsy6978sf1nmrl____
2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel    | 156 - org.eclipse.jetty.util -
9.3.21.v20170918 |
//10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards
<http://10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards>____
org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]____
        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[134:org.apache.shiro.core:1.3.2]____
        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____
        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]____
        at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[189:org.jolokia.osgi:1.5.0]____
        at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[440:org.ops4j.pax.web.pax-web-runtime:6.0.9]____
        at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
[151:org.eclipse.jetty.security:9.3.21.v20170918]____
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[154:org.eclipse.jetty.servlet:9.3.21.v20170918]____
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at
org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at org.eclipse.jetty.io <http://org.eclipse.jetty.io>.FillInterest.fillable(FillInterest.java:108)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at java.lang.Thread.run(Thread.java:748) [?:?]____
__ __
____
Thanks,____
Faseela____
__ __
*From:*Tom Pantelis [mailto:tompantelis@... <mailto:tompantelis@...>]
*Sent:* Monday, July 02, 2018 10:29 PM
*To:* Sam Hague <shague@... <mailto:shague@...>>
*Cc:* Vishal Thapar <vthapar@... <mailto:vthapar@...>>; odl netvirt dev
<netvirt-dev@... <mailto:netvirt-dev@...>>; aaa-dev@...
<mailto:aaa-dev@...>; Faseela K <faseela.k@... <mailto:faseela.k@...>>
*Subject:* Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT____
__ __
jolokia has a different security code path than restconf that interfaces with shiro. It starts with:____
__ __
....____
at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[130:org.apache.shiro.core:1.3.2]____
            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____
            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]____
            at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[179:org.jolokia.osgi:1.5.0]____
            at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[312:org.ops4j.pax.web.pax-web-runtime:6.0.9]____
            at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[310:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
__ __
__ __
On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@... <mailto:shague@...>> wrote:____
Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?____
__ __
[1] https://jira.opendaylight.org/browse/CONTROLLER-1838 <https://jira.opendaylight.org/browse/CONTROLLER-1838>____
__ __
On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:____
I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even
post ownership change.____
__ __
On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>> wrote:____
Hello aaa-dev,____
   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting
with some of the 3 node netvirt CSIT activities [0].____
   The HTTP requests work most of the cases, but sometimes hit the below error randomly?____
   Any pointers?____
____
<html> <head> <meta http-equiv="Content-Type" content="text/html; <title>Error 500 </title> </head>
<body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing
/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards.
Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id
[4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>____
____
____
Thanks,____
Faseela____
____
[0]
https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz
<https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz>____
____
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____
__ __
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____
_______________________________________________
aaa-dev mailing list
aaa-dev@... <mailto:aaa-dev@...>
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
<https://lists.opendaylight.org/mailman/listinfo/aaa-dev>____
__ __
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>
_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

Join z.archive.aaa-dev@lists.opendaylight.org to automatically receive all group messages.