Re: [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT


Michael Vorburger <vorburger@...>
 

On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@...> wrote:

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.


   2018-07-02T02:47:54,463 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id 3u5gk3qqbpby1c13f3hpivnge

2018-07-02T02:47:54,470 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id yev7cqw2kyjsy6978sf1nmrl

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel                      | 156 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [134:org.apache.shiro.core:1.3.2]

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]

        at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [189:org.jolokia.osgi:1.5.0]

        at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [440:org.ops4j.pax.web.pax-web-runtime:6.0.9]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [151:org.eclipse.jetty.security:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [154:org.eclipse.jetty.servlet:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at java.lang.Thread.run(Thread.java:748) [?:?]

 

   

Thanks,

Faseela

 

From: Tom Pantelis [mailto:tompantelis@...]
Sent: Monday, July 02, 2018 10:29 PM
To: Sam Hague <shague@...>
Cc: Vishal Thapar <vthapar@...>; odl netvirt dev <netvirt-dev@lists.opendaylight.org>; aaa-dev@...; Faseela K <faseela.k@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

 

....

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]

            at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]

            at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]

            at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]

 

 

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?

 

 

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

 

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

 

_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


Join z.archive.aaa-dev@lists.opendaylight.org to automatically receive all group messages.