Re: [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT


Vishal Thapar <vthapar@...>
 



On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@...> wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@...> wrote:

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.


​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives information about if all services are up or not. It is used in monitoring tool of ODLTools.


Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be addressed.


   2018-07-02T02:47:54,463 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id 3u5gk3qqbpby1c13f3hpivnge

2018-07-02T02:47:54,470 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id yev7cqw2kyjsy6978sf1nmrl

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel                      | 156 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [134:org.apache.shiro.core:1.3.2]

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]

        at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [189:org.jolokia.osgi:1.5.0]

        at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [440:org.ops4j.pax.web.pax-web-runtime:6.0.9]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [151:org.eclipse.jetty.security:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [154:org.eclipse.jetty.servlet:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at java.lang.Thread.run(Thread.java:748) [?:?]

 

   

Thanks,

Faseela

 

From: Tom Pantelis [mailto:tompantelis@...]
Sent: Monday, July 02, 2018 10:29 PM
To: Sam Hague <shague@...>
Cc: Vishal Thapar <vthapar@...>; odl netvirt dev <netvirt-dev@...t.org>; aaa-dev@...; Faseela K <faseela.k@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

 

....

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]

            at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]

            at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]

            at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]

 

 

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?

 

 

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

 

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

 

_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev



_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


Join z.archive.aaa-dev@lists.opendaylight.org to automatically receive all group messages.