1. Topics
  2. [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

[netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

Vishal Thapar <vthapar@...>
 

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
Sam Hague <shague@...>
 

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?


On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:
I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@...
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
netvirt-dev mailing list
netvirt-dev@...
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
Tom Pantelis
 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

....
at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]
at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]
at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]
at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]
at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]


On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:
Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?


On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:
I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
Faseela K <faseela.k@...>
 

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :

 

   2018-07-02T02:47:54,463 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id 3u5gk3qqbpby1c13f3hpivnge

2018-07-02T02:47:54,470 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id yev7cqw2kyjsy6978sf1nmrl

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel                      | 156 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [134:org.apache.shiro.core:1.3.2]

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]

        at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [189:org.jolokia.osgi:1.5.0]

        at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [440:org.ops4j.pax.web.pax-web-runtime:6.0.9]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [151:org.eclipse.jetty.security:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [154:org.eclipse.jetty.servlet:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at java.lang.Thread.run(Thread.java:748) [?:?]

 

   

Thanks,

Faseela

 

From: Tom Pantelis [mailto:tompantelis@...]
Sent: Monday, July 02, 2018 10:29 PM
To: Sam Hague <shague@...>
Cc: Vishal Thapar <vthapar@...>; odl netvirt dev <netvirt-dev@...>; aaa-dev@...; Faseela K <faseela.k@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

 

....

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]

            at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]

            at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]

            at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]

 

 

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?

 

 

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

 

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@...
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

 

_______________________________________________
netvirt-dev mailing list
netvirt-dev@...
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

 
Michael Vorburger <vorburger@...>
 

On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@...> wrote:

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.


   2018-07-02T02:47:54,463 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id 3u5gk3qqbpby1c13f3hpivnge

2018-07-02T02:47:54,470 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id yev7cqw2kyjsy6978sf1nmrl

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel                      | 156 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [134:org.apache.shiro.core:1.3.2]

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]

        at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [189:org.jolokia.osgi:1.5.0]

        at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [440:org.ops4j.pax.web.pax-web-runtime:6.0.9]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [151:org.eclipse.jetty.security:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [154:org.eclipse.jetty.servlet:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at java.lang.Thread.run(Thread.java:748) [?:?]

 

   

Thanks,

Faseela

 

From: Tom Pantelis [mailto:tompantelis@...]
Sent: Monday, July 02, 2018 10:29 PM
To: Sam Hague <shague@...>
Cc: Vishal Thapar <vthapar@...>; odl netvirt dev <netvirt-dev@lists.opendaylight.org>; aaa-dev@...; Faseela K <faseela.k@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

 

....

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]

            at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]

            at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]

            at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]

 

 

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?

 

 

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

 

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

 

_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
Jamo Luhrsen <jluhrsen@...>
 

On 07/03/2018 12:31 PM, Michael Vorburger wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:
Tom,____
   I couldn’t follow your response below completely.____
    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the
shiro error.____
   When I updated the CSIT to retry on failure, it works on subsequent retries.____
    Below is the exception stack trace :
I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.
even if that is a better thing for CSIT to do, it would just be a workaround
to whatever problem is really there.

JamO
____
2018-07-02T02:47:54,463 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id 3u5gk3qqbpby1c13f3hpivnge____
2018-07-02T02:47:54,470 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id yev7cqw2kyjsy6978sf1nmrl____
2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel    | 156 - org.eclipse.jetty.util -
9.3.21.v20170918 |
//10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards
<http://10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards>____
org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]____
        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148)
[134:org.apache.shiro.core:1.3.2]____
        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[134:org.apache.shiro.core:1.3.2]____
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[134:org.apache.shiro.core:1.3.2]____
        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____
        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]____
        at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[189:org.jolokia.osgi:1.5.0]____
        at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[440:org.ops4j.pax.web.pax-web-runtime:6.0.9]____
        at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
[151:org.eclipse.jetty.security:9.3.21.v20170918]____
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[154:org.eclipse.jetty.servlet:9.3.21.v20170918]____
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at
org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____
        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at org.eclipse.jetty.io <http://org.eclipse.jetty.io>.FillInterest.fillable(FillInterest.java:108)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____
        at java.lang.Thread.run(Thread.java:748) [?:?]____
__ __
____
Thanks,____
Faseela____
__ __
*From:*Tom Pantelis [mailto:tompantelis@... <mailto:tompantelis@...>]
*Sent:* Monday, July 02, 2018 10:29 PM
*To:* Sam Hague <shague@... <mailto:shague@...>>
*Cc:* Vishal Thapar <vthapar@... <mailto:vthapar@...>>; odl netvirt dev
<netvirt-dev@... <mailto:netvirt-dev@...>>; aaa-dev@...
<mailto:aaa-dev@...>; Faseela K <faseela.k@... <mailto:faseela.k@...>>
*Subject:* Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT____
__ __
jolokia has a different security code path than restconf that interfaces with shiro. It starts with:____
__ __
....____
at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[130:org.apache.shiro.core:1.3.2]____
            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[130:org.apache.shiro.core:1.3.2]____
            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____
            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]____
            at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[179:org.jolokia.osgi:1.5.0]____
            at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[312:org.ops4j.pax.web.pax-web-runtime:6.0.9]____
            at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[310:org.ops4j.pax.web.pax-web-jetty:6.0.9]____
__ __
__ __
On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@... <mailto:shague@...>> wrote:____
Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?____
__ __
[1] https://jira.opendaylight.org/browse/CONTROLLER-1838 <https://jira.opendaylight.org/browse/CONTROLLER-1838>____
__ __
On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:____
I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even
post ownership change.____
__ __
On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>> wrote:____
Hello aaa-dev,____
   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting
with some of the 3 node netvirt CSIT activities [0].____
   The HTTP requests work most of the cases, but sometimes hit the below error randomly?____
   Any pointers?____
____
<html> <head> <meta http-equiv="Content-Type" content="text/html; <title>Error 500 </title> </head>
<body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing
/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards.
Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id
[4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>____
____
____
Thanks,____
Faseela____
____
[0]
https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz
<https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz>____
____
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____
__ __
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____
_______________________________________________
aaa-dev mailing list
aaa-dev@... <mailto:aaa-dev@...>
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
<https://lists.opendaylight.org/mailman/listinfo/aaa-dev>____
__ __
_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>
_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
Faseela K <faseela.k@...>
 

Michael,

The shard list is being dumped to make sure our single shard configuration is proper at different places of the CSIT. Not to check if cluster is ready.

Thanks,
Faseela

-----Original Message-----
From: Jamo Luhrsen [mailto:jluhrsen@...]
Sent: Wednesday, July 04, 2018 3:31 AM
To: Michael Vorburger <vorburger@...>; Faseela K <faseela.k@...>
Cc: Sam Hague <shague@...>; aaa-dev@...; odl netvirt dev <netvirt-dev@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT



On 07/03/2018 12:31 PM, Michael Vorburger wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:

Tom,____

   I couldn’t follow your response below completely.____

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the
shiro error.____

   When I updated the CSIT to retry on failure, it works on
subsequent retries.____

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from
/jolokia/read/org.opendaylight.controller:type=DistributedOperationalD
atastore,Category=ShardManager,name=shard-manager-operational/LocalSha
rds, but I don't suppose it could (now) be replaced with GET
/diagstatus from https://jira.opendaylight.org/browse/GENIUS-138
?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.
even if that is a better thing for CSIT to do, it would just be a workaround to whatever problem is really there.

JamO
____


2018-07-02T02:47:54,463 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id 3u5gk3qqbpby1c13f3hpivnge____

2018-07-02T02:47:54,470 | INFO  |
org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer |
LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s)
with id yev7cqw2kyjsy6978sf1nmrl____

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel    | 156 - org.eclipse.jetty.util -
9.3.21.v20170918 |
//10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

<http://10.30.170.46:8181/jolokia/read/org.opendaylight.controller:typ
e=DistributedOperationalDatastore,Category=ShardManager,name=shard-man
ager-operational/LocalShards>____

org.apache.shiro.session.UnknownSessionException: There is no
session with id [bf682403-082b-4520-993f-987a8804bd31]____

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170)
[134:org.apache.shiro.core:1.3.2]____

        at
org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222)
[134:org.apache.shiro.core:1.3.2]____

        at

org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSes
sion(AbstractValidatingSessionManager.java:118)
[134:org.apache.shiro.core:1.3.2]____

        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148)
[134:org.apache.shiro.core:1.3.2]____

        at
org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[134:org.apache.shiro.core:1.3.2]____

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[134:org.apache.shiro.core:1.3.2]____

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____

        at
Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source)
[?:?]____

        at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[189:org.jolokia.osgi:1.5.0]____

        at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[440:org.ops4j.pax.web.pax-web-runtime:6.0.9]____

        at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
[151:org.eclipse.jetty.security:9.3.21.v20170918]____

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[154:org.eclipse.jetty.servlet:9.3.21.v20170918]____

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at
org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
[438:org.ops4j.pax.web.pax-web-jetty:6.0.9]____

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at
org.eclipse.jetty.server.Server.handle(Server.java:534)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[153:org.eclipse.jetty.server:9.3.21.v20170918]____

        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____

        at org.eclipse.jetty.io <http://org.eclipse.jetty.io>.FillInterest.fillable(FillInterest.java:108)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____

        at org.eclipse.jetty.io
<http://org.eclipse.jetty.io>.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[145:org.eclipse.jetty.io:9.3.21.v20170918]____

        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____

        at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[156:org.eclipse.jetty.util:9.3.21.v20170918]____

        at java.lang.Thread.run(Thread.java:748) [?:?]____

__ __

____

Thanks,____

Faseela____

__ __

*From:*Tom Pantelis [mailto:tompantelis@... <mailto:tompantelis@...>]
*Sent:* Monday, July 02, 2018 10:29 PM
*To:* Sam Hague <shague@... <mailto:shague@...>>
*Cc:* Vishal Thapar <vthapar@... <mailto:vthapar@...>>; odl netvirt dev
<netvirt-dev@... <mailto:netvirt-dev@...>>; aaa-dev@...
<mailto:aaa-dev@...>; Faseela K <faseela.k@... <mailto:faseela.k@...>>
*Subject:* Re: [Aaa-dev] [netvirt-dev] Random errors while
fetching JOLOKIA URLs in 3 node CSIT____

__ __

jolokia has a different security code path than restconf that
interfaces with shiro. It starts with:____

__ __

....____

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283)
[130:org.apache.shiro.core:1.3.2]____

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
[130:org.apache.shiro.core:1.3.2]____

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42)
[188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]____

            at
Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source)
[?:?]____

            at
org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72)
[179:org.jolokia.osgi:1.5.0]____

            at
org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46)
[312:org.ops4j.pax.web.pax-web-runtime:6.0.9]____

            at
org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
[310:org.ops4j.pax.web.pax-web-jetty:6.0.9]____

__ __

__ __

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...
<mailto:shague@...>> wrote:____

Could this be related to the clustering bugs Vic and Jamo are
following like [1] that deal with 500 responses?____

__ __

[1] https://jira.opendaylight.org/browse/CONTROLLER-1838
<https://jira.opendaylight.org/browse/CONTROLLER-1838>____

__ __

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar
<vthapar@... <mailto:vthapar@...>> wrote:____

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even
post ownership change.____

__ __

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K
<faseela.k@... <mailto:faseela.k@...>> wrote:____

Hello aaa-dev,____

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting
with some of the 3 node netvirt CSIT activities
[0].____

   The HTTP requests work most of the cases, but
sometimes hit the below error randomly?____

   Any pointers?____

____

<html> <head> <meta http-equiv="Content-Type" content="text/html; <title>Error 500 </title> </head>
<body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing
/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards.
Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id
[4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr
/> </body> </html>____

____

____

Thanks,____

Faseela____

____

[0]
https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

<https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelane
tvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-p
lugin/log_full.html.gz>____

____


_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____

__ __

_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>____


_______________________________________________
aaa-dev mailing list
aaa-dev@... <mailto:aaa-dev@...>
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
<https://lists.opendaylight.org/mailman/listinfo/aaa-dev>____

__ __


_______________________________________________
netvirt-dev mailing list
netvirt-dev@... <mailto:netvirt-dev@...>
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
<https://lists.opendaylight.org/mailman/listinfo/netvirt-dev>




_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
Vishal Thapar <vthapar@...>
 



On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@...> wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@...> wrote:

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.


​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives information about if all services are up or not. It is used in monitoring tool of ODLTools.

https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170

Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be addressed.


   2018-07-02T02:47:54,463 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@38d77047Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id 3u5gk3qqbpby1c13f3hpivnge

2018-07-02T02:47:54,470 | INFO  | org.ops4j.pax.web.service.jetty.internal.LateInvalidatingHashSessionManager@19c0aeb1Timer | LateInvalidatingHashSessionManager | 438 - org.ops4j.pax.web.pax-web-jetty - 6.0.9 | Timing out for 1 session(s) with id yev7cqw2kyjsy6978sf1nmrl

2018-07-02T02:48:05,934 | WARN  | qtp1371496041-116 | HttpChannel                      | 156 - org.eclipse.jetty.util - 9.3.21.v20170918 | //10.30.170.46:8181/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards

org.apache.shiro.session.UnknownSessionException: There is no session with id [bf682403-082b-4520-993f-987a8804bd31]

        at org.apache.shiro.session.mgt.eis.AbstractSessionDAO.readSession(AbstractSessionDAO.java:170) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSessionFromDataSource(DefaultSessionManager.java:236) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DefaultSessionManager.retrieveSession(DefaultSessionManager.java:222) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractValidatingSessionManager.doGetSession(AbstractValidatingSessionManager.java:118) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupSession(AbstractNativeSessionManager.java:148) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.lookupRequiredSession(AbstractNativeSessionManager.java:152) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.AbstractNativeSessionManager.getHost(AbstractNativeSessionManager.java:237) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.mgt.DelegatingSession.getHost(DelegatingSession.java:111) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.session.ProxiedSession.getHost(ProxiedSession.java:93) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [134:org.apache.shiro.core:1.3.2]

        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [134:org.apache.shiro.core:1.3.2]

        at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [198:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

        at Proxyb44c2d91_1f07_4b19_be55_2f3cb0209c21.authenticate(Unknown Source) [?:?]

        at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [189:org.jolokia.osgi:1.5.0]

        at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [440:org.ops4j.pax.web.pax-web-runtime:6.0.9]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [151:org.eclipse.jetty.security:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [154:org.eclipse.jetty.servlet:9.3.21.v20170918]

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80) [438:org.ops4j.pax.web.pax-web-jetty:6.0.9]

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.Server.handle(Server.java:534) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [153:org.eclipse.jetty.server:9.3.21.v20170918]

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [145:org.eclipse.jetty.io:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [156:org.eclipse.jetty.util:9.3.21.v20170918]

        at java.lang.Thread.run(Thread.java:748) [?:?]

 

   

Thanks,

Faseela

 

From: Tom Pantelis [mailto:tompantelis@...]
Sent: Monday, July 02, 2018 10:29 PM
To: Sam Hague <shague@...>
Cc: Vishal Thapar <vthapar@...>; odl netvirt dev <netvirt-dev@...t.org>; aaa-dev@...; Faseela K <faseela.k@...>
Subject: Re: [Aaa-dev] [netvirt-dev] Random errors while fetching JOLOKIA URLs in 3 node CSIT

 

jolokia has a different security code path than restconf that interfaces with shiro. It starts with:

 

....

at org.apache.shiro.subject.support.DefaultSubjectContext.resolveHost(DefaultSubjectContext.java:270) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSubjectFactory.createSubject(DefaultSubjectFactory.java:45) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.doCreateSubject(DefaultSecurityManager.java:369) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:344) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.createSubject(DefaultSecurityManager.java:183) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:283) [130:org.apache.shiro.core:1.3.2]

            at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256) [130:org.apache.shiro.core:1.3.2]

            at org.opendaylight.aaa.authenticator.ODLAuthenticator.authenticate(ODLAuthenticator.java:42) [188:org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT]

            at Proxy02007817_e19a_4a1a_8178_e2f238fd17a0.authenticate(Unknown Source) [?:?]

            at org.jolokia.osgi.security.ServiceAuthenticationHttpContext.handleSecurity(ServiceAuthenticationHttpContext.java:72) [179:org.jolokia.osgi:1.5.0]

            at org.ops4j.pax.web.service.internal.WebContainerContextWrapper.handleSecurity(WebContainerContextWrapper.java:46) [312:org.ops4j.pax.web.pax-web-runtime:6.0.9]

            at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) [310:org.ops4j.pax.web.pax-web-jetty:6.0.9]

 

 

On Mon, Jul 2, 2018 at 12:44 PM, Sam Hague <shague@...> wrote:

Could this be related to the clustering bugs Vic and Jamo are following like [1] that deal with 500 responses?

 

 

On Mon, Jul 2, 2018 at 2:27 AM Vishal Thapar <vthapar@...> wrote:

I've seen this in my local testing during bringup or during ownership change. Sometimes it persists even post ownership change.

 

On Mon, Jul 2, 2018 at 11:24 AM, Faseela K <faseela.k@...> wrote:

Hello aaa-dev,

   I am trying to dump some of the MDSAL exposed MBeans with jolokia over HTTP , while am experimenting with some of the 3 node netvirt CSIT activities [0].

   The HTTP requests work most of the cases, but sometimes hit the below error randomly?

   Any pointers?

 

  <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards. Reason: <pre> org.apache.shiro.session.UnknownSessionException: There is no session with id [4050ca12-d12d-4965-8204-d3c11cb16e1b]</pre></p> <hr /> </body> </html>

 

 

Thanks,

Faseela

 

[0] https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/faseelanetvirt-csit-3node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev

 

_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev


_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev

 


_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev



_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
Tom Pantelis
 



On Tue, Jul 3, 2018 at 9:39 PM, Vishal Thapar <vthapar@...> wrote:


On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@...> wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@...> wrote:

Tom,

   I couldn’t follow your response below completely.

    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it throws the shiro error.

   When I updated the CSIT to retry on failure, it works on subsequent retries.

    Below is the exception stack trace :


I don't know the details of what this CSIT wants from /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards, but I don't suppose it could (now) be replaced with GET /diagstatus from https://jira.opendaylight.org/browse/GENIUS-138 ?  That requires no authentication, so it would never suffer from this particular problem. Just a thought.


​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives information about if all services are up or not. It is used in monitoring tool of ODLTools.


Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be addressed.




The exception emanates from shiro code so that would be the place to start. Anyone have cycles now to try to look into this?
Jamo Luhrsen <jluhrsen@...>
 

Here is a jira to track this one:

https://jira.opendaylight.org/browse/CONTROLLER-1850


JamO

On 07/03/2018 07:16 PM, Tom Pantelis wrote:
On Tue, Jul 3, 2018 at 9:39 PM, Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:
On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@... <mailto:vorburger@...>> wrote:
On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:
Tom,____
   I couldn’t follow your response below completely.____
    But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it
throws the shiro error.____
   When I updated the CSIT to retry on failure, it works on subsequent retries.____
    Below is the exception stack trace :
I don't know the details of what this CSIT wants from
/jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards,
but I don't suppose it could (now) be replaced with GET /diagstatus from
https://jira.opendaylight.org/browse/GENIUS-138 <https://jira.opendaylight.org/browse/GENIUS-138> ?  That
requires no authentication, so it would never suffer from this particular problem. Just a thought.
​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives
information about if all services are up or not. It is used in monitoring tool of ODLTools.
https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170
<https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170>
Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be
addressed.
____
The exception emanates from shiro code so that would be the place to start. Anyone have cycles now to try to look into this?
_______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
Tom Pantelis
 



On Thu, Jul 5, 2018 at 6:29 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Here is a jira to track this one:

 https://jira.opendaylight.org/browse/CONTROLLER-1850


This is really a AAA issue so moved to https://jira.opendaylight.org/browse/AAA-176
 



JamO

On 07/03/2018 07:16 PM, Tom Pantelis wrote:


On Tue, Jul 3, 2018 at 9:39 PM, Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:



    On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@... <mailto:vorburger@...>> wrote:

        On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:

            Tom,____

                I couldn’t follow your response below completely.____

                 But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it
             throws the shiro error.____

                When I updated the CSIT to retry on failure, it works on subsequent retries.____

                 Below is the exception stack trace :


        I don't know the details of what this CSIT wants from
        /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards,
        but I don't suppose it could (now) be replaced with GET /diagstatus from
         https://jira.opendaylight.org/browse/GENIUS-138 <https://jira.opendaylight.org/browse/GENIUS-138> ?  That
        requires no authentication, so it would never suffer from this particular problem. Just a thought.


    ​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives
    information about if all services are up or not. It is used in monitoring tool of ODLTools.

    https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170
    <https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170>

    Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be
    addressed.

             ____




The exception emanates from shiro code so that would be the place to start. Anyone have cycles now to try to look into this?


 _______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev
Ariel Adam <aadam@...>
 

Ryan, can you take a look?

Thanks. 

On Fri, Jul 6, 2018 at 6:29 AM, Tom Pantelis <tompantelis@...> wrote:


On Thu, Jul 5, 2018 at 6:29 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Here is a jira to track this one:

 https://jira.opendaylight.org/browse/CONTROLLER-1850


This is really a AAA issue so moved to https://jira.opendaylight.org/browse/AAA-176
 



JamO

On 07/03/2018 07:16 PM, Tom Pantelis wrote:


On Tue, Jul 3, 2018 at 9:39 PM, Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:



    On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@... <mailto:vorburger@...>> wrote:

        On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:

            Tom,____

                I couldn’t follow your response below completely.____

                 But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it
             throws the shiro error.____

                When I updated the CSIT to retry on failure, it works on subsequent retries.____

                 Below is the exception stack trace :


        I don't know the details of what this CSIT wants from
        /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards,
        but I don't suppose it could (now) be replaced with GET /diagstatus from
         https://jira.opendaylight.org/browse/GENIUS-138 <https://jira.opendaylight.org/browse/GENIUS-138> ?  That
        requires no authentication, so it would never suffer from this particular problem. Just a thought.


    ​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives
    information about if all services are up or not. It is used in monitoring tool of ODLTools.

    https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170
    <https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170>

    Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be
    addressed.

             ____




The exception emanates from shiro code so that would be the place to start. Anyone have cycles now to try to look into this?


 _______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev



_______________________________________________
netvirt-dev mailing list
netvirt-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
Tom Pantelis
 



On Sun, Jul 8, 2018 at 4:33 AM, Ariel Adam <aadam@...> wrote:
Ryan, can you take a look?

Thanks. 

On Fri, Jul 6, 2018 at 6:29 AM, Tom Pantelis <tompantelis@...> wrote:


On Thu, Jul 5, 2018 at 6:29 PM, Jamo Luhrsen <jluhrsen@...> wrote:
Here is a jira to track this one:

 https://jira.opendaylight.org/browse/CONTROLLER-1850


This is really a AAA issue so moved to https://jira.opendaylight.org/browse/AAA-176

I have an idea for this - will push patch soon. 
 
 



JamO

On 07/03/2018 07:16 PM, Tom Pantelis wrote:


On Tue, Jul 3, 2018 at 9:39 PM, Vishal Thapar <vthapar@... <mailto:vthapar@...>> wrote:



    On Wed, Jul 4, 2018 at 1:01 AM, Michael Vorburger <vorburger@... <mailto:vorburger@...>> wrote:

        On Tue, Jul 3, 2018 at 7:21 AM, Faseela K <faseela.k@... <mailto:faseela.k@...>>wrote:

            Tom,____

                I couldn’t follow your response below completely.____

                 But, the URL whatever I am using in CSIT works most of the times, and only in some random cases it
             throws the shiro error.____

                When I updated the CSIT to retry on failure, it works on subsequent retries.____

                 Below is the exception stack trace :


        I don't know the details of what this CSIT wants from
        /jolokia/read/org.opendaylight.controller:type=DistributedOperationalDatastore,Category=ShardManager,name=shard-manager-operational/LocalShards,
        but I don't suppose it could (now) be replaced with GET /diagstatus from
         https://jira.opendaylight.org/browse/GENIUS-138 <https://jira.opendaylight.org/browse/GENIUS-138> ?  That
        requires no authentication, so it would never suffer from this particular problem. Just a thought.


    ​It gives information about which node is leader and which is follower for a given shard while diagstatus only gives
    information about if all services are up or not. It is used in monitoring tool of ODLTools.

    https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170
    <https://github.com/opendaylight/odltools/blob/master/odltools/monitor/monitor_odl_cluster.py#L170>

    Unless every bit of information we get from jolokia can be provided through an alternate rest call, this needs to be
    addressed.

             ____




The exception emanates from shiro code so that would be the place to start. Anyone have cycles now to try to look into this?


 _______________________________________________
aaa-dev mailing list
aaa-dev@...
https://lists.opendaylight.org/mailman/listinfo/aaa-dev



_______________________________________________
netvirt-dev mailing list
netvirt-dev@....org
https://lists.opendaylight.org/mailman/listinfo/netvirt-dev
1 - 13 of 13