Connection Mgr issue


Luis Gomez
 

Hi ovsdb devs,

I am writing to you because since some time ago I am seeing some OSGI console errors when running the OVSDB test in Base edition:


And after some research I found out there is an issue in the connection manager the first time it connects to OVSDB server in mininet after controller starts. The issue is that the existing OF connections get bounced as you can see in this printout:

osgi> 2014-04-22 12:32:54.573 PDT [SwitchEvent Thread] INFO  o.o.c.p.o.core.internal.Controller - 00:00:00:00:00:00:00:02 is removed
2014-04-22 12:32:54.575 PDT [ControllerI/O Thread] INFO  o.o.c.p.o.core.internal.Controller - Switch:10.1.19.32:59989 is connected to the Controller
2014-04-22 12:32:54.765 PDT [SwitchEvent Thread] INFO  o.o.c.p.o.core.internal.Controller - 00:00:00:00:00:00:00:03 is removed
2014-04-22 12:32:54.770 PDT [ControllerI/O Thread] INFO  o.o.c.p.o.core.internal.Controller - Switch:10.1.19.32:59990 is connected to the Controller
2014-04-22 12:32:54.898 PDT [SwitchEvent Thread] INFO  o.o.c.p.o.core.internal.Controller - 00:00:00:00:00:00:00:01 is removed
2014-04-22 12:32:54.902 PDT [ControllerI/O Thread] INFO  o.o.c.p.o.core.internal.Controller - Switch:10.1.19.32:59991 is connected to the Controller

The steps to reproduce this are very simple:

- start controller base edition ./run.sh
- start mininet -tree,3
- Use connection mgr REST API to connect to OVS manager in mininet:, i.e. http://10.1.19.33:8080/controller/nb/v2/connectionmanager/node/HOST1/address/10.1.19.32/port/6640/

Please let me know if you see this problem too and whether I should open a bug to ovsdb or controller project to fix it.

Thanks/Luis


On Apr 22, 2014, at 7:59 AM, Brent Salisbury <brent.salisbury@...> wrote:

Thanks for the info Andre. Really appreciate you following up. Will get some better logging there. 

Thanks,
Brent

On Apr 21, 2014, at 7:20 PM, André Martins <aanm@...> wrote:

Hello Madhu,
sorry to bother you guys but I was around this for 2 days.
The fault is all mine, I thought you change the port of the controller from 6633 to 6640, but the 6640 is for the ovsdb.
I was running mininet as
$ sudo mn --switch ovsk --controller remote,ip=127.0.0.1,port=6640 --topo single,3
So the openvswitch was trying to contact the ovsdb.
Thank you so much for the help.
André Martins

On 04/21/2014 02:51 PM, Madhu Venugopal wrote:
Hi Andre,

Thanks for the info. I see a potential issue in using the ByteSourceJsonBootstrapper for detecting the encoding.
This class looks for the BOM in the JSON stream to determine the encoding. But in addition, it also "auto-detects"
the encoding when BOM is absent. I think that auto-detection can cause issues with the way we are handling the
data stream. Since the OVSDB specification is strictly UTF-8 & there are no BOMs sent, am thinking of removing
the piece of code that checks for the encoding.

Can you please apply this patch : https://gist.github.com/11143238 in your local repo and give it a try ?
Essentially, i have just commented out the piece of code that checks for the encoding in the decode() method of JsonRpcDecoder.java

BTW, the wireshark capture that you sent didnt capture the response to the "get_schema" RPC call. But I see the FIN
immediately after the get_schema call.

Please let us know if this patch works for you. We will continue to look into the issue.

Thanks,
Madhu

On 4/21/14, 2:29 AM, André Martins wrote:
Hello Madhu,
I've tried with the netty 4.0.10 and the problem persists.
Inside the ovsdb/ovsdb/src/main/java/org/opendaylight/ovsdb/lib/jsonrpc/JsonRpcDecoder.java line 82, I've put                 logger.trace("jsonEncoding.getJavaName() = {}", jsonEncoding.getJavaName());
Also, the buff content is 0x01, 0x0, 0x0, 0x08.
I'm using the 2.1.0 version of openvswith.
Thanks,
André Martins

On 04/21/2014 06:14 AM, Madhu Venugopal wrote:
Hi Andre,

As you notice in the Exception, we use Netty & we have a custom decoder : JsonRpcDecoder which handles the Initial
JSON decoding on the Netty pipeline before handing over to Jackson.

As per RFC 7047, OVSDB uses JSON-RPCv1.0 and only UTF-8 encoding is supported from the ovsdb server side.
Hence, the JsonRpcDecoder.java performs a strict check on the data format and throw an InvalidEncodingException
exception if it detects a non UTF-8 format. So, this is a deliberate exception and should be expected if the data format
is anything other than UTF-8.

Can you please help explain how you debugged the problem to spot the UTF-16LE and are you using stock OpenVSwitch
when you hit this problem ?

It could be unrelated, but in March timeframe there was a commit that bumped the Netty version to 4.0.17 in the controller
project (and persumably on integration as well).

Thanks,
Madhu

On 4/20/14, 8:45 PM, André Martins wrote:
Thannk you for the response Brent,
I've just download the zip from that link with ./run.sh -of13 -debug
I'm developing a bundle to opendaylight but I didn't pull the controller repo since around April 1st, before that everything worked.
I've tried to checkout the controller's repo to a commit before April 1st but after I do both controller and integration project
a mvn clean install, the problem doesn't go away.
$ sudo ovs-vsctl show
1019d2bd-473d-4def-9a6b-d97f6344ea70
$ Sudo ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.1.0
Compiled Apr 20 2014 02:12:48

$ sudo ovsdb-client dump
see http://paste.fedoraproject.org/95592/13980508/

Wireshark capture is attached in this email and the opendaylight.log
I could debug that where this exception is throw the data is UTF-16LE.
Thanks,
André Martins


On 04/21/2014 04:04 AM, Brent Salisbury wrote:
Hi Andre,
If you can provide some additional information about your environment we can figure out the root of the issue.

- QDescription how you are using the OVSDB plugin, e.g. API calls via Postman, the AD_SAL API,  Neutron (doubtful since you are using the base edition) etc.
$ sudo ovs-vsctl show  -- (Share command outputs via a public gist/pastebin/etc or attach as a text file)
$ sudo ovs-vsctl --version 
$ sudo ovsdb-client dump 
- if up to it, upload a packet capture to a public fileshare containing the unhandled transaction to see what is on the wire and not getting deserialized by Netty. In Wireshark filtering on " tcp.port == 6640 " without quotes and exporting only the filtered packets will give us the relevant information about the problematic method. 


I am adding the ovsdb-dev@... listserv as this will be an OVSDB plugin specific issue if you will drop controller-dev on your reply. Leaving it for this email to communicate the handoff. Also if more convenient for you we can work through this realtime on irc.freenode.net channel #opendaylight-ovsdb 

Thanks for the assistance and interest,
-Brent


From: André Martins <aanm@...>
Date: Sunday, April 20, 2014 at 1:56 PM
To: <controller-dev@...>
Subject: [controller-dev] Huge problem - InvalidEncodingException: currently only UTF-8 is supported

Hello,
I'm having a huge problem running opendaylight with openvswitch.
When openvswitch tries to connect opendaylight, this happens in opendaylight's terminal.

2014-04-20 07:06:14.161 WEST [nioEventLoopGroup-6-4] WARN  i.n.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.opendaylight.ovsdb.lib.jsonrpc.InvalidEncodingException: currently only UTF-8 is supported
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:259) ~[na:na]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:192) ~[na:na]
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelInactive(DefaultChannelHandlerContext.java:240) [bundlefile:4.0.17.Final]
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelInactive(DefaultChannelHandlerContext.java:226) [bundlefile:4.0.17.Final]
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) [bundlefile:4.0.17.Final]
        at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:213) [bundlefile:4.0.17.Final]
        at io.netty.channel.DefaultChannelHandlerContext.invokeChannelInactive(DefaultChannelHandlerContext.java:240) [bundlefile:4.0.17.Final]
        at io.netty.channel.DefaultChannelHandlerContext.fireChannelInactive(DefaultChannelHandlerContext.java:226) [bundlefile:4.0.17.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:767) [bundlefile:4.0.17.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$5.run(AbstractChannel.java:567) [bundlefile:4.0.17.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354) [bundlefile:4.0.17.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353) [bundlefile:4.0.17.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [bundlefile:4.0.17.Final]
        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: org.opendaylight.ovsdb.lib.jsonrpc.InvalidEncodingException: currently only UTF-8 is supported
        at org.opendaylight.ovsdb.lib.jsonrpc.JsonRpcDecoder.decode(JsonRpcDecoder.java:84) ~[na:na]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:228) ~[na:na]
        ... 13 common frames omitted


I'm running the https://jenkins.opendaylight.org/integration/job/integration-project-centralized-integration/lastSuccessfulBuild/artifact/distributions/base/target/distributions-base-0.1.2-SNAPSHOT-osgipackage.zip
with the ./run.sh -of13 -debug options
Thanks,
André Martins
_______________________________________________ controller-dev mailing list controller-dev@... https://lists.opendaylight.org/mailman/listinfo/controller-dev



_______________________________________________
ovsdb-dev mailing list
ovsdb-dev@...
https://lists.opendaylight.org/mailman/listinfo/ovsdb-dev




_______________________________________________
controller-dev mailing list
controller-dev@...
https://lists.opendaylight.org/mailman/listinfo/controller-dev

Join integration-dev@lists.opendaylight.org to automatically receive all group messages.