|
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks, -Moiz
|
|
I think more than one component might
find the change of mastership (or primary) significant - hence
they should be able to register a listener on Clustering Service
-- which would notify the change in mastership. Its upto the
component how to handle the change in mastership event.
thx
-Basheeruddin
On 06/04/2014 01:10 PM, Moiz Raja wrote:
The Clustering design in Helium is based on a Master-Slave model
where all writes essentially go to the primary replica. In looking
at the OF 1.3 Spec it appears that OF also supports the notion of
Master-Slave controller and it seems that it would be the right
mode of operation for us to use.
To properly support Master-Slave mode I believe the
openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller
the master (makeMaster). This method will be invoked by
an external component. Since the clustering scheme is based on
data shards we could come up with a simple rule that a member
that has the inventory shard will also use the openflowplugin.
When the external component notices that the primary shard for
inventory has changed to the current cluster member then it will
invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message
to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to
re-register all routed Rpc’s that were registered by the
previous master. I haven’t thought through exactly how the new
master would discover those Rpc’s - but one way would be enhance
the Rpc registration API to include some meta information which
could be used to query for those Rpc registrations. Another
option could be enhance the RpcProvider to switch Rpc
Registrations to the new provider.
Please let me know if you believe this scheme would have
problems.
Thanks,
-Moiz
--
Thx,
-Basheeruddin
|
|
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this node. Does openflow spec define any event
that “old” master would get when its NOT master anymore?
-Abhishek
On Jun 4, 2014, at 1:10 PM, Moiz Raja < moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and it seems that it would be the right
mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that
has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks,
-Moiz
|
|
+1
On Jun 4, 2014, at 2:05 PM, Basheeruddin Ahmed < syedbahm@...> wrote:
I think more than one component might find the change of mastership (or primary) significant - hence they should be able to register a listener on Clustering Service -- which would notify the change in mastership. Its upto the component
how to handle the change in mastership event.
thx
-Basheeruddin
On 06/04/2014 01:10 PM, Moiz Raja wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and it seems that it would be the right
mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that
has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks,
-Moiz
--
Thx,
-Basheeruddin
|
|
Basheer,
In general I agree that components should register with ClusteringService to find if a member is up or down. The subtle difference I see is that we need some component that is "shard aware" to make that decision and I do not think openflowplugin should
be that "shard aware" component and so I find introducing/using an RPC to be a better choice.
-Moiz
toggle quoted message
Show quoted text
From: Basheeruddin Ahmed (syedbahm)
Sent: Wednesday, June 04, 2014 2:05 PM
To: Moiz Raja (moraja); openflowplugin-dev@...; openflowjava-dev@...
Cc: Abhishek Kumar (abhishk2)
Subject: Re: Clustering impact on openflowplugin
I think more than one component might find the change of mastership (or primary) significant - hence they should be able to register a listener on Clustering Service -- which would notify the change in mastership. Its upto the component
how to handle the change in mastership event.
thx
-Basheeruddin
On 06/04/2014 01:10 PM, Moiz Raja wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and it seems
that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that
has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks,
-Moiz
--
Thx,
-Basheeruddin
|
|
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously the downed node will not be able to unregister the rpcs and not be able to handle a "role change"
notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master from a Slave controller is that slaves do not receive any asynchronous
messages from the switch.
-Moiz
toggle quoted message
Show quoted text
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc: openflowplugin-dev@...; openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would
be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this node. Does openflow spec define any event
that “old” master would get when its NOT master anymore?
-Abhishek
On Jun 4, 2014, at 1:10 PM, Moiz Raja < moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and
it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that
has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks,
-Moiz
|
|
Yes, but that falls into routed rpc clustering component. Openflow plugin need not worry.
On Jun 4, 2014, at 2:35 PM, Moiz Raja (moraja) < moraja@...> wrote:
toggle quoted message
Show quoted text
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously the downed node will not be able to unregister the rpcs and not be able to handle a "role change"
notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master from a Slave controller is that slaves do not receive any asynchronous
messages from the switch.
-Moiz
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc: openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would
be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node. OFPlugin folks -
please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this node. Does openflow spec define any event
that “old” master would get when its NOT master anymore?
-Abhishek
On Jun 4, 2014, at 1:10 PM, Moiz Raja < moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller
and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that
has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration
API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks,
-Moiz
|
|
What does member indicate here?
-Prasanna
toggle quoted message
Show quoted text
From: openflowplugin-dev-bounces@... [mailto:openflowplugin-dev-bounces@...]
On Behalf Of Moiz Raja (moraja)
Sent: Thursday, June 05, 2014 3:05 AM
To: Abhishek Kumar (abhishk2)
Cc: openflowjava-dev@...; openflowplugin-dev@...
Subject: Re: [openflowplugin-dev] Clustering impact on openflowplugin
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc: openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
|
|
toggle quoted message
Show quoted text
From: Prasanna Huddar [prasanna.huddar@...]
Sent: Wednesday, June 04, 2014 10:29 PM
To: Moiz Raja (moraja); Abhishek Kumar (abhishk2)
Cc: openflowjava-dev@...; openflowplugin-dev@...
Subject: RE: Clustering impact on openflowplugin
What does member indicate here?
-Prasanna
From: openflowplugin-dev-bounces@... [mailto:openflowplugin-dev-bounces@...]
On Behalf Of Moiz Raja (moraja)
Sent: Thursday, June 05, 2014 3:05 AM
To: Abhishek Kumar (abhishk2)
Cc: openflowjava-dev@...; openflowplugin-dev@...
Subject: Re: [openflowplugin-dev] Clustering impact on openflowplugin
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc:
openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
|
|
Wait.
This should all be a performance optimization, right? That is if any node can access any MD-SAL data element, it's just it might be slow if it is remote. Similarly, any node can make any RPC call, it just might be slow if it's remote.
We might want to try to co-locate the data for a node on the same instance that has the connection (and thus handles the RPCs) for that node, but it shouldn't affect correctness. Am I wrong?
--Colin
On Thu, Jun 5, 2014 at 12:47 PM, Moiz Raja (moraja) <moraja@...> wrote:
A cluster member.
What does member indicate here?
-Prasanna
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc:
openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
|
|
Not wrong. It is a performance optimization only.
-Moiz
toggle quoted message
Show quoted text
From: Colin Dixon [colin@...]
Sent: Thursday, June 05, 2014 11:12 AM
To: Moiz Raja (moraja)
Cc: Prasanna Huddar; Abhishek Kumar (abhishk2); openflowjava-dev@...; openflowplugin-dev@...
Subject: Re: [openflowjava-dev] Clustering impact on openflowplugin
Wait.
This should all be a performance optimization, right? That is if any node can access any MD-SAL data element, it's just it might be slow if it is remote. Similarly, any node can make any RPC call, it just might be slow if it's remote.
We might want to try to co-locate the data for a node on the same instance that has the connection (and thus handles the RPCs) for that node, but it shouldn't affect correctness.
Am I wrong?
--Colin
|
|
So, really, we'd like for there to be some way to figure out what controller instance is currently attached to a given device so that we could give it's data affinity to that device. Does the OpenFlow plugin expose the information about which switches are attached to which instances?
--Colin
On Thu, Jun 5, 2014 at 1:22 PM, Moiz Raja (moraja) <moraja@...> wrote:
Not wrong. It is a performance optimization only.
-Moiz
Wait.
This should all be a performance optimization, right? That is if any node can access any MD-SAL data element, it's just it might be slow if it is remote. Similarly, any node can make any RPC call, it just might be slow if it's remote.
We might want to try to co-locate the data for a node on the same instance that has the connection (and thus handles the RPCs) for that node, but it shouldn't affect correctness.
Am I wrong?
--Colin
|
|
Moiz,
Couple of weeks ago Chris Price and I were discussing the implementation along the lines you are talking about. Master-slave - one of the controller cluster members is a master for a particular switch. The others indicate via set-async configuration that they are not interested in the async messages like packet-ins. Also the OpenFlow switch will likely be connecting to multiple controllers (from the cluster) - whichever controller gets connected first - will be the master for that switch. We assumed that there would be an arbiter which would decide which of the controller instance is a master. So we would need the OpenFlow plugin to let know the arbiter when handshake with a switch is completed. Then the arbiter can call this makeMaster method.
While this (master-slave - switches configured with multiple controllers IP addresses) is one possible implementation - another possible implementation could be the switches actually have a single IP address for the controller (cluster) and connect to that IP address. I assume we are not thinking of that model for clustering - correct?
- Abhijit
On Wed, Jun 4, 2014 at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
Thanks, -Moiz
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
|
|
Currently we don't have multi controller support in the OpenFlow plugin - so does not expose the information which switches are attached to which instance.
So, really, we'd like for there to be some way to figure out what controller instance is currently attached to a given device so that we could give it's data affinity to that device. Does the OpenFlow plugin expose the information about which switches are attached to which instances?
--Colin
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
|
|
Abhijit,
The option I was thinking of using was the one where we had an arbiter decide which controller would be master. How it decides could be based on various possible options like (a) is this controller colocated with inventory/topology (b) Does this controller
have the capacity to handle another openflow connection and so on.
The other option that you proposed I wasn't thinking about but it is an interesting option. I am assuming you just want to use a VIP for the controller nodes and point all switches to that - am I right? In this case when a controller that was connected
to a set of switches goes down and the switches reconnect to another controller is there anything different that the openflowplugin would need to do?
-Moiz
toggle quoted message
Show quoted text
From: Abhijit Kumbhare [abhijitkoss@...]
Sent: Thursday, June 05, 2014 3:57 PM
To: Moiz Raja (moraja)
Cc: openflowplugin-dev@...; openflowjava-dev
Subject: Re: [openflowjava-dev] Clustering impact on openflowplugin
Moiz,
Couple of weeks ago Chris Price and I were discussing the implementation along the lines you are talking about. Master-slave - one of the controller cluster members is a master for a particular switch. The others indicate via set-async configuration that
they are not interested in the async messages like packet-ins. Also the OpenFlow switch will likely be connecting to multiple controllers (from the cluster) - whichever controller gets connected first - will be the master for that switch. We assumed that there
would be an arbiter which would decide which of the controller instance is a master. So we would need the OpenFlow plugin to let know the arbiter when handshake with a switch is completed. Then the arbiter can call this makeMaster method.
While this (master-slave - switches configured with multiple controllers IP addresses) is one possible implementation - another possible implementation could be the switches actually have a single IP address for the controller (cluster) and connect to
that IP address. I assume we are not thinking of that model for clustering - correct?
- Abhijit
|
|
Kamal Rameshan (kramesha) <kramesha@...>
Hi
For multi controller support, the open flow plugin would need an arbiter component.
Please let know if any questions/concerns
-Kamal
toggle quoted message
Show quoted text
What does member indicate here?
-Prasanna
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc:
openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
|
|
Great. Why don't you talk about it a bit tomorrow?
On Mon, Jul 21, 2014 at 3:24 PM, Kamal Rameshan (kramesha) <kramesha@...> wrote:
Hi
For multi controller support, the open flow plugin would need an arbiter component.
Please let know if any questions/concerns
-Kamal
A cluster member.
What does member indicate here?
-Prasanna
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc:
openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
|
|
Kamal Rameshan (kramesha) <kramesha@...>
Yeah sure. Will present the high level design tomorrow.
-Kamal
Great. Why don't you talk about it a bit tomorrow?
toggle quoted message
Show quoted text
Hi
For multi controller support, the open flow plugin would need an arbiter component.
Please let know if any questions/concerns
-Kamal
A cluster member.
What does member indicate here?
-Prasanna
Abhishek,
In the case where a member crashes rpcs will not get unregistered with the system. I think we need to handle that scenario as well - in this scenario obviously
the downed node will not be able to unregister the rpcs and not be able to handle a "role change" notification from the switch - which as per the OpenFlow Spec would not be sent to the slave controller anyway. One of the things which distinguishes a OF Master
from a Slave controller is that slaves do not receive any asynchronous messages from the switch.
From: Abhishek Kumar (abhishk2)
Sent: Wednesday, June 04, 2014 2:08 PM
To: Moiz Raja (moraja)
Cc:
openflowplugin-dev@...;
openflowjava-dev@...; Basheeruddin Ahmed (syedbahm)
Subject: Re: Clustering impact on openflowplugin
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
When a new node is discovered by the open flow plugin, it does routed rpc registration for this node with broker. When connection with new node is lost, the rpcs are unregistered for this node.
OFPlugin folks - please correct me if I am wrong.
With above in place, when a controller is “made” master for a node, the OFPlugin on that controller can register for routed rpcs for this node. And similarly, the old master needs to unregister the rpcs for this
node. Does openflow spec define any event that “old” master would get when its NOT master anymore?
On Jun 4, 2014, at 1:10 PM, Moiz Raja <moraja@...> wrote:
The Clustering design in Helium is based on a Master-Slave model where all writes essentially go to the primary replica. In looking at the OF 1.3 Spec it appears that OF also supports the notion of Master-Slave
controller and it seems that it would be the right mode of operation for us to use.
To properly support Master-Slave mode I believe the openflowplugin needs to do the following,
a. Support a global RPC method to make a specific controller the master (makeMaster). This method will be invoked by an external component. Since the clustering scheme is based on data shards we could come
up with a simple rule that a member that has the inventory shard will also use the openflowplugin. When the external component notices that the primary shard for inventory has changed to the current cluster member then it will invoke the makeMaster RPC on
the local openflowplugin.
b. The makeMaster implementation would need to send a message to the switch requesting that it’s role be set to master
c. The makeMaster implementation would also need to re-register all routed Rpc’s that were registered by the previous master. I haven’t thought through exactly how the new master would discover those Rpc’s - but
one way would be enhance the Rpc registration API to include some meta information which could be used to query for those Rpc registrations. Another option could be enhance the RpcProvider to switch Rpc Registrations to the new provider.
Please let me know if you believe this scheme would have problems.
_______________________________________________
openflowjava-dev mailing list
openflowjava-dev@...
https://lists.opendaylight.org/mailman/listinfo/openflowjava-dev
|
|
