Hi All,

As presented/discussed in the ODL TSC meeting held on 22^nd Friday 10.30 AM IST, posting this email to highlight the issues on ODL clustering use cases encountered during the performance testing.

Details and configurations as follows:

• Requirement : ODL clustering for high availability (HA) on data distribution
• Env Configuration:

• 3 node k8s Cluster ( 1 master & 3 worker nodes) with 3 ODL instances running on each node
• CPU :  8 Cores
• RAM : 20GB
• Java Heap size : Min – 512MB Max – 16GB
• JDK version : 11
• Kubernetes version : 1.19.1
• Docker version : 20.10.7

• ODL features installed to enable clustering:

• odl-netconf-clustered-topology
• odl-restconf-all

• Device configured : Netconf devices , all devices having same schema(tested with 250 devices)
• Use Case:

• Fail Over/High Availability:

• Expected : In case any of the ODL instance gets down/restarted due to network splits or internal error, other instance in cluster should be available and functional. If the affected instance is having master mount, the other instance who is elected as master by re-election should be able to re-register the devices and resume the operations. Once the affected instance comes up, it should be able to join the cluster as member node and register the slave mounts.
• Observation : When the odl instance which is having the master mount restarts, election happens among the other node in the cluster and elects the new leader. Now the new leader is trying to re-register the master mount but failed at a point due to the termination of the Akka Cluster Singleton Actor. Hence the cluster goes to idle state and failed to assign owner for the device DOM entity. In this case, the configuration of already mounted device/ new mounts will fail.  

• JIRA reference : https://jira.opendaylight.org/browse/CONTROLLER-2035  
• Akka configuration of all the nodes attached. (Increased the gossip-interval time to 5s in akka.conf file to avoid Akka AskTimedOut issue while mounting multiple devices at a time.)

Requesting your support to identify if there is any mis-configurations or any known solution for the issue .

Please let us know if any further information required.

Note : We have tested the single ODL instance without enabling cluster features in K8s cluster. In case of K8s node failure, ODL instance will be re-scheduled in other available K8s node and operations will be resumed.  

Thanks & Regards,

Rohini

Hi All,

As presented/discussed in the ODL TSC meeting held on 22^nd Friday 10.30 AM IST, posting this email to highlight the issues on ODL clustering use cases encountered during the performance testing.

Details and configurations as follows:

• Requirement : ODL clustering for high availability (HA) on data distribution
• Env Configuration:

• 3 node k8s Cluster ( 1 master & 3 worker nodes) with 3 ODL instances running on each node
• CPU :  8 Cores
• RAM : 20GB
• Java Heap size : Min – 512MB Max – 16GB
• JDK version : 11
• Kubernetes version : 1.19.1
• Docker version : 20.10.7

• ODL features installed to enable clustering:

• odl-netconf-clustered-topology
• odl-restconf-all

• Device configured : Netconf devices , all devices having same schema(tested with 250 devices)
• Use Case:

• Fail Over/High Availability:

• Expected : In case any of the ODL instance gets down/restarted due to network splits or internal error, other instance in cluster should be available and functional. If the affected instance is having master mount, the other instance who is elected as master by re-election should be able to re-register the devices and resume the operations. Once the affected instance comes up, it should be able to join the cluster as member node and register the slave mounts.
• Observation : When the odl instance which is having the master mount restarts, election happens among the other node in the cluster and elects the new leader. Now the new leader is trying to re-register the master mount but failed at a point due to the termination of the Akka Cluster Singleton Actor. Hence the cluster goes to idle state and failed to assign owner for the device DOM entity. In this case, the configuration of already mounted device/ new mounts will fail.  

• JIRA reference : https://jira.opendaylight.org/browse/CONTROLLER-2035  
• Akka configuration of all the nodes attached. (Increased the gossip-interval time to 5s in akka.conf file to avoid Akka AskTimedOut issue while mounting multiple devices at a time.)

Requesting your support to identify if there is any mis-configurations or any known solution for the issue .

Please let us know if any further information required.

Note : We have tested the single ODL instance without enabling cluster features in K8s cluster. In case of K8s node failure, ODL instance will be re-scheduled in other available K8s node and operations will be resumed.  

Thanks & Regards,

Rohini