On 30/10/2019 12:05, JayaPr via Lists.Opendaylight.Org wrote:

Hi Jamo,

 

There are 5 blockers for netvirt in Sodium-SR1 which are of 2 category:

1. 2 jobs of dcgw –Both the jobs has 3 failures

1 fails for exceptions caught in teardown : broken recently

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/685/JP-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-dcgw-sodium/4/robot-plugin/log_full.html.gz

no exception in sept12 result.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/83/
is the first one to show uptick in failures.

2019-10-06T22:11:07,771 | ERROR | org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.access.control.list.rev160218.access.lists.Acl_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | AsyncDataTreeChangeListenerBase | 274 - org.opendaylight.genius.mdsalutil-api - 0.7.1 | Thread terminated due to uncaught exception: org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.access.control.list.rev160218.access.lists.Acl_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0
java.lang.NullPointerException: null
at org.opendaylight.netvirt.aclservice.listeners.AclEventListener.remove(AclEventListener.java:110) ~[?:?]
at org.opendaylight.netvirt.aclservice.listeners.AclEventListener.remove(AclEventListener.java:50) ~[?:?]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:158) ~[274:org.opendaylight.genius.mdsalutil-api:0.7.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]

NullPointerExceptions Are Always Bugs (TM)

In this particular case I think what we are looking at is List<Ace>
becoming empty through a transaction, prompting disappearance of
AccessListEntries and thus:

List<Ace> aceList = acl.getAccessListEntries().getAce();

results in NPE. The code needs to take the path equivalent to aceList
being empty and this is nothing new:
https://wiki.opendaylight.org/view/Neon_platform_upgrade#DataTree_removes_empty_lists.2C_choices_and_augmentations

Jamo: please file an issue with Netvirt, the fix needs to go to all
maintained branches.

Looking at
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-1cmb-0ctl-0cmp-openstack-rocky-upstream-stateful-itm-direct-tunnels-sodium/93/odl_1/odl1_karaf.log.gz,
there's at least one more:

2019-10-16T16:44:44,433 | ERROR | jobcoordinator-main-task-6 | VpnSubnetRouteHandler | 385 - org.opendaylight.netvirt.vpnmanager-impl - 0.9.1 | SUBNETROUTE: onInterfaceUp: Unable to handle interface up event for port a8a7cd0e-337a-4540-8749-1372ce63f7b2 in subnet 65fe623a-5c49-45ce-a82a-673b7ff99199
java.lang.NullPointerException: null
at org.opendaylight.netvirt.vpnmanager.VpnSubnetRouteHandler.onInterfaceUp(VpnSubnetRouteHandler.java:623) ~[?:?]
at org.opendaylight.netvirt.vpnmanager.SubnetRouteInterfaceStateChangeListener.lambda$add$0(SubnetRouteInterfaceStateChangeListener.java:115) ~[?:?]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$call$2(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:39) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:398) ~[?:?]
at org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$run$0(ClassLoaders.java:26) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.call(ClassLoaders.java:47) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.run(ClassLoaders.java:25) ~[293:org.opendaylight.infrautils.util:1.6.1]
at org.opendaylight.infrautils.utils.ClassLoaders.lambda$wrap$3(ClassLoaders.java:54) ~[293:org.opendaylight.infrautils.util:1.6.1]
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) [?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:?]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:?]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:?]

Regards,
Robert