Re: TSC agenda: quick update on tpce issues


Guillaume Lambert
 

Hi

Jakub, thanks a lot for the explanation.
This is coherent with my observation when investigating the problem and also explains, at least partially, the last issue we had.
I confirm it works well when upgrading testtool to Fluorine - that's one of the first thing we've tried.
I was a bit reluctant to do this without more explanation - in my opinion, the best practice is to test new features with fieldproven simulators.
But since we know where the problem comes from, it is fair enough.

FYI, I have also watched the same openEXI/EXIficient compatibility problem when trying to connect our other mock (honeynode) which is based on FD.io honeycomb.
This is logic since honeycomb is based on an older ODL version that was still using openEXI.

For tpce, this means that we cannot merge in Fluorine a part of our functests based on honeynode ( at least for the moment ).
Shweta have reproduced in ATT labs those tests on real equipment to ensure they pass in Fluorine too.
I've undrafted related changes so that you can look at them if needed.

BR
Guillaume

-----Original Message-----
From: Jakub Morvay [mailto:jmorvay@...]
Sent: jeudi 2 août 2018 22:37
To: LAMBERT Guillaume IMT/OLN; Tom Pantelis
Cc: Ariel Adam; netconf-dev; Jakub Toth; Abhijit Kumbhare; Abhijit Kumbhare; RENAIS Olivier IMT/OLN; VACHHANI, SHWETA (sv111y@...); Robert Varga; <tsc@...>; COULIBALY Martial Ext IMT/OLN
Subject: Re: TSC agenda: quick update on tpce issues

Hi all,

There is also another private conversation regarding the issue, but I am going to reply just here and I am going to talk just about the netconf testtool, since I believe other issues were answered by Tom.

I've checked netconf's CSIT jobs and there seems to be an issue connecting netconf testtool. CSIT jobs are using testtool with version 1.3.2. In the logs I can see that there is a problem with decoding the message in EXI format on testool's side. I believe the problem is in the fact that we are using different EXI implementation in Fluorine and different implementation in Nitrogen. In Fluorine we use EXIficient, in Nitrogen (and that is in testtool 1.3.2) it is OpenEXI implementation. I've investigated this little bit further and It seems, that EXIficient produces the different encoding of the same message than OpenEXI does. Testtool with version 1.3.2 uses OpenEXI to decode the message and fails to do so.

I tried this with the Fluorine testool and it works fine. I am able to mount the device successfully. So I think we should use the Fluorine testtool in Fluorine CSIT jobs. Furthermore, I am going to try to mount some other netconf devices, so I can see if there are no EXI decoding issues there. Also we should investigate this further to be sure where the issue is exactly hidden and we should be sure that the bug is on the OpenEXI side and not on the EXIficient side.

So please, use Fluorine testtool for your test setup, this should get you going.

About the NETCONF-522.. I don't believe this is related to anything discussed here. I will reply to the issue directly.

Jakub Morvay


On 08/01/2018 06:25 PM, guillaume.lambert@... wrote:
Thanks for confirming the complete removal of the controller-config mountpoint. We will remove that from our test bench.
Since it was still mentioned in the oxygen and fluorine netconf user guides, I was afraid to have missed something.

About the netconf CLI, I am not sure to understand your point.
We already use the netconf-topology REST API in our functests to mount netconf nodes.
Do you mean the netconf CLI will be removed too ?

About the third issue, I’ve seen Jakub just posted something very similar in Jira NETCONF-552 but with testtool 1.4.3.
https://jira.opendaylight.org/browse/NETCONF-552
I will keep an eye on this.



From: Tom Pantelis [mailto:tompantelis@...]
Sent: mercredi 1 août 2018 15:41
To: LAMBERT Guillaume IMT/OLN
Cc: Ariel Adam; netconf-dev; Jakub Morvay; Jakub Toth; Abhijit
Kumbhare; Abhijit Kumbhare; RENAIS Olivier IMT/OLN; VACHHANI, SHWETA
(sv111y@...); Robert Varga; <tsc@...>;
COULIBALY Martial Ext IMT/OLN
Subject: Re: TSC agenda: quick update on tpce issues


On Wed, Aug 1, 2018 at 7:00 AM, <guillaume.lambert@...<mailto:guillaume.lambert@...>> wrote:
Hi all and thanks a lot for proposing help The first 2 issues are easy
to understand and quite simple to reproduce.
Since oxygen, we do not see anymore the “controller-config” node automatically connected at netconf connector start-up.
Has this feature abandoned ? We had a basic tests its connection, currently commented out. Can we remove it ?
https://git.opendaylight.org/gerrit/#/c/74340/

The controller-config mount point was part of the config system which was previously deprecated in Carbon and is now completely removed.

The second one is the impossibility since Oxygen to use the netconf CLI until a netconf node has been connected with the REST API.
I wonder if this issues is not somehow linked to the previous one…

The netconf device must be mounted by creating a node in the netconf-topology. This mechanism replaced the controller-config mechanism several releases ago although controller-config was kept for a while for backwards compatibility.


The third issue is more tricky, but hopefully fixed now even if it is not clear why.
Here is how to reproduce it.
We are using python –nose + tox to run our functests. All is in the tests folder. You can look at our tox.ini file for details.
In change 74468, we used to mock our openroadm devices with a patched
version of netconf-testtool 1.3.1 (the patch merges config and
operational datastore and is solely for read-only tests) This mock is
notably used by the portmapping functests (cd tests && tox –e portmapping) Tox runs the script build.sh or buildTesttool.sh to download, compile, patch and configure testtool in our tests folder.
With the bump to Fluorine, the old testtool configuration does not work anymore with the controller.
We’ve commented the issue in change 74296 PS 5
https://git.opendaylight.org/gerrit/#/c/74296/5
Martial’s first feeling was a capability / get-schema problem.
But when I looked at the log and snooped the netconf session, it appeared the problem is deeper.
We had a java exception and EXI malformed message errors.
My feeling is that the netconf southbound connector requires now more features support from netconf devices that it used to in Nitrogen or Oxygen.
We upgraded testtool to the 1.5.0-SNAPSHOT in change 74508 https://git.opendaylight.org/gerrit/74508 and this fixed the issue.
But we haven’t fully understood why we got those errors with testtool
1.3.1

The last issue is not yet fixed.
We are now in the process of switching to honeycomb because testtool have a limited support of operational datastore and write operations.
This has already been done successfully in Nitrogen with changes 74182
and 6801
https://git.opendaylight.org/gerrit/74182
https://git.opendaylight.org/gerrit/68021
However, cherry-picks in Oxygen and master reports some unexpected functests failures.
https://git.opendaylight.org/gerrit/#/q/status:open+project:transportp
ce+branch:stable/oxygen+topic:honeynode
We have also observed sometimes unstabilities in the old testtool tox tests suite when bumping to oxygen.
Karaf 4 boot process related issues ?
I am still trying to characterize that more clearly.

Any idea or help is welcome.




BR
Gullaume

From: Tom Pantelis
[mailto:tompantelis@...<mailto:tompantelis@...>]
Sent: mardi 31 juillet 2018 20:21
To: Ariel Adam
Cc: LAMBERT Guillaume IMT/OLN; netconf-dev; Jakub Morvay; Jakub Toth;
Abhijit Kumbhare; Abhijit Kumbhare; RENAIS Olivier IMT/OLN; VACHHANI,
SHWETA (sv111y@...<mailto:sv111y@...>); Robert Varga;
<tsc@...<mailto:tsc@...>>

Subject: Re: TSC agenda: quick update on tpce issues



On Tue, Jul 31, 2018 at 12:09 PM, Ariel Adam <aadam@...<mailto:aadam@...>> wrote:
Guilaume, that is great news :-).
Adding the Netconf team (Jakub, Jakub and Tom who I see are listed).

Guys, how can we help Guilaume and his team to git this through (1 week go to go)?

I'm trying to follow everything here.... Guilaume, so what is the remaining issue(s) re: netconf southbound connector dependencies? Can you provide steps on how to reproduce it?


Thanks.


On Tue, Jul 31, 2018 at 6:53 PM <guillaume.lambert@...<mailto:guillaume.lambert@...>> wrote:
Hi Ariel

Good news.
I just managed to bump our master to Fluorine by repackaging correctly
in tpce the ietf-network draft models removed from MD-SAL Cf
https://git.opendaylight.org/gerrit/#/c/74296/
So we have a chance to participate the formal release.
Jenkins-releng currently failed because we have a minor issue with renderer Junit tests induced by the bump.
Martial or Doha should be able to fix it quickly when they will be back tomorrow.
Those tests have been tested successfully today in Oxygen
https://git.opendaylight.org/gerrit/#/c/74691/

Otherwise, we still have netconf / karaf issues with the netconf southbound connector dependencies.
We tried to bump our mock version without success even if it improved the situation.
https://git.opendaylight.org/gerrit/#/c/74508/
I didn’t see any feedback to my previous email last week about that in the netconf mailing-list.
I will try to precise that tomorrow since the situation evolved.

BR
Guillaume

From: Ariel Adam [mailto:aadam@...<mailto:aadam@...>]
Sent: dimanche 29 juillet 2018 06:50
To: LAMBERT Guillaume IMT/OLN
Cc: Abhijit Kumbhare
(abhijit.kumbhare@...<mailto:abhijit.kumbhare@...>);
Abhijit Kumbhare
(abhijitkoss@...<mailto:abhijitkoss@...>); RENAIS Olivier
IMT/OLN; VACHHANI, SHWETA (sv111y@...<mailto:sv111y@...>)
Subject: Re: TSC agenda: quick update on tpce issues

Sounds good Guillaume.
Given that you said the team are also on vacation from mid August recommend you guys push hard to have time left for fixing last minute problems (which always appear).

Also update us ASAP if your not getting assistance.

Thanks.

On Fri, Jul 27, 2018 at 8:20 PM, <guillaume.lambert@...<mailto:guillaume.lambert@...>> wrote:
Hi

Thanks for your help at yesterday TSC.
As suggested, I've undrafted our work and created some JIRA issues about our current difficulties.
They are referenced under TRNSPRTPCE-29 and TRNSPRTPCE-32 I linked the
related gerrit changes and TRNSPRTPCE-29 to the issues TSC-124 and NETCONF-550 initiated by Robert.
Let me know if they need to be referenced elsewhere.

We will try what Robert proposes and keep everyone up to date, probably at the TWS Monday.

Best regards
Guillaume


-----Original Message-----
From: LAMBERT Guillaume IMT/OLN
Sent: jeudi 26 juillet 2018 10:02
To: Abhijit Kumbhare
(abhijit.kumbhare@...<mailto:abhijit.kumbhare@...>);
Abhijit Kumbhare (abhijitkoss@...<mailto:abhijitkoss@...>)
Cc: Ariel Adam (aadam@...<mailto:aadam@...>)
Subject: TSC agenda: quick update on tpce issues

Hi

Abhijit, we've talked with Ariel and we would like to have a 10min slot to discuss tpce integration issues in Fluorine to day at the TSC meeting.
They are mostly due to the need of repackaging the ietf-network
revision 2015-06-08 models recently dropped in MD-SAL Cf changes 74296
and 74468 https://git.opendaylight.org/gerrit/#/c/74296/
https://git.opendaylight.org/gerrit/#/c/74468/

BR
Guillaume

______________________________________________________________________
___________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
exploites ou copies sans autorisation. Si vous avez recu ce message
par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or
privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


______________________________________________________________________
___________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les
messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or
privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.


______________________________________________________________________
___________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les
messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or
privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.


______________________________________________________________________
___________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc pas etre diffuses,
exploites ou copies sans autorisation. Si vous avez recu ce message
par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or
privileged information that may be protected by law; they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Join TSC@lists.opendaylight.org to automatically receive all group messages.