Date   

Silicon GA release approval

Daniel de la Rosa
 

Dear TSC

Now that CSIT check has been completed, please proceed to approve Silicon GA release


Thanks

Daniel de la Rosa
ODL Release Manager


Re: [integration-dev] Decommision Ubuntu 16.04 images on ODL Jenkins and update to Ubuntu 18.04 images on releng/builder repo.

Luis Gomez
 

Sorry this is confusing:

- Are all the mininet VMs converging to this one?

"ZZCI - Ubuntu 18.04 - mininet-ovs-25 - x86_64 - 20200813-035525.790" (Label: ubuntu1804-mininet-ovs-28-1c-4g)

- If so, why it says ovs-25 in the name but ovs-28 in the label?

BR/Luis


On Mar 27, 2021, at 5:59 PM, Anil Belur <abelur@...> wrote:

Please review CR to remove Ubuntu 16.04 packer jobs:


Cheers,
Anil

On Fri, Mar 26, 2021 at 11:51 AM Anil Belur via lists.opendaylight.org <abelur=linuxfoundation.org@...> wrote:
Hello all,

Ubuntu 16.04 is going to be EOL by April 2021 and these images would no longer be supported on our infrastructure to receive updates/new builds. The version of python 2.7 that is available on Ubuntu 16.04 is already EOL and does not works with some of the tox/verify jobs. Therefore, it's required to update existing jobs to use the Ubuntu 18.04 image and decommission Ubuntu 16.04 images.

The below Ubuntu 16.04 images are in use in the releng/builder repo CSIT jobs:
"ZZCI - Ubuntu 16.04 - mininet-ovs-25 - x86_64 - 20200813-044931.688" (Label: ubuntu1604-mininet-ovs-25-1c-4g)
"ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - 20200813-051640.743" (Label: ubuntu1604-mininet-ovs-28-1c-4g)

Which needs to be updated to:
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-4c-4g)
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-2c-2g)
"ZZCI - Ubuntu 18.04 - mininet-ovs-25 - x86_64 - 20200813-035525.790" (Label: ubuntu1804-mininet-ovs-28-1c-4g)

Note: To test the CSIT jobs with the Ubuntu 18.04 images, update the image labels on the JJB/project before pushing the job to ODL Jenkins sandbox. 


Regards,
Anil Belur





Re: Silicon General Release CSIT check

Robert Varga
 

On 28/03/2021 21:47, Luis Gomez wrote:
Looking at past runs of MRI CSIT, only the cluster netty replicate test
fails consistently:

https://jenkins.opendaylight.org/releng/job/mdsal-csit-3node-netty-replicate-only-silicon/10/
<https://jenkins.opendaylight.org/releng/job/mdsal-csit-3node-netty-replicate-only-silicon/10/>

*16:41:52* 2021-03-20T23:39:15,618 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Constructor with 0 arguments not found. Component will fail.
*16:41:52* 2021-03-20T23:39:15,619 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Error during instantiation of the implementation object: Constructor not found.
*16:41:52* 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Constructor with 0 arguments not found. Component will fail.
*16:41:52* 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Error during instantiation of the implementation object: Constructor not found.


Not sure if this is something we have to take a look.
I think we can safely punt this to SR1.

Regards,
Robert


Re: Silicon General Release CSIT check

Luis Gomez
 

I already added some comments to both failing CSIT, nothing looks critical or blocking the release.

Now, since we already have a good amount of MRI CSIT, going forward I would recommend to manual trigger this test for our release candidates:


Looking at past runs of MRI CSIT, only the cluster netty replicate test fails consistently:


16:41:52 2021-03-20T23:39:15,618 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink             | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Constructor with 0 arguments not found. Component will fail.
16:41:52 2021-03-20T23:39:15,619 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSink             | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSink(45)] : Error during instantiation of the implementation object: Constructor not found.
16:41:52 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource           | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Constructor with 0 arguments not found. Component will fail.
16:41:52 2021-03-20T23:39:15,621 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-20 | NettyReplicationSource           | 265 - org.opendaylight.mdsal.replicate-netty - 7.0.5 | bundle org.opendaylight.mdsal.replicate-netty:7.0.5 (265)[org.opendaylight.mdsal.replicate.netty.NettyReplicationSource(46)] : Error during instantiation of the implementation object: Constructor not found.

Not sure if this is something we have to take a look.

BR/Luis



On Mar 26, 2021, at 9:29 PM, Daniel de la Rosa <ddelarosa0707@...> wrote:

Great, thanks for the update @Anil Belur  I have picked Silicon AR#236 and here is the update CSIT spreadsheet

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing

It looks like we only have one issue with BGPCEP and JSONRPC. The rest (GENIUS, LISP) can be ignored... Richard, are you still supporting JSONRPC? Rangan, can you check the BGP issue.

Thanks




On Thu, Mar 25, 2021 at 6:01 PM Anil Belur <abelur@...> wrote:

On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert











Re: [integration-dev] Decommision Ubuntu 16.04 images on ODL Jenkins and update to Ubuntu 18.04 images on releng/builder repo.

Anil Belur
 

Please review CR to remove Ubuntu 16.04 packer jobs:


Cheers,
Anil

On Fri, Mar 26, 2021 at 11:51 AM Anil Belur via lists.opendaylight.org <abelur=linuxfoundation.org@...> wrote:
Hello all,

Ubuntu 16.04 is going to be EOL by April 2021 and these images would no longer be supported on our infrastructure to receive updates/new builds. The version of python 2.7 that is available on Ubuntu 16.04 is already EOL and does not works with some of the tox/verify jobs. Therefore, it's required to update existing jobs to use the Ubuntu 18.04 image and decommission Ubuntu 16.04 images.

The below Ubuntu 16.04 images are in use in the releng/builder repo CSIT jobs:
"ZZCI - Ubuntu 16.04 - mininet-ovs-25 - x86_64 - 20200813-044931.688" (Label: ubuntu1604-mininet-ovs-25-1c-4g)
"ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - 20200813-051640.743" (Label: ubuntu1604-mininet-ovs-28-1c-4g)

Which needs to be updated to:
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-4c-4g)
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-2c-2g)
"ZZCI - Ubuntu 18.04 - mininet-ovs-25 - x86_64 - 20200813-035525.790" (Label: ubuntu1804-mininet-ovs-28-1c-4g)

Note: To test the CSIT jobs with the Ubuntu 18.04 images, update the image labels on the JJB/project before pushing the job to ODL Jenkins sandbox. 


Regards,
Anil Belur




$PROJECT Termination review (based on inactivity)

Anil Belur
 

Hello TSC team:

As per section TSC Charter 2.3.5 [1.], the TSC is initiating a termination review for lack of activity (no commits in the last 18 months).

The list of ODL $PROJECT's that are candidates for archival, with the commit date for the last commit.
1. usc:
Total 3674 (delta 0), reused 3674 (delta 0)
* master - 56eca9d - Merge "Remove obsolete Maven Site configuration" - A H (2 years, 11 months ago)
2. packetcable:
* master - 71996a6 - Bump to odlparent 3.1.0 and yangtools 2.0.3 - Stephen Kitt (3 years ago)
3. opflex:
* master - 46b19e9 - Fix issues with config directory watching - Rob Adams (3 years, 1 month ago)

Once the review is approved by the TSC:
1. All jobs for the $PROJECT would be purged from Jenkins releng/builder repo.
2. The $PROJECT will be set to an inactive state on Gerrit.
3. The $PROJECT will be renamed/prefixed as "archived-$PROJECT" on Github/Gerrit.  

Please add if I missed any projects, refer to the logs attached. 

[1.] https://www.opendaylight.org/about/lifecycle-releases

Regards,
Anil


Re: [E] Re: Release Interview

Daniel de la Rosa
 

I'll try to join as well. Adding TSC mailing list as requested

Thanks

On Fri, Mar 26, 2021 at 10:31 AM <guillaume.lambert@...> wrote:

Hi Casey

 

I am glad we agree to say we need to refine this process.

I can join during the Monday TWS slot. And I’ll do my best to compile something before.

Though, I am pretty sure we won’t have every input by this date.

 

Best Regards

Guillaume

 

 

De : Casey Cain [mailto:ccain@...]
Envoyé : jeudi 25 mars 2021 19:09
À : navid.ghazisaidi@...
Cc : Robert Varga; Daniel de la Rosa; Luis Gomez; LAMBERT Guillaume TGI/OLN; Brandon Wick
Objet : Re: [E] Re: Release Interview

 

After today's discussion on building the release marking info, can we sync at the next TWS meeting?

Would we be able to have all of the necessary pages up to date at that point and then we can focus on the interview?

 

I agree that we need to refine our reporting process to ensure that it is built into the release, but for Silicon, this manual process will have to do with our timeframe. 

 

Best,

Casey Cain

Technical Program Manager / Community Architect

Linux Foundation

_________________

IRC: CaseyLF

WeChat: okaru6

Voice: +1.408.641.0193

 

 

On Thu, Mar 25, 2021 at 9:18 AM <navid.ghazisaidi@...> wrote:

Thanks Robert!

I was using this page (https://docs.opendaylight.org/projects/integration-distribution/en/latest/platform-versions.html) which just shows the targeted release number but as you mentioned it didn't cover all previous versions. How about other projects? If you agree we should start mapping them to the main release version (in this case silicon)? I remember we discussed this before but can you help me recall the reason why we can't use the main release name for MRI projects?

Thanks,
Navid


On 3/25/21, 3:24 AM, "Robert Varga" <nite@...> wrote:

    On 25/03/2021 06:14, navid.ghazisaidi@... wrote:
    > All,
    >
    > 
    >
    > I just updated the filters of
    > https://wiki.opendaylight.org/display/ODL/Silicon+Release
    > <https://wiki.opendaylight.org/display/ODL/Silicon+Release> to pick up
    > other tickets for netconf, etc.

    Thanks.

    > Note that for some project (e.g., ODL parent, YANG tools, and
    > infrautils) I don’t see any results, not sure if there is any jira
    > ticket that I can’t capture it or there is nothing…

    The reason for that is that the filter needs to include *all* versions
    of MRI projects as integrated into int/dist at the time of release, i.e.
    the filter for odlparent is not 'is 8.1.1', but rather (as of this
    writing) 'in [8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1]'.

    For obvious reasons I am not willing to maintain the list of versions
    anywhere but a git repository.

    Regards,
    Robert


_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


Calling all PTL.. Update Silicon page

Daniel de la Rosa
 

Hello ODL PTL

Please update Silicon release page with any highlights for the Marketing release



It seems that AAA, BGPCEP, JSONRPC and TransportPCE were the only with new features and/or improvements, so please confirm ASAP





Thanks


Re: Silicon General Release CSIT check

Daniel de la Rosa
 

Great, thanks for the update @Anil Belur  I have picked Silicon AR#236 and here is the update CSIT spreadsheet

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing

It looks like we only have one issue with BGPCEP and JSONRPC. The rest (GENIUS, LISP) can be ignored... Richard, are you still supporting JSONRPC? Rangan, can you check the BGP issue.

Thanks




On Thu, Mar 25, 2021 at 6:01 PM Anil Belur <abelur@...> wrote:

On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert










Re: Release Interview

Anil Belur
 



On Thu, Mar 25, 2021 at 11:09 PM Robert Varga <nite@...> wrote:
Hello everyone,

I would also like to raise two concerns:

1) this particular conversation is point-to-point and not on mailing
list. I do not see *any* reason for it to be private, so can everyone
here *default* to mailing lists, please?

2) we are slipping more and more into *only* using meetings. There is
very little in terms of asynchronous (and public) communication going on
-- and very quickly 'on meetings' is becoming the only place where any
work is being done.

If you look at LFX, the second trend is clearly visible in graphs when
you look at on which days things tend to happen.

I am not sure what 'interview' is meant to mean here.

Honestly, as the guy who contributed 53% of changes in past 6 months and
83.5% of changes in last 90 days, I just do not have the cycles to
edit/maintain wiki pages -- this is effectively creating reports and as
such it *must* be automated. Any and all maintenance must be a natural
part of the development workflow -- i.e. things like 'which versions are
in Silicon MRI' absolutely must have a single place in Gerrit and that
place needs to be updated as part of the version bump process -- just
like
https://docs.opendaylight.org/projects/integration-distribution/en/latest/platform-versions.html
is.

I have raised the issue of our release notes templates being unusable to
pretty much everyone at least three times over the course of past three
years, yet nothing has changed.

They are useless to developers, simply because things are not tied to
anything developer-tangible. It is unusable to marketing, because it
deals with 'features' and similar technical constructs. It is useless to
our end users, as it is not tied to use cases.

Each and every single release we go through the same exercise and basics
do not change.

Per-project release notes should look something like:
https://www.eclipse.org/xtend/releasenotes.html#/releasenotes/2021/03/02/version-2-25-0
-- note how pretty much everything in that is easily scripted and automated.

The overall SimRel notes should look something like:
https://www.eclipse.org/eclipse/news/4.19/

Both of those should be created in a collaborative and open fashion and
ultimately hosted in a git repository (probably docs).

As for timing, please understand that these documents come to life at
SimRel start and *NOT* just before SimRel is released -- they need to be
*finalized* in the few weeks between code freeze and release
announcement. What that means in concrete terms is that for Silicon
these should have been started on 2020-09-17 at the latest -- and
preferably just after 2020-08-03.

Regards,
Robert

Hello Robert: 

I agree with the way release notes are managed for ODL $project might need to be changed, the versions number by themselves are meaningless without more information. 
For a while for lf-releng/global-jjb and releng/lftools repos we are managing the release notes with reno.

Examples: 

Maybe ODL projects should move to using this tool or something similar?

Cheers,
Anil


Re: [E] RE: Jira tickets fields

Anil Belur
 

+1, It would also be good if these fields (description) can have some kind of default templates, so its easier for users to raise any issue and provide comprehensive information in the ticket. reducing the challenge-response. 

On Thu, Mar 25, 2021 at 2:09 AM navid.ghazisaidi via lists.opendaylight.org <navid.ghazisaidi=verizon.com@...> wrote:

I forgot that, thanks for reminding me, Guillaume!

 

Should we put it under https://docs.opendaylight.org/en/latest/contributor-guides/newcomers-guide.html ?

 

Thanks,

Navid

 

 

From: "guillaume.lambert@..." <guillaume.lambert@...>
Date: Wednesday, March 24, 2021 at 3:28 AM
To: Navid <navid.ghazisaidi@...>, "ODL Docs (documentation@...)" <documentation@...>, "TSC@..." <TSC@...>
Cc: "Casey Cain (ccain@...)" <ccain@...>, Daniel de la Rosa <ddelarosa0707@...>
Subject: [E] RE: Jira tickets fields

 

Thanks Navid

 

For information, most of best practices have now been migrated to the documentation contributor guides

at https://docs.opendaylight.org/en/latest/contributor-guides/index.html

 

Guillaume

 

 

 

De : TSC@... [mailto:TSC@...] De la part de navid.ghazisaidi via lists.opendaylight.org
Envoyé : mardi 23 mars 2021 21:08
À : ODL Docs (documentation@...); TSC@...
Cc : Casey Cain (ccain@...); Daniel de la Rosa
Objet : [OpenDaylight TSC] Jira tickets fields

 

Hi All,

 

Here is the list of fields I believe we should make them mandatory in any jira ticket. I have added description with some example values for each field. As mentioned in our last TSC call, this may help automating the project release tracking and release notes publication.

 

please let me know what your thoughts are.

 

We should also find a good place in wiki to publish it. Maybe under Best Practices (https://wiki-archive.opendaylight.org/view/BestPractices)?

 

Thanks,

Navid

 

 

Project:

description: project name

values: aaa/bgpcep/controller/daexim/distribution/infrautils/jsonrpc/lispflowmapping/netconf/netvirt/openflowplugin/ovsdb/serviceutils/tpce

 

Priority:

description: the importance of ticket

values: Highest, High, Medium, Low, Lowest

 

Type:

description: issue type

values: Bug/New Feature/Task/Improvement/Deprecate

 

Status:

description: the status of ticket

values: Open/In Progress/In Review/Confirmed/Verified/Resolved/Closed/Done/Open

 

affectedVersion:

description: the version that the issue was seen (it’s only applicable to bugs)

values: aluminum/silicon/etc.; “None” to be used for anything other than “Bug”

 

fixVersion:

description: the version that includes the bug fix, new feature, behavior/feature change (improvement), or deprecated feature (the first time that the feature is being disappeared)

values: aluminum/silicon/etc.

 

 

_________________________________________________________________________________________________________________________
 
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
 
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.




Re: ODL Helm chart

Anil Belur
 

Hello Luis: 

ONAP has a few jobs that publish helm charts onto Nexus, and we have some scripts in place that can be reused. 

PS let me know if these bits can be reused. I can setup time to discuss the details of usecase for ODL
and get an understanding of which of the ODL $projects would require helm charts.

Regards,
Anil


On Fri, Mar 26, 2021 at 4:51 AM Luis Gomez <ecelgp@...> wrote:
As I brought in today's TSC meeting, any decent IT app nowadays has a helm chart to quickly install in K8s.

Looking at some open source example:

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

It seems like we need to hold some helm definitions in our code repo for folks to perform:

helm repo add odl https://opendaylight.github.io/helm-charts

Also we need to publish docker containers somewhere so they can be used in a K8s Deployment file like this:

    spec:
      containers:
      - name: odl
        image: quay.io/opendaylight/netconf:v0.13.0
...

I am not an expert in K8s so let me know if I miss anything.

Anil, do you see any problem in existing ODL infra to achieve the above?

BR/Luis


Re: Decommision Ubuntu 16.04 images on ODL Jenkins and update to Ubuntu 18.04 images on releng/builder repo.

Anil Belur
 

Hello all,

Ubuntu 16.04 is going to be EOL by April 2021 and these images would no longer be supported on our infrastructure to receive updates/new builds. The version of python 2.7 that is available on Ubuntu 16.04 is already EOL and does not works with some of the tox/verify jobs. Therefore, it's required to update existing jobs to use the Ubuntu 18.04 image and decommission Ubuntu 16.04 images.

The below Ubuntu 16.04 images are in use in the releng/builder repo CSIT jobs:
"ZZCI - Ubuntu 16.04 - mininet-ovs-25 - x86_64 - 20200813-044931.688" (Label: ubuntu1604-mininet-ovs-25-1c-4g)
"ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - 20200813-051640.743" (Label: ubuntu1604-mininet-ovs-28-1c-4g)

Which needs to be updated to:
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-4c-4g)
"ZZCI - Ubuntu 18.04 - docker - x86_64 - 20200901-040308.994" (Label: ubuntu1804-docker-2c-2g)
"ZZCI - Ubuntu 18.04 - mininet-ovs-25 - x86_64 - 20200813-035525.790" (Label: ubuntu1804-mininet-ovs-28-1c-4g)

Note: To test the CSIT jobs with the Ubuntu 18.04 images, update the image labels on the JJB/project before pushing the job to ODL Jenkins sandbox. 


Regards,
Anil Belur


Re: Silicon General Release CSIT check

Anil Belur
 


On Wed, Mar 24, 2021 at 6:08 AM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis





Hello team:

The change in [3.] is no longer required as it's already addressed in releng/global-jjb CR (https://gerrit.linuxfoundation.org/infra/c/releng/global-jjb/+/67014).
The CR was waiting on a review/merge on the CR 67014 before proceeding with the releng/global-jjb release. 

Now CR 67014 is merged and releng/global-jjb v0.62.0 is released. Please merge releng/builder CR for the global-jjb 

Cheers,
Anil

 
On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert










Re: [E] Re: Release Interview

Robert Varga
 

On 25/03/2021 17:18, navid.ghazisaidi@... wrote:
Thanks Robert!

I was using this page (https://docs.opendaylight.org/projects/integration-distribution/en/latest/platform-versions.html) which just shows the targeted release number but as you mentioned it didn't cover all previous versions.
Yes, because that page is end-user collateral for Simultaneous Release.
As such, it is updated during each MRI bump. For example in Silicon, the
first update was here:
https://git.opendaylight.org/gerrit/c/integration/distribution/+/92972/4/docs/platform-versions.rst
as part of https://git.opendaylight.org/gerrit/q/topic:"silicon-mri",
just as
https://git.opendaylight.org/gerrit/c/docs/+/92934/3/docs/release-notes/upgrade-process.rst
was.

This sort of thing happens in every release between "Release Integrated
Deadline" and "Version Bump Checkpoint".

How about other projects? If you agree we should start mapping them to
the main release version (in this case silicon)? I remember we discussed
this before but can you help me recall the reason why we can't use the
main release name for MRI projects?

Quick answer to that is: because MRI projects are released way before
Simultaneous Release. As an example: Silicon is not released yet, but
yangtools-6.0.0 was released on October 8th, 2020.

The entire idea of a monolithic autorelease-based Simultaneous Release
(where you build all the projects and release them at the same time)
does not actually have a basis in our governance and has been introduced
as a reaction to, and a stop-gap solution for, how badly our very first
release (Hydrogen) went. That in turn was a result of us having been
SNAPSHOT-integrated across all projects and our pom.xml being a horrible
mess. Neither of those is true anymore.

Regards,
Robert


Thanks,
Navid


On 3/25/21, 3:24 AM, "Robert Varga" <nite@...> wrote:

On 25/03/2021 06:14, navid.ghazisaidi@... wrote:
> All,
>
>
>
> I just updated the filters of
> https://wiki.opendaylight.org/display/ODL/Silicon+Release
> <https://wiki.opendaylight.org/display/ODL/Silicon+Release> to pick up
> other tickets for netconf, etc.

Thanks.

> Note that for some project (e.g., ODL parent, YANG tools, and
> infrautils) I don’t see any results, not sure if there is any jira
> ticket that I can’t capture it or there is nothing…

The reason for that is that the filter needs to include *all* versions
of MRI projects as integrated into int/dist at the time of release, i.e.
the filter for odlparent is not 'is 8.1.1', but rather (as of this
writing) 'in [8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.1.0, 8.1.1]'.

For obvious reasons I am not willing to maintain the list of versions
anywhere but a git repository.

Regards,
Robert



Re: [ODL Discuss] ODL Helm chart

Daniel de la Rosa
 

+ Jeff Hartley


On Thu, Mar 25, 2021 at 11:52 AM Luis Gomez <ecelgp@...> wrote:
As I brought in today's TSC meeting, any decent IT app nowadays has a helm chart to quickly install in K8s.

Looking at some open source example:

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

It seems like we need to hold some helm definitions in our code repo for folks to perform:

helm repo add odl https://opendaylight.github.io/helm-charts

Also we need to publish docker containers somewhere so they can be used in a K8s Deployment file like this:

    spec:
      containers:
      - name: odl
        image: quay.io/opendaylight/netconf:v0.13.0
...

I am not an expert in K8s so let me know if I miss anything.

Anil, do you see any problem in existing ODL infra to achieve the above?

BR/Luis



ODL Helm chart

Luis Gomez
 

As I brought in today's TSC meeting, any decent IT app nowadays has a helm chart to quickly install in K8s.

Looking at some open source example:

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

It seems like we need to hold some helm definitions in our code repo for folks to perform:

helm repo add odl https://opendaylight.github.io/helm-charts

Also we need to publish docker containers somewhere so they can be used in a K8s Deployment file like this:

spec:
containers:
- name: odl
image: quay.io/opendaylight/netconf:v0.13.0
...

I am not an expert in K8s so let me know if I miss anything.

Anil, do you see any problem in existing ODL infra to achieve the above?

BR/Luis


Re: Silicon General Release CSIT check

Luis Gomez
 

Anil, please let us know when global-jib patch is merged as this is blocking CSIT jobs update.

BR/Luis

On Mar 24, 2021, at 8:59 PM, Daniel de la Rosa <ddelarosa0707@...> wrote:

Hello all.. Sounds like we need to wait for @Anil Belur  to work on [2] and [3] to pick the next Silicon AR?  or we can move forward with just the NETCONF fix

Thanks

On Tue, Mar 23, 2021 at 1:08 PM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis


On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert








Re: Release Interview

Robert Varga
 

Hello everyone,

I would also like to raise two concerns:

1) this particular conversation is point-to-point and not on mailing
list. I do not see *any* reason for it to be private, so can everyone
here *default* to mailing lists, please?

2) we are slipping more and more into *only* using meetings. There is
very little in terms of asynchronous (and public) communication going on
-- and very quickly 'on meetings' is becoming the only place where any
work is being done.

If you look at LFX, the second trend is clearly visible in graphs when
you look at on which days things tend to happen.

I am not sure what 'interview' is meant to mean here.

Honestly, as the guy who contributed 53% of changes in past 6 months and
83.5% of changes in last 90 days, I just do not have the cycles to
edit/maintain wiki pages -- this is effectively creating reports and as
such it *must* be automated. Any and all maintenance must be a natural
part of the development workflow -- i.e. things like 'which versions are
in Silicon MRI' absolutely must have a single place in Gerrit and that
place needs to be updated as part of the version bump process -- just
like
https://docs.opendaylight.org/projects/integration-distribution/en/latest/platform-versions.html
is.

I have raised the issue of our release notes templates being unusable to
pretty much everyone at least three times over the course of past three
years, yet nothing has changed.

They are useless to developers, simply because things are not tied to
anything developer-tangible. It is unusable to marketing, because it
deals with 'features' and similar technical constructs. It is useless to
our end users, as it is not tied to use cases.

Each and every single release we go through the same exercise and basics
do not change.

Per-project release notes should look something like:
https://www.eclipse.org/xtend/releasenotes.html#/releasenotes/2021/03/02/version-2-25-0
-- note how pretty much everything in that is easily scripted and automated.

The overall SimRel notes should look something like:
https://www.eclipse.org/eclipse/news/4.19/

Both of those should be created in a collaborative and open fashion and
ultimately hosted in a git repository (probably docs).

As for timing, please understand that these documents come to life at
SimRel start and *NOT* just before SimRel is released -- they need to be
*finalized* in the few weeks between code freeze and release
announcement. What that means in concrete terms is that for Silicon
these should have been started on 2020-09-17 at the latest -- and
preferably just after 2020-08-03.

Regards,
Robert

On 25/03/2021 12:04, guillaume.lambert@... wrote:
Hi all

 

Brandon, Casey, I’d be happy to give you my thought but if you really
need them by this Friday, the best I can do is Friday 10pm Paris Time (=
2pm  LA time).

If Robert wants to make the interview, I am also fine with that as long
as the whole TSC is OK.

 

I share Casey’s concern that this topic risks to absorb the whole TSC
meeting.

Daniel, is that fine if we limit this talk to 10 to review the deadlines
and organize ourselves ?.

 

If we did talk about this marketing message a fortnight ago, some
projects did only have one meeting to react since then.

And we have still pending technical issues to fix before broadcasting
any marketing message.

Silicon Managed projects have not been released or approved yet. SM
projects are still working hard to wrap up.

So I can understand some of them have not filled the wiki yet.

 

As a PTL, I’ve raised the topics several times to the TransportPCE
contributors during our syncs but they are all pretty busy.

AT&T contributors were struggling with a bug to prepare their demo for
the OFC conference and Orange was preparing another demo for MDONS.

During this release, Nokia and Eurecom also proposed additional
interesting contributions that were not planned initially and that we
just integrated.

Thus, we have to rework our message and if it is now ready from the
Orange side,  we are still validating its contents with the other
companies at that moment.

I don’t think it would be fair to not include everyone.

 

Also let’s not mistake haste for speed.

In the past, these people noticed their message was modified by the
marketing and they didn’t have a chance to react to these modifications.
For example in Neon, they told me that they would have liked to change
the term “degree” which has a different meaning in WDM.

If the message was really OK for non-optical expert, it was really
confusing for the optical professionals.

I’d really like that we take the time to talk on how we can avoid this
kind of situation for future releases.

 

Best Regards

Guillaume

 

 

*De :*Daniel de la Rosa [mailto:ddelarosa0707@...]
*Envoyé :* jeudi 25 mars 2021 05:36
*À :* Casey Cain; Robert Varga; Luis Gomez; navid.ghazisaidi@...
*Cc :* LAMBERT Guillaume TGI/OLN; Brandon Wick
*Objet :* Re: Release Interview

 

Indeed we talked about it so let me add Robert so he can let us know why
this wiki page doesn't include NETCONF and other of his projects. Let me
also add Navid and Luis since we all talked about this during a TSC
meeting. 

 

On Wed, Mar 24, 2021 at 11:55 AM Casey Cain <ccain@...
<mailto:ccain@...>> wrote:

We discussed this at the last 2 TSC meetings and even mentioned it a
few times before.  We have an outstanding action item for the
Release Manager and the PTLs to ensure that the information on this
wiki page <https://wiki.opendaylight.org/x/GRsF> is accurate and
complete prior to March 24th.

The purpose of the Interview is to give a chance for the face of the
community, the TSC chair to provide their thoughts on the release
and to highlight important changes.

 

In the past when we tried to have the "Release Interview" with the
entire TSC we have had issues with staying on topic and developing
relevant marketing messaging.  If you think it would be best to
approach Robert to do the release interview, please let me know.

 

Best,

Casey Cain

Technical Program Manager / Community Architect

Linux Foundation

_________________

IRC: CaseyLF

WeChat: okaru6

Voice: +1.408.641.0193

Book a Meeting w/ Casey <http://calendly.com/caseylf>

 

 

On Wed, Mar 24, 2021 at 11:34 AM <guillaume.lambert@...
<mailto:guillaume.lambert@...>> wrote:

No objection.

I let you propose it at the meeting start.

 

*De :*Daniel de la Rosa [mailto:ddelarosa0707@...
<mailto:ddelarosa0707@...>]
*Envoyé :* mercredi 24 mars 2021 15:06
*À :* LAMBERT Guillaume TGI/OLN
*Cc :* Casey Cain; Brandon Wick
*Objet :* Re: Release Interview

 

Hello all. I'm not available either but also, we are going to
need Robert and potentially others since they are more fully
aware of what's new in Silicon. Maybe we can just use the whole
TSC meeting to review this?

 

Thanks

 

On Wed, Mar 24, 2021 at 1:23 AM <guillaume.lambert@...
<mailto:guillaume.lambert@...>> wrote:

Hi all

 

My agenda is already full this Friday afternoon ( 7am PST is
3pm Paris Time this week)

Can it wait for Monday ? Potentially anytime at your
convenience.

 

Best Regards

Guillaume

 

------------------------------------------------------------------------

*De :*Casey Cain [ccain@...
<mailto:ccain@...>]
*Envoyé :* mardi 23 mars 2021 21:32
*À :* LAMBERT Guillaume TGI/OLN; Daniel de la Rosa
*Cc :* Brandon Wick
*Objet :* Release Interview

Hi, Guillaume and Daniel.

 

As we discussed previously there is a need to confirm the
release features of Silicon and set up an interview with
yourself for marketing purposes.  Can we schedule this
interview for this Friday at 7 am PST?

I will also confirm with you on this week's TSC call that
the necessary information has already been collected and
posted to the wiki.

 

Thanks!!

Casey Cain

Technical Program Manager / Community Architect

Linux Foundation

_________________

IRC: CaseyLF

WeChat: okaru6

Voice: +1.408.641.0193

Book a Meeting w/ Casey <http://calendly.com/caseylf>

_________________________________________________________________________________________________________________________

 

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

 

This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________

 

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

 

This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


Re: Silicon General Release CSIT check

Daniel de la Rosa
 

Hello all.. Sounds like we need to wait for @Anil Belur  to work on [2] and [3] to pick the next Silicon AR?  or we can move forward with just the NETCONF fix

Thanks

On Tue, Mar 23, 2021 at 1:08 PM Luis Gomez <ecelgp@...> wrote:
I think you patch [1] fixed the issue:


We can maybe pick next AR build to see all netconf tests passing.

Also we need another patch [2] to move aaa and netconf CSIT to MRI once this issue [3] gets addressed in global-jib.

BR/Luis


On Mar 23, 2021, at 12:03 AM, Robert Varga <nite@...> wrote:

On 19/03/2021 06:44, Daniel de la Rosa wrote:
Hello TSC

We have picked a Silicon AR #228 as release candidate. 

https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-silicon-mvn35-openjdk11/228/>

Here is the CSIT checklist so please review it ASAP

https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing
<https://docs.google.com/spreadsheets/d/1zvD4xgiMlWCgg5ZBvxSRONY_LtLtzzNUeuA06-i7ukc/edit?usp=sharing>

It looks like the netconf test suite is busted:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-silicon/250/robot-plugin/log.html.gz

Suite setup failed:
Artifact "netconf-testtool" in component "netconf" could not be downloaded from https://nexus.opendaylight.org/content/repositories//autorelease-4768/org/opendaylight/netconf/netconf-testtool/1.13.1/netconf-testtool-1.13.1-rest-perf-client.jar

Also suite teardown failed:
Variable '${testtool}' not found.

Since NETCONF is an MRI project in Silicon, the testtool should not be
downloaded from autorelease staging repo but rather from normal release
repo.

I do not know how to achieve, can someone from int/test take a look?

Regards,
Robert






661 - 680 of 14243