Date   

Phosphorus GA release approval

Daniel de la Rosa
 

Dear TSC

As you could read from the other email thread, we have can proceed to approve Phosphorus GA release


Thanks

Daniel de la Rosa
ODL Release Manager


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Daniel de la Rosa
 




On Thu, Sep 30, 2021 at 9:12 AM Robert Varga <nite@...> wrote:
On 30/09/2021 17:38, Luis Gomez wrote:
> I think netconf devs got confused on how our CSIT branches work, I do
> not see any CSIT job for phosphorous and therefore I do not think we are
> testing netconf in this release:
>
> https://jenkins.opendaylight.org/releng/view/netconf/
> <https://jenkins.opendaylight.org/releng/view/netconf/>

I think netconf devs would be less confused if you could share some
writeup of how exactly is test execution wired across integration/test
and releng/builder :)

Please note that NETCONF is an MRI project, hence its naming follows its
versioning scheme since Silicon GA.

Since we are after autorelease has branched stable/phosphorus, but
netconf.git has _not_ branched 2.0.x, netconf's master is still hosting
netconf-2.0.x and that in turn is integrated into *both* Phosphorus and
Sulfur.

The same is true for odlparent, mdsal, controller, aaa and bgpcep.
Yangtools is a bit ahead of the curve and has 7.0.x for Phosphorus and
master is hosting yangtools-8.0.0-to-be (and destined for Sulfur).

I hope that paints the current state of git branches enough to
understand where we are.

Currently all testing of MRI projects is still wired on the assumption
that what we are testing int/dist with that particular MRI project. See
bgpcep tests for an example.

That is quite wrong, as MRI projects' CSIT must be able to execute on
whatever artifacts are produced, for example,
{mriproject}-maven-stage-{branch} job.

Tomas has sunk a significant chunk of his time to correct this and we
are still kicking the wheels on that here:

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-maven-mri-stage-master/
. What that job does is build and stage netconf for release (just link
autorelease does for openflowplugin) and then triggers

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-distribution-mri-test-master/
. This job runs the NETCONF CSIT, but not on int/dist's karaf.tar.gz,
but rather on netconf's netconf-karaf.tar.gz.

Which is pretty much what should be happening for all MRI projects, as
that way we get fully-vetted artifacts before we decide to run our
{project}-release-merge job to publish them to Nexus (step done manually
by Anil for autorelease).

Based on all of this, please start saying goodbye to the idea that MRI
projects have CSIT job names bearing SimRel names -- currently that
would mean executing netconf/master CSIT twice (sulfur and phosphorus).

Also, please be patient while we bring all of this long-overlooked task
to fruition.

As for netconf-csit-1node-scale-max-devices-only job, specifically, that
one underwent a rewrite and is still pending stabilization. The test
never really worked as it should have.

As for overall Phosphorus GA release, I just do not see the point
holding it up anymore. Once it is out, we will be finally able to wipe
Aluminium, reducing cognitive load significantly due to the genius
Netvirt no longer being in the picture.

If there is anything wrong in GA, we have SR1 scheduled exactly 4 weeks
from today.

Ok. I've sent an email to TSC so the current phosphorus candidate can be approved


Thanks

 

Regards,
Robert

>
> BR/Luis
>
>
>> On Sep 30, 2021, at 8:19 AM, Daniel de la Rosa
>> <ddelarosa0707@... <mailto:ddelarosa0707@...>> wrote:
>>
>>
>>
>> On Thu, Sep 30, 2021 at 4:26 AM Robert Varga <nite@...
>> <mailto:nite@...>> wrote:
>>
>>     On 29/09/2021 20:29, Daniel de la Rosa wrote:
>>     >
>>     >
>>     > On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@...
>>     <mailto:nite@...>
>>     > <mailto:nite@... <mailto:nite@...>>> wrote:
>>     >
>>     >     On 27/09/2021 21:18, Daniel de la Rosa wrote:
>>     >      > It seems that we are still having Bgp and pcep issues but are
>>     >     they all
>>     >      > critical ? Or can we fix them later so we can release
>>     phosphorus  ?
>>     >      >
>>     >      > On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
>>     <mailto:nite@...>
>>     >     <mailto:nite@... <mailto:nite@...>>
>>     >      > <mailto:nite@... <mailto:nite@...> <mailto:nite@...
>>     <mailto:nite@...>>>> wrote:
>>     >      >
>>     >      >     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>>     >      >      >
>>     >      >      > Phosphorus AR#212 integration #169 has only one
>>     test case
>>     >     failed
>>     >      >      >
>>     >      >      > bgpcep
>>     >      >
>>     >
>>     https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>>     >   
>>      <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>>     >      >
>>     >     
>>      <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>
>>     >      >
>>     >     
>>      <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>>     >      >
>>     >     
>>      <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>>
>>     >      >      >
>>     >      >      > so looks to me like a good RC candidate
>>     >      >
>>     >      >     It seems we have a few regressions in both BGP and
>>     PCEP. I
>>     >     suspect I
>>     >      >     know the culprit behind at least one of the failures,
>>     should
>>     >     have an
>>     >      >     updated bgpcep ready later today.
>>     >
>>     >     Okay, so we have one remaining issue in BGPCEP, but after
>>     spending
>>     >     today
>>     >     trying to make sense of what is going on, it is not a
>>     regression, just
>>     >     shifted timing in code.
>>     >
>>     >     It may have been detected transiently before, but now it is
>>     getting
>>     >     caught in every built.
>>     >
>>     >     The underlying problem has been there since late 2017, maybe
>>     even as a
>>     >     day-0 issue.
>>     >
>>     >       From BGPCEP perspective we are good to release
>>     >
>>     https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
>>     <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>
>>     >   
>>      <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
>>     <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>>
>>     >
>>     >
>>     > Thanks Robert.. I've come up with this  phosphorus release
>>     approval for TSC
>>     >
>>     >
>>     https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
>>     <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>
>>
>>     >
>>     <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
>>     <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>>
>>     >
>>     > but I couldn't find the right mri test ... it is not 39 right?
>>     >
>>     >
>>     https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
>>     <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>
>>
>>     >
>>     <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
>>     <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>>
>>
>>     So the mri-test-phosphorus does not kick off of autorelease, but runs
>>     periodically. I have kicked off #40, but #39 should be accurate.
>>
>>
>> both #40 and #39 is having this issue
>>
>> ERROR: Build aborted. Can't trigger undefined projects. 1 of the below project(s) can't be resolved:
>>   > netconf-csit-1node-scale-max-devices-only-master
>>
>>
>>
>>
>>     Regards,
>>     Robert
>>
>>
>>
>>
>


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 30/09/2021 17:38, Luis Gomez wrote:
I think netconf devs got confused on how our CSIT branches work, I do not see any CSIT job for phosphorous and therefore I do not think we are testing netconf in this release:
https://jenkins.opendaylight.org/releng/view/netconf/ <https://jenkins.opendaylight.org/releng/view/netconf/>
I think netconf devs would be less confused if you could share some writeup of how exactly is test execution wired across integration/test and releng/builder :)

Please note that NETCONF is an MRI project, hence its naming follows its versioning scheme since Silicon GA.

Since we are after autorelease has branched stable/phosphorus, but netconf.git has _not_ branched 2.0.x, netconf's master is still hosting netconf-2.0.x and that in turn is integrated into *both* Phosphorus and Sulfur.

The same is true for odlparent, mdsal, controller, aaa and bgpcep. Yangtools is a bit ahead of the curve and has 7.0.x for Phosphorus and master is hosting yangtools-8.0.0-to-be (and destined for Sulfur).

I hope that paints the current state of git branches enough to understand where we are.

Currently all testing of MRI projects is still wired on the assumption that what we are testing int/dist with that particular MRI project. See bgpcep tests for an example.

That is quite wrong, as MRI projects' CSIT must be able to execute on whatever artifacts are produced, for example, {mriproject}-maven-stage-{branch} job.

Tomas has sunk a significant chunk of his time to correct this and we are still kicking the wheels on that here:

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-maven-mri-stage-master/ . What that job does is build and stage netconf for release (just link autorelease does for openflowplugin) and then triggers

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-distribution-mri-test-master/ . This job runs the NETCONF CSIT, but not on int/dist's karaf.tar.gz, but rather on netconf's netconf-karaf.tar.gz.

Which is pretty much what should be happening for all MRI projects, as that way we get fully-vetted artifacts before we decide to run our {project}-release-merge job to publish them to Nexus (step done manually by Anil for autorelease).

Based on all of this, please start saying goodbye to the idea that MRI projects have CSIT job names bearing SimRel names -- currently that would mean executing netconf/master CSIT twice (sulfur and phosphorus).

Also, please be patient while we bring all of this long-overlooked task to fruition.

As for netconf-csit-1node-scale-max-devices-only job, specifically, that one underwent a rewrite and is still pending stabilization. The test never really worked as it should have.

As for overall Phosphorus GA release, I just do not see the point holding it up anymore. Once it is out, we will be finally able to wipe Aluminium, reducing cognitive load significantly due to the genius Netvirt no longer being in the picture.

If there is anything wrong in GA, we have SR1 scheduled exactly 4 weeks from today.

Regards,
Robert

BR/Luis

On Sep 30, 2021, at 8:19 AM, Daniel de la Rosa <ddelarosa0707@... <mailto:ddelarosa0707@...>> wrote:



On Thu, Sep 30, 2021 at 4:26 AM Robert Varga <nite@... <mailto:nite@...>> wrote:

On 29/09/2021 20:29, Daniel de la Rosa wrote:
>
>
> On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@...
<mailto:nite@...>
> <mailto:nite@... <mailto:nite@...>>> wrote:
>
>     On 27/09/2021 21:18, Daniel de la Rosa wrote:
>      > It seems that we are still having Bgp and pcep issues but are
>     they all
>      > critical ? Or can we fix them later so we can release
phosphorus  ?
>      >
>      > On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
<mailto:nite@...>
>     <mailto:nite@... <mailto:nite@...>>
>      > <mailto:nite@... <mailto:nite@...> <mailto:nite@...
<mailto:nite@...>>>> wrote:
>      >
>      >     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>      >      >
>      >      > Phosphorus AR#212 integration #169 has only one
test case
>     failed
>      >      >
>      >      > bgpcep
>      >
>
https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>      >
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>
>      >
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>      >
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>>
>      >      >
>      >      > so looks to me like a good RC candidate
>      >
>      >     It seems we have a few regressions in both BGP and
PCEP. I
>     suspect I
>      >     know the culprit behind at least one of the failures,
should
>     have an
>      >     updated bgpcep ready later today.
>
>     Okay, so we have one remaining issue in BGPCEP, but after
spending
>     today
>     trying to make sense of what is going on, it is not a
regression, just
>     shifted timing in code.
>
>     It may have been detected transiently before, but now it is
getting
>     caught in every built.
>
>     The underlying problem has been there since late 2017, maybe
even as a
>     day-0 issue.
>
>       From BGPCEP perspective we are good to release
>
https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
<https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>
>
 <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
<https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>>
>
>
> Thanks Robert.. I've come up with this  phosphorus release
approval for TSC
>
>
https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
<https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>

>
<https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
<https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>>
>
> but I couldn't find the right mri test ... it is not 39 right?
>
>
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>

>
<https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
<https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>>

So the mri-test-phosphorus does not kick off of autorelease, but runs
periodically. I have kicked off #40, but #39 should be accurate.


both #40 and #39 is having this issue

ERROR: Build aborted. Can't trigger undefined projects. 1 of the below project(s) can't be resolved:
> netconf-csit-1node-scale-max-devices-only-master




Regards,
Robert



Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Luis Gomez
 

I think netconf devs got confused on how our CSIT branches work, I do not see any CSIT job for phosphorous and therefore I do not think we are testing netconf in this release:


BR/Luis


On Sep 30, 2021, at 8:19 AM, Daniel de la Rosa <ddelarosa0707@...> wrote:



On Thu, Sep 30, 2021 at 4:26 AM Robert Varga <nite@...> wrote:
On 29/09/2021 20:29, Daniel de la Rosa wrote:
>
>
> On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@...
> <mailto:nite@...>> wrote:
>
>     On 27/09/2021 21:18, Daniel de la Rosa wrote:
>      > It seems that we are still having Bgp and pcep issues but are
>     they all
>      > critical ? Or can we fix them later so we can release phosphorus  ?
>      >
>      > On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
>     <mailto:nite@...>
>      > <mailto:nite@... <mailto:nite@...>>> wrote:
>      >
>      >     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>      >      >
>      >      > Phosphorus AR#212 integration #169 has only one test case
>     failed
>      >      >
>      >      > bgpcep
>      >
>     https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>
>      >      >
>      >      > so looks to me like a good RC candidate
>      >
>      >     It seems we have a few regressions in both BGP and PCEP. I
>     suspect I
>      >     know the culprit behind at least one of the failures, should
>     have an
>      >     updated bgpcep ready later today.
>
>     Okay, so we have one remaining issue in BGPCEP, but after spending
>     today
>     trying to make sense of what is going on, it is not a regression, just
>     shifted timing in code.
>
>     It may have been detected transiently before, but now it is getting
>     caught in every built.
>
>     The underlying problem has been there since late 2017, maybe even as a
>     day-0 issue.
>
>       From BGPCEP perspective we are good to release
>     https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
>     <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>
>
>
> Thanks Robert.. I've come up with this  phosphorus release approval for TSC
>
> https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
> <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>
>
> but I couldn't find the right mri test ... it is not 39 right?
>
> https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
> <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>

So the mri-test-phosphorus does not kick off of autorelease, but runs
periodically. I have kicked off #40, but #39 should be accurate.

both #40 and #39 is having this issue

ERROR: Build aborted. Can't trigger undefined projects. 1 of the below project(s) can't be resolved:
 > netconf-csit-1node-scale-max-devices-only-master



 

Regards,
Robert






Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Daniel de la Rosa
 



On Thu, Sep 30, 2021 at 4:26 AM Robert Varga <nite@...> wrote:
On 29/09/2021 20:29, Daniel de la Rosa wrote:
>
>
> On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@...
> <mailto:nite@...>> wrote:
>
>     On 27/09/2021 21:18, Daniel de la Rosa wrote:
>      > It seems that we are still having Bgp and pcep issues but are
>     they all
>      > critical ? Or can we fix them later so we can release phosphorus  ?
>      >
>      > On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
>     <mailto:nite@...>
>      > <mailto:nite@... <mailto:nite@...>>> wrote:
>      >
>      >     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>      >      >
>      >      > Phosphorus AR#212 integration #169 has only one test case
>     failed
>      >      >
>      >      > bgpcep
>      >
>     https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>      >   
>       <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>
>      >      >
>      >      > so looks to me like a good RC candidate
>      >
>      >     It seems we have a few regressions in both BGP and PCEP. I
>     suspect I
>      >     know the culprit behind at least one of the failures, should
>     have an
>      >     updated bgpcep ready later today.
>
>     Okay, so we have one remaining issue in BGPCEP, but after spending
>     today
>     trying to make sense of what is going on, it is not a regression, just
>     shifted timing in code.
>
>     It may have been detected transiently before, but now it is getting
>     caught in every built.
>
>     The underlying problem has been there since late 2017, maybe even as a
>     day-0 issue.
>
>       From BGPCEP perspective we are good to release
>     https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
>     <https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>
>
>
> Thanks Robert.. I've come up with this  phosphorus release approval for TSC
>
> https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval
> <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>
>
> but I couldn't find the right mri test ... it is not 39 right?
>
> https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/
> <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>

So the mri-test-phosphorus does not kick off of autorelease, but runs
periodically. I have kicked off #40, but #39 should be accurate.

both #40 and #39 is having this issue

ERROR: Build aborted. Can't trigger undefined projects. 1 of the below project(s) can't be resolved:
 > netconf-csit-1node-scale-max-devices-only-master



 

Regards,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 29/09/2021 20:29, Daniel de la Rosa wrote:
On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@... <mailto:nite@...>> wrote:
On 27/09/2021 21:18, Daniel de la Rosa wrote:
> It seems that we are still having Bgp and pcep issues but are
they all
> critical ? Or can we fix them later so we can release phosphorus  ?
>
> On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
<mailto:nite@...>
> <mailto:nite@... <mailto:nite@...>>> wrote:
>
>     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>      >
>      > Phosphorus AR#212 integration #169 has only one test case
failed
>      >
>      > bgpcep
>
https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>
 <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/ <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>>
>      >
>      > so looks to me like a good RC candidate
>
>     It seems we have a few regressions in both BGP and PCEP. I
suspect I
>     know the culprit behind at least one of the failures, should
have an
>     updated bgpcep ready later today.
Okay, so we have one remaining issue in BGPCEP, but after spending
today
trying to make sense of what is going on, it is not a regression, just
shifted timing in code.
It may have been detected transiently before, but now it is getting
caught in every built.
The underlying problem has been there since late 2017, maybe even as a
day-0 issue.
 From BGPCEP perspective we are good to release
https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/
<https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/>
Thanks Robert.. I've come up with this  phosphorus release approval for TSC
https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval <https://wiki.opendaylight.org/display/ODL/Phosphorus+Formal+Release+Approval>
but I couldn't find the right mri test ... it is not 39 right?
https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/ <https://jenkins.opendaylight.org/releng/view/autorelease/job/integration-distribution-mri-test-phosphorus/39/>
So the mri-test-phosphorus does not kick off of autorelease, but runs periodically. I have kicked off #40, but #39 should be accurate.

Regards,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Daniel de la Rosa
 



On Wed, Sep 29, 2021 at 9:52 AM Robert Varga <nite@...> wrote:
On 27/09/2021 21:18, Daniel de la Rosa wrote:
> It seems that we are still having Bgp and pcep issues but are they all
> critical ? Or can we fix them later so we can release phosphorus  ?
>
> On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...
> <mailto:nite@...>> wrote:
>
>     On 23/09/2021 05:54, Daniel de la Rosa wrote:
>      >
>      > Phosphorus AR#212 integration #169 has only one test case failed
>      >
>      > bgpcep
>     https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
>     <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>      >
>      > so looks to me like a good RC candidate
>
>     It seems we have a few regressions in both BGP and PCEP. I suspect I
>     know the culprit behind at least one of the failures, should have an
>     updated bgpcep ready later today.

Okay, so we have one remaining issue in BGPCEP, but after spending today
trying to make sense of what is going on, it is not a regression, just
shifted timing in code.

It may have been detected transiently before, but now it is getting
caught in every built.

The underlying problem has been there since late 2017, maybe even as a
day-0 issue.

 From BGPCEP perspective we are good to release
https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/

Thanks Robert.. I've come up with this  phosphorus release approval for TSC

 

but I couldn't find the right mri test ... it is not 39 right?




Regards,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 27/09/2021 21:18, Daniel de la Rosa wrote:
It seems that we are still having Bgp and pcep issues but are they all critical ? Or can we fix them later so we can release phosphorus  ?
On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@... <mailto:nite@...>> wrote:
On 23/09/2021 05:54, Daniel de la Rosa wrote:
>
> Phosphorus AR#212 integration #169 has only one test case failed
>
> bgpcep
https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/> <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>
> so looks to me like a good RC candidate
It seems we have a few regressions in both BGP and PCEP. I suspect I
know the culprit behind at least one of the failures, should have an
updated bgpcep ready later today.
Okay, so we have one remaining issue in BGPCEP, but after spending today trying to make sense of what is going on, it is not a regression, just shifted timing in code.

It may have been detected transiently before, but now it is getting caught in every built.

The underlying problem has been there since late 2017, maybe even as a day-0 issue.

From BGPCEP perspective we are good to release https://jenkins.opendaylight.org/releng/job/autorelease-release-phosphorus-mvn35-openjdk11/221/

Regards,
Robert


TSC Meeting for September 30, 2021 at 10 pm Pacific

Guillaume Lambert
 

Hello OpenDaylight Community,


Next TSC meeting is September 30, 2021 at 10 pm Pacific Time.
The agenda proposal and the connection details for this meeting are available at the following URL:

 

If you need to add anything, please let me know or add it there.
The meeting minutes will be at the same location after the meeting is over.

Best Regards
Guillaume

 
_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 28/09/2021 13:33, Robert Varga wrote:
On 28/09/2021 05:24, Robert Varga wrote:
On 27/09/2021 21:18, Daniel de la Rosa wrote:
It seems that we are still having Bgp and pcep issues but are they all critical ? Or can we fix them later so we can release phosphorus  ?
Yeah, it's three test cases, all of them are PCEP-related. They are failing reliably, which seems to indicate a systemic problem.

I'll try to see if I can debug/repro it tomorrow. We might punt to SR1 (which is around the corner) if it ends up being something hard.
Alright, this looks like a pccmock issue:
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/bgpcep-csit-1node-userfeatures-all-phosphorus/179/robot-plugin/log.html.gz#s1-s2-k2-k3 https://jira.opendaylight.org/browse/BGPCEP-981 tracks it.
Okay, I think I found the culprint. A fixed bgpcep should be out in about two hours or so and should be reflect in next AR build.

Bye,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 28/09/2021 05:24, Robert Varga wrote:
On 27/09/2021 21:18, Daniel de la Rosa wrote:
It seems that we are still having Bgp and pcep issues but are they all critical ? Or can we fix them later so we can release phosphorus  ?
Yeah, it's three test cases, all of them are PCEP-related. They are failing reliably, which seems to indicate a systemic problem.
I'll try to see if I can debug/repro it tomorrow. We might punt to SR1 (which is around the corner) if it ends up being something hard.
Alright, this looks like a pccmock issue:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/bgpcep-csit-1node-userfeatures-all-phosphorus/179/robot-plugin/log.html.gz#s1-s2-k2-k3

https://jira.opendaylight.org/browse/BGPCEP-981 tracks it.

Regards,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 27/09/2021 21:18, Daniel de la Rosa wrote:
It seems that we are still having Bgp and pcep issues but are they all critical ? Or can we fix them later so we can release phosphorus  ?
Yeah, it's three test cases, all of them are PCEP-related. They are failing reliably, which seems to indicate a systemic problem.

I'll try to see if I can debug/repro it tomorrow. We might punt to SR1 (which is around the corner) if it ends up being something hard.

Regards,
Robert




On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@... <mailto:nite@...>> wrote:
On 23/09/2021 05:54, Daniel de la Rosa wrote:
>
> Phosphorus AR#212 integration #169 has only one test case failed
>
> bgpcep
https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/> <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/
<https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>>
>
> so looks to me like a good RC candidate
It seems we have a few regressions in both BGP and PCEP. I suspect I
know the culprit behind at least one of the failures, should have an
updated bgpcep ready later today.
Regards,
Robert


Re: [release] [releng][TSC] phosphorus release status - master branch has been locked

Daniel de la Rosa
 

It seems that we are still having Bgp and pcep issues but are they all critical ? Or can we fix them later so we can release phosphorus  ?

On Thu, Sep 23, 2021 at 5:08 AM Robert Varga <nite@...> wrote:
On 23/09/2021 05:54, Daniel de la Rosa wrote:
>
> Phosphorus AR#212 integration #169 has only one test case failed
>
> bgpcep        https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/  <https://jenkins.opendaylight.org/releng/job/bgpcep-csit-1node-userfeatures-all-phosphorus/167/>
>
> so looks to me like a good RC candidate

It seems we have a few regressions in both BGP and PCEP. I suspect I
know the culprit behind at least one of the failures, should have an
updated bgpcep ready later today.

Regards,
Robert


Re: Phosphorus release, input to marketing

Robert Varga
 

On 24/09/2021 06:23, Daniel de la Rosa wrote:
On Thu, Sep 23, 2021 at 1:32 PM Robert Varga <nite@... <mailto:nite@...>> wrote:
On 20/09/2021 05:57, Daniel de la Rosa wrote:
> Hello TSC and all
Hey Daniel,

> As we agree, I have updated the filters on this page.
>
> https://wiki.opendaylight.org/display/ODL/Phosphorus+Release
<https://wiki.opendaylight.org/display/ODL/Phosphorus+Release>
> <https://wiki.opendaylight.org/display/ODL/Phosphorus+Release
<https://wiki.opendaylight.org/display/ODL/Phosphorus+Release>>
>
> to try to capture the bug fixes and enhancements for each of the
managed
> projects since phosphorus started ( 3/17/2021).
>
> Please  let us know if this list of jira tickets is
representative of
> the work for each of your projects in this release. Additionally and
> more important, please  provide a summary on the page or reply
with this
> email, with the highlights for each of your projects so LFN
> marketing can get ready for this release.
As I stated before, I do not believe Confluence is the right tool here
for long term.
Since we are already maintaining release notes here:
https://docs.opendaylight.org/en/latest/release-notes/
<https://docs.opendaylight.org/en/latest/release-notes/>
I think that page should be our primary "release notes" collateral.
I have started updating individual project release notes, like
Controller:
https://docs.opendaylight.org/en/latest/release-notes/project/controller.html
<https://docs.opendaylight.org/en/latest/release-notes/project/controller.html>
yes i agree, this does look better than the confluence page ...  and I think you meant this page right?
https://docs.opendaylight.org/en/latest/release-notes/projects/controller.html <https://docs.opendaylight.org/en/latest/release-notes/projects/controller.html>
which looks great, so lets go with this page. I'll just link the confluence page to this page instead
I am glad you like it, all MRI projects have been updated to the best of my knowledge.

As for marketing ... what is really the marketing strategy being
executed? Is it just a simple article on www.opendaylight.org
<http://www.opendaylight.org>?
Yes, an article that highlights new features, enhancements and fixes In Phosphorus. We agree on the last TSC to dedicate time on the next meeting to review this as  a group. Hoep you can join us to review these release notes
Ah, yeah, my timing was off last time, joined exactly one hour late :(

Regards,
Robert


Re: [releng][TSC] phosphorus release status - master branch has been locked

Anil Belur
 

Hi Robert: 

The jjb-merge job failed since a new branch is added and the CR changes a lots of job configs. This translates to updating all the job configs on Jenkins and requires more time to complete. 
In most cases, I generally increase the timeout for the jjb-merge job during the branch cut (which I missed this time). PS note: This can be fixed by "remerge" comment on the CR.

- Anil  


On Fri, Sep 24, 2021 at 3:10 AM Robert Varga <nite@...> wrote:
On 23/09/2021 18:24, Robert Varga wrote:
> On 21/09/2021 02:01, Anil Belur wrote:
>> Hello all,
>>
>> The master branch is locked for Phosphorus code freeze and branch
>> cutting ("stable/phosphorus") and master branch will be promoted to
>> next (Sulphur). Once the version bump and the release activities are
>> complete, I will notify the status here.
>
> It seems Jenkins/JJB integration is busted. I have filed
> https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-22956
> for that.

To explain: it is failing since Sep 21, hence Jenkins jobs do not
reflect Gerrit branches mapping:
- all -phosphorus jobs still refer to master branch
- there are no -sulfur jobs

Regards,
Robert


The road to Java 17

Robert Varga
 

Hello everyone,

as you might have noticed, Java 17 has been released: https://jdk.java.net/17/ with reference implementation here: https://jdk.java.net/java-se-ri/17

OpenDaylight currently requires Java 11 at compile-time and *should* be able to run on everything up to Java 17. This *should* is currently not enforced by our CI, but I am not aware of any reasons this would not be the case.

Java 17 is the next LTS release, which there are multiple support options available, with at least 8 years of support being available.

As per our usual OpenDaylight support policy, we are currently supporting Java 17 runtime on a best-effort policy: any issues found will be dealt with to the extent considered feasible.

Going forward, though, we will require Java 17 as both compile-time and runtime very soon, simply because of the language feature options becoming available:
- https://openjdk.java.net/jeps/361 (switch expressions)
- https://openjdk.java.net/jeps/371 (hidden classes)
- https://openjdk.java.net/jeps/378 (text blocks)
- https://openjdk.java.net/jeps/394 (instanceof pattern matching)
- https://openjdk.java.net/jeps/395 (records)
- https://openjdk.java.net/jeps/409 (sealed classes)
- https://openjdk.java.net/jeps/415 (deserialization filters)
- https://jdk.java.net/17/release-notes#JDK-8251989 (improved CHA)

Furthermore, there are a ton of runtime improvements, which we can take into implementation considerations, like https://bugs.openjdk.java.net/browse/JDK-8266074. We want to take advantage to these ASAP.

IIUC, there are only a few issues which prevents us from adopting JDK 17 as a requirement:
- maven-xtend-plugin compatibility (due to Guice, what a surprise), which should be solved in 2.26.0, whenever that is available
- SpotBugs compatibility, which should be addressed in 4.4.x series


With all this in picture, I believe the proper course in OpenDaylight is to have:
- Sulfur (22.03) supporting both JDK11 and JDK17 at compile-time, with artifacts compatible with JDK11+
- All of Sulfur being validated with JDK17
- Chlorine (22.09) to require JDK17+

Unless there are any objections, this is the current plan of record. If you disagree, please holler now.

Regards,
Robert


Re: Phosphorus release, input to marketing

Daniel de la Rosa
 


On Thu, Sep 23, 2021 at 1:32 PM Robert Varga <nite@...> wrote:
On 20/09/2021 05:57, Daniel de la Rosa wrote:
> Hello TSC and all

Hey Daniel,

> As we agree, I have updated the filters on this page.
>
> https://wiki.opendaylight.org/display/ODL/Phosphorus+Release
> <https://wiki.opendaylight.org/display/ODL/Phosphorus+Release>
>
> to try to capture the bug fixes and enhancements for each of the managed
> projects since phosphorus started ( 3/17/2021).
>
> Please  let us know if this list of jira tickets is representative of
> the work for each of your projects in this release. Additionally and
> more important, please  provide a summary on the page or reply with this
> email, with the highlights for each of your projects so LFN
> marketing can get ready for this release.

As I stated before, I do not believe Confluence is the right tool here
for long term.

Since we are already maintaining release notes here:
https://docs.opendaylight.org/en/latest/release-notes/

I think that page should be our primary "release notes" collateral.

I have started updating individual project release notes, like Controller:
https://docs.opendaylight.org/en/latest/release-notes/project/controller.html

yes i agree, this does look better than the confluence page ...  and I think you meant this page right?


which looks great, so lets go with this page. I'll just link the confluence page to this page instead 
 


The end result is very maintainable structure, which can be just updated
(perhaps subject to automation), rather than recreated every time anew.

As for marketing ... what is really the marketing strategy being
executed? Is it just a simple article on www.opendaylight.org?

Yes, an article that highlights new features, enhancements and fixes In Phosphorus. We agree on the last TSC to dedicate time on the next meeting to review this as  a group. Hoep you can join us to review these release notes 

 

Regards,
Robert


Re: [releng][TSC] phosphorus release status - master branch has been locked

Daniel de la Rosa
 


On Thu, Sep 23, 2021 at 10:10 AM Robert Varga <nite@...> wrote:
On 23/09/2021 18:24, Robert Varga wrote:
> On 21/09/2021 02:01, Anil Belur wrote:
>> Hello all,
>>
>> The master branch is locked for Phosphorus code freeze and branch
>> cutting ("stable/phosphorus") and master branch will be promoted to
>> next (Sulphur). Once the version bump and the release activities are
>> complete, I will notify the status here.
>
> It seems Jenkins/JJB integration is busted. I have filed
> https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-22956
> for that.

To explain: it is failing since Sep 21, hence Jenkins jobs do not
reflect Gerrit branches mapping:
- all -phosphorus jobs still refer to master branch
- there are no -sulfur jobs

it seems that the issue has been fixed by LF IT so are we good to pick up Phosphorus AR #216 integration #174 is showing some BGPCEP failed test cases


So I'm assuming that we do want to fix those?

 

Regards,
Robert


Re: Phosphorus release, input to marketing

Robert Varga
 

On 20/09/2021 05:57, Daniel de la Rosa wrote:
Hello TSC and all
Hey Daniel,

As we agree, I have updated the filters on this page.
https://wiki.opendaylight.org/display/ODL/Phosphorus+Release <https://wiki.opendaylight.org/display/ODL/Phosphorus+Release>
to try to capture the bug fixes and enhancements for each of the managed projects since phosphorus started ( 3/17/2021).
Please  let us know if this list of jira tickets is representative of the work for each of your projects in this release. Additionally and more important, please  provide a summary on the page or reply with this email, with the highlights for each of your projects so LFN marketing can get ready for this release.
As I stated before, I do not believe Confluence is the right tool here for long term.

Since we are already maintaining release notes here: https://docs.opendaylight.org/en/latest/release-notes/

I think that page should be our primary "release notes" collateral.

I have started updating individual project release notes, like Controller:
https://docs.opendaylight.org/en/latest/release-notes/project/controller.html

The end result is very maintainable structure, which can be just updated (perhaps subject to automation), rather than recreated every time anew.

As for marketing ... what is really the marketing strategy being executed? Is it just a simple article on www.opendaylight.org?

Regards,
Robert


Re: [releng][TSC] phosphorus release status - master branch has been locked

Robert Varga
 

On 23/09/2021 18:24, Robert Varga wrote:
On 21/09/2021 02:01, Anil Belur wrote:
Hello all,

The master branch is locked for Phosphorus code freeze and branch cutting ("stable/phosphorus") and master branch will be promoted to next (Sulphur). Once the version bump and the release activities are complete, I will notify the status here.
It seems Jenkins/JJB integration is busted. I have filed https://jira.linuxfoundation.org/plugins/servlet/theme/portal/2/IT-22956 for that.
To explain: it is failing since Sep 21, hence Jenkins jobs do not reflect Gerrit branches mapping:
- all -phosphorus jobs still refer to master branch
- there are no -sulfur jobs

Regards,
Robert

421 - 440 of 14296