Abhijit, can we bring this topic in tonight’s TSC?

BR/Luis

Begin forwarded message:

From: "Allan" <aclarke@...>

Subject: [OpenDaylight TSC] Issue with Linux Foundation and expectations on ODL projects

Date: March 5, 2020 at 8:05:00 AM PST

To: tsc <TSC@...>

Cc: Thanh Ha <thanh.ha@...>

Greetings!

 

I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk. 

 

*Summary*

 

LF is not allowing newer release of Plastic 2.1.7 that fixes 3^rdparty vulnerabilities already in 2.1.6 release.

LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available. 

Both LF tooling (Nexus IQ) and vulnerability process is broken.

 

*Details*

 

Here is the sequence of events AFAIKT:

 

• Plastic 2.1.6 released (months ago)
• LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
• Plastic 2.1.7 candidate build successful but not available for testing
• LF helpdesk says Nexus IQ violations making build unavailable
• (Older version of Guava and Groovy have exposure on features not used by Plastic)
• LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
• New version of Plastic 2.1.7 candidate build (violations fixed)
• Build still not available due to “violations” (what???)
• Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
• Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
• LF says because security violations are involved, the ODL Security group needs waivers.

 

*Conclusion*

 

I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3^rd party libraries.

It has been languishing and holding up the next major release.

I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.

 

Allan Clarke

Plastic PTL

 

Hi Jamo,

Few outcomes from yesterday TSC call (at least for me):

1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.
4) ODL community needs more and better communication on how things work and change in LF.

BR/Luis


> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>
> I missed the TSC meeting last night. Can someone recap over email what came of
> this? I see one note in the minutes that Casey indicated that no change has
> occurred in the policy regarding security violations, but that doesn't seem
> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
> on security policies. Maybe I don't understand something, but that feels like
> something changed...
>
> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
> logs and messages about v2.1.6. Allan has mentioned that multiple times and
> it keeps getting ignored.
>
> Thanks,
> JamO
>
> On 3/5/20 7:01 PM, Thanh ha wrote:
>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>> Greetings!
>>
>> 
>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>
>> 
>> *Summary*
>>
>> 
>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>
>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>
>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>
>> 
>> *Details*
>>
>> 
>> Here is the sequence of events AFAIKT:
>>
>> 
>>      • Plastic 2.1.6 released (months ago)
>>      • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>      • Plastic 2.1.7 candidate build successful but not available for testing
>>      • LF helpdesk says Nexus IQ violations making build unavailable
>>      • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>      • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>      • New version of Plastic 2.1.7 candidate build (violations fixed)
>>      • Build still not available due to “violations” (what???)
>>      • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>      • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>      • LF says because security violations are involved, the ODL Security group needs waivers.
>> 
>> *Conclusion*
>>
>> 
>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>
>> It has been languishing and holding up the next major release.
>>
>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>
>> 
>> Allan Clarke
>>
>> Plastic PTL
>>
>> 
>>
>>
>> Greetings Allan:
>>
>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>> 
>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release.
>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting.
>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>
>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>
>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>
>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>
>> Regards,
>> Thanh
>>
>>
>>
>
>

Well, that is what LF is investigating now per my understanding but this should not block Allan to use the security team avenue.

BR/Luis

On Mar 6, 2020, at 3:26 PM, Jamo Luhrsen <jluhrsen@...> wrote:

thanks Luis, Abhijit.

Anil, Casey, still wondering about the v2.1.6 vs v2.1.7 weirdness that keeps getting
unanswered.

JamO

On 3/6/20 3:18 PM, Abhijit Kumbhare wrote:

Also:

• Casey Cain to work with LF IT to address communications to the community regarding security issues. 06 Mar 2020 
• Casey Cain to follow up with Allan Clarke Thanh Ha and the community on Plastic security issues 06 Mar 2020 

On Fri, Mar 6, 2020 at 3:03 PM Luis Gomez <ecelgp@...> wrote:

Hi Jamo,

Few outcomes from yesterday TSC call (at least for me):

1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.
4) ODL community needs more and better communication on how things work and change in LF.

BR/Luis


> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>
> I missed the TSC meeting last night. Can someone recap over email what came of
> this? I see one note in the minutes that Casey indicated that no change has
> occurred in the policy regarding security violations, but that doesn't seem
> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
> on security policies. Maybe I don't understand something, but that feels like
> something changed...
>
> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
> logs and messages about v2.1.6. Allan has mentioned that multiple times and
> it keeps getting ignored.
>
> Thanks,
> JamO
>
> On 3/5/20 7:01 PM, Thanh ha wrote:
>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>> Greetings!
>>
>> 
>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>
>> 
>> *Summary*
>>
>> 
>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>
>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>
>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>
>> 
>> *Details*
>>
>> 
>> Here is the sequence of events AFAIKT:
>>
>> 
>>      • Plastic 2.1.6 released (months ago)
>>      • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>      • Plastic 2.1.7 candidate build successful but not available for testing
>>      • LF helpdesk says Nexus IQ violations making build unavailable
>>      • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>      • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>      • New version of Plastic 2.1.7 candidate build (violations fixed)
>>      • Build still not available due to “violations” (what???)
>>      • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>      • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>      • LF says because security violations are involved, the ODL Security group needs waivers.
>> 
>> *Conclusion*
>>
>> 
>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>
>> It has been languishing and holding up the next major release.
>>
>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>
>> 
>> Allan Clarke
>>
>> Plastic PTL
>>
>> 
>>
>>
>> Greetings Allan:
>>
>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>> 
>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release.
>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting.
>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>
>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>
>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>
>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>
>> Regards,
>> Thanh
>>
>>
>>
>
>

On 07/03/2020 00:03, Luis Gomez wrote:
> Hi Jamo,
>
> Few outcomes from yesterday TSC call (at least for me):
>
> 1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
> 2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
> 3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.

I am sorry, but I just cannot agree with this point.

Each normal[0] OpenDaylight project is created sovereign.

That means that each project is completely and ultimately responsible
for defining and executing any processes regarding any and all aspects
of its governance [1].

While a project may waive its right to release individually -- such as
MSI projects, who have agreed to trade that right in for the convenience
autorelease provides -- every project can reverse such a decision at any
moment, subject *only* to its own governance.

To make it very clear and unambiguous, because this is something people
tend to forget:

NEITHER THE OPENDAYLIGHT TSC NOR THE LINUX FOUNDATION HOLD ANY SWAY
WHATSOVEVER WHICH CAN BE USED TO OPPOSE, OR EVEN BE CONSTRUED TO
CONSTRAIN, A PROJECT'S DECISION TO RELEASE ITS ARTIFACTS.

While this Nexus IQ thing can be used as an advice or a warning,
providing any waiver (or whatever) needs to be as simple as triggering
the $project-release-merge job, perhaps with an additional checkbox
selected.

A review by any entity outside of Plastic's committers is *BY
DEFINITION* purely optional and *CANNOT* be imposed on Plastic unless
its committers *EXPLICITLY* agree to it. Furthermore (as implied above),
Plastic's committers are free to rescind any such agreement at any time,
effective immediately, subject to no other approval.

I understand it may be quite frustrating to think about projects this
way, but this has always been our governance and any sort of top-down
management goes directly against both the spirit and the wording of our
governing documents.

Regards,
Robert

[0] there are projects created by the TSC to provide execution arm for
its authority -- integration/* and autorelease are examples of such projects
[1] https://www.merriam-webster.com/dictionary/sovereign

> 4) ODL community needs more and better communication on how things work and change in LF.
>
> BR/Luis
>
>
>> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>>
>> I missed the TSC meeting last night. Can someone recap over email what came of
>> this? I see one note in the minutes that Casey indicated that no change has
>> occurred in the policy regarding security violations, but that doesn't seem
>> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
>> on security policies. Maybe I don't understand something, but that feels like
>> something changed...
>>
>> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
>> logs and messages about v2.1.6. Allan has mentioned that multiple times and
>> it keeps getting ignored.
>>
>> Thanks,
>> JamO
>>
>> On 3/5/20 7:01 PM, Thanh ha wrote:
>>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>>> Greetings!
>>>
>>> 
>>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>>
>>> 
>>> *Summary*
>>>
>>> 
>>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>>
>>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>>
>>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>>
>>> 
>>> *Details*
>>>
>>> 
>>> Here is the sequence of events AFAIKT:
>>>
>>> 
>>>     • Plastic 2.1.6 released (months ago)
>>>     • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>>     • Plastic 2.1.7 candidate build successful but not available for testing
>>>     • LF helpdesk says Nexus IQ violations making build unavailable
>>>     • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>>     • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>>     • New version of Plastic 2.1.7 candidate build (violations fixed)
>>>     • Build still not available due to “violations” (what???)
>>>     • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>>     • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>>     • LF says because security violations are involved, the ODL Security group needs waivers.
>>> 
>>> *Conclusion*
>>>
>>> 
>>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>>
>>> It has been languishing and holding up the next major release.
>>>
>>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>>
>>> 
>>> Allan Clarke
>>>
>>> Plastic PTL
>>>
>>> 
>>>
>>>
>>> Greetings Allan:
>>>
>>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>>> 
>>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release.
>>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting.
>>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>>
>>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>>
>>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>>
>>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>>
>>> Regards,
>>> Thanh
>>>
>>>
>>>
>>
>>
>
>
>
>

FYI, I have found this in the ODL charter [1], it is about License violations rather than security vulnerabilities (I believe Nexus IQ detects both):

Paragraph c and d in section 7. Intellectual Property Policy:

The TSC may approve the use of an alternative license or licenses for inbound or outbound contributions on an exception basis. To request an exception, please describe the contribution, the alternative open source license(s), and the justification for using an alternative open source license for the Project. License exceptions must be approved by a two-thirds vote of the entire TSC.

Contributed files should contain license information, such as SPDX short form identifiers, indicating the open source license or licenses pertaining to the file. License compliance review may be conducted at any time, including prior to contribution, to verify compliance of contributions with the terms of the Charter, pursuant to the Inbound Code Review policy, available on opendaylight.org.

I guess the above also applies to third party SW used in ODL?

BR/Luis

[1] https://www.opendaylight.org/wp-content/uploads/sites/14/2018/01/ODL-Technical-Charter-LF-Projects-LLC-FINAL.pdf

On Mar 7, 2020, at 10:07 AM, Luis Gomez via Lists.Opendaylight.Org <ecelgp=gmail.com@...> wrote:

I think there is a mismatch between this and what Casey communicated last Thursday, I wish you were there. My understanding of ODL project governance is exactly what you described in your email, however in the meeting it was explained the security enforcement to release an artifact is part of the LF standard policies, not from now but since long time. I guess we will have to look at ODL records and bylaws to clarify the project governance model and resolve this issue for good.

BR/Luis

On Fri, Mar 6, 2020, 7:21 PM Robert Varga <nite@...> wrote:

On 07/03/2020 00:03, Luis Gomez wrote:
> Hi Jamo,
> 
> Few outcomes from yesterday TSC call (at least for me):
> 
> 1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
> 2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
> 3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.

I am sorry, but I just cannot agree with this point.

Each normal[0] OpenDaylight project is created sovereign.

That means that each project is completely and ultimately responsible
for defining and executing any processes regarding any and all aspects
of its governance [1].

While a project may waive its right to release individually -- such as
MSI projects, who have agreed to trade that right in for the convenience
autorelease provides -- every project can reverse such a decision at any
moment, subject *only* to its own governance.

To make it very clear and unambiguous, because this is something people
tend to forget:

NEITHER THE OPENDAYLIGHT TSC NOR THE LINUX FOUNDATION HOLD ANY SWAY
WHATSOVEVER WHICH CAN BE USED TO OPPOSE, OR EVEN BE CONSTRUED TO
CONSTRAIN, A PROJECT'S DECISION TO RELEASE ITS ARTIFACTS.

While this Nexus IQ thing can be used as an advice or a warning,
providing any waiver (or whatever) needs to be as simple as triggering
the $project-release-merge job, perhaps with an additional checkbox
selected.

A review by any entity outside of Plastic's committers is *BY
DEFINITION* purely optional and *CANNOT* be imposed on Plastic unless
its committers *EXPLICITLY* agree to it. Furthermore (as implied above),
Plastic's committers are free to rescind any such agreement at any time,
effective immediately, subject to no other approval.

I understand it may be quite frustrating to think about projects this
way, but this has always been our governance and any sort of top-down
management goes directly against both the spirit and the wording of our
governing documents.

Regards,
Robert

[0] there are projects created by the TSC to provide execution arm for
its authority -- integration/* and autorelease are examples of such projects
[1] https://www.merriam-webster.com/dictionary/sovereign

> 4) ODL community needs more and better communication on how things work and change in LF. 
> 
> BR/Luis
> 
> 
>> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>>
>> I missed the TSC meeting last night. Can someone recap over email what came of
>> this? I see one note in the minutes that Casey indicated that no change has
>> occurred in the policy regarding security violations, but that doesn't seem
>> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
>> on security policies. Maybe I don't understand something, but that feels like
>> something changed...
>>
>> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
>> logs and messages about v2.1.6. Allan has mentioned that multiple times and
>> it keeps getting ignored.
>>
>> Thanks,
>> JamO
>>
>> On 3/5/20 7:01 PM, Thanh ha wrote:
>>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>>> Greetings!
>>>
>>>  
>>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>>
>>>  
>>> *Summary*
>>>
>>>  
>>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>>
>>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>>
>>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>>
>>>  
>>> *Details*
>>>
>>>  
>>> Here is the sequence of events AFAIKT:
>>>
>>>  
>>>     • Plastic 2.1.6 released (months ago)
>>>     • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>>     • Plastic 2.1.7 candidate build successful but not available for testing
>>>     • LF helpdesk says Nexus IQ violations making build unavailable
>>>     • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>>     • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>>     • New version of Plastic 2.1.7 candidate build (violations fixed)
>>>     • Build still not available due to “violations” (what???)
>>>     • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>>     • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>>     • LF says because security violations are involved, the ODL Security group needs waivers.
>>>  
>>> *Conclusion*
>>>
>>>  
>>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>>
>>> It has been languishing and holding up the next major release.
>>>
>>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>>
>>>  
>>> Allan Clarke
>>>
>>> Plastic PTL
>>>
>>>  
>>>
>>>
>>> Greetings Allan:
>>>
>>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>>>  
>>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release. 
>>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting. 
>>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>>
>>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>>
>>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>>
>>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>>
>>> Regards,
>>> Thanh
>>>
>>>
>>>
>>
>>
> 
> 
> 
> 

Hi, everyone.

I would like to share what information we have at this time.  Plastic 2.1.6 was able to release on Jan 29th.  After that time, the staging repository has seemed to be reporting at least 1 critical security warning.  Nexus IQ did gate the builds after Jan 29th and it should not have.  At this time, we are unable to determine how the repository was set up to gate builds for CVE vulnerabilities.  This could be the result of policy bleed over from ONAP as they are working on enforcing new policies that gate on any warnings from Nexus IQ.  The Plastic repo was configured with this policy in error and it has been resolved.  When the Plastic team will not need a new build number, however, when the project rebuilds its staging repository, the issue will no longer be gated.

The LF will not gate projects for security reasons.  However, it is our goal to keep the Projects informed when CVEs are detected in the codebase.  It is the responsibility of the Projects and the TSCs to determine the security policies and if any gating should happen.  I have updated the Security page on the new wiki with the previously agreed upon vulnerability management policy, but it may need to be updated.

LF IT Management has also promised me to look at who looks at how escalated tickets are addressed and how they are notified to ensure that they get more attention and appropriate response.  I will be following up with them over the next few weeks to get a better understanding of any changes that are made and report them to the community.  IT Management teams have also met to discuss how security-related tickets are handled to ensure that they fall into Project Governance policies. 

Best,

Casey Cain

Technical Program Manager / Community Architect

Linux Foundation

_________________

IRC - CaseyLF

WeChat - okaru6

Book a Meeting w/ Casey

On Sun, Mar 8, 2020 at 10:46 AM Luis Gomez <ecelgp@...> wrote:

FYI, I have found this in the ODL charter [1], it is about License violations rather than security vulnerabilities (I believe Nexus IQ detects both):

Paragraph c and d in section 7. Intellectual Property Policy:

The TSC may approve the use of an alternative license or licenses for inbound or outbound contributions on an exception basis. To request an exception, please describe the contribution, the alternative open source license(s), and the justification for using an alternative open source license for the Project. License exceptions must be approved by a two-thirds vote of the entire TSC.

Contributed files should contain license information, such as SPDX short form identifiers, indicating the open source license or licenses pertaining to the file. License compliance review may be conducted at any time, including prior to contribution, to verify compliance of contributions with the terms of the Charter, pursuant to the Inbound Code Review policy, available on opendaylight.org.

I guess the above also applies to third party SW used in ODL?

BR/Luis

[1] https://www.opendaylight.org/wp-content/uploads/sites/14/2018/01/ODL-Technical-Charter-LF-Projects-LLC-FINAL.pdf

On Mar 7, 2020, at 10:07 AM, Luis Gomez via Lists.Opendaylight.Org <ecelgp=gmail.com@...> wrote:

I think there is a mismatch between this and what Casey communicated last Thursday, I wish you were there. My understanding of ODL project governance is exactly what you described in your email, however in the meeting it was explained the security enforcement to release an artifact is part of the LF standard policies, not from now but since long time. I guess we will have to look at ODL records and bylaws to clarify the project governance model and resolve this issue for good.

BR/Luis

On Fri, Mar 6, 2020, 7:21 PM Robert Varga <nite@...> wrote:

On 07/03/2020 00:03, Luis Gomez wrote:
> Hi Jamo,
> 
> Few outcomes from yesterday TSC call (at least for me):
> 
> 1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
> 2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
> 3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.

I am sorry, but I just cannot agree with this point.

Each normal[0] OpenDaylight project is created sovereign.

That means that each project is completely and ultimately responsible
for defining and executing any processes regarding any and all aspects
of its governance [1].

While a project may waive its right to release individually -- such as
MSI projects, who have agreed to trade that right in for the convenience
autorelease provides -- every project can reverse such a decision at any
moment, subject *only* to its own governance.

To make it very clear and unambiguous, because this is something people
tend to forget:

NEITHER THE OPENDAYLIGHT TSC NOR THE LINUX FOUNDATION HOLD ANY SWAY
WHATSOVEVER WHICH CAN BE USED TO OPPOSE, OR EVEN BE CONSTRUED TO
CONSTRAIN, A PROJECT'S DECISION TO RELEASE ITS ARTIFACTS.

While this Nexus IQ thing can be used as an advice or a warning,
providing any waiver (or whatever) needs to be as simple as triggering
the $project-release-merge job, perhaps with an additional checkbox
selected.

A review by any entity outside of Plastic's committers is *BY
DEFINITION* purely optional and *CANNOT* be imposed on Plastic unless
its committers *EXPLICITLY* agree to it. Furthermore (as implied above),
Plastic's committers are free to rescind any such agreement at any time,
effective immediately, subject to no other approval.

I understand it may be quite frustrating to think about projects this
way, but this has always been our governance and any sort of top-down
management goes directly against both the spirit and the wording of our
governing documents.

Regards,
Robert

[0] there are projects created by the TSC to provide execution arm for
its authority -- integration/* and autorelease are examples of such projects
[1] https://www.merriam-webster.com/dictionary/sovereign

> 4) ODL community needs more and better communication on how things work and change in LF. 
> 
> BR/Luis
> 
> 
>> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>>
>> I missed the TSC meeting last night. Can someone recap over email what came of
>> this? I see one note in the minutes that Casey indicated that no change has
>> occurred in the policy regarding security violations, but that doesn't seem
>> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
>> on security policies. Maybe I don't understand something, but that feels like
>> something changed...
>>
>> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
>> logs and messages about v2.1.6. Allan has mentioned that multiple times and
>> it keeps getting ignored.
>>
>> Thanks,
>> JamO
>>
>> On 3/5/20 7:01 PM, Thanh ha wrote:
>>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>>> Greetings!
>>>
>>>  
>>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>>
>>>  
>>> *Summary*
>>>
>>>  
>>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>>
>>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>>
>>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>>
>>>  
>>> *Details*
>>>
>>>  
>>> Here is the sequence of events AFAIKT:
>>>
>>>  
>>>     • Plastic 2.1.6 released (months ago)
>>>     • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>>     • Plastic 2.1.7 candidate build successful but not available for testing
>>>     • LF helpdesk says Nexus IQ violations making build unavailable
>>>     • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>>     • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>>     • New version of Plastic 2.1.7 candidate build (violations fixed)
>>>     • Build still not available due to “violations” (what???)
>>>     • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>>     • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>>     • LF says because security violations are involved, the ODL Security group needs waivers.
>>>  
>>> *Conclusion*
>>>
>>>  
>>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>>
>>> It has been languishing and holding up the next major release.
>>>
>>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>>
>>>  
>>> Allan Clarke
>>>
>>> Plastic PTL
>>>
>>>  
>>>
>>>
>>> Greetings Allan:
>>>
>>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>>>  
>>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release. 
>>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting. 
>>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>>
>>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>>
>>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>>
>>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>>
>>> Regards,
>>> Thanh
>>>
>>>
>>>
>>
>>
> 
> 
> 
> 

Hi Casey,

Thanks for promptly and moving this issue forward. I can confirm that after a rebuild the new plastic-1007 staging repo was successfully able to close.

Allan,

You should be clear to move forward with your release now via the new staging repo plastic-1007 built this morning.

https://nexus.opendaylight.org/content/repositories/plastic-1007/

Since this is a rebuild I'd recommend you double check that everything is in order with this new repo before proceeding with the release.

Regards,
Thanh

On Mon, 9 Mar 2020 at 18:00, Casey Cain <ccain@...> wrote:

Hi, everyone.

I would like to share what information we have at this time.  Plastic 2.1.6 was able to release on Jan 29th.  After that time, the staging repository has seemed to be reporting at least 1 critical security warning.  Nexus IQ did gate the builds after Jan 29th and it should not have.  At this time, we are unable to determine how the repository was set up to gate builds for CVE vulnerabilities.  This could be the result of policy bleed over from ONAP as they are working on enforcing new policies that gate on any warnings from Nexus IQ.  The Plastic repo was configured with this policy in error and it has been resolved.  When the Plastic team will not need a new build number, however, when the project rebuilds its staging repository, the issue will no longer be gated.

The LF will not gate projects for security reasons.  However, it is our goal to keep the Projects informed when CVEs are detected in the codebase.  It is the responsibility of the Projects and the TSCs to determine the security policies and if any gating should happen.  I have updated the Security page on the new wiki with the previously agreed upon vulnerability management policy, but it may need to be updated.

LF IT Management has also promised me to look at who looks at how escalated tickets are addressed and how they are notified to ensure that they get more attention and appropriate response.  I will be following up with them over the next few weeks to get a better understanding of any changes that are made and report them to the community.  IT Management teams have also met to discuss how security-related tickets are handled to ensure that they fall into Project Governance policies. 

Best,

Casey Cain

Technical Program Manager / Community Architect

Linux Foundation

_________________

IRC - CaseyLF

WeChat - okaru6

Book a Meeting w/ Casey

On Sun, Mar 8, 2020 at 10:46 AM Luis Gomez <ecelgp@...> wrote:

FYI, I have found this in the ODL charter [1], it is about License violations rather than security vulnerabilities (I believe Nexus IQ detects both):

Paragraph c and d in section 7. Intellectual Property Policy:

The TSC may approve the use of an alternative license or licenses for inbound or outbound contributions on an exception basis. To request an exception, please describe the contribution, the alternative open source license(s), and the justification for using an alternative open source license for the Project. License exceptions must be approved by a two-thirds vote of the entire TSC.

Contributed files should contain license information, such as SPDX short form identifiers, indicating the open source license or licenses pertaining to the file. License compliance review may be conducted at any time, including prior to contribution, to verify compliance of contributions with the terms of the Charter, pursuant to the Inbound Code Review policy, available on opendaylight.org.

I guess the above also applies to third party SW used in ODL?

BR/Luis

[1] https://www.opendaylight.org/wp-content/uploads/sites/14/2018/01/ODL-Technical-Charter-LF-Projects-LLC-FINAL.pdf

On Mar 7, 2020, at 10:07 AM, Luis Gomez via Lists.Opendaylight.Org <ecelgp=gmail.com@...> wrote:

I think there is a mismatch between this and what Casey communicated last Thursday, I wish you were there. My understanding of ODL project governance is exactly what you described in your email, however in the meeting it was explained the security enforcement to release an artifact is part of the LF standard policies, not from now but since long time. I guess we will have to look at ODL records and bylaws to clarify the project governance model and resolve this issue for good.

BR/Luis

On Fri, Mar 6, 2020, 7:21 PM Robert Varga <nite@...> wrote:

On 07/03/2020 00:03, Luis Gomez wrote:
> Hi Jamo,
> 
> Few outcomes from yesterday TSC call (at least for me):
> 
> 1) Thanh will open a ticket to LF to work on a new feature to notify users upfront if there is a security problem in the staging build. I think Allan was waiting long time to get this information.
> 2) LF will look at the current Nexus IQ problem of reporting old SW security issues in new SW scans.
> 3) In the situations where it is believed Nexus IQ is not doing the right thing (like this one), the impacted project/person will rise a mail to ODL security team (security@...) with all the details. The ODL security will analyze the problem and if it turns out to be a tool problem, the security team will provide an immediate waiver. If there is a real sec issue, the project/person will have to fix or ask a waiver to the TSC.

I am sorry, but I just cannot agree with this point.

Each normal[0] OpenDaylight project is created sovereign.

That means that each project is completely and ultimately responsible
for defining and executing any processes regarding any and all aspects
of its governance [1].

While a project may waive its right to release individually -- such as
MSI projects, who have agreed to trade that right in for the convenience
autorelease provides -- every project can reverse such a decision at any
moment, subject *only* to its own governance.

To make it very clear and unambiguous, because this is something people
tend to forget:

NEITHER THE OPENDAYLIGHT TSC NOR THE LINUX FOUNDATION HOLD ANY SWAY
WHATSOVEVER WHICH CAN BE USED TO OPPOSE, OR EVEN BE CONSTRUED TO
CONSTRAIN, A PROJECT'S DECISION TO RELEASE ITS ARTIFACTS.

While this Nexus IQ thing can be used as an advice or a warning,
providing any waiver (or whatever) needs to be as simple as triggering
the $project-release-merge job, perhaps with an additional checkbox
selected.

A review by any entity outside of Plastic's committers is *BY
DEFINITION* purely optional and *CANNOT* be imposed on Plastic unless
its committers *EXPLICITLY* agree to it. Furthermore (as implied above),
Plastic's committers are free to rescind any such agreement at any time,
effective immediately, subject to no other approval.

I understand it may be quite frustrating to think about projects this
way, but this has always been our governance and any sort of top-down
management goes directly against both the spirit and the wording of our
governing documents.

Regards,
Robert

[0] there are projects created by the TSC to provide execution arm for
its authority -- integration/* and autorelease are examples of such projects
[1] https://www.merriam-webster.com/dictionary/sovereign

> 4) ODL community needs more and better communication on how things work and change in LF. 
> 
> BR/Luis
> 
> 
>> On Mar 6, 2020, at 10:37 AM, JamO Luhrsen <jluhrsen@...> wrote:
>>
>> I missed the TSC meeting last night. Can someone recap over email what came of
>> this? I see one note in the minutes that Casey indicated that no change has
>> occurred in the policy regarding security violations, but that doesn't seem
>> accurate since Plastic v2.1.6 was successfully released, but v2.1.7 is blocked
>> on security policies. Maybe I don't understand something, but that feels like
>> something changed...
>>
>> Also, nobody is answering why the nexus IQ reports are failing v2.1.7 with
>> logs and messages about v2.1.6. Allan has mentioned that multiple times and
>> it keeps getting ignored.
>>
>> Thanks,
>> JamO
>>
>> On 3/5/20 7:01 PM, Thanh ha wrote:
>>> On Thu, 5 Mar 2020 at 16:45, Anil Belur <abelur@...> wrote:
>>> On Fri, Mar 6, 2020 at 2:05 AM Allan <aclarke@...> wrote:
>>> Greetings!
>>>
>>>  
>>> I have been trying to get a new release of the Plastic sub-project out for 3 weeks now and have been clogged up in LF helpdesk.
>>>
>>>  
>>> *Summary*
>>>
>>>  
>>> LF is not allowing newer release of Plastic 2.1.7 that fixes 3rd party vulnerabilities already in 2.1.6 release.
>>>
>>> LF wants ODL after-the-fact approval for Plastic 2.1.6 that is already publicly available.
>>>
>>> Both LF tooling (Nexus IQ) and vulnerability process is broken.
>>>
>>>  
>>> *Details*
>>>
>>>  
>>> Here is the sequence of events AFAIKT:
>>>
>>>  
>>>     • Plastic 2.1.6 released (months ago)
>>>     • LF silently changes release process to require Nexus IQ passing (no docs, no links to help PTLs BTW)
>>>     • Plastic 2.1.7 candidate build successful but not available for testing
>>>     • LF helpdesk says Nexus IQ violations making build unavailable
>>>     • (Older version of Guava and Groovy have exposure on features not used by Plastic)
>>>     • LF finally gets accounts and links fixed so Nexus IQ reports are available to PTL
>>>     • New version of Plastic 2.1.7 candidate build (violations fixed)
>>>     • Build still not available due to “violations” (what???)
>>>     • Nexus IQ report is all screwed up in the UI, mixing 2.1.6 and 2.1.7 up, mislabeling, clearly buggy
>>>     • Nexus IQ report appears to be showing old violations against ALREADY RELEASED 2.1.6
>>>     • LF says because security violations are involved, the ODL Security group needs waivers.
>>>  
>>> *Conclusion*
>>>
>>>  
>>> I would like to release Plastic 2.1.7. It has improvements and fixes violations in 3rd party libraries.
>>>
>>> It has been languishing and holding up the next major release.
>>>
>>> I would like a release process that understands when “the horse has left the barn” and that thinks it is a good thing when a new release fixes vulnerabilities in an old release.
>>>
>>>  
>>> Allan Clarke
>>>
>>> Plastic PTL
>>>
>>>  
>>>
>>>
>>> Greetings Allan:
>>>
>>> I understand the concerns on the delay in releasing Plastic, please find my response below.
>>>  
>>> 1. The policy violations are security issues outside the scope of LF/RE's to determine if they can be waived.
>>> 2. The policy violations showing up on stage repositories is a "feature" and not a "bug", and a proactive way to ensure these violations are going to be addressed early on before a release. 
>>> 3. The violations seen recently in plastic is a result of updating the "Nexus platform plugin", which is working as expected and not a result of any to Nexus repository level setting. 
>>> 4. It's seldom up to LF or the PTL's to waive IQ policy violations, waivers can only come from the ODL security team as approved by the TSC.
>>>
>>> The concerning thing here for me is LF deployed a new release process requiring someone or group of people to analyse and provide a security waiver to projects before allowing the project to be released and actively blocks the project from doing so. While the goal of this process might be fine and proactive the way LF decided to unleash this to the world is not.
>>>
>>> The community was not informed of the new process, nor has there been any communication to the communitiy to setup a process for providing security waivers to projects (if there has I apologies as I have not seen it and would welcome someone to correct me otherwise). The problem is today there is no way for a PTL to have this security waiver approved.
>>>
>>> I'm not sure how LF can justify implementing a new "feature" without first implementing any processes around to support the feature with the community, then seemingly telling the community it is their problem to handle it.
>>>
>>> Regards,
>>> Thanh
>>>
>>>
>>>
>>
>>
> 
> 
> 
>