Re: IT-22529 Vexxhost stack instantiation failed


Robert Varga
 

So this again is interplay between CSIT, Vexxhost, openstack and us.

There were wide-scale Jenkins problems during past 24 hours, as for example:

1.
https://jenkins.opendaylight.org/releng/job/integration-distribution-test-silicon/390/
spent 12 hours waiting in the queue. This job normally executes within
90 minutes.

2.
https://jenkins.opendaylight.org/releng/view/distribution/job/integration-distribution-test-aluminium/507/
has 2 new failures in three-node scenarios

3.
https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/
took 13 hours to build.
It usually completes in 20 minutes.

https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/console
reveals this:

01:03:35 -----END_OF_BUILD-----
This is after end of build, i.e. post-build actions which the ODL
community has no way of affecting.

01:03:35 [JaCoCo plugin] Loading packages..
12:21:28 [JaCoCo plugin] Done.
i.e. it took 11 hours and 18 minutes...

12:21:28 [JaCoCo plugin] Loading packages..
12:26:24 [JaCoCo plugin] Done.
... then 5 minutes ...

12:26:24 [JaCoCo plugin] Loading packages..
13:43:35 [JaCoCo plugin] Done.
... and some 76 minutes ...

13:43:35 [JaCoCo plugin] Loading packages..
14:25:10 [JaCoCo plugin] Done.
... and some 42 minutes ...

14:25:10 [JaCoCo plugin] Loading packages..
14:25:20 [JaCoCo plugin] Done.
14:25:20 [PostBuildScript] - [ERROR] An error occured during post-build processing.
14:25:32 org.jenkinsci.plugins.postbuildscript.PostBuildScriptException: java.lang.InterruptedException
and this is me, taking that job out back. It probably would still be
running, holding back retriggers (of which 7 have since passed
successfully).

While all of this was going on, https://status.linuxfoundation.org/
shows OpenDaylight infra to be happy as a clam the last 24 hours.

Something is seriously not adding up here and it is Not Fun[*].

Regards,
Robert

[*] All of this delayed Phosphorus CSIT results by ~10 hours, i.e. no
new data during morning CET.


On 21/07/2021 22:22, Project Services wrote:
—-—-—-—
—-—-—-—
Reply above this line.

Hello *Robert Varga*,

*Eric Ball* has added a comment on your request *IT-22529*
<https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>:

_The resource usage is currently low, so I was able to successfully
re-run the ovsdb job. I'm re-running the openflowplugin job now.

I'll also submit a request to Vexxhost to expand or remove the
memory cap. We have a limit to the total number of instances active
at once, which should keep our resource usage from hitting any of
the usage caps. But it seems that the memory cap doesn't take into
account the ram used by stacks dynamically spun up by the jobs, so a
large number of those jobs running concurrently can still have this
issue._

You may reply to their comment *here
<https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>*

------------------------------------------------------------------------

Hello *Robert Varga*,

*System Administrator* changed the status of your request *IT-22529
<https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>*
to *Waiting for customer*.

View request
<https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>
· Turn off this request's notifications
<https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529/unsubscribe?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJxc2giOiI0Nzc2OWVjZGY5OTNhYzM4ZjczYTNlMWZhNTY0YWViOGZmODM0ZjYzZWYyZjEzMjUzMTQ1NjU0OTg5ZGYwMTMyIiwiaXNzIjoic2VydmljZWRlc2stand0LXRva2VuLWlzc3VlciIsImNvbnRleHQiOnsidXNlciI6InJvdmFyZ2EiLCJpc3N1ZSI6IklULTIyNTI5In0sImV4cCI6MTYyOTMxODEwNCwiaWF0IjoxNjI2ODk4OTA0fQ.Dw61-8vZTZJFaz5psnA6RauRWMALf-ctSJIQun61XwY>


This is shared with ODL and Robert Varga.

Help Center, powered by Jira Service Desk
<https://www.atlassian.com/software/jira/service-desk/powered-by?utm_medium=email&utm_source=service-desk_email-notification_server&utm_campaign=service-desk_email-notification_server>,
sent you this message.

Join TSC@lists.opendaylight.org to automatically receive all group messages.