Hi Abhijit,

 

For netvirt we need the below jobs:

netvirt-csit-hwvtep-openstack
netvirt-csit-multi-openstack-upgrade
netvirt-csit-multi-openstack

 

I am working with Anil Belur on optimizing the scripts to be run on Daily/Weekly basis as per the table below:

 

ODL	Openstack	Weekly/Daily
Aluminium	Stein	Daily
Aluminium	Queens	Daily
Aluminium	Rocky	Daily
Magnesium	Stein	Weekly
Magnesium	Rocky	Daily
Magnesium	Queens	Weekly
Sodium	Rocky	Weekly
Sodium	Queens	Weekly

 

https://wiki.opendaylight.org/display/ODL/2020-05-27+Integration+Meeting+Minutes

 

Thanks,

Srinivas

+91-9243478719

 

From: TSC@... <TSC@...> On Behalf Of Abhijit Kumbhare
Sent: 11 June 2020 11:37
To: Jamo Luhrsen <jluhrsen@...>
Cc: Lori Jakab <lorand.jakab+odl@...>; tsc <tsc@...>; Release <release@...>; app-dev@...; kernel-dev@...
Subject: Re: [OpenDaylight TSC] [release] High cost CSIT jobs

 

Folks,

 

What about the other jobs?

 

centos7-autorelease-8c-8g
centos7-builder-8c-8g
netvirt-csit-hwvtep-openstack
netvirt-csit-multi-openstack-upgrade
netvirt-csit-multi-openstack

 

Any thoughts from NetVirt as well as on those centos7 jobs? We will discuss in the TSC meeting tomorrow - but it would be better to have some discussion before hand over the email.

 

Thanks,

Abhijit

 

 

On Wed, Jun 10, 2020 at 3:11 PM Jamo Luhrsen <jluhrsen@...> wrote:

https://git.opendaylight.org/gerrit/c/releng/builder/+/90384

On 6/8/20 9:10 AM, Abhijit Kumbhare wrote:

Thanks for the response Lori. It is a good point about using LISP performance for monitoring the comparative historical data. We will have to investigate that further if we want to go down that path. However it is a good information that you provided about the status of lispflowmapping-csit-performance jobs. 

 

On Mon, Jun 8, 2020 at 8:27 AM Lori Jakab <lorand.jakab+odl@...> wrote:

Hi folks,

 

We no longer actively monitor lispflowmapping performance numbers, so lispflowmapping-csit-performance can be removed completely. Unless you think it has value for keeping comparative historical data on the evolution of general MD-SAL/ODL performance, since we don't modify LFM code.

 

Regards,

-Lori

 

On Sun, Jun 7, 2020 at 9:31 PM Abhijit Kumbhare <abhijitkoss@...> wrote:

Hi folks,

 

Sending this email as per my action in the TSC meeting. We have a few high cost test jobs that we should look into to check whether 

1) they are no longer needed

2) they can be run with lower CPU core / memory VMs. e.g. instead of 8c-8g (8 core, 8 GB) - can they be run with lower cores / possibly lower GB.

3) they still are needed 

 

This exercise will help us optimize our costs - we do not need to spend more if we do not need to. Also we could possibly use the savings somewhere else (for other jobs that we find more relevant).

 

The jobs with the highest costs that Robert had identified were:

 

centos7-autorelease-8c-8g
centos7-builder-8c-8g
lispflowmapping-csit-performance
netvirt-csit-hwvtep-openstack
netvirt-csit-multi-openstack-upgrade
netvirt-csit-multi-openstack

 

See Robert's email below for more details, but compared to 8c-8g:

 

4c-16g is 30% cheaper 
4c-4g is 50% cheaper
2c-8g is 66% cheaper

1c-4g/2c-1g nodes -- are 37%/82%(!) cheaper than 2c-2g.

 

Regards,

Abhijit

 

Robert's email is below this:

 

===========================

 

Hello everyone,

On 27/05/2020 16:47, Casey Cain wrote:

[snip]

Thanks for the break-down, see inline for instance sizing.

> ------------------------------------------------------
> 18581.97 instance-hours (odl-standard-2)

2c-8g

> 24918.98 instance-hours (odl-highcpu-4) 

4c-4g

> 2355.07 instance-hours (odl-standard-4) 

4c-16g

> 13692.02 instance-hours (odl-highcpu-2) 

2c-2g

> 1150.23 instance-hours (v1-standard-1) 

2c-1g

> 35611.03 instance-hours (odl-highcpu-8) 

8c-8g

> 1377.23 instance-hours (odl-standard-1) 

1c-4g

> 7.60 instance-hours (v2-highcpu-1) 
> 474.25 instance-hours (odl-standard-8) 

I cannot find these described, but they are pretty much outliers.

odl-highcpu-8 accounts for 56% of our spending, so we really should be
looking at jobs which are using those -- which boils down to:

centos7-autorelease-8c-8g
centos7-builder-8c-8g
lispflowmapping-csit-performance
netvirt-csit-hwvtep-openstack
netvirt-csit-multi-openstack-upgrade
netvirt-csit-multi-openstack

I think we should be looking very hard at a number of things:
  1) sizing of openstack nodes, i.e. "control_1"
  2) sizing of ODL nodes in netvirt CSIT
  3) use of "build-node: centos7-builder-8c-8g"
  4) getting performance stats out of individual CSIT nodes
  5) automating idle CPU/mem calculation (based on 4)

The point is: why do we need 8 cores for these functions?

To illustrate:
4c-16g is 30% cheaper
4c-4g is 50% cheaper
2c-8g is 66% cheaper

Also, I think we should start looking at more aggressively using
1c-4g/2c-1g nodes -- they are 37%/82%(!) cheaper than 2c-2g.

Abhijit, can we get this scheduled at the next TSC meeting (without
actual numbers)?

Thanks,
Robert