On Aug 2, 2018, at 1:38 AM, Dibya Das <dibyadas998@...> wrote:

Hi Luis,

I noticed that the build number ordering is incorrect. It detects the test-run as a keyword. It's a minor fix and I will send the patch soon.

The other thing is, index pattern. There are two ways we can fix this. One is to put build data of different types like fluorine, oxygen, etc under one index and have a source filter that can filter out the relevant build type and show in the visualization. The other way is to store the different build type data under different index and have separate index pattern set for the visualization. I'll weight both the options and decide which one will be fit for our case.

One more thing left is to check if the visualization to push exists or not. Here, to check the visualization exists takes 1 request. If it's present then nothing else. But if it's not then, 2 request. So in worst case, it takes 2 requests. But if we push the visualization format regardless, it creates if it's doesn't exist and updates if it doesn't. So we can take advantage of this and modify the dashboard structure at any build. 

Also, where do I add the documentation of the visualization config format? In the ODL wiki?

Thanks,

Dibya

On Sun, Jul 29, 2018, 5:09 AM Dibya Das <dibyadas998@...> wrote:

Thanks for the review Luis. :) I'll add the docs in the next patch. I might need some help with the creation of the sanbox job but I'll try it myself first. I'll get back if I'm not able to make it work.

Dibya

On Sun, Jul 29, 2018 at 4:18 AM, Luis Gomez <ecelgp@...> wrote:

OK Dibya, no major comments from me at this moment. Only thing I miss is some in-line documentation for some of the python functions you create (you can work on that in next patch if you want) and a sandbox run link to verify the code works.

BR/Luis

On Jul 26, 2018, at 10:54 AM, Dibya Das <dibyadas998@...> wrote:

Hi Luis and the Integration team,

I have updated the patch with new edits https://git.opendaylight.org/gerrit/#/c/73921/. Please do suggest changes if any. Apologies for the size of the patch.

Also, I'll add a documentation about the format of the visualization and dashboard configuration file through another patch once this is merged.

Thanks,

Dibya

On Thu, Jul 19, 2018 at 1:23 AM, Dibya Das <dibyadas998@...> wrote:

Hi Luis,

I went through your suggestions. I also felt the same way about grouping the config files in central templates and project specific dashboard config. It helps in less redundancy and more organization. So I'll push a patch implementing that soon. I have a little doubt regarding the requirement of another job to push the dashboard config. I did not get why that is necessary when the same job that pushes the data, can also push the configs as well.

Thanks,

Dibya

On Wed, Jul 18, 2018 at 12:33 AM, Luis Gomez <ecelgp@...> wrote:

Thanks for the update :)

On Jul 17, 2018, at 5:38 AM, Dibya Das <dibyadas998@...> wrote:

Hi Luis,

My traveling is done and I'll be resuming my work. I'll will test your suggestions and get back to you by tomorrow.

Thanks,

Dibya

On Fri, Jul 13, 2018 at 12:38 AM, Luis Gomez <ecelgp@...> wrote:

On Jul 12, 2018, at 1:45 AM, Dibya Das <dibyadas998@...> wrote:

Hi,

Thanks, Luis, for you review and suggestions. It'll take me some time to test the suggestions you have mentioned. I'll properly go through them and get back to you. And I tried removing the co_ords and it seems that Kibana cannot figure it out on it's own. Hence it's necessary to pass it as well. (Personally, I feel that this is good thing because I can configure the dashboard to look the way I want even if I don't have access to Kibana to modify it.)

OK, then lets leave that as a project choice.

Also, I'll be traveling on Monday and might be bit occupied over the weekends.

Now worries, you have done good work so far :)

Thanks :)

Dibya

On Thu, Jul 12, 2018 at 10:48 AM, Luis Gomez <ecelgp@...> wrote:

Hi Dibya,

Very good work, after looking at the yaml config files, I can clearly see what is required to configure the dashboards in Kibana :)

First, I would like to ask you couple of questions on the configuration:

1) This "field" in the visualization:

field: performance.pass-tests

I guess this is the field containing the data for the y-axis. I am not sure where "performance" above is coming from but what we really need is the "pass-test" value for all data records belonging to same job (e.g. openflowplugin-csit-1node-perf-bulkomatic-only-oxygen).

2) The coordinates params for the visualizations in the dashboard:

      co_ords:
        h: 3
        w: 6
        x: 0
        y: 3

is this something kibana can figure out by itself or do we have to always pass these?

Second, some thoughts after looking at your patch:

- I start to think we will need one dedicated jenkins job for pushing the dashboard configuration. This job will run independently from the jobs feeding the data (e.g. existing csit jobs).
- This new job will read the dashboard configuration files (see later) in an specific folder (e.g. test/dashboard) and will update kibana with the new configuration.
- We want projects in ODL (e.g. openflowplugin) to manipulate some aspect of the dashboards (e.g. title, plot labels, etc) but in general we want to have central control of the dashboard settings with an efficient amount of config settings (e.g. not creating individual visualizations for every single graph.

So based on my last thought above I think we can better organize the dashboard config files in 2 groups (note the examples below are to illustrate the idea, so do not take them literally):

1) Centralized visualization templates file (e.g. viz-templates.yaml) used to configure the visualizations. This has most of the visualization configuration parameters except for a few project specific values:

viz-template:
    type: histogram
    index_pattern: opendaylight-test-*
    id: opendaylight-test-feature
    desc: Test Pass/Fail results
    num_cat_axes: 1
    value_axes:
        y-axis:
            position: left
            title: No. of test cases
    seriesParams:
        pass:
            data_type: histogram
            mode: stacked
            label: Pass tests
            agg_id: 1
            value_axis: 1
        fail:
            data_type: histogram
            mode: stacked
            label: Fail tests
            agg_id: 2
            value_axis: 1
    aggs:
        pass:
            field: performance.pass-tests
            custom_label: Pass tests
            schema: metric
        fail:
            field: performance.fail-tests
            custom_label: Fail tests
            schema: metric   
        build:
            field: performance.test-run.keyword
            custom_label: Build Number
            schema: segment

viz-template:
    type: line
    index_pattern: opendaylight-test-*
    id: opendaylight-test-performance
    num_cat_axes: 1
    value_axes:
        y-axis:
            position: left
    seriesParams:
        y-axis:
            data_type: line
            mode: normal
            agg_id: 1
            value_axis: 1
    aggs:
        build:
            field: performance.test-run.keyword
            custom_label: Build Number
            schema: segment
        y-axis:
            schema: metric

Note that with a template definition like above we only need 2 templates for the time being.


2) Project specific dashboard file (e.g. openflowplugin-dashboard.yaml) used to configure the visualizations and the dashboard for a given project (e.g. openflowplugin):

dashboard:
  id: openflowplugin
  title: openflowplugin
  desc: Dashboard for openflowplugin project
  viz:
   1:
      id: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen-test
      title: Performance Bulkomatic Test Oxygen
      viz-template: opendaylight-test-feature
      job-name: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen
      co_ords:
        h: 3
        w: 6
        x: 0
        y: 3
    2:
      id: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen-rate
      title: Performance Bulkomatic Flow Add Rate Oxygen
      desc: Test to measure controller flow add rate
      viz-template: opendaylight-test-performance
      job-name: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen
      y-axis:
        title: sec
        series:
          1:
            field: performance.plots.rate.Config DS
            label: Config DS
          2:
            field: performance.plots.rate.OVS Switch
            label: OVS Switch
          3:
            field:  performance.plots.rate.Operational DS
            label: Operational DS
      co_ords:
        h: 3
        w: 6
        x: 6
        y: 3
    3:
      id: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen-time
      title: Performance Bulkomatic Flow Add Time Oxygen
      desc: Test to measure controller flow add time
      viz-template: opendaylight-test-performance
      job-name: openflowplugin-csit-1node-perf-bulkomatic-only-oxygen
      y-axis:
        title: sec
        series:
          1:
            field: performance.plots.time.Config DS
            label: Config DS
          2:
            field: performance.plots.time.OVS Switch
            label: OVS Switch
          3:
            field:  performance.plots.time.Operational DS
            label: Operational DS
      co_ords:
        h: 3
        w: 12
        x: 0
        y: 0

Let me know what you think about the above and any question you may have.

BR/Luis

> On Jul 11, 2018, at 3:29 AM, Dibya Das <dibyadas998@...> wrote:
>
> Hi,
>
> Here's the patch that I have sent https://git.opendaylight.org/gerrit/#/c/73921/. Please review and suggest improvements.
>
> Thanks,
> Dibya
>
> On Wed, Jul 11, 2018 at 11:42 AM, Luis Gomez <ecelgp@...> wrote:
> Thanks Dibya, looking forward to reviewing your patch :)
>
>
>> On Jul 10, 2018, at 2:25 PM, Dibya Das <dibyadas998@...> wrote:
>>
>> Hi,
>>
>> Work Update: The creation and pushing of dashboard and visualizations is finally completed! :D  I've made some modifications. The visualizations and dashboard configuration are written on a YAML file. Then, those YAML files are parsed to generate the corresponding JSON to be fed to Kibana. Since it's YAML, the dashboard and visualizations can be fairly easily configured by anyone. Tomorrow, I'll push and send a patch for these changes. Here are the config files [ visualization, dashboard ] in case you are interested.
>>
>> I've faced some roadblocks as well. Here is an issue that I raised in Kibana repo for a bug that I encountered. They mentioned that an experimental API exists that allows programmatically editing the saved objects a.k.a visualizations, dashboard &  index-patterns in Kibana but since, it's experimental, it'll go through breaking changes, I presume. Also, in the future versions, directly editing the .kibana index might not be allowed anymore depending on the environment.
>>
>> Thanks,
>> Dibya
>>
>>
>> On Mon, Jul 9, 2018 at 11:38 AM, Dibya Das <dibyadas998@...> wrote:
>> Hi Luis,
>>
>> That's great! I'll incorporate this as soon as I finish automating the basic part of a dashboard creation.
>>
>> Thanks,
>> Dibya
>>
>> On Mon, Jul 9, 2018 at 4:32 AM, Luis Gomez <ecelgp@...> wrote:
>> OK Dibya, here is how we can retrieve the y-axis value from jenkins:
>>
>> 1) Get the job configuration, e.g. for openflowplugin-csit-1node-perf-bulkomatic-only-oxygen job, fetch URL:
>>
>> https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-perf-bulkomatic-only-oxygen/config.xml
>>
>> 2) Now use an XML parser (e.g. xpath) to find the y-axis (and title) for a given csv file:
>>
>> echo `xpath config.xml '//file[contains(text(),"rate.csv")]/../../../yaxis/text()' 2> /dev/null`
>> flow_mods/sec
>> echo `xpath config.xml '//file[contains(text(),"rate.csv")]/../../../title/text()' 2> /dev/null`
>> Flow Add rate
>> echo `xpath config.xml '//file[contains(text(),"time.csv")]/../../../yaxis/text()' 2> /dev/null`
>> seconds
>> echo `xpath config.xml '//file[contains(text(),"time.csv")]/../../../title/text()' 2> /dev/null`
>> Flow Add time
>>
>> Let me know if you have further questions.
>>
>> BR/Luis
>>
>> > On Jul 6, 2018, at 3:13 PM, Luis Gomez <ecelgp@...> wrote:
>> >
>> > Correct, for now lets assume all that is under "plots" in the json update are performance metrics. Now as you have probably figured out there is a piece of information missing in the json body which is the y-axis units (e.g. flowmod/sec and seconds) in the performance dashboards. I am not very sure at this moment how we are going to retrieve these values from jenkins (only place this is known), I will give some thought over the weekend and will let you know.
>> >
>> >> On Jul 6, 2018, at 11:26 AM, Dibya Das <dibyadas998@...> wrote:
>> >>
>> >> On another note, I would really like to know what  Flow Rate and Time mean and their sub-labels -> Config DS, Operational DS and OVS Switch, that appear in Jenkins. As much as I can understand, I believe these are some metrics that indicate performance of the tests.
>> >>
>> >> Thanks,
>> >> Dibya
>> >>
>> >>
>> >> On Fri, Jul 6, 2018 at 11:46 PM, Dibya Das <dibyadas998@...> wrote:
>> >> Hi Luis and the Integration team,
>> >>
>> >> I've created a dashboard to show how the data would look there. Here is the link to the screenshot. Right now, it's just dummy data I've added to make the trend noticeable. I'll now proceed to automate this and re-write parts of the automation script from scratch. Please do suggest changes to the way you would like to see the data presented in the dashboard.
>> >>
>> >> Thanks,
>> >> Dibya
>> >>
>> >>
>> >> On Wed, Jul 4, 2018 at 12:10 PM, Dibya Das <dibyadas998@...> wrote:
>> >> Hi Luis,
>> >>
>> >> Thanks for this detail. I'm probably going to start some parts from scratch. I'll post doubts here if I face any.
>> >>
>> >> Regards,
>> >> Dibya
>> >>
>> >>
>> >>
>> >> On Wed, Jul 4, 2018 at 2:16 AM, Luis Gomez <ecelgp@...> wrote:
>> >> Hi Dibya,
>> >>
>> >> Lets open this conversation to the integration list. As I told you in private I do not think the automatic dashboard creation was tested in the last internship, so let me give you some context of what we are looking for here.
>> >>
>> >> For a job like this: https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-perf-bulkomatic-only-oxygen/293/ (you already have been working on this in sandbox)
>> >>
>> >> We expect the framework to automatically create 3 dashboards (if not created already) in kibana:
>> >>
>> >> - Test result trend (pass test + fail test) for a build: https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-perf-bulkomatic-only-oxygen/293/robot/graph?zoomSignificant=true&failedOnly=false&criticalOnly=false&maxBuildsToShow=0&hd=true
>> >>
>> >> - Flow Add Rate in https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-perf-bulkomatic-only-oxygen/plot/Java%20Performance/
>> >>
>> >> - Flow Add Time in https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-perf-bulkomatic-only-oxygen/plot/Java%20Performance/
>> >>
>> >> The y-axis (pass test, fail-test, rate, time) as well as the x-axis (test-run) values for the above are in the json update we send to ES database:
>> >>
>> >> {
>> >>
>> >> 
>> >>     "jenkins-silo": "releng",
>> >>
>> >> 
>> >>     "start-time": "20180606 03:27:11.640",
>> >>
>> >> 
>> >>     "@timestamp": "2018-06-06T03:28:41.221224Z",
>> >>
>> >> 
>> >>     "project": "opendaylight",
>> >>
>> >> 
>> >>     "pass-tests": "9",
>> >>
>> >> 
>> >>     "fail-tests": "0",
>> >>
>> >>      "test-name": "openflowplugin-csit-1node-per
>> >> f-bulkomatic-only-oxygen",
>> >>
>> >> 
>> >>     "test-type": "performance",
>> >>
>> >> 
>> >>     "plots": {
>> >>          "rate": {
>> >>
>> >> 
>> >>             "Config DS": 4633.49087202,
>> >>
>> >> 
>> >>             "OVS Switch": 4132.06065865,
>> >>
>> >> 
>> >>             "Operational DS": 2114.83557153
>> >>
>> >> 
>> >>         "time": {
>> >>
>> >> 
>> >>             "Config DS": 21.582,
>> >>
>> >> 
>> >>             "OVS Switch": 24.201,
>> >>
>> >> 
>> >>             "Operational DS": 47.285
>> >>
>> >> 
>> >>         }
>> >>
>> >> 
>> >>     },
>> >>
>> >> 
>> >>     "test-run": "265",
>> >>
>> >>      "id": "openflowplugin-csit-1node-per
>> >> f-bulkomatic-only-oxygen-265",
>> >>
>> >> 
>> >>     "duration": "0:01:17.883000",
>> >>
>> >> 
>> >>     "subject": "test"
>> >>
>> >>  }
>> >>
>> >> Please let me know if you have any question.
>> >>
>> >> BR/Luis
>> >>
>> >>
>> >>> On Jul 3, 2018, at 6:58 AM, Dibya Das <dibyadas998@...> wrote:
>> >>>
>> >>> Hi Luis,Thanh,
>> >>>
>> >>> Seems like there are more problems than I anticipated. Even after modifying the JSON to satisfy the new mapping types, there is yet another error that is appearing. This & this describes the error I'm facing. I'm working on it to fix it. I have a doubt now. Was the script written by the previous intern working and able to display the viz in the earlier version of ES?  I just tried that, to be sure, and it did not work.
>> >>>
>> >>> Regards,
>> >>> Dibya
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>
>