Date   

Re: Issue regarding Multiple Certificates in Odl KeyStore- Looking for guidance.

Ajay Deep Singh <ajay.deep.singh@...>
 

Hi All,

To add more on below mail:
I have excatly tried adding Certificates in the Format it was reflected in :  Endpoint.
I am not able to find if Multiple set of Certificates are supported in ODL or not .?

Please can anyone have a look and point out if am doing something wrong..?
ODL version : NEON.

Followed Format:  as on Rest endpoint

**(config)private-key
{
  "netconf-keystore:private-key": [
    {
      "netconf-keystore:name": "Optional.empty",
      "netconf-keystore:data": "Optional.empty",
      "netconf-keystore:certificate-chain": [
        "Optional.empty"
      ]
    }
  ]
}

**(config)trusted-certificate

{
  "netconf-keystore:trusted-certificate": [
    {
      "netconf-keystore:name": "Optional.empty",
      "netconf-keystore:certificate": "Optional.empty"
    }
  ]
}

Exception I encounter in Karaf Logs : 

2020-05-05T12:44:38,973 | INFO  | nioEventLoopGroupCloseable-3-4 | AbstractNetconfSessionNegotiator | 352 - org.opendaylight.netconf.netty-util - 1.6.1 |  -  | Unexpected error during
 negotiation
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[61:io.netty.codec:4.1.34.Final]

2020-05-05T12:44:38,980 | WARN  | nioEventLoopGroupCloseable-3-4 | AbstractChannelHandlerContext    | 63 - io.netty.common - 4.1.34.Final |  -  | An exception 'java.lang.IllegalStateE
xception: complete already: NetconfSessionPromise@4fca1aee(failure: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: unknown_ca)' [enable DEB
UG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: unknown_ca
 
2020-05-05T12:44:38,988 | WARN  | nioEventLoopGroupCloseable-3-5 | DefaultPromise                   | 63 - io.netty.common - 4.1.34.Final |  -  | An exception was thrown by org.openda
ylight.netconf.nettyutil.AbstractNetconfSessionNegotiator$$Lambda$1162/51785401.operationComplete()
java.lang.IllegalStateException: Ssl handshake was not successful
at com.google.common.base.Preconditions.checkState(Preconditions.java:507) ~[38:com.google.guava:25.1.0.jre]


Regards,
Ajay


From: Ajay Deep Singh
Sent: Tuesday, May 5, 2020 1:45 PM
To: dev@... <dev@...>; integration-dev@... <integration-dev@...>
Cc: Tobias Mole <tobias.mole@...>
Subject: Issue regarding Multiple Certificates in Odl KeyStore- Looking for guidance.
 
Hi All,

Hope everyone is doing good.

I am running into issue where experiencing Connection Failure() after uploading two sets of Certificates in ODl KeyStore and try connecting  Netconf supported external Node(Simulator) over TLS on to ODL.  

Also when I individualy test below Certificates and try adding external Node Connection is successful.

Exception I encounter in Karaf Logs : 

Made Post Request on below Endpoint and we can find output via Get Endpoint.


{
  "keystore": {
    "trusted-certificate": [
      {
        "name": "xNF_CA_certificate_0_1",
        "certificate": "MIIDeDCCAmCgAwIBAgIUV4anCcc2IeB4jcc7j5UKa2T/fkMwDQYJKoZIhvcNAQEL\nBQAwOjEYMBYGA1UEAwwPTXlfTWFuYWdlbWVudENBMREwDwYDVQQKDAhQcmltZUtl\neTELMAkGA1UEBhMCU0UwHhcNMjAwNTA1MDk1MzQzWhcNMjEwNTA1MDk1MzQzWjA6\nMRgwFgYDVQQDDA9NeV9NYW5hZ2VtZW50Q0ExETAPBgNVBAoMCFByaW1lS2V5MQsw\nCQYDVQQGEwJTRTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANoyp/+h\n8vWcsMmbfouD+GdObxGiagfp/RQAIhFZDpu1f/zL8UZxmYd0D6LLNY5GI3BkC1Th\nXSbnPFBXr4Oh4AYSBnASCM1q1lynV+od1Ge49OlzsQpU96+4x+8GMayT7lMQD3Qc\nDJn8BXQvfo04sH9P8pYuPYiaaCTL4W9ezvYzflDwuGu6MyYeVVRmUhtU028Oaa7G\nVdLJyFab00pzefIk26RW0CC8FcjTOZnZK/9KgfUIn6HAA7twBIAssNuiHW08269v\nvHV6OMcVCFKpuw8flchONixjAyxTh4vv+x7RN14E8OD8vlnvf52QbiiSCPNLDL7g\nEEsjOn+9QtCgn+cCAwEAAaN2MHQwDwYDVR0TAQH/BAUwAwEB/zAfBgNVHSMEGDAW\ngBTWROzosR50oxCOjVHkj886aQbM5jARBgNVHSAECjAIMAYGBFUdIAAwHQYDVR0O\nBBYEFNZE7OixHnSjEI6NUeSPzzppBszmMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG\n9w0BAQsFAAOCAQEAbhAskQ6TrFQhRWhZroopTE58MexO6tl0AXGn1p6wdXcWDQOZ\ndgWVOwUcad2gcIAvj+XRRNaW5HGxmpY2syxCMzjVa/kCXCllGTQXhhTS23lJQc4n\nWRQDJFeEhEP1KrXBytyZpb8nawDmpgtpAMqIdvytMZYryvjZvguUzZ+mlZWAs9lV\nwlLO5rCANaGWeEQRArn0Cv2oktoEQUWVdZvlL+hJ16FwEGEPDz6lnANyJVv5U+T1\nTJ9ps91ZOOt1aOF5/rPXFTTAH9nWpbBn0P7u+AZegtvM3nlZ6CTpl3Pwb/jfrPtU\nj+EsQzUmNFHuyscbpZbfeFECSBcJ3nhKcKMiXg=="
      },
      {
        "name": "xNF_CA_certificate_0_0",
        "certificate": "MIID7TCCAtWgAwIBAgIJAMtE1NGAR5KoMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYD\nVQQGEwJDWjEWMBQGA1UECAwNU291dGggTW9yYXZpYTENMAsGA1UEBwwEQnJubzEP\nMA0GA1UECgwGQ0VTTkVUMQwwCgYDVQQLDANUTUMxEzARBgNVBAMMCmV4YW1wbGUg\nQ0ExIjAgBgkqhkiG9w0BCQEWE2V4YW1wbGVjYUBsb2NhbGhvc3QwHhcNMTQwNzI0\nMTQxOTAyWhcNMjQwNzIxMTQxOTAyWjCBjDELMAkGA1UEBhMCQ1oxFjAUBgNVBAgM\nDVNvdXRoIE1vcmF2aWExDTALBgNVBAcMBEJybm8xDzANBgNVBAoMBkNFU05FVDEM\nMAoGA1UECwwDVE1DMRMwEQYDVQQDDApleGFtcGxlIENBMSIwIAYJKoZIhvcNAQkB\nFhNleGFtcGxlY2FAbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEArD3TDHPAMT2Z84orK4lMlarbgooIUCcRZyLe+QM+8KY8Hn+mGaxPEOTS\nL3ywszqefB/Utm2hPKLHX684iRC14ID9WDGHxPjvoPArhgFhfV+qnPfxKTgxZC12\nuOj4u1V9y+SkTCocFbRfXVBGpojrBuDHXkDMDEWNvr8/52YCv7bGaiBwUHolcLCU\nbmtKILCG0RNJyTaJpXQdAeq5Z1SJotpbfYFFtAXB32hVoLug1dzl2tjG9sb1wq3Q\naDExcbC5w6P65qOkNoyym9ne6QlQagCqVDyFn3vcqkRaTjvZmxauCeUxXgJoXkyW\ncm0lM1KMHdoTArmchw2Dz0yHHSyDAQIDAQABo1AwTjAdBgNVHQ4EFgQUc1YQIqjZ\nsHVwlea0AB4N+ilNI2gwHwYDVR0jBBgwFoAUc1YQIqjZsHVwlea0AB4N+ilNI2gw\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAI/1KH60qnw9Xs2RGfi0/\nIKf5EynXt4bQX8EIyVKwSkYKe04zZxYfLIl/Q2HOPYoFmm3daj5ddr0ZS1i4p4fT\nUhstjsYWvXs3W/HhVmFUslakkn3PrswhP77fCk6eEJLxdfyJ1C7Uudq2m1isZbKi\nh+XF0mG1LxJaDMocSz4eAya7M5brwjy8DoOmA1TnLQFCVcpn+sCr7VC4wE/JqxyV\nhBCk/MuGqqM3B1j90bGFZ112ZOecyE0EDSr6IbiRBtmeNbEwOFjKXhNLYdxpBZ9D\n8A/368OckZkCrVLGuJNxK9UwCVTe8IhotHUqU9EqFDmxdV8oIdU/OzUwwNPA/Bd/\n9g=="
      }
    ],
    "private-key": [
      {
        "name": "ODL_private_key_1",
        "certificate-chain": [
          "MIID8zCCAtugAwIBAgIUZzoKqogek7AqXwZM0jz65H1szLswDQYJKoZIhvcNAQEL\nBQAwOjEYMBYGA1UEAwwPTXlfTWFuYWdlbWVudENBMREwDwYDVQQKDAhQcmltZUtl\neTELMAkGA1UEBhMCU0UwHhcNMjAwNTA1MDk1NTA1WhcNMjEwNTA1MDk1MzQzWjB8\nMRYwFAYDVQQDDA1zZG5jLm9uYXAub3JnMQ0wCwYDVQQLDARPTkFQMRkwFwYDVQQK\nDBBMaW51eC1Gb3VuZGF0aW9uMRYwFAYDVQQHDA1TYW4tRnJhbmNpc2NvMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAJ6wPqEZVkc5nI8nGed3E/4O0LPUKNp9MFucONrHVQvrwzEv\nvsLHoOnx4esXhqq5pF+lhRPBuEUN5GUIOVzZGsU0DPpebBbnRDqaQv3O6UkOZRPH\ngj0CWzuwEfd7Jsl4WJEHfgynaWD+rRHS4nqUCjhMZvDhHhLke6ORptC/akUq5kPC\nfSZP9JgC22Ry+P/snmR7J1kIcD6w9WQh2H2LfQR4pL61dV5fJTgvbolLg9EGyRkE\njmd6um/C4KE1j9nZcVtvrFg4ge2DPzYWDFpnN6pQf5Kx8Idx7P1n5ZGu4dcn1Txs\n5yVk/63jm7LUn94eXqHWiDT8omYZ93dV3Fq8sm8CAwEAAaOBrjCBqzAMBgNVHRMB\nAf8EAjAAMB8GA1UdIwQYMBaAFNZE7OixHnSjEI6NUeSPzzppBszmMCIGA1UdEQQb\nMBmCC2V4YW1wbGUuY29tggpzYW1wbGUuY29tMCcGA1UdJQQgMB4GCCsGAQUFBwMC\nBggrBgEFBQcDBAYIKwYBBQUHAwEwHQYDVR0OBBYEFI1XdhOJ7RzXWgN45evIYtEL\n2OplMA4GA1UdDwEB/wQEAwIF4DANBgkqhkiG9w0BAQsFAAOCAQEAVpPWQtOWrYsH\nov1jAt+fAgVXkgzinKILPiwNITzJBnP3Y329vROLVbktie0HAxd89xhZX02E5wxa\nK2bTKqrwXA8H7inWK9U4fcQ9j9qPhwzkpd6AZAOdWFZ6gCrQpz5jLaJlHA7ePTxt\nVLQiD/7DHtPTPpVMXCVG4R9zIZaWpC8EntlMy9Fdhge4+PMni4OdwCNnjnrdPbzU\nuNGz8Ri3PVNn/o+v/N3MkcvlqVJpzcryldWZIqdZnVA5liiT7deQB4HzMhoPABuH\npUstTj8Gk1b4aSFkGxSlgDhBdY80XB9n0VzySpDn+MVgAOGZ1GrCpnZiOHE5Ujld\nVblGRGOFdQ=="
        ],
        "data": "MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCesD6hGVZHOZyP\nJxnndxP+DtCz1CjafTBbnDjax1UL68MxL77Cx6Dp8eHrF4aquaRfpYUTwbhFDeRl\nCDlc2RrFNAz6XmwW50Q6mkL9zulJDmUTx4I9Als7sBH3eybJeFiRB34Mp2lg/q0R\n0uJ6lAo4TGbw4R4S5HujkabQv2pFKuZDwn0mT/SYAttkcvj/7J5keydZCHA+sPVk\nIdh9i30EeKS+tXVeXyU4L26JS4PRBskZBI5nerpvwuChNY/Z2XFbb6xYOIHtgz82\nFgxaZzeqUH+SsfCHcez9Z+WRruHXJ9U8bOclZP+t45uy1J/eHl6h1og0/KJmGfd3\nVdxavLJvAgMBAAECggEAULr/EbwvHYscr+NHlW8KF7z3xhTQN99qg6Cld//ir27g\n3ZwZo0QmA/6PXaeZVi+dhR9S5WsvUMoxgIi/jhHRQbQqShFRUhGTr3CdVjw2iHkv\nSPG9u0YAngwEB/B6GhMEk7vJD6z3umW2LJQprgy0alHMKQ2N3DSrBdrqj9ZhvQ+5\njPTISU1CggkRMyCEOFA9CNffWl+gLWDKhlwTzS87nCbikxjimde1uuRN36PJofbr\nAOYTeLPJ3nF0IwmZ0QaH6nYPePeFn5p4+v1YgJ3HhEaaP7MvNb5tbeN90CMcv+tk\nI/vBgKR/BsXCCzuvnWxeTZjob8cfBkih6goFe7s5qQKBgQDXjORY+ZJsPh+DnePf\nXJSIAZg6SjSjmnPU8WvrEJcm72Kf/6YNPOI3VIjRG4auLYZhfEdIgBZFl/rfCi23\no4TRkCL2l3PoqC+Bkn5cWjngPwy1ZjUKMgUwA1nb42UNA+ZiQ+JrxPpj7VkaitU9\n80zgykMwOkEoVNsVG/xCjgNB5QKBgQC8d7BM5kPtHD4my2/U608ES7TgzTOemFxc\n0C5MQej1bQFzXeVykU+5HLUg7Tj1QnHJHpu/PqVXPdSIWsQljtW78U6dW+oWRYDC\n7vedlq0PaX92AK4kFSU20ARhgd7SF237fqJpSJ0qGMnjxooyzYK5AM5Zw2ArNXN+\ncz7C5zdtwwKBgHfxWo94MNYyq26dyVNdPPmBiafF6wyuwY3Y2WhwTuvHgWXu6AOt\nwWNwksFpQ8RgaMqOUUwymjIXo+3AyDfLsfi67/mvxTvNesHrNo1FV7dE3lmCijkm\nJGTXOIV3P5/Cb8x2YeP+NCauyEpQvuA+JqVfZ4vKnr9GmLJiZDjUtVNRAoGAfmRK\nMJgwL6DEMaI3InSYLkpTSHt1RcPqFThsBCojgAwVb/glezrmWz9cXXOZwilIQ9eq\nHYx2t9Y4ZSVx67oprnv6+977UupTmudF2kKFVxOqLknwfzRLG8JPXdEJzJ133bSk\nWLUg6W1sX7vzTHS0WTuoJ1y2buPOy5ugGHeVnZ8CgYB2B1vCvafjkrbl9iNcfVs6\nxEvQqkrW4UMHRCci54XT1f7H7Oih7Bi08OL9wyEdwaNK2wqb03WGZoaqz7fdAiU/\naQSBar6Dr3gTnZuJz+JGNJsS+VAbr6aGtC+bHjwqGdnIvmGSEeDX6fdDOewT2Q2s\nXZ+LHtO3BIfCILGGmog1rA=="
      },
      {
        "name": "ODL_private_key_0",
        "certificate-chain": [
          "MIIECTCCAvGgAwIBAgIBBzANBgkqhkiG9w0BAQsFADCBjDELMAkGA1UEBhMCQ1ox\nFjAUBgNVBAgMDVNvdXRoIE1vcmF2aWExDTALBgNVBAcMBEJybm8xDzANBgNVBAoM\nBkNFU05FVDEMMAoGA1UECwwDVE1DMRMwEQYDVQQDDApleGFtcGxlIENBMSIwIAYJ\nKoZIhvcNAQkBFhNleGFtcGxlY2FAbG9jYWxob3N0MB4XDTE1MDczMDA3MjcxOFoX\nDTM1MDcyNTA3MjcxOFowgYUxCzAJBgNVBAYTAkNaMRYwFAYDVQQIDA1Tb3V0aCBN\nb3JhdmlhMQ8wDQYDVQQKDAZDRVNORVQxDDAKBgNVBAsMA1RNQzEXMBUGA1UEAwwO\nZXhhbXBsZSBjbGllbnQxJjAkBgkqhkiG9w0BCQEWF2V4YW1wbGVjbGllbnRAbG9j\nYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAueCQaNQWoNmF\nK6LKu1p8U8ZWdWg/PvDdLsJyzfzl/Qw4UA68SfFNaY06zZl8QB9W02nr5kWeeMY0\nVA3adrPgOlvfx3oWlFbkETnMaN4OT3WTQ0Wt6jAWZDzVfopwpJPAzRPxACDftIqF\nGagYcF32hZlVNqqnVdbXh0S0EViweqp/dbG4VDUHSNVbglc+u4UbEzNIFXMdEFsJ\nZpkynOmSiTsIATqIhb+2srkVgLwhfkC2qkuHQwAHdubuB07ObM2z01UhyEdDvEYG\nHwtYAGDBL2TAcsI0oGeVkRyuOkV0QY0UN7UEFI1yTYw+xZ42HgFx3uGwApCImxhb\nj69GBYWFqwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVu\nU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUXGpLeLnh2cSDARAV\nA7KrBxGYpo8wHwYDVR0jBBgwFoAUc1YQIqjZsHVwlea0AB4N+ilNI2gwDQYJKoZI\nhvcNAQELBQADggEBAJPV3RTXFRtNyOU4rjPpYeBAIAFp2aqGc4t2J1c7oPp/1n+l\nZvjnwtlJpZHxMM783e2ryDQ6dkvXDf8kpwKlg3U3mkJ3xKkDdWrM4QwghXdCN519\naa9qmu0zdFL+jUAaWlQ5tsceOrvbusCcbMqiFGk/QfpHqPv52SVWbYyUx7IX7DE+\nUjgsLHycfV/tlcx4ZE6soTzl9VdgSL/zmzG3rjsr58J80rXckLgBhvijgBlIAJvW\nfC7D0vaouvBInSFXymdPVoUDZ30cdGLf+hI/i/TfsEMOinLrXVdkSGNo6FXAHKSv\nXeB9oFKSzhQ7OPyRyqvEPycUSw/qD6FVr80oDDc="
        ],
        "data": "MIIEpAIBAAKCAQEAueCQaNQWoNmFK6LKu1p8U8ZWdWg/PvDdLsJyzfzl/Qw4UA68\nSfFNaY06zZl8QB9W02nr5kWeeMY0VA3adrPgOlvfx3oWlFbkETnMaN4OT3WTQ0Wt\n6jAWZDzVfopwpJPAzRPxACDftIqFGagYcF32hZlVNqqnVdbXh0S0EViweqp/dbG4\nVDUHSNVbglc+u4UbEzNIFXMdEFsJZpkynOmSiTsIATqIhb+2srkVgLwhfkC2qkuH\nQwAHdubuB07ObM2z01UhyEdDvEYGHwtYAGDBL2TAcsI0oGeVkRyuOkV0QY0UN7UE\nFI1yTYw+xZ42HgFx3uGwApCImxhbj69GBYWFqwIDAQABAoIBAQCZN9kR8DGu6V7y\nt0Ax68asL8O5B/OKaHWKQ9LqpVrXmikZJOxkbzoGldow/CIFoU+q+Zbwu9aDa65a\n0wiP7Hoa4Py3q5XNNUrOQDyU/OYC7cI0I83WS0lJ2zOJGYj8wKae5Z81IeQFKGHK\n4lsy1OGPAvPRGh7RjUUgRavA2MCwe07rWRuDb/OJFe4Oh56UMEjwMiNBtMNtncog\nj1vr/qgRJdf9tf0zlJmLvUJ9+HSFFV9I/97LJyFhb95gAfHkjdVroLVgT3Cho+4P\nWtZaKCIGD0OwfOG2nLV4leXvRUk62/LMlB8NI9+JF7Xm+HCKbaWHNWC7mvWSLV58\nZl4AbUWRAoGBANyJ6SFHFRHSPDY026SsdMzXR0eUxBAK7G70oSBKKhY+O1j0ocLE\njI2krHJBhHbLlnvJVyMUaCUOTS5m0uDw9hgSsAqeSL3hL38kxVZw+KNG9Ouno1Fl\nKnE/xXHlPQyeGs/P8nAMzHZxQtEsQdQayJEhK2XXHTsy7Q3MxDisfVJ1AoGBANfD\n34gB+OMx6pwj7zk3qWbYXSX8xjCZMR0ciko+h4xeMP2N8B0oyoqC+v1ABMAtJ3wG\nsGZd0hV9gwM7OUM3SEwkn6oeg1GemWLcn4rlSmTnZc4aeVwrEWlnSNFX3s4g9l4u\nk8Ugu4MVJYqH8HuDQ5Ggl6/QAwPzMSEdCW0O+jOfAoGAIBRbegC5+t6m7Yegz4Ja\ndxV1g98K6f58x+MDsQu4tYWV4mmrQgaPH2dtwizvlMwmdpkh+LNWNtWuumowkJHc\nakIFo3XExQIFg6wYnGtQb4e5xrGa2xMpKlIJaXjb+YLiCYqJDG2ALFZrTrvuU2kV\n9a5qfqTc1qigvNolTM0iaaUCgYApmrZWhnLUdEKV2wP813PNxfioI4afxlpHD8LG\nsCn48gymR6E+Lihn7vuwq5B+8fYEH1ISWxLwW+RQUjIneNhy/jjfV8TgjyFqg7or\n0Sy4KjpiNI6kLBXOakELRNNMkeSPopGR2E7v5rr3bGD9oAD+aqX1G7oJH/KgPPYd\nVl7+ZwKBgQDcHyWYrimjyUgKaQD2GmoO9wdcJYQ59ke9K+OuGlp4ti5arsi7N1tP\nB4f09aeELM2ASIuk8Q/Mx0jQFnm8lzRFXdewgvdPoZW/7VufM9O7dGPOc41cm2Dh\nyrTcXx/VmUBb+/fnXVEgCv7gylp/wtdTGHQBQJHR81jFBz0lnLj+gg=="
      }
    ]
  }
}





Regards,
Ajay


Issue regarding Multiple Certificates in Odl KeyStore- Looking for guidance.

Ajay Deep Singh <ajay.deep.singh@...>
 

Hi All,

Hope everyone is doing good.

I am running into issue where experiencing Connection Failure() after uploading two sets of Certificates in ODl KeyStore and try connecting  Netconf supported external Node(Simulator) over TLS on to ODL.  

Also when I individualy test below Certificates and try adding external Node Connection is successful.

Exception I encounter in Karaf Logs : 

Made Post Request on below Endpoint and we can find output via Get Endpoint.


{
  "keystore": {
    "trusted-certificate": [
      {
        "name": "xNF_CA_certificate_0_1",
        "certificate": "MIIDeDCCAmCgAwIBAgIUV4anCcc2IeB4jcc7j5UKa2T/fkMwDQYJKoZIhvcNAQEL\nBQAwOjEYMBYGA1UEAwwPTXlfTWFuYWdlbWVudENBMREwDwYDVQQKDAhQcmltZUtl\neTELMAkGA1UEBhMCU0UwHhcNMjAwNTA1MDk1MzQzWhcNMjEwNTA1MDk1MzQzWjA6\nMRgwFgYDVQQDDA9NeV9NYW5hZ2VtZW50Q0ExETAPBgNVBAoMCFByaW1lS2V5MQsw\nCQYDVQQGEwJTRTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANoyp/+h\n8vWcsMmbfouD+GdObxGiagfp/RQAIhFZDpu1f/zL8UZxmYd0D6LLNY5GI3BkC1Th\nXSbnPFBXr4Oh4AYSBnASCM1q1lynV+od1Ge49OlzsQpU96+4x+8GMayT7lMQD3Qc\nDJn8BXQvfo04sH9P8pYuPYiaaCTL4W9ezvYzflDwuGu6MyYeVVRmUhtU028Oaa7G\nVdLJyFab00pzefIk26RW0CC8FcjTOZnZK/9KgfUIn6HAA7twBIAssNuiHW08269v\nvHV6OMcVCFKpuw8flchONixjAyxTh4vv+x7RN14E8OD8vlnvf52QbiiSCPNLDL7g\nEEsjOn+9QtCgn+cCAwEAAaN2MHQwDwYDVR0TAQH/BAUwAwEB/zAfBgNVHSMEGDAW\ngBTWROzosR50oxCOjVHkj886aQbM5jARBgNVHSAECjAIMAYGBFUdIAAwHQYDVR0O\nBBYEFNZE7OixHnSjEI6NUeSPzzppBszmMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG\n9w0BAQsFAAOCAQEAbhAskQ6TrFQhRWhZroopTE58MexO6tl0AXGn1p6wdXcWDQOZ\ndgWVOwUcad2gcIAvj+XRRNaW5HGxmpY2syxCMzjVa/kCXCllGTQXhhTS23lJQc4n\nWRQDJFeEhEP1KrXBytyZpb8nawDmpgtpAMqIdvytMZYryvjZvguUzZ+mlZWAs9lV\nwlLO5rCANaGWeEQRArn0Cv2oktoEQUWVdZvlL+hJ16FwEGEPDz6lnANyJVv5U+T1\nTJ9ps91ZOOt1aOF5/rPXFTTAH9nWpbBn0P7u+AZegtvM3nlZ6CTpl3Pwb/jfrPtU\nj+EsQzUmNFHuyscbpZbfeFECSBcJ3nhKcKMiXg=="
      },
      {
        "name": "xNF_CA_certificate_0_0",
        "certificate": "MIID7TCCAtWgAwIBAgIJAMtE1NGAR5KoMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYD\nVQQGEwJDWjEWMBQGA1UECAwNU291dGggTW9yYXZpYTENMAsGA1UEBwwEQnJubzEP\nMA0GA1UECgwGQ0VTTkVUMQwwCgYDVQQLDANUTUMxEzARBgNVBAMMCmV4YW1wbGUg\nQ0ExIjAgBgkqhkiG9w0BCQEWE2V4YW1wbGVjYUBsb2NhbGhvc3QwHhcNMTQwNzI0\nMTQxOTAyWhcNMjQwNzIxMTQxOTAyWjCBjDELMAkGA1UEBhMCQ1oxFjAUBgNVBAgM\nDVNvdXRoIE1vcmF2aWExDTALBgNVBAcMBEJybm8xDzANBgNVBAoMBkNFU05FVDEM\nMAoGA1UECwwDVE1DMRMwEQYDVQQDDApleGFtcGxlIENBMSIwIAYJKoZIhvcNAQkB\nFhNleGFtcGxlY2FAbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEArD3TDHPAMT2Z84orK4lMlarbgooIUCcRZyLe+QM+8KY8Hn+mGaxPEOTS\nL3ywszqefB/Utm2hPKLHX684iRC14ID9WDGHxPjvoPArhgFhfV+qnPfxKTgxZC12\nuOj4u1V9y+SkTCocFbRfXVBGpojrBuDHXkDMDEWNvr8/52YCv7bGaiBwUHolcLCU\nbmtKILCG0RNJyTaJpXQdAeq5Z1SJotpbfYFFtAXB32hVoLug1dzl2tjG9sb1wq3Q\naDExcbC5w6P65qOkNoyym9ne6QlQagCqVDyFn3vcqkRaTjvZmxauCeUxXgJoXkyW\ncm0lM1KMHdoTArmchw2Dz0yHHSyDAQIDAQABo1AwTjAdBgNVHQ4EFgQUc1YQIqjZ\nsHVwlea0AB4N+ilNI2gwHwYDVR0jBBgwFoAUc1YQIqjZsHVwlea0AB4N+ilNI2gw\nDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQUFAAOCAQEAI/1KH60qnw9Xs2RGfi0/\nIKf5EynXt4bQX8EIyVKwSkYKe04zZxYfLIl/Q2HOPYoFmm3daj5ddr0ZS1i4p4fT\nUhstjsYWvXs3W/HhVmFUslakkn3PrswhP77fCk6eEJLxdfyJ1C7Uudq2m1isZbKi\nh+XF0mG1LxJaDMocSz4eAya7M5brwjy8DoOmA1TnLQFCVcpn+sCr7VC4wE/JqxyV\nhBCk/MuGqqM3B1j90bGFZ112ZOecyE0EDSr6IbiRBtmeNbEwOFjKXhNLYdxpBZ9D\n8A/368OckZkCrVLGuJNxK9UwCVTe8IhotHUqU9EqFDmxdV8oIdU/OzUwwNPA/Bd/\n9g=="
      }
    ],
    "private-key": [
      {
        "name": "ODL_private_key_1",
        "certificate-chain": [
          "MIID8zCCAtugAwIBAgIUZzoKqogek7AqXwZM0jz65H1szLswDQYJKoZIhvcNAQEL\nBQAwOjEYMBYGA1UEAwwPTXlfTWFuYWdlbWVudENBMREwDwYDVQQKDAhQcmltZUtl\neTELMAkGA1UEBhMCU0UwHhcNMjAwNTA1MDk1NTA1WhcNMjEwNTA1MDk1MzQzWjB8\nMRYwFAYDVQQDDA1zZG5jLm9uYXAub3JnMQ0wCwYDVQQLDARPTkFQMRkwFwYDVQQK\nDBBMaW51eC1Gb3VuZGF0aW9uMRYwFAYDVQQHDA1TYW4tRnJhbmNpc2NvMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAJ6wPqEZVkc5nI8nGed3E/4O0LPUKNp9MFucONrHVQvrwzEv\nvsLHoOnx4esXhqq5pF+lhRPBuEUN5GUIOVzZGsU0DPpebBbnRDqaQv3O6UkOZRPH\ngj0CWzuwEfd7Jsl4WJEHfgynaWD+rRHS4nqUCjhMZvDhHhLke6ORptC/akUq5kPC\nfSZP9JgC22Ry+P/snmR7J1kIcD6w9WQh2H2LfQR4pL61dV5fJTgvbolLg9EGyRkE\njmd6um/C4KE1j9nZcVtvrFg4ge2DPzYWDFpnN6pQf5Kx8Idx7P1n5ZGu4dcn1Txs\n5yVk/63jm7LUn94eXqHWiDT8omYZ93dV3Fq8sm8CAwEAAaOBrjCBqzAMBgNVHRMB\nAf8EAjAAMB8GA1UdIwQYMBaAFNZE7OixHnSjEI6NUeSPzzppBszmMCIGA1UdEQQb\nMBmCC2V4YW1wbGUuY29tggpzYW1wbGUuY29tMCcGA1UdJQQgMB4GCCsGAQUFBwMC\nBggrBgEFBQcDBAYIKwYBBQUHAwEwHQYDVR0OBBYEFI1XdhOJ7RzXWgN45evIYtEL\n2OplMA4GA1UdDwEB/wQEAwIF4DANBgkqhkiG9w0BAQsFAAOCAQEAVpPWQtOWrYsH\nov1jAt+fAgVXkgzinKILPiwNITzJBnP3Y329vROLVbktie0HAxd89xhZX02E5wxa\nK2bTKqrwXA8H7inWK9U4fcQ9j9qPhwzkpd6AZAOdWFZ6gCrQpz5jLaJlHA7ePTxt\nVLQiD/7DHtPTPpVMXCVG4R9zIZaWpC8EntlMy9Fdhge4+PMni4OdwCNnjnrdPbzU\nuNGz8Ri3PVNn/o+v/N3MkcvlqVJpzcryldWZIqdZnVA5liiT7deQB4HzMhoPABuH\npUstTj8Gk1b4aSFkGxSlgDhBdY80XB9n0VzySpDn+MVgAOGZ1GrCpnZiOHE5Ujld\nVblGRGOFdQ=="
        ],
        "data": "MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCesD6hGVZHOZyP\nJxnndxP+DtCz1CjafTBbnDjax1UL68MxL77Cx6Dp8eHrF4aquaRfpYUTwbhFDeRl\nCDlc2RrFNAz6XmwW50Q6mkL9zulJDmUTx4I9Als7sBH3eybJeFiRB34Mp2lg/q0R\n0uJ6lAo4TGbw4R4S5HujkabQv2pFKuZDwn0mT/SYAttkcvj/7J5keydZCHA+sPVk\nIdh9i30EeKS+tXVeXyU4L26JS4PRBskZBI5nerpvwuChNY/Z2XFbb6xYOIHtgz82\nFgxaZzeqUH+SsfCHcez9Z+WRruHXJ9U8bOclZP+t45uy1J/eHl6h1og0/KJmGfd3\nVdxavLJvAgMBAAECggEAULr/EbwvHYscr+NHlW8KF7z3xhTQN99qg6Cld//ir27g\n3ZwZo0QmA/6PXaeZVi+dhR9S5WsvUMoxgIi/jhHRQbQqShFRUhGTr3CdVjw2iHkv\nSPG9u0YAngwEB/B6GhMEk7vJD6z3umW2LJQprgy0alHMKQ2N3DSrBdrqj9ZhvQ+5\njPTISU1CggkRMyCEOFA9CNffWl+gLWDKhlwTzS87nCbikxjimde1uuRN36PJofbr\nAOYTeLPJ3nF0IwmZ0QaH6nYPePeFn5p4+v1YgJ3HhEaaP7MvNb5tbeN90CMcv+tk\nI/vBgKR/BsXCCzuvnWxeTZjob8cfBkih6goFe7s5qQKBgQDXjORY+ZJsPh+DnePf\nXJSIAZg6SjSjmnPU8WvrEJcm72Kf/6YNPOI3VIjRG4auLYZhfEdIgBZFl/rfCi23\no4TRkCL2l3PoqC+Bkn5cWjngPwy1ZjUKMgUwA1nb42UNA+ZiQ+JrxPpj7VkaitU9\n80zgykMwOkEoVNsVG/xCjgNB5QKBgQC8d7BM5kPtHD4my2/U608ES7TgzTOemFxc\n0C5MQej1bQFzXeVykU+5HLUg7Tj1QnHJHpu/PqVXPdSIWsQljtW78U6dW+oWRYDC\n7vedlq0PaX92AK4kFSU20ARhgd7SF237fqJpSJ0qGMnjxooyzYK5AM5Zw2ArNXN+\ncz7C5zdtwwKBgHfxWo94MNYyq26dyVNdPPmBiafF6wyuwY3Y2WhwTuvHgWXu6AOt\nwWNwksFpQ8RgaMqOUUwymjIXo+3AyDfLsfi67/mvxTvNesHrNo1FV7dE3lmCijkm\nJGTXOIV3P5/Cb8x2YeP+NCauyEpQvuA+JqVfZ4vKnr9GmLJiZDjUtVNRAoGAfmRK\nMJgwL6DEMaI3InSYLkpTSHt1RcPqFThsBCojgAwVb/glezrmWz9cXXOZwilIQ9eq\nHYx2t9Y4ZSVx67oprnv6+977UupTmudF2kKFVxOqLknwfzRLG8JPXdEJzJ133bSk\nWLUg6W1sX7vzTHS0WTuoJ1y2buPOy5ugGHeVnZ8CgYB2B1vCvafjkrbl9iNcfVs6\nxEvQqkrW4UMHRCci54XT1f7H7Oih7Bi08OL9wyEdwaNK2wqb03WGZoaqz7fdAiU/\naQSBar6Dr3gTnZuJz+JGNJsS+VAbr6aGtC+bHjwqGdnIvmGSEeDX6fdDOewT2Q2s\nXZ+LHtO3BIfCILGGmog1rA=="
      },
      {
        "name": "ODL_private_key_0",
        "certificate-chain": [
          "MIIECTCCAvGgAwIBAgIBBzANBgkqhkiG9w0BAQsFADCBjDELMAkGA1UEBhMCQ1ox\nFjAUBgNVBAgMDVNvdXRoIE1vcmF2aWExDTALBgNVBAcMBEJybm8xDzANBgNVBAoM\nBkNFU05FVDEMMAoGA1UECwwDVE1DMRMwEQYDVQQDDApleGFtcGxlIENBMSIwIAYJ\nKoZIhvcNAQkBFhNleGFtcGxlY2FAbG9jYWxob3N0MB4XDTE1MDczMDA3MjcxOFoX\nDTM1MDcyNTA3MjcxOFowgYUxCzAJBgNVBAYTAkNaMRYwFAYDVQQIDA1Tb3V0aCBN\nb3JhdmlhMQ8wDQYDVQQKDAZDRVNORVQxDDAKBgNVBAsMA1RNQzEXMBUGA1UEAwwO\nZXhhbXBsZSBjbGllbnQxJjAkBgkqhkiG9w0BCQEWF2V4YW1wbGVjbGllbnRAbG9j\nYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAueCQaNQWoNmF\nK6LKu1p8U8ZWdWg/PvDdLsJyzfzl/Qw4UA68SfFNaY06zZl8QB9W02nr5kWeeMY0\nVA3adrPgOlvfx3oWlFbkETnMaN4OT3WTQ0Wt6jAWZDzVfopwpJPAzRPxACDftIqF\nGagYcF32hZlVNqqnVdbXh0S0EViweqp/dbG4VDUHSNVbglc+u4UbEzNIFXMdEFsJ\nZpkynOmSiTsIATqIhb+2srkVgLwhfkC2qkuHQwAHdubuB07ObM2z01UhyEdDvEYG\nHwtYAGDBL2TAcsI0oGeVkRyuOkV0QY0UN7UEFI1yTYw+xZ42HgFx3uGwApCImxhb\nj69GBYWFqwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVu\nU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUXGpLeLnh2cSDARAV\nA7KrBxGYpo8wHwYDVR0jBBgwFoAUc1YQIqjZsHVwlea0AB4N+ilNI2gwDQYJKoZI\nhvcNAQELBQADggEBAJPV3RTXFRtNyOU4rjPpYeBAIAFp2aqGc4t2J1c7oPp/1n+l\nZvjnwtlJpZHxMM783e2ryDQ6dkvXDf8kpwKlg3U3mkJ3xKkDdWrM4QwghXdCN519\naa9qmu0zdFL+jUAaWlQ5tsceOrvbusCcbMqiFGk/QfpHqPv52SVWbYyUx7IX7DE+\nUjgsLHycfV/tlcx4ZE6soTzl9VdgSL/zmzG3rjsr58J80rXckLgBhvijgBlIAJvW\nfC7D0vaouvBInSFXymdPVoUDZ30cdGLf+hI/i/TfsEMOinLrXVdkSGNo6FXAHKSv\nXeB9oFKSzhQ7OPyRyqvEPycUSw/qD6FVr80oDDc="
        ],
        "data": "MIIEpAIBAAKCAQEAueCQaNQWoNmFK6LKu1p8U8ZWdWg/PvDdLsJyzfzl/Qw4UA68\nSfFNaY06zZl8QB9W02nr5kWeeMY0VA3adrPgOlvfx3oWlFbkETnMaN4OT3WTQ0Wt\n6jAWZDzVfopwpJPAzRPxACDftIqFGagYcF32hZlVNqqnVdbXh0S0EViweqp/dbG4\nVDUHSNVbglc+u4UbEzNIFXMdEFsJZpkynOmSiTsIATqIhb+2srkVgLwhfkC2qkuH\nQwAHdubuB07ObM2z01UhyEdDvEYGHwtYAGDBL2TAcsI0oGeVkRyuOkV0QY0UN7UE\nFI1yTYw+xZ42HgFx3uGwApCImxhbj69GBYWFqwIDAQABAoIBAQCZN9kR8DGu6V7y\nt0Ax68asL8O5B/OKaHWKQ9LqpVrXmikZJOxkbzoGldow/CIFoU+q+Zbwu9aDa65a\n0wiP7Hoa4Py3q5XNNUrOQDyU/OYC7cI0I83WS0lJ2zOJGYj8wKae5Z81IeQFKGHK\n4lsy1OGPAvPRGh7RjUUgRavA2MCwe07rWRuDb/OJFe4Oh56UMEjwMiNBtMNtncog\nj1vr/qgRJdf9tf0zlJmLvUJ9+HSFFV9I/97LJyFhb95gAfHkjdVroLVgT3Cho+4P\nWtZaKCIGD0OwfOG2nLV4leXvRUk62/LMlB8NI9+JF7Xm+HCKbaWHNWC7mvWSLV58\nZl4AbUWRAoGBANyJ6SFHFRHSPDY026SsdMzXR0eUxBAK7G70oSBKKhY+O1j0ocLE\njI2krHJBhHbLlnvJVyMUaCUOTS5m0uDw9hgSsAqeSL3hL38kxVZw+KNG9Ouno1Fl\nKnE/xXHlPQyeGs/P8nAMzHZxQtEsQdQayJEhK2XXHTsy7Q3MxDisfVJ1AoGBANfD\n34gB+OMx6pwj7zk3qWbYXSX8xjCZMR0ciko+h4xeMP2N8B0oyoqC+v1ABMAtJ3wG\nsGZd0hV9gwM7OUM3SEwkn6oeg1GemWLcn4rlSmTnZc4aeVwrEWlnSNFX3s4g9l4u\nk8Ugu4MVJYqH8HuDQ5Ggl6/QAwPzMSEdCW0O+jOfAoGAIBRbegC5+t6m7Yegz4Ja\ndxV1g98K6f58x+MDsQu4tYWV4mmrQgaPH2dtwizvlMwmdpkh+LNWNtWuumowkJHc\nakIFo3XExQIFg6wYnGtQb4e5xrGa2xMpKlIJaXjb+YLiCYqJDG2ALFZrTrvuU2kV\n9a5qfqTc1qigvNolTM0iaaUCgYApmrZWhnLUdEKV2wP813PNxfioI4afxlpHD8LG\nsCn48gymR6E+Lihn7vuwq5B+8fYEH1ISWxLwW+RQUjIneNhy/jjfV8TgjyFqg7or\n0Sy4KjpiNI6kLBXOakELRNNMkeSPopGR2E7v5rr3bGD9oAD+aqX1G7oJH/KgPPYd\nVl7+ZwKBgQDcHyWYrimjyUgKaQD2GmoO9wdcJYQ59ke9K+OuGlp4ti5arsi7N1tP\nB4f09aeELM2ASIuk8Q/Mx0jQFnm8lzRFXdewgvdPoZW/7VufM9O7dGPOc41cm2Dh\nyrTcXx/VmUBb+/fnXVEgCv7gylp/wtdTGHQBQJHR81jFBz0lnLj+gg=="
      }
    ]
  }
}





Regards,
Ajay


Re: [release] [kernel-dev] Regression in Mg SR1

Daniel de la Rosa
 


On Mon, May 4, 2020 at 1:34 PM Jamo Luhrsen <jluhrsen@...> wrote:

On 5/4/20 1:12 PM, Robert Varga wrote:
> On 04/05/2020 17:38, Daniel de la Rosa wrote:
>> Thank @Jamo Luhrsen <mailto:jluhrsen@...> So @Robert Varga
>> <mailto:nite@...> i think we should merge this revert ASAP so we can
>> respin and get a new Magnesium SR1 release ASAP
> The revert is in, next autorelease should have it.

It'll be this one to look at:
https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-magnesium-mvn35-openjdk11/261


Thanks,
JamO

> Regards,
> Robert
>


--
Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


Re: [release] [kernel-dev] Regression in Mg SR1

JamO Luhrsen
 

On 5/4/20 1:12 PM, Robert Varga wrote:
On 04/05/2020 17:38, Daniel de la Rosa wrote:
Thank @Jamo Luhrsen <mailto:jluhrsen@...> So @Robert Varga
<mailto:nite@...> i think we should merge this revert ASAP so we can
respin and get a new Magnesium SR1 release ASAP
The revert is in, next autorelease should have it.
It'll be this one to look at:
https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-magnesium-mvn35-openjdk11/261


Thanks,
JamO

Regards,
Robert


Re: [kernel-dev] Regression in Mg SR1

Robert Varga
 

On 01/05/2020 23:36, JamO Luhrsen wrote:

On 5/1/20 1:01 PM, Luis Gomez wrote:
FYI this change merged in April 27th:

https://git.opendaylight.org/gerrit/c/controller/+/87586
here's the revert:
https://git.opendaylight.org/gerrit/c/controller/+/89554
Merged.


Produced regression in all these many suites:

https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-magnesium/
https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-ask-all-magnesium/
https://jenkins.opendaylight.org/releng/job/controller-csit-1node-akka1-all-magnesium/
This one seems easiest to try to debug. The gist of the problem is this:

- bring up older controller version and do some configs
- copy snapshots/ and *journal/ folders off to new controller version
- start new controller version
- notice that the data/config is not there (404 on cars:cars)

That's all I have though, by looking at the robot logs. Looking at the
karaf log, it's
weirdly silent after the new controller boots up like normal. All that's
there are
the two log statements we write to it from robot:

2020-05-01T01:47:57,330 | INFO | pipe-log:log "ROBOT MESSAGE: Starting
test controller-akka1.txt.Verify_Data_Is_Restored" | core | 123 -
org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test
controller-akka1.txt.Verify_Data_Is_Restored 2020-05-01T01:51:01,859 |
INFO | pipe-log:log "ROBOT MESSAGE: Starting test
controller-akka1.txt.Archive_Older_Karaf_Log" | core | 123 -
org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test
controller-akka1.txt.Archive_Older_Karaf_Log
I looked at the BGPCEP job and this part is quite weird:

2020-05-04T16:10:03,320 | ERROR | opendaylight-cluster-data-shard-dispatcher-39 | Shard | 298 - org.opendaylight.controller.sal-clustering-commons - 1.11.1 | member-1-shard-default-config: Log entry not found for index 0
2020-05-04T16:10:03,383 | ERROR | opendaylight-cluster-data-shard-dispatcher-39 | Shard | 298 - org.opendaylight.controller.sal-clustering-commons - 1.11.1 | member-1-shard-default-config: failed to apply payload org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload$Simple@3d3565da
also, similar things are going down in the ask-all job:

2020-05-04T15:29:10,295 | INFO | opendaylight-cluster-data-shard-dispatcher-40 | Shard | 297 - org.opendaylight.controller.sal-clustering-commons - 1.11.1 | member-1-shard-prefix-configuration-shard-config (Follower): The log is not empty but the prevLogIndex 22 was not found in it - lastIndex: 21, snapshotIndex: -1, snapshotTerm: -1
2020-05-04T15:29:10,296 | INFO | opendaylight-cluster-data-shard-dispatcher-40 | Shard | 297 - org.opendaylight.controller.sal-clustering-commons - 1.11.1 | member-1-shard-prefix-configuration-shard-config (Follower): Follower is out-of-sync so sending negative reply: AppendEntriesReply [term=54, success=false, followerId=member-1-shard-prefix-configuration-shard-config, logLastIndex=21, logLastTerm=5, forceInstallSnapshot=false, needsLeaderAddress=false, payloadVersion=11, raftVersion=4, recipientRaftVersion=3]
so something is definitely off with journal :-/

The bgp jobs seem to be even more broken though. More ERRORs, etc. Not sure
if we need to look at those separately or not.
Probably yes, as they seem to indicate an inconsistency in cluster
singleton -- but that may be related (although I am not sure how).

Regards,
Robert


Re: [release] [kernel-dev] Regression in Mg SR1

Robert Varga
 

On 04/05/2020 17:38, Daniel de la Rosa wrote:
Thank @Jamo Luhrsen <mailto:jluhrsen@...> So @Robert Varga
<mailto:nite@...> i think we should merge this revert ASAP so we can
respin and get a new Magnesium SR1 release ASAP
The revert is in, next autorelease should have it.

Regards,
Robert


Re: [release] [kernel-dev] Regression in Mg SR1

Daniel de la Rosa
 

Thank @Jamo Luhrsen So @Robert Varga i think we should merge this revert ASAP so we can respin and get a new Magnesium SR1 release ASAP

Thanks

On Fri, May 1, 2020 at 10:27 PM JamO Luhrsen <jluhrsen@...> wrote:


On 5/1/20 2:36 PM, JamO Luhrsen via lists.opendaylight.org wrote:


On 5/1/20 1:01 PM, Luis Gomez wrote:
FYI this change merged in April 27th:

here's the revert:
https://git.opendaylight.org/gerrit/c/controller/+/89554



Produced regression in all these many suites:

This one seems easiest to try to debug. The gist of the problem is this:

- bring up older controller version and do some configs
- copy snapshots/ and *journal/ folders off to new controller version
- start new controller version
- notice that the data/config is not there (404 on cars:cars)

That's all I have though, by looking at the robot logs. Looking at the karaf log, it's
weirdly silent after the new controller boots up like normal. All that's there are
the two log statements we write to it from robot:

2020-05-01T01:47:57,330 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored 2020-05-01T01:51:01,859 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log


The bgp jobs seem to be even more broken though. More ERRORs, etc. Not sure
if we need to look at those separately or not.


These suites are in some degree dealing with the snapshot folder that might have changed after the mentioned patch.
Did snapshot change? I know journal did, but we addressed that here:
https://git.opendaylight.org/gerrit/c/integration/test/+/88658

I am not sure at this moment we should investigate the issues + repair the test (it can take a while) or just revert and try next SR.

I would guess some folks might have also reservations in introducing this change in an SR.
Once the revert I created gives me a distribution, I'll run it through these four
jobs in the sandbox. If those all pass like expected, and we don't get any quick
fix on the CSIT side, it might make sense to merge the revert and get moving
on a new release candidate.

FYI, The four jobs reported above were all fine with the revert.

JamO


Thanks,
JamO

BR/Luis



    



--
Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


Re: [release] [kernel-dev] Regression in Mg SR1

JamO Luhrsen
 


On 5/1/20 2:36 PM, JamO Luhrsen via lists.opendaylight.org wrote:


On 5/1/20 1:01 PM, Luis Gomez wrote:
FYI this change merged in April 27th:

here's the revert:
https://git.opendaylight.org/gerrit/c/controller/+/89554



Produced regression in all these many suites:

This one seems easiest to try to debug. The gist of the problem is this:

- bring up older controller version and do some configs
- copy snapshots/ and *journal/ folders off to new controller version
- start new controller version
- notice that the data/config is not there (404 on cars:cars)

That's all I have though, by looking at the robot logs. Looking at the karaf log, it's
weirdly silent after the new controller boots up like normal. All that's there are
the two log statements we write to it from robot:

2020-05-01T01:47:57,330 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored 2020-05-01T01:51:01,859 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log


The bgp jobs seem to be even more broken though. More ERRORs, etc. Not sure
if we need to look at those separately or not.


These suites are in some degree dealing with the snapshot folder that might have changed after the mentioned patch.
Did snapshot change? I know journal did, but we addressed that here:
https://git.opendaylight.org/gerrit/c/integration/test/+/88658

I am not sure at this moment we should investigate the issues + repair the test (it can take a while) or just revert and try next SR.

I would guess some folks might have also reservations in introducing this change in an SR.
Once the revert I created gives me a distribution, I'll run it through these four
jobs in the sandbox. If those all pass like expected, and we don't get any quick
fix on the CSIT side, it might make sense to merge the revert and get moving
on a new release candidate.

FYI, The four jobs reported above were all fine with the revert.

JamO


Thanks,
JamO

BR/Luis



    


Re: [kernel-dev] Regression in Mg SR1

JamO Luhrsen
 


On 5/1/20 1:01 PM, Luis Gomez wrote:
FYI this change merged in April 27th:

here's the revert:
https://git.opendaylight.org/gerrit/c/controller/+/89554



Produced regression in all these many suites:

This one seems easiest to try to debug. The gist of the problem is this:

- bring up older controller version and do some configs
- copy snapshots/ and *journal/ folders off to new controller version
- start new controller version
- notice that the data/config is not there (404 on cars:cars)

That's all I have though, by looking at the robot logs. Looking at the karaf log, it's
weirdly silent after the new controller boots up like normal. All that's there are
the two log statements we write to it from robot:

2020-05-01T01:47:57,330 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Verify_Data_Is_Restored 2020-05-01T01:51:01,859 | INFO | pipe-log:log "ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log" | core | 123 - org.apache.karaf.log.core - 4.2.6 | ROBOT MESSAGE: Starting test controller-akka1.txt.Archive_Older_Karaf_Log


The bgp jobs seem to be even more broken though. More ERRORs, etc. Not sure
if we need to look at those separately or not.


These suites are in some degree dealing with the snapshot folder that might have changed after the mentioned patch.
Did snapshot change? I know journal did, but we addressed that here:
https://git.opendaylight.org/gerrit/c/integration/test/+/88658

I am not sure at this moment we should investigate the issues + repair the test (it can take a while) or just revert and try next SR.

I would guess some folks might have also reservations in introducing this change in an SR.
Once the revert I created gives me a distribution, I'll run it through these four
jobs in the sandbox. If those all pass like expected, and we don't get any quick
fix on the CSIT side, it might make sense to merge the revert and get moving
on a new release candidate.
Thanks,
JamO

BR/Luis


Re: integration meeting agenda topics for tomorrow (4/29/2020)?

JamO Luhrsen
 

ok, sounds good.

On 4/28/20 9:14 AM, Daniel De La Rosa wrote:

lets review magnesium status ...

Thanks

On Tue, Apr 28, 2020 at 9:02 AM JamO Luhrsen <jluhrsen@...> wrote:
Do we have any topics to hold an integration meeting tomorrow 9pm PST?

Thanks,
JamO




--
Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


Re: integration meeting agenda topics for tomorrow (4/29/2020)?

Daniel de la Rosa
 

lets review magnesium status ...

Thanks

On Tue, Apr 28, 2020 at 9:02 AM JamO Luhrsen <jluhrsen@...> wrote:
Do we have any topics to hold an integration meeting tomorrow 9pm PST?

Thanks,
JamO




--
Daniel de la Rosa
Customer Support Manager
Lumina Networks Inc.
e: ddelarosa@...
m:  +1 408 7728120


integration meeting agenda topics for tomorrow (4/29/2020)?

JamO Luhrsen
 

Do we have any topics to hold an integration meeting tomorrow 9pm PST?

Thanks,
JamO


Re: [release] jenkins job logs not stored

Luis Gomez
 

I am not committer of global-jjb, Jamo?

On Apr 23, 2020, at 9:29 PM, Anil Belur <abelur@...> wrote:

Hello Jamo, Luis:

This is because of a update to deploy-logs which is missing the an update to the regex in global-jjb. I have fixed it here.

Please merge, we'll need to do a hotfix for this.

I have tested the fix here:

On Fri, Apr 24, 2020 at 10:12 AM Anil Belur via lists.opendaylight.org<abelur=linuxfoundation.org@...> wrote:
I'll have a look into this an get back to you, there has been some recent updates to global-jjb.

On Fri, Apr 24, 2020 at 8:14 AM JamO Luhrsen <jluhrsen@...> wrote:

Actually, the logs are there:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-flow-services-all-aluminium/48/

but the link is not there in jenkins like we expect.

I don't see any smoking gun in the console logs, but the global-jjb
version upgrade happened between when we last had logs and
when it stopped working. I'm sure that's it.

Here's a revert, if we want to try that:

https://git.opendaylight.org/gerrit/c/releng/builder/+/89282


but I suppose it could be some other change in jenkins (e.g., a plugin update)


JamO


On 4/23/20 2:52 PM, Luis Gomez wrote:
Has anything changed lately in the infra so that no log is stored anymore:


BR/Luis





Re: [release] jenkins job logs not stored

Anil Belur
 

Hello Jamo, Luis:

This is because of a update to deploy-logs which is missing the an update to the regex in global-jjb. I have fixed it here.

Please merge, we'll need to do a hotfix for this.

I have tested the fix here:

On Fri, Apr 24, 2020 at 10:12 AM Anil Belur via lists.opendaylight.org <abelur=linuxfoundation.org@...> wrote:

I'll have a look into this an get back to you, there has been some recent updates to global-jjb.

On Fri, Apr 24, 2020 at 8:14 AM JamO Luhrsen <jluhrsen@...> wrote:

Actually, the logs are there:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-flow-services-all-aluminium/48/

but the link is not there in jenkins like we expect.

I don't see any smoking gun in the console logs, but the global-jjb
version upgrade happened between when we last had logs and
when it stopped working. I'm sure that's it.

Here's a revert, if we want to try that:

https://git.opendaylight.org/gerrit/c/releng/builder/+/89282


but I suppose it could be some other change in jenkins (e.g., a plugin update)


JamO


On 4/23/20 2:52 PM, Luis Gomez wrote:
Has anything changed lately in the infra so that no log is stored anymore:


BR/Luis



    



Re: [release] jenkins job logs not stored

Anil Belur
 

I'll have a look into this an get back to you, there has been some recent updates to global-jjb.


On Fri, Apr 24, 2020 at 8:14 AM JamO Luhrsen <jluhrsen@...> wrote:

Actually, the logs are there:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-flow-services-all-aluminium/48/

but the link is not there in jenkins like we expect.

I don't see any smoking gun in the console logs, but the global-jjb
version upgrade happened between when we last had logs and
when it stopped working. I'm sure that's it.

Here's a revert, if we want to try that:

https://git.opendaylight.org/gerrit/c/releng/builder/+/89282


but I suppose it could be some other change in jenkins (e.g., a plugin update)


JamO


On 4/23/20 2:52 PM, Luis Gomez wrote:
Has anything changed lately in the infra so that no log is stored anymore:


BR/Luis



    


Re: [release] jenkins job logs not stored

JamO Luhrsen
 

Actually, the logs are there:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-flow-services-all-aluminium/48/

but the link is not there in jenkins like we expect.

I don't see any smoking gun in the console logs, but the global-jjb
version upgrade happened between when we last had logs and
when it stopped working. I'm sure that's it.

Here's a revert, if we want to try that:

https://git.opendaylight.org/gerrit/c/releng/builder/+/89282


but I suppose it could be some other change in jenkins (e.g., a plugin update)


JamO


On 4/23/20 2:52 PM, Luis Gomez wrote:

Has anything changed lately in the infra so that no log is stored anymore:


BR/Luis



    


jenkins job logs not stored

Luis Gomez
 

Has anything changed lately in the infra so that no log is stored anymore:


BR/Luis


integration meeting tonight (4/22/2020) @ 9pm PST

JamO Luhrsen
 

It may be short, but we can at least catch up on release status.

Thanks,
JamO


Re: [opendaylight-dev] Build step 'Execute shell' marked build as failure

JamO Luhrsen
 

It's not always obvious, but you have to dig through the console log to see
what's wrong. Here's what I found:

  [ERROR] Non-resolvable import POM: Could not find artifact org.opendaylight.netconf:netconf-artifacts:pom:1.8.0-SNAPSHOT

That is magnesium official release version, but we are now working
on SR1, which is version 1.8.1-SNAPSHOT. The common/pom.xml in your
patch's tree is here:

  https://git.opendaylight.org/gerrit/gitweb?p=transportpce.git;a=blob;f=common/pom.xml;h=cba96a6a173fe8a65dae37a99fb9f9c721273f3b;hb=9c08179c83415e9dd87f12fd46a2652cf07d9c62#l27

I'm guessing all you need to do is rebase to get past this error.

Hope it helps,
JamO

On 4/20/20 11:27 PM, shaaban.eltanany.ext via lists.opendaylight.org wrote:


Re: ODL:Master(Aluminium): Unable to bring karaf distribution locally - Reg

Robert Varga
 

On 21/04/2020 11:23, Karthikeyan wrote:
*      <type>test-jar</type>
      <scope>compile</scope>*
This combination is always wrong in a production artifact's pom.xml.

Bye,
Robert