Issues with bundle artifacts vs. jar artifacts


David Karr
 

I was hoping to get some input on the various issues involved with https://bugs.opendaylight.org/show_bug.cgi?id=5940 . I now understand this better, but i wanted to get other POVs.

I want to ensure that users have an easy path to get artifacts with Yang models to import.

If they click the "Add" button in Dependencies in the pom and enter "ietf-yang-types" in the search field, they get artifacts from a couple of different groupIds.  The "old" artifacts are in "org.opendaylight.yangtools.model", and the "new" artifacts are in "org.opendaylight.mdsal.model".  The "old" artifacts are all "jar" artifacts, and the "new" artifacts are all "bundle" artifacts.  It seems reasonable to assume that they would want the newer artifacts.

If I select the bundle artifact (for instance, version "2010.09.24.8.1-Beryllium-SR1") and then try to add an import, the available completions don't list "ietf-yang-types" (they do at this moment for me, but I think that's a caching issue that I'm also a bit concerned about).  If I instead edit the properties of the dependency and change it to type "jar" and then check completions again, it has the "ietf-yang-types" offering.

One thing I'm wondering is WHY m2e is offering the "bundle" type for some artifacts and only the "jar" type for other artifacts. Looking at two sample "bundle" and "jar" artifacts, I'm guessing having "packaging-type" set to "bundle" is the distinguishing factor.

The problem is, although m2e apparently knows what a bundle is, the underlying pom doesn't necessarily know that, unless you tell it.

I found the following thread, which talks about this: http://stackoverflow.com/questions/9932923/why-cant-maven-find-an-osgi-bundle-dependency .

The "ad hoc" correct answer (the correct answer wasn't marked as such) indicates that the POM has to use the "maven-bundle-plugin" in order to know about bundle artifacts.  If I add that to my yang project, it's able to get Yang models from bundle artifacts.

I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.

The other funny issue is that m2e will happily offer up "bundle" artifacts into a POM that doesn't understand bundle artifacts, resulting in a guaranteed build failure. I'm having a conversation on m2e-users about this, but I'm getting a blank stare back.


Michael Vorburger
 

On Wed, May 25, 2016 at 7:43 PM, David M. Karr <davidmichaelkarr@...> wrote:

I could use your input on the various issues involved with https://bugs.opendaylight.org/show_bug.cgi?id=5940 (...)

If they click the "Add" button in Dependencies in the pom and enter "ietf-yang-types" in the search field, they get artifacts from a couple of different groupIds. (...)

Are you aware that this will anyway only work for artifacts which have been previously installed or already downloaded  into local m2 repo? M2E has no way of known all artifacts which have YANG. (There is a thing for a Nexus index, but this too big and too slow; it's disabled by default for good reason, and is that makes M2E say this in Add Dependency: "Index downloads are disabled, search results may be incomplete."

If I select the bundle artifact (for instance, version "2010.09.24.8.1-Beryllium-SR1") and then try to add an import, the available completions don't list "ietf-yang-types" (they do at this moment for me, but I think that's a caching issue that I'm also a bit concerned about).

They do for me as well. What makes you say that there is a caching issue? Are you sure you are not chasing a phantom problem - to me, this actually seems to work...
 
If I instead edit the properties of the dependency and change it to type "jar" and then check completions again, it has the "ietf-yang-types" offering.

One thing I'm wondering is WHY m2e is offering the "bundle" type for some artifacts and only the "jar" type for other artifacts. Looking at two sample "bundle" and "jar" artifacts, I'm guessing having "packaging-type" set to "bundle" is the distinguishing factor.

The problem is, although m2e apparently knows what a bundle is, the underlying pom doesn't necessarily know that, unless you tell it.

I found the following thread, which talks about this: http://stackoverflow.com/questions/9932923/why-cant-maven-find-an-osgi-bundle-dependency .

The "ad hoc" correct answer (the correct answer wasn't marked as such) indicates that the POM has to use the "maven-bundle-plugin" in order to know about bundle artifacts.  If I add that to my yang project, it's able to get Yang models from bundle artifacts.

I can see that the Effective POM for e.g. the hello-api archetype indeed has a maven-bundle-plugin with <extensions>true, so that would confirm that this is the right thing to do.
 
I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.  I have a query out to m2e-users to ask why m2e is offering bundle artifacts for poms that don't know about them.

Yeah... although personally, I would never have built a New Project wizard for YANG Project to begin with. If it was up to me, I would just completely remove that entire feature. I don't want to be involved in supporting it. Instead, I would do all "example" type wizards (incl. tutorials etc.) through Maven archetypes, exclusively. This has the advantage that such projects "kickstarters" are real code, that can be integration tested in the build (Maven archetypes have support for this), instead of being hidden inside yangide.ui/resources, and programmatically created by code. My understanding is that the already existing odl-model-project archetype in Controller is for exactly this. It appears to be currently broken, but I've just fixed it as part of https://git.opendaylight.org/gerrit/#/c/39484/

BTW in this context also note my recent edit to https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Startup_Project_Archetype#Part_1_-_Build_with_a_simple_.27Example.27_module : "If you are using the GettingStarted: Eclipse fully automated set up, then you can now use menu File > New > Project > Maven > Maven Project, Next; Advanced: Name template: [groupId].[artifactId], and then check [X] Include snapshot archetypes, and choose Artifact Id opendaylight-startup-archetype. (Note that opendaylight-eclipse-setup already has a catalog with the required archetypeRepository from ODL.)"  FYI this works because I did https://github.com/vorburger/opendaylight-eclipse-setup/commit/94c89d68c7a4bb9c4aa0e95f77d4a507f6f3198e.


Tx,
M.
--
Michael Vorburger <vorburger@...> | IRC: vorburger @freenode | G/Hangout: michael.vorburger | Skype: mike_vorburger | ~ = http://vorburger.ch


On Thu, May 26, 2016 at 5:17 PM, David M. Karr <davidmichaelkarr@...> wrote:
I was hoping to get some input on the various issues involved with https://bugs.opendaylight.org/show_bug.cgi?id=5940 . I now understand this better, but i wanted to get other POVs.

I want to ensure that users have an easy path to get artifacts with Yang models to import.

If they click the "Add" button in Dependencies in the pom and enter "ietf-yang-types" in the search field, they get artifacts from a couple of different groupIds.  The "old" artifacts are in "org.opendaylight.yangtools.model", and the "new" artifacts are in "org.opendaylight.mdsal.model".  The "old" artifacts are all "jar" artifacts, and the "new" artifacts are all "bundle" artifacts.  It seems reasonable to assume that they would want the newer artifacts.

If I select the bundle artifact (for instance, version "2010.09.24.8.1-Beryllium-SR1") and then try to add an import, the available completions don't list "ietf-yang-types" (they do at this moment for me, but I think that's a caching issue that I'm also a bit concerned about).  If I instead edit the properties of the dependency and change it to type "jar" and then check completions again, it has the "ietf-yang-types" offering.

One thing I'm wondering is WHY m2e is offering the "bundle" type for some artifacts and only the "jar" type for other artifacts. Looking at two sample "bundle" and "jar" artifacts, I'm guessing having "packaging-type" set to "bundle" is the distinguishing factor.

The problem is, although m2e apparently knows what a bundle is, the underlying pom doesn't necessarily know that, unless you tell it.

I found the following thread, which talks about this: http://stackoverflow.com/questions/9932923/why-cant-maven-find-an-osgi-bundle-dependency .

The "ad hoc" correct answer (the correct answer wasn't marked as such) indicates that the POM has to use the "maven-bundle-plugin" in order to know about bundle artifacts.  If I add that to my yang project, it's able to get Yang models from bundle artifacts.

I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.

The other funny issue is that m2e will happily offer up "bundle" artifacts into a POM that doesn't understand bundle artifacts, resulting in a guaranteed build failure. I'm having a conversation on m2e-users about this, but I'm getting a blank stare back.

_______________________________________________
yangide-dev mailing list
yangide-dev@...
https://lists.opendaylight.org/mailman/listinfo/yangide-dev



David Karr
 

On 05/26/2016 09:13 AM, Michael Vorburger wrote:
On Wed, May 25, 2016 at 7:43 PM, David M. Karr <davidmichaelkarr@...> wrote:

I could use your input on the various issues involved with https://bugs.opendaylight.org/show_bug.cgi?id=5940 (...)

If they click the "Add" button in Dependencies in the pom and enter "ietf-yang-types" in the search field, they get artifacts from a couple of different groupIds. (...)

Are you aware that this will anyway only work for artifacts which have been previously installed or already downloaded  into local m2 repo? M2E has no way of known all artifacts which have YANG. (There is a thing for a Nexus index, but this too big and too slow; it's disabled by default for good reason, and is that makes M2E say this in Add Dependency: "Index downloads are disabled, search results may be incomplete."

Sorry, I don't understand this.  The search facility uses the indexing data Eclipse builds from the defined repos.  The "Index downloads are disabled" message is when the repository entry in "Maven Repositories" doesn't have indexing enabled, but you might already know that.

I don't know what "M2E has no way of known all artifacts which have YANG" means.  I'm guessing "known" was supposed to be "knowing".  Yes, M2E has no way to know which artifacts have Yang models.  I didn't expect it to.  I have to know what artifacts I want to get.  That's normal operating procedure.

If I select the bundle artifact (for instance, version "2010.09.24.8.1-Beryllium-SR1") and then try to add an import, the available completions don't list "ietf-yang-types" (they do at this moment for me, but I think that's a caching issue that I'm also a bit concerned about).

They do for me as well. What makes you say that there is a caching issue? Are you sure you are not chasing a phantom problem - to me, this actually seems to work...

I've been looking at this for quite a while, and I've verified the behavior many times.  Generally, if I don't specify the artifact dependency properly, the Yang model won't be available as an import choice.

 
If I instead edit the properties of the dependency and change it to type "jar" and then check completions again, it has the "ietf-yang-types" offering.

One thing I'm wondering is WHY m2e is offering the "bundle" type for some artifacts and only the "jar" type for other artifacts. Looking at two sample "bundle" and "jar" artifacts, I'm guessing having "packaging-type" set to "bundle" is the distinguishing factor.

The problem is, although m2e apparently knows what a bundle is, the underlying pom doesn't necessarily know that, unless you tell it.

I found the following thread, which talks about this: http://stackoverflow.com/questions/9932923/why-cant-maven-find-an-osgi-bundle-dependency .

The "ad hoc" correct answer (the correct answer wasn't marked as such) indicates that the POM has to use the "maven-bundle-plugin" in order to know about bundle artifacts.  If I add that to my yang project, it's able to get Yang models from bundle artifacts.

I can see that the Effective POM for e.g. the hello-api archetype indeed has a maven-bundle-plugin with <extensions>true, so that would confirm that this is the right thing to do.

Acknowledged. I've already tested the changes to implement this, but I wanted to discuss it first.
 
I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.  I have a query out to m2e-users to ask why m2e is offering bundle artifacts for poms that don't know about them.

Yeah... although personally, I would never have built a New Project wizard for YANG Project to begin with. If it was up to me, I would just completely remove that entire feature. I don't want to be involved in supporting it. Instead, I would do all "example" type wizards (incl. tutorials etc.) through Maven archetypes, exclusively. This has the advantage that such projects "kickstarters" are real code, that can be integration tested in the build (Maven archetypes have support for this), instead of being hidden inside yangide.ui/resources, and programmatically created by code. My understanding is that the already existing odl-model-project archetype in Controller is for exactly this. It appears to be currently broken, but I've just fixed it as part of https://git.opendaylight.org/gerrit/#/c/39484/.

The issue is, a very large potential user base for this is people who are not programmers, but Yang designers.  I've had many inquiries about this application, asking if they can avoid using Maven entirely. I'm not going to remove an entire feature set that allows those people to avoid dealing with some aspects of Maven configuration.


BTW in this context also note my recent edit to https://wiki.opendaylight.org/view/OpenDaylight_Controller:MD-SAL:Startup_Project_Archetype#Part_1_-_Build_with_a_simple_.27Example.27_module : "If you are using the GettingStarted: Eclipse fully automated set up, then you can now use menu File > New > Project > Maven > Maven Project, Next; Advanced: Name template: [groupId].[artifactId], and then check [X] Include snapshot archetypes, and choose Artifact Id opendaylight-startup-archetype. (Note that opendaylight-eclipse-setup already has a catalog with the required archetypeRepository from ODL.)"  FYI this works because I did https://github.com/vorburger/opendaylight-eclipse-setup/commit/94c89d68c7a4bb9c4aa0e95f77d4a507f6f3198e.


Tx,
M.
--
Michael Vorburger <vorburger@...> | IRC: vorburger @freenode | G/Hangout: michael.vorburger | Skype: mike_vorburger | ~ = http://vorburger.ch


On Thu, May 26, 2016 at 5:17 PM, David M. Karr <davidmichaelkarr@...> wrote:
I was hoping to get some input on the various issues involved with https://bugs.opendaylight.org/show_bug.cgi?id=5940 . I now understand this better, but i wanted to get other POVs.

I want to ensure that users have an easy path to get artifacts with Yang models to import.

If they click the "Add" button in Dependencies in the pom and enter "ietf-yang-types" in the search field, they get artifacts from a couple of different groupIds.  The "old" artifacts are in "org.opendaylight.yangtools.model", and the "new" artifacts are in "org.opendaylight.mdsal.model".  The "old" artifacts are all "jar" artifacts, and the "new" artifacts are all "bundle" artifacts.  It seems reasonable to assume that they would want the newer artifacts.

If I select the bundle artifact (for instance, version "2010.09.24.8.1-Beryllium-SR1") and then try to add an import, the available completions don't list "ietf-yang-types" (they do at this moment for me, but I think that's a caching issue that I'm also a bit concerned about).  If I instead edit the properties of the dependency and change it to type "jar" and then check completions again, it has the "ietf-yang-types" offering.

One thing I'm wondering is WHY m2e is offering the "bundle" type for some artifacts and only the "jar" type for other artifacts. Looking at two sample "bundle" and "jar" artifacts, I'm guessing having "packaging-type" set to "bundle" is the distinguishing factor.

The problem is, although m2e apparently knows what a bundle is, the underlying pom doesn't necessarily know that, unless you tell it.

I found the following thread, which talks about this: http://stackoverflow.com/questions/9932923/why-cant-maven-find-an-osgi-bundle-dependency .

The "ad hoc" correct answer (the correct answer wasn't marked as such) indicates that the POM has to use the "maven-bundle-plugin" in order to know about bundle artifacts.  If I add that to my yang project, it's able to get Yang models from bundle artifacts.

I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.

The other funny issue is that m2e will happily offer up "bundle" artifacts into a POM that doesn't understand bundle artifacts, resulting in a guaranteed build failure. I'm having a conversation on m2e-users about this, but I'm getting a blank stare back.

_______________________________________________
yangide-dev mailing list
yangide-dev@...
https://lists.opendaylight.org/mailman/listinfo/yangide-dev




Michael Vorburger
 

On Thu, May 26, 2016 at 6:25 PM, David M. Karr <davidmichaelkarr@...> wrote:
On 05/26/2016 09:13 AM, Michael Vorburger wrote:
On Wed, May 25, 2016 at 7:43 PM, David M. Karr <davidmichaelkarr@...> wrote:
I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.  I have a query out to m2e-users to ask why m2e is offering bundle artifacts for poms that don't know about them.

Yeah... although personally, I would never have built a New Project wizard for YANG Project to begin with. If it was up to me, I would just completely remove that entire feature. I don't want to be involved in supporting it. Instead, I would do all "example" type wizards (incl. tutorials etc.) through Maven archetypes, exclusively. This has the advantage that such projects "kickstarters" are real code, that can be integration tested in thy e build (Maven archetypes have support for this), instead of being hidden inside yangide.ui/resources, and programmatically created by code. My understanding is that the already existing odl-model-project archetype in Controller is for exactly this. It appears to be currently broken, but I've just fixed it as part of https://git.opendaylight.org/gerrit/#/c/39484/
The issue is, a very large potential user base for this is people who are not programmers, but Yang designers.  I've had many inquiries about this application, asking if they can avoid using Maven entirely. I'm not going to remove an entire feature set that allows those people to avoid dealing with some aspects of Maven configuration.

Sure. What you could also do then is mixed, and pull the org.opendaylight.controller.archetypes:odl-model-project into yangide.ui a build time (simple Maven dependency copy should be able to do that), if we would like to avoid duplicating that pom.xml with the correct parent and all.

FYI I actually "stole" yangide.ui/resources/yang/acme-system.yang for archetypes/odl-model-project/src/main/resources/archetype-resources/src/main/yang/acme-system.yang


David Karr
 

On 05/26/2016 11:02 AM, Michael Vorburger wrote:
On Thu, May 26, 2016 at 6:25 PM, David M. Karr <davidmichaelkarr@...> wrote:
On 05/26/2016 09:13 AM, Michael Vorburger wrote:
On Wed, May 25, 2016 at 7:43 PM, David M. Karr <davidmichaelkarr@...> wrote:
I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.  I have a query out to m2e-users to ask why m2e is offering bundle artifacts for poms that don't know about them.

Yeah... although personally, I would never have built a New Project wizard for YANG Project to begin with. If it was up to me, I would just completely remove that entire feature. I don't want to be involved in supporting it. Instead, I would do all "example" type wizards (incl. tutorials etc.) through Maven archetypes, exclusively. This has the advantage that such projects "kickstarters" are real code, that can be integration tested in thy e build (Maven archetypes have support for this), instead of being hidden inside yangide.ui/resources, and programmatically created by code. My understanding is that the already existing odl-model-project archetype in Controller is for exactly this. It appears to be currently broken, but I've just fixed it as part of https://git.opendaylight.org/gerrit/#/c/39484/
The issue is, a very large potential user base for this is people who are not programmers, but Yang designers.  I've had many inquiries about this application, asking if they can avoid using Maven entirely. I'm not going to remove an entire feature set that allows those people to avoid dealing with some aspects of Maven configuration.

Sure. What you could also do then is mixed, and pull the org.opendaylight.controller.archetypes:odl-model-project into yangide.ui a build time (simple Maven dependency copy should be able to do that), if we would like to avoid duplicating that pom.xml with the correct parent and all.
I understand this from a high level, including the value of it, but I don't see how to do this.

FYI I actually "stole" yangide.ui/resources/yang/acme-system.yang for archetypes/odl-model-project/src/main/resources/archetype-resources/src/main/yang/acme-system.yang
Not sure what you mean.  Are you saying you did this with maven-dependency-plugin like you're suggesting above?


David Karr
 

On 05/26/2016 11:02 AM, Michael Vorburger wrote:
On Thu, May 26, 2016 at 6:25 PM, David M. Karr <davidmichaelkarr@...> wrote:
On 05/26/2016 09:13 AM, Michael Vorburger wrote:
On Wed, May 25, 2016 at 7:43 PM, David M. Karr <davidmichaelkarr@...> wrote:
I think the conclusion to this is that we need to change the default pom we construct for a Yang project to include the "maven-bundle-plugin" reference.  I have a query out to m2e-users to ask why m2e is offering bundle artifacts for poms that don't know about them.

Yeah... although personally, I would never have built a New Project wizard for YANG Project to begin with. If it was up to me, I would just completely remove that entire feature. I don't want to be involved in supporting it. Instead, I would do all "example" type wizards (incl. tutorials etc.) through Maven archetypes, exclusively. This has the advantage that such projects "kickstarters" are real code, that can be integration tested in thy e build (Maven archetypes have support for this), instead of being hidden inside yangide.ui/resources, and programmatically created by code. My understanding is that the already existing odl-model-project archetype in Controller is for exactly this. It appears to be currently broken, but I've just fixed it as part of https://git.opendaylight.org/gerrit/#/c/39484/
The issue is, a very large potential user base for this is people who are not programmers, but Yang designers.  I've had many inquiries about this application, asking if they can avoid using Maven entirely. I'm not going to remove an entire feature set that allows those people to avoid dealing with some aspects of Maven configuration.

And just to be clear, I have no problem with redundant strategies, like using archetypes to build the same structure, but we need the hand-holding path for the non-programmers.  If we can use other strategies to reduce code duplication (as described below), even better.


Sure. What you could also do then is mixed, and pull the org.opendaylight.controller.archetypes:odl-model-project into yangide.ui a build time (simple Maven dependency copy should be able to do that), if we would like to avoid duplicating that pom.xml with the correct parent and all.

FYI I actually "stole" yangide.ui/resources/yang/acme-system.yang for archetypes/odl-model-project/src/main/resources/archetype-resources/src/main/yang/acme-system.yang