Re: [alto] Interop test


Y. Richard Yang
 

Wendy,

Good proposal! Two comments:

1. I liked that the interop is for a setting that is close to real-life deployment. In particular, my understanding of the proposal is that the interop should leave as much unspecified as possible. A first reaction then is how to validate the correctness. For example, the response of resource-id is not known ahead of time. I assume that then the interop participants will need a second channel (e.g., human in the loop) to verify that a client c1 gets the correct response from a server s1. Is this the intended setting?

2. A second comment is that it will be interesting if the interop design will also become a starting point of ALTO compliance tests. We partition the tests into MUST and optional so that a test case is a MUST case if and only if (iff) it is a MUST in RFC7285. In the current proposal below, I see some mix (e.g., some services in the interop is not a MUST in RFC7285). Make sense?

Richard

On Fri, May 29, 2015 at 10:24 AM, Wendy Roome <w.roome@...> wrote:
Folks,

Rather than simply update the last test, I suggest a new approach,
outlined below.

**** General ****

The interop test defines two network maps: the default network map and an
alternate network map. We specify the PIDs and CIDRs for each map. For
each network map, the interop test defines the numerical-mode values for
the routingcost and hopcount cost-metrics. The interop test does not
define ordinal values for those metrics. Instead, each ALTO server may
assign ordinal values however it wants, as long as the ordinal values
preserve the numerical values.

The interop test also defines a custom endpoint property, say
"priv:ietf-type", and specifies the values.


**** Approach & Philosophy ****

Rather than specifying every detail, as in the previous interop test, this
iteration only specifies what is absolutely necessary. That is, the PIDs
in the network maps, the costs between PIDs, and the enpdoint property
values. Everything else -- resource ids, uris, cost-type names, etc. -- is
up to the server. Thus each server is completely characterized by two
parameters:

* The URI of its IRD.
* The resource id of its alternate map (if provided).

When a client is paired with a server, the client is given those two
parameters. The client is expected to determine everything else it needs
(e.g., the URIs of the default network map and the numerical routingcost
cost map) from the IRD.

Client-writers are encouraged to script their client so that it validates
the server's responses. That is, rather than just printing a cost map, the
client is encouraged to verify that it has the expected values. The
interop test will specify the maps in enough detail that clients will know
exactly what the server should return for any request.

Server-writers are encouraged to configure their server (especially the
IRD) so as to stress-test clients. That is, do things which are legal, but
unusual. For example, instead of having all resources in one IRD, the
server might have several linked (cascaded) IRDs. Or instead of offering
one filtered cost map, which handles all four combinations of cost metrics
& modes, plus constraints, the server might offer four separate filtered
cost map resources: one which handles numerical mode routingcost &
hopcount, but does not accept constraints, a second that handles the same
cost types, but which does accept constraints, and two more for ordinal
mode costs. This would verify that a client is able to select the correct
resource for a given request.


**** Server Requirements ****

Each server MUST offer the following resources:
* A full network map for the default network.
* Four full cost maps for the default network map: the routingcost &
hopcount metrics, in both numerical & ordinal modes.
* An endpoint property service for the pid property for the default
network map.

Each server MAY offer the following optional services:
* A filtered network map for the default network.
* A filtered cost map for the default network, optionally allowing cost
constraints.
* An endpoint cost service, covering the 4 cost metric/mode combinations,
optionally allowing cost constraints. The values must be derived from the
default network & cost maps.
* The filtered cost map & endpoint cost services may support constraints.
* An alternate network map plus the 4 associated full cost maps.
* An endpoint property service for the alternate network's pid property.
This may be the same EPS resource as for the default network, or a
different resource.
* A filtered network map for the alternate network.
* A filtered cost map for the alternate network map.

If offered, the filtered cost map & endpoint cost resources MUST cover all
four metric/mode combinations.

If offered, the costs for the endpoint cost service MUST must be derived
from the default network map.


**** Client Requirements ****

When a client is paired with a server, the client is given the URI of the
server's IRD, and the resource id of its alternate map (if the server &
client support alternate maps). The client must be able to extract all
other information from the IRD -- full map resources, their URIs, etc.

Each client MUST:

* Fetch the IRD.
* Locate & fetch the default network map and its four associated full cost
maps.
* Locate the EPS for the default network map's pid resource, and determine
the pid for a set of endpoints specified by the interop test.

Note that clients are not required to fetch the alternate network map (if
present). But every client must be able to parse an IRD with an alternate
network map, and distinguish the default network map, and its associate
resources, from the alternate map.

Optional client requirements:
* Use the filtered network map for the default network map to fetch PIDs
specified by the interop test.
* Use the filtered cost map for the default network map to fetch the costs
for several combinations of source pids, destination pids, metrics/modes,
and optional constraints, as specified by the interop test.
* Use the endpoint cost service to fetch the costs for several
combinations of source addresses, destination addresses, metrics/modes,
and optional constraints, as specified by the interop test.
* Locate & fetch the alternate map and its four full cost maps.
* Use the filtered network map for the alternate network map to fetch PIDs
specified by the interop test.
* Use the filtered cost map for the alternate network map to fetch the
costs for several combinations of source pids, destination pids,
metrics/modes, and optional constraints, as specified by the interop test.
* Fetch the custom endpoint property for a set of endpoints defined by the
interop test.


**** Error Testing ****

We will develop "torture test" scripts of bad client requests, and feed
them to each server to make sure the server gives reasonable error
responses. Participants are encouraged to contribute bad client requests
up to the time of the test. We will not specify the error tests in
advance, but will publish them after the interop is complete.

Note that the emphasis in on verifying that servers survive client errors.
It does not seem important to verify that clients survive server errors.
If you disagree, speak now or forever hold your peace.

    - Wendy Roome


_______________________________________________
alto mailing list
alto@...
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_alto&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=4G36iiEVb2m_v-0RnP2gx9KZJjYQgfvrOCE3789JGIA&m=kXIGnWFHVmEKkrU-3jzE_hUpglOe7Bj7H4Ys164dZBE&s=eXl0C-Yj1ZVYLA6lRFfpWyP1zKA5InHCoayBzITypGQ&e=

Join z.archive.alto-dev@lists.opendaylight.org to automatically receive all group messages.