[alto] Interop test


Y. Richard Yang
 

Wendy,

Good proposal! Two comments:

1. I liked that the interop is for a setting that is close to real-life deployment. In particular, my understanding of the proposal is that the interop should leave as much unspecified as possible. A first reaction then is how to validate the correctness. For example, the response of resource-id is not known ahead of time. I assume that then the interop participants will need a second channel (e.g., human in the loop) to verify that a client c1 gets the correct response from a server s1. Is this the intended setting?

2. A second comment is that it will be interesting if the interop design will also become a starting point of ALTO compliance tests. We partition the tests into MUST and optional so that a test case is a MUST case if and only if (iff) it is a MUST in RFC7285. In the current proposal below, I see some mix (e.g., some services in the interop is not a MUST in RFC7285). Make sense?

Richard

On Fri, May 29, 2015 at 10:24 AM, Wendy Roome <w.roome@...> wrote:
Folks,

Rather than simply update the last test, I suggest a new approach,
outlined below.

**** General ****

The interop test defines two network maps: the default network map and an
alternate network map. We specify the PIDs and CIDRs for each map. For
each network map, the interop test defines the numerical-mode values for
the routingcost and hopcount cost-metrics. The interop test does not
define ordinal values for those metrics. Instead, each ALTO server may
assign ordinal values however it wants, as long as the ordinal values
preserve the numerical values.

The interop test also defines a custom endpoint property, say
"priv:ietf-type", and specifies the values.


**** Approach & Philosophy ****

Rather than specifying every detail, as in the previous interop test, this
iteration only specifies what is absolutely necessary. That is, the PIDs
in the network maps, the costs between PIDs, and the enpdoint property
values. Everything else -- resource ids, uris, cost-type names, etc. -- is
up to the server. Thus each server is completely characterized by two
parameters:

* The URI of its IRD.
* The resource id of its alternate map (if provided).

When a client is paired with a server, the client is given those two
parameters. The client is expected to determine everything else it needs
(e.g., the URIs of the default network map and the numerical routingcost
cost map) from the IRD.

Client-writers are encouraged to script their client so that it validates
the server's responses. That is, rather than just printing a cost map, the
client is encouraged to verify that it has the expected values. The
interop test will specify the maps in enough detail that clients will know
exactly what the server should return for any request.

Server-writers are encouraged to configure their server (especially the
IRD) so as to stress-test clients. That is, do things which are legal, but
unusual. For example, instead of having all resources in one IRD, the
server might have several linked (cascaded) IRDs. Or instead of offering
one filtered cost map, which handles all four combinations of cost metrics
& modes, plus constraints, the server might offer four separate filtered
cost map resources: one which handles numerical mode routingcost &
hopcount, but does not accept constraints, a second that handles the same
cost types, but which does accept constraints, and two more for ordinal
mode costs. This would verify that a client is able to select the correct
resource for a given request.


**** Server Requirements ****

Each server MUST offer the following resources:
* A full network map for the default network.
* Four full cost maps for the default network map: the routingcost &
hopcount metrics, in both numerical & ordinal modes.
* An endpoint property service for the pid property for the default
network map.

Each server MAY offer the following optional services:
* A filtered network map for the default network.
* A filtered cost map for the default network, optionally allowing cost
constraints.
* An endpoint cost service, covering the 4 cost metric/mode combinations,
optionally allowing cost constraints. The values must be derived from the
default network & cost maps.
* The filtered cost map & endpoint cost services may support constraints.
* An alternate network map plus the 4 associated full cost maps.
* An endpoint property service for the alternate network's pid property.
This may be the same EPS resource as for the default network, or a
different resource.
* A filtered network map for the alternate network.
* A filtered cost map for the alternate network map.

If offered, the filtered cost map & endpoint cost resources MUST cover all
four metric/mode combinations.

If offered, the costs for the endpoint cost service MUST must be derived
from the default network map.


**** Client Requirements ****

When a client is paired with a server, the client is given the URI of the
server's IRD, and the resource id of its alternate map (if the server &
client support alternate maps). The client must be able to extract all
other information from the IRD -- full map resources, their URIs, etc.

Each client MUST:

* Fetch the IRD.
* Locate & fetch the default network map and its four associated full cost
maps.
* Locate the EPS for the default network map's pid resource, and determine
the pid for a set of endpoints specified by the interop test.

Note that clients are not required to fetch the alternate network map (if
present). But every client must be able to parse an IRD with an alternate
network map, and distinguish the default network map, and its associate
resources, from the alternate map.

Optional client requirements:
* Use the filtered network map for the default network map to fetch PIDs
specified by the interop test.
* Use the filtered cost map for the default network map to fetch the costs
for several combinations of source pids, destination pids, metrics/modes,
and optional constraints, as specified by the interop test.
* Use the endpoint cost service to fetch the costs for several
combinations of source addresses, destination addresses, metrics/modes,
and optional constraints, as specified by the interop test.
* Locate & fetch the alternate map and its four full cost maps.
* Use the filtered network map for the alternate network map to fetch PIDs
specified by the interop test.
* Use the filtered cost map for the alternate network map to fetch the
costs for several combinations of source pids, destination pids,
metrics/modes, and optional constraints, as specified by the interop test.
* Fetch the custom endpoint property for a set of endpoints defined by the
interop test.


**** Error Testing ****

We will develop "torture test" scripts of bad client requests, and feed
them to each server to make sure the server gives reasonable error
responses. Participants are encouraged to contribute bad client requests
up to the time of the test. We will not specify the error tests in
advance, but will publish them after the interop is complete.

Note that the emphasis in on verifying that servers survive client errors.
It does not seem important to verify that clients survive server errors.
If you disagree, speak now or forever hold your peace.

    - Wendy Roome


_______________________________________________
alto mailing list
alto@...
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_alto&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=4G36iiEVb2m_v-0RnP2gx9KZJjYQgfvrOCE3789JGIA&m=kXIGnWFHVmEKkrU-3jzE_hUpglOe7Bj7H4Ys164dZBE&s=eXl0C-Yj1ZVYLA6lRFfpWyP1zKA5InHCoayBzITypGQ&e=


Wendy Roome <w.roome@...>
 

I was assuming the interop test would specify the network & cost maps. E.g., the pid names, their CIDRs, and the costs between PIDs. Given that, and given the IRD entry for a resource (e.g., its media-type, accepts, uses and capabilities), a validating client should know exactly what that resource will return.

Okay, there are a few exceptions, such as the tag for a network map and the values of an ordinal cost map. But a client can verify that the network map vtag in a cost map matches the vtag in the network map, and it can verify that ordinal cost values are consistent with the order of the known numerical values.

Operationally, the ideal validating client would print the results from the server, followed by a polite "C'est bon" or a big "I DISAGREE!!!"  

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 11:40
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

1. I liked that the interop is for a setting that is close to real-life deployment. In particular, my understanding of the proposal is that the interop should leave as much unspecified as possible. A first reaction then is how to validate the correctness. For example, the response of resource-id is not known ahead of time. I assume that then the interop participants will need a second channel (e.g., human in the loop) to verify that a client c1 gets the correct response from a server s1. Is this the intended setting?


Y. Richard Yang
 

Wendy,

On Fri, May 29, 2015 at 12:11 PM, Wendy Roome <w.roome@...> wrote:
I was assuming the interop test would specify the network & cost maps. E.g., the pid names, their CIDRs, and the costs between PIDs. Given that, and given the IRD entry for a resource (e.g., its media-type, accepts, uses and capabilities), a validating client should know exactly what that resource will return.

Okay, there are a few exceptions, such as the tag for a network map and the values of an ordinal cost map. But a client can verify that the network map vtag in a cost map matches the vtag in the network map,

This makes sense. We can specify the validation condition for the interop:

- For network map, the only unspecified value then is the "vtag" field, and the interop specifies the assert, which is based on consistency between data and IRD.

- For cost map, the only unspecified is the exact "dependent-vtags", and your validation makes sense.

 
and it can verify that ordinal cost values are consistent with the order of the known numerical values.

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.

Richard

 

Operationally, the ideal validating client would print the results from the server, followed by a polite "C'est bon" or a big "I DISAGREE!!!"  

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 11:40
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

1. I liked that the interop is for a setting that is close to real-life deployment. In particular, my understanding of the proposal is that the interop should leave as much unspecified as possible. A first reaction then is how to validate the correctness. For example, the response of resource-id is not known ahead of time. I assume that then the interop participants will need a second channel (e.g., human in the loop) to verify that a client c1 gets the correct response from a server s1. Is this the intended setting?


Wendy Roome <w.roome@...>
 


But I have not seen IETF compliance tests for other protocols. Does the IETF officially sponsor/endorse them? Or must they be done outside of the IETF?

- Wendy Roome

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 11:40
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

2. A second comment is that it will be interesting if the interop design will also become a starting point of ALTO compliance tests. We partition the tests into MUST and optional so that a test case is a MUST case if and only if (iff) it is a MUST in RFC7285. In the current proposal below, I see some mix (e.g., some services in the interop is not a MUST in RFC7285). Make sense?


Y. Richard Yang
 


On Fri, May 29, 2015 at 12:26 PM, Wendy Roome <w.roome@...> wrote:

Super cool!
 
But I have not seen IETF compliance tests for other protocols. Does the IETF officially sponsor/endorse them? Or must they be done outside of the IETF?


I guess this is a question that the chairs/AD can provide the guidance :-)

Richard

 
- Wendy Roome

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 11:40
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

2. A second comment is that it will be interesting if the interop design will also become a starting point of ALTO compliance tests. We partition the tests into MUST and optional so that a test case is a MUST case if and only if (iff) it is a MUST in RFC7285. In the current proposal below, I see some mix (e.g., some services in the interop is not a MUST in RFC7285). Make sense?


Wendy Roome <w.roome@...>
 

I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

But if we take off our lawyer hats, and look at RFC 7285 as an agreement between friends, rather than a formal contract between adversaries, then I think it is reasonable to say that if the ordinal mode costs do not preserve the order of the numerical costs, then that server is wrong.

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)

And looking at it from a real-world perspective, the whole concept is irrelevant. The only reason for defining ordinal mode is to allow a server to hide the numerical costs. For example, if the numerical costs to three PIDs are 10, 11 and 100, a client can deduce that the middle pid is close. If the costs are 10, 99 and 100, a client can deduce that the middle pid is distant. If the ordinal costs are 1,2,3, a client cannot deduce anything other than 1 is better than 2 is better than 3.

So if a server offers numerical costs, there is no advantage for it to also offer ordinal mode costs.

And if numerical costs are available, there is no advantage to a client to use ordinal costs. Maybe if the client could assume the ordinal costs are 1,2,3,… -- but the client cannot.

So in practice, no server will offer both numerical & ordinal mode for the same metric.

That means a formal compliance test should only require one mode or the other, but not both. However, keep the interop simple, unless someone objects, I suggest we require both modes. 

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 12:24
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

 
and it can verify that ordinal cost values are consistent with the order of the known numerical values.

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.


Y. Richard Yang
 



On Saturday, May 30, 2015, Wendy Roome <w.roome@...> wrote:
I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

I remembered that the removal of the consistency requirement was a conscious decision, not an omission, to reduce the load of consistency checking. I buy your point below of a server supporting only one, in the sense of providing the maximum allowed by the privacy concern.

Regarding supporting both in the interop, I will be happy to wait a bit to hear others' opinions.

Richard

But if we take off our lawyer hats, and look at RFC 7285 as an agreement between friends, rather than a formal contract between adversaries, then I think it is reasonable to say that if the ordinal mode costs do not preserve the order of the numerical costs, then that server is wrong.

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)

And looking at it from a real-world perspective, the whole concept is irrelevant. The only reason for defining ordinal mode is to allow a server to hide the numerical costs. For example, if the numerical costs to three PIDs are 10, 11 and 100, a client can deduce that the middle pid is close. If the costs are 10, 99 and 100, a client can deduce that the middle pid is distant. If the ordinal costs are 1,2,3, a client cannot deduce anything other than 1 is better than 2 is better than 3.

So if a server offers numerical costs, there is no advantage for it to also offer ordinal mode costs.

And if numerical costs are available, there is no advantage to a client to use ordinal costs. Maybe if the client could assume the ordinal costs are 1,2,3,… -- but the client cannot.

So in practice, no server will offer both numerical & ordinal mode for the same metric.

That means a formal compliance test should only require one mode or the other, but not both. However, keep the interop simple, unless someone objects, I suggest we require both modes. 

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 12:24
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

 
and it can verify that ordinal cost values are consistent with the order of the known numerical values.

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.


--
Richard


Bertz, Lyle T [CTO] <Lyle.T.Bertz@...>
 

Wendy,

 

On Friday, May 29, 2015, Wendy Roome <w.roome@...> wrote:

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)”

 

I am pretty sure someone has actually tried to sell me one of these and can find it in the lab (but was not the purchaser).  To date its loss rate is close to 100%.  It is treated more like a very secure firewall than a router and is placed beside a tin pale that is a bit bucket someone purchased long ago. :D

 

On a serious note, I agree with you that only one mode would be supported on the server .   Please note that an ordinal ranking of a metric may exist if there are too many entries that have the same value in the original metric.  We do this quite often as an operator but we *always* give this metric a different name in order to emphasize the ranking aspect.  

 

I have a much larger question about ordinal rank in 7285, is it the expectation that ordinal ranks are true metrics in practice.  In other words are they or even the original metrics true mathematical metrics, i.e. non-negative, have symmetry, coincidence axiom and the triangle inequality.  Further are they formally ultrametrics or intrinsic?  

 

I only ask because depending on the answer we can add services for max distance, sum distance and the like that apply graph theory to the resulting maps.  Who know, I may even be able to apply my only graph theories to them but I will keep such aspirations low at the moment.

 

Lyle

 

PS – Are we looking to perform an interop in July or the next meeting?

 

From: Wendy Roome [mailto:w.roome@...]
Sent: Friday, May 29, 2015 12:06 PM
To: Y. Richard Yang
Cc: alto@...; Bertz, Lyle T [CTO]; Hans Seidel; alto-dev@...
Subject: Re: [alto] Interop test

 

I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

 

But if we take off our lawyer hats, and look at RFC 7285 as an agreement between friends, rather than a formal contract between adversaries, then I think it is reasonable to say that if the ordinal mode costs do not preserve the order of the numerical costs, then that server is wrong.

 

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)

 

And looking at it from a real-world perspective, the whole concept is irrelevant. The only reason for defining ordinal mode is to allow a server to hide the numerical costs. For example, if the numerical costs to three PIDs are 10, 11 and 100, a client can deduce that the middle pid is close. If the costs are 10, 99 and 100, a client can deduce that the middle pid is distant. If the ordinal costs are 1,2,3, a client cannot deduce anything other than 1 is better than 2 is better than 3.

 

So if a server offers numerical costs, there is no advantage for it to also offer ordinal mode costs.

 

And if numerical costs are available, there is no advantage to a client to use ordinal costs. Maybe if the client could assume the ordinal costs are 1,2,3,… -- but the client cannot.

 

So in practice, no server will offer both numerical & ordinal mode for the same metric.

 

That means a formal compliance test should only require one mode or the other, but not both. However, keep the interop simple, unless someone objects, I suggest we require both modes. 

 

                - Wendy

 

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 12:24
To: Wendy Roome <
w.roome@...>
Cc: "
alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

 

 

and it can verify that ordinal cost values are consistent with the order of the known numerical values.

 

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.




This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.


Wendy Roome <w.roome@...>
 

Interesting point!  RFC 7285 does NOT require cost metrics -- numerical or ordinal -- to follow the requirements for a formal metric. Other than the values must be non-negative.

In particular, there is no requirement that d(x,x) = 0, or that d(x,y) = 0 iff x = y.

Costs are directed, so symmetry isn't even appropriate.

I guess I would expect a numeric metric to follow the triangle inequality, more or less, but there is no formal requirement for it do so.

Incidentally, for ordinal mode costs, the values are only comparable to other costs in the same request. In other words, if an ordinal cost in a filtered cost map is 0, that just means it is the lowest cost for the set of sources & destinations you requested. It is NOT the lowest cost for the full map.

- Wendy Roome


From: "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>
Date: Fri, May 29, 2015 at 15:21
To: Wendy Roome <w.roome@...>, "Y. Richard Yang" <yry@...>
Cc: "alto@..." <alto@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: RE: [alto] Interop test

I have a much larger question about ordinal rank in 7285, is it the expectation that ordinal ranks are true metrics in practice.  In other words are they or even the original metrics true mathematical metrics, i.e. non-negative, have symmetry, coincidence axiom and the triangle inequality.  Further are they formally ultrametrics or intrinsic?  

 

I only ask because depending on the answer we can add services for max distance, sum distance and the like that apply graph theory to the resulting maps.  Who know, I may even be able to apply my only graph theories to them but I will keep such aspirations low at the moment.



Bertz, Lyle T [CTO] <Lyle.T.Bertz@...>
 

I had not put together the comparison applying to the same requests.   That makes matters convenient as the included data only applies to the response. 

 

Without formal metric compliance though there are a lot of algorithms that could not be reliably applied.  I think we should consider this as we move forward in developing ALTO.  It may be worthwhile to clarify matters in a future update.

 

From: Wendy Roome [mailto:w.roome@...]
Sent: Friday, May 29, 2015 2:45 PM
To: Bertz, Lyle T [CTO]; Y. Richard Yang
Cc: alto@...; Hans Seidel; alto-dev@...
Subject: Re: [alto] Interop test

 

Interesting point!  RFC 7285 does NOT require cost metrics -- numerical or ordinal -- to follow the requirements for a formal metric. Other than the values must be non-negative.

 

In particular, there is no requirement that d(x,x) = 0, or that d(x,y) = 0 iff x = y.

 

Costs are directed, so symmetry isn't even appropriate.

 

I guess I would expect a numeric metric to follow the triangle inequality, more or less, but there is no formal requirement for it do so.

 

Incidentally, for ordinal mode costs, the values are only comparable to other costs in the same request. In other words, if an ordinal cost in a filtered cost map is 0, that just means it is the lowest cost for the set of sources & destinations you requested. It is NOT the lowest cost for the full map.

 

                - Wendy Roome

 

 

From: "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>
Date: Fri, May 29, 2015 at 15:21
To: Wendy Roome <
w.roome@...>, "Y. Richard Yang" <yry@...>
Cc: "
alto@..." <alto@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: RE: [alto] Interop test

 

I have a much larger question about ordinal rank in 7285, is it the expectation that ordinal ranks are true metrics in practice.  In other words are they or even the original metrics true mathematical metrics, i.e. non-negative, have symmetry, coincidence axiom and the triangle inequality.  Further are they formally ultrametrics or intrinsic?  

 

I only ask because depending on the answer we can add services for max distance, sum distance and the like that apply graph theory to the resulting maps.  Who know, I may even be able to apply my only graph theories to them but I will keep such aspirations low at the moment.






This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.


Wendy Roome <w.roome@...>
 

Lyle,

I haven't used metric spaces since the days of keypunches, so it took a while to blow the cobwebs out of that memory bank. :-)

I don't think ALTO cost metrics could ever meet the requirements for a metric space. The problems:

* A true metric d(x,y) must be defined for all x & y. ALTO does not require that the cost be defined for all pairs.

* For a metric, d(x,y) = d(y,x) for all x,y. ALTO costs represent communication links, and are potentially asymmetric. Eg,  the download bandwidth can be higher than the upload bandwidth.

* For a metric, d(x,x) = 0 for all x. For ALTO, the cost within a PID frequently is greater than 0.

* For a metric, d(x,y) = 0 means x = y. We *might* be able to define ALTO costs so that is possible, but I have my doubts.

* For a metric, d(x,z) <= d(x,y) + d(y,z) for all y. For ALTO, that is tempting requirement, and I cannot think of an obvious counter example. But I doubt that would help without the other requirements.

So we could not require *every* ALTO cost metric to qualify as a mathematical metric. But we could define a particular cost as satisfying those rules, and (say) identify it in the IRD.  Can you explain how that would benefit clients or servers? One obvious advantage is that the cost matrix could be smaller, because its symmetric and the center diagonal is 0.

- Wendy Roome

From: "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>
Date: Fri, May 29, 2015 at 16:03
To: Wendy Roome <w.roome@...>, "Y. Richard Yang" <yry@...>
Cc: "alto@..." <alto@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: RE: [alto] Interop test

I had not put together the comparison applying to the same requests.   That makes matters convenient as the included data only applies to the response. 

 

Without formal metric compliance though there are a lot of algorithms that could not be reliably applied.  I think we should consider this as we move forward in developing ALTO.  It may be worthwhile to clarify matters in a future update.

 

From: Wendy Roome [mailto:w.roome@...]
Sent: Friday, May 29, 2015 2:45 PM
To: Bertz, Lyle T [CTO]; Y. Richard Yang
Cc: alto@...; Hans Seidel; alto-dev@...
Subject: Re: [alto] Interop test

 

Interesting point!  RFC 7285 does NOT require cost metrics -- numerical or ordinal -- to follow the requirements for a formal metric. Other than the values must be non-negative.

 

In particular, there is no requirement that d(x,x) = 0, or that d(x,y) = 0 iff x = y.

 

Costs are directed, so symmetry isn't even appropriate.

 

I guess I would expect a numeric metric to follow the triangle inequality, more or less, but there is no formal requirement for it do so.

 

Incidentally, for ordinal mode costs, the values are only comparable to other costs in the same request. In other words, if an ordinal cost in a filtered cost map is 0, that just means it is the lowest cost for the set of sources & destinations you requested. It is NOT the lowest cost for the full map.

 

                - Wendy Roome

 

 

From: "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>
Date: Fri, May 29, 2015 at 15:21
To: Wendy Roome <
w.roome@...>, "Y. Richard Yang" <yry@...>
Cc: "
alto@..." <alto@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: RE: [alto] Interop test

 

I have a much larger question about ordinal rank in 7285, is it the expectation that ordinal ranks are true metrics in practice.  In other words are they or even the original metrics true mathematical metrics, i.e. non-negative, have symmetry, coincidence axiom and the triangle inequality.  Further are they formally ultrametrics or intrinsic?  

 

I only ask because depending on the answer we can add services for max distance, sum distance and the like that apply graph theory to the resulting maps.  Who know, I may even be able to apply my only graph theories to them but I will keep such aspirations low at the moment.






This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.


Hans Seidel <hseidel@...>
 

Richard

On 29.05.2015 20:24, Y. Richard Yang wrote:


On Saturday, May 30, 2015, Wendy Roome <w.roome@...> wrote:
I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

I remembered that the removal of the consistency requirement was a conscious decision, not an omission, to reduce the load of consistency checking. I buy your point below of a server supporting only one, in the sense of providing the maximum allowed by the privacy concern.

Regarding supporting both in the interop, I will be happy to wait a bit to hear others' opinions.

I agree that providing both numerical and ordinal mode for the same cost metric makes little sense from a real world perspective. For the interop, I think both should be covered. Since Wendy already proposed cost maps for routingcost and hopcount in her initial mail, I suggest providing a numerical cost map for one metric and an ordinal for the other.

Hans


Richard

But if we take off our lawyer hats, and look at RFC 7285 as an agreement between friends, rather than a formal contract between adversaries, then I think it is reasonable to say that if the ordinal mode costs do not preserve the order of the numerical costs, then that server is wrong.

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)

And looking at it from a real-world perspective, the whole concept is irrelevant. The only reason for defining ordinal mode is to allow a server to hide the numerical costs. For example, if the numerical costs to three PIDs are 10, 11 and 100, a client can deduce that the middle pid is close. If the costs are 10, 99 and 100, a client can deduce that the middle pid is distant. If the ordinal costs are 1,2,3, a client cannot deduce anything other than 1 is better than 2 is better than 3.

So if a server offers numerical costs, there is no advantage for it to also offer ordinal mode costs.

And if numerical costs are available, there is no advantage to a client to use ordinal costs. Maybe if the client could assume the ordinal costs are 1,2,3,… -- but the client cannot.

So in practice, no server will offer both numerical & ordinal mode for the same metric.

That means a formal compliance test should only require one mode or the other, but not both. However, keep the interop simple, unless someone objects, I suggest we require both modes. 

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 12:24
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

 
and it can verify that ordinal cost values are consistent with the order of the known numerical values.

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.


--
Richard


Y. Richard Yang
 



On Tuesday, June 2, 2015, Hans Seidel <hseidel@...> wrote:
Richard

On 29.05.2015 20:24, Y. Richard Yang wrote:

On Saturday, May 30, 2015, Wendy Roome <w.roome@...> wrote:
I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

I remembered that the removal of the consistency requirement was a conscious decision, not an omission, to reduce the load of consistency checking. I buy your point below of a server supporting only one, in the sense of providing the maximum allowed by the privacy concern.

Regarding supporting both in the interop, I will be happy to wait a bit to hear others' opinions.

I agree that providing both numerical and ordinal mode for the same cost metric makes little sense from a real world perspective. For the interop, I think both should be covered. Since Wendy already proposed cost maps for routingcost and hopcount in her initial mail, I suggest providing a numerical cost map for one metric and an ordinal for the other. 

 This proposal makes sense to me. I support it.

Richard 


Hans


Richard

But if we take off our lawyer hats, and look at RFC 7285 as an agreement between friends, rather than a formal contract between adversaries, then I think it is reasonable to say that if the ordinal mode costs do not preserve the order of the numerical costs, then that server is wrong.

Similarly, a router is allowed to drop packets, and I do not think there is any formal requirement that it cannot drop *every* packet. So theoretically you could glue eight jacks to a block of hardwood and market it as a router. Maybe you could avoid getting charged for fraud. Just don't expect anyone to buy more than one! :-)

And looking at it from a real-world perspective, the whole concept is irrelevant. The only reason for defining ordinal mode is to allow a server to hide the numerical costs. For example, if the numerical costs to three PIDs are 10, 11 and 100, a client can deduce that the middle pid is close. If the costs are 10, 99 and 100, a client can deduce that the middle pid is distant. If the ordinal costs are 1,2,3, a client cannot deduce anything other than 1 is better than 2 is better than 3.

So if a server offers numerical costs, there is no advantage for it to also offer ordinal mode costs.

And if numerical costs are available, there is no advantage to a client to use ordinal costs. Maybe if the client could assume the ordinal costs are 1,2,3,… -- but the client cannot.

So in practice, no server will offer both numerical & ordinal mode for the same metric.

That means a formal compliance test should only require one mode or the other, but not both. However, keep the interop simple, unless someone objects, I suggest we require both modes. 

- Wendy

From: "Y. Richard Yang" <yry@...>
Date: Fri, May 29, 2015 at 12:24
To: Wendy Roome <w.roome@...>
Cc: "alto@..." <alto@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, Hans Seidel <hseidel@...>, "alto-dev@..." <alto-dev@...>
Subject: Re: [alto] Interop test

 
and it can verify that ordinal cost values are consistent with the order of the known numerical values.

This is a reasonable validation. An issue is that RFC7285 specifies only that "An ALTO server MUST support at least one of the following modes:  numerical and ordinal." I believe that RFC7285 chose to not specify the consistency between the two modes. Hence, this is beyond compliance, right? Hence, I feel that separating RFC7285-conforming and beyond can be helpful.


--
Richard





--
Richard


Wendy Roome <w.roome@...>
 

I suggest a slightly different approach: we specify the numerical values for routingcost & hopcount. Then let each server decide whether it wants to present numerical or ordinal versions. The only requirement is that a server MUST provide a routingcost cost map, either numerical or ordinal. Servers can provide whatever additional resources they want, and clients can fetch & validate any resources they recognize.

We do not define ordinal values, but allow servers to assign whatever values they want as long as the ordering is consistent with the numerical values we specify. That is sufficient to allow a client to verify the ordinal values.

- Wendy Roome

From: Hans Seidel <hseidel@...>
Date: Tue, June 2, 2015 at 03:00
To: "Y. Richard Yang" <yry@...>
Cc: Wendy Roome <w.roome@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, "alto-dev@..." <alto-dev@...>, "alto@..." <alto@...>
Subject: Re: [alto] Interop test

On 29.05.2015 20:24, Y. Richard Yang wrote:
On Saturday, May 30, 2015, Wendy Roome <w.roome@...> wrote:
I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

I remembered that the removal of the consistency requirement was a conscious decision, not an omission, to reduce the load of consistency checking. I buy your point below of a server supporting only one, in the sense of providing the maximum allowed by the privacy concern.

Regarding supporting both in the interop, I will be happy to wait a bit to hear others' opinions.
I agree that providing both numerical and ordinal mode for the same cost metric makes little sense from a real world perspective. For the interop, I think both should be covered. Since Wendy already proposed cost maps for routingcost and hopcount in her initial mail, I suggest providing a numerical cost map for one metric and an ordinal for the other. 

Hans


郭华明 <guohuaming@...>
 

I think that we just specify the numerical & ordinal values for routingcost cost map, then let each server decide whether it wants to present numerical or ordinal versions. The hopcount cost map is no need to specify, as it is not a MUST in
RFC 7285.


-----原始邮件-----
发件人:"Wendy Roome" <w.roome@...>
发送时间:2015-06-02 22:17:46 (星期二)
收件人: "Hans Seidel" <hseidel@...>, "Y. Richard Yang" <yry@...>
抄送: "alto-dev@..." <alto-dev@...>, "alto@..." <alto@...>
主题: Re: [alto] Interop test

I suggest a slightly different approach: we specify the numerical values for routingcost & hopcount. Then let each server decide whether it wants to present numerical or ordinal versions. The only requirement is that a server MUST provide a routingcost cost map, either numerical or ordinal. Servers can provide whatever additional resources they want, and clients can fetch & validate any resources they recognize.

We do not define ordinal values, but allow servers to assign whatever values they want as long as the ordering is consistent with the numerical values we specify. That is sufficient to allow a client to verify the ordinal values.

- Wendy Roome

From: Hans Seidel <hseidel@...>
Date: Tue, June 2, 2015 at 03:00
To: "Y. Richard Yang" <yry@...>
Cc: Wendy Roome <w.roome@...>, "Bertz, Lyle T [CTO]" <Lyle.T.Bertz@...>, "alto-dev@..." <alto-dev@...>, "alto@..." <alto@...>
Subject: Re: [alto] Interop test

On 29.05.2015 20:24, Y. Richard Yang wrote:
On Saturday, May 30, 2015, Wendy Roome <w.roome@...> wrote:
I thought RFC 7285 required order-consistency between numerical & ordinal modes for the same metric. But I cannot find that requirement. Too bad! I would have added that if I realized it wasn't there.

I remembered that the removal of the consistency requirement was a conscious decision, not an omission, to reduce the load of consistency checking. I buy your point below of a server supporting only one, in the sense of providing the maximum allowed by the privacy concern.

Regarding supporting both in the interop, I will be happy to wait a bit to hear others' opinions.
I agree that providing both numerical and ordinal mode for the same cost metric makes little sense from a real world perspective. For the interop, I think both should be covered. Since Wendy already proposed cost maps for routingcost and hopcount in her initial mail, I suggest providing a numerical cost map for one metric and an ordinal for the other. 

Hans

--
----------------
Best!


Huaming Guo

China Academy of Information and Communications Technology (CAICT)

No.36 A Nanlishi Road, Xicheng District, Beijing 100037, China