From f0fe323fa1e32555aeb21d9332ad30ff319084b7 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Fri, 3 Sep 2021 15:14:09 -0400 Subject: [PATCH 1/5] Simplify the immutable share reading interface --- docs/proposed/http-storage-node-protocol.rst | 25 +++++++------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/docs/proposed/http-storage-node-protocol.rst b/docs/proposed/http-storage-node-protocol.rst index d7a41e827..fcf1e43fe 100644 --- a/docs/proposed/http-storage-node-protocol.rst +++ b/docs/proposed/http-storage-node-protocol.rst @@ -509,28 +509,21 @@ For example:: [1, 5] -``GET /v1/immutable/:storage_index?share=:s0&share=:sN&offset=o1&size=z0&offset=oN&size=zN`` -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +``GET /v1/immutable/:storage_index/:share_number?offset=:offset&length=:length +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -Read data from the indicated immutable shares. -If ``share`` query parameters are given, selecte only those shares for reading. -Otherwise, select all shares present. -If ``size`` and ``offset`` query parameters are given, -only the portions thus identified of the selected shares are returned. -Otherwise, all data is from the selected shares is returned. +Read a contiguous sequence of bytes from one share in one bucket. +If the ``offset`` query parameter is given then it is interpreted as a base 10 representation of an integer giving the position at which to begin reading. +If it is not given then begin reading at the beginning of the share. +If the ``length`` query parameter is given then it is interpreted as a base 10 representation of an integer giving the maximum number of bytes to read and return. +If it is not given then bytes will be read until the end of the share is reached. -The response body contains a mapping giving the read data. -For example:: - - { - 3: ["foo", "bar"], - 7: ["baz", "quux"] - } +The response body is the raw share data (i.e., ``application/octet-stream``). Discussion `````````` -Offset and size of the requested data are specified here as query arguments. +Offset and length of the requested data are specified here as query arguments. Instead, this information could be present in a ``Range`` header in the request. This is the more obvious choice and leverages an HTTP feature built for exactly this use-case. However, HTTP requires that the ``Content-Type`` of the response to "range requests" be ``multipart/...``. From 895c77fecf9559a31a7770ae98841846124e995e Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Fri, 3 Sep 2021 15:15:48 -0400 Subject: [PATCH 2/5] news fragment --- newsfragments/3777.documentation | 1 + 1 file changed, 1 insertion(+) create mode 100644 newsfragments/3777.documentation diff --git a/newsfragments/3777.documentation b/newsfragments/3777.documentation new file mode 100644 index 000000000..7635cc1e6 --- /dev/null +++ b/newsfragments/3777.documentation @@ -0,0 +1 @@ +The Great Black Swamp proposed specification now has a simplified interface for reading data from immutable shares. From 18cbff1789c6f3525c3f8127b629f534bc5c9a9f Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Fri, 3 Sep 2021 15:22:29 -0400 Subject: [PATCH 3/5] minimal discussion about this change --- docs/proposed/http-storage-node-protocol.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/proposed/http-storage-node-protocol.rst b/docs/proposed/http-storage-node-protocol.rst index fcf1e43fe..fb327b0a8 100644 --- a/docs/proposed/http-storage-node-protocol.rst +++ b/docs/proposed/http-storage-node-protocol.rst @@ -534,6 +534,15 @@ There are many drawbacks to this framing technique: 2. It is resource-intensive to parse. 3. It is complex to parse safely [#]_ [#]_ [#]_ [#]_. +A previous revision of this specification allowed requesting one or more contiguous sequences from one or more shares. +This *superficially* mirrored the Foolscap based interface somewhat closely. +The interface was simplified to this version because this version is all that is required to let clients retrieve any desired information. +It only requires that the client issue multiple requests. +This can be done with pipelining or parallel requests to avoid an additional latency penalty. +In the future, +if there are performance goals, +benchmarks can demonstrate whether they are achieved by a more complicated interface or some other change. + Mutable ------- From 82f94ae5af4791ff8d322f908432ae8c1fc7b2d0 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 7 Sep 2021 14:17:38 -0400 Subject: [PATCH 4/5] Yay use the Range request header --- docs/proposed/http-storage-node-protocol.rst | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/docs/proposed/http-storage-node-protocol.rst b/docs/proposed/http-storage-node-protocol.rst index eb5154e45..5f67fcaa6 100644 --- a/docs/proposed/http-storage-node-protocol.rst +++ b/docs/proposed/http-storage-node-protocol.rst @@ -539,24 +539,20 @@ For example:: [1, 5] -``GET /v1/immutable/:storage_index/:share_number?offset=:offset&length=:length -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +``GET /v1/immutable/:storage_index/:share_number +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Read a contiguous sequence of bytes from one share in one bucket. -If the ``offset`` query parameter is given then it is interpreted as a base 10 representation of an integer giving the position at which to begin reading. -If it is not given then begin reading at the beginning of the share. -If the ``length`` query parameter is given then it is interpreted as a base 10 representation of an integer giving the maximum number of bytes to read and return. -If it is not given then bytes will be read until the end of the share is reached. - The response body is the raw share data (i.e., ``application/octet-stream``). +The ``Range`` header may be used to request exactly one ``bytes`` range. +Interpretation and response behavior is as specified in RFC 7233 ยง 4.1. +Multiple ranges in a single request are *not* supported. Discussion `````````` -Offset and length of the requested data are specified here as query arguments. -Instead, this information could be present in a ``Range`` header in the request. -This is the more obvious choice and leverages an HTTP feature built for exactly this use-case. -However, HTTP requires that the ``Content-Type`` of the response to "range requests" be ``multipart/...``. +Multiple ``bytes`` ranges are not supported. +HTTP requires that the ``Content-Type`` of the response in that case be ``multipart/...``. The ``multipart`` major type brings along string sentinel delimiting as a means to frame the different response parts. There are many drawbacks to this framing technique: From a988f126bee53252c0692dda40b3c0262000cba4 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Tue, 7 Sep 2021 16:12:01 -0400 Subject: [PATCH 5/5] fix markup error --- docs/proposed/http-storage-node-protocol.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/proposed/http-storage-node-protocol.rst b/docs/proposed/http-storage-node-protocol.rst index 5f67fcaa6..a84d62176 100644 --- a/docs/proposed/http-storage-node-protocol.rst +++ b/docs/proposed/http-storage-node-protocol.rst @@ -539,8 +539,8 @@ For example:: [1, 5] -``GET /v1/immutable/:storage_index/:share_number -!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! +``GET /v1/immutable/:storage_index/:share_number`` +!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Read a contiguous sequence of bytes from one share in one bucket. The response body is the raw share data (i.e., ``application/octet-stream``).