Ulf Wiger
b2e9af0acb
Gajumaru Serialization Tests / tests (push) Successful in 48m24s
Introduce gmser_schema_export to emit portable ASN.1 type definitions from tag/version/template triples (RLP remains the wire format). Export tag/1 and rev_tag/1 for discovery and naming, with docs and eunit coverage.
272 lines
9.5 KiB
Markdown
272 lines
9.5 KiB
Markdown
# GM Serialization
|
|
|
|
Serialization helpers for the Gajumaru.
|
|
|
|
For an overview of the static serializer, see [this document](doc/static.md).
|
|
|
|
To export static templates as portable ASN.1 type definitions (for codegen
|
|
in other languages; wire format remains RLP), see
|
|
[doc/schema_export.md](doc/schema_export.md) and module `gmser_schema_export`.
|
|
|
|
## Build
|
|
|
|
$ rebar3 compile
|
|
|
|
|
|
## Test
|
|
|
|
$ rebar3 eunit
|
|
|
|
## Dynamic encoding
|
|
|
|
The module `gmser_dyn` offers dynamic encoding support, encoding most 'regular'
|
|
Erlang data types into an internal RLP representation.
|
|
|
|
Main API:
|
|
* `encode(term()) -> iolist()`
|
|
* `encode_typed(template(), term()) -> iolist()`
|
|
* `decode(iolist()) -> term()`
|
|
|
|
* `serialize(term()) -> binary()`
|
|
* `serialize_typed(template(), term()) -> binary()`
|
|
* `deserialize(binary()) -> term()`
|
|
|
|
In the examples below, we use the `decode` functions, to illustrate
|
|
how the type information is represented. The fully serialized form is
|
|
produced by the `serialize` functions.
|
|
|
|
The basic types supported by the encoder are:
|
|
* `integer()` (`anyint`, code: 246)
|
|
* `neg_integer()` (`negint`, code: 247)
|
|
* `non_neg_integer()` (`int` , code: 248)
|
|
* `binary()` (`binary`, code: 249)
|
|
* `boolean()` (`bool` , code: 250)
|
|
* `list()` (`list` , code: 251)
|
|
* `map()` (`map` , code: 252)
|
|
* `tuple()` (`tuple` , code: 253)
|
|
* `gmser_id:id()` (`id` , code: 254)
|
|
* `atom()` (`label` , code: 255)
|
|
|
|
(The range of codes is chosen because the `gmser_chain_objects` codes
|
|
range from 10 to 200, and also to stay within 1 byte.)
|
|
|
|
When encoding `map` types, the map elements are first sorted.
|
|
|
|
When specifying a map type for template-driven encoding, use
|
|
the `#{items => [{Key, ValueType} | {opt, Key, ValueType}]}` construct.
|
|
The key names are included in the encoding, and are match against the item
|
|
specs during decoding. If the key names don't match, the decoding fails, unless
|
|
for an `{opt, K, V}` item, in which case that item spec is skipped.
|
|
|
|
```erlang
|
|
T = #{items => [{a,int},{opt,b,int},{c,int}]}
|
|
E1 = gmser_dyn:encode_typed(T, #{a => 1, b => 2, c => 3}) ->
|
|
[<<0>>,<<1>>,[<<252>>,
|
|
[[[<<255>>,<<97>>],[<<248>>,<<1>>]],
|
|
[[<<255>>,<<98>>],[<<248>>,<<2>>]],
|
|
[[<<255>>,<<99>>],[<<248>>,<<3>>]]]]]
|
|
E2 = gmser_dyn:encode_typed(T, #{a => 1, c => 3}) ->
|
|
[<<0>>,<<1>>,[<<252>>,
|
|
[[[<<255>>,<<97>>],[<<248>>,<<1>>]],
|
|
[[<<255>>,<<99>>],[<<248>>,<<3>>]]]]]
|
|
gmser_dyn:decode_typed(T,E2) ->
|
|
#{c => 3,a => 1}
|
|
```
|
|
|
|
## Labels
|
|
|
|
Labels correspond to (existing) atoms in Erlang.
|
|
Decoding of a label results in a call to `binary_to_existing_atom/2`, so will
|
|
fail if the corresponding atom does not already exist.
|
|
|
|
This behavior can be modified using the option `#{missing_labels => fail | create | convert}`,
|
|
where `fail` is the default, as described above, `convert` means that missing atoms are
|
|
converted to binaries, and `create` means that the atom is created dynamically.
|
|
|
|
The option can be passed e.g.:
|
|
```erlang
|
|
gmser_dyn:deserialize(Binary, gmser_dyn:set_opts(#{missing_labels => convert}))
|
|
```
|
|
|
|
or
|
|
```erlang
|
|
gmser_dyn:deserialize(Binary, gmser_dyn:set_opts(#{missing_labels => convert}, Types))
|
|
```
|
|
|
|
By calling `gmser_dyn:register_types/1`, after having added options to the type map,
|
|
the options can be made to take effect automatically.
|
|
|
|
|
|
It's possible to cache labels for more compact encoding.
|
|
Note that when caching labels, the same cache mapping needs to be used on the
|
|
decoder side.
|
|
|
|
Labels are encoded as `[<<255>>, << AtomToBinary/binary >>]`.
|
|
If a cached label is used, the encoding becomes `[<<255>, [Ix]]`, where
|
|
`Ix` is the integer-encoded index value of the cached label.
|
|
|
|
## Examples
|
|
|
|
Dynamically encoded objects have the basic structure `[<<0>>,V,Obj]`, where `V` is the
|
|
integer-coded version, and `Obj` is the top-level encoding on the form `[Tag,Data]`.
|
|
|
|
```erlang
|
|
E = fun(T) -> io:fwrite("~w~n", [gmser_dyn:encode(T)]) end.
|
|
|
|
E(17) -> [<<0>>,<<1>>,[<<248>>,<<17>>]]
|
|
E(<<"abc">>) -> [<<0>>,<<1>>,[<<249>>,<<97,98,99>>]]
|
|
E(true) -> [<<0>>,<<1>>,[<<250>>,<<1>>]]
|
|
E(false) -> [<<0>>,<<1>>,[<<250>>,<<0>>]]
|
|
E([1,2]) -> [<<0>>,<<1>>,[<<251>>,[[<<248>>,<<1>>],[<<248>>,<<2>>]]]]
|
|
E({1,2}) -> [<<0>>,<<1>>,[<<253>>,[[<<248>>,<<1>>],[<<248>>,<<2>>]]]]
|
|
E(#{a=>1, b=>2}) ->
|
|
[<<0>>,<<1>>,[<<252>>,[[[<<255>>,<<97>>],[<<248>>,<<1>>]],[[<<255>>,<<98>>],[<<248>>,<<2>>]]]]]
|
|
E(gmser_id:create(account,<<1:256>>)) ->
|
|
[<<0>>,<<1>>,[<<254>>,<<1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1>>]]
|
|
```
|
|
|
|
Note that tuples and list are encoded the same way, except for the initial type tag.
|
|
Maps are encoded as `[<Map>, [KV1, KV2, ...]]`, where `[KV1, KV2, ...]` is the sorted
|
|
list of key-value tuples from `map:to_list(Map)`, but with the `tuple` type tag omitted.
|
|
|
|
## Template-driven encoding
|
|
|
|
Templates can be provided to the encoder by either naming an already registered
|
|
type, or by passing a template directly. In both cases, the encoder will enforce
|
|
the type information in the template.
|
|
|
|
If the template has been registered, the encoder uses the registered type specification
|
|
to drive the encoding. The code of the registered template is embedded in the encoded
|
|
output:
|
|
|
|
```erlang
|
|
gmser_dyn:encode_typed({int,int,int}, {1,2,3}) ->
|
|
[<<0>>,<<1>>,[<<253>>,
|
|
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>]]]]
|
|
|
|
Types = gmser_dyn_types:add_type(t3,1013,{int,int,int}).
|
|
gmser_dyn:encode_typed(t3, {1,2,3}, Types) ->
|
|
[<<0>>,<<1>>,[[<<3,245>>,<<253>>],
|
|
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>]]]]
|
|
```
|
|
|
|
Note that the original `<<253>>` type code is wrapped as `[<<3,245>>,<<253>>]`,
|
|
where `<<3,245>>` corresponds to the custom code `1013`.
|
|
|
|
Using the default option `#{strict => true}`, the decoder will extract the custom
|
|
type spec, and validate the encoded data against it. If the custom code is missing,
|
|
the decoder aborts. Using `#{strict => false}`, the custom code is used if it exists,
|
|
but otherwise, it's ignored, and the encoded data is decoded using the dynamic type
|
|
info.
|
|
|
|
### Alternative types
|
|
|
|
The dynamic encoder supports a few additions to the `gmserialization` template
|
|
language: `any`, `#{list => Type}`, `#{alt => [AltTypes]}` and `#{switch => [AltTypes]}`.
|
|
|
|
#### `any`
|
|
|
|
The `any` type doesn't have an associated code, but enforces dynamic encoding.
|
|
|
|
#### `list`
|
|
|
|
The original list type notation expects a key-value list, e.g.
|
|
|
|
`[{name, binary}, {age, int}]`
|
|
|
|
```erlang
|
|
EL = gmser_dyn:encode_typed([{name,binary},{age,int}], [{name,<<"Ulf">>},{age,29}]) ->
|
|
[<<0>>,<<1>>,[<<251>>,
|
|
[[<<253>>,[[<<255>>,<<110,97,109,101>>],[<<249>>,<<85,108,102>>]]],
|
|
[<<253>>,[[<<255>>,<<97,103,101>>],[<<248>>,<<29>>]]]]]]
|
|
```
|
|
Note that the encoding explicitly lays out a `[{Key, Value}]` structure, all
|
|
dynamically typed. This means it can be dynamically decoded without templates.
|
|
|
|
```erlang
|
|
gmser_dyn:decode(EL).
|
|
[{name,<<"Ulf">>},{age,29}]
|
|
```
|
|
|
|
In order to specify something like Erlang's `[integer()]` type, we can use
|
|
the following:
|
|
|
|
```erlang
|
|
gmser_dyn:encode_typed(#{list => int}, [1,2,3,4]) ->
|
|
[<<0>>,<<1>>,[<<251>>,
|
|
[[<<248>>,<<1>>],[<<248>>,<<2>>],[<<248>>,<<3>>],[<<248>>,<<4>>]]]]
|
|
```
|
|
|
|
#### `alt`
|
|
|
|
The `#{alt => [Type]}` construct also enforces dynamic encoding, and will try
|
|
to encode as each type in the list, in the specified order, until one matches.
|
|
|
|
```erlang
|
|
gmser_dyn:encode_typed(#{alt => [negint,int]}, 5) -> [<<0>>,<<1>>,[<<247>>,<<5>>]]
|
|
gmser_dyn:encode_typed(#{alt => [negint,int]}, 5) -> [<<0>>,<<1>>,[<<248>>,<<5>>]]
|
|
|
|
gmser_dyn:encode_typed(anyint,-5) -> [<<0>>,<<1>>,[<<246>>,[<<247>>,<<5>>]]]
|
|
gmser_dyn:encode_typed(anyint,5) -> [<<0>>,<<1>>,[<<246>>,[<<248>>,<<5>>]]]
|
|
```
|
|
|
|
#### `switch`
|
|
|
|
The `switch` type allows for encoding a 'tagged' object, where the tag determines
|
|
the type.
|
|
|
|
```erlang
|
|
E1 = gmser_dyn:encode_typed(#{switch => #{name => binary, age => int}}, #{age => 29}) ->
|
|
[<<0>>,<<1>>,[<<252>>,[[[<<255>>,<<97,103,101>>],[<<248>>,<<29>>]]]]]
|
|
gmser_dyn:decode_typed(#{switch => #{name => binary, age => int}}, E1) ->
|
|
#{age => 29}
|
|
E2 = gmser_dyn:encode_typed(#{switch => #{name => binary, age => int}}, #{name => <<"Ulf">>}) ->
|
|
[<<0>>,<<1>>,[<<252>>,[[[<<255>>,<<110,97,109,101>>],[<<249>>,<<85,108,102>>]]]]]
|
|
gmser_dyn:decode_typed(#{switch => #{name => binary, age => int}}, E1) ->
|
|
#{name => <<"Ulf">>}
|
|
```
|
|
|
|
A practical use of `switch` would be in a protocol schema:
|
|
|
|
```erlang
|
|
t_msg(_) ->
|
|
#{switch => #{ call => t_call
|
|
, reply => t_reply
|
|
, notification => t_notification }}.
|
|
|
|
t_call(_) ->
|
|
#{items => [ {id, anyint}
|
|
, {req, t_req} ]}.
|
|
|
|
t_reply(_) ->
|
|
#{alt => [#{items => [ {id, anyint}
|
|
, {result, t_result} ]},
|
|
#{items => [ {id, anyint}
|
|
, {code, anyint}
|
|
, {message, binary} ]}
|
|
]}.
|
|
```
|
|
|
|
In this scenario, messages are 'taggged' as 1-element maps, e.g.:
|
|
|
|
```erlang
|
|
async_request(Msg) ->
|
|
Id = erlang:unique_integer(),
|
|
gmmp_cp:to_server(
|
|
whereis(gmmp_core_connector),
|
|
#{call => #{ id => Id
|
|
, req => Msg }}),
|
|
Id.
|
|
```
|
|
|
|
### Notes
|
|
|
|
Note that `anyint` is a standard type. The static serializer supports only
|
|
positive integers (`int`), as negative numbers are forbidden on-chain.
|
|
For dynamic encoding e.g. in messaging protocols, handling negative numbers can
|
|
be useful, so the `negint` type was added as a dynamic type. To encode a full-range
|
|
integer, the `alt` construct is needed.
|
|
|
|
(Floats are not supported, as they are non-deterministic. Rationals and fixed-point
|
|
numbers could easily be handled as high-level types, e.g. as `{int,int}`.)
|