Agent versioning
Agent versioning
Introduction
To preserve the state across program and type changes, gall agents tag their state type with an explicit version number.
+$ state-1 [%1 my-state]
This enables the agent to discriminate the current state, and use a correct procedure for upgrade.
If an agent never communicated with the outside world, that would be sufficient. However, as soon as two parties are communicating with types, it is necessary to either make sure everyone is up to date, or use a versioning scheme for these types. The former is only feasible in a strictly local API. As soon as remote parties are involved, it is no longer possible to ensure they always stay in sync. In that case, we can either resort to blocking, or we may introduce a running version tag for each communicated type, that would allow the remote party to detect and deal with a possible incompatibility.
:: an example of two versions of a type
::
+$ diary-1 [content=(list @t)]
+$ diary-2 [content=(list @t) pages=@ud]
In addition, since the agent state type will usually depend, directly or indirectly, on at least some of these versioned types, a change of a type will propagate through the network of type dependencies, increasing versions of the affected types.
Versioning scheme
In a hoon application the following program components may be versioned.
- Types
- Application state
- Marks
- Scry paths
- Watch paths
Among these, the types are the most fundamental component and usually influence the version of the remaining four components.
Three kinds of types
Types are the woven fabric of an application. A change in a single type may ripple to affect a number of other types, sometimes effecting a change of the application state type. If a type is transferred over a network, a mark or a path may be affected as well. However, it is the application state that is almost always linked with type changes.
We distinguish the following three kinds of types.
- State type – it is the underlying type of a particular version of the application state.
- Anchored type – a type dependency of a state type. This dependency can be direct or indirect.
- Loose type – a type which is neither a state type nor anchored type.
Versioning of types
We now introduce a type versioning scheme that crucially rests on the version of the application state type.
The state type
The version of the state begins at %0 and advances in two cases.
- A change in any one of the anchored types affects the overarching state type, and thus requires a version number increase. The change here can be direct, if the state type changes by itself, or indirect if the change is buried deeper in the type hierarchy.
- Less commonly, there arises a need to apply a state value change during
+on-load, such as when fixing a bug. This too requires a version change to distinguish instances where the change had already come into effect.
Example: state type upgrade
:: /app/talk: simply chat
::
/- t=talk
::
+$ state-4
$: %4
banned=(set ship)
messages=(map msg-id:t message:v2:t)
==
+$ state-3
$: %3
banned=(set ship)
messages=(map msg-id:t message:v2:t)
==
+$ state-2
$: %2
banned=(set ship)
messages=(map msg-id:t message:v1:t)
==
+$ state-1
$: %1
banned=(set ship)
messages=(map msg-id:t message:v0:t)
==
+$ state-0
$: %0
messages=(map msg-id:t message:v0:t)
==
The state change from v0 to v1 is direct – a new field is added to the state, the set of banned ships. The changes from v1 to v2 and v2 to v3 are indirect. One of the anchored types, $message, is versioned at v1 and later at v2. This change is a consequence of a change deeper in the type hierarchy. The change from v3 to v4 must to be a state value change – the state type is not affected.
Example: state value upgrade
:: talk: simply chat
::
++ on-load
|^ |= =vase
^+ ta-core
=+ !<(any=any-state vase)
=? any ?=(%0 -.any) (state-0-to-1 any)
=? any ?=(%1 -.any) (state-1-to-2 any)
=? any ?=(%2 -.any) (state-2-to-3 any)
:: v3 -> v4
::
=? any ?=(%3 -.any)
:: fix a bug where we could ban ourselves
%_ any
- %4
banned (~(del in banned.any) our.bowl)
==
=. state any
ta-core
::
+$ any-state ?(state-0 state-1 state-2 state-3)
--
An excerpt from the talk agent upgrade logic. The upgrade from v3 to v4 is a state value change due to a bug in the application logic. Some users banned themselves. The fix, when deployed, removes the host from the banned list and bumps the state version to mark the instances where the fix has already come into effect.
Anchored types
A version change of an anchored type must increase the version of the application state. Thus, it is natural to consider the state version to be an anchor for the whole set of anchored types. Simple types which are unlikely to change may remain implicitly anchored and never gain an explicit version anchor. When a type becomes explicitly anchored, it gains an anchor at the earliest state version that the type was defined. A type once anchored should never lose its anchor.
:: /sur/talk/hoon
::
|%
+$ msg-id @ud
+$ message [time=@da archived=? story=(list verse)]
+$ verse
$% [%text p=@t]
[%mention p=ship]
==
+$ all-messages (list [id=msg-id =message])
:: version aliases
::
++ v2 .
++ v1 v1:ver
++ v0 v0:ver
:: |ver: version core
::
++ ver
|%
++ v1
=, v0
|%
++ message [time=@da archived=? story=(list verse)]
++ messages (list [id=msg-id =message])
--
++ v0
|%
++ message [time=@da story=(list verse)]
++ verse @t
++ messages (list [id=msg-id =message])
--
--
--
The structure file of the talk agent. The $msg-id, $message, and $verse are all anchored types. $msg-id and $message are directly anchored in the state type. $verse is an indirect anchored types as a type dependency of the $message type. $all-messages is a loose type, presumably a return type from a scry request.
The $message and $verse types have undergone a change, and gained an anchor at v0. $msg-id is a simple type that is unlikely to change. It remains implicitly anchored at v0. Were $msg-id one day be redefined, its old version would need to be anchored under the |v0 core.
The use of namespace import in the |v1 anchor core is an example of an anchor ladder. It allows the developer to surgically change types, while inheriting the types from the lower version anchor.
Loose types
Loose types, since they are not dependent on any of the anchored types, can be robustly versioned using a running version tag if more than one version of the type is required. If at one point a loose type is anchored, it loses the version tag and becomes anchored at the current version anchor.
In the preceding example, $all-messages is a loose type – it is not a part of the state type hierarchy. It can be a return type of a scry request. If a new version of the type were required, it would be version as $all-messages-1. An unversioned loose type is implicitly versioned at 0.
Comparison with other approaches
Versioning with separate files
A common approach to versioning is to simply keep a separate structure file for each bundle of type changes deployed under a common version state. This approach has an advantage: it makes any errors in not explicitly declaring type changes impossible, as can happen when anchor ladders are used in the same structure file. A significant drawback of this approach is the loss of intelligible information about successive state changes and simultaneous build-up of many files even for trivial changes of a single type.
Since the very point of anchor ladder method is the inheritance of unchanged types, failing to define a type defined at the top level and affected by a change under an appropriate version anchor could cause the type change to implicitly propagate to old version cores. Such a mistake would not always be reported by the compiler, as some changes could occur in a compatible manner.
This, however, can be guarded against simply by disallowing inheritance of types from the top level by separating the version core into a separate structure file. This should be the usual practice in production agents of enough complexity.
:: /sur/talk/hoon
::
/- ver=talk-ver
|%
+$ msg-id @ud
+$ message [time=@da archived=? story=(list verse)]
+$ verse
$% [%text p=@t]
[%mention p=ship]
==
+$ all-messages (list [id=msg-id =message])
:: version aliases
::
++ v2 .
++ v1 v1:ver
++ v0 v0:ver
--
:: /sur/talk-ver/hoon
::
|%
++ v1
=, v0
|%
++ message [time=@da archived=? story=(list verse)]
++ messages (list [id=msg-id =message])
--
++ v0
|%
++ msg-id @ud
++ message [time=@da story=(list verse)]
++ verse @t
++ messages (list [id=msg-id =message])
--
--
Versioning of marks
Historically, application marks were versioned in a way that is
unrelated to the version of the underlying type. The mark version number
would start either at 0 or 1, and would increase any time the underlying
mark type has changed. This has the obvious disadvantage of having to keep
track of, and manage, yet another version number.
A mark version change is not induced by any other change except that of the underlying type – change in the mark core implementation (the +grab, +grad and +jump arms) do not induce mark version changes, unless it becomes necessary to fix a bug.
We need to evaluate a mark versioning scheme where the mark version reflects the version of the underlying type. This can be done without introducing spurious marks – when the state version advances, this will, in general not advance the mark version. On a subsequent mark change, its version advances to the current state version.
The price that we pay for the clear association of the mark version with its type version are gaps in the version sequence. (It is worth noting that gaps would also appear in the anchored type versioning scheme, if we did not use anchor ladders.)
%talk-action-1 <-> talk-action:v1
%talk-action-4 <-> talk-action:v4
%talk-action-10 <-> talk-action:v10
Association between marks and their types. Version number can increase in jumps.
Current version marks
Some agents use a protocol negotiation library to be able to upgrade their types without adjusting endpoints and marks. When this is the case, the marks used in communication shielded with protocol negotiation are not versioned, and use the current version of the type.
Versioning of paths
Path version can change both as a consequence of a mark-type change, or a change in behavior. Furthermore, scry paths are sometimes found to return types constructed inline, for which no version number can be suitably defined. Thus, it is not possible to tie the path version with a type in the same way as with marks, neither it is desirable to do so.
Path versions begin at v1. v0 is reserved as an implicit default for
nonversioned paths. The version tag should always be the first component of a path,
to allow an easy discrimination between different versions of semantically equivalent paths.
Each time a path is affected by a change in type or other incompatible change,
its version number increases.
:: scry paths to return a list of messages.
:: note the endpoint version does not, in general,
:: correspond to the version of the return type. any matches
:: are coincidental.
::
/v1/talk/messages -> $messages:v0:g
/v2/talk/messages -> $messages:v2:g :: versions equal by coincidence
:: watch paths that publish messages sent to a particular ship
::
/v1/talk/ship=@ -> %talk-message-0
:: same fact mark but different behavior, and thus a different version
:: publish messages both received from, and sent to a ship.
::
/v2/talk/ship=@ -> %talk-message-0
Versioning of wires
Wires, on the other hand, are usually not versioned. An agent will typically want to handle subscriptions to a single version of a path at a time. The received facts themselves are versioned by the version of the mark (and thus by the corresponding version of the mark type). It is therefore ergonomic to handle the upgrade to a new endpoint by detecting and adjusting for a new mark, or discriminating among marks if necessary.
The only case where we should distinguish between versions of the same wire is if the agent truly requires subscriptions to different versions of the same path. In such cases, versioning becomes necessary, as duplicate wires are an application error.