You are reading content from Scuttlebutt
Feed of @Dominic

A handsome ocean wanderer with wonderfully comic pink feet.


author field in message signing format instead.

@aljoscha has proposed that the author field be removed from the signing and hashing format. He has pointed out that an implementor may still choose to keep the message in the database format, and even include the author in the transport format.

My understanding is that the benefit of removing the author field is:

  • reduced bandwidth, if the author field is not sent
  • more "perfection" because something unnecessary* is not included.

@aljoscha please clarify if this is correct, or if I have missed something.

* (note, I'm not yet convinced of this)

counter suggestion

I think security should be the primary consideration of the signing/hashing format, and the second should be straightforwardness. It should be easy to verify that we have produced the same implementation that does the same thing. and it should also be easy to be confidant that all implementations will handle the same message the same way.
Not including the author in signing makes this harder to reason about. I want security to be easy to reason about.

However, the optionality of the author can be shifted into the transport layer.
A transport layer could drop repeated fields that are deducable by context (according to some protocol), and then those could be reconstructed to verify the signature.

There are some signature formats that can extract the key from the signature. for example, secp256k1, as used by bitcoin and ethereum. but this extracts two public keys one of which corresponds to the private key used to sign the message, and the other one is a ghost, with no corresponding private key. I consider this a weird edge case. I'm not sure if ed25519 keys can be extracted from signatures, but none of the libraries support it.

If the public key (or hash of public key) is included in the signed format, then I can be sure I am verifying the signature with the same key it was signed with. If the signature is not included, then I am worried about ways insecure messages may be constructed, such that they can be verified with multiple public keys. I could go into detail here, after some research, but I don't feel the onus should be on me to prove that this will be secure, given that including the public key obviously will be.

Basically, the question is: "is it possible for anyone to generate another public key that also verifies this message" (where that someone may be the private key holder) we need to answer that question 100% "no". Discussing this with @keks, we both agree that we'd need quite a bit more math knowledge to verify this, and we could not verify it for other signature algorithms.

Algorithm agility is an important consideration. Asymmetric cryptography (signatures) are considered much weaker than hashes, so we'll need to upgrade them at some point, and we can't perfectly anticipate what properties they have.


just to back up the idea that there are signature algorithms with weird properties:

BLS aggregate signatures
more bls

Re: %mBiz5k4Dx

@arj yes the db implementor always has the ability to store in a different format, as long as they are able to reconstruct the signing format. It's impossible for the protocol to enforce the storage format. I would simply prefer a signing format that happens to works well for my database design.


format upgrade policy guide

I'd like to document a series of design values that will make upgrading the format easy.
This is both because implementing in a currently used server implementation will be more difficult,
and also because if we can't present messages to clients in a backwards compatible way, then it will make
front end clients difficult to upgrade and that will delay actually rolling out the new protocol to users.


  • protocol layer - the bytes to be sent and accepted
  • platform layer - local interface, exposed to applications
  • application layer - the way that applications format messages, etc

We are currently discussing changes to both the protocol and platform layer.

semantics of protocol layer

I think we should first agree on what fields are present, what their names are, and what they do. Then how that is encoded is a separate discussion.

It's easier to add new field names than remove field names.
Field names should not be changed. Removing a field needs strong justification.

For example, in bamboo, I would prefer backlink to be previous. lippmalink is okay, because it's a new field.
I do not agree we should remove the author field.

I'm okay with shifting timestamp and type into the message content, and not validating it.
If the message type is not present, it can just be defaulted to "about"
(trigger warning: previous sentence was a joke ;)

There is a large amount of code that I maintain,
and the less changes I need to make the easier this will be to roll out.
Also, there are many other people that maintain code that depends on my code, they mostly are not taking part in this discussion. They have not committed to rewriting their code and I don't want to thrust that on them.

encoding for signing and hashing

I'm okay with a separate encoding format for message metadata than the content. I'll accept protobuf or a custom format, or the same format as recommended for the content. Mainly, I want the bytes sent, signed, and stored to be the same, or as close as possible.

If we are signing content hash, instead of including the content, then we cannot validate the content encoding. This means that content is firmly in the platform layer now, not something that can be enforced by the protocol. If the message is sent without content, then we can't validate it until some future point when we receive the content. That means the content is just arbitrary bytes. That means a content format is just recommended content format. That also means that implementations must store the content as received. There can be no parsing and re-serializing. If I use non-cannonical json, it must stay non-cannonical.

I previously argued for non-cannonical encoding

However, implementors of the platform layer can form a consensus that they will use a particular format.

I support this design, but we must accept it's consequences.

okay the call is about to start so I'm gonna post this...

Wrote something private
Re: %9vW9vgkE1

I tried to find a photo of that future planning board but I couldn't find it... someone must have taken a photo of it though... #scuttlecamp

Re: %9vW9vgkE1

At scuttlecamp there was a bit where there was a planning thing on the wall, and we were all to put post-it notes with the various things we envisioned for the future. there was one color for "must have" and another one for "would be cool" @dangerousbeans you wrote "scuttleship" on "must have" color, and asked me where to put it, in the "3-5 year section"? I was like, nah put it definitely next year. I was sort of joking, I thought it was a crazy idea, but maybe this made it more likely to happen.

and here we are!

Voted # Scuttlebutt has bought it's first ship ![alchemist_ssb.jpg](&s0u8mCiNICu
Wrote something private
Wrote something private
Re: %da6rw16rU

I'm gonna be in UK soon. Barncamp got cancelled... but I'm still coming.

Show whole feed
Join Scuttlebutt now