You are reading content from Scuttlebutt
@bundy

What are the layers of Scuttlebutt?

I'm feeling inspired to write a subjective post on the state of scuttlebutt, but it seems like the first step would be to draw up a reasonable taxonomy for the components that make Scuttlebutt. I've seen some lovely stack diagrams in the past, but these seem more focused on the implementation layers of Scuttlebutt rather than the features.

Diagram of the implementation layers of Scuttlebutt, focused on secret-stack and secure-scuttlebutt (now ssb-db) implementation layers.

I'm going to brain-dump some layers, feedback and contributions very welcome!

  • Storage: How and where to save the data that we create or receive.
  • Identity: Pair of corresponding public and private keys that are used for identification and verification.
  • Message: Small piece of data that's hashed and signed with our identity.
    • Content schema: Shared understanding of what a message intends.
    • Private: Encrypted message that can only be read by the recipient identities.
  • Blob: Content-addressable data that may referenced by a message via metadata.
  • Feed: Singly-linked list of messages from an identity that forms a signature chain.
  • Peer: Any other instance of Scuttlebutt that we can connect to and communicate with.
  • Discovery: Collection of methods for discovering peers and their identities.
  • Connection: Method of networking with peers so that we can communicate.
  • Communication: Shared language and expectations for how peers communicate with each other.
  • Replication: Method of communicating feeds and blobs with peers over a connection.

Is there anything I'm missing, or are there any suggestions on how to categorize this better? In the future I'd like to go through each category and explore the good/bad/ugly, but it seems important to nail down a useful taxonomy first.

@The Person You Have Called
Voted this
@bundy

Oops, one clarification that I'd like to make: this is for any implementation that we can all agree is Scuttlebutt, so I've purposefully left out any specifics of the JavaScript implementation in 2019. For example, this doesn't mention JSON because if we all switched to something else then I don't think it'd stop being Scuttlebutt.

alt implementation folks I'd especially love feedback from, cc: @aljoscha @cryptix @keks @mikey @piet @matt @christoph @seanb

@Connor
Voted this
@bobhaugen
Voted this
@aran
Voted this
@punkmonk.termux
{
  "type": "queue",
  "message": "%zy1rB14YAwIfL1wtpVCbcStkiGV3BICpShbMwxCVjlM=.sha256",
  "queue": true
}
@Dan Hassan
Voted this
@Dan Hassan
Voted this
@Dan Hassan

wave.jpg love this so here is an image of support for you to rest your eyes on...

@piet
Voted this
@mikey
Voted this
@Justin Abrahms
{
  "type": "tag",
  "version": 1,
  "tagged": true,
  "message": "%zy1rB14YAwIfL1wtpVCbcStkiGV3BICpShbMwxCVjlM=.sha256",
  "root": "%n+Sp77wulNwEwa0n7aSsZ1nWOzHp394joVzkuRFN4Lc=.sha256",
  "branch": []
}
@teq mobile
Voted this
@bundy
Voted this
@David Wales
Voted this
@cft

Nice, @Christian Bundy
There is one thing which I think is missing: data structures.
All implementations will come up (internally or with equivalent libraries) with abstractions like:

  • the whole set of SSB logs
  • the set of SSB logs within your social horizon
  • a set of SSB logs selected by other criteria
  • a discussion thread (with methods to browse and append to, and dialable/based on one of the above views)
  • a closed group
  • a discussion thread inside a closed group
  • a user directory
  • extensible user record
  • an eventually consistent key-value store (easy then to have the user record)
  • an eventually consistent bag or set (e.g. collect all threads for some hashtag channel)
  • eventually consistent counters (e.g. single-ballot voting, for closed groups as well as open, cleartext as well as secret vote)
  • walkable tree
  • a notification stream for changes of any of the above

#ssb-datastructures

@elavoie
Voted this
@Dan Hassan
Voted this
@darius
Voted this
User has chosen not to be hosted publicly
@Dominic

this diagram was correct a month ago, but recently scuttlebot was renamed to ssb-server, and secure-scuttlebutt renamed to ssb-db, and then ssb-db was refactored to just be a secret-stack plugin. ssb-server is just a bundle of plugins now, and a script to start a server, and use cli commands.

@condiosluzverde/mobile

Might I suggest that the taxonomy of implementation-free names - to replace all instances of "what we all agree upon" that makes "it" secure scuttlebutt - could br referred to as the Secure ScuttleButt Protocol?

More concisely: ssb protocol.

Then that is what you might organise into layers, again implenentation-free.

What do you think?

@enkiv2
Voted this
@bundy

@cft

Interesting, thanks! In this model do you think that "data structures" should be built on top of "storage"? My goal is to be able to write about the state of Scuttlebutt on each of these components, but I'm unsure how I'd review the state of Scuttlebutt data structures. Is there a better way to organize them into this taxonomy?


@dominic

Yeah, and it's also specific to the JavaScript implementation. I included it to give an example of a stack diagram, but the bit I'm looking to iterate on is the list below with the general Scuttlebutt components.

@Cosmic Love Guru
Voted this
@xj9
Voted this
@keks
Voted this
@Nicolas Stampf (phone)
Voted this
@forward-looking
Voted this
@The Scuttleverse Herald
Re: %TVCRMYOIg

@Christian Bundy put together a nice overview of the layers of Scuttlebutt.

Diagram of the implementation layers of Scuttlebutt, focused on secret-stack and secure-scuttlebutt now ssb-db implementation layers


@andrestaltz_phone linked to

Really interesting hashing alternative to Merkle trees, devised by Facebook. The problem they describe (propagating updates in a distributed network) sounds quite close to Scuttlebutt.

Andre's thread is here and the post on "homomorphic hashing" is here.

-----------------------------------

This week’s summary was curated from 298 threads by 136 authors by @masukomi, with the support of my lovely patrons. If you'd like to help continue this creation you can become a patron on Patreon.

post 3 of 3

@tulsi
Voted this
@rabble
Voted this
@blaine
Voted this
@Antoine Cailly
{
  "type": "tag",
  "version": 1,
  "tagged": true,
  "message": "%zy1rB14YAwIfL1wtpVCbcStkiGV3BICpShbMwxCVjlM=.sha256",
  "root": "%6TJHwANg7Y+qFIZgSP3GWx99VlmjnG8124lIoKDzukY=.sha256",
  "branch": [
    "%eH2oWTmpSgMxDqnM4dxjVSBXE6e8V+TZqwJBmjmlXYk=.sha256"
  ]
}
@seanb

@Christian Bundy

I think the layers you've defined are a good level of abstraction to introduce SSB, and to discuss potential improvements. What would have been useful for me getting started (and still useful now) is a layer map that's specific enough to call out the concrete protocols that are used, but isn't necessarily tied to how things are organized into node modules. So, something with the layers you've defined, where each layer or category discusses the current protocols, eg Communication is {secret handshake, box stream, rpc}, Storage is (usually) {flume offset log, views}, Replication includes some discussion about how this currently works.

I'd love to hear your subjective thoughts about the current state. I've spent a lot of time trying to get a read on the collective feelings about the current pieces, and how they might evolve, but it's been tough. I recently discovered
@bobhaugen's collection of threads on such topics, which gives me a lot more reading to do (thanks, Bob!) I'll develop my own feelings as I implement or touch more of the stack, but a lot of people here have had experience shaping ssb, and watching it grow, which I'll always lack.

@Pipon
Voted this
@bundy
Voted this
@stevie
Voted this
@habitatm45
Voted this
@maged
Voted this
@w
Voted this
@Aadil Ayub
Voted this
User has chosen not to be hosted publicly
Join Scuttlebutt now