Willow & Earthstar's spring
Published on June 13, 2024 by Sam Gwilym
My last project update was in October. Whoops. It’s not for a lack of news.
In the last year Willow has grown from an idea to a fully specified family of protocols with implementations.
willowprotocol.org
In February Aljoscha and I launched willowprotocol.org, which we’d been working on for about eight months. After meeting in Berlin the previous summer to present Willow at p4p@offline, we came away feeling energised to specify Willow and get it out into the world. Between last June and this February, what took us so long in finishing the website?
Some of the reasons are obvious:
- Thinking things through enough to specify them uncovered many areas we’d not considered.
- Implementing ideas as we went along proved some ideas to be impractical.
- Exploring ideas uncovered completely new systems along the way, like a whole capability system (thanks to Ar Nazeh for prompting this exploration!)
Some other reasons it took so long are completely self-inflicted:
- Building an undocumented, experimental macro system to enable cross-referencing with content-addressed previews, generated RSS feeds, and possibly too much more.
- Insisting on adding hand-drawn diagrams and illustrations to many of the pages.
- Covering all our bases and doing write-ups for questions we knew we’d get, e.g.Timestamps, really?
It was a lot of work, but it seems that effort was appreciated. On the day we published the site, willowprotocol.org made it to the top of the orange site and our server took approximately one million requests over 24 hours.
We were very happy to see that our designs for resilient, equitable, and efficient networking resonated with so many people!
The aforementioned undocumented, experimental macro system is being formalised into a new library we have affectionately named Macromania. It’s been designed with an eye to producing arbitrary output — so not just websites, but documents like academic papers, too. Hopefully more to share on that soon!
TypeScript implementations
Shortly following the website we published our first implementation, meadowcap-js. A few months later we published willow-js, which has implementations of the Willow data model (store included), and the Willow General Purpose sync protocol.
Earthstar
With the first Willow implementation published, we started work on incorporating Willow into Earthstar.
Earthstar is the precursor to Willow, and many of Willow’s concepts (e.g. the three dimensional user / path / timestamp data model) are formalised versions of Earthstar’s. Now we get to complete the circle and rebuild Earthstar upon the conceptual foundations Willow provides.
Here are a few highlights of what Earthstar 11 will gain from using Willow:
- A fine-grained capability system with user-specific permissions.
- Efficient, partial sync.
- A secure private set intersection algorithm for sync.
Right now, building upon Willow has two parts to it:
The first part is Earthstar’s new specification (currently a proposal) , which presents itself as an instantiation of the Willow data model, Meadowcap, and using the Willow general purpose sync protocol for sync. It’s much lighter and far more rigorous than the previous specification.
The second part of that is a new beta channel for Earthstar 11 , which you can try today in browsers, Deno, and Node.
Sideloading specification
We’re building Willow as the basis for resilient networking systems that can be adapted to resource-constrained contexts. Our syncing spec presumes infrastructure which establishes connections between peers. What if no such infrastructure is available?
In May we published the Willow Sideloading protocol as a possible solution. The sideloading protocol seeks to sidestep the extremely annoying, persistent question of how peers actually find each other, and builds on the tradition of sneakernets so that Willow data can e.g. be securely put onto a USB key and dropped off in the woods.
willow-js has an implementation of the sideloading protocol as of 0.5.0, and an Earthstar 11 beta will gain this functionality shortly.
Willow in Rust
All of the above concludes our NGI Assure project with NLnet, announced back in September of 2022. We have now begun a new project with €50K funding from NGI Core and NLnet.
So far our implementations of Willow have been written in TypeScript. This is so that we could easily integrate it with the existing Earthstar implementation (which is also built in TypeScript), and because it was expedient to do so.
However, TypeScript is not able to take advantage of many potential optimisations Willow enables, e.g. peers being able to communicate their resource constraints and control the amount of memory they use during a sync session. JavaScript is also not famous for producing the most efficient programs.
We aim to address this with Rust implementations of the Willow data model, Meadowcap, and Willow General Purpose Sync protocol. We’re very fortunate in that we already have a few partial implementations of Willow in Rust to learn from.
Joining me in the implementation work is glyph, who’s previously contributed onPeachcloud, solar, Cabal / Cable, and p2panda. Aljoscha Meyer continues to provide his guidance on all things Willow (and now Rust). I am so fortunate to be learning from, and working with, these two.
We’ll also be interviewing a few potential users of this implementation regarding the problems they’re trying to solve, how they hope Willow can help with those, their technical requirements, and publishing our findings.
Finally, we’ll also produce two new websites for our TypeScript and Rust implementations. These websites will have tutorials, guides, and API documentation.
In closing…
A lot of good work was produced this last year. It feels like Willow has resonated with a lot more people than Earthstar was previously able to. Considering all that vibing, and the conversations we’ve been having with different organisations interested in using Willow, I feel cautiously optimisitic that Willow can make an impact.