RocksDB indexes are done, open-sourcing Yakread next

Last time on Biff: The Newsletter:

I've just barely started to re-implement the entire index feature by using RocksDB directly for the secondary indexes. RocksDB is a mutable KV store, so we can update documents in place without retaining the history—a useful feature for our main source-of-truth database; less critical for these indexes.

That work is now done. Check out the code and the tests. From users' perspective, the main change is that your indexer functions—which take in your XTDB transactions as input, and create custom indexes/derived data/materialized views as output—now return maps instead of XTDB transactions. The map keys and values are serialized to bytes (via Nippy for the values, and via a custom function for keys since we need the serialization to be stable) and stored in RocksDB.

On the usage side, there's a nifty biff/open-db-with-index function that's similar to xt/open-db, except that the db value it returns also lets you query a snapshot of your RocksDB index:

(with-open [db (biff/open-db-with-index ctx)]
  ;; Query xt
  (xt/q db ...)
  ;; Read from your index
  (biff/index-get db :my-index-id :some-key))

And Biff ensures that the two snapshots are consistent; i.e. the Biff index snapshot was built from all the transactions that the XT snapshot was and none more.

Now that indexes are ready to go, I'm back to working on Yakread. I'm doing a complete rewrite using both the new indexes feature and Pathom to make the code base more maintainable. When the rewrite is done I'll make another release of Biff with all the index + Pathom features, then I'll release Yakread as open-source and use it as a "flagship" example Biff app. I probably won't emphasize indexes or Pathom in e.g. the tutorial, since for a lot of Biff users they would probably just complicate things unnecessarily. But I will likely write some documentation about using Biff for "serious"/commercial projects that are expected to grow large.

So far I'm organizing Yakread's source code into three main folders/namespaces:

• com.yakread.model.*: this contains indexes and Pathom resolvers. The resolvers handle most of the database/index access for your application.

• com.yakread.app.*: this part also contains a bunch of Pathom resolvers, but instead of returning "regular" data, they return HTML (hiccup) snippets. Each page you navigate to in the app is generated by wiring up a bunch of different resolvers. This is where I also put the non-GET HTTP routes, scheduled tasks, and other application-feature type things.

• com.yakread.lib.*: this is where most of the business logic/helper functions go. Each child namespace (e.g. com.yakread.lib.user) is treated as its own independent library, with a public API and a private implementation. model and app code can both require namespaces from lib, but not the other way around.

Once this is all done there a few other things I'd like to explore in the realm of making-Biff-suitable-for-large-apps, like "how do you handle complex forms with dozens of fields and conditional logic etc" and "how do you gradually add more client-side interactivity to your app without rewriting the whole thing in React?"