Twelve years of distributed SQLite
In September 2014 I published a short post announcing rqlite. The idea was simple: take SQLite, put a Raft log in front of it, and replicate every write to a cluster so that the loss of a node did not mean the loss of the data. The source went on GitHub that same weekend. As far as I can tell, that was the first open-source system to combine Raft and SQLite. Nobody else was doing it, and for several years nobody else tried.
Today “distributed SQLite” is a category with a dozen projects in it, a few of them venture-funded, two of them acquired. Since the people now writing about the category tend to start the story around 2021, I thought I should write down the version that starts in 2014. I was there for all of it.
The need was concrete. A clustered system I worked on at jut.io needed a small amount of relational storage, and the question of which node should hold it had no good answer. Replicating SQLite with a consensus protocol solved it. We went another way at work, but I kept building rqlite because the problem was general and the solution was small. The first version used goraft, the same library InfluxDB and etcd used at the time (I subsequently joined InfluxDB as a core developer around the same time). In early 2016 I replaced it with Hashicorp’s Raft implementation which rqlite still uses.
The design decision that mattered most was made then and never changed. rqlite is a CP system. Writes must go through the leader, a majority must acknowledge, and a partitioned minority refuses writes. Everything else, the HTTP API, the read-consistency levels, the snapshotting, followed from that.
Two things deserve mention for honesty’s sake. Expensify had been replicating SQLite inside its own infrastructure for years with a Paxos-style protocol; they released it as Bedrock in 2016. And ActorDB, an Erlang system that gave each “actor” its own SQLite database replicated by Raft, appeared around the same time as rqlite. Neither was widely adopted outside its origin, but both show the idea was in the air. What rqlite did was make it a standalone, general-purpose, open-source thing anyone could run.
Canonical built dqlite to get LXD and later MicroK8s an embedded replicated store without running etcd. Mark Shuttleworth actually called me once, and we talked, hoping that rqlite could be made to work, but it never quite fit. dqlite began in Go and was rewritten in C with its own Raft library. It is the project most similar to rqlite in intent, and the main difference is of one of packaging: dqlite is a library you embed, rqlite is a server you run. Canonical’s need was a database inside their products: mine was a database beside anyone’s product.
Ben Johnson (whom I worked with at InfluxDB) released Litestream in early 2021. Litestream does not use consensus at all. It watches the SQLite WAL and streams it to S3, giving you disaster recovery for a single-node SQLite database at almost no cost. Ben’s argument was that most applications do not need a cluster; they use one machine and need a way to recover if it dies. He was right about a great many applications, and the “SQLite is enough” movement of the next few years, with blog posts and conference talks, owes a lot to that argument.
Litestream and rqlite answer different questions. Litestream: what if the machine dies? rqlite: what if the machine dies and I cannot wait to restore? The first question is more common, the second is the one people have when they come to rqlite.
Fly.io hired Ben in 2022 and shipped LiteFS, which replicates SQLite at filesystem level with leases from Consul. It was the first attempt to make replicated SQLite a platform feature rather than a thing you deploy. And in a final twist, Litestream inspired rqlite’s v8.0 approach to handling the SQLite WAL.
Two projects in 2022 took approaches that neither rqlite nor Litestream had. cr-sqlite added CRDTs to SQLite as an extension, so that multiple writers could diverge and merge without a leader. That is the AP corner of the design space, and it suits local-first applications where rqlite’s CP guarantees would be a hindrance.
libSQL forked SQLite itself. ChiselStrike, later Turso, wanted to change things inside the engine that the SQLite project, which does not accept outside contributions, and would not change. On top of the fork they built a hosted service with replicas at the edge. Turso raised venture money on that, and in 2026 Supabase acquired them.
Around the same time, SQLite Cloud appeared, offering hosted SQLite replicated with Raft. It is the project closest to rqlite’s original design, wrapped in a cloud service.
Cloudflare announced D1 in 2022, SQLite as a serverless database inside their edge network, and later put SQLite storage inside Durable Objects. This is replicated SQLite as a managed product from a company that does not need to tell you how it works. Turso’s service is the same shape. The category had gone from “a thing you run” to “a thing you buy.”
Three things, I think.
The core was right. SQLite as a storage engine, with replication added on the outside, has been reinvented by at least eight teams with different needs and different protocols. Nobody has found a reason to stop doing it.
The design space has three corners, and they appeared in order. Consensus first (rqlite, dqlite): one leader, strong consistency, a cluster. Streaming second (Litestream): one writer, cheap recovery, no cluster. Merge third (cr-sqlite, and parts of the local-first movement): many writers, eventual consistency. Each is correct for a different question, and most of the arguments are arguments between their users about which question is being asked.
And the money went to the platforms. rqlite, dqlite, and Litestream are open source and free. The projects that raised the capital or got acquired are the ones that turned replicated SQLite into a service with a bill. That is the usual fate of infrastructure ideas, and I do not mind it. But it is worth recording that the idea was open source for eight years before anyone sold it.
rqlite is still here. It does the same thing it did in 2014, a good deal better, and it is still the thing you reach for when you want a fault-tolerant relational database you can run yourself, understand fully, and not pay anyone for. Twelve years in, I am content with that position in the story.
ai aws bleve book review C++ chatgpt cloud conference database design distributed systems docker ec2 ekanite elasticsearch go google hashicorp influxdb java kafka kubernetes laptop leadership linux logs meetups node.js open-source operations percolate podcast programming python quality raft rqlite search speaking sqlite storm syslog time-series video whitepaper
Thank you for this blog! It was very interesting.
Thank you for all your great work Philip.
Rqlite is absolutely wonderful, and as far as how to be a good maintainer goes, you are absolutely a role model to me.
Thank you Phillip! I have been familiar with all of the projects you’ve mentioned. At my previous company I’ve worked in we also built a distributed layer on top of SQLite. It also was in Go using hashicorp library. One thing that I’ve been working on there is distributed transactions. At first it was implemented as every stmt in a tx is a different command on the ledger. I changed it so there’s at most 2 commands . One for distributed begin and then the node that initiated it does the entire tx locally. Then it accumulates all of the tx stmts and at the end it rollbacks and sends a commit command with the entire batch
Your email address will not be published. Required fields are marked *
Save my name, email, and website in this browser for the next time I comment.
Notify me of follow-up comments by email.
Notify me of new posts by email.