Dat was developed with the idea of making sharing large files or folders securely an easy task. Without the need to rely on a central “cloud” service (like dropbox) or through semi-cumbersome means like rsync or scp, Dat can make it easy to store, share, and track large volumes of data. This is accomplished through a peer-to-peer (p2p) network (much like BitTorrent). This brings along several of the benefits of a p2p architecture. There is no central host where availability (or security/privacy leanings) would be a concern, you can share across a local network (LAN), and bandwidth scales with the growth of demand as downloads will be distributed across the peers.
Scientists and researchers were the initial target audience for Dat, allowing them to easily archive and share data. I first caught wind of Dat in Februrary of 2017 when it came to light that the data stored at open.whitehouse.gov was being inexplicably deleted. Data from open.whitehouse.gov was scraped and stored in a Dat archive so that the information would not be lost. You can see this and a few other endeavors cataloged on the Dat website.
You can read more about Dat, and forgo further clumsy explanation from me, in their documentation.
So Dat is a protocol to serve files. Viewing a website is nothing more than files served to a browser that then renders them. So what if we pipe files recieved via the Dat protocol to a browser for rendering? Well you got a peer-to-peer web goin’!
Clearly it’s a bit more complex than that, but that is the gist of what Beaker Browser does. Beaker is a project developed by Tara Vancil and Paul Frazee, and is a self described peer-to-peer web browser. Its still under very active development, but after some semi-regular use I’ve found it both useable and exciting.
Recently Tara and Paul did a talk for the local group Bleeding Edge Web here in Austin. The talk is enlightening; it gives a window into their vision of the project. If you have some free time, its worth a watch.
Paul also has a brief video introduction to Beaker:
The peer-to-peer web is, arguably, still in its infancy. However, seeing and using a working model is exciting, with potentially far-reaching ramifications. Since many of the current “gatekeepers” to the web are taken out of the equation in a peer-to-peer model, I think there is a potential to reclaim a lot of the magic and excitement present in the early internet. In this spirit, I’ve made this site available to Beaker via Dat at dat://electro.pizza or
dat://485d1f5c6cc1970ee29318f3be66c6c9bd0fb5894e757fdc926785ae6a725346 if you like hashes.
While setting this simple site up a few things stood out to me and I’d like to note them here. For the sake of time (and the length of this post), I’m not going to touch on all of the features with both Dat and Beaker. Rather, I will try to limit the focus to what I encountered while setting up and publishing this simple site. Many of these features are quite interesting and exciting in their own right, and if you are interested I encourage you to check them out on the relavent project pages. Most notable of these to me are:
- Built in versioning on the history of changes in a DAT site.
- Forking of Dat sites (along with new potential workflows when working with a team).
- New Web API’s in Beaker to build peer-to-peer applications.
But for now, some Beaker basics:
Initial creation of a site/hash is extremely simple; the dialogues in the browser walk you through things. Beaker handles setting up the
.dat folder and keys behind the scenes so you don’t have to futz with it. If you don’t want to keep the site code in the default location (
/home/<user>/Sites in Linux), you need to take the extra step of changing the directory to point at the public files of your site.
Here is an example of the process:
Live reloading should be nothing new to most web developers, however its nice to have this ready to go “out of the box”. It behaves like you would expect, watching site files for changes and refreshing. This makes pairing with something like a static site generator in a watch mode quite nice.
This was glossed over in the site generation example above, but the publishing mechanism may be the most exciting (in the context of workflow) with Beaker. As you would expect, when you make changes to site files locally they are reflected locally in Beaker. Traditionally, when you are ready to publish these changes to the web, you would perform a push of the code through various mechanisms, ie. pushing to a CI system or manually scp/rsync/sftp files to a centralized server somewhere. However, with Beaker all you need do is click the big green publish button, and the file changes are then propigated through to all peers. No server concerns at all. Refreshing!
In this example Beaker is on the left and Firefox on the right. Changes published from Beaker are reflected via a remote peer serving over HTTPS to Firefox.
Now wait a minute. Firefox? HTTPS? A *gasp* SERVER!? Weren’t we talking about a peer-to-peer web? Well we are, but we don’t have a widely adopted peer-to-peer web yet! So, we need a stop-gap to solve a couple of issues:
First, electro.pizza isn’t wildly popular. Often my local Beaker Browser installation will be the only peer, thus the only source for anyone to view the site. Therefore, for this site to have constant “uptime”, I would need my Beaker installation to always be running with a network connection. Not ideal.
Second, the vast majority of people surfing the web are using more traditional browsers (Firefox, Chrome, Edge, etc.), and I would like those folks to be able to view my site through the (currently) standard HTTPS web protocol.
Dathttpd solves this problem by acting as both a peer for a Dat site, as well as a small web server. This allows you to keep a Dat site up without having to leave Beaker running. Further, you get the added bonus of being able to point DNS at that server which allows for shortnames for your site. Configuration is simple and detailed on the projects github page.
I was caught up in a couple of small snags while setting up dathttpd. Nothing crazy, but its probably worth a note.
Once the dathttpd config is set up and things seem to be working as expected, it is suggested to use add-to-systemd to daemonize the process. This works quite well, however I’ve found add-to-systemd works best if nodejs is installed on a system level (rather than something like NVM). Use your distro’s package manager to install nodejs greater than version 6.0. On distros like Ubuntu, you’ll likely have to add a PPA for the newer nodejs version.
After having dathttpd running for a few days then returning to Beaker to make site updates, I found that dathttpd and Beaker had trouble reconnecting with each other to propigate updates. I watched traffic for a bit, and it seemed to be a firewall issue with the server. Turns out the machine that is running dathttpd needs to have port 3282/tcp open. After making this adjustment everything worked as expected. I have a bit of a nagging concern that any machine running Beaker also needs port 3282 opened. So far this does not seem to be the case, but I’m not completely convinced.