blog.tibidat.com

erli has been a great project to experiment and learn erlang and webmachine. In a way, its complete success as a learning tool has been the cause of its never ending ‘beta’ status: every time its state closed in on the ‘good enough’ mark, motivation to finish dried up. It would bit-rot for a while, until I found a new aspect of erlang to explore. This is the third time that I return to it in an attempt to finish it.

Now that erli is to play a central role in my newest learning endeavour, there is no way around that 1.0. The specific components that erli still requires to function as part of ‘Hœnir’ are:

• An API endpoint for a binary protocol, to be used by the dart & polymer based frontend
• Removal or isolation of the thumbnail generation component, since it has zero test coverage and is highly experimental
• Test suites for both the HTTP and binary APIs
• Configurable (and optional) integration with kafka
• A new resource for ‘key’ URLs. These should behave similar to paths without shared targets, no visit statistics and no discoverability. My current plan is to point ‘key’ URLs to a note with specific URL parameters or request headers (communication will be via HTTPS only) added. These params would be used to decrypt the note’s contents. I realize that this is not a secure design. A better solution would be to actually use the ‘key’ URLs as keys since this would avoid having any kind of relation between keys and notes. I haven’t decided on which approach to pursue, but it will most likely depend upon frontend usability.
• A release

I’m really looking forward to implementing these. The binary API endpoint should be especially fun.

So I guess the tldr of this post is: if you can’t get yourself to finish a pet project, try to incorporate it into a larger, newer, pet project.

I recently stumbled across an amazing blog post on logging by Jay Kreps of LinkedIn. Beside being extremely well written and informative, it got me really interested in building an application to try out some of the technologies at the heart of ‘scaling’.

The plan is to (finally) complete the erli rewrite and combine its functionality with that of ephemeral. The output will be an application that allows (anonymous) users to create secret notes. I want the stored notes to be server-side encrypted. My current idea is to generate two URLs for each note, one pointing to the note and the other being used as key to encrypt the data. Since there would be no state or any other connection between a note and a key URL (as key URLs could be generated ahead of time or on the client side) this should provide ‘good enough’ secrecy.

Both URLs are subject to expiry, so if you want your note to expire after one view, both the note and key URL are each only available for exactly one view. Upon expiry, the relevant database contents are erased and the URL will return a 410.

This is the core functionality of the application and should be a pretty fun exercise to implement. In addition, I would like to add Kafka and hadoop integration to provide an activity feed and simple statistics and aggregates regarding service usage.

Ideally, the whole application should be docker based to facilitate testing and experimenting with ‘scale’. I code-named the project ‘Hœnir’, after the god of silence in Norse mythology.