RemoteStorage.js

I am currently working on a piece of offline-first cross-platform survey software. The project uses the Ionic 3 framework with Cordova and Electron to enable completely cross-platform availability.

Creating an offline-first application can be challenging; there are a lot of concepts that have to be perfectly implemented for the app to function. You need:

  • Local (in-app) storage – IndexedDB is usually the best choice for cross-platform web apps
  • Server-side storage – something fast and powerful that can manage changes, conflicts and security
  • Synchronisation

PouchDB/CouchDB

PouchDB with CouchDB is a very common solution to offline-first. PouchDB is a JavaScript library used for the client-side. It's incredibly fast, versatile (it supports plug-in modules), works very conistently, and has a lot of  community support.

CouchDB is used on the server side. It's build using Node.js, and is a general-purpose NoSQL database with built-in server capabilities, which is awesome for setting up APIs quickly.

I have one major issue with this "stack": The server side. In order to have this work, you will need a server capable of running Node.js with a lot of storage and bandwidth. Most cheap servers will provide shared hosting on Linux for Apache, PHP, MySQL development (great for WordPress-like sites), but will restrict system use specifically to prevent Node.js from running on them.

So you end up getting a VPS with a lot of storage, bandwidth, and now need to fork out a considerable amount for anything decent (something without fair use policies, because, believe me, they will not like PouchDB running 24/7).

Enter RemoteStorage.js

I found out about remotestorage.js (rs.js) a few months ago, but wasn't too interested in what the library had to offer. Its design was all over the place, and seemed to be mostly a way of showing off their remotestorage protocol as opposed to something more fully-fledged. But then 1.0 released last week, and I thought I'd give it a go.

What is it?

RemoteStorage.js is a JavaScript library aimed at bringing offline-first per-user storage that is compatible with a variety of storage solutions straight to the browser.

Now there is a rather large caveat to rs.js: it definitely isn't perfect for all applications. Because rs is per-user all data available to a user is only that user's. So there is no easy way of doing any form of social networking.

But that's not really who it is aimed at. The point of the app is to provide data security (in the sense that it won't be obliterated the second you leave a website) to applications that are specifically meant for one user. A great example of where it would apply is an online word processor: when the user leaves the website, they expect their documents they just wrote to still be there – and they will be.

What's so freaking awesome about this is you can create applications that actually store user data without forcing them to make yet another account.

But that's something any offline-first app can do. What makes rs.js special?

Google Drive & Dropbox integration

Awesome. Simply awesome. Remotestorage.js lets users connect their Google Drive or DropBox account to your application to back up their local data and sync it across devices.

At the moment Google Drive and Dropbox are the only Cloud storage providers, but it is possible to manually create one yourself, and they are working on adding more as we speak.

With this feature, I can create an app that requires literally no server. This makes the app incredibly cost-effective for me, requires less maintenance from me, and can transition the savings and provide some nifty benefits to the end-user:

  • The user can choose their own storage provider. They don't have to trust me with their data.
  • DropBox and Google Drive are far more secure, scalable, robust and cheaper than I could possibly offer the same amount of storage for.
  • The data is always their own. I can provide my application as a service, the user knowing that – even if my product tanks – their data will always be available to them.

How it works (basically)

The Remotestorage protocol combines HTTP, CORS, WebFinger and OAuth 2 together to form a protocol designed for remote storage access and offline-first synchronisation that isn't an absolute pain in the ass.

Firstly, the app has to have local storage. Rs.js is pretty great at figuring out what the browser's capabilities are. It will default to IndexedDB, but can fall back to localStorage.

Next, it needs to perform OAuth. This is done very cleverly, differently on each platform. In the browser it simply redirects to the OAuth page with a redirect URI of the app, where it receives the authentication token. On Cordova, the story is different: It uses the InAppBrowser plug-in to open the OAuth page, wait for redirect to an authenticated redirect URI (not necessarily the app's URI), and intercepts the redirect URI, extracting the token from it. It then closes the InAppBrowser window.

RS.js provides "modules", which are essentially categories of information. Within each module, types can be defined. Each type works similarly to how a table works in traditional SQL. A schema is defined using JSON schema (love it) for the type, and is used to validate data stored in the module.

A simple, powerful, promise-based (yes!) storage interface is used to perform requests on the storage. There are functions like getAll(), getObject(path, maxAge) and setObject(path, data, maxAge), which is just bliss to use.

What's great is when you define a module, you can set the exports to whatever you want, enabling a really easy-to-use way of managing your data. Example:

await this.remoteStorage.documents.save({
'title': 'My document',
'body': 'Hello, world! Lorem ipsum...'
});

Will save a document to the documents module with the given information. Within the save function defined in module's exports, we can perform very useful operations. For my application I implemented a UUID generator for each document, the ability to update rather than save if the ID is specified, and create timestamps for when the object is created or updated.

Current issues/complaints

There are some problems with the project in its current state. The primary issue I have encountered is their support for Cordova. Google Drive recently decided to no-longer let webviews authenticate OAuth, so rs.js are busy working on an alternative way of performing OAuth.

I don't particularly like the way that modules are defined. Modules themselves are awesome, but they have to be defined using a JSON object that has a "builder" function. This function gives a reference to the privateClient variable, which needs to be used to do pretty much anything. Getting a reference to this variable outside the function is annoying, making it a bit more difficult to construct modules in an object-oriented manner (doing things like extending modules and overriding their exported methods).

Conclusion

RemoteStorage.js is an amazing option for per-user storage that can be cheap, effective and possibly bring in more users knowing that their data is their own (or at least as "their own" as cloud storage can be).

Copyright © Matthew Evans 2017