PWAs in Haskell - Elements of a web framework

Part of series Bachelor's thesis (post 2 of 2)

Posted on January 14, 2019

After a long delay where I didn’t work on my thesis at all, I started again after long-overdue check-in with my adviser. The finish line looms closer - it’s now the middle of January and the deadline is the middle of May, so I have about four months remaining.

I have a few goals in this post. The first is to find the features and tools developers require in a web framework in general, a surface survey of many language ecosystems. The second is to clarify the goals I have for the framework I’m building, and to prioritize its features in a few tiers: things I want to see in the end product at any cost, things that would be nice to have but not necessary, and also non-goals, features I explicitly won’t be working on.

Languages and their frameworks

Both frontend and backend applications are written in a wide variety of languages. On the backend, you usually have the luxury of choosing the best language for your problem, whereas on the frontend you’re limited to languages that either are JavaScript, or that compile to JavaScript. Then there are the rare ‘full-stack frameworks’ that attempt to wrap both front- and backend into a single application - unsurprisingly written in JavaScript. Fortunately, JavaScript’s stranglehold is soon to be broken with the advent of WebAssembly, a compiler target that any language can compile to (especially those that support LLVM). WebAssembly is however still not widely used, so I will keep to JavaScript or languages that compile it on the frontend.

The following is in no way meant to be a comprehensive overview of every popular framework. While I’ve tried to find a representative sample of the popular ones and a few of the niche ones, I have no experience with web development in C#, Java, Ruby, Groovy or Elixir at all, so there will be mistakes in those sections. I have more experience with the ecosystem of the other languages, but I haven’t used all of the frameworks - that would be a subject for a review-focused bachelor thesis all by itself.

Starting with the big names like Angular, React, or Django, we have the most popular languages for web development - JavaScript and Python. Going on we have the still popular PHP; C# and Java, the enterprise giants; Ruby, with the original web framework; Scala and Groovy on the JVM; Elixir, an Erlang successor, and finally Haskell, the language of choice for my thesis. I’ve also included Elm and Purescript, two Haskell derivatives, to see what their ecosystem looks like.

In each language I’ve tried to choose both big and popular frameworks, as well as some less popular but conceptually interesting frameworks. For each framework I’ll try to find what are their advertised best features, what their users think of them, what they like or dislike, and also have a look at which features or tools am I missing compared to other frameworks or languages.

(Edit: It seems I’ve skipped Go, the newcomer to the web scene. I was going by the list on https://hotframeworks.com/, where Go is only on the second page, and I’ve skipped it when adding Haskell-related or conceptually interesting technologies not on that list. Oh well…)

Features

– A description of the things I’ll be looking for, partially from Wikipedia’s Comparison of web frameworks and Comparison of JavaScript frameworks, partially from my own experience, and partially from the feature lists of the frameworks that I’ll be looking into.

– Of course, in a Turing-complete language anything is possible. I’m looking for either out-of-the-box features, or at least doesn’t obstruct their implementation - if you need to fork the framework to do something that didn’t occur to the author, then that’s a bad sign

Tooling

I’ll start with the things you encounter first when setting up a project, its tools. Developers have wildly differing levels of expectations from their tools. A Python developer might expect just a text editor and an interpreter, whereas a JVM developer might not be satisfied with anything less than a full-featured IDE.

Code generators or scaffolding tools start with creating a package manifest and a src/ directory, going on to generators that set up a few different types of projects based on templates, all the way to tools that can add an entire website module, perhaps even with database migrations.

Build tools range from a set of conventions on how to use your build tool that might get formalized in your Makefile, through a CLI tool that takes care of building, testing and perhaps even deploying your project, to the way of the IDE where anything you can think of is just a few clicks away.

Debugging tools also come in many flavors and for many purposes. On the side of the server, you have all the usual tools for the language, plus a few more - a toolbar with an overview of everything that goes on in a page render or an AJAX call, or the option to remotely connect to a running process and to debug live. Client-side, we have the now irreplaceable DevTools with a built-in debugger and profiler, but some frameworks go even further and provide a framework-specific tools - React’s component tree, or Elm’s time-traveling debugger.

Quality assurance tools have many sub-categories. From static code analysis tools or linters, as they are commonly known; through tests - unit, integration, end-to-end tests, or more exotic ones like marble tests or visual regression tests; to profilers - runtime or allocation measurements, frontend performance measurements, or more involved tools like performance evolution tracking.

Common

Pluggable transports

  • HTTP (via XmlHttpRequest or fetch API) is standard
  • WS widely used for real-time applications
  • Server-Sent Events also not uncommon
  • then we have more exotic transports like WebRTC channels
  • that’s not even touching the topic of transport encoding
    • JSON is standard
    • but what about gRPC, protobufs, …?

Configuration management

  • starting simple - loading from env, from file, from consul, …
  • different configs per execution environment (dev, staging, prod, test)
  • configuring the framework vs configuring the application
  • API design necessary - anti-example = Yesod…

Authentication, authorization

  • authz
    • developers expect many auth methods built-in
    • local user DB
    • OAuth/SSO
    • LDAP
    • other non-OAuth external providers
  • authn
    • RBAC (roles) - for centrally administered systems
    • ACL (users manage their resources) - for larger systems with explicit sharing

Secure by default

  • security is implicit, user has to opt-out to be insecure
  • see RoR’s (or was it Django?) approach
  • frontend - JS or HTML injection, …
  • backend - SQL injection, auth escalation, …

Frontend

Templating

By templating, I mean a way to write the HTML that makes up an application, usually including some render logic and variable interpolation. In some frameworks the whole program is a template (see React), some have templates in separate files and compile them during runtime (see Angular). Templates sometimes contain CSS as well (see the new CSS-in-JS trend)

Forms

There are a few layers of abstraction at which a framework can decide to implement forms - starting at raw DOM manipulation, going on to data containers with validation (but manual rendering), all the way up to form builders, manual or automatic. Under ‘forms’ I count a way to render a form, to validate user input, and collect the result.

Routing

  • History API
  • transitions between pages
  • showing the correct page on page load

Internationalization

  • starting from simple string translations, pluralization, word order
  • going on to RTL, date/time formats, currency, time zones
  • extra: vertical text

Native mobile support

  • it’s common now to provide wrapper applications around web apps via Cordova or similar, usually only a shell app though
  • can be more responsive (as in speed), faster to load, can access device-specific APIs not exposed via Web Platform APIs

Native desktop support

  • it’s quite simple nowadays to wrap a web app into an Electron shell and provide a desktop application as well
  • benefits are the same as with a native/hybrid mobile application

Accessibility

  • the key word now is ARIA = support for screen readers
  • also, semantic elements, text contrast, customization
  • also also, keyboard-accessibility (shortcuts, every clickable element accessible via keyboard = tabindex)
  • accessibility testing (automatic as well, see aXe)

Optimistic updates

  • one of the things I want to focus on
  • broadly, expecting that every network request will be successful and updating the GUI accordingly
  • rolling back app state in case of failure, with notifications

Web Platform

  • location, camera, touch, vibration, …

Pre-render

  • one approach to shortening start-up times
  • serving HTML with all the content already inside, no need for more requests to the backend for the initial page load
  • JS takes over and uses what’s already been loaded
  • can be static or dynamic:
    • static = JAM stack, serving a bunch of files rendered at compile-time
    • dynamic = rendering the HTML at runtime

Backend

Templating

  • static pages or server-side rendering
  • exports - XSLX tables, PDFs, …

Form data definition & validation

  • same as above, an easy way of going from a data structure to a way to validate, process, and save forms

Pub/Sub

  • bidirectional client-server communication
  • server-sent events or unidirectional real-time updates
  • client-to-client communication proxied over server
  • a herd requirement for many complex applications

ORM

  • in Haskell, it won’t be really ‘object’-relational, but whatever
  • in general, an abstraction layer over the DB, see ActiveRecord for the prototypical implementation
  • design consideration - where to place the abstraction, what to expose and what to abstract over, are there exit hatches?

Migrations

  • schema management (versioning, perhaps branching)
  • seed data (vs. fixtures)
  • DB migrations (what about scale? blue/green migrations, …)

Named routes

  • a way of approximating type-safe routes
  • named routes means there is no room for typos in URIs
  • advanced version includes URI parameters, perhaps as a function

Scalability

  • not really a feature by itself
  • I’m looking for a clear story about the limitations of a single server and how to overcome them, how to deal with stateful requests, …

Analysis

– The analysis of the frameworks I’ve listed above, categorized by language; a general description (perhaps history?), “Notable features”, “Notably absent”

Frontend

Angular (JavaScript)

On a first look, Angular looks like a well thought-out frontend framework. Written in Typescript with comprehensive documentation and great tooling, it seems that the authors have learned from their mistakes with AngularJS.

Some notable features:

  • command line tool, ng - it streamlines setting up the entire project - scaffolding, preparing build and testing tools, starting a development server, …
  • runtime environments - from server-side rendering, PWAs with ServiceWorkers, to native and desktop applications, it seems that Angular tries to cover every possible use-case
  • tooling other than the ng tool - browser extensions for runtime debugging, IDEs and others. I haven’t thought of a tool I would miss, but I’m used to minimalism in tooling from the Haskell world…

Some negatives that developers complain about:

  • Angular is intimidating for a new developer, it’s too complex and there’s a lot to learn
  • Too much ‘magic’ - related to the previous point, there’s a lot of abstraction and it’s not easy to understand all the layers
  • Code bloat - the amount of boilerplate and also the size of the resulting bundle
  • Too opinionated - if you don’t like ‘the angular way’, you’re out of luck here
  • scattered documentation - too many articles and tutorials out there for AngularJS that can’t work with the new Angular

React (JavaScript)

React is not a framework in itself. Rather, it’s a library that focuses on a single thing and does it in a unique enough way that there’s sprung up an entire ecosystem around it. In it, there are groups of libraries that build upon React, each focusing on a single feature - UI components, state management, forms etc.

There’s a large jungle of libraries, each one with a different scope and focus. Choosing a library that fits your problem can sometimes take many attempts. Add to it the fact that libraries, frameworks and tools come and go quite quickly - the main cause of the so-called “JavaScript fatigue - and the fact that in JavaScript, it’s fashionable to write extremely small libraries, and you have a recipe for a quite unpleasant development experience.

I’ll try to go through some of the most popular ‘frameworks’ that build on React, though each one is more of a pre-built toolkit of libraries and tools and bits of glue in between, rather than cohesive frameworks. In general, the React world is a lot more mix-and-match than developers used to enterprise frameworks would expect.

Create-react-app, nwb, Razzle, and Neutrino all cover only the build process. Next.js is the first one that I’ve found that goes a step beyond just pre-configuring Webpack and other build tools - it provides other features that are starting become standard - server-side runtime rendering, link prefetching, and build-time prerendering. It’s also the first tool I found that considers that a website can consist of multiple applications, via its ‘zones’ feature.

Gatsby (JavaScript)

One rather unique framework I found - and this is a framework in a strong sense, not like the React tools above - is Gatsby. It’s unique in the sense that while it’s a frontend framework, it’s not supposed to run in a browser. It’s a part of a growing movement centered around the ‘JAM stack’ - “JavaScript, APIs, and Markup”. That doesn’t tell you much, but the main feature is that at build-time, you fetch data from your APIs, and render the application to plain HTML files, so that you don’t need a server other than an S3 bucket or similar.

It’s a framework targeted at a specific subset of website - not single-page applications, but more blogs or e-shops, and a workflow exemplified by Netlify. This means it doesn’t need to concern itself with many features that would be missing in a frontend framework intended for a browser, and those are delegated to a different part of the stack.

Vue.js (JavaScript)

Polymer (JavaScript)

Elm

Halogen (Purescript)

Reflex (Haskell)

Miso (Haskell)

Concur (Haskell)

Transient (Haskell)

Backend

Symfony (PHP)

  • CLI - scaffold, dev-server, migrations, code generation, deps security checker, dump routes, models
  • templates with pipes & logic, auto escaping, whitespace control
  • debug toolbar
  • routes, localized routing, advanced triggers (‘reqs’), named routes
  • both automatic and manual DI
  • config: env-based, local overrides for dev
  • forms: autobuild, template override, many field types, events, auto CSRF
  • annotations-based validation
  • auth: many UserProviders, firewalls (= ACLs), roles
  • ORM (doctrine), autogenerator, DQL
  • + caching, i18n, …

Laravel (PHP)

Zend (PHP)

Phalcon (PHP)

Django (Python)

  • CLI (manage.py) - inspect, ORM shell, migrations, model from DB, test mail, scaffold, user mgmt, graph models
  • ORM (best-in-class)
  • templates with logic, filters
  • auth system - groups, ACL
  • debug toolbar (= server middleware)
  • forms (automatic from models)
  • named URLs
  • DB events (‘signals’)
  • i18n (via gettext?)
  • security by default

Flask (Python)

Express (JavaScript)

Sails.js (JavaScript)

Koa (JavaScript)

Feathers (JavaScript)

  • server-side only?
  • REST API, WS pub/sub, built-in pagination
  • auth systems - session, JWT, OAuth, mixed-auth endpoints
  • built-in clustering
  • pluggable transports (server and client both)
  • hook-able services

Ruby on Rails (Ruby)

Sinatra (Ruby)

ASP.NET (C#)

ASP.NET MVC (C#)

Spring (Java)

Play (Scala, Java)

Phoenix (Elixir)

  • CLI tool - generators, nested projects, dev server, compile, migration mgmt
  • basics - routing, pipeline, templating
  • channels - WS, green threads, pub/sub, clients for Java, Swift, C#; buffering + reconnect, “Mind the Gap”
  • “presence” - CRDTs
  • built-in ORM, simplified SQL API, ‘changesets’

Grails (Groovy)

Yesod (Haskell)

Happstack (Haskell)

Snap (Haskell)

Magicbane (Haskell)

Hybrid/full-stack

Meteor (JavaScript)

  • CLI - meteor (compiler, plugins), devserver, debugger, scaffold, auth, package manager, platform (build target management, db shell), ‘mup’ (deploy over ssh, rollback, via docker)
  • ORM = mongo API, optional schemas, migrations
  • ‘methods’ (RPC with ACL)
  • Spacebars templates
  • shared code (‘isomorphic’), iso packaging (single package for back- and frontend both)
  • mobile-ready via Cordova, hot code push
  • real-time framework - pub/sub (one-directional) via DDP, limited field lists
  • optimistic updates
  • built-in auth (in-app, OAuth), roles
  • iso testing - e2e tests, unified reporting
  • rate limiting

Prioritizing

First the goals I have for this framework as a whole:

  • easy to use and to learn
  • defaults that ‘just work’ for quick prototypes
  • expressive enough for larger products
  • secure by default
  • offline-capable as a default

– Why have I chosen these?

From previous notes - must-haves:

  • pluggable transports
  • real-time updates -> sync + offline
  • auth -> roles, ACL
  • forms
  • tests (unit, e2e)
  • security
  • API -> interoperability with other languages
  • routing
  • CLI: scaffold, runner, (generators, shells)
  • i18n

Features I want:

  • authorization - in-house, SSO
  • forms - HTML -> transport -> DB without boilerplate
  • authorization - roles, ?
  • internationalization - frontend, backend exports
  • routing
    • type-safe
    • doubly parametric (URL and parameters)? Vs singly parametric a la servant’s approach
  • static pre-render (a la jam-stack)
  • real-time communication = pub/sub, synchronization
  • tests
    • doctests
    • frontend and backend unit tests
    • integration tests with mocked networking
    • end-to-end tests with all components
    • (marble tests)
    • (visual regression tests)
  • benchmarks
    • GHC - criterion (timing), weigh (allocations)
    • frontend load-times
    • performance evolution (see Tweag’s hyperion)
  • exposing an API - via swagger, protobuf?
  • migrations

Killer features:

  • auto-generated administration a la Django
  • static pre-rendered frontend
  • and the main one: sync and offline-capable, as automatic as possible
    • CRDTs? event sourcing, plain old bidi sync?

Bonus features:

  • component generation (not only scaffolding, say “add a new page at route /xyz”)
  • more transports between front- and backend than just HTTP and WS
  • desktop app
  • clustering, scaling to at least 10,000 users