Roadmap to new architecture

2024 update: this thread is no longer valid. Please see last post for current status.

I am not happy with Misago's current architecture. It sure took me far, but it's showing its limitations and is slowing me down.

I would like to separate client and API parts completely and have Python part only implement a GraphQL API for internet forum, and client part be TypeScript React.js isomorphic application that renders both on server and then if user has JavaScript enabled, also in client to improve interactivity. Basically what Remix and Next.js deliver. I would also want to have plugin system enabling developers to either extend or plain out replace the parts of site that they don't like.

This new architecture was initially realized as ambitious v4 rewrite. But this v4 plan was too ambitious and too big to deliver without hurting existing version, so I've decided to scrap it a while ago.

But I've been thinking a lot about evolving existing codebase into this new architecture in smaller steps. Yesterday I've settled on 10 steps roadmap on getting there:

github.com/users/rafalp/projects/3/views/1

This roadmap is still years of work, but now in smaller and saner chunks. So when there's no higher priority work, I will work on items from this roadmap, slowly pushing the cart towards the goal.

Those items have different complexity and completeness criteria. For example, for filters based plugin system I just have to copy filters abstractions from v4 branch and change them a little to make them work with existing codebase, then setup developer documentation for plugin hooks. At this point this task will be considered completed and closed, but it will only be a beginning of the process for adding extension hooks using those abstractions to existing Misago features. But implementation of GraphQL API will be a months-long process spread over multiple releases as new features are added to the GraphQL API piecemeal and individual parts of the UI are changed to use those instead of old restful API.

I've updated the roadmap as some of my plans have changed (eg. I want custom RPC API instead of GraphQL) and end of year nears, where usually I have a lot of time for Misago.

My immediate goal is to get Misago 0.37 released. After that I will focus on implementing foundations of first four items of roadmap:

1. Filters based plugin system
2. AST-based parser from v4
3. New permission system
4. RPC API

It's impossible to complete those features quickly (besides the new parser). Moving whole codebase to new plugin system/permission system or implementing new RPC API that has feature parity with current REST API is potentially years of work. But completing foundations for those features is not a large effort and should be possible to achieve sometime before Q3 of 2024.

The downside of this is that Misago will have a transition period where there are two plugin systems, two permission systems and two API systems. But this will only be noticeable to plugin developers and site admins.

Just to be clear, this doesn't mean that after Misago 0.37 is released I will hide myself in a cave and only come out with Misago 0.38 half a year later. I will still try to release as often as I am able to.

Also to elaborate, my current plan is to go the way of Discourse and Flarum. So when user enters a forum, they get barebones HTML to keep devices without JavaScript support happy, and then JS will load and render a proper site. This will be slower than current approach, but lighter on resources than original plan of having Next.js or Remix.js proxy running together with Django in docker-compose setup.

It should still be fast enough to don't annoy your users, but with added advantage of no longer having to maintain two sets of same HTML templates and translations in both Django and React.js.

This plan has separate thread here.

Also, my reason for going with custom RPC API instead of GraphQL is having backend control whats returned to client, so Python plugins can easily customize API payloads that JavaScript gets.

Please note that plan to implement API and have React.js UI with no Django backend has been scrapped a while ago.

Current plan is to instead remove React.js from the UI, convert Misago back to the Multi Page Application (MPA) and use HMTX for improved interactivity, based on usage of partial templates swapping out parts of HTML:

Removing React.js from the codebase and adapting HTMX for UI interactivity

GitHub roadmap mentioned above has been updated to reflect this new direction.