New message parser

For the abandoned Misago v4 rewrite I've implemented a two-step parser:

1. Parse a document into AST and Metadata.
2. Convert AST and Metadata into a final structure (HTML, JSON, Search document or cleaned up Markdown again).

This approach is a massive upgrade over current parser used in Misago 0.x. Instead of having to hack around internal data structures and final HTML, you just modified the AST.

This approach is also very easy to extend and customize through plugins.

GitHub Issue: #1480
Pull Request: #1720

On v4 branch I've used Mistune parser because it emits AST and supports markdown out of the box. But in latest version that library makes some decisions that I would have to implement a layer around to keep from Misago (like HTML support), so I've decided to write my own AST parser instead.

There's quite a lot of syntax that needs to be implemented in this parser, but the parser logic itself seems simple enough to implement. No ETA when this will be finished tho.

I think I have core functionality of the parser done. Now I need to implement a support for different formats. There's quite a few of those to do, but I have a list for tracking progress.

It took me a while, but all supported syntax (short of dedicated markup for posts attachment) has been implemented. Todo left is gathering items metadata and rendering markup to plaintext and HTML.

Metadata is done. Now onto rendering to plaintext and HTML.

I've did a quick experiment and plugged new parser to the "preview text" feature of Misago's editor.

Markup:

# New parser

This is Misago's new parser. It emits the `AST` ("**A**bstract **S**yntax **T**ree") representation of the markup, which is then rendered to one of multiple formats, like HTML or plain text for use in <meta type="description"> HTML tag or as a search document for PostgreSQL.

Abstract Syntax Tree

[
  {
    "type": "heading",
    "level": 1,
    "children": [
      {
        "type": "text",
        "text": "New parser"
      }
    ]
  },
  {
    "type": "paragraph",
    "children": [
      {
        "type": "text",
        "text": "This is Misago's new parser. It emits the "
      },
      {
        "type": "code-inline",
        "code": "AST"
      },
      {
        "type": "text",
        "text": " (\""
      },
      {
        "type": "strong",
        "children": [
          {
            "type": "text",
            "text": "A"
          }
        ]
      },
      {
        "type": "text",
        "text": "bstract "
      },
      {
        "type": "strong",
        "children": [
          {
            "type": "text",
            "text": "S"
          }
        ]
      },
      {
        "type": "text",
        "text": "yntax "
      },
      {
        "type": "strong",
        "children": [
          {
            "type": "text",
            "text": "T"
          }
        ]
      },
      {
        "type": "text",
        "text": "ree\") representation of the markup, which is then rendered to one of multiple formats, like HTML or plain text for use in <meta type=\"description\"> HTML tag or as a search document for PostgreSQL."
      }
    ]
  }
]

HTML

<h1>New parser</h1>\n<p>This is Misago&#x27;s new parser. It emits the <code>AST</code> (&quot;<strong>A</strong>bstract <strong>S</strong>yntax <strong>T</strong>ree&quot;) representation of the markup, which is then rendered to one of multiple formats, like HTML or plain text for use in &lt;meta type=&quot;description&quot;&gt; HTML tag or as a search document for PostgreSQL.</p>

Screenshot

New message parser has been merged 🎉

I will update the existing codebase to use it instead of old parser as I progress with other tasks.