clo/sitegen
1
0
Fork 0
Code Issues 16 Pull requests Projects Releases Packages Wiki Activity Actions

framework: remove @mdx-js/mdx and replace with custom implementation #1

New issue
Open
opened 2025-06-08 16:13:51 -07:00 by clo · 1 comment
clo commented 2025-06-08 16:13:51 -07:00
Owner
Copy link

task list

  • implement a base CommonMark parser that is close enough for all practical purposes, with a plugin system
    • inline parsing
      • link
      • image
      • code
      • emphasis
      • html
      • autolinks
      • linebreak
    • block parsing
      • heading
      • list
      • indented codeblock
      • fenced codeblock
      • blockquote
      • html
      • paragraph
    • link references
    • insecure characters handling
    • get the test suite to run so i have a notion of how far off i am. i do not expect to pass all tests
  • get q+a to use this instead of simple-markdown (delete 1250 lines of code, in return i get a better markdown engine)
    • question paragraph
    • qref
    • aref
    • extra auto linking rules
    • deadname
    • clover flavored @html blocks
  • implement GFM extensions
    • strikethru
    • tables (with rowspan and colspan extensions)
    • admonition
    • bare autolink
  • syntax highlighted code blocks
  • markodown/mdx adapter

the two biggest dependencies of the framework are

  • @mdx-js/mdx
  • marko (separate issue, unlikely to take action)
MDX has 111 total dependencies. which is insane because it is not doing anything revolutionary 
@mdx-js/mdx@3.1.0
+-- @types/estree-jsx@1.0.5
+-- @types/estree@1.0.8
+-- @types/hast@3.0.4
| `-- @types/unist@3.0.3
+-- @types/mdx@2.0.13
+-- collapse-white-space@2.1.0
+-- devlop@1.1.0
| `-- dequal@2.0.3
+-- estree-util-is-identifier-name@3.0.0
+-- estree-util-scope@1.0.0
+-- estree-walker@3.0.3
+-- hast-util-to-jsx-runtime@2.3.6
| +-- comma-separated-tokens@2.0.3
| +-- hast-util-whitespace@3.0.0
| +-- mdast-util-mdx-expression@2.0.1
| | `-- mdast-util-to-markdown@2.1.2
| |   +-- longest-streak@3.1.0
| |   +-- mdast-util-phrasing@4.1.0
| |   +-- micromark-util-classify-character@2.0.1
| +-- mdast-util-mdx-jsx@3.2.0
| | +-- ccount@2.0.1
| | +-- parse-entities@4.0.2
| | | +-- @types/unist@2.0.11
| | | +-- character-entities-legacy@3.0.0
| | | +-- character-reference-invalid@2.0.1
| | | +-- is-alphanumerical@2.0.1
| | | | +-- is-alphabetical@2.0.1
| | | +-- is-decimal@2.0.1
| | | `-- is-hexadecimal@2.0.1
| | +-- stringify-entities@4.0.4
| | | +-- character-entities-html4@2.1.0
| +-- mdast-util-mdxjs-esm@2.0.1
| +-- property-information@7.1.0
| +-- space-separated-tokens@2.0.2
| +-- style-to-js@1.1.16
| | `-- style-to-object@1.0.8
| |   `-- inline-style-parser@0.2.4
| +-- unist-util-position@5.0.0
| `-- vfile-message@4.0.2
+-- markdown-extensions@2.0.0
+-- recma-build-jsx@1.0.0
| +-- estree-util-build-jsx@3.0.1
+-- recma-jsx@1.0.0
| +-- acorn-jsx@5.3.2
| | `-- acorn@8.15.0
| +-- estree-util-to-js@2.0.0
| | +-- astring@1.9.0
| +-- recma-parse@1.0.0
| | +-- esast-util-from-js@2.0.1
| | | +-- esast-util-from-estree@2.0.0
| | | | +-- estree-util-visit@2.0.0
+-- recma-stringify@1.0.0
+-- rehype-recma@1.0.0
| `-- hast-util-to-estree@3.1.3
|   +-- estree-util-attach-comments@3.0.0
|   `-- zwitch@2.0.4
+-- remark-mdx@3.1.0
| +-- mdast-util-mdx@3.0.0
| `-- micromark-extension-mdxjs@3.0.0
|   +-- micromark-extension-mdx-expression@3.0.1
|   | +-- micromark-factory-mdx-expression@2.0.3
|   | +-- micromark-factory-space@2.0.1
|   | +-- micromark-util-character@2.1.1
|   | +-- micromark-util-events-to-acorn@2.0.3
|   +-- micromark-extension-mdx-jsx@3.0.2
|   +-- micromark-extension-mdx-md@2.0.0
|   +-- micromark-extension-mdxjs-esm@3.0.0
|   | +-- micromark-core-commonmark@2.0.3
|   | | +-- micromark-factory-destination@2.0.1
|   | | +-- micromark-factory-label@2.0.1
|   | | +-- micromark-factory-title@2.0.1
|   | | +-- micromark-factory-whitespace@2.0.1
|   | | +-- micromark-util-html-tag-name@2.0.1
|   +-- micromark-util-combine-extensions@2.0.1
|   | +-- micromark-util-chunked@2.0.1
+-- remark-parse@11.0.0
| +-- @types/mdast@4.0.4
| +-- mdast-util-from-markdown@2.0.2
| | +-- decode-named-character-reference@1.1.0
| | | `-- character-entities@2.0.2
| | +-- mdast-util-to-string@4.0.0
| | +-- micromark-util-decode-numeric-character-reference@2.0.2
| | +-- micromark-util-decode-string@2.0.1
| | +-- micromark-util-normalize-identifier@2.0.1
| | +-- micromark-util-symbol@2.0.1
| | +-- micromark@4.0.2
| | | +-- @types/debug@4.1.12
| | | | `-- @types/ms@2.1.0
| | | +-- debug@4.4.1
| | | | `-- ms@2.1.3
| | | +-- micromark-util-encode@2.0.1
| | | +-- micromark-util-resolve-all@2.0.1
| | | +-- micromark-util-subtokenize@2.1.0
| +-- micromark-util-types@2.0.2
+-- remark-rehype@11.1.2
| +-- mdast-util-to-hast@13.2.0
| | +-- @ungap/structured-clone@1.3.0
| | +-- micromark-util-sanitize-uri@2.0.1
| | +-- trim-lines@3.0.1
+-- source-map@0.7.4
+-- unified@11.0.5
| +-- bail@2.0.2
| +-- extend@3.0.2
| +-- is-plain-obj@4.1.0
| +-- trough@2.2.0
+-- unist-util-position-from-estree@2.0.0
+-- unist-util-stringify-position@4.0.0
+-- unist-util-visit@5.0.0
| +-- unist-util-is@6.0.0
| `-- unist-util-visit-parents@6.0.1
`-- vfile@6.0.3
for completion, marko has 71 dependencies. not off the hook but this dependency provides a lot of value currently. 
marko@6.0.21
+-- @marko/compiler@5.39.21
| +-- @babel/code-frame@7.27.1
| | +-- @babel/helper-validator-identifier@7.27.1
| | +-- js-tokens@4.0.0
| | `-- picocolors@1.1.1
| +-- @babel/core@7.27.4
| | +-- @ampproject/remapping@2.3.0
| | +-- @babel/helper-compilation-targets@7.27.2
| | | +-- @babel/compat-data@7.27.5
| | | +-- @babel/helper-validator-option@7.27.1
| | | +-- browserslist@4.25.0
| | | | +-- caniuse-lite@1.0.30001721
| | | | +-- electron-to-chromium@1.5.165
| | | | +-- node-releases@2.0.19
| | | | `-- update-browserslist-db@1.1.3
| | | |   +-- escalade@3.2.0
| | | +-- lru-cache@5.1.1
| | | | `-- yallist@3.1.1
| | +-- @babel/helper-module-transforms@7.27.3
| | | +-- @babel/helper-module-imports@7.27.1
| | +-- @babel/helpers@7.27.6
| | +-- @babel/template@7.27.2
| | +-- convert-source-map@2.0.0
| | +-- debug@4.4.1
| | | `-- ms@2.1.3
| | +-- gensync@1.0.0-beta.2
| | +-- json5@2.2.3
| | `-- semver@6.3.1
| +-- @babel/generator@7.27.5
| | +-- @jridgewell/gen-mapping@0.3.8
| | | +-- @jridgewell/set-array@1.2.1
| | +-- @jridgewell/trace-mapping@0.3.25
| | | +-- @jridgewell/resolve-uri@3.1.2
| +-- @babel/parser@7.27.5
| +-- @babel/plugin-syntax-typescript@7.27.1
| | `-- @babel/helper-plugin-utils@7.27.1
| +-- @babel/plugin-transform-modules-commonjs@7.27.1
| +-- @babel/plugin-transform-typescript@7.27.1
| | +-- @babel/helper-annotate-as-pure@7.27.3
| | +-- @babel/helper-create-class-features-plugin@7.27.1
| | | +-- @babel/helper-member-expression-to-functions@7.27.1
| | | +-- @babel/helper-optimise-call-expression@7.27.1
| | | +-- @babel/helper-replace-supers@7.27.1
| | +-- @babel/helper-skip-transparent-expression-wrappers@7.27.1
| +-- @babel/runtime@7.27.6
| +-- @babel/traverse@7.27.4
| | `-- globals@11.12.0
| +-- @babel/types@7.27.6
| | +-- @babel/helper-string-parser@7.27.1
| +-- @luxass/strip-json-comments@1.4.0
| +-- complain@1.6.1
| | `-- error-stack-parser@2.1.4
| |   `-- stackframe@1.3.4
| +-- he@1.2.0
| +-- htmljs-parser@5.5.4
| +-- jsesc@3.1.0
| +-- kleur@4.1.5
| +-- lasso-package-root@1.0.1
| | `-- lasso-caching-fs@1.0.2
| |   `-- raptor-async@1.1.3
| +-- raptor-regexp@1.0.1
| +-- raptor-util@3.2.0
| +-- relative-import-path@1.0.0
| +-- resolve-from@5.0.0
| +-- self-closing-tags@1.0.1
| `-- source-map-support@0.5.21
|   +-- buffer-from@1.1.2
|   `-- source-map@0.6.1
+-- csstype@3.1.3
`-- magic-string@0.30.17
  `-- @jridgewell/sourcemap-codec@1.5.0

there is already an extensible markdown parser in ./src/q+a/simple-markdown.ts. this file probably needs a lot of cleanup, because it is old. it can be moved into the framework. the JSX segments can be emitted verbatim since the file can go through esbuild. this also lets us try a "markodown" version where marko syntax is used instead of JSX. there is a good chance that would be much more preferable than JSX since it is universally less verbose to write.

**task list** - [ ] implement a base CommonMark parser that is close enough for all practical purposes, with a plugin system - [ ] inline parsing - [x] link - [ ] image - [x] code - [ ] emphasis - [ ] html - [ ] autolinks - [x] linebreak - [ ] block parsing - [ ] heading - [ ] list - [ ] indented codeblock - [ ] fenced codeblock - [ ] blockquote - [ ] html - [ ] paragraph - [ ] link references - [ ] insecure characters handling - [ ] get the test suite to run so i have a notion of how far off i am. i do not expect to pass all tests - [ ] get q+a to use this instead of simple-markdown (delete 1250 lines of code, in return i get a better markdown engine) - [ ] question paragraph - [ ] qref - [ ] aref - [ ] extra auto linking rules - [ ] deadname - [ ] clover flavored `@html` blocks - [ ] implement GFM extensions - [ ] strikethru - [ ] tables (with rowspan and colspan extensions) - [ ] admonition - [ ] bare autolink - [ ] syntax highlighted code blocks - [ ] markodown/mdx adapter --- the two biggest dependencies of the framework are - `@mdx-js/mdx` - `marko` (separate issue, unlikely to take action) <details> <summary>MDX has 111 total dependencies. which is insane because it is not doing anything revolutionary</summary> ``` @mdx-js/mdx@3.1.0 +-- @types/estree-jsx@1.0.5 +-- @types/estree@1.0.8 +-- @types/hast@3.0.4 | `-- @types/unist@3.0.3 +-- @types/mdx@2.0.13 +-- collapse-white-space@2.1.0 +-- devlop@1.1.0 | `-- dequal@2.0.3 +-- estree-util-is-identifier-name@3.0.0 +-- estree-util-scope@1.0.0 +-- estree-walker@3.0.3 +-- hast-util-to-jsx-runtime@2.3.6 | +-- comma-separated-tokens@2.0.3 | +-- hast-util-whitespace@3.0.0 | +-- mdast-util-mdx-expression@2.0.1 | | `-- mdast-util-to-markdown@2.1.2 | | +-- longest-streak@3.1.0 | | +-- mdast-util-phrasing@4.1.0 | | +-- micromark-util-classify-character@2.0.1 | +-- mdast-util-mdx-jsx@3.2.0 | | +-- ccount@2.0.1 | | +-- parse-entities@4.0.2 | | | +-- @types/unist@2.0.11 | | | +-- character-entities-legacy@3.0.0 | | | +-- character-reference-invalid@2.0.1 | | | +-- is-alphanumerical@2.0.1 | | | | +-- is-alphabetical@2.0.1 | | | +-- is-decimal@2.0.1 | | | `-- is-hexadecimal@2.0.1 | | +-- stringify-entities@4.0.4 | | | +-- character-entities-html4@2.1.0 | +-- mdast-util-mdxjs-esm@2.0.1 | +-- property-information@7.1.0 | +-- space-separated-tokens@2.0.2 | +-- style-to-js@1.1.16 | | `-- style-to-object@1.0.8 | | `-- inline-style-parser@0.2.4 | +-- unist-util-position@5.0.0 | `-- vfile-message@4.0.2 +-- markdown-extensions@2.0.0 +-- recma-build-jsx@1.0.0 | +-- estree-util-build-jsx@3.0.1 +-- recma-jsx@1.0.0 | +-- acorn-jsx@5.3.2 | | `-- acorn@8.15.0 | +-- estree-util-to-js@2.0.0 | | +-- astring@1.9.0 | +-- recma-parse@1.0.0 | | +-- esast-util-from-js@2.0.1 | | | +-- esast-util-from-estree@2.0.0 | | | | +-- estree-util-visit@2.0.0 +-- recma-stringify@1.0.0 +-- rehype-recma@1.0.0 | `-- hast-util-to-estree@3.1.3 | +-- estree-util-attach-comments@3.0.0 | `-- zwitch@2.0.4 +-- remark-mdx@3.1.0 | +-- mdast-util-mdx@3.0.0 | `-- micromark-extension-mdxjs@3.0.0 | +-- micromark-extension-mdx-expression@3.0.1 | | +-- micromark-factory-mdx-expression@2.0.3 | | +-- micromark-factory-space@2.0.1 | | +-- micromark-util-character@2.1.1 | | +-- micromark-util-events-to-acorn@2.0.3 | +-- micromark-extension-mdx-jsx@3.0.2 | +-- micromark-extension-mdx-md@2.0.0 | +-- micromark-extension-mdxjs-esm@3.0.0 | | +-- micromark-core-commonmark@2.0.3 | | | +-- micromark-factory-destination@2.0.1 | | | +-- micromark-factory-label@2.0.1 | | | +-- micromark-factory-title@2.0.1 | | | +-- micromark-factory-whitespace@2.0.1 | | | +-- micromark-util-html-tag-name@2.0.1 | +-- micromark-util-combine-extensions@2.0.1 | | +-- micromark-util-chunked@2.0.1 +-- remark-parse@11.0.0 | +-- @types/mdast@4.0.4 | +-- mdast-util-from-markdown@2.0.2 | | +-- decode-named-character-reference@1.1.0 | | | `-- character-entities@2.0.2 | | +-- mdast-util-to-string@4.0.0 | | +-- micromark-util-decode-numeric-character-reference@2.0.2 | | +-- micromark-util-decode-string@2.0.1 | | +-- micromark-util-normalize-identifier@2.0.1 | | +-- micromark-util-symbol@2.0.1 | | +-- micromark@4.0.2 | | | +-- @types/debug@4.1.12 | | | | `-- @types/ms@2.1.0 | | | +-- debug@4.4.1 | | | | `-- ms@2.1.3 | | | +-- micromark-util-encode@2.0.1 | | | +-- micromark-util-resolve-all@2.0.1 | | | +-- micromark-util-subtokenize@2.1.0 | +-- micromark-util-types@2.0.2 +-- remark-rehype@11.1.2 | +-- mdast-util-to-hast@13.2.0 | | +-- @ungap/structured-clone@1.3.0 | | +-- micromark-util-sanitize-uri@2.0.1 | | +-- trim-lines@3.0.1 +-- source-map@0.7.4 +-- unified@11.0.5 | +-- bail@2.0.2 | +-- extend@3.0.2 | +-- is-plain-obj@4.1.0 | +-- trough@2.2.0 +-- unist-util-position-from-estree@2.0.0 +-- unist-util-stringify-position@4.0.0 +-- unist-util-visit@5.0.0 | +-- unist-util-is@6.0.0 | `-- unist-util-visit-parents@6.0.1 `-- vfile@6.0.3 ``` </details> <details> <summary> for completion, marko has 71 dependencies. not off the hook but this dependency provides a lot of value currently. </summary> ``` marko@6.0.21 +-- @marko/compiler@5.39.21 | +-- @babel/code-frame@7.27.1 | | +-- @babel/helper-validator-identifier@7.27.1 | | +-- js-tokens@4.0.0 | | `-- picocolors@1.1.1 | +-- @babel/core@7.27.4 | | +-- @ampproject/remapping@2.3.0 | | +-- @babel/helper-compilation-targets@7.27.2 | | | +-- @babel/compat-data@7.27.5 | | | +-- @babel/helper-validator-option@7.27.1 | | | +-- browserslist@4.25.0 | | | | +-- caniuse-lite@1.0.30001721 | | | | +-- electron-to-chromium@1.5.165 | | | | +-- node-releases@2.0.19 | | | | `-- update-browserslist-db@1.1.3 | | | | +-- escalade@3.2.0 | | | +-- lru-cache@5.1.1 | | | | `-- yallist@3.1.1 | | +-- @babel/helper-module-transforms@7.27.3 | | | +-- @babel/helper-module-imports@7.27.1 | | +-- @babel/helpers@7.27.6 | | +-- @babel/template@7.27.2 | | +-- convert-source-map@2.0.0 | | +-- debug@4.4.1 | | | `-- ms@2.1.3 | | +-- gensync@1.0.0-beta.2 | | +-- json5@2.2.3 | | `-- semver@6.3.1 | +-- @babel/generator@7.27.5 | | +-- @jridgewell/gen-mapping@0.3.8 | | | +-- @jridgewell/set-array@1.2.1 | | +-- @jridgewell/trace-mapping@0.3.25 | | | +-- @jridgewell/resolve-uri@3.1.2 | +-- @babel/parser@7.27.5 | +-- @babel/plugin-syntax-typescript@7.27.1 | | `-- @babel/helper-plugin-utils@7.27.1 | +-- @babel/plugin-transform-modules-commonjs@7.27.1 | +-- @babel/plugin-transform-typescript@7.27.1 | | +-- @babel/helper-annotate-as-pure@7.27.3 | | +-- @babel/helper-create-class-features-plugin@7.27.1 | | | +-- @babel/helper-member-expression-to-functions@7.27.1 | | | +-- @babel/helper-optimise-call-expression@7.27.1 | | | +-- @babel/helper-replace-supers@7.27.1 | | +-- @babel/helper-skip-transparent-expression-wrappers@7.27.1 | +-- @babel/runtime@7.27.6 | +-- @babel/traverse@7.27.4 | | `-- globals@11.12.0 | +-- @babel/types@7.27.6 | | +-- @babel/helper-string-parser@7.27.1 | +-- @luxass/strip-json-comments@1.4.0 | +-- complain@1.6.1 | | `-- error-stack-parser@2.1.4 | | `-- stackframe@1.3.4 | +-- he@1.2.0 | +-- htmljs-parser@5.5.4 | +-- jsesc@3.1.0 | +-- kleur@4.1.5 | +-- lasso-package-root@1.0.1 | | `-- lasso-caching-fs@1.0.2 | | `-- raptor-async@1.1.3 | +-- raptor-regexp@1.0.1 | +-- raptor-util@3.2.0 | +-- relative-import-path@1.0.0 | +-- resolve-from@5.0.0 | +-- self-closing-tags@1.0.1 | `-- source-map-support@0.5.21 | +-- buffer-from@1.1.2 | `-- source-map@0.6.1 +-- csstype@3.1.3 `-- magic-string@0.30.17 `-- @jridgewell/sourcemap-codec@1.5.0 ``` </details> there is already an extensible markdown parser in `./src/q+a/simple-markdown.ts`. this file probably needs a lot of cleanup, because it is old. it can be moved into the framework. the JSX segments can be emitted verbatim since the file can go through esbuild. this also lets us try a "markodown" version where marko syntax is used instead of JSX. there is a good chance that would be much more preferable than JSX since it is universally less verbose to write.
clo commented 2025-07-08 01:11:14 -07:00
Author
Owner
Copy link

new code is here. it'll probably move to engine since it is another way to use the SSR engine instead of a markdown parser

https://git.paperclover.net/clo/sitegen/src/branch/master/framework/lib/markdown.tsx

new code is here. it'll probably move to engine since it is another way to use the SSR engine instead of a markdown parser https://git.paperclover.net/clo/sitegen/src/branch/master/framework/lib/markdown.tsx
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
typefix
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: clo/sitegen#1
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?