WIP: feat(lib): Incremental Build Tool API

clo commented

2026-03-23 22:56:02 -07:00

Owner

resolves #27
resolves #88
blocked by #90 because i don't want to configure a JSX plugin

been playing with node:vm's experimental modules. it is neat because it gives us a hook into the module system that no other runtime has:

able to reflect what files are pointed to by each file
can adjust how files are processed before loading
can unload/reload modules, with old ones being garbage collected

this commit rebuilds hot.ts and incremental.ts as a new system that only depends on these node builtins. here is a big todo list for progress

module loading incremental.graph.import("src/main.ts");
- node.js ESM resolution
- plugin system to load extensions like .tsx, .marko, .svelte
- commonjs (stub that is not actually incremental, just don't use cjs please)
tracing the files used in resolution and initialization
- module resolution that traces what files it read
- linking modules to each other
- tracing a list of css files from an entrypoint
allowing arbitrary jobs that cache when re-run
- buffer-focused data serialization
- arbitrary dependencies on other Work, other files, and other directory contents
recomputing work when it should be
creating a filesystem watcher to detect changes
- reloading parts of the graph
- optional bait: support import.meta.hot since it's so easy
serializing internal state to disk
- save and revive work entries
- save and revive graph state
- save and revive transpiled files
- save and revive the v8 cache

by implementing @clo/lib/incremental, clover sitegen can:

stop using and patching node.js internals. everything is now fair-game APIs
drop the esbuild powered hot.ts, using real ESM whenever possible
provide a base for more features (issue 4 and 5), which could possibly be solved as their own top-level repos.

but most importantly, it makes the incremental api available for anyone who wants to use it, intead of it being a very hyper specific component to clover sitegen.

this pr also drops msgpackr. since lib cannot have dependencies, data serialization is unified into the core format instead of being a stupid layer on top; the incremental state is a rich sqlite database. this should help with large incremental graphs that contain many megabytes of files.

resolves #27 resolves #88 blocked by #90 because i don't want to configure a JSX plugin been playing with `node:vm`'s experimental modules. it is neat because it gives us a hook into the module system that no other runtime has: - able to reflect what files are pointed to by each file - can adjust how files are processed before loading - can unload/reload modules, with old ones being garbage collected this commit rebuilds `hot.ts` and `incremental.ts` as a new system that only depends on these node builtins. here is a big todo list for progress - [x] module loading `incremental.graph.import("src/main.ts");` - [x] node.js ESM resolution - [x] plugin system to load extensions like `.tsx`, `.marko`, `.svelte` - [x] commonjs (stub that is not actually incremental, just don't use cjs please) - [ ] tracing the files used in resolution and initialization - [x] module resolution that traces what files it read - [ ] linking modules to each other - [ ] tracing a list of css files from an entrypoint - [ ] allowing arbitrary jobs that cache when re-run - [x] buffer-focused data serialization - [ ] arbitrary dependencies on other `Work`, other files, and other directory contents - [ ] recomputing work when it should be - [ ] creating a filesystem watcher to detect changes - [ ] reloading parts of the graph - [ ] optional bait: support `import.meta.hot` since it's so easy - [ ] serializing internal state to disk - [ ] save and revive work entries - [ ] save and revive graph state - [ ] save and revive transpiled files - [ ] save and revive the v8 cache by implementing `@clo/lib/incremental`, clover sitegen can: - stop using and patching node.js internals. everything is now fair-game APIs - drop the `esbuild` powered `hot.ts`, using real ESM whenever possible - provide a base for more features (issue 4 and 5), which could possibly be solved as their own top-level repos. but most importantly, it makes the incremental api available for anyone who wants to use it, intead of it being a very hyper specific component to clover sitegen. this pr also drops msgpackr. since lib cannot have dependencies, data serialization is unified into the core format instead of being a stupid layer on top; the incremental state is a rich sqlite database. this should help with large incremental graphs that contain many megabytes of files.

clo added 1 commit

2026-03-23 22:56:02 -07:00

WIP: feat(lib): Incremental Module Loader API 2477fbe717

been playing with `node:vm`'s experimental modules. it is neat because
it gives us a hook into the module system that no other runtime has:

- able to reflect what files are pointed to by each file
- can adjust how files are processed before loading
- can unload/reload modules, with old ones being garbage collected

this commit rebuilds `hot.ts` and `incremental.ts` as a new system that
only depends on these node builtins, which (TODO) provides:

- simple module loading `new Incremental().import("src/main.ts");`
- plugin system to load extensions like `.tsx`, `.marko`, `.svelte`
- tracing the files used in resolution and initialization
  - allowing arbitrary dependencies to be added
- allowing arbitrary jobs that cache when re-run
- creating a filesystem watcher to detect changes
  - performing trivial module reloading (maybe full HMR)
- serializing the graph state to disk, along with v8 cache data

by implementing this, clover sitegen can:

- stop patching node.js internals
- drop the server-side esbuild code (frontend still needs it)
- use real es modules
- provide a base for more features (issue 4 and 5), which could possibly
  be solved as their own top-level projects.

this is also a great time to solve issue 27 (incremntal tracking)

clo added the

lib

label

2026-03-23 22:56:14 -07:00

clo commented

2026-03-29 13:25:31 -07:00

Author

Owner

neat idea:

// run.js would use something like this
incremental.reviveAndExecute(import.meta.resolve("./framework/generate.ts"));

incremental.graph.import("./") // untraced?

incremental.graph.transaction() // similar to incr.compile? this is where ref comes in?

// inside of generate.ts
const result = await inc.cachedWork([key], async (io) => {
  // do a task. rules are to not pollute globals
  io.import("...");
});

low key think this should be named something else top level

neat idea: ``` // run.js would use something like this incremental.reviveAndExecute(import.meta.resolve("./framework/generate.ts")); incremental.graph.import("./") // untraced? incremental.graph.transaction() // similar to incr.compile? this is where ref comes in? // inside of generate.ts const result = await inc.cachedWork([key], async (io) => { // do a task. rules are to not pollute globals io.import("..."); }); ``` low key think this should be named something else top level

clo force-pushed incremental from 2477fbe717 to 1e02d3c568

2026-03-31 00:29:12 -07:00

Compare

clo changed title from ~~WIP: feat(lib): Incremental Module Loader API~~ to WIP: feat(lib): Incremental Build Tool API

2026-03-31 00:30:22 -07:00

clo force-pushed incremental from 1e02d3c568 to 26c98740bb

2026-04-07 23:20:08 -07:00

Compare

clo force-pushed incremental from 26c98740bb to 46ce321c04

2026-05-14 21:44:16 -07:00

Compare

clo force-pushed incremental from 46ce321c04 to 6a4e985a96

2026-05-28 04:16:01 -07:00

Compare

This pull request has changes conflicting with the target branch.

framework/marko.ts
lib/progress.ts
lib/readme.md
run.js

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin incremental:incremental

git switch incremental

Merge

Merge the changes and update on Forgejo.

git switch master

git merge --no-ff incremental

git switch incremental

git rebase master

git switch master

git merge --ff-only incremental

git switch incremental

git rebase master

git switch master

git merge --no-ff incremental

git switch master

git merge --squash incremental

git switch master

git merge --ff-only incremental

git switch master

git merge incremental

git push origin master

Rows
Columns

WIP: feat(lib): Incremental Build Tool API #96

Checkout

Merge