Journals for 2022-10

Effectum tests are underway and progress has been good so far. Found a couple issues but nothing major. It'll be nice to have this done and have a background job framework with no external dependencies, ready to drop into any future project.

Learning

Ran into a issue today with a Svelte component that dispatches an event when it first loads. It turns out that if you dispatch immediately then the event doesn't get handled, but you can work around this by dispatching in onMount instead. I created this example Svelte REPL to demonstrate the behavior.

The MVP of Effectum is almost ready. Still a few more features to add, and lots of tests to write before it's truly ready!

Links

• The creator of the swc Typescript transpiler has open-sourced his Typescript type checker project. Looking forward to seeing what comes out of this, since Typescript performance is a big pain point in modern development on large projects.

Michael Lynch's post today on migrating from Cypress to Playwright struck a chord with me, as I had recently been setting up Cypress at work and encountered many of the same pain points that he mentioned. Having tried Playwright now, I'm a fan.

Github Copilot continues to be quite nice. It even appears to be aware of multiple buffers at once. When I was converting my Cypress tests, it actually suggested a nearly proper Playwright equivalent. Very impressive, and it saved me at least five minutes per test.

I signed up for the Github Copilot trial today. Pretty impressed overall. The longer suggestions in Rust frequently need some tweaking to be completely correct (usually around specific type coercion), but it still is saving a lot of time.

Learning

Wrote a note on Atomic Memory Ordering Models since it was finally time to understand Acquire and Release.

Links

• Falcon is a library for visualizing millions of rows of data within the browser.

Back from vacation, and while I didn't get much done on Effectum, I did come closer to figuring out the structure of the job registry. By wrapping each job runner function with a closure, I can abstract away the need to represent all the different possible return types of each task, and instead just handle cleaning up the task and processing its result inside the closure.

This thread on the Rust forum describes the proper way to save reference a closure that also returns a future. The main thing to remember here is that you have to manually pin the future.

The remaining part here is figuring out how to deal with shared state (things like database pools and so on). The easiest way would probably be to set up a single context type for a worker, which all task types share, and use it as a generic type parameter.

Ideally, each task runner could have its own structure for the shared state, instead of them all needing to use the same one. Something like the Extension structure that various Rust web frameworks use could work. For the first version I'll probably punt on this though, just to get something up and running.

Links

• Jason Cohen, founder of WPEngine, is blogging again and has written a bunch of good articles on planning out work. JIT selection from independent streams: An alternative to the “big backlog” of work and The practical application of "Rocks, Pebbles, Sand" are two good selections of many.

Links

• Navigate, don't search, A GPS for the mind, and Hyperlink maximalism by Linus are three good posts on how software can be more of a partner to our thinking process. Linus is one of the more original and interesting writers in the overcrowded "tools for thought" space and his writing is worth checking out even if you're not interested in the topic as a whole.

Progress on Effectum is coming along as I find a couple hours here and there to work on it. Job heartbeat and checkpoint functionality is written. Next up will be some thought about ergonomic ways to start workers that can run multiple types of tasks. I may do something inspired by the job registry in sqlxmq.

Links

• Haki Benita wrote a short article about writing SQL in a future-proof way by adding exhaustive checking to CASE statements. This way if an unexpected new value is added to a column, a query will throw an error instead of just ignoring it.

We have a fair amount of legacy code at work which we're slowly converting to modernity, and one aspect of that is making Typescript pass in strict mode. I've found it useful to enable strict mode in the main tsconfig.json, while using a second tsconfig file to actually build the project that extends from the main file but turns off strict mode. This way we can still run checks and get the full set of diagnostics when developing, while allowing the build to still pass.

Links

• A neat trick to move an existing process into a tmux session. Looks like the utility needed to accomplish it doesn't work on Mac, but this seems like something most used on remote servers anyway.

Implemented job insertion on Effectum today. I tried to be clever at first and prepare all the queries in advance when the Queue is created, but ran into issues where prepared rusqlite statements are not Send. That's ok though. The rusqlite prepared statement cache is close enough.

Links

• LibSQL is a new SQLite fork. Whereas SQLite famously does not accept outside contributions, LibSQL is approaching it from a more community-based open source viewpoint. They're aiming toward some more "modern" features such as support of io_uring and perhaps WASM user-defined functions. We'll have to see how this turns out. SQLite's strict development process has also lead to a famously stable product, and I expect that it will be difficult to keep that same level of rigor. Still, if this picks up steam it could be quite promising.

Implemented my first Rust future by hand today for Effectum . Wasn’t too bad actually, for simple cases. I ended up going a different route but it was nice to get the experience. The main job waiter loop is coming together, and next up will be adding new jobs and pulling them off the queue to assign to workers.

I switched back from Warp to iTerm2+tmux once again. There are still some rough edges with Neovim, but overall Warp has improved a lot in the past six months and I'm looking forward to the next time I try it again.

Links

• Intuitive is a new Rust library geared toward declarative terminal user interfaces. Still in development, it seems, but it looks promising.

Starting yet another side project because why not :) Effectum will be a SQLite-based task queue embeddable via a Rust library, and eventually it will also be a standalone project, which can be used over HTTP or gRPC. Just putting together the requirements now. This is mostly an excuse to play with SQLite a bit, but once this is in decent condition I'll probably start using it for both Ergo and Pic Store.