Added conditional format conversions to Pic Store today. This lets you configure it to do things like generate PNGs only if the input was also a PNG, so that you don't do silly things like output a PNG of a photo. Next up will be the Pic Store integration mentioned in the previous post.
Last night I watched Join Order Optimization with (almost) no Statistics by Tom Ebergen. In this video, Tom describes enhancements he made to the join planner in DuckDB, and how it applies both to native tables where you have some extra metadata about the contents of each column, and on external files such as Parquet or CSV, where you don't know much about the actual contents. Good watching if you have any interest in database systems.
Thinking about how to better represent the progress and plans for my various projects. I moved a lot of my internal TODO lists over to Github issues, which works all right, and I can even create a view that shows issues from multiple repositories at once. This is especially nice now that I'm building up a small eco system of projects, some of which are using others as dependencies.
I also tried Linear briefly and while I love the interface, there doesn't seem to be any way to "build in public" with it. So I'm torn about that. Maybe I'll make a small program to export issues from the API and write out some Markdown with the project status, which I can then publish here.
Implemented job recovery in Effectum today, so that if the process restarts unexpectedly the jobs can be retried. I also did a bit of cleanup to prepare for the eventual addition of a server mode. The aim is to allow switching from an embedded task queue to one hosted on a server, and not have to significantly change the API, so that you can keep complexity down at first but then scale up with minimal fuss when necessary.
Pic Store is now running on Axum 0.6, which was just released. For the feature set I was using, it was a pretty seamless change. I like that there's a lot more clarity now between extractors that consume the request body (e.g. Json parsing) and those that don't touch it at all.
Didn't have time to mention it here last night but the Let's Encrypt DNS challenge tool is all working. It starts the challenge, adds the required DNS entry to Vercel, and then uploads the resulting certificate to DigitalOcean and enables it.
It's a bit over-engineered — I mean, a shell script run every 60 days would probably have been sufficient — but this will make it easier to add support for more hosts and services types in the future for other projects.
I spent an hour trying to figure out why Let's Encrypt was rejecting my DNS record, and in the end turned out to be just a typo in my DNS record (_acme_challenge instead of _acme-challenge). But overall the process wasn't too bad.
I'll get around to a full writeup in the near future, I hope, but in the meantime here's the Github repo.
In getting S3 upload for Pic Store to work, I found that DigitalOcean Spaces defaults all files to private, with no way to change the default from the web UI. But you can use s3cmd to do it by setting the S3 bucket policy. First, create a policy JSON file:
Pic Store image uploading is all working now, but I ran into issues with actually setting up a CDN to back it. I initially wanted to use Backblaze B2 as the object store, which is set up to work with Cloudflare CDN. But Cloudflare want you to switch your whole domain over to them for it to work, and I don't want to switch my whole DNS setup over since I'm using Vercel.
DigitalOcean also prefers that you switch your nameservers to them, but they do offer an alternative option, if you supply your own SSL certificate.
So today's free time will be spent building a small utility that talks to LetsEncrypt, Vercel DNS, and DigitalOcean CDN to handle generating the certificate and installing it. There are a few Rust crates that handle the ACME protocol already, so hopefully it should be quick project. I'll write up a small article once it's done.
Logseq's latest release no longer uses its pages-metadata.edn file, which my note exporter was using to track the created and updated dates of each exported page. Since this file is no longer updated, I updated the program to track them in a SQLite database.
For each page, I calculate a hash, and if the page's hash has changed, it updates the database with the file's last modified time. Nice and easy. For older pages, it can still import the initial data from pages-metadata.edn too. Integrating SQLite into the exporter program was a pretty smooth experience; I'm hoping to use it more often.
Of course, now that I have all the pages in a database, I can add other data, and that brings up more fun questions about what else I can do with it.
I set up Let's Encrypt at work today. Our nginx configuration is somewhat complex and we also run it inside a Docker container, so the process was a bit unusual. I wrote about it here.
Copilot works well for writing repetitive CRUD endpoints in Pic Store, but better is to not have to write them at all. I've been experimenting with using macros to abstract out the details of CRUD operations. This includes the relevant permissions checks, filtering on team/project, and so on.
Seems to be working well so far, and it removes a lot of the boilerplate. I may take this a step further and generate the entire endpoint function with a macro, but I haven't decided yet.
Got the Pic Store bootstrap command working today. The Postgres ability to defer constraints checking until the end of a transaction is very handy here, when you're loading a bunch of JSON files and don't want to have to sort them topologically according to the foreign arrangement of the tables.
I've also been getting a steady stream of Github notification emails for the past hour, all from the same PR at work that someone else is working on. I had an idea a while back to make an Email Digest Service and this is pushing me closer to doing just that. At least Pic Store is getting close enough to MVP that I can push through and start using it soon.
AWS Route53 lets you create "alias" records that work like a CNAME, but just redirect to another AWS resource (including other Route53 entries). This incurs less cost than a normal CNAME, since resolving the alias doesn't count as an extra lookup.
Today I wrote image reading and writing tests for Pic Store. I came across a number of issues reading AVIF files, which eventually were fixed by switching directly to the libavif crate which seems to do a better job parsing certain AVIF files that aren't quite up to spec.
I also made a fork of the `imageinfo` crate to help with detecting these files, but I need to go back and see if my fix is really correct before I submit it upstream.
Started up development on Pic Store again. I switched out the task queue to Effectum and I got it to the point where the server starts, creates the database, and a simple authenticated request succeeds. A lot of the core code for the project is already implemented, so there will be some testing to see if stuff actually works and then I can start working on a simple web interface, and tools to make it convenient to use.
I'm thinking of a Vite plugin that will automatically upload the images if needed and generate a full <picture> tag. Also some CLI/GUI utilities that can make it convenient to upload and reference images from other contexts, such as when writing a document.
If it's possible, I would love to be able to drag an image straight into Logseq, and have that handle the upload and URL pasting. Will have to see if the plugin system allows intercepting events like that.
Since I already have a working JS engine in the project, the state machines will continue to support bits of JS to assist in evaluating conditions and such. Tasks with heavier scripting will be runnable as actions that spawn external processes, and can then return values that can trigger further actions in the state machines.
From there, I can create other types of tasks that are basically wrappers around certain types of state machines, and hopefully come up with something that's both intuitive to use and actually useful.
The initial release of Effectum is out! Pretty happy with how it turned out, and my first application will be to use it for background image processing in Pic Store. Look for more about that in the coming weeks.
One interesting issue I immediately encountered after releasing v0.1.0 was that the trait bounds on the job runner were overly strict. I had required Sync on the future returned by the job runner function, when actually nothing in the library required that. But that meant that any job runner function could not hold a non-Sync value across an await point. Fortunately this was just a matter of removing the trait bound and adding a test whose job runner function holds a Cell across an await.
When it comes time to actually document code for public consumption, Rust's missing_docs lint is great. I don't have to search out every place that needs documentation; I can just let the compiler yell at me about it.
In related news, Effectum first release is basically done. All the basic tests are there and I got a big performance improvement by batching database operations together, so it can process about 50,000 do-nothing jobs per second on my laptop. Going to clean some things up and then the first actual release should come early next week.
Some future features will include auto-recurring jobs, the ability to cancel or modify jobs that haven't started running yet, and support for running the task queue as its own server.
Effectum tests are underway and progress has been good so far. Found a couple issues but nothing major. It'll be nice to have this done and have a background job framework with no external dependencies, ready to drop into any future project.
Ran into a issue today with a Svelte component that dispatches an event when it first loads. It turns out that if you dispatch immediately then the event doesn't get handled, but you can work around this by dispatching in onMount instead. I created this example Svelte REPL to demonstrate the behavior.
Michael Lynch's post today on migrating from Cypress to Playwright struck a chord with me, as I had recently been setting up Cypress at work and encountered many of the same pain points that he mentioned. Having tried Playwright now, I'm a fan.
Github Copilot continues to be quite nice. It even appears to be aware of multiple buffers at once. When I was converting my Cypress tests, it actually suggested a nearly proper Playwright equivalent. Very impressive, and it saved me at least five minutes per test.
I signed up for the Github Copilot trial today. Pretty impressed overall. The longer suggestions in Rust frequently need some tweaking to be completely correct (usually around specific type coercion), but it still is saving a lot of time.
Back from vacation, and while I didn't get much done on Effectum, I did come closer to figuring out the structure of the job registry. By wrapping each job runner function with a closure, I can abstract away the need to represent all the different possible return types of each task, and instead just handle cleaning up the task and processing its result inside the closure.
This thread on the Rust forum describes the proper way to save reference a closure that also returns a future. The main thing to remember here is that you have to manually pin the future.
The remaining part here is figuring out how to deal with shared state (things like database pools and so on). The easiest way would probably be to set up a single context type for a worker, which all task types share, and use it as a generic type parameter.
Ideally, each task runner could have its own structure for the shared state, instead of them all needing to use the same one. Something like the Extension structure that various Rust web frameworks use could work. For the first version I'll probably punt on this though, just to get something up and running.
Navigate, don't search, A GPS for the mind, and Hyperlink maximalism by Linus are three good posts on how software can be more of a partner to our thinking process. Linus is one of the more original and interesting writers in the overcrowded "tools for thought" space and his writing is worth checking out even if you're not interested in the topic as a whole.
Progress on Effectum is coming along as I find a couple hours here and there to work on it. Job heartbeat and checkpoint functionality is written. Next up will be some thought about ergonomic ways to start workers that can run multiple types of tasks. I may do something inspired by the job registry in sqlxmq.
Haki Benita wrote a short article about writing SQL in a future-proof way by adding exhaustive checking to CASE statements. This way if an unexpected new value is added to a column, a query will throw an error instead of just ignoring it.
We have a fair amount of legacy code at work which we're slowly converting to modernity, and one aspect of that is making Typescript pass in strict mode. I've found it useful to enable strict mode in the main tsconfig.json, while using a second tsconfig file to actually build the project that extends from the main file but turns off strict mode. This way we can still run checks and get the full set of diagnostics when developing, while allowing the build to still pass.
Implemented job insertion on Effectum today. I tried to be clever at first and prepare all the queries in advance when the Queue is created, but ran into issues where prepared rusqlite statements are not Send. That's ok though. The rusqlite prepared statement cache is close enough.
LibSQL is a new SQLite fork. Whereas SQLite famously does not accept outside contributions, LibSQL is approaching it from a more community-based open source viewpoint. They're aiming toward some more "modern" features such as support of io_uring and perhaps WASM user-defined functions. We'll have to see how this turns out. SQLite's strict development process has also lead to a famously stable product, and I expect that it will be difficult to keep that same level of rigor. Still, if this picks up steam it could be quite promising.
Implemented my first Rust future by hand today for Effectum . Wasn’t too bad actually, for simple cases. I ended up going a different route but it was nice to get the experience. The main job waiter loop is coming together, and next up will be adding new jobs and pulling them off the queue to assign to workers.
I switched back from Warp to iTerm2+tmux once again. There are still some rough edges with Neovim, but overall Warp has improved a lot in the past six months and I'm looking forward to the next time I try it again.
Intuitive is a new Rust library geared toward declarative terminal user interfaces. Still in development, it seems, but it looks promising.
Starting yet another side project because why not :) Effectum will be a SQLite-based task queue embeddable via a Rust library, and eventually it will also be a standalone project, which can be used over HTTP or gRPC. Just putting together the requirements now. This is mostly an excuse to play with SQLite a bit, but once this is in decent condition I'll probably start using it for both Ergo and Pic Store.
Started up Warp Terminal once again to check out the latest updated. The various displays in Neovim all look correct now, which is great. And the notifications on finishing of long-running commands is welcome, since I'm frequently running long tasks in other tabs at work.
They've also added a quick session switcher (Cmd+Shift+P). Still would be nice to be able to switch between entire sets of tabs, like tmux lets you do, but I'm ready to give it another serious try.