Building

• Filigree
  • Got the object storage abstraction and configuration finished up over the weekend.
  • Now I"m starting on a "File" type of model, which will allow the template system to generate code that handles uploads, and optionally stores things like file hash, image type, etc. in the database.

Building

• Filigree
  • Starting on object storage support. Once done, this will add a File type to the database model, set up the client, and also make it easy to do things like stream files to and from the client. And of course the template system will initialize everything for you.

Building

• Smelter
  • The worker framework now handles SIGINT for you, and gives your code a channel that will change when the task should be cancelled.
  • Published v0.1.0 of the Smelter crates, since it's about to go into production at work!
• Effectum
  • More work on outbox pattern. The code is pretty much done, just needs tests.

Building

• Effectum
  • Started on support for transactional outbox pattern, since I'm starting to add background worker support to Filigree
• Smelter
  • The AWS Rust SDK doesn't handle the ThrottlingError returned from the ECS API, and so they don't get retried. I added a custom `ClassifyRetry` layer to make this work.
  • Improved job cancellation on error, specifically around ensuring that there is time to send cancel requests to all the workers. There's still room for improvement around the ergonomics here I think, but it's an improvement.

Building

• Smelter
  • Worker containers collect statistics such as RAM used and load average, and return these to the job manager.
  • Added comments to all the public exports, in preparation for the first published version on crates.io

Building

• Smelter
  • Added better job cancellation support
  • Each job now as its own UUID
  • QOL improvements around status messages

Smelter had been on hold for a while, but this week at work a need came up for parallel processing, so I finished it up and ran the first real job!

This data processing pipeline formerly took 6 hours to run with a single process on a 24 vCPU machine, and now runs in 23 minutes using 32 8vCPU containers on Fargate.

The original vision for Smelter was to run it with Lambdas and other similar FaaS runtimes, but because it uses an adapter model to support new platforms, I was able to add Fargate with no changes to the existing code. Really happy how this turned out.

I do still want to implement the originally envisioned massively-parallel Lambda computation at some point too, for more interactive use cases.

Building

• Filigree
  • Started on many-to-many relationships using a through table and model
• Smelter
  • Tested out the Fargate orchestration code with real containers and made some fixes
  • Ergonomic improvements to spawning Fargate tasks
  • First real run of Smelter! This is a data processing pipeline at work that formerly ran with a single process, now running with 32 containers in parallel across the dataset.

I finally had a real-world need for Smelter, so I've revived the project and it now supports AWS Fargate. With the adapter model I'm using, implementing it for Lambda as well (the original vision) shouldn't be a big lift.

Building

• Filigree
  • Finished up tests for populated child models

Links

• PRQL is a data query language that compiles to SQL. There are a number of these projects, with Malloy being the other one you hear about the most.

I've been reorganizing my code snippets repo, and among other things I've updated my dark mode support for modern SvelteKit.

This supports SSR by persisting both the user's choice and the default setting in cookies, to avoid that "flash of light". Check it out here.

I also set up CI and websites for PromptBox and sqlweld, where they can easily be downloaded.

Links

• AWS announced a new S3 storage tier designed for low-latency analytics among other things. If I ever start working on Smelter again this will be useful. 7x the cost of normal S3 but I'm sure that's worth it for certain systems.
• Placekey looks like a nice solution for address entity resolution. It has a generous free tier and cheap paid tiers as well. Heard about it on from the Mapscaping Podcast.

Got the first version of Glance up and running. A bit more work to be done on design and then I'll be ready to write more mini-apps to feed in all the info I want to see... at a glance.

Spent some time yesterday and today updating my sqlx JSON companion crate to support sqlx 0.7. I also added a type that makes it easier to get a Box<RawValue> out of the query.

Also finally added tests! And wrote up some notes on using JSON in sqlx.

I created a small utility named sqlweld for assembling SQL query files from liquid templates with partials, so I can get statically-defined queries while still allowing reuse for common things like permissions checks. This way I can write raw SQL in my projects while still getting code reuse on my queries.

The first real use of this will be on Glance, which I'm finally starting on in earnest. Check out sqlweld on Github!

Updated SBBP so that I can add videos directly from the app, and also queue up multiple videos for download. Whisper currently doesn't do great on technical terms, so that needs some improvement. I think there's a way to do that with OpenAI's API, but it's not obvious how to do it with the Huggingface high-level packages, so I may need to dive in to the internals some to see if that can be done.

Somehow I missed that the WebP format includes a separate lossless encoding mode. I've made two updates to Pic Store as a result.

1. Added configurable quality to the conversion profile, and a 100 quality will do lossless encoding when the input is a PNG.
2. Updated the "reconvert" endpoint to read the conversion profile again, so that changes will take effect.

With this, I can finally upload a good screenshot from SBBP as well, where I used it to read through a MotherDuck talk on hybrid query execution and see the slides alongside.

I also sped up the image similarity process by 100x, by just running it all in one Python execution instead of starting a new invocation for every checked pair of images.

Overall I'm really happy with how this is turning out. Still need to make the design better but it's already very useful for things like long conference talks.

I added structural image similarity to SBBP , so that screenshots from the video that are too similar to the previous one are automatically dropped out. This makes the viewing experience much better since you only see a new image when something has really changed since the previous one.

Learning

Calculating Structural Similarity of Images

Turns out scikit-image in Python makes this really easy.

import cv2
from skimage import structured_similarity as ssim
img1 = cv2.imread(sys.argv[1])
img2 = cv2.imread(sys.argv[2])
sim = ssim(img1, img2, channel_axis=2)

I started a new project called "Should have been a Blog Post," or SBBP for short. This application downloads a Youtube video, extracts screenshots every 10 seconds, and runs the audio through Whisper. The text is then presented alongside the screenshots, so you can just read through the transcript and see the visuals alongside, instead of needing to spend an hour watching a single video.

Check it out on Github.

Added the ability in PromptBox to append strings from the command line and from stdin in addition to what's in the prompt template. Next up will be ability to submit images to GPT4 vision and Ollama (once the pending PR for it is merged).

I uploaded some sample prompt templates as well.

Links

• Spent the weekend playing some with Dagster for downloading and processing medical device approval data from the FDA. It's a nice system; the data asset model and partitioning fits well with how I like to think about things. Definitely worth checking out if you need to write data pipelines of some complexity.

Wrote a small note on how to Preview a CSV In-Browser with Papa Parse.

Links

• pipx is a program that exists solely to install and run Python executable packages, each in their own venv. Takes a lot of pain out of the nonsense and frustration that continues to characterize using Python.
• Rye does seem to work better for some packages, particularly those that refuse to run on the latest version of Python, which is basically anything ML-related for a few months after the release.

I added a Wiki to the svelte-maplibre Github page today. The initial purpose of this will be to collect useful patterns, such as this one to improve hover usability when rendering thin lines on a map.

I’ve been playing with using PromptBox for code generation given context from the repository. Having some trouble figuring out a good output format though.

I have been able to get it to output diffs, sort of. The main problem is that it loses track of how long the diff is supposed to be compared to what it puts in the header, so patch rejects it.

Maybe there’s some better way? Maybe I just need a post processing step to fix the diff or apply it some other way? I'll have to play with it some more.

SoundQueue

Every year I help out with the sound at my church's Christmas play. This year we're low on help, and the play is much more involved than your average Christmas play, so I find myself both running the sound board and triggering the music/effects. This was a good excuse to automate the latter task, and a good opportunity to explore Tauri a bit.

SoundQueue is a small program that reads in a manifest of a bunch of sound files, and lets you easily play them one at a time with the press of the space bar, queueing up the next sound when one finishes playing. It allows custom volume and in/out points for each sound as well.

I was pretty happy with the productivity of Tauri. Despite not being too familiar with it, this app took only a few hours to make.

I've spent the last few days building PromptBox, a utility allows maintaining libraries of LLM prompt templates which can be filled in and submitted from the command line. The templates are just TOML files like this.

# File: summarize.pb.toml

description = "Summarize some files"

# This can also be template_path to read from another file.
template = '''
Create a {{style}} summary of the below files
which are on the topic of {{topic}}. The summary should be about {{ len }} sentences long.

{% for f in file -%}
File {{ f.filename }}:
{{ f.contents }}


{%- endfor %}
'''

[model]
# These model options can also be defined in a config file to apply to the whole directory of templates.
model = "gpt-3.5-turbo"
temperature = 0.7
# Also supports top_p, frequency_penalty, presence_penalty, stop, and max_tokens

[options]
len = { type = "int", description = "The length of the summary", default = 4 }
topic = { type = "string", description = "The topic of the summary" }
style = { type = "string", default = "concise" }
file = { type = "file", array = true, description = "The files to summarize" }

Each of these options becomes a CLI option which can help fill in the template.

It works with OpenAI for the usual case, but you can also run it against LM Studio or Ollama if you like local LLMs. If you give it a try, let me know what you think!

Links

• Database of Databases, a list of almost 1,000 databases administered by Carnegie Mellon University

Learning

You can use git log --diff-filter D --name-only | grep .changeset to find the names of all the changeset files that were ever in a repository. Found on Waylon Walker's Site

❯ git log --diff-filter D --name-only | grep .changeset
.changeset/mean-spiders-promise.md
.changeset/slow-lizards-sell.md
.changeset/bright-falcons-study.md
.changeset/funny-pigs-drop.md
.changeset/big-olives-serve.md
.changeset/eighty-bags-smash.md
.changeset/proud-cougars-tease.md
.changeset/tall-trainers-smash.md
.changeset/tender-turtles-accept.md
.changeset/slimy-fireants-prove.md
.changeset/cold-knives-search.md
.changeset/neat-horses-rule.md
.changeset/tasty-ways-report.md
.changeset/giant-points-study.md
.changeset/good-shirts-accept.md
.changeset/heavy-drinks-flash.md
.changeset/long-mice-try.md
.changeset/purple-jars-rescue.md

Links

• pg_bm25 is a PostgreSQL extension that uses Tantivy to provide BM25-based full text search in PostgreSQL.

I had an idea today for a Github bot that you can enable to automatically reply to new PRs with some message, for when you’re on vacation or otherwise unable to look at it for a while.

The downside, which any seasoned open source maintainer will sense, is the implication that at any other time you will always respond promptly.

Wrote up a short article today on Svelte 5, focusing not on the ergonomic changes but on the long-standing bugs that will be fixed by the move to runes. Read it here: How Svelte 5 Runes will Fix Long-Standing Svelte Bugs

Links

• Andrew Lee from Shortwave gave a webinar a few days ago describing how their new AI Email Assistant works.

Today marks the end of my nine-year tenure with Carevoyance. It's been quite a ride with lots of ups and downs, but a great experience overall, and since the company has been handed over to H1 and scaled up, I've been getting that "founder itch" to start something new.

For the short term, I'm going to take some time off and work on a bunch of side projects and other stuff, but will likely be back at it again before too long!

Yesterday I released v0.4 of svelte-maplibre, and this is the first release for which I didn’t write any of the code. Great to have other contributors joining in!

I'm playing with the DeBERTa-v3-base-mnli-fever-anli model a bit to do intent detection for Buzzy. The idea here is that I can figure out if I should do a web search to add some context to help answer a factual question, or maybe a bit of math, or if the question doesn't require any of that. First impressions are that it should work very well.