Journals for 2024-03

Building

• Filigree
  • Got the object storage abstraction and configuration finished up over the weekend.
  • Now I"m starting on a "File" type of model, which will allow the template system to generate code that handles uploads, and optionally stores things like file hash, image type, etc. in the database.

Building

• Filigree
  • Starting on object storage support. Once done, this will add a File type to the database model, set up the client, and also make it easy to do things like stream files to and from the client. And of course the template system will initialize everything for you.

Building

• Smelter
  • The worker framework now handles SIGINT for you, and gives your code a channel that will change when the task should be cancelled.
  • Published v0.1.0 of the Smelter crates, since it's about to go into production at work!
• Effectum
  • More work on outbox pattern. The code is pretty much done, just needs tests.

Building

• Effectum
  • Started on support for transactional outbox pattern, since I'm starting to add background worker support to Filigree
• Smelter
  • The AWS Rust SDK doesn't handle the ThrottlingError returned from the ECS API, and so they don't get retried. I added a custom `ClassifyRetry` layer to make this work.
  • Improved job cancellation on error, specifically around ensuring that there is time to send cancel requests to all the workers. There's still room for improvement around the ergonomics here I think, but it's an improvement.

Building

• Smelter
  • Worker containers collect statistics such as RAM used and load average, and return these to the job manager.
  • Added comments to all the public exports, in preparation for the first published version on crates.io

Building

• Smelter
  • Added better job cancellation support
  • Each job now as its own UUID
  • QOL improvements around status messages

Smelter had been on hold for a while, but this week at work a need came up for parallel processing, so I finished it up and ran the first real job!

This data processing pipeline formerly took 6 hours to run with a single process on a 24 vCPU machine, and now runs in 23 minutes using 32 8vCPU containers on Fargate.

The original vision for Smelter was to run it with Lambdas and other similar FaaS runtimes, but because it uses an adapter model to support new platforms, I was able to add Fargate with no changes to the existing code. Really happy how this turned out.

I do still want to implement the originally envisioned massively-parallel Lambda computation at some point too, for more interactive use cases.

Building

• Filigree
  • Started on many-to-many relationships using a through table and model
• Smelter
  • Tested out the Fargate orchestration code with real containers and made some fixes
  • Ergonomic improvements to spawning Fargate tasks
  • First real run of Smelter! This is a data processing pipeline at work that formerly ran with a single process, now running with 32 containers in parallel across the dataset.