Building

• Most of my time lately has been spent between my new startup and my new baby, but when I can squeeze in some free time I'm trying my hand at an autonomous programming agent based around Aider. First attempts with prompts I found from elsewhere were just mediocre, but I'm thinking something like this:
• Aside from Aider's internal repo map, I'll create a list of all the files in the repository and use an LLM to create a short description of each one.
• Core loop:
  • Get a task from plan.md
  • Use file_map.yaml to ask about relevant files, add these to Aider.
  • Generate a plan to implement this task. Lay out the interfaces between the frontend and backend, but do not generate any code except for type definitions.
  • Generate integration tests for the plan
  • Generate the frontend code based on the plan.
  • Generate the backend code based on the plan and the generated frontend code.
  • Check types and edit until type checks pass
  • Generate unit tests for the code
  • run/edit loop on tests
  • once tests pass, run format step
  • run postmortem step updating docs and lessons
  • update file_map.yaml file to describe any newly added files or files that
• As I make progress on this I may need to also come up with a system for developing detailed plan documents for each task. It can be done in ChatGPT/Claude but it gets a bit hard to manage all the different feature-specific chats all in one, and it would be nice to have this live alongside the actual coding agent and be informed by the repository structure.

• This is a Rust library which, instead of embedding a JavaScript engine directly into the application, communicates with a persistent pool of engine instances that are set up to execute code.
• It's somewhat difficult right now to embed a well-equipped JS engine instance into your application if you want to have full API availability, and so this gets around that problem while avoiding the overhead of starting a new process for every expression evaluation.

• Task List

  • Up Next

  • Soon

    • Allow passing additional dependencies which can be used in the scripts
      • This would probably be paths to somewhere on the filesystem that contains a library with package.json and all.
      • Or maybe just a simple package.json with node_modules directory and then the script can be run with that as the CWD
    • Support callbacks from the JS side to the Rust side
      • e.g. to do expensive lookups, mutate state, etc.

  • Later/Maybe

    • Consider using worker_threads instead of cluster
      • This should use a bit less memory (though probably not much less), but is complicated by the lack of ability to send sockets across thread boundaries, unlike cluster.
      • The bigger advantage is that it will make it easier to maintain a single code cache, and share data between threads in a zero-copy manner, using SharedArrayBuffers.
      • One approach is to have a single master thread create a new MessagePort pair for each connection. A thread gets the port and then the master shuttles buffers back and forth between them.
        • Mostly implemented here https://www.phind.com/search?cache=zdy35n834rpmds1cmiiszwt8.
        • Feels bad for latency though
      • Another way would be for each thread to have its own socket and then the Rust side communicates with each one and handles the load balancing. This would also potentially make it easier to share other things like persistent contexts but that is probably over complicating it.
        • This isn't great as a general approach but works well for this case because we know that there's just a single client (the Rust process) with exact knowledge of how things are structured.
        • I'll probably go this route.
      • Need to benchmark these approaches
    • Support multiple engines (Bun, Deno)
      • Bun doesn't support node:cluster yet
      • Deno should work with minimal changes
    • Run entire projects
      • with node_modules and so on
    • Support running Boa or QuickJS in-process with the same interface
    • Allow control over which Node built-in libraries can be imported
      • Currently none can be imported, this wouldn't be hard to change but making it configurable would be great.
    • Network sandbox
      • This can be done most easily by replacing fetch with a function that wraps fetch and checks the URL before passing on the request
    • Filesystem sandbox
      • Same as network, but for the fs functions. This is a bit more difficult to get right since there are so many functions.
      • Ideally we could just run the isolate in a chroot, but I don't think that's possible.

  • Done

    • Benchmarks
    • Support V8 code cache when evaluating code
      • Need some kind of content-addressable LRU caching so we don't hold on to code cache values forever
    • Customizable number of workers
    • Improve Rust API for an engine connection
      • Single function to run a script and wait for its result
    • Start a single engine
    • JS code to handle requests, run code, and respond
    • Send requests to the engine, get a response
    • Support one-way communication of a script back to the Rust side (e.g. progress messages)
    • Ability to reuse a context across multiple requests without reinitializing it each time.
    • Hook into console so that it goes back to Rust in a way that is linked to each request.
    • Run a pool of engines and load balance requests between them.

• Basic Design

  • Rust Side

    • Start and maintain a pool of engines
    • Allow passing JS code to the engine, either as strings or as filesystem references
    • Support callbacks which

  • JS Side

    • The core is a small application which takes in requests
    • Support auto-importing modules from the filesystem before running the expression.
    • Support compilation snapshots, if possible
      • Looks like Node.js support this via createCachedData. It doesn't cache stored data but does cache functions.
    • Library which can callback to the Rust side to take actions

• Alternatives

  • Why not Deno?

    • In my experience in the past this worked out ok, but
      • you have to set up a lot of runtime stuff yourself
      • Updating all the Deno crates together is a pain and there were often breaking changes to be handled.
    • Some of this might be better now as Deno itself has matured, but overall it seemed that embedding a "full-featured Deno" is not really as easy as it should be.

  • Why not bindings to QuickJS/Boa/etc?

    • Runtime compatibility. Some applications may need to do things that only really work in the most popular JS engines, such as talking to Postgres.
    • Harder to include arbitrary NPM packages or similar, without bundling
    • For applications that don't need this, I do think it's worth providing a mode that just uses bindings and won't have to start a separate sidecar process.
    • As QuickJS gets more WinterCG compatibility this also may be less of an issue.

Learning

Today I read StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows. This paper talks about using state machines to explicitly model the problem that an agent is trying to solve. They got great results, and I agree that it's a good way to go about solving a problem, but I'm not sure what's really new here. Maybe it's just my background, but using state machines to direct the high-level behavior of your agent feels like a very obvious thing to do, and a lot of people in the LLM space have been talking about it for a while now.

Learning

Today I presented a review of The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey at the Latent Space paper club. While it wasn't recorded, my slides are published on Google Slides.

• The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey

• Introduction

  • The paper defines agents as a system that uses planning, loops, reflection, and other control structures, as well as leveraging the model's reasoning abilities to accomplish a task.
  • This paper focuses mostly on the difference between single-agent vs. multiple-agent architectures.
  • The multiple agent architectures are then subdivided into vertical and horizontal architectures.
  • Each agent has a persona which is basically the system prompt as well as the tools that the agent has access to.
    • In addition to the instructions for the task, the persona may define a specific role such as an expert coder or a manager or a reviewer and so on.
  • Tools of course are external function calls that the model can request, such as editing a document, searching the web, or other actions that the model is not able to do inside its computations.
  • The paper defines a single agent architecture as those powered by a single language model which performs all the tasks on its own with no feedback from other models or agents. There may be feedback from humans though.
  • In a multi-agent setup, each agent typically has a different persona.
  • In a vertical multi-agent architecture, one agent access to leader and other agents report to it. There could be multiple levels of hierarchy as well. But the main distinction is the clear division of labor between the different sub-agents.
  • In a horizontal architecture, the agents are all more or less equal and are part of a single discussion about the task. Communication between agents is shared between all of the agents. And agents can volunteer themselves to complete certain tasks or call tools.

• Key Considerations

  • Reasoning

    • Reasoning is basically the same thing that we humans do where we think critically about a problem, understand how it fits into the world around us, and make a decision.
    • For a model, reasoning is what allows it to go beyond its training data and learn new tasks or make decisions under new circumstances.

  • Planning

    • Planning is an application of reasoning.
    • And there are five major approaches to it. Task decomposition, multi-plan selection, external modulated planning, reflection and refinement, and memory augmented planning. See understanding the planning of LLM agents.
    • Most agents have a dedicated planning step, which they run before executing any actions. There are many ways to do this. The paper particularly calls out Graph-enhanced Large Language Models in Asynchronous Plan Reasoning AKA "Plan like a graph," and tree of thought as examples which allow the agent to execute multiple steps in parallel.
      • Although my recollection of Tree of Thought was that it was more about trying different permutations of problem solving and not so much about planning.

  • Tool Calling

    • Tool calling goes hand in hand with reasoning and is what really allows the model to make effective and informed decisions.
    • Many agents use some iterative process of planning, reasoning, tool calling, and then breaking up the task into further sub steps with more planning and so on.
    • But some papers point out that single agent architectures often have trouble with these long chains of subtasks.
      • https://arxiv.org/pdf/2403.03031
      • https://arxiv.org/pdf/2401.17464

• Single Agent Architectures

  • Proper planning and self-correction is paramount here.
  • A big risk with single agent architectures is that because they don't have any external method of automatically correcting themselves, they may get stuck in an infinite loop where they reason the same step over and over again with the same result.
  • ReAct
    • ReAct: Synergizing Reasoning and Acting in Language Models was one of the first single agent methods designed to improve over single-step prompting. In React, which stands for Reason Plus Act, the agent has a cycle of thinking about a task, performing an action based on that thought and observing the output.
    • Aside from improved reliability, one big advantage of this method over previous single-prompt methods was that the sequence of thoughts and actions are all there to see, so it's easier to figure out how the model arrived at its conclusion.
    • But ReAct is susceptible to the infinite loops mentioned above.
  • RAISE
    • RAISE, as described in From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models, stands for Reasoning and Acting through Scratch pad and Examples. There's no 'I' but I guess they thought RAISE sounded better than RASPE or something.
    • It's based on the ReAct method, but adds the scratchpad for short-term storage and a data set of similar previous examples for long-term storage.
      • NOTE How does this work?
    • One interesting problem that the race paper found was that agents would often exceed their defined roles such as a sales agent role which ended up writing Python code. The authors also cited problems with hallucinations and difficulty understanding complex logic.
  • Reflexion
    • Reflexion: Language Agents with Verbal Reinforcement Learning
    • Reflexion is a method in which the agent is asked to reflect on its own performance with certain metrics such as success state and if the current trajectory matches the agent's desired task.
    • NOTE look at the paper to determine more about these
    • Some limitations cited by the authors
      • Reflextion is prone to falling into non-optimal local minima
      • The agent's memory is simply stored in the model's context with a sliding window and so older important items may be forgotten
  • AutoGPT + P
    • AutoGPT+P: Affordance-based Task Planning with Large Language Models
    • AutoGPT+P is a technique specifically designed for use in robotics. It uses computer vision to detect the objects present in a scene. And then can use four tools to try to complete its task.
      • Plan Tool
      • Partial Plan Tool
      • Suggest Alternative Tool
      • Explore Tool
    • The model also works in concert with a traditional planning tool using PDDL or planning domain definition language. This planner helps with translating the model's instructions into things that the robot is actually able to do given its physical limitations.
    • As with many of the above approaches, it does have some problems such as sometimes choosing the wrong tools or getting stuck in loops. And at least as described in the paper, there's no opportunity for human interaction such as the agent asking for clarification or the human interrupting if the robot starts to do something wrong.
  • LATS
    • Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
    • LATS is an algorithm based on Monte Carlo Tree Search. You can read the link for more details, but basically, it's inspired by Monte Carlo simulation in which you do a bunch of random runs to get a better idea of the probability space and the best action to take.
    • But as you can imagine, doing a bunch of random runs down a tree with language models can be very slow and expensive. Also, the paper doesn't tackle particularly complex scenarios.

• Multi Agent Architectures

  • Common themes with multi-agent architectures
    • Leadership of agent teams
    • Dynamic creation of agent teams between stages
    • Information sharing between team members
  • Embodied LLM Agents Learn to Cooperate in Organized Teams
    • This method uses a hybrid approach that is mostly a horizontal team, but has a leader agent over the rest of the team.
    • They found that teams with a leader finished their tasks about 10% faster and that without a leader the agents spend about half of their time giving orders to each other. Whereas with a single designated leader, the leader spends 60% of its messages giving directions, while the other agents can focus more on actual exchange of useful information.
  • DyLAN
    • Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization
    • DyLAN is a dynamic team method which uses elimination rounds to remove the agents that have contributed the least to the task.
    • Team Optimization
      • Each agent is asked to rank the other agents’ results
      • These ratings are aggregated
      • An “Agent Importance Score” is calculated
      • Low-performing agents are removed from the system
  • AgentVerse
    • AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
    • Agentverse uses a four-stage process
      • Recruitment, which uses a “recruiter” agent to generate personas for a set of agents to work on this iteration, based on the current goal state.
      • Collaborative decision-making between the agents.
        • This can be vertical or horizontal arrangement, depending on the task.
      • Independent action execution by each agent
        • Each agent uses a ReAct loop with up to 10 iterations to get to the desired output
      • Evaluation of how close the current state is to the goal.
    • This process can be repeated until the goal is reached.
    • One important finding here is that agent feedback is not always reliable.
      • Even if an agent’s feedback is not valid, the receiving agent may incorporate it anyway.
  • MetaGPT
    • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
    • MetaGPT focuses on using structured outputs to communicate between agents instead of plain text in order to reduce unproductive chatter and inefficiencies, such as "how are you? I'm fine".
    • It also implements a message bus which allows agents to publish their information to a common place but only listen to information that is relevant to them.

• Discussion

  • Single agent patterns tend to work best with a narrowly defined list of tools and well defined processes. They're also easier to implement because there's only one agent and set of tools, and they don't face the limitations of multi-agent systems like poor feedback from other agents or unrelated chatter from other team members. But they are more likely to get stuck in loops and fail to make progress if they find themselves in a situation that does not match their reasoning strengths.
  • Multi-agent architectures work best when feedback from different personas helps to accomplish the task, such as drafting a document and then reviewing or proofreading it. They're also useful for performing parallel execution when there are distinct independent subtasks. Multi-agent architecture is particularly advantageous when no examples of the task have been provided.
  • Feedback can be very helpful, but it's not a panacea. The AgentVerse paper notes a case where an agent gave invalid feedback to another agent, but it was still incorporated. Similarly, human feedback may conflict with the desired behavior of the agent, but because the language models tend to be willing to please, they may incorporate it anyway.

  • Information Sharing

    • Information sharing in a horizontal multi-agent system is very useful, but also has issues. For example, agents can too closely simulate a human when assigned a persona and start asking the other agents small-talk questions such as "how are you?" Agents may also be exposed to information that is irrelevant to their particular task, so systems that allow subscribing or filtering incoming information can be helpful for keeping an agent on task.
    • Vertical architectures tend to not have as many of these issues, but can encounter problems when the managing agent does not send enough information to its team for them to do the job. The paper recommends using prompting techniques to help with this.
  • Careful design of the system prompt for the persona can help to keep an agent on task and reduce the amount of unnecessary chatter between agents.
  • Dynamic team creation where agents are brought in and out of the system can be a big help because it excludes irrelevant agents from adding noise a particular stage of the problem.

• Limitations

  • Evaluating agents is difficult and there are not very many good standard benchmarks.
  • Many papers introduce their own benchmarks alongside a new agent system, which makes it difficult to compare agent systems beyond those tested in that particular paper.
  • Many agent evals are complex and require manual scoring, which can be tedious, limits the size of the evaluation set, and adds the possibility of evaluator bias. The complexity of agents also leads to a lot more variation in their outputs, so it's more difficult to properly determine if an agent's answer is correct or not.
  • As with language model evaluations, data set contamination is a problem, where the tasks that the agents are trying to work on can be found in their training data.
  • Many standard benchmarks designed for language model testing, such as MMLU and GSM8K, are not applicable to agents because they don't really exercise an agent's ability to reason beyond what you would find in a single call to a language model.
  • Some agent eval systems use simpler answers such as yes or no, which are easier to evaluate, but this limits the real world applicability of the eval, where most tasks require more complex answers. More complex benchmarks that use logic puzzles or video games come closer, but even in those cases it's questionable how much it translates to the real world, where tasks are less well-defined and data is dirtier.
  • The paper mentions WildBench and SWE-bench as a couple of benchmarks that use real-world data, though WildBench doesn't seem to be designed for agent testing.

Building

• Fixed some issues with svelte-maplibre and Mapbox's Draw plugin today. Both of these turned out to be general incompatibilities with Maplibre, where the Mapbox plugin (understandably) assumed the presence of certain Mapbox CSS classes. I released v0.9.9 of the package which includes some new CSS which allows the mouse pointers to work properly in Maplibre. And an example is also now up on the website.

One of the best parts about tools like Phind is you can ask it a question, follow up with "I'm on a Mac and this didn't work" and it will just tell you why.

Turns out Mac tools often use libedit instead of readline, so your keybindings need to go in .editrc instead of .inputrc.

Building

• I've managed to avoid learning Kubernetes for years, but it's finally time, so I'm setting up SBBP to run with it. I'll probably put Caddy in front. Aside from the learning, the main purpose of this is just so I can easily add the transcribed videos to Readwise.

Learning

SIEVE is a smarter change eviction algorithm that also manages to be simpler and faster than most other algorithms. A clever idea that optimizes for quick eviction, taking advantage of the property that in many cache use cases, a large proportion of the fetched objects are actually not fetched often.

Links

• This podcast with Flo Crivello has a lot of good information on how Lindy is training their AI agents. Most interesting to me was the talk about explicit training phases for agents when they encounter new tasks, where the user can give quick feedback on how the agent is performing on a set of initial work items, and have it adapt in real-time based on that feedback until the performance is acceptable.

Learning

Hit an unusual problem today where reading pixels back from a canvas HTML element won't always give you the same value you wrote, if Brave's anti-fingerprinting measures are on. (Anti-aliasing is an issue too, but we knew about that already.) I remember reading something about how browser-fingerprinting libraries can sometimes use techniques like this to determine unique users, but was baffled for a while until turning off Brave's privacy features made the bug go away.

• Aviator is a tool for managing stacked branches and PRs. Their CLI can be used separately from the rest of the product. Here are some useful commands for it:
• av stack sync rebases branches within the current stack and pushes
• av stack sync --trunk is like the above, but also brings the stack up to date with the trunk branch
• av stack sync --prune deletes branches for merged PRs
• av stack sync --all --trunk --prune brings everything in the repository up to date, deletes merged branches, etc.
• av stack submit to create and sync PRs for every branch in the current stack
• av pr create creates a PR for the current branch
• av stack branch creates a new branch off of the current branch
• av stack branch-commit -b <BRANCHNAME> createe a new branch and commits the currently staged files (see —help on this one for more options)
• av stack switch is a stack-aware TUI for switching branches.
• av stack tree shows the same diagram used by switch but only prints it.
• av stack reorder to move commits between branches in a stack, or also merge and split branches.
• Moving branches between stacks requires a few commands
  • git checkout <BRANCH TO MOVE>
  • git rebase <NEW_PARENT_BRANCH>
  • av stack sync --parent <NEW_PARENT_BRANCH>

• Provides a proxy which can be called instead of the normal URL, then passes a request to the LLM provider and returns the response. Chronicle can be embedded directly into a Rust application or can run as a standalone server.

• Task List

  • Up Next

    • Write documentation
    • Basic UI for visualizing runs, steps, etc.

  • Soon

    • Option to skip logging model messages
    • Python client
      • This should be both a normal client and have the ability to wrap other common clients such as the OpenAI SDK. Needs to be compatible with tools like instructor, DSPy, etc.
    • Add fixture tests for Fireworks
    • Add fixture tests for Together
    • Add fixture tests for Ollama
    • Add fixture tests for Anyscale
    • Add fixture tests for DeepInfra
    • When waiting to retry, detect if the request has disconnected and cancel it
    • Add test for JSON response_format

  • Later/Maybe

    • Function to validate configuration and return a list of errors
    • API Management
      • Endpoints to manage aliases
      • Endpoints to manage API keys
    • Watch and reload config file
    • Request Management
      • Allow passing through user-agent header
      • Simple Request caching
        • Cache responses based on provider/model/messages
        • Support in-memory, disk, database, Redis
      • Fetch secrets from AWS secrets store
      • When looping around providers with retries, omit providers who had an unrecoverable error.
      • Option for provider-level or model-level rate limiting
        • So that new requests coming in will automatically wait or fallback
      • Monitor error rates from providers and auto-switch
        • This can be built into the alias system perhaps
    • New Providers
      • New Provider: Cohere
        • Very different API than the others, it’s different enough from a feature perspective that I’m not sure it’s worth translating between request formats.
      • New Provider: OpenRouter
      • New Provider: Google Gemini
      • OctoAI
      • Lepton
    • Specific Provider Upgrades
      • Claude: support "is_error" flag in tool results
      • Claude: support images in tool results
    • Other Modalities
      • Support binary upload APIs like Deepgram as well
    • Logging
      • Send logged data to arbitrary HTTP endpoint
        • This should be done in a way that it can sent to something like Kafka, Elasticsearch, or Clickhouse using only the configuration, no custom code
      • Send logged data to S3
        • As JSON files? As Parquet?
      • Figure out what to do about large data
        • saving input and outputs is useful but can start taking up a lot of space. Ideally we can have something that places records in cloud storage, just need to figure out the formats and so on and if/how to make things queryable before they land in storage.
    • Analysis
      • visualize by arbitrary metadata
      • Ability to create database indexes on arbitrary metadata even in JSON fields
    • Price Tracking
      • Associate each provider and its calls with a pricing plan
      • Fetch and update prices for each provider

  • Done

    • Support prompt caching — Aug 25th, 2024
    • Support tools with Ollama
    • New Provider: AWS Bedrock — Jul 8th, 2024
      • https://lib.rs/crates/aws-sdk-bedrockruntime
      • https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock_Runtime.html
    • Run and Step tracing — Jun 30th, 2024
    • Add Mistral support — Jun 19th, 2024
    • Streaming with Groq and Ollama — Jun 1st, 2024
    • Enhance test suite with real-world cases — May 31st, 2024
      • This uses streaming and non-streaming responses from various provider types, for both regular text and tool calls.
    • Streaming with Claude — May 31st, 2024
    • Streaming support — May 31st, 2024
    • "Simple" API can build for Postgres
      • Dropped the "full web app" version of the API. This will come back at some later time
    • Anthropic now supports "required" tool mode
    • Recover from Groq function calling failure
    • Endpoint for generic event logging — May 8th, 2024
      • Take the same metadata that we use for LLM calls, store them in a different table with just an event type and data json blob.
    • Support tool use fields — May 1st, 2024
    • Simpler API server that supports SQLite — Apr 30th, 2024
      • This will just use the built-in proxy tables, but is better for simpler use since it writes to SQLite
      • Autoload config files from the XDG directories
    • Javascript client — Apr 29th, 2024
      • This comes with a Chronicle-specific client, and can also redirect clients such as the OpenAI SDK using a custom fetch function.
    • Submit request metadata (org/user/workflow id) via HTTP headers — Apr 29th, 2024
    • API should have default to do everything without authorization
      • Do this by not only setting up a default user, but also adding it as the anonymous fallback
    • Testing — Apr 26th, 2024
    • For API mode, add data tables as Filigree models instead of using the built-in tables
    • When multiple providers are in use, keep retrying even on normally un-retryable errors
    • Allow configuring fallback provider and model on retry. — Apr 24th, 2024
      • This is part of the model alias configuration. Basically instead of a single provider and model there's an array of provider/model/apikey tuples
    • Support model/provider aliases — Apr 23rd, 2024
    • Support api keys — Apr 23rd, 2024
      • These can only be referenced by aliases
    • Save metadata into SQLite or Postgres — Apr 22nd, 2024
    • Load model and provider definitions from a configuration file — Apr 22nd, 2024
    • Store and load model and provider definitions from the database — Apr 22nd, 2024
    • Configurable user agent for HTTP client — Apr 21st, 2024
    • Link requests to internal users/orgs/projects — Apr 21st, 2024
    • Configurable timeout — Apr 20th, 2024
    • Common format chat messages and responses — Apr 19th, 2024
    • Automatic retry with rate-limit support — Apr 19th, 2023
    • Endpoint that proxies the call — Apr 19th, 2024
    • Send all relevant metadata as Otel traces — Apr 19th, 2024
• Probably take some code from Promptbox and change that to use this as a library, since it already has some of the needed functionality
• Maintain a price sheet with input/output token price per provider and model
  • Each price sheet entry as an active flag
  • When prices are updated for a model, add a new entry and mark it active
  • In the future have a scraper or other mechanism of getting latest price data for each model
• Support multiple methods of output:
  • Record in a postgres table
  • Output OpenTelemetry
• Consider allowing metadata such as org and user ID can be sent in a cookie or in HTTP headers in addition to the body. Not sure how useful this is though.
• For each entry, record:
  • Org ID
  • User ID
  • Run ID (ID linking related prompt calls together)
  • Workflow Name
  • Workflow Step
  • Arbitrary other metadata
  • endpoint called
  • provider and model used
  • input text
  • output text
  • input token count
  • output token count
  • which price sheet row was used
  • response time

• Ramus is an in-process DAG and state machine executor designed for running agents with a focus on observability.

• Task List

  • Up Next

    • Probably drop this and use LangGraph instead?
    • Completely serialize and restore an agent
      • This needs to account for tool context as well, shared state between nodes, nested state machines, etc.
        • Probably split up "context" and "tools" where "context" always needs to be serializable and tools can contain functions.
        • Could go more complicated with serialize/deserialize functions but this gets messy in JS.
    • Agent visualization
      • This should be both a generic application that can visualize agents in any database with a compliant schema, and also a set of components that make it easy to do so in a customized fashion.

  • Soon

    • Helper function for simple chain -> DAG
    • Chatbot interface that can be adapted to web, Slack, Discord, etc.
      • Standard set of incoming and outgoing events for this case
      • Implement serialization
      • Multiple levels of verbosity
    • Serializable runner functions
      • This has the runner in the config as an object instead of a function, where the object contains a key into some runner registry that points to the function, and additional arguments to the function.
      • This will be very useful for DAGs generated by agents that may not be fixed in the code.
    • Dynamic node type that can expand its inputs into a DAG or state machine
    • Cache system
      • Mainly for tools that make network requests and for LLMs. This is mostly a debugging feature but in some cases will probably be useful for other things.
      • LLM request caching could be built into chronicle
      • Support in-memory, file-system, and network like Redis
    • Central orchestrator for multiple sub-state machines and agents.
      • This can be a convenient point for human in the loop and other types of things like that.
      • There should be some affordance here for cases where we want to allow the user to bail out of a sub-agent.

  • Later/Maybe

    • Replay a DAG from any point in its execution
      • Basically just rerunning a node from its inputs and setting things up so that the descendant nodes will all run appropriately too.
    • Replay a state machine from any point in its execution
      • At each step, save the input/output and a diff of the context.
    • Update inputs and rerun only affected parts of a DAG
      • This adds a new config key to nodes, where they can indicate which keys in rootInput they care about
    • Ability to edit a node's output that wasn't what the user wanted
      • Say, if a node got the wrong result and the user wants to correct it.
      • Ability to cancel just a single node and its downstream
      • Modify output of any node and rerun downstream nodes in the DAG
      • Make "cancelled" state non-permanent. Nodes return to a ready state or something when a cancel occurs.
    • Dynamic modification of a DAG or state machine
      • Add new nodes to the existing DAG or state machine. This shouldn't be too hard to do and will help with introspection if doing dynamic planning.
      • This should also be able to take another DAG configuration and merge it into the current one, along with support for mapping or prefixing the node names so that multiple copies of a DAG can be merged.
      • Where do the outputs of the merged DAG go? Need to think more about how this makes sense and fits together.
      • A simple version of this is the node that is dynamically configured as a subgraph. This probably makes the most sense as a longer-term solutiona s well.
    • Look at integrating with https://github.com/transitive-bullshit/agentic tools
    • Short term memory system
      • Sometimes we want to do RAG or something while an agent executes but won't need most of this data afterward. This basically becomes the ability to purge and/or archive the RAG data after we don't need it anymore.
      • One-file-per-agent using LanceDB or something like that could work well here.
    • Script to parse DAG dependencies and generate a Typescript interface with proper types for the inputs and outputs of each node
    • Manage multiple related state machines
      • e.g. multiple agents running similar tasks on different input for a usre
      • or multiple parts of a single plan all executing concurrently
    • Send an event to multiple state machines based on some criteria
      • This should allow some level of arbitrary criteria, like we want to be able to select based on an organization, a user, workflow type, start time, some other tag
    • Global rate limiting of active tasks
    • Evals
      • Design conversation
      • Eval database tables
        • Eval job - name, time, source (CI, manual, etc), user, optional description
        • Eval item - a single item from an eval job that links to a run. Contains additional info around the run
        • Eval score - one score for an eval run
        • Eval data set - inputs and outputs
        • Need ability to attach extra scores by manual curation.
        • Associate metadata with a run
      • Views:
        • compare results of a particular eval across different runs
        • Results for a particular eval over time
        • Total results for a particular category of run over time (e.g. for a particular workflow, the trend of success rate)
        • All results for a run
        • Filter by whatever, including metadata values

  • Done

    • Chronicle function for converting metadata into span attributes
    • Convert to multipackage monorepo
    • Allow overriding name of DAG or state machine on each run
    • Basic State machine support
      • Configuration
      • Function to execute one step of the state machine
      • Ability to call a DAG, other state machines, or an arbitrary promise-returning function in a state
      • Tracing/events on all transitions
      • Configurable transitions
      • Global error handler/state
      • Configurable override of error transition on a per-state basis
      • Send events to state machines
    • Semaphores per class of node
    • Stepwise mode for DAGs
    • Some way to send events out while still running (e.g. to send updates to the user)
    • Tracing spans for every state
    • Log state transitions, inputs, and outputs to a database
    • DAG excutor
    • Spawn state machines or DAGs from an existing task and wait for them to finish
      • This is just done via normal code, nothing special.
• The main features are:
  • "human in the loop" states
  • cyclic workflow graphs
    • with some control on the number of steps taken
  • Fully embrace "workflow as state machine" rather than just using them behind the scenes
  • arbitrary delays
  • rate limiting on how quickly an agent can make requests?
• What do we really need a workflow engine for?
  • Arbitrary delays e.g. resume after a month
  • Stop to get user input, and resume
    • Sort of. This makes it somewhat easier but a simpler system can accomplish the same thing.
  • Global rate limiting
  • Long-running tasks so we can resume from somewhere on failure
    • This can be somewhat emulated with checkpoints or by doing a state machine inside the worker and having it save its state on every transition
  • Events that can be sent to multiple dormant workflows based on some tag
    • Can be done manually though
• A good first effort would be to write agents as state machines where the state machine runs in the agent itself instead of being externally orchestrated
  • This simplifies the running a lot, especially for agents that don't need to sleep for a long time.
  • This leaves the external component as something that can resume and run a state machine, which doesn't have to be generic at first. It can just be a normal call to the service that runs it.
  • Downsides:
    • Not resilient to full process crash, although with an external sweeper this could be mollified somewhat
    • Are there any downsides around spawning sub-state machines? I think the agent software would have to be designed to allow resuming from a state like this, but otherwise it wouldn't be a problem.
  • Take a look at napi-rs or neon or maybe even just in WASM, to see if we can write a lot of this in Rust. Would simplify code reuse and reliability in some ways and look forward to more full-featured plans. Might be more complex than it's worth though, need to see.
  • Initial Layout
    • This is a library which can run an arbitrary state machine and persist the config and state into a database
    • Two modes
      • State machine
      • DAG
        • The difference is that because a DAG is non-looping, we can just run all the root nodes concurrently and then run other nodes as soon as their parent dependencies are done.
    • Each node in a state machine or DAG can spawn other DAGs and state machines, and wait for them to finish
    • Support running either preconfigured machine from the database or taking an ad hoc state machine
    • Some way to send events to a paused state machine to start it up again
    • State machines can run other state machines
      • Need to figure out how much explicit support this really needs
    • Ensure that we're never processing more than one state machine event at a time (except perhaps a cancel event)
    • When entering a state, supply
      • Previous state name
      • Current machine context
      • Data emitted from the previous state for this state
      • Data emitted from any submachine results that ran between the previous and current state.
    • When leaving a state, the function returns
      • Next state
      • Data for that state
      • Context updates
      • Submachines to run before transitioning, if any, and their inputs
    • Create a tracing span for every state
    • Global error state
    • States can override the global error state with some other state
    • Parallel states?
      • Need to figure out the right configuration for merging these back into a single state again though.
      • Probably easier if either
        • These are just run from within a state
          • But then it's just a normal call. Works but not great
        • These are submachines which run to completion and there's a fixed state on the other end that receives all the results.
  • Library V2
    • Eventually have preconfigured blocks for certain common things, that can be called without any custom code
      • Maybe do this sooner than later depending on how agent planning goes
    • Built-in loop control with loop count limit.
      • Probably do loops as a submachine since the scoped context is useful here
    • Submachines
      • Scoped context for groups of states
      • Also useful for embedding state machines from other sources into a larger workflow.
• A particular instance of a workflow can have its own priority
• Workflows can stream events out to the caller
• Optionally use a time-weighted priority system for queuing
• Model workflows as a Moore state machine
  • Probably will end up with a hybrid Moore/Mealy though
• State executor code can run child state machines
  • Have some way to time out on these, like we run 10 searches with a timeout of 15 seconds and when we reach the timeout use whatever results have come back.
  • Each submachine will run in parallel
  • I think it's best if these child machines are actually their own separate machines which can be spawned from the state, rather than part of the parent state machine config. This is because many of these child machines will be used by multiple workflows.
• States have categories (eg makes an LLM call or not)
• Rate limits global, per workflow type, and per category
• Scoped context so that a set of sub-machines can have its own context separate from the global context
• Workflow and state priority
• States can define a transition to a particular state on timeout
• Support websockets or 2-way gRPC for simpler one-off runs where we don't really want to integrate a whole web server
• Support running as a library for cases where the agent is in Rust
• States and Transitions can log events for analysis
  • Things like "agent required correction", "this is an error", "user accepted the answer"
• Each state (or state transition? whatever runs code) also has an "error" property which defines the state that it goes to upon an uncaught error. The state machine as a whole also has an error state which can perform some action.
• jobs register their state machine with a version and a place to call back when triggering states. Then anything can start the job with some initial context.
  • Allow upserting a config where it makes a new version if the config is different from the latest version.
• Some way to generate typescript types from state machine config
  • This should include a list of permitted next states from any particular state
• Each step returns:
  • next state. There should be some way of type checking the returned value from the worker.
  • Input for the next state, specific to that state?
  • Conditions to go to next state:
    • Optional delay before next transition. This can be encoded in the state machine config but also overridden
    • Some kind of polling of an endpoint?
      • Maybe the orchestrator calls an endpoint on some backoff schedule and depending on status code it either goes ot the next state or tries again. Need to think about this it might not be too useful
  • Context updates

• Bot Abstraction

  • Keep track of conversations.
  • Each "server" is a mapping of a platform (Discord, Slack, etc) and the ID of the team (Guild, Team, Org, etc.) on that platform to one of our orgs.
  • Each conversation has:
    • Our own ID
    • The server it belongs to
    • Which user started it
    • The ID of the conversation in the platform. This should be JSON since it could have multiple components (e.g. a channel and thread ID in Discord) but also should be easy to look up.
      • Might be best to just use separate tables for each platform for the platform-specific part. We do still need an indication in the main conversation table of which platform a conversation belongs to so we don't have to look it up in every table.
  • Sending events
    • Figure out which platform the conversation belongs to
    • Send the conversation ID and the standardized event to the platform adapter
    • Platform adapter translates it to the platform-specific format and sends it.
  • Receiving an event
    • Look up the conversation, if one exists
    • Translate the event to the standard format
    • send the conversation ID and the event to the main layer

• Notes from Phind

  • Here are some suggestions to improve and expand on your notes for a workflow orchestrator that models workflows as Moore state machines: 1. State Machine Definition:
    • Allow defining state machines using a declarative format (e.g., YAML or JSON) for easy configuration and versioning.
    • Include additional properties for each state, such as:
      • State name and description
      • Input and output data schemas for type checking and validation
      • Error handling and retry mechanisms
      • Timeouts and maximum execution time
    • Support nested state machines or sub-workflows for modular and reusable components. 2. Step Categories and Rate Limiting:
    • Define a clear set of step categories based on the type of operation (e.g., LLM calls, database queries, external API calls).
    • Implement rate limiting at both the global and category levels to control resource utilization and prevent abuse.
    • Allow configuring rate limits per category and provide options to adjust them dynamically based on load or priority. 3. Job Priority and Scheduling:
    • Introduce a priority system for jobs to determine their execution order and resource allocation.
    • Support different priority levels (e.g., low, medium, high) and allow assigning priorities to individual jobs or workflows.
    • Implement a scheduler that takes into account job priorities and available resources to optimize execution order and minimize waiting times. 4. Callback Mechanism:
    • Provide a flexible callback mechanism for triggering state transitions and notifying external systems.
    • Allow registering multiple callback endpoints or functions for different events (e.g., state entry, state exit, job completion).
    • Support various callback protocols (e.g., HTTP, WebSocket, message queues) to integrate with different systems. 5. Context Management:
    • Enhance the context object to store and pass data between states and steps.
    • Allow defining default context values and merging them with job-specific context during execution.
    • Provide methods to update, retrieve, and delete context variables within steps. 6. Error Handling and Recovery:
    • Implement robust error handling mechanisms to handle failures gracefully and prevent workflow disruptions.
    • Allow defining error handling strategies per state or step (e.g., retry, skip, abort).
    • Support compensating actions or rollback mechanisms to revert partial changes in case of failures. 7. Monitoring and Observability:
    • Provide built-in monitoring and logging capabilities to track workflow execution, performance metrics, and errors.
    • Expose APIs or dashboards for real-time monitoring and visualization of workflow progress and status.
    • Integrate with external monitoring and alerting systems for centralized observability and incident management. 8. Scalability and Distribution:
    • Design the orchestrator to be scalable and distributed to handle a large number of concurrent workflows.
    • Implement load balancing and horizontal scaling mechanisms to distribute workload across multiple nodes or workers.
    • Consider using distributed locking and coordination mechanisms (e.g., ZooKeeper, etcd) for managing shared resources and ensuring consistency. 9. Testing and Debugging:
    • Provide testing utilities and frameworks to facilitate unit testing and integration testing of workflows and steps.
    • Allow simulating and debugging workflows in a controlled environment without affecting production systems.
    • Implement logging and tracing capabilities to aid in troubleshooting and identifying issues during development and production. 10. Documentation and Examples:
      • Create comprehensive documentation that explains the concepts, architecture, and usage of the workflow orchestrator.
      • Provide clear examples and tutorials demonstrating how to define state machines, implement steps, and integrate with external systems.
      • Maintain a collection of common workflow patterns and best practices to guide users in designing efficient and maintainable workflows. These suggestions aim to enhance the functionality, reliability, and usability of your workflow orchestrator. They cover aspects such as state machine definition, rate limiting, job scheduling, error handling, monitoring, scalability, testing, and documentation. Incorporating these features and improvements will make your orchestrator more robust, flexible, and suitable for a wide range of workflow orchestration scenarios.

Building

• Filigree
  • Got the object storage abstraction and configuration finished up over the weekend.
  • Now I"m starting on a "File" type of model, which will allow the template system to generate code that handles uploads, and optionally stores things like file hash, image type, etc. in the database.

Building

• Filigree
  • Starting on object storage support. Once done, this will add a File type to the database model, set up the client, and also make it easy to do things like stream files to and from the client. And of course the template system will initialize everything for you.

Building

• Smelter
  • The worker framework now handles SIGINT for you, and gives your code a channel that will change when the task should be cancelled.
  • Published v0.1.0 of the Smelter crates, since it's about to go into production at work!
• Effectum
  • More work on outbox pattern. The code is pretty much done, just needs tests.

Building

• Effectum
  • Started on support for transactional outbox pattern, since I'm starting to add background worker support to Filigree
• Smelter
  • The AWS Rust SDK doesn't handle the ThrottlingError returned from the ECS API, and so they don't get retried. I added a custom `ClassifyRetry` layer to make this work.
  • Improved job cancellation on error, specifically around ensuring that there is time to send cancel requests to all the workers. There's still room for improvement around the ergonomics here I think, but it's an improvement.

Building

• Smelter
  • Worker containers collect statistics such as RAM used and load average, and return these to the job manager.
  • Added comments to all the public exports, in preparation for the first published version on crates.io

Building

• Smelter
  • Added better job cancellation support
  • Each job now as its own UUID
  • QOL improvements around status messages

• AWS sets up a bunch of convenient things in the default VPC for an account, which you may need to recreate when making a new VPC. Here's how to do that in Terraform.
• First, the VPC and subnets:

  • resource "aws_vpc" "my_app" {
      cidr_block = var.my_app_cidr
      tags = {
        Name = "my_app"
      }
    }
    
    resource "aws_subnet" "my_app" {
      vpc_id = aws_vpc.my_app.id
      cidr_block = var.my_app_cidr
      # Set appropriately for your needs
      map_public_ip_on_launch = true
      availability_zone = var.az
      tags = {
        Name = "my_app"
      }
    }
    
    

• For tasks that access the internet, you also need an internet gateway and a routing table to use it.
  • You can also use a NAT gateway or something, but this is the simplest and cheapest way to go.
  • Here we'll also set up a VPC endpoint to link directly into S3, which saves egress charges for going through public routes.

  • resource "aws_internet_gateway" "my_app" {
      vpc_id = aws_vpc.my_app.id
    
      tags = {
        Name = "my_app"
      }
    }
    
    resource "aws_route_table" "my_app" {
      vpc_id = aws_vpc.my_app.id
    
      route {
        cidr_block = "0.0.0.0/0"
        gateway_id = aws_internet_gateway.my_app.id
      }
    
      tags = {
        Name = "my_app"
      }
    }
    

• Finally, we set up a VPC endpoint to communicate directly with S3, without needing to go through the public internet. This can improve performance and save money.

  • resource "aws_vpc_endpoint" "my_app_s3" {
      vpc_id       = aws_vpc.my_app.id
      service_name = "com.amazonaws.us-west-2.s3"
    
      tags = {
        Name = "my_app_s3"
      }
    }
    
    resource "aws_vpc_endpoint_route_table_association" "my_app_s3" {
      vpc_endpoint_id = aws_vpc_endpoint.my_app_s3.id
      route_table_id = aws_route_table.my_app.id
    }
    
    

Smelter had been on hold for a while, but this week at work a need came up for parallel processing, so I finished it up and ran the first real job!

This data processing pipeline formerly took 6 hours to run with a single process on a 24 vCPU machine, and now runs in 23 minutes using 32 8vCPU containers on Fargate.

The original vision for Smelter was to run it with Lambdas and other similar FaaS runtimes, but because it uses an adapter model to support new platforms, I was able to add Fargate with no changes to the existing code. Really happy how this turned out.

I do still want to implement the originally envisioned massively-parallel Lambda computation at some point too, for more interactive use cases.

Building

• Filigree
  • Started on many-to-many relationships using a through table and model
• Smelter
  • Tested out the Fargate orchestration code with real containers and made some fixes
  • Ergonomic improvements to spawning Fargate tasks
  • First real run of Smelter! This is a data processing pipeline at work that formerly ran with a single process, now running with 32 containers in parallel across the dataset.