Ergo

Written — Updated
  • Ergo was an interesting experiment, but ultimately tried to do too many things at once and never quite found a real purpose. Most of what I actually wanted from this project will be done with Glance instead, with a much more modular, maintainable approach.
  • This is sort of similar to Zapier, IFTTT, etc., with some added low-code features.
    • Tasks can incorporate Javascript for more custom behavior.
    • Tasks can be persistent and update state or trigger actions in response to incoming events.
  • Developing on Github
  • Rework of Design
    • Developing complex JS within a web UI is a hassle. So discourage inline Javascript tasks.
    • Instead tasks will mostly be state machines that can optionally have snippets of Javascript in them.
    • External actions will all be done through raw commands, docker containers, or task orchestrators
      • Need to figure out a good way for tasks to communicate their results back to Ergo.
      • A few options
        • Provide an HTTP endpoint to hit with the output payload.
          • This has the downside that you have to provide a URL that is actually resolvable by the task.
          • Providing the base hostname or IP in the Ergo config would be a good way to do this.
        • Provide a file path to write with the output payload. For docker containers this would be a volume mounted into the container. For task orchestrators, can we do something similar?
        • Force use of WASM for tasks that need to return data more than a success or failure. This is appealing, but I need to see what is the best way to go about this since these tasks would probably want some sort of data persistence, and that's easier to do when the task is just something that runs externally.
    • Another possibility is to do a block-based design, similar to natto.dev, but where everything continues running on the server when you close the tab. This is slightly different from a state machine, more of a data flow type of thing, but fits well with the input-based model, and could probably reuse a lot of the UI design between the two.
      • Some blocks can be tables and graphs, and these will be automatically exposed in a “view” mode and can show up in some form in the dashboard as well.
  • Current Tasks
    • Need to drop a lot of this given the above design rework.
    • Move complex queries into files for better maintainability
    • Package import and export that contains tasks and the various other objects they reference.
      • Allow cross-instance import
      • Link imported objects to the original package to facilitate re-imports and other things.
    • Think about using row locking instead of serializable transaction mode when handling inputs
    • Enough to get the weather task working
      • New Action Screen
        • Actually make fields editable
        • New action screen
      • Action Account Management
      • Actual testing of periodic tasks
      • Script simulator
    • Task triggers can have a built-in payload filter
      • this would just check for equality on certain fields of the body, nothing complicated
      • or it could be Javascript?
    • Task triggers can be configured to fire periodically
      • Each periodic task trigger can be associated with a payload that must conform to the input schema of the task trigger's input.
      • Add a new table of periodic triggers
        • Each one links to a specific task trigger.
        • Each entry can be configured to run the trigger on a schedule with a particular payload.
          • The payload can be anything that conforms to the associated input payload's schema.
        • Trigger schedule formats
          • Cron format
          • Weekday schedules like "2nd tuesday of month"
            • Done via human-readable descriptions of
      • Track scheduled inputs using the input logs
        • Add a "scheduled_at" field to the input logs table.
      • Re-enqueue task triggers
        • Upon running successfully
        • On error if the maximum number of retries is hit.
        • With a sweeper that makes sure things get re-enqueued, just in case.
        • Test it
      • Test API code for creating/deleting/updating task triggers
      • Provide cheerio or something as scraping logic. (npm/skypack/pika/deno package imports will work for this)
      • Old plan
        • The task triggers already sort of support this so perhaps what we want is a generic main input type that the task triggers can link to.
        • How do we define a payload for the trigger? One option is to not have a payload or do make it fixed, like a timestamp and the trigger name. Since the triggers only belong to one task this is probably fine.
        • The next step is to make triggers run on a repeating basis. This could be done with a "periodic" field on the task trigger. Then these just get enqueued like anything else but with a scheduled "run at" time, perhaps with the job id being the task trigger id to make it easy to modify the job in the queue.
        • Can we do this as Nomad tasks? Probably not the right way to go.
          • How would a Nomad task communicate back to the server?
            • Create a temporary ID and the task can call an endpoint with the ID and whatever value was fetched?
            • There's a way to read a file out of the allocation. Not sure how reliably we can know that the allocation will still be there though after the process ends.
        • Optionally perform automatic dedupe against previous fetched payload, so that we don't trigger tasks if nothing has changed.
          • Actually just make this easy to do with functions that work with the task context.
    • UI Work
      • Input editor
      • Action editor
        • Screen is implemented but this is mostly readonly right now
      • Task action editor
        • Simple management is done. Need to add template editor
      • Task trigger editor
      • Task script editor
        • Allow writing tasks in Typescript? This would also help with type checking the context function and action definitions.
        • Run simulated inputs in the client and show the result, along with validating the actions taken.
          • ability to save input payloads
      • Text Editor Work
        • Javascript editor
        • JSON/JSON5 editor with schema validation
        • Validate use of payload/input fields?
          • Probably possible based on the schema for each input/action
          • Now that this is done in Rust it's just a matter of making that work properly.
      • State machine editor
        • Text editor as JSON with schema validation
        • Simulation and visualization of states
          • Can we take stuff from stately.ai? Probably more fun to build my own.
      • API key generation
      • Permissions editor for API keys and users
    • Task Queue Upgrades
      • Doesn't seem to run more than one task at a time
      • Use listener for Postgres queue checker for better response times.
      • Can I use a listener for Redis queue checker? Probably use the pubsub for this
      • Scheduled Task Sweeper
      • Move all queue staging to the generic model
      • Expired Task Sweeper
      • Just move to Prefect
    • Permissions system upgrades
      • Create an "owner" permission
        • This assigns full rights to an object
      • Think about how to better systematize permissions checks
        • Problems right now:
          • Permissions checks need to be built into every query manually
          • Failed permissions checks are indistinguishable from the "not found" case.
        • Possible solutions:
          • Do a separate query that does permissions checks before or concurrent with the primary data fetch.
            • This is less good for performance but better for abstracting out the permissions checks into one module.
          • Generate a separate boolean for the permissions check
            • This works ok. Doesn't really solve a lot of the problems but does let us separate permissions errors from "not found" errors.
            • Did this in Pic Store, it woks nicely.
      • Object tags
        • These should be not only a method of organizing objects, but permissions should be assignable to tags
        • Assigning roles to permissions on tags should be the primary method of arranging permissions.
        • A user should be able assign a tag to an object if:
          • The user owns the object or the user owns the tag.
          • The user has write permissions (or is in a role with write permissions) to objects with that tag.
          • Or org admins can do anything.
        • Should all user-created tags participate in the permissions system, or just admin-created tags? (Probably the former)
      • Role inheritance
      • Do we want to have something more expressive like casbin, or does the roles -> tags arrangement get us everything we need?
    • Scripting v1
      • Script executor
      • Action templates can be scripts instead of templates.
      • Implement scripts in state machine conditions
      • Tasks can be serialized scripts
        • This requires some database modifications.
        • Currently it's assumed that tasks are state machines. We can change the column names of the state machine config and data to reuse them for task scripts and task data.
      • Task context management
        • This is a simple method of keeping state for a task.
        • Tasks can set their context
          • Almost done. Need to write the context back to Postgres
          • localStorage is a common interface but it sucks since it's all string based
          • IndexedDB is too complex
          • Perhaps best to implement something simple myself for now...
            • setTaskContext(context)
            • This will then let us do whatever serialization is needed to preserve maps and things.
            • It would be easiest to use the raw serde functionality, but that prevents introspection.
            • Choice: TODO How to preserve task context
        • Feed context into the script on each run
          • Fetchable from in the script with Ergo.getContext()
      • Scripts can fire actions
        • This won't run it right away, but will add it to a list of things to do, which are then all enqueued at the end of the script.
        • Provide a function something like Ergo.runAction('actionName', args).
        • Enqueue the actions from the list
        • Testing
      • Implement module loaders.
        • Doing this on the client for now via Rollup bundling
    • User-definable inputs that trigger multiple tasks.
      • Currently inputs can only be created by admins but we want to have inputs that are per-org, with task triggers subscribing to those inputs triggering. This really brings up a few new features
        • Creation of org-specific inputs
        • Task triggers can listen to these inputs
        • Inputs can be triggered globally, for globally-owned inputs, or per-org, for org-owned inputs.
          • Inputs should be marked as global, which means that they can be triggered apart from a task, and will instead trigger any task trigger attached to the input.
        • Inputs can be triggered by other tasks
  • Future Tasks
    • tasks rescheduling themselves
      • How should this work? This feels like a different level where we are scheduling one-off input sends but it's kind of the same thing as periodic tasks. Probably this becomes an action/executor type to send an input at a scheduled time.
    • Figure out versioning
      • The ground work in the data structures has been done. Actual support for working with it still remains
    • Dashboards
      • Array of buttons to trigger tasks with certain inputs
      • Show recent runs
    • (MAYBE) Templates may need a way to interpolate arrays and objects into values instead of just strings
      • e.g. { "obj_field": object_argument }
    • (MAYBE) Tasks can access a KV store like redis or PG for more complex needs than the context system provides
    • Other Executors
      • Docker executor
      • Discord executor
        • Send a discord message. Useful for notifications.
        • This actually should be implemented as an action on top of the HTTP executor.
      • "Send an input" executor
        • To the triggering task, to a particular task, or to all tasks once global input listeners are ready.
        • And this needs to interact with the permission system.
          • See if the user can send inputs to a certain task.
          • Adding a global trigger to a task should require a permission.
        • Testing
        • New job to run a container image or run an existing Nomad job.
      • Kubernetes Executor
        • I'm not really going to use this but it needs to be done if this project is going to get a lot of external pickup.
      • WASM executor?
        • This would load and run a WASM module. Might need to wait until WASI supports socket operations for this to really be useful. Might also be obviated by the Docker/Nomad/Kubernetes executors.
  • Done Tasks
    • Connection Pool Changes
      • Switch to sqlx default pool
        • This wasn't as good of a fit back when I was doing Vault auto-reload, but I think this is more hassle than it's worth. Better to just kill and restart the container if you're doing this kind of thing.
      • See about switching from redis pool to a single multiplexed connection.
        • Status: So far this looks like a hassle since despite the conection manager supporting multiplexing, you need mut access to the connection to do anything which makes everything harder and it's not really worth it to implement interior mutability compared to the deadpool solution which already works.
        • Since redis is single-threaded anyway it's not like multiple connections gets us much.
    • Client-side Task Validation
      • Put task_types crate back into the normal tasks crate and figure out how to make that work
      • Add validation that referenced inputs, actions, states, etc. all exist
        • Base work is done
    • Endpoint to get recent logs
      • Go from inputs to actions
    • Dashboard home page
      • First version just shows recent logs.
    • Improve test coverage
    • API input HTTP endpoint can have the API key as part of the path. (done: query string)
    • Events v0.1
      • Table to store events
      • Table to store tasks
      • Table to store actions
      • Trigger events from endpoint
        • Simple state machine implementation
        • Figure out atomic updates for state machines
          • Maybe have a flag for single-state state machines that aren't keeping any state so we don't have to touch the database.
          • Use Postgres SELECT FOR UPDATE, or is it better to use a serializable transaction? I think the latter is better because it won't block runs that don't actually update any internal task state.
            • Serializable transaction
          • Queue up actions to be done in a postgres table so that the transaction itself doesn't have to wait for them to run
    • User Management v0.1
      • Simple users/roles/orgs tables
      • API keys for users
    • Pull actions out of the queue and run them.
    • Event logging on inputs
    • Discord webhooks for inputs and actions
    • Execute actions
      • Raw Command Executor
      • Http Request Executor
        • Add a per-task cookie jar or something? (probably not but is this useful ever?)
    • Each input and each action execution gets an ID which is created when it enters the queue, and those ID ends up in the action log so we can read back the results later and link actions to the inputs that most directly caused them.
    • Make sure all components are instantiated in the aio-server.
    • Clean up ergonomics of getting a connection from the pool March 19th, 2021
    • Single SIGINT listener that closes a broadcast channel, and all other things using sigint listeners listen to that instead March 17th, 2021
  • Why?
    • There are already good alternatives out there, but it’s fun to do my own.
    • It’s mostly an excuse to learn Rust async and server stuff
  • Packages to Use
    • Web Server
    • sqlx for database access
    • reqwest for http client? How does this compare to using raw Hyper?
  • Use Cases
    • Weather alerts using data from https://open-meteo.com/en
    • Scrape web sources and send emails based on some content in them
    • Receive a payload and run youtube-dl on the provided URL
    • Periodically download Roam database
    • Some sort of Twitter list archiving?
    • Scrape TradeMachine and load potential trades into database
    • Fetch filled brokerage orders
  • Data Model
    • Inputs
      • Inputs come into a task and trigger some behavior in the task's state machines.
      • Input Schemas
        • These just define a schema that a particular input can conform to.
          • Weather Info
          • A URL
          • A stock ticker and price
          • Any (for task-specific triggers only)
        • Certain fields an be defined as listenable, and then task inputs can filter on those
      • Periodic Input Checker
        • Some input schemas are linked to one or more periodic checkers that can automatically pull some data from somewhere else and then trigger inputs as necessary.
        • Eventually this should be automatically linked to adding inputs that need them, and check only the things that inputs care about. Probably will be manual at first.
      • Task Triggers
        • A task trigger is linked to an input schema and can listen to incoming events for that schema.
        • Task inputs can also just be specific to the task and triggerable by REST endpoint.
          • Eventually these inputs will have a way to define their own data schema.
    • Actions
      • Actions do something and can also fire more events
      • Some actions spawn docker containers
      • Others are built-in actions and can run commands, hit HTTP endpoints, and so on.
      • The action data model has four components
        • Executor
          • This is the "type" of the action, such as sending an HTTP request, running a Nomad job, and so on.
          • An executor defines some low-level templates to fill in specific arguments.
            • HTTP URL, Method, Body
            • Nomad job, the entire argument hash
        • Accounts
          • Account info for external services
          • These fill in templates in the action.
          • API Keys, URL components, etc.
        • Action
          • An action is a particular instance of an executor.
            • Run youtube-dl with some URL
            • Send an HTTP request to Slack.
          • Actions support zero or more account types
          • Actions have their own templates which fill in the executor template with a combination of constant data defined on the action and information from the task action and the state machine that triggers the task action.
        • Task Action
          • This is part of a task and is triggered from a task. It links to an action and fills in the specific values that it requires.
          • A task action links to an account, if applicable, and can also include default values to be used when executing the underlying action. These values can be overridden by the state machine that triggers the action.
    • Tasks
      • A task is a collection of state machines that listen to inputs and fire actions.
      • State machines can have persistent state across calls.
      • Tasks can fork themselves, for instance to launch a chatbot for a particular user in response to an inquiry.
        • How do we deal with routing future incoming requests to the correct task clone?
          • Ideas:
            • State machine can have a routing table.
              • This starts out as just an object in the context that it can update.
              • Later, expose a KV store abstraction to the task that it can use to maintain a routing table.
            • State machine provides an extractor to use on certain inputs (can be JSONPath or a script) that provides a key used to route to clones, so then the routing happens automatically.
              • The event that comes in then checks the routing table first, and on routing misses it goes to the state machine which can then spawn a new state machine or handle it some other way.
              • I think this will be the right way to go.
        • Forked tasks should be differentiated in the API and UI so they don't clutter up the main list.
        • Forked tasks need a way to "end"
    • Logs
      • Log each trigger, event, task, action, etc.
  • Scripting
    • Action Templates can be scripts
    • Task state machine handlers can be scripts.
    • Scripts need a timeout.
    • Ended up choosing Deno core
    • May still want to consider WASM at some point
  • Event Loop
    • Events and actions are each in a separate queue
    • Separate from the web server, there will be listener processes for events and actions. (Currently all-in-one)
    • The events queue listener will actually run the tasks
    • The actions queue listener is just responsible for running actions.
  • GUI Ideas
    • Since each task will be a network of connected nodes, how can we make that easier to deal with?
      • Zoom in on state machines.
      • Allow grouped nodes and autolayout of just those nodes within the group, and treat group as a single entity for autolayout.
      • Sometimes like vim easymotion to jump between different nodes
    • Simulate state machines on the client
    • Verify JS code on the client
  • State Machine abstractions
    • One thing that would make this much more useful is various ways to define state machines that don't look like state machines, but internally compile down to them.
    • Chat bot decision tree
      • This doesn't stray too far from a state machine, but does abstract over the existence of states and hardcodes the event types. Probably a good first start.
    • Imperative programming compiled to state machine, where each call to an action is a continuation stopping point (see Temporal workflows for inspiration).
      • Look at async/await/continuation compilation, since these internally compile down to state machines.
    • Some way for tasks to wait for multiple inputs to arrive and then to act on all of them at once.
      • Essentially this saves the incoming payload of each input in the context and then transitions to a new state that processes them all. Just need to figure out the exact semantics on handling concurrency and time delay between arrival of inputs here.
  • Version Control
    • Tasks should be versioned in some way so that the user can compare with previous versions or restore to them.
    • It should be possible to create staging versions of tasks that can then be promoted to the "real" version after some testing.
      • This means that for each task, there will be an external_task_id for each version, as well as an external_task_id for the "active" version.
    • Questions
      • How do we decide when to create a "new" version? Is this just done explicitly by the user?
      • How do we manage these versions?
        • I guess just store them in Postgres.
      • Granularity of versioning?
        • Task-level is the most obvious, but there's a good argument to do it on some sort of "task group" level too. Probably just task-level though.
      • Might be cool to have some way to just sync a bunch of task definitions stored in Git to the system.

Thanks for reading! If you have any questions or comments, please send me a note on Twitter.