Filigree

Task List

• Up Next

  • Make test objects easier to manage
    • Instead of a one-size-fits-all bootstrapping we should just do it with a custom function per file, which will speed up the tests overall a bit, and also greatly simplify testing of through models where we need to make sure that the IDs all line up between the different objects.

• Soon

  • Investigate making it easier to auto-forward page actions through SvelteKit to the API
  • Finish tests for through relationships
    • Do this after Make test objects easier to manage
  • Evaluate faster API authentication methods between SvelteKit Backend and Rust Backend
    • Basically to make it work better to make multiple API calls from a single request without hitting the database every time to look up the user/roles/etc.
    • Instead of just a session ID, have an option to generate a short-lived JWT or similar token that contains all the auth info. Store the JWT in the cookie, on each request check the expiration time and get a new one if getting close to expiring.
    • Might also need some hook for client-side requests that can do the renewal if they talk directly to the API.
  • Use docker compose with separate containers for API and SvelteKit parts
    • More trouble than it's worth to force the two parts into a single container
  • More flexible configuration
    • Switch from clap to config or something
    • Main impetus here is to make it possible to read configuration from ConfigMap-sourced volumes in K8S and similar systems.

• Later/Maybe

  • Use a build script to inject the current filigree version into the value used by add_deps
    • Minor thing but makes the release process a bit smoother. Right now this has to be updated manually.
  • Simplify permissions system when using custom auth
    • Since we aren't creating the roles and users, we need a way to make sure that people have reasonable permissions when they start.
    • Perhaps just a function that adds default permissions or something like that when we first see a user.
  • organization_members needs to be extendable like users is
    • This is so that there's a place to attach per-org metadata and fields
    • I think the proper thing to do here is that once we fully support Joining tables — Jul 22nd, 2024 we can just put this into the system as a joining table.
  • Improve API error printing
    • Some errors use source and error_stack's default printer ignores it. Should walk the source chain as well. This seems difficult though without using nightly features
  • Metrics tracking and alerts
    • Consistently high CPU, RAM usage, etc.
    • Might just use a service for this, need to see
  • Option to only allow connections from certain IP ranges
    • eg https://www.cloudflare.com/ips/ when using Cloudflare
  • Helpers for retrying requests and other operations
    • rate limit retry-after
    • exponential retry with jitter, etc. for regular retryable failures
    • ability to not retry for certain errors
    • Maybe just fork backon for this?
  • QoL Issues Encountered
    • Conflict in src/main.rs on first run
      • Maybe detect if this is the first run by looking at absence of .state directory and ask if it's ok to just replace it
  • Add auto-bootstrap option
    • When appropriate environment variables are set, allows the API to do the bootstrapping itself when it first starts up
  • Ops helpers
    • This isn't part of the core framework, but a set of tools for automatically doing ops-related tasks.
    • Back up database to cloud storage on a regular basis
    • Deploy on Git push
    • Preview environments?
    • Set up Postgres slow statement tracking and create a report on what was slow
    • Drop partitions of old data
  • Frontend
    • SvelteKit: Error page
    • Add dependencies to package.json like we do for the Rust side
    • App shell with nav bar, etc.
    • Password reset page
    • Profile management page
    • Manage your organization
    • Default Model editor page
    • Default Model list page
    • Admin: invite users to app
    • Admin: administer organizations
    • 404 page
    • TS Type Enhancements
      • Permission names per model
  • HTMX support improvements
    • Debugging option to show a toast whenever a livereload finishes or an htmx error occurs
    • move client-side code from SBBP into Filigree for reuse by other projects
    • theming, dark/light mode with persistence via localStorage and header script to inject dark class
      • DaisyUI provides a lot of this already
    • Island architecture support for Svelte components
      • Each of these should be its own entry point in the Vite config, and from there we can use the manifest support to get the necessary tags
    • Flash/toast support
      • Can probably be done with HX-Trigger header, or maybe an extension
      • Adapter function for actions to have errors return a toast
    • Implement fancy dropdown menu in Alpine with floating-ui
      • https://github.com/awcodes/alpine-floating-ui may work well
      • https://alpinejs.dev/plugins/anchor is the official one, seems to support fewer options
    • Improve livereload performance
      • Not sure how much opportunity there is here but it feels like it can be made 20% faster
      • Dioxus is working on a solution that I should be able to adapt once it's public.
  • Models Backend
    • Move a lot of the auth queries like add_permissions_to_role into the template system, so they can be placed on the model and/or customized
      • Everything in fiiligree/src/users except add_user_email_login which is used by the OAuth system and so is harder to move.
    • Generate endpoints for grandchild objects
      • Probably need to make this generally recursive which may difficult to do via the current template system, might need a macro that can call itself or some fancier fragment rendering.
      • I think what we want to do here is for each item in children to potentially have its own children, instead of recalculating each one independently. Also need some way to walk up the stack of children here for URL generation (or do we? Probably it doesn't matter to have the grandparent in the URL itself).
    • Make it easier for differing get/list model types to be used interchangeably
      • This matters more for HTMX mode
      • Basically we might have a function that renders something for a particular object, but it gets difficult when the input can be either the Get or List result types when they are different.
      • One option may be to generate a type of view struct which can take a reference to either type and then generate a bunch of methods for the shared fields which can return either value. This is a lot of boilerplate, but its all autogenerated, but these aren't used too often
    • Allow joining models to have their own extra data fields
    • For runtime queries, create query builders that use SeaQuery to add the auth clause automatically
      • Add an enum of columns in the table
      • Allow adding any population of data or id at runtime
      • Each child population can be seen as a column as well (but probably don't do this, it should )
      • Add functions for "all base columns" to drop into the query
    • tests for list endpoint filtering/pagination/sorting
    • Models with non-writable, but required, fields don't generate valid tests
      • This can happen when the normal "create" workflow happens in some other way than calling a REST create endpoint.
      • Also consider detecting this and just not generating an "insert" script that can't work.
        • The criteria is a model that has at least one field which
          • is not fixed
          • is not nullable
          • has no default (SQL or Rust)
          • is not writable by owner
    • Support file uploads in multipart form submissions
      • A lot of the underlying framework for this is ready but the code itself needs to be written. Main issue is that the current FormOrJson can't be used in that case. Instead we probably need something that even on a JSON file looks like a multipart submission, but just doesn't return any files.
    • Test populated get and list when updatewithparent is false
    • Models can define extra permissions
  • CLI / Template System
    • Bug: CLI doesn't throw an error if a field is defined twice
    • Use toml_edit to allow adding features to Cargo.toml
    • When generating migrations, read in the actual app migrations and ignore tables that we don't care about.
      • This lets us see things like model tables having been removed if, say, a migration file was deleted since it did the wrong thing.
      • Do need to figure out how to handle extra columns added in the model via manual SQL migrations here.
    • Command to print config and diff files against last gen and current config gen
    • Consider using something like sea_query to generate queries instead of a bunch of fixed queries
      • Will add flexibility, and could make a lot of things simpler. Need to see if there's still a good way to pre-generate queries when running the CLI
    • Data-driven default org contents, even if just a JSON blob to start
      • This can piggyback off of the fixture system
      • Considering not doing this since it may be simpler to just do it in a function.
    • Some way to easily add hooks to CRUD operations
      • Developer should probably just edit the generated code here.
      • Maybe add stub functions to each model to allow adding pre/post save hooks
    • Make it easy for additional queries to use the permissions macros
      • Probably moving away from SQL and toward Rust code for permissions checks so this won't matter as much
      • Can suggest using sqlweld here
      • See if there's a path to using a query builder like sea_query instead, maybe as an addition to the existing thing.
    • Autogenerate belongs_to field if necessary when the inverse has is detected without a through table.
      • Currently this returns an error instead
  • Authentication and Registration
    • Make organization_members a real model instead of just a table
      • Once joining tables work properly
    • Make SessionBackend work with custom auth
      • It mostly works but when the tables are in another schema then it doesn't. Need a less-fixed query method here.
    • Rate limiting for login-related endpoints
      • https://docs.rs/governor/latest/governor/
      • https://lib.rs/crates/tower_governor
      • https://github.com/brandur/redis-cell if i need this to be distributed
        • DragonflyDB is a redis-compatible that has this built-in
    • Passwordless login should include a code that can be pasted into the UI to log in instead.
    • Password requirements
      • length
      • mix of character types
    • Prevent reuse of recent passwords
    • Expire passwords that need a rehash due to upgraded standards
    • Option to expire passwords after a certain amount of time
      • This isn't recommended practice but some enterprise clients require this in order to sign with you.
    • Passkeys
      • https://passkeys.dev/
      • https://github.com/1Password/passkey-rs
    • RSA timecode Authenticator app
    • Authenticator Account recovery codes
    • Account self-deletion
    • Optional TOS acceptance at registration
    • Service accounts
    • Record info about sessions (geoip, user-agent, etc.)
    • Allow viewing other sessions for your user, and potentially logging them out
    • Invite users to your team
    • Invite users to the app, but not in your team
    • Link a new OAuth login that may not have a matching email to an existing account
    • When logging in via OAuth, fetch all known emails from the OAuth provider and see if any of them match an existing account if we haven't seen the account before.
      • Need to handle the case where there are multiple emails and they match different accounts. This probably requires an interstitial after the login page to select the matching account.
    • Waitlist functionality
    • Put auth code behind an adapter so it can be swapped out for a 3rd-party service
  • Permissions and Authorization
    • Project-based Access Control
    • Consider separating permissions checks from the SQL queries themselves
      • This has the upside that it would simplify a lot of checking for permissions and make more complex permissions easier
      • Upsides
        • SQL templates become simpler
        • Much easier to implement ABAC
        • Easier to differentiate authz errors from missing objects
      • Downsides
        • Requires remembering to actually check for permissions, though that's alleviated some by the template system since the checks can be placed in the access functions.
        • We also end up reading objects multiple times in update/delete situations, though this isn't that big of a deal. For project-based permissions we could potentially load all the projects though would have to see about that.
      • Should probably do this as permissions is becoming more and more complex as more options are added.
      • Maybe consider something like casbin or biscuits too. Feels like overkill at this point for most things though, probably more appropriate for an API-first product.
      • list query would still need permissions checks in the SQL
    • API Key lookup query needs to change to make permissions a subset of the user's permissions when inherituserpermissions is false
      • This way if a user has some permissions removed, they can't bypass those limits using an old API key which still had those permissions
      • This is actually a good use case for casbin or biscuit, though could still be implemented without it.
    • Feature flags
    • Add "all" model permissions so that some roles have permissions on everything even when new models are added.
      • This needs some more thought though. Some models like user, org, etc don't want to have this blanket rule applied. Might be better to just do this via migrations.
      • Better: Have some models use global org-wide permissions of just "read," "write," etc. and only specific models have their own set of permissions.
    • Ability to apply limits on how many objects are created
      • There's a lot of flexibility here, since the limits could be something simple like number of objects, or some derived metric such as "number of minutes" for an audio-based model.
      • The source for the limits could also come from a number of different places, most commonly pricing plan but could be other stuff such as number of community contributions, upvotes, etc.
      • So the question is what's the proper framework to make it easy to write code that accomplishes all this.
        • Make it easy to fetch the limits data.
        • Make it easy to enforce rules around these limits.
    • API call quotas
      • e.g. 3000 calls per month, 20 calls per hour
      • This should be able to group endpoints by categories where they all share a quota
      • Might need to do this with a combination of event tracking and a rate limiter
    • Usage-based pricing
    • Ability to add rate limiting on any endpoint
      • keyed by org
      • keyed by user
      • endpoints can share a bucket
      • Local tracking is fine at first, should eventually support doing it in Redis or similar to support multiple API servers.
    • Ability for admins to assume permissions of other users
      • This needs to be audited as well even if nothing else is audited
  • Workers
    • Ability to run worker tasks in some other process/VM/FaaS
      • Need to figure out configuration for this... this might be better implemented in the task queue itself
    • Outbox pattern support for transactional starting of background jobs
      • Not important for now since Effectum only runs in-process but at some point it will have a server mode and then it will make more sense.
  • Block Storage and File Uploads
    • Option for Uploads via signed URL
    • Generate tests for file upload endpoints
    • When retainfileon_delete is set, optionally add an entry to a table that lists orphaned files.
      • The idea being that we might want to hold on to orphaned files for some period of time, and then delete them later.
    • Image statistics on upload
      • format, width, height
        • Can detect on upload using code from Pic Store
    • Generate image thumbnail and blurhash
      • Best done in background worker
    • Potentially need a way to upload a file before the parent object exists. Will probably wait for a real use case on this one.
      • Could make the parent ID nullable here and then add an endpoint to "claim" a file for a particular parent object
      • Also could just do the Github style thing and upload them to a publicly available bucket. We don't maintain a file database entry in this case, and just return the public URL for whatever use it is.
    • Allow specifying storage provider presets from env vars, not just in config
  • Notifications
    • Notifications: Other types of notifications?
    • Notifications Internal notifications
      • e.g. Discord webhook notifying myself about events
      • This should have notification templates and also destinations for the notifications
  • Event logging
    • Should support arbitrary types of events and also allow setting up queries that filter on various event sequences for a user/org
  • Workflow System
    • Build the workflow system
      • A lot from Ergo can be reused
    • State chart implementation
    • Workflows can "sleep" by enqueueing a scheduled task that will continue running when it wakes up
    • Endpoint to send an event to a workflow
    • Define workflow actions
    • Ability to define workflows in the app config
    • Scripting for Workflows
      • Piggyback on Ergo but maybe switch to QuickJS or saghul/txiki.js: A tiny JavaScript runtime
  • Hosting
    • Not sure if I'll do any of this, it's really better handled by a reverse proxy
    • LetsEncrypt support
    • TLS/HTTPS support
    • HTTP -> HTTPS redirect
  • User model list/get functions need to return users which are in multiple orgs, and whose current org is a different one
  • Easy setup for commercialization
  • Collect and export metrics to some tool
    • Looks like Honeycomb supports Prometheus format
  • Basic Analytics
  • More complex workflow/funnel analytics
  • Data Hooks and Similar
    • Delete log table for possibility of undelete later
    • Option for auditing on all operations
  • Password confirmation flow for "dangerous" actions
  • User profile photos
  • Copy all tests in test-app into filigree crate where it makes sense
    • Some tests are more convenient to test in the test app, but rely on minimal scaffolded functionality. Should add a minimal server/migration setup in filigree crate tests so that they can run there as well.
  • Support SQLite
    • Not sure how much work it will be to support both Postgres and SQLite, could turn out to be a big hassle due to differing feature support and syntax, but I'll see at some point
    • Things I use that are different in SQLite
      • AL JOIN (it supports them, just not with this syntax)
      • No native uuid and timestamptz types
      • No array_agg - this is probably the biggest one
      • jsonb functions are a bit different
      • parameter binding syntax is different

• Done

  • When using "string auth IDs" or custom auth in general, make it a different field that references the user/organization/role ID in the external system, and continue using a normal ObjectId type internally — Jul 22nd, 2024
    • This simplifies a whole bunch of code that otherwise has to go back and forth depending on the actual underlying type of the auth object.
  • Joining tables — Jul 22nd, 2024
  • Model should have functions for adding/updating/removing child objects — Jul 17th, 2024
  • Rework query template system — Jul 16th, 2024
    • Create more of the query template stuff in the Rust code instead of in Tera
      • This will make certain things a lot simpler especially when dealing with binding parameters
      • When generating a static query, keep a mapping of binding parameter to position, and have some method for the template system to generate a bunch of bindings from an object.
        • This allows the query system to call a single function with data generated along with the bindings, which adds the relevant parameter bindings and doesn't require us to keep the two places in sync.
    • Remove _permission from all the structs
      • This removes the ability to have the custom Serialize impl that only serializes fields visible to the Owner if the permission is write.
      • How useful is it anyway to have fields which are only viewable by the owner? Maybe it doesn't matter.
    • Remove permissions checks from queries for global auth scope cases where we already know the permissions from the roles
      • project-level and object-level still need it
    • Change all the query functions into objects on a struct. Consider making the struct into a thing that takes a reference to the AuthInfo.
      • This mostly just makes the typing experience nicer since instead of post::queries::create you call Post::create.
  • Make user/org/role/etc. models optional — May 26th, 2024
  • Allow user/org/role IDs to be strings when using custom auth — May 26th, 2024
  • Add more test apps — May 24th, 2024
    • Add more as big new features are added
    • These should mostly be .gitignored except for the files that filigree doesn't generate
  • Bug: Unique fields need to be unique per org, not globally — May 23rd, 2024
  • Pluggable auth system — May 22nd, 2024
    • Support talking to some other auth microservice, external service, using JWTs or Biscuits, etc.
    • This requires
      • A new function on the AuthQueries trait that can take a request Parts instead of a pre-parsed API key or session ID
        • Or maybe just change the api_key to pass the raw Bearer token, and make it easy to unpack that into an API key for standard uses
      • trait functions should return some error type other than an sqlx::Error. An AuthError is probably the way to go or maybe even just start using Report there so we can save the internal error
  • Interactive bootstrap command — Apr 26th, 2024
  • Command to list all the environment variables that will be read along with their defaults if not provided
    • Might be a bit complex but worth it since there can be a lot
  • Bug: when deleting a model the files under src/model/NAME/ are not removed.
    • Probably need to read the .state directory to see what files we generated before but aren't being generated this time
    • Also don't delete files which have been edited
  • Anonymous user mode
    • Anonymous users get a default Authed with a specific org and user ID created just for the anonymous user.
  • Allow specifying a user ID for anonymous users to impersonate. This lets them have specific permissions, see certain objects, etc. — Apr 16th, 2024
  • Add init command that runs Cargo init, create-svelte, and creates basic config.toml? — Apr 16th, 2024
  • Add a command that creates a new Postgres user, password, and database for your project — Apr 16th, 2024
  • Support maud/htmx as a SvelteKit alternative for less complicated apps — Apr 16th, 2024
  • Switch out deprecated Jaeger tracing crate for opentelemetry-otlp — Apr 15th, 2024
  • When a type and its populated type are the same, don't generate e.g "ListAndPopulatedListResult", just "ListResult" — Apr 15th, 2024
  • htmx: manifest alterations need to be a vite plugin so they work in dev mode — Apr 12th, 2024
  • htmx: live reload — Apr 12th, 2024
  • htmx: asset pipeline config to bundle CSS and scripts, compress assets, etc. — Apr 11th, 2024
  • htmx: build JS with vite or esbuild, read from file manifest to get list of static files with hash — Apr 11th, 2024
  • htmx: Simple bundling with vite — Apr 11th, 2024
  • htmx: Read vite manifest to support hashed values — Apr 11th, 2024
  • htmx: Instantiate manifest object in app, set up watcher in dev mode — Apr 11th, 2024
  • htmx: obfuscate error middleware should only run on JSON responses (although maybe do HTML obfuscation too) — Apr 10th, 2024
  • htmx: Potentially a new extractor that lives alongside FormOrJson that always returns the form — Apr 10th, 2024
  • htmx: Customize / route in pages — Apr 10th, 2024
  • htmx: Tailwind CSS support — Apr 10th, 2024
  • Auth failures need to redirect to login page instead of just returning a 404 error - — Apr 9th, 2024
    • Errors in general need to do that
  • Skip rendering sveltekit files when it is not enabled — Apr 8th, 2024
  • Remove "sync types" and TS generation code when not using sveltekit — Apr 8th, 2024
  • htmx: Page wrapper that gets standard <head> tag contents for all — Apr 8th, 2024
  • htmx: Allow defining pages and actions — Apr 8th, 2024
    • This works similar to the existing endpoints stuff but made for HTML app
    • Autogenerate a rust module for each path and its actions
    • Should this be in a nested directory structure or just by filenames? Maybe nested directories
  • Backend Error reporting — Apr 7th, 2024
    • Almost done, just need to hook up the error reporting interface into FiligreeServerState so that I can report errors outside of the automatic system.
  • Frontend error reporting — Apr 7th, 2024
  • Generate JS code to talk to the API
    • This is done for custom endpoint functions, not yet for CRUD
  • Add configuration to enable trace export — Apr 6th, 2024
  • Add SQL functions to go between an object ID text and UUID forms. — Apr 5th, 2024
    • Should have these left over from Ergo
  • Frontend: Generate client-side validation — Apr 5th, 2024
    • This is mostly done with the generated zod schema, but some things are needed such as checking against create or update schema as appropriate
    • Right now the UI doesn't update right on a client-side validation failure
  • Set it up to work in production — Apr 5th, 2024
    • Dockerfile
    • handleFetch hook
    • Rust side has a fallback route which directs data to the SvelteKit server
      • Most uses will also have a reverse proxy in front but this will let it work without that
  • Command to print all the environment variables that the generated application will check — Apr 2nd, 2024
  • Specify custom endpoints per model — Mar 29th, 2024
    • This generates an endpoint function, a route, path/query/body parameters, input/output/query structures, and corresponding Typescript functions and interfaces to use it.
  • Allow omitting certain normal fields on list — Mar 27th, 2024
    • e.g. if there's a large JSON field not needed for the list page we should omit it.
  • Easy generation of TS types from Rust types — Mar 25th, 2024
  • Background jobs v1 — Mar 23rd, 2024
    • Define background jobs in config
      • Option to set recurrence in config
    • Global and per-job concurrency settings
      • Actually do this via multiple worker configurations. Can configure workers equal to your concurrency desires.
    • Generate a payload structure for the background job
      • No need to define the contents in the config, the user can just fill in what they need
    • Generate a helper to enqueue a background job
    • Generate a stub payload and function that runs the job
    • Hook it all up
      • Add queue to ServerState
      • Make environment variables
  • Option to allow setting an object ID when creating an object — Mar 22nd, 2024
    • This can help with scenarios where we want to create an object and potentially write it to the database later, but have the ID to use beforehand.
  • Generate Typescript interfaces for models — Mar 19th, 2024
  • Generate tests for child object url endpoints — Mar 17th, 2024
  • File model type — Mar 15th, 2024
    • Files are a separate model, and when they need to be referenced by another model they can just form a parent-child relationship.
    • Add a function for upload
      • where should this function go? probably a new file which handles upload, delete, and whatever else might be necessary
    • Add an endpoint to process an upload directly
    • Delete file when model is deleted
    • Delete files when parent model is deleted
    • Special file configuration
      • upload path template
      • a bucket from storage.bucket to store an upload
      • permissions for download or not
    • Extra optional fields
      • hash
      • file size
      • original filename
  • Initial Block Storage Support — Mar 10th, 2024
    • Block storage abstraction
      • use object_store crate for the underlying interface
    • Storage configuration
      • The idea here is that each storage location can be configured but the details should be completely configurable at runtime.
        • Or should it always be environment based? Probably yes actually except maybe some basics like bucket name, endpoint, etc.
      • something like STORAGE_${location_name}_BUCKET and so on for each provider
    • Use storage configuration in template
    • Stream request body into object storage
    • Stream multipart files into object storage
    • Allow examining data as it streams through
    • Stream file from object storage as Response body
  • Create endpoints should return children when doing updatewithparent — Feb 29th, 2024
  • Store .gen files in a tar file instead - Cancelled Feb 28th, 2024
    • This reduces the huge number of files in the state directory, but makes merge conflicts more difficult to deal with.
  • Parent/child model relationships — Feb 27th, 2024
    • Configuration
    • Add fields for belongs_to relationships to model structures
    • Generate composite primary key for joining tables and update queries appropriately
    • handle has_one relationships in fetch queries with ID fetch
    • handle has_one relationships in fetch queries with data fetch
    • handle has_many relationships in fetch queries with ID fetch
    • handle has_many relationships in fetch queries with data fetch
    • Allow populating references fields on fetch when configured
    • Updates for has_one relationships with data payload
    • Updates for has_many relationships with data payload
    • Generate CRUD endpoints for children of a particular object
      • This is basically the same as the normal endpoint, but with the parent ID specified in the URL instead of in the payload.
      • When it's not a many relationship, omit the child ID parameter
    • Populated query functions
      • Call populated functions from endpoints when appropriate
    • SQL Queries for child objects
      • List by parent ID
        • This could be done as just another filter on the list query instead of a new query template
      • insert/update multiple objects
      • Delete by parent ID where ID is not in the set of IDs just inserted/deleted
    • Ensure that each model is not referenced my more than one non-through has relationship
    • Sort CREATE TABLE statements so that parents come first
    • First round of Tests
      • Create a test model with children
        • Post
        • has one Poll
        • has many Comments
        • has many Reactions
      • populated queries with has_one
      • populated queries with has_many
      • insert with child object
      • update with child object
    • create and update functions should return the resulting objects
  • Test passwordless login in Glance — Feb 14th, 2024
  • Test login with password in Glance — Feb 13th, 2024
  • Frontend: Handle validation errors from server — Feb 11th, 2024
  • On validation failure, return the sent data
    • This helps when SvelteKit has forwarded the request straight to the server, since it allows Kit to return the form data back again to repopulate the form
  • Command to bootstrap initial org and user — Feb 11th, 2024
    • This should have the ability to take admin user's email from either the command line or the environment
  • Initial OAuth login support — Feb 10th, 2024
    • Update createnewuser to implement UserCreator trait
    • Allow creating a user without an email address
    • Look for client id/secret environment variables for each provider and create the ones that are found — Feb 3rd, 2024
    • Hook up oauth code to endpoints — Feb 3rd, 2024
    • Link to existing account when it's a new OAuth login but a known email — Feb 2nd, 2024
    • Client-side code for OAuth login — Feb 9th, 2024
  • Login page — Feb 10th, 2024
  • Easier way to add new fields to base models than redefining the whole model — Jan 29th, 2024
  • Integrate into Glance
  • User Creation updates — Jan 27th, 2024
    • Add default role field to org, for users who are added without any set of roles
    • Add roles field to email_invites
  • Validation for form submissions — Jan 22nd, 2024
  • Use hosts list in post-login redirect processing
  • JSON validation that checks multiple fields
    • do this using axum-jsonschema
    • Derive schemars::JsonSchema on every structure that might need to be validated
    • Might want to add additional validation in the future with an extension to this
  • CORS configuration — Jan 17th, 2024
    • Host allow list for apps that want to lock to the app's host
    • Use CorsLayer::permissive() on /api for apps that want to expose the API to browsers
  • Initial SQL migration should only be written once — Jan 17th, 2024
  • New models should be added in new SQL migrations — Jan 17th, 2024
  • Column changes to models should generate new SQL migrations — Jan 17th, 2024
    • sql-migration-sim crate is written, just need diff functionality
    • Add index tracking to sql-migration-sim
  • New runs diff against previous run and only apply diffed changes — Jan 16th, 2024
    • When running the CLI, save the generated templates (post-formatter) to a state directory
    • On each run, diff the generated templates against the previous state in that directory
    • Apply that diff to the actual output file
      • Need a patch style that matches against nearby lines instead of just using line numbers.
    • Point out conflicts as they occur, probably use git style merge conflict markers
    • diffy supports three-way merge so that's also worth looking at
    • This system allows the user to make changes to the generated files but still update the config.
    • Need a command to rewrite a particular file
    • Need a command to just regenerate the state cache, for when changing formatter settings
  • Ability to make signup open to the public or admin-only — Jan 12th, 2024
  • signup endpoint to create a new org that also creates default roles, adds a user, etc. — Jan 12th, 2024
  • ability to sign up via passwordless email auth if enabled — Jan 12th, 2024
  • Password forgot/reset endpoints — Jan 12th, 2024
  • Passwordless email auth — Jan 11th, 2024
  • Email template system — Jan 11th, 2024
  • List all permissions in a file with descriptions — Jan 9th, 2024
  • For models with global permissions, add permissions check in route layer before hitting database — Jan 9th, 2024
  • MVP Basic email sending, enough to do password management and such — Jan 9th, 2024
  • Integrate with email sending services — Jan 9th, 2024
  • Endpoints for retrieving and updating your own user — Jan 7th, 2024
  • Auto-add appropriate dependencies to Cargo.toml — Jan 7th, 2024
  • Tests for users without proper permissions for each action — Jan 6th, 2024
  • Generate tests for models — Jan 6th, 2024
  • Tests for login/logout — Jan 5th, 2024
  • Generate fixtures for tests — Jan 5th, 2024
  • Write functions needed to bootstrap test data — Jan 4th, 2024
    • These should basically be the version that the actual app will use, just deferring all the surrounding functionality that isn't needed to write basic tests
    • Add org
    • Add roles
    • Add user to org with roles/permissions
    • Add new org (using all the above)
    • Add API key for user
  • Password login/logout endpoints — Dec 29th, 2023
  • Get server to start — Dec 27th, 2023
  • Add permissions checks to endpoints — Dec 27th, 2023
  • Add app-specific AuthInfo object — Dec 23rd, 2023
  • Add AuthQueries implementation — Dec 23rd, 2023
  • Select queries need to return the current permission level on each object for project/object-level permissions — Dec 22nd, 2023
  • Add middleware layer to do permissions check — Dec 22nd, 2023
  • Add authz middleware to do generic predicate check — Dec 22nd, 2023
  • query to load user/roles/org for auth info — Dec 21st, 2023
  • Generate router for all model endpoints — Dec 21st, 2023
  • Create endpoints for the model — Dec 20th, 2023
  • Bearer token auth middleware — Dec 20th, 2023
  • Session cookie auth middleware — Dec 20th, 2023
  • Figure out querystring filter parameters — Dec 18th, 2023
  • Pagination — Dec 18th, 2023
  • Rename team to organization
  • Create Rust structures for each model
  • Bootstrap CLI utility
  • Figure out main config format
  • Figure out model definition
  • Create team, user, role, permissions, etc. tables
  • Create SQL migration
  • Create SQL queries to read/write the model

Setup

• This isn't really full documentation., just notes on how I set things up. More of this will be automated and rough edges smoothed out over time.

• Initial Setup

  • Create a git repository
  • Create a Rust project in it
  • Create a SvelteKit project, add filigree-web as a dependency
  • Add the filigree directory with config.toml and other files
  • Create a rustfmt.toml
  • Delete the default src/main.rs
  • Run filigree to write the source code
  • Add this to the vite config

    •   server: {
          proxy: {
            '/api': {
              target: 'http://localhost:7823',
            },
          },
        },
      
      

• Set up Database

  • Create Database and Database User
    • cat /dev/urandom | head | shasum for an easy random password

    • -- Create the user and database
      create role USER login password 'PASSWORD';
      create database THE_DB with owner USER;
      

    • Set DATABASE_URL in your .env
    • cd YOUR_API_DIRECTORY && sqlx migrate run
  • Bootstrap the Database

    • cargo run --release -- db bootstrap \
        --email YOUR_EMAIL \
        --password 'PASSWORD' \
        --name 'YOUR NAME' \
        --org-name 'ORG NAME'
      

    • The only required option is email. Without a password you can log in via OAuth or through a passwordless email login.
    • You can also supply a prehashed password by using the --password-hash argument instead, and hashes can be created by running cargo run --release -- util hash-password YOURPASSWORD.

• Concepts
  • Organization
  • User
    • Users can potentially be in multiple organizations
  • Role
    • Basically a collection of permissions
  • Group
    • This is similar to Role
    • Might not do this, but it can be useful to make this different when more advanced sharing options are created.
  • Actor - all the things (users, roles, groups) that can have permissions on objects
    • When looking at permissions we gather up all the actors that apply to the current user, and use that whole list for permissions checks
• The permissions table is then a list of the permissions each actor has.
• Permissions
  • These should be definable per model once more than one is supported.
  • Tables
    • Global permissions
      • A many-to-many mapping of actor to permission
    • Object permissions
      • A mapping of actor to object and permission
      • For a project-based model, the projects are also just objects
  • Models
    • Role-based Access Control (RBAC)
      • Each role has permissions on whether it can read/write/admin certain types of objects.
      • This is the first one we'll support
      • Might be useful to have two sets of roles, some which can have the global permissions and are only created by administrators, and others which are generic groups of people for managing sharing of projects and objects
        • Need to think about how much this needs to be in the data model vs. just something in the UI. Probably just a single boolean flag is sufficient.
    • Resource-based access control
      • Read/write/admin on individual objects
      • Each role/user has specific permissions for each object
      • More complex, need to make sure the relevant entries get added for each object when it is created
      • Best for a model where objects are not shared by default.
    • Project-based Access Control
      • This combines with the above concepts but is a middle level of granularity compared to the all-or-nothing of plain RBAC and the often-overly-granular resource-based system.
      • A project acts as a container for objects, and actors can have permissions on the project instead of the individual objects.
      • Will probably implement this after RBAC, just need additional functionality for project management and ability to
      • Both individuals and roles can be added to projects
    • Attribute-based Access Control (ABAC)
      • Objects have attributes and actors
    • Public sharing
      • Some sites are built around publicly sharing your stuff.
      • This is basically the same as the resource-based access control model, but with the ability to share with an "anyone" user, and then all the permissions checks check for that actor ID as well.
      • The main change here is that the organization ID checks need to be relaxed when looking for publicly-shared objects.
  • Types of default permissions
    • creator only until shared
    • Creator gets write access, everyone else gets read access
    • everyone can see, only creator can write
    • Everyone in the project can see
    • Everyone in the project can edit

• Types
  • parent-child one to many
  • parent-child one to one
  • Many to one
    • Usually this isn't really a parent-child paradigm so much as related objects. This is actually similar to ActiveRecord belongs_to on the parent object, but without the meaning implied by the word "belongs".
    • e.g. Image uploads can all reference a single conversion configuration, but any image can be switched to a different configuration.
    • The main thing that we want here is the ability to easily fetch the associated objects instead of just their IDs
  • Many to many
    • Almost always best implemented with a join table
• Configuration keywords
  • has - For one-to-one or one-to-many parent-child relationships.
  • has with through for many-to-many relationships
    • In this case through is the name of the model that holds the joins. Making it a full-fledged model allows adding other information to the relationship, if applicable.
    • Need a special mode for through models that will omit the normal id field since its primary key will be a composite of the IDs of the two other models.
  • references for many-to-one relationships
    • This is actually accomplished by the existing references member on a model field definition.
    • Added 2 extra fields that allow setting the option to automatically populate the field on list or get
• Where does the linking field actually go?
  • When there's a linking model, it goes into that model's table, of course.
  • Otherwise for a has relationship it goes into the child object. This is slightly inefficient in a true has_one case but those are rare in practice. In general those turn into either a complex JSON field inside the model itself, or it would be a has_many.

• Add pricing tiers and permissions that go along with those tiers - this should mostly be data driven
  • Need ability for numeric values as well (e.g. max 3 projects, 500MB storage, up to 60 minutes of something, etc.)
• Need ability to do grandfathering of pricing tiers and permissions
  • Essentially this means that both price plans and permissions sets should be versioned, and separately.
  • And there should be the ability to update existing plans and all previous versions

• Workflows are basically state machine definitions which can be instantiated with some data, linked to users/orgs/etc.
• Workflows can perform some set of predefined actions, which may include sending emails, notifying a user, spawning other workflows, etc.
• Workflows can define transitions to run after a certain amount of time
• Consider a scripting engine or something for checking state when transitioning (perhaps just reuse a bunch of code from Ergo for this)

Old Notes

• This should be made modular somehow so it's easy to maintain and add new stuff.
  • Modules define dependencies
• Config
  • formatters to use
  • SQL dialect (postgres or sqlite)
• Model definition
  • Write in TOML
  • Each field has
    • SQL type
    • nullable (default not null)
    • Rust type (can be inferred from the SQL type, but can be overridden)
  • Define Indexes and primary key
    • indexes can maybe just be defined in raw sql
  • Can have child relationships, one -to-one and one-to-many
  • Can have JSON fields, which deserialize to either JSON values or other Rust structs
    • The rust structs can be defined in the config as strings or something like that
  • List of other query functions to generate (find by field, etc.)
  • Permissions
    • Automatic checking on all operations
    • Ability to define which permissions are needed for what
  • some kind of hook on update? or is this better done by just changing the generated code myself
• Consider some kind of codegen for even easier crud endpoint generation
  • Need ability to generate migrations that add/remove columns too
    • this can definitely be manual to start though.
• Authentication
  • Need a user/account mapping to support multiple auth methods per user.
• Scrapped attempt to switch to SeaORM
  • The experience starting to try this ended up with Rework query template system — Jul 16th, 2024
  • Problems
    • exec_with_returning only supports returning a single model, this isn't an issue with raw sqlx
    • find_with_related only supports a single related model, not an issue with the current query building
  • But the nice things were...
    • Easy to add auth query with a wrapper
    • Easy dynamic query construction
    • ActiveModel is pretty nice
    • Easier to decide at runtime which things to populate
    • Templates were easier to deal with, no more gnarly templates with a bunch of ifs generating SQL or functions to call queries.
    • Partial Models were nice