HYTRADBOI Conference

Incremental Execution

• Using incremental execution for realtime machine learning
  • They were looking at storing unstructured data in the database and extracting data from it using a model.
  • The data ends up in separate columns in the same row as the unstructured data, with confidence values for each one.
  • Their UI then lets you do corrections which feed back into the model and immediately recalculate values from it.
  • This then feeds into incremental materialized views that update only the parts which need updating.
• Incremental stream processing with Differential Datalog.
  • https://github.com/vmware/differential-datalog
  • VMWare was running queries to check consistency constraints within their networks.
  • For better performance they wanted to have queries that update incrementally, on only the data that has changed.
  • On top of Datalog, they added more data types such as structs and collections, and fragments of imperative code as well.
  • They’re compiling DDLog to use differential dataflow for the compute engine, and are now working on their own compute engine.

• GUI vs SQL
• No need for extra join table for many-to-many relationships. It's handled in the background
  • probably by all relationships being many to many internally? Or just array fields maybe, but then the hiding ofthe implementation is the point.
• The nice thing about Airtable is that the GUI primitives turn into a cheap business logic without much effort and no real GUI-building required.
• Unlike a spreadsheet, you're still using database concepts such as actual relations between rows in tables and uniqueness constraints, which makes things more scalable and prevents the all-too-familiar gigantic unmaintainable Excel spreadsheet.
• You can add triggers to perform certain tasks in the database when changing certain pieces of data.
  • The trigger scripts can be actual Javascript with access to a full API.

• This is a general purpose database application, but with better priimitives for building functionality on top of it than the normal database tool.
• Can show data for joins on each row.