Introducing ReMemory

The solution to your re-rendering woes.

Introduction

Those familiar with the C3kit Bucket library have come to love the c3kit.bucket.memory implementation. It allows us to easily mock out our database when writing tests on the backend simply by switching implementations, and on the front end we can use a Reagent atom for the database store and our components will re-render automagically whenever data in the database changes. It also means we are using the same db/find and db/tx syntax whether we are interacting with the actual backend database or the front end state.

While this Reagent usage is convenient, it's not without its share of problems. The fact that the database store is a Reagent atom means that if anything changes in the database, it will cause all components that are doing a db/find to re-render, even if they are querying for data unrelated to the data that was modified. This means that something as simple as opening a modal could cause every component on the page to re-render!

But what if it was smarter? What if the components that were querying for :user entities only re-rendered if the :user entities were changed, and ignore any other updates to the database store? Even better, what if the components could listen only to the specific attributes of the entities they needed to render and ignored all other information in the database store?

With that goal in mind, we are excited to announce a new implementation of C3kit Bucket for Reagent front ends: ReMemory.

Why Make a Whole New Implementation?

In order to fully understand the need for a Reagent-specific implementation, let's dive a little deeper into the complication of using the current memory implementation with Reagent.

As mentioned above, the current implementation makes it so all components re-render if anything in the database changes. For example, a component querying for :user entities will re-render if an :order entity is modified, even if the component has no connection with the :order entity that was modified. This is because doing a db/find is essentially the same thing as deref'ing the Reagent atom that is the database store. Therefore, every interaction with the front end database has unwanted side effects that we now have to try to mitigate in the way we write components. The most common tactic to try to mitigate this is to wrap the db/find call in a reagent/track, but this approach has its own set of problems.

Why Not Use a Track?

Consider the following example:

(defn my-component []
  (r/with-let [orders (r/track db/find-by :order :user @user/id)]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

Here, @user/id is the logged in user's ID stored in the page state. However, the user/id deref-able is not what is being passed into the db/find-by function, just the deref'ed value at the time the component mounted. That means if the logged in user changes, that track is not going to re-run and the component will not re-render. The deref needs to happen inside of the function passed to the track. We could "fix" this by using an anonymous function:

(defn my-component []
  (r/with-let [orders (r/track #(db/find-by :order :user @user/id))]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

But this is an incorrect use of tracks. Part of the benefit of tracks is that they actually track which functions are called with which arguments. If multiple components are calling the same function with the same arguments, Reagent knows to run the calculation once and then re-render all necessary components. With an anonymous function, this performance optimization is lost and the calculation will happen n times for each component doing the lookup.

Even still, the tracks are just a Band-Aid fix. Every time anything in the database store changes, the tracks will re-calculate because the Reagent atom has been modified. If the return value of the query doesn't change, the re-render doesn't happen, but the underlying calculation that would have caused the re-render to happen still occurred. This means that when we open React Dev Tools we may no longer see our components re-rendering, but our app hasn't gotten any faster. The tracks have simply hidden the side-effect of the calculation (the re-render) without actually stopping the unnecessary calculation. This is almost worse - it leaves us with a front end that looks like it has good state management because components aren't re-rendering, but is still doing all of the calculation involved in re-rendering all the components. The fact that the re-renders are no longer observable makes the performance issue even harder to debug.

Re-renders are side effects of state calculations. If components are re-rendering when they shouldn't be re-rendering, it's because there's a calculation that's happening that shouldn't be happening. We need to stop the calculation, not its side effects.

Good tools should improve our applications while also making code easier to write, not introduce side effects and performance problems that we need to actively mitigate. We should simply be able to to tell our components to look up specific information and then that component, without any extra song and dance, only re-renders if that specific information changes. And that is exactly what ReMemory does.

The Bucket API - in ReMemory

All functions from c3kit.bucket.api are supported by ReMemory with the same syntax; the only difference is the re-render behavior of the components that use them. And the best part is, there's no need for tracks, reagent/with-let forms, or inner function in our components. Let's take a look at each function and its usage.

entity

(defn my-component []
  (let [order (db/entity 123456)]
    [:div (:date order)
      [:div "Your order has " (count (:items order)) " items"]]))

This component will only re-render if that one specific entity with that ID is modified and will ignore changes to all other entities.

find

(defn my-component []
  (let [orders (db/find :order)]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

This component will only re-render if any entities of kind :order are modified and will ignore any changes to entities of any other kind.

find-by

(defn my-component []
  (let [orders (db/find-by :order :id [12345 67890 54321] :user @user/id)]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

The behavior of find-by is the same as find, with one useful exception: when querying by an ID or a list of IDs, it will work like entity, i.e. the re-renders will be scoped to edits to those specific entities and will ignore all others. In the example above, the component will re-render if any one of those three entities are modified but will ignore changes to all other entities, even of the same kind.

Since we aren't using tracks and with-let forms or inner functions, if the value of user/id changes, the component will re-render and the query will update accordingly.

ffind and count

These functions will yield the same re-rendering behavior as find.

ffind-by and count-by

These functions will yield the same re-rendering behavior as find-by.

tx and tx*

These functions continue to work as expected. Remember that it's the querying of data that will cause re-renders; if no components are querying the front end database, transacting data into it won't cause any re-renders.

ReMemory-Specific Additions (Our Secret Sauce)

While the new implementation already yields huge performance benefits, we wanted to go even further. In many cases, components only actually use a handful of all the attributes on an entity. For example, a user entity may have a whole slew of fields relating to social media links, email addresses, etc., but perhaps the component being rendered only needs their name. What if we could make our components only re-render if specific attributes of the returned entities were modified and ignore changes to all the attributes our component doesn't need? For that, we've provided select-find-by and its complementary functions.

select-find-by

(require '[c3kit.bucket.re-memory :as re])

(defn my-component []
  (let [orders (re/select-find-by :order :user 1423 :store 5431)]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

This component will only re-render if the :user attribute or :store attribute on any :order entities is modified. If given a list of IDs, the re-renders will be further scoped to those specific :order entities rather than all of them.

If we would like to select more attributes than those being queried, we can provide a sequence of keywords between the kind and the options:

(require '[c3kit.bucket.re-memory :as re])

(defn my-component []
  (let [orders (re/select-find-by :order [:products] :user 1423 :store 5431)]
    (ccc/for-all [order @orders]
      [:div (:date order)
        [:div "Your order has " (count (:items order)) " items"]])))

This may feel syntactically familiar, as it is inspired by clojure.core's select-keys function, and that is the reason why the function has been named select-find-by. Also similar to select-keys, select-find-by will only return the attributes selected and queried, along with the kind and ID. For example:

(re/select-find-by :order [:products] :user 1423 :store 5431)
=> ({:kind :order :id 8071 :user 1423 :store 5431 :products [8765 9735]})

(re/select-find-by :order :user 1423 :store 5431)
=> ({:kind :order :id 8071 :user 1423 :store 5431})

This is important to remember, because db/tx and db/tx* will delete the values of any attributes that are not present. That means if we use select-find-by to query two attributes of an entity that has ten attributes and then db/tx that entity, eight of those attributes' values will be cleared. To rectify this, we've provided select-tx and select-tx*, which will reload the full entity when transacting to ensure data integrity.

Along with select-find-by, ReMemory also provides select-find, select-ffind-by, select-count, and select-count-by.

IMPORTANT

select-find-by will NOT return all attributes of the entity, so using db/tx will lead to data loss! Only use select-tx or select-tx* when transacting an entity that was queried with select-find-by. Of course, if your components don't need that data, it's not really a problem, as you aren't changing any data in the actual backend database.

Syntactic Sugar

There may be times where we want to select all the attributes of an entity except one or two. In this case, it would be very cumbersome to have to manually select all of the desired attributes. To address this, we've provided a dissoc option:

(re/select-find-by :order ['dissoc :products] :user 1423 :store 5431)

Getting Started with ReMemory

To start using ReMemory, simply set {:impl :re-memory} in your Bucket configuration and ensure that your application is pulling in Reagent as a dependency. It will create the store as a Reagent atom automatically, so it's not necessary to include {:store (r/atom {})} in the Bucket configuration.

ReMemory does NOT provide the Reagent dependency. If your application is using c3kit.wire, the Reagent dependency will come in with Wire, otherwise you will need to include Reagent in your deps.edn or project.clj.

At the time of writing, c3kit.bucket.re-memory is tested with Reagent v1.3.0 and the current version of c3kit.bucket is v2.9.0. ReMemory in its current form was released with Bucket v2.70.

All of the select- functions provided by ReMemory can be used a la carte by importing the c3kit.bucket.re-memory namespace even if your Bucket implementation is set to :memory, provided that your database store is a Reagent atom.

A Note About Testing

With ReMemory, there will be a lot less components re-rendering than with the standard memory implementation. This may mean that there are some failing tests when migrating to ReMemory because the data being modified in the test isn't the proper data to cause the component being tested to re-render. Tests failing when switching implementations does not necessarily indicate a problem with ReMemory, and if you get all of your tests passing with ReMemory, they will still be passing with the old memory implementation.

Conclusion

We hope you're as excited about this new implementation as we are. When Connor Kilgore and I first presented this to the rest of the team at Clean Coders, we hot-swapped one of our internal applications from the memory implementation to ReMemory and witnessed an immediate 6x speed increase in scripting and re-rendering time in a particularly bogged-down part of the application. The best part is, we didn't have to re-write any of the front end code or pull a new dependency like ReFrame into the project to achieve this.

c3kit.bucket is an open-source repository, and pull requests are welcome. Please let us know your experience, and happy coding!