New Product Feature: Introducing Persistence Engine

When it comes to making business decisions based on data, time is of the essence. In order for data to be valuable in decision-making, it needs to be available when it counts. However, when data sets are heavy and when you’re creating an increasing number of models, load times for reports and dashboards can begin to lag and hold you back. This happens because your warehouse is under too much pressure and is over-performing at query time. If limits are reached within your warehouse, some queries may not even work anymore.

This is where Whaly’s Persistence Engine comes in. This new feature speeds things up by materializing models as views or tables, depending on the configuration for each model. Whaly’s Persistence Engine will compute the lineage of your Models, and depending on the materialization configuration for each model, it will write down its results in your warehouse. Future queries are then done on top of these materialization results, making load times faster.

Another benefit of our Persistence Engine is that it allows you to reuse a Whaly model in an alternative tool, such as another visualization tool or in a reverse ETL. This is particularly useful for businesses that have invested in multiple tools and want to use the same model in different applications.

Whaly’s Persistence Engine can materialize each model in three ways: 1) No materialization, 2) Materialization as a view, and 3) Materialization as a table.

• No materialization means that the model will remain "virtual" and won't be written to your warehouse.
• Materialization as a view will persist only the SQL query of the model in your warehouse. Each future read will still execute the SQL query. While this won’t save you compute costs, it will expose the models in your warehouse, which means that each query will always return the most up-to-date results.
• Materialization as a table, on the other hand, will persist the results of the SQL query in your warehouse. Future reads won't execute the model SQL again, but only read the previous result. This is the best cost-saving option since compute costs can be reduced, however, the downside is that Data Consumers won’t have access to the most up-to-date data — only the data of the last materialization job run. This should be used cautiously since data consumers may not know when data is stale, and it could lead to reporting issues and bad decisions.

We’re excited about our new Persistence Engine feature since it will speed up your load times, and will allow you to use Whaly models in alternate tools. With its three materialization options, it provides businesses with the flexibility they need to manage their data effectively while prioritizing speed. Try it out today!

👉 Check out our Persistence Engine documentation here

👉 Start your 14-day free trial of Whaly here. If you already use Whaly, write us in the Whalers slack group to get this new feature!