For a long time now, PostgreSQL has had an extensible Index Access Method API (called AM), which has stood the test of time and enabled numerous robust extensions to provide their own index types. For example: rum, pgvector, bloom, zombodb and others. PostgreSQL 12 introduced the Table AM API, promising equivalent flexibility for table access methods.

Despite PostgreSQL's Table AM API being available since version 12 and ongoing criticisms of its built-in storage engine — particularly the MVCC model ([1], [2], [3]) — it remains surprising that no fully featured transactional storage engine has yet emerged purely as an extension.

Since the table AM and index AM APIs are tightly coupled, this is an issue for both implementations.

The features most in demand for alternative PostgreSQL table engines are:

1. Alternative MVCC implementations, e.g.., UNDO-log-based storages. The motivation to provide this is well-discussed in Uber blog post, Andy Pavlo blog post, and many other sources.
2. Non-heap-like storages. For example, in index-organized tables, the index is not an optional addition to the table that speeds up requests but a necessary layer the table storage uses internally. Consequently, the table tuple is a part of a complex data structure and can’t be addressed by a fixed-length address like page number and offset number. It requires to be addressed by variable length identifier like index key.

The API extension to provide #2 is more or less understandable. This could be done by replacing ctid with an arbitrary sequence of bytes in all the APIs. However, #1 seems rather complex and requires much clarification.

As an example that illustrates the motivation for the changes to the table and index AM API comes OrioleDB. It is an extension providing a table access method we developed to address many well-known shortcomings of the built-in storage engine. However, OrioleDB is not yet a drop-in extension; it requires several patches to the PostgreSQL Core.

Besides these two things, which will be discussed below, there are numerous needs for further API improvements, like pointer squizzling and alternative WAL-logging, that are outside the scope of this post.

OrioleDB is a storage extension for PostgreSQL which uses PostgreSQL's pluggable storage system. Designed as a drop-in replacement for PostgreSQL's existing Heap storage, OrioleDB aims to overcome scalability bottlenecks and fully utilize modern hardware capabilities. By integrating seamlessly with PostgreSQL, it offers improved performance, efficiency, and scalability without sacrificing the robustness and reliability that PostgreSQL is known for.

Today we’re releasing OrioleDB version beta7. This marks a significant step in delivering a high-performance, next-generation storage engine for Postgres users. OrioleDB is designed to extract the full potential of modern hardware, offering superior performance, efficiency, and scalability.

OrioleDB design choices​

OrioleDB is built from the ground up to leverage modern hardware, reduce maintenance needs, and enhance distributed capabilities. The key technical decisions forming the foundation of OrioleDB are:

1. Elimination of Buffer Mapping and Lock-less Page Reading: In OrioleDB, in-memory pages are directly linked to storage pages, eliminating the need for buffer mapping and its associated bottlenecks. Additionally, in-memory page reading is performed without atomic operations, enabling lock-less access. Together, these design choices significantly elevate vertical scalability for PostgreSQL.
2. MVCC Based on the UNDO Log Concept: OrioleDB employs a Multi-Version Concurrency Control (MVCC) mechanism based on an undo log. Old versions of tuples are evicted into undo logs—forming undo chains—rather than causing bloat in the main storage system. Page-level undo records allow the system to promptly reclaim space occupied by deleted tuples. Combined with page merging, these mechanisms eliminate bloat in most cases. As a result, dedicated vacuuming of tables is unnecessary, removing a common cause of system performance degradation and database outages.
3. Copy-on-Write Checkpoints and Row-Level WAL: OrioleDB utilizes copy-on-write checkpoints to provide structurally consistent snapshots of data at all times. This approach is friendly to modern SSDs and enables row-level Write-Ahead Logging (WAL). Row-level WAL is easy to parallelize (already implemented), compact, and suitable for active-active multi-master configurations (planned).

Benchmarking OrioleDB vs PostgreSQL Heap​

To illustrate the performance characteristics of OrioleDB we used TPC-C benchmarking, a complex test simulating real database workloads that is considered a modern standard in database applications.

PostgreSQL, a powerful open-source object-relational database system, has been lauded for its robustness, functionality, and flexibility. However, it is not without its challenges – one of which is the notorious VACUUM process. However, the dawn of a new era is upon us with OrioleDB, a novel engine designed for PostgreSQL that promises to eliminate the need for the resource-consuming VACUUM.

Long story short, OrioleDB alpha version was released more than year ago. More than 200 bugs were fixed since then. Now, OrioleDB reached beta stage. That means we recommend OrioleDB for pre-production testing. The most interesting workloads for testing could include: high transaction throughput, high volume of updates, high volume of in-memory operations, lock bottlenecks and other extreme cases.

Postgres is great… until it's not... In the past few years, there have been a few major public outages of a cloud service, like MailChimp's Mandril, Joyent's Manta Storage, and Sentry's monitoring service. Invariably, the outages affect applications that use Postgres in write-heavy workloads.

Traditional database engines were designed in the '80s and early '90s. At that time CPUs were much slower than they are today. Even worse was storage, hard drive head positioning time was enormous¹. And CPU (or, at most, a few single-core CPUs) was assumed to be infinitely fast in comparison to IOPS. Therefore, systems were designed to save IOPS as much as possible, while CPU overhead was considered a secondary optimization target.