Skip to main content

OrioleDB Source Code Structure

The OrioleDB source code structure comprises various components, including workflows for CI/CD, documentation, tests, and C-source files. It is structured to facilitate development, testing (including under Valgrind), and deployment of the OrioleDB extension for PostgreSQL. A heavy focus is placed on concurrency testing and performance analysis through stop events and CI workflows.

File structure​

orioledb
|- .github
|  |- workflows
|  |  |- check.yml      -- build & test each commit
|  |  |- docker.yml     -- build docker images for each release
|  |  |- dockertest.yml -- tests the docker images
|  |  |- pgindent.yml   -- checks C-source code formatting
|  |  |- rpm.yml        -- build CentOS 7 packages on demand
|  |  |- static.yml     -- runs static analysis
|  |- FUNDING.yml
|
|- ci  -- scripts for building, testing, and CI automation
|- doc -- documentation
|- docker
|  |- tests -- Docker images tests
|  |- Dockerfile -- Alpine based Dockerfile used within docker.yml and dockertest.yml workflows
|  |- Dockerfile.ubuntu -- Ubuntu based Dockerfile used within docker.yml and dockertest.yml workflows
|  |- README.md -- Docker images tests documentation
|  |- docker-entrypoint.sh -- entrypoint file for Docker images
|  |- orioledb-config.sh -- Docker images tests configuration
|  |- postgresql.docker.conf -- configuration file used by Docker images
|- include -- C-headers of extension
|- sql -- installation scripts with definitions of the extension's SQL-level objects
|- src -- C-sources of extension
|- test
|  |- expected -- expected output for regression and isolation tests
|  |- specs -- isolation tests
|  |- sql -- regression tests
|  |- t -- python tests
|  |- orioledb_isolation.conf -- configuration file used during isolation tests
|  |- orioledb_regression.conf -- configuration file used during regression tests
|- LICENSE -- defines PostgreSQL-licence for the project
|- Makefile -- defines make targets
|- README.md -- main documentation entrypoint
|- orioledb.control -- extension control file
|- stopevents.txt -- list of "stop event"
|- stopevents_gen.py -- generates include/utils/stopevents_(defs|data).h from stopevents.txt
|- typedefs_gen.py -- generates list of C-symbols in orioledb.so to orioledb.typedefs
|- valgrind.supp -- suppression rules for valgrind checks

Makefile targets​

All test-related targets accept the VALGRIND=1 argument, which runs the tests under valgrind. Valgrind makes tests about ~100 times slower but catches uninitialized memory access. Another positive side-effect of Valgrind's super-slow test runs is the altered timing, which can expose various types of concurrency errors.

  • regresscheck -- run SQL-tests (see below).
  • isolationcheck -- run isolation tests (see below).
  • testgrescheck -- run testgres tests (see below).
  • testgrescheck_part_1 -- first half of testgres checks. Testgres tests are split into two nearly equal halves to avoid overly long individual CI runs.
  • testgrescheck_part_2 -- second half of testgres checks.
  • installcheck -- run all types of tests when installed using the PostgreSQL extension system (USE_PGXS=1).
  • check -- run all types of tests when installed from the contrib folder of the PostgreSQL source code.
  • pgindent -- automatically indents OrioleDB sources. The pgindent tool should be available in $PATH. Note that you need to install pg_bsd_indent first. Also, you need GNU Objdump available in $PATH as objdump or gobjdump, or specified via the OBJDUMP environment variable.

Extension SQL scripts​

OrioleDB extension files are organized using a base version installation script, followed by upgrade scripts for respective minor versions. For instance, when installing version 1.7, PostgreSQL first applies the 1.0 script and then sequentially runs all upgrade scripts from 1.0 up to 1.7.

To bump the extension version (for example, bumping from 1.6 to 1.7), follow these steps:

  1. Edit the default_version parameter in the orioledb.control file to match the new version.
  2. Manually create development and production upgrade scripts in the sql directory:
    • ./sql/orioledb--1.6--1.7_prod.sql
    • ./sql/orioledb--1.6--1.7_dev.sql
  3. Add the corresponding headers to the scripts and populate the changes:
    • Changes intended only for the development environment (such as test functions that should not be exposed on production systems) go into _dev.sql.
    • All standard changes go into the _prod.sql file.
note

The build system (make) utilizes both files to automatically generate the final ./sql/orioledb--1.6--1.7.sql script. Therefore, you do not need to create this final file manually. The contents of _dev.sql will only be included when the IS_DEV flag is specified during the build. There is no need to duplicate changes between the two files.

  1. Add the generated file (./sql/orioledb--1.6--1.7.sql) to both .gitignore and .dockerignore.

Tests​

OrioleDB has 3 groups of tests described below.

  • SQL tests are located in the test/sql folder. These are the simplest types of tests. The SQL file is passed to psql and the result is compared to the reference output in the test/expected folder. Note that there might be multiple reference outputs for one input file. For instance, collate.sql contains collation-aware tests. The result may match collate.out, collate_1.out, or collate_2.out depending on database encoding and the presence of libicu.

  • Isolation tests are located in the specs folder. These tests simulate multiple connections running simultaneously. See the README in the PostgreSQL source tree. These tests are especially powerful in conjunction with stop events.

  • Python testgres tests are located in t. These are the most powerful and complex tests. Additionally to the ability to simulate multiple simultaneous connections, they can perform actions with the whole PostgreSQL instance such as start, stop, backup, replication, etc. See the testgres docs for details.

note

For proper integration into the test suite, the test file names should end with _test.py.

CI​

OrioleDB uses GitHub CI. The CI workflows are described below.

check.yml​

This workflow runs the following tests for each of the two compilers (gcc and clang) and each of the supported PostgreSQL major versions (16 and 17).

  • normal -- run tests without asserts and without debug symbols.
  • debug -- run tests with asserts and with debug symbols.
  • sanitize -- run tests with asserts, with debug symbols, with alignment, and other sanitizers. This replaces running tests on strict alignment architectures, providing even somewhat stricter checks (for instance, it traps you on accessing a properly-aligned member of an improperly aligned structure, which real hardware wouldn't do).
  • check_page -- runs tests with asserts, with debug symbols, and with the CHECK_PAGE_STRUCT macro enabled. This macro provides the page structure check on every page unlock.
  • valgrind_1 -- runs regresscheck, isolationcheck, and testgrescheck_part_1 under valgrind with asserts and with debug symbols.
  • valgrind_2 -- runs testgrescheck_part_2 under Valgrind with asserts and with debug symbols.
  • static -- runs clang-analyzer or cppcheck over sources.

docker.yml​

This workflow builds docker images for amd64 and arm64v8 architectures under Alpine and Ubuntu Linux. This Dockerfile is a slightly adjusted PostgreSQL Dockerfile. See our dockerhub for details.

dockertest.yml​

This workflow tests the Docker images. It runs on every push and pull request.

pgindent.yml​

This workflow checks the code formatting using pgindent. It runs on every push and pull request.

rpm.yml​

This workflow builds RPM packages for CentOS 7 using the specification from the orioledb/pgrpms repository, a fork of the pgrpms repository.

static.yml​

This workflow runs static analysis on the code. It runs on every push and pull request.

Stop events​

Stop events are special places in the code where execution can be paused based on specific conditions. Stop events are used for the reliable reproduction of concurrency issues. OrioleDB isolation and testgres tests use stop events.

A stop event exposes a set of parameters, encapsulated into a jsonb value. The conditions on stop event parameters are defined using the SQL/JSON path language.

The SQL-level functions and variables for stop events manipulation are listed below.

  • orioledb.enable_stopevents -- enables stop events checking for the process. Stop events checking is expensive and significantly affects performance. This is why stop events are disabled by default.

  • orioledb.trace_stopevents -- enables logging of all stop events. Disabled by default.

  • pg_stopevent_set(eventname text, condition jsonpath) RETURNS void -- set the condition for the stop event. Once the function is executed, all processes that run a given stop event with parameters satisfying the given jsonpath condition will be stopped.

  • pg_stopevent_reset(eventname text) RETURNS bool -- reset the stop event. All processes previously stopped on the given stop event will continue execution.

  • pg_stopevents(OUT stopevent text, OUT condition jsonpath, OUT waiter_pids int[]) RETURNS SETOF record -- returns all the stop events currently set with their conditions and waiter process PIDs.

At the C level, the following macros are provided for managing stop events.

  • STOPEVENTS_ENABLED() -- checks if stop events are enabled.
  • STOPEVENT(event_id, params) -- raises the given stop event with the given jsonb parameters.
  • STOPEVENT_CONDITION(event_id, params) -- checks the stop event condition without stopping the execution. Used for error simulation.

The list of stop events is defined in stopevents.txt file. The stopevents_gen.py script generates include/utils/stopevents_defs.h (macros) and include/utils/stopevents_data.h (name strings) files with C definitions of the stop events list.