Monday, 23 June 2025

Partition in AWS Aurora PostgreSQL


When your PostgreSQL tables start growing into the millions (or even billions) of rows, performance can take a hit — especially for read-heavy or write-heavy workloads. That’s where partitioningcomes in. It’s a technique that breaks a big table into smaller chunks (called partitions) based on time or ID, so queries run faster and maintenance becomes much easier.

If you're using Amazon Aurora PostgreSQL, you can supercharge this with the pg_partman extension. It automates pretty much everything: creating new partitions as data comes in, managing indexes, and even cleaning up old data with retention rules. It’s a great fit for use cases like logging, audit trails, analytics, and time-series data — basically anything that grows over time and can be logically split.

But automation doesn’t stop there.

Pair pg_partman with pg_cron, another powerful extension available in Aurora, and you can schedule routine maintenance jobs directly inside the database. For example, you can schedule pg_partman.run_maintenance() to run every hour or day, so new partitions are created and old ones are cleaned up — all without any manual effort or external scripts.

Why this combo is so useful:

  • pg_partman keeps your tables partitioned, indexed, and tidy.

  • pg_cron makes sure those maintenance tasks happen regularly and automatically.

  • Together, they reduce manual DBA work and help your database scale smoothly.

So if you're working with large or growing datasets on Aurora, setting up pg_partman with pg_cron is a no-brainer. It’s like setting your database on autopilot.

PostgreSQL pg_partman


Used to automate the creation and maintenance of table partitions.

We can configure pg_partman with the following settings:


Table to be partitioned

  • Partition type
  • Partition key
  • Partition granularity
  • Partition pre-creation and management options


A partitioned table can be registered to pg_partman by calling a function `create_parent`. This creates necessary partitions based on the input given by us


Enabling the pg_partman extension


The below command has to be executed on each database where we need to partition tables.


CREATE SCHEMA partman;

CREATE EXTENSION pg_partman WITH SCHEMA partman;



 


Note: We need superuser privilege to run the above command. No DB restart is needed.


Configuring partitions using the create_parent function


By executing the function create_parent we can add partitions to the table.


SELECT partman.create_parent( p_parent_table => 'public.job_logs',

 p_control => 'created_at',

 p_type => 'native',

 p_interval=> 'daily',

 p_premake => 10);


p_parent_table – The parent partitioned table. This table must already exist and be fully qualified, including the schema.

p_control – The column on which the partitioning is to be based. The data type must be an integer or time-based.

p_type – The type is either 'native' or 'partman'. We typically use the native type for its performance improvements and flexibility. The partman type relies on inheritance.

p_interval – The time interval or integer range for each partition. Example values include daily, hourly, and so on.

p_premake – The number of partitions to create in advance to support new inserts.


Example


The below example illustrates how to add a partition for a table


CREATE TABLE job_logs

  (

     event_id        BIGSERIAL,

     operation       CHAR(1),

     value           FLOAT(24),

     parent_event_id BIGINT,

     event_type      VARCHAR(25),

     created_at      TIMESTAMP,

     CONSTRAINT pk_job_logs_event PRIMARY KEY (event_id, created_at)

  ) partition BY range (created_at); 

CREATE INDEX idx_event_type ON  job_logs(event_type);







To create partitions, this will create partitions based on create date and the interval will be for 1 day, This means for each day we will have a partitioned table.


SELECT partman.create_parent(

p_parent_table => 'public.job_logs', 

p_control => 'created_at', 

p_type => 'native', 

p_interval=> 'quarter-hour', 

p_premake => 2);






In the above, you can see 6 partitions have been created, as we chose to premake as 2 it has created 2 future partitions from the current time. Also, we could see a new partition table job_logs_default, this table will store all the data which are out of the partitioned data.



Info-message


Note: When first running create_parent() to create a partition set, intervals less than a day round down when determining what the first partition to create will be. Intervals less than 24 hours but greater than 1 minute use the nearest hour rounded down. Intervals less than 1 minute use the nearest minute rounded down. However, enough partitions will be made to support up to what the real current time is. This means that when create_parent() is run, more previous partitions may be made than expected and all future partitions may not be made. The first run of run_maintenance() will fix the missing future partitions. This happens due to the nature of being able to support custom time intervals. Any intervals greater than or equal to 24 hours should set things up as would be expected.



Testing by adding data to the partition,






Configuring partition maintenance using the run_maintenance_proc function


We can run partition maintenance operations to automatically create new partitions, detach partitions, or remove old partitions. Partition maintenance relies on the run_maintenance_proc function of the pg_partman extension .


We need to make some changes in the existing partition config based on our needs.


UPDATE partman.part_config 

SET infinite_time_partitions = true,

    retention = '15 minutes', 

    retention_keep_table=false,

    retention_keep_index=false

WHERE parent_table = 'public.job_logs';


With the above query, we will update the existing info which was created using function partman.create_parent


Explanation for above query,


infinite_time_partitions = true, – Configures the table to be able to automatically create new partitions without any limit.

retention = '15 minutes', – Configures the table to have a maximum retention of 15 minutes.

retention_keep_table = false – Configures the table so that when the retention period is due, the tables are dropped. While setting it to true, the table isn't deleted automatically. Instead, partitions that are older than the retention period are only detached from the parent table.

retention_keep_index = false - Same like retention_keep_table where index of the partitioned tables are dropped.When set to true, pg_partman retains the index names of dropped partitions in its internal tracking tables.






Execute the below command from the database.


select partman.run_maintenance(p_parent_table => 'public.job_logs');





From the above o/p we can see it has created new partitions (job_logs_p2025_06_23_1230,job_logs_p2025_06_23_1245,job_logs_p2025_06_23_1300) with our limit of 2 new partitions from the current time and removed the older partitions.



How to remove a table from a partition

In order to remove a table from being partitioned, we should remove the table details from the partman schema tables.


delete from partman.part_config;

drop table partman.template_public_job_logs;



Pg_partman can help in managing the partitions, but if we need to automate it we can achieve this by using pg_cron extension on our database.


PostgreSQL pg_cron


PostgreSQL pg_cron extension to schedule maintenance commands within a PostgreSQL database.


Setting up the pg_cron extension


Add pg_cron to the shared_preload_libraries parameter value.

DB restart is needed for the change to take place

Execute the below command on the database, default connect to Postgres database.


CREATE EXTENSION pg_cron;


Use the following command to check the value of the cron.log_run parameter.


postgres=> SHOW cron.log_run;


Scheduling pg_cron jobs


Partition maintenance relies on the run_maintenance_proc function of the pg_partman extension and the pg_cron extension, which initiates an internal scheduler. The pg_cron scheduler automatically executes SQL statements, functions, and procedures defined in your databases.


To add a partition configuration under pg_cron


INSERT INTO cron.job (schedule, command, nodename, nodeport, database, username, active,jobname)

VALUES (

    '*/15 * * * *',

    'SELECT partman.run_maintenance(p_parent_table => ''public.job_logs'');',

    'localhost',

    5432,

    'btest',

    'root',

    true,

    'job_logs_partition'

);


To disable a cron


Update the job ID with a basic SQL command;


update from cron.job set active = false where jobid=107; 


To remove a cron


Delete the job ID with a basic SQL command;


delete from cron.job where jobid=107;

 


How to monitor the pg_cron and partitions


The table cron.job_run_details contains historical information about past scheduled jobs that ran. This is useful to investigate the status, return messages, and start and end times of the job that ran. 

Sunday, 28 May 2023

PostgreSQL Autovacuum: Maintaining Database Performance

In the world of relational databases, PostgreSQL stands out as a powerful and feature-rich open-source option. One critical aspect of database management is the maintenance of data storage to ensure optimal performance. PostgreSQL's autovacuum feature plays a vital role in this process, automatically managing table bloat and ensuring efficient use of disk space. In this blog post, we will explore PostgreSQL autovacuum in depth, discussing its purpose, configuration options, and best practices for maintaining a healthy database.



What is Autovacuum?

Autovacuum is a feature in PostgreSQL that automatically frees up space occupied by deleted or outdated data within tables. When rows are deleted or updated, the old versions of the rows remain in the database until they are vacuumed. Over time, this can lead to table bloat, resulting in degraded performance, increased disk usage, and slower queries. Autovacuum solves this problem by automatically initiating the vacuuming process, reclaiming space, and updating database statistics to optimize query planning.


Understanding Autovacuum Parameters:

To configure autovacuum behavior, PostgreSQL provides us several parameters that can be adjusted according to specific database requirements. Some important parameters include:


  1. autovacuum_vacuum_scale_factor: This parameter defines the threshold at which autovacuum should start working on a table. When the number of updated, inserted, or deleted tuples exceeds this threshold, autovacuum kicks in to reclaim space.

  2. autovacuum_analyze_scale_factor: This parameter determines when autovacuum should analyze a table's statistics. Analyzing statistics helps the query planner make better decisions when generating query execution plans.

  3. autovacuum_vacuum_cost_limit and autovacuum_vacuum_cost_delay: These parameters control the speed and resource consumption of the autovacuum process. By tweaking these values, you can adjust the trade-off between maintenance and query performance.

Best Practices for Autovacuum: To ensure efficient autovacuum operations and maintain a healthy database, consider the following best practices:

Monitor and Tune Autovacuum Configuration: Regularly monitor autovacuum activity, including the number of tuples and pages processed, as well as the time taken for vacuuming. Adjust the configuration parameters to strike a balance between resource utilization and maintenance requirements. Periodic Manual Vacuuming: Although autovacuum handles most vacuuming tasks automatically, there may be situations where manual intervention is necessary. Schedule periodic manual vacuums for heavily updated or deleted tables to complement the autovacuum process. Analyze Regularly: Ensure that the autovacuum_analyze_scale_factor parameter is appropriately set to trigger regular table statistics analysis. Accurate statistics are crucial for efficient query planning and optimization. Separate Busy Tables: Consider placing heavily updated or deleted tables on separate tablespaces to isolate their vacuuming activity. This separation can prevent excessive contention and improve overall performance. Monitor Disk Space: Keep a close eye on disk space usage, as autovacuum requires free disk space to operate effectively. Running out of disk space can lead to disrupted autovacuum operations and potential database issues.

PostgreSQL's autovacuum feature is a powerful mechanism for maintaining optimal performance in a relational database. By automatically reclaiming space and updating statistics, autovacuum helps prevent table bloat, ensuring efficient use of disk space and improved query execution. By understanding and fine-tuning autovacuum parameters, and following best practices, database administrators can maintain a healthy PostgreSQL database and provide a smooth user experience for their applications.

Monday, 2 August 2021

PostgreSQL 9.0.x source installation

In this blog i will cover how to do a source installation PostgreSQL lower (outdated) version.

This will be helpful when you want to test your setup other than prod. 


We have all source packages available here, https://www.postgresql.org/ftp/source/


PostgreSQL Version : 9.0.23

OS : Opensuse Leap 15.3


Installing python package (make sure you have gcc devel packages installed)


zypper install python-base



Download uuid and extract it  (for source installation we need )


wget https://src.fedoraproject.org/repo/pkgs/uuid/uuid-1.6.2.tar.gz/5db0d43a9022a6ebbbc25337ae28942f/uuid-1.6.2.tar.gz

tar -xf uuid-1.6.2.tar.gz


Configure and install it 


cd uuid-1.6.2/

./configure

make -j20

make install


Download source package and extract it 


wget https://ftp.postgresql.org/pub/source/v9.0.23/postgresql-9.0.23.tar.bz2

tar -xf postgresql-9.0.23.tar.bz2


Configure and install it 


cd postgresql-9.0.23/

./configure --with-python

make -j20

make install

cd contrib

make all

make install

export PATH=/usr/local/pgsql/bin:$PATH



During configure if you get below readline error please follow the fix given below 


Expected error

 


Fix (need to install readline-devel)


zypper install readline-devel



Create the user and required folders


groupadd postgres  

useradd -G postgres postgres

passwd postgres

mkdir /home/postgres/ 

mkdir /usr/local/pgsql/data/ 

mkdir /var/log/postgres/ 

chown postgres.postgres /home/postgres/ 

chown postgres.postgres /usr/local/pgsql/data/ 

chown postgres.postgres /var/log/postgres

export PATH=/usr/local/pgsql/bin:$PATH



Switch to postgres user and initialize the db and we can start the service 


su postgres

/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data --encoding=UTF8 --lc-collate=en_US.UTF-8 --locale=en_US.utf8

/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start




You can login to the DB ,