pg_tiktoken_c

Overview

Built from upstream main snapshot fa2957b; bundles five vocabularies and includes DESTDIR and correctness patches. Upstream README declares Apache-2.0, but the pinned snapshot omits the referenced LICENSE file.

Version

Build

You can build the RPM / DEB packages for pg_tiktoken_c using pig build:

pig build pkg pg_tiktoken_c         # build RPM / DEB packages

Install

You can install pg_tiktoken_c directly. First, make sure the PGDG and PIGSTY repositories are added and enabled:

pig repo add pgsql -u          # Add repo and update cache

Install the extension using pig or apt/yum/dnf:

pig install pg_tiktoken_c;          # Install for current active PG version

pig ext install -y pg_tiktoken_c -v 18  # PG 18
pig ext install -y pg_tiktoken_c -v 17  # PG 17
pig ext install -y pg_tiktoken_c -v 16  # PG 16
pig ext install -y pg_tiktoken_c -v 15  # PG 15
pig ext install -y pg_tiktoken_c -v 14  # PG 14

dnf install -y pg_tiktoken_c_18       # PG 18
dnf install -y pg_tiktoken_c_17       # PG 17
dnf install -y pg_tiktoken_c_16       # PG 16
dnf install -y pg_tiktoken_c_15       # PG 15
dnf install -y pg_tiktoken_c_14       # PG 14

apt install -y postgresql-18-pg-tiktoken-c   # PG 18
apt install -y postgresql-17-pg-tiktoken-c   # PG 17
apt install -y postgresql-16-pg-tiktoken-c   # PG 16
apt install -y postgresql-15-pg-tiktoken-c   # PG 15
apt install -y postgresql-14-pg-tiktoken-c   # PG 14

Create Extension:

CREATE EXTENSION pg_tiktoken_c;

Usage

Sources:

• pg_tiktoken_c README at the packaged revision
• Version 1.1 SQL API
• Extension control file
• Bundled vocabulary data

pg_tiktoken_c implements OpenAI-compatible tiktoken encoding in C inside PostgreSQL. Use it to count or materialize tokens near stored text and to split text into token-bounded chunks before embedding or model requests.

Create the Extension

CREATE EXTENSION pg_tiktoken_c;

The implementation depends on PCRE2 10.30 or later at build time. It does not require shared_preload_libraries; vocabulary data is loaded and cached per backend as encodings are used.

Encode and Count

SELECT tiktoken_encode('cl100k_base', 'PostgreSQL search');
SELECT tiktoken_count('cl100k_base', 'PostgreSQL search');

tiktoken_encode returns a bigint array of token identifiers. tiktoken_count returns the token count without requiring the caller to retain the token array.

The bundled selectors include cl100k_base, o200k_base, r50k_base, p50k_base, and p50k_edit, together with aliases documented by the project. Choose the encoding required by the target model rather than assuming all models share a vocabulary.

Chunk Text

Return chunks as an array:

SELECT chunk_text(
  'long document text',
  chunk_size => 512,
  chunk_overlap => 64,
  encoding => 'cl100k_base'
);

Or return one row per chunk:

SELECT *
FROM chunk_text_table(
  'long document text',
  chunk_size => 512,
  chunk_overlap => 64,
  encoding => 'cl100k_base'
);

chunk_text_table returns chunk_index, chunk, and token_count. The chunk index is zero-based. Overlap repeats boundary tokens between neighboring chunks and must be smaller than the chunk size.

Function Index

• tiktoken_encode(selector, text) returns bigint[] token identifiers.
• tiktoken_count(selector, text) returns bigint token count.
• chunk_text(input_text, chunk_size, chunk_overlap default 0, encoding default cl100k_base) returns text[].
• chunk_text_table(input_text, chunk_size, chunk_overlap default 0, encoding default cl100k_base) returns one row per chunk with its index and token count.

The SQL functions are declared immutable and parallel safe. They can therefore be used in generated expressions or parallel plans only when the selected vocabulary files are deployed consistently across every server.

Operational Notes

• Tokenization is model-encoding specific. Confirm both the encoding name and the model’s current context limits in the application.
• Counting or chunking large text consumes backend CPU and memory; batch large corpora and monitor query latency.
• Backend-local caches avoid repeated parsing but increase memory use in sessions that touch several vocabularies.
• The upstream README’s compatibility list can lag packaging. Test the exact pg_tiktoken_c build against the target PostgreSQL major version instead of inferring support from a different binary.