pg_cjk_parser

Overview

PGSTY applies a PG_CONFIG build-selection patch.

Version

Build

You can build the RPM / DEB packages for pg_cjk_parser using pig build:

pig build pkg pg_cjk_parser         # build RPM / DEB packages

Install

You can install pg_cjk_parser directly. First, make sure the PGDG and PIGSTY repositories are added and enabled:

pig repo add pgsql -u          # Add repo and update cache

Install the extension using pig or apt/yum/dnf:

pig install pg_cjk_parser;          # Install for current active PG version

pig ext install -y pg_cjk_parser -v 18  # PG 18
pig ext install -y pg_cjk_parser -v 17  # PG 17
pig ext install -y pg_cjk_parser -v 16  # PG 16
pig ext install -y pg_cjk_parser -v 15  # PG 15
pig ext install -y pg_cjk_parser -v 14  # PG 14

dnf install -y pg_cjk_parser_18       # PG 18
dnf install -y pg_cjk_parser_17       # PG 17
dnf install -y pg_cjk_parser_16       # PG 16
dnf install -y pg_cjk_parser_15       # PG 15
dnf install -y pg_cjk_parser_14       # PG 14

apt install -y postgresql-18-pg-cjk-parser   # PG 18
apt install -y postgresql-17-pg-cjk-parser   # PG 17
apt install -y postgresql-16-pg-cjk-parser   # PG 16
apt install -y postgresql-15-pg-cjk-parser   # PG 15
apt install -y postgresql-14-pg-cjk-parser   # PG 14

Create Extension:

CREATE EXTENSION pg_cjk_parser;

Usage

Sources:

• Official v0.1.0 README
• v0.1.0 release notes
• v0.1.0 control file
• v0.1.0 SQL functions

pg_cjk_parser is a PostgreSQL full-text-search parser derived from the built-in parser. In a UTF-8 database it keeps the default behavior for non-CJK text while emitting overlapping 2-gram tokens for Chinese, Japanese, and Korean text. The extension installs parser support functions; you create the text-search parser and configuration that use them.

Core Workflow

CREATE EXTENSION pg_cjk_parser;

CREATE TEXT SEARCH PARSER public.pg_cjk_parser (
    START = prsd2_cjk_start,
    GETTOKEN = prsd2_cjk_nexttoken,
    END = prsd2_cjk_end,
    LEXTYPES = prsd2_cjk_lextype,
    HEADLINE = prsd2_cjk_headline
);

CREATE TEXT SEARCH CONFIGURATION public.config_2_gram_cjk (
    PARSER = public.pg_cjk_parser
);

SELECT alias, description, token
FROM ts_debug(
    'public.config_2_gram_cjk',
    'PostgreSQL 全文検索和中文检索'
);

Use the configuration explicitly in generated tsvector columns and queries, or set it as the session default:

SET default_text_search_config = 'public.config_2_gram_cjk';

SELECT to_tsvector('public.config_2_gram_cjk', '日本語全文検索');

Important Objects

• prsd2_cjk_start, prsd2_cjk_nexttoken, prsd2_cjk_end, prsd2_cjk_lextype, and prsd2_cjk_headline: support functions used by CREATE TEXT SEARCH PARSER.
• cjk_zht2zhs(text): converts mapped Traditional Chinese characters to Simplified Chinese while leaving other characters unchanged.
• Parser token type cjk: emits overlapping CJK bigrams; CJK punctuation is emitted as a unigram.

SELECT cjk_zht2zhs('漢語');
-- 汉语

Version Notes and Caveats

• Version 0.1.0 fixes incorrect cjk_zht2zhs scanning across mixed-width UTF-8 characters and corrects handling of four-byte CJK code points.
• Upstream supports PostgreSQL 11 through 18 at this release.
• The database must use UTF-8 for CJK bigram behavior. With another encoding, the parser behaves like the PostgreSQL default parser.
• Creating a text-search parser requires elevated privileges. Decide mappings, dictionaries, stop words, and ranking separately; the example configuration defines only the parser.