Module: JUICE

• 1: Configuration
• 2: Parameters
• 3: Playbook
• 4: Administration
• 5: Monitoring
• 6: FAQ

flowchart LR
    subgraph Client["App/User"]
        app["POSIX Access"]
    end

    subgraph JUICE["JUICE"]
        jfs["JuiceFS Mount"]
    end

    subgraph PGSQL["PGSQL"]
        meta["Metadata DB"]
    end

    subgraph Object["Object Storage (optional)"]
        s3["S3 / MinIO"]
    end

    subgraph INFRA["INFRA (optional)"]
        vm["VictoriaMetrics"]
    end

    app --> jfs
    jfs --> meta
    jfs -.-> s3
    jfs -->|/metrics| vm

    style JUICE fill:#5B9CD5,stroke:#4178a8,color:#fff
    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style Object fill:#FCDB72,stroke:#d4b85e,color:#333
    style INFRA fill:#999,stroke:#666,color:#fff

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567

./juice.yml -l <host>

1 - Configuration

Concepts and Implementation

JuiceFS consists of a metadata engine and data storage. In Pigsty v4.1, meta is passed through to juicefs as the metadata engine URL, and PostgreSQL is typically used in production. Data storage is defined by data options passed to juicefs format.

JUICE module core commands:

# Format (only effective on first creation)
juicefs format --no-update <data> "<meta>" "<name>"

# Mount
juicefs mount <mount> --cache-dir <juice_cache> --metrics 0.0.0.0:<port> <meta> <path>

Notes:

• --no-update ensures existing filesystems are not overwritten.
• data is only used for initial format; it does not affect existing filesystems.
• mount is only used during mount, you can pass cache and concurrency options.

Module Parameters

JUICE module has only two parameters:

• juice_cache: shared local cache directory for all instances, default /data/juice
• juice_instances: instance-level dict, key is filesystem name; an empty dict means no instances are managed

Instance Configuration

Each entry in juice_instances represents a JuiceFS instance:

Example:

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567

Storage Backends

data is appended to juicefs format, any supported backend works. Common examples:

PostgreSQL Large Objects

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta

MinIO Object Storage

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage minio --bucket http://10.10.10.10:9000/juice --access-key minioadmin --secret-key minioadmin

S3-Compatible Storage

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage s3 --bucket https://s3.amazonaws.com/my-bucket --access-key AKIAXXXXXXXX --secret-key XXXXXXXXXX

Typical Configurations

Multi-Instance (Same Node)

juice_instances:
  pgfs:
    path: /pgfs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567
  shared:
    path: /shared
    meta: postgres://dbuser_meta:[email protected]:5432/shared
    data: --storage minio --bucket http://10.10.10.10:9000/shared
    port: 9568
    owner: postgres
    group: postgres

Shared Mount Across Nodes

Mount the same JuiceFS on multiple nodes:

app:
  hosts:
    10.10.10.11: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }
    10.10.10.12: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }

Only one node needs to format the filesystem; others will skip via --no-update.

Notes

• port is exposed on 0.0.0.0. Use firewall or security group to restrict access.
• Changing data will not update an existing filesystem; handle migration manually.

2 - Parameters

JUICE module has 2 parameters:

• juice_cache: shared cache directory
• juice_instances: instance definition dict

Parameter Overview

Level: C = cluster level, I = instance level.

Default Parameters

Defined in roles/juice/defaults/main.yml:

#-----------------------------------------------------------------
# JUICE
#-----------------------------------------------------------------
juice_cache: /data/juice
juice_instances: {}

juice_cache

Parameter: juice_cache, type: path, level: C

Shared local cache directory for all JuiceFS instances, default /data/juice. JuiceFS isolates caches by filesystem UUID under this directory.

juice_cache: /data/juice

juice_instances

Parameter: juice_instances, type: dict, level: I

Instance definition dict, usually defined at instance level. Default is an empty dict (meaning no instances are deployed). Key is filesystem name, value is instance config object.

juice_instances:
  jfs:
    path: /fs
    meta: postgres://u:p@h:5432/db
    data: --storage postgres --bucket ...
    port: 9567

Instance fields:

3 - Playbook

JUICE module provides juice.yml playbook to deploy and remove JuiceFS instances.

juice.yml

Task structure in juice.yml:

juice_id        : validate config, check port conflicts
juice_install   : install juicefs package
juice_cache     : create shared cache dir
juice_clean     : remove instance (state=absent)
juice_instance  : create instance (state=create)
  - juice_init  : format filesystem (--no-update)
  - juice_dir   : create mount dir
  - juice_config: render env file and systemd unit
  - juice_launch: start service and wait for metrics port
juice_register  : register to VictoriaMetrics targets

Scope

Examples:

./juice.yml -l 10.10.10.10                 # deploy all instances on the node
./juice.yml -l 10.10.10.10 -e fsname=jfs   # only deploy jfs instance

Common Tags

Config Updates

Render config only (no restart):

./juice.yml -l <host> -t juice_config

Update config and ensure service is online (without force restart):

./juice.yml -l <host> -t juice_config,juice_launch

If you need new mount options to take effect immediately, manually restart the instance service:

systemctl restart juicefs-<name>

Remove Instance

Removal flow:

1. Set instance state to absent
2. Run juice_clean

juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    state: absent

./juice.yml -l <host> -t juice_clean
./juice.yml -l <host> -e fsname=jfs -t juice_clean

Removal includes: stop service, lazy unmount, remove systemd unit/env files, reload systemd. PostgreSQL metadata and object storage data are not deleted.

Monitoring Registration

juice_register writes target file on infra node:

/infra/targets/juice/<hostname>.yml

To re-register manually:

./juice.yml -l <host> -t juice_register

4 - Administration

Common operations:

• Initialize Instance
• Reconfigure
• Remove Instance
• Add New Instance
• Shared Mount Across Nodes
• PITR Recovery
• Troubleshooting
• Performance Tuning

See FAQ for more.

Initialize Instance

./juice.yml -l <host>
./juice.yml -l <host> -e fsname=<name>

Initialization steps:

• Install juicefs package
• Create shared cache dir (default /data/juice)
• Run juicefs format --no-update (only effective on first creation)
• Create mount point and set permissions
• Render systemd unit and env files
• Start service and wait for metrics port
• Register to VictoriaMetrics (if infra node exists)

Reconfigure

After changing config, it’s recommended to run (update config and ensure service is online):

./juice.yml -l <host> -t juice_config,juice_launch

Render config without touching service state:

./juice.yml -l <host> -t juice_config

Notes:

• juice_config,juice_launch ensures service is started, but does not force-restart an already running instance
• data only takes effect on the first format
• After changing mount options, manually restart the instance service (systemctl restart juicefs-<name>)

Remove Instance

1. Set instance state to absent
2. Run juice_clean

juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    state: absent

./juice.yml -l <host> -t juice_clean
./juice.yml -l <host> -e fsname=jfs -t juice_clean

Removal actions:

• Stop systemd service
• umount -l lazy unmount
• Remove unit and env files
• Reload systemd

PostgreSQL metadata and object storage data are not deleted.

Add New Instance

Add a new instance in config, ensure unique port:

juice_instances:
  newfs:
    path: /newfs
    meta: postgres://...
    data: --storage minio --bucket http://minio:9000/newfs
    port: 9568

Deploy:

./juice.yml -l <host> -e fsname=newfs

Shared Mount Across Nodes

Configure the same meta and instance name on multiple nodes:

app:
  hosts:
    10.10.10.11: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }
    10.10.10.12: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }

Only one node needs to format the filesystem; others will skip via --no-update.

PITR Recovery

When data is also stored in PostgreSQL (--storage postgres), filesystem PITR can be done via PG PITR:

# Stop services on all nodes
systemctl stop juicefs-jfs

# Restore metadata DB with pgBackRest
pb restore --stanza=meta --type=time --target="2024-01-15 10:30:00"

# Start PostgreSQL
systemctl start postgresql

# Start JuiceFS service
systemctl start juicefs-jfs

If data is stored in MinIO/S3, only metadata is rolled back; objects will not.

Troubleshooting

Mount Fails

systemctl status juicefs-jfs
journalctl -u juicefs-jfs -f
mountpoint /fs

Metadata Connection Issues

psql "postgres://dbuser_meta:[email protected]:5432/meta" -c "SELECT 1"

Metrics Port Check

ss -tlnp | grep 9567
curl http://localhost:9567/metrics

Performance Tuning

Pass juicefs mount options via mount:

juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    mount: --cache-size 102400 --prefetch 3 --max-uploads 50

Key metrics to watch:

• juicefs_blockcache_hits/juicefs_blockcache_miss: cache hit ratio
• juicefs_object_request_durations_histogram_seconds: object storage latency
• juicefs_transaction_durations_histogram_seconds: metadata transaction latency

5 - Monitoring

JuiceFS instances expose Prometheus metrics via juicefs mount --metrics. In JUICE, metrics listen on 0.0.0.0:<port>, default port 9567.

Monitoring Architecture

JuiceFS Mount (metrics: 0.0.0.0:<port>)
    ↓
VictoriaMetrics (scrape)
    ↓
Grafana Dashboard

If INFRA is deployed, juice_register writes scrape targets to:

/infra/targets/juice/<hostname>.yml

Target File Example

- labels: { ip: 10.10.10.10, ins: "node-jfs", cls: "jfs" }
  targets: [ 10.10.10.10:9567 ]

To register manually:

./juice.yml -l <host> -t juice_register

Key Metrics

Object Storage

Cache

Metadata Transactions

Common PromQL

Cache hit ratio:

rate(juicefs_blockcache_hits[5m]) /
(rate(juicefs_blockcache_hits[5m]) + rate(juicefs_blockcache_miss[5m]))

Object storage P99 latency:

histogram_quantile(0.99, rate(juicefs_object_request_durations_histogram_seconds_bucket[5m]))

6 - FAQ

Port Conflicts?

Multiple instances on the same node must use different port values. Example:

juice_instances:
  fs1:
    path: /fs1
    meta: postgres://...
    port: 9567
  fs2:
    path: /fs2
    meta: postgres://...
    port: 9568

Why does changing data not take effect?

data is only used by juicefs format --no-update. After filesystem creation it will not change. To switch backend, migrate data and reformat manually.

How to add a new instance?

1. Add instance definition in config
2. Run:

./juice.yml -l <host> -e fsname=<name>

How to remove an instance?

1. Set instance state to absent
2. Run:

./juice.yml -l <host> -t juice_clean

Removal does not delete PostgreSQL metadata or object storage data.

Where is file data stored?

Depends on data:

• --storage postgres: data in PostgreSQL pg_largeobject
• --storage minio/s3: data in object storage bucket

Metadata is stored in the metadata engine defined by meta (in Pigsty production scenarios, this is usually PostgreSQL).

Multi-node mount notes?

• Use the same meta and instance name on all nodes
• Only one node needs to format; others will skip
• Ensure port does not conflict on each node

Monitoring target not generated?

juice_register only writes /infra/targets/juice/ when infra group exists. You can run manually:

./juice.yml -l <host> -t juice_register

How to change mount options?

After updating mount in the instance, refresh config first and then manually restart the service:

./juice.yml -l <host> -t juice_config,juice_launch
systemctl restart juicefs-<name>