7.1.5. groonga-httpd — Groonga v16.0.5 documentation

Skip to main content
`Ctrl`+`K`

# 7.1.5. groonga-httpd

Warning

groonga-httpd has been extracted as groonga-nginx since Groonga 13.0.3.

## 7.1.5.1. Summary

groonga-httpd is a program to communicate with a Groonga server using the HTTP protocol. It functions as same as Groonga HTTP server. Although Groonga HTTP server has limited support for HTTP with a minimal built-in HTTP server, groonga-httpd has full support for HTTP with an embedded nginx. All standards-compliance and features provided by nginx is also available in groonga-httpd.

groonga-httpd has an Web-based administration tool implemented with HTML and JavaScript. You can access to it from http://hostname:port/.

## 7.1.5.2. Syntax

    groonga-httpd [nginx options]

## 7.1.5.3. Usage

### 7.1.5.3.1. Set up

First, you’ll need to edit the groonga-httpd configuration file to specify a database. Edit /etc/groonga/httpd/groonga-httpd.conf to enable the `groonga_database` directive like this:

    # Match this to the file owner of groonga database files if groonga-httpd is
    # run as root.
    #user groonga;
    ...
    http {
      ...
      # Don't change the location; currently only /d/ is supported.
      location /d/ {
        groonga on; # <= This means to turn on groonga-httpd.

        # Specify an actual database and enable this.
        groonga_database /var/lib/groonga/db/db;
      }
      ...
    }

Then, run groonga-httpd. Note that the control immediately returns back to the console because groonga-httpd runs as a daemon process by default.:

    % groonga-httpd

### 7.1.5.3.2. Request queries

To check, request a simple query (status).

Execution example:

    $ curl http://localhost:10041/d/status
    [
      [
        0,
        1337566253.89858,
        0.000355720520019531
      ],
      {
        "alloc_count": 29,
        "starttime": 1696558618,
        "start_time": 1696558618,
        "uptime": 0,
        "version": "2.9.1",
        "n_queries": 0,
        "cache_hit_rate": 0.0,
        "command_version": 1,
        "default_command_version": 1,
        "max_command_version": 3,
        "n_jobs": 0,
        "features": {
          "nfkc": true,
          "mecab": true,
          "message_pack": true,
          "mruby": true,
          "onigmo": true,
          "zlib": true,
          "lz4": true,
          "zstandard": true,
          "kqueue": false,
          "epoll": true,
          "poll": false,
          "rapidjson": false,
          "apache_arrow": true,
          "xxhash": true,
          "blosc": true,
          "bfloat16": true,
          "h3": true,
          "simdjson": true,
          "llama.cpp": true,
          "back_trace": true,
          "reference_count": false
        },
        "apache_arrow": {
          "version_major": 2,
          "version_minor": 9,
          "version_patch": 1,
          "version": "2.9.1"
        },
        "memory_map_size": 2929,
        "n_workers": 0,
        "default_n_workers": 0,
        "os": "Linux",
        "cpu": "x86_64"
      }
    ]

### 7.1.5.3.3. Loading data by POST

You can load data by POST JSON data.

Here is an example `curl` command line that loads two users `alice` and `bob` to `Users` table:

    % curl --data-binary '[{"_key": "alice"}, {"_key": "bob"}]' -H "Content-Type: application/json" "http://localhost:10041/d/load?table=Users"

If you loads users from JSON file, prepare JSON file like this:

    [
    {"_key": "alice"},
    {"_key": "bob"}
    ]

Then specify JSON file in `curl` command line:

    % curl -X POST 'http://localhost:10041/d/load?table=Users' -H 'Content-Type: application/json' -d @users.json

### 7.1.5.3.4. Browse the administration tool

Also, you can browse Web-based administration tool at http://localhost:10041/.

### 7.1.5.3.5. Shut down

Finally, to terminate the running groonga-httpd daemon, run this:

    % groonga-httpd -s stop

## 7.1.5.4. Configuration directives

This section describes only important directives. They are groonga-httpd specific directives and performance related directives.

The following directives can be used in the groonga-httpd configuration file. By default, it’s located at /etc/groonga/httpd/groonga-httpd.conf.

### 7.1.5.4.1. Groonga-httpd specific directives

The following directives aren’t provided by nginx. They are provided by groonga-httpd to configure groonga-httpd specific configurations.

#### 7.1.5.4.1.1. `groonga`

Syntax:

    groonga on | off;

Default

  `groonga off;`

Context

  `location`

Specifies whether Groonga is enabled in the `location` block. The default is `off`. You need to specify `on` to enable groonga.

Examples:

    location /d/ {
      groonga on;  # Enables groonga under /d/... path
    }

    location /d/ {
      groonga off; # Disables groonga under /d/... path
    }

#### 7.1.5.4.1.2. `groonga_database`

Syntax:

    groonga_database /path/to/groonga/database;

Default

  `groonga_database /usr/local/var/lib/groonga/db/db;`

Context

  `http`, `server`, `location`

Specifies the path to a Groonga database. This is the required directive.

#### 7.1.5.4.1.3. `groonga_database_auto_create`

Syntax:

    groonga_database_auto_create on | off;

Default

  `groonga_database_auto_create on;`

Context

  `http`, `server`, `location`

Specifies whether Groonga database is created automatically or not. If the value is `on` and the Groonga database specified by groonga_database doesn’t exist, the Groonga database is created automatically. If the Groonga database exists, groonga-httpd does nothing.

If parent directory doesn’t exist, parent directory is also created recursively.

The default value is `on`. Normally, the value doesn’t need to be changed.

#### 7.1.5.4.1.4. `groonga_base_path`

Syntax:

    groonga_base_path /d/;

Default

  The same value as `location` name.

Context

  `location`

Specifies the base path in URI. Groonga uses `/d/command?parameter1=value1&...` path to run `command`. The form of path in used in groonga-httpd but groonga-httpd also supports `/other-prefix/command?parameter1=value1&...` form. To support the form, groonga-httpd removes the base path from the head of request URI and prepend `/d/` to the processed request URI. By the path conversion, users can use custom path prefix and Groonga can always uses `/d/command?parameter1=value1&...` form.

Normally, this directive isn’t needed. It is needed for per command configuration.

Here is an example configuration to add authorization to shutdown command:

    groonga_database /var/lib/groonga/db/db;

    location /d/shutdown {
      groonga on;
      # groonga_base_path is needed.
      # Because /d/shutdown is handled as the base path.
      # Without this configuration, /d/shutdown/shutdown path is required
      # to run shutdown command.
      groonga_base_path /d/;
      auth_basic           "manager is required!";
      auth_basic_user_file "/etc/managers.htpasswd";
    }

    location /d/ {
      groonga on;
      # groonga_base_path doesn't needed.
      # Because location name is the base path.
    }

#### 7.1.5.4.1.5. `groonga_log_path`

Syntax:

    groonga_log_path path | off;

Default

  `/var/log/groonga/httpd/groonga.log`

Context

  `http`, `server`, `location`

Specifies Groonga log path in the `http`, `server` or `location` block. The default is `/var/log/groonga/httpd/groonga.log`. You can disable logging to specify `off`.

Examples:

    location /d/ {
      groonga on;
      # You can disable log for groonga.
      groonga_log_path off;
    }

#### 7.1.5.4.1.6. `groonga_log_level`

Syntax:

    groonga_log_level none | emergency | alert | critical | error | warning | notice | info | debug | dump;

Default

  `notice`

Context

  `http`, `server`, `location`

Specifies Groonga log level in the `http`, `server` or `location` block. The default is `notice`. You can disable logging by specifying `none` as log level.

Examples:

    location /d/ {
      groonga on;
      # You can customize log level for groonga.
      groonga_log_level notice;
    }

#### 7.1.5.4.1.7. `groonga_query_log_path`

Syntax:

    groonga_query_log_path path | off;

Default

  `/var/log/groonga/httpd/groonga-query.log`

Context

  `http`, `server`, `location`

Specifies Groonga’s query log path in the `http`, `server` or `location` block. The default is `/var/log/groonga/httpd/groonga-query.log`. You can disable logging to specify `off`.

Examples:

    location /d/ {
      groonga on;
      # You can disable query log for groonga.
      groonga_query_log_path off;
    }

Query log is useful for the following cases:

> - Detecting slow query.

> - Debugging.

You can analyze your query log by groonga-query-log package. The package provides useful tools.

For example, there is a tool that analyzing your query log. It can detect slow queries from your query log. There is a tool that replaying same queries in your query log. It can test the new Groonga before updating production environment.

### 7.1.5.4.2. Performance related directives

The following directives are related to the performance of groonga-httpd.

#### 7.1.5.4.2.1. `worker_processes`

For optimum performance, set this to be equal to the number of CPUs or cores. In many cases, Groonga queries may be CPU-intensive work, so to fully utilize multi-CPU/core systems, it’s essential to set this accordingly.

This isn’t a groonga-httpd specific directive, but an nginx’s one. For details, see http://wiki.nginx.org/CoreModule#worker_processes.

By default, this is set to 1. It is nginx’s default.

#### 7.1.5.4.2.2. `groonga_cache_limit`

Syntax:

    groonga_cache_limit limit;

Default

  `groonga_cache_limit 100;`

Context

  `http`, `server`, `location`

Specifies Groonga’s limit of query cache in the `http`, `server` or `location` block. The default value is 100. You can disable query cache to specify 0 to `groonga_cache_limit` explicitly.

Examples:

    location /d/ {
      groonga on;
      # You can customize query cache limit for groonga.
      groonga_cache_limit 100;
    }

#### 7.1.5.4.2.3. `groonga_cache_base_path`

Syntax:

    groonga_cache_base_path path | off;

Default

  `groonga_cache_base_path off;`

Context

  `http`, `server`, `location`

Specifies the base path of query cache in the `http`, `server` or `location` block.

It’s recommended that you specify this configuration when you use multi-workers configuration.

If the base path is specified, you can use persistent cache instead of on memory cache. If you use persistent cache, workers share query cache. It’s efficient for multi-workers configuration because the same response is cached only once in multiple workers.

There is one more merit for persistent cache. You don’t need to warm up cache after groonga-httpd is restarted. Persistent cache isn’t cleared when groonga-httpd is down. groonga-httpd can use existing persistent cache again.

The default value is `off`. It means that persistent cache is disabled. On memory cache is used. On memory cache is independent in each worker. It’s not efficient for multi-workers configuration because two or more workers may keeps the same response separately.

Persistent cache is a bit slower than on memory cache. Normally, the difference has little influence on performance.

You must specify the base path on memory file system. If you specify the base path on disk, your cache will be slow. It’s not make sense.

Examples:

    location /d/ {
      groonga on;
      # You can customize query cache limit for groonga.
      groonga_cache_base_path /dev/shm/groonga-httpd-cache;
    }

#### 7.1.5.4.2.4. `proxy_cache`

In short, you can use nginx’s reverse proxy and cache mechanism instead of Groonga’s built-in query cache feature.

##### 7.1.5.4.2.4.1. Query cache

Groonga has query cache feature for select command. The feature improves performance in many cases.

Query cache feature works well on groonga-httpd except you use cache_limit command on 2 or more workers. Normally, cache_limit command isn’t used. So there is no problem on many cases.

Here is a description about a problem of using cache_limit command on 2 or more workers.

Groonga’s query cache is available in the same process. It means that workers can’t share the cache. If you don’t change cache size, it isn’t a big problem. If you want to change cache size by cache_limit command, there is a problem.

There is no portable ways to change cache size for all workers.

For example, there are 3 workers:

                                       +-- worker 1
    client -- groonga-httpd (master) --+-- worker 2
                                       +-- worker 3

The client requests cache_limit command and the worker 1 receives it:

                                       +-> worker 1 (changed!)
    client -> groonga-httpd (master) --+-- worker 2
                                       +-- worker 3

The client requests cache_limit command again and the worker 1 receives it again:

                                       +-> worker 1 (changed again!!!)
    client -> groonga-httpd (master) --+-- worker 2
                                       +-- worker 3

In this case, the worker 2 and the worker 3 aren’t received any requests. So they don’t change cache size.

You can’t choose a worker. So you can’t change cache sizes of all workers by cache_limit command.

##### 7.1.5.4.2.4.2. Reverse proxy and cache

You can use nginx’s reverse proxy and cache feature for query cache:

                                                                +-- worker 1
    client -- groonga-httpd (master) -- reverse proxy + cache --+-- worker 2
                                                                +-- worker 3

You can use the same cache configuration for all workers but you can’t change cache configuration dynamically by HTTP.

Here is a sample configuration:

    ...
    http {
      proxy_cache_path /var/cache/groonga-httpd levels=1:2 keys_zone=groonga:10m;
      proxy_cache_valid 10m;
      ...
      # Reverse proxy and cache
      server {
        listen 10041;
        ...
        # Only select command
        location /d/select {
          # Pass through groonga with cache
          proxy_cache groonga;
          proxy_pass http://localhost:20041;
        }

        location / {
          # Pass through groonga
          proxy_pass http://localhost:20041;
        }
      }

      # groonga
      server {
        location 20041;
        location /d/ {
          groonga on;
          groonga_database /var/lib/groonga/db/db;
        }
      }
      ...
    }

See the following nginx documentations for parameter details:

> - http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path

> - http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid

> - http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache

> - http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

Note that you need to remove cache files created by nginx by hand after you load new data to Groonga. For the above sample configuration, run the following command to remove cache files:

    % groonga DB_PATH < load.grn
    % rm -rf /var/cache/groonga-httpd/*

If you use Groonga’s query cache feature, you don’t need to expire cache by hand. It is done automatically.

### 7.1.5.4.3. nginx log related directives

#### 7.1.5.4.3.1. `access_log`

We can output nginx’s log by the `access_log` directive. The following example is the default nginx’s log settings:

    http {
      log_format  combined  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';

      access_log  access.log  combined;
    }

We can specify log format by `log_format` directive. For the details of available items of the log, see Module ngx_http_log_module.

nginx’s access log is outputted when a requests is processed. If we want to output the request processing time, we need to set configuration as below:

    http {
      log_format  combined  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for" '
                        '"$request_time"';

      access_log  access.log  combined;
    }

## 7.1.5.5. Available nginx modules

All standard HTTP modules are available. HttpRewriteModule is disabled when you don’t have PCRE (Perl Compatible Regular Expressions). For the list of standard HTTP modules, see nginx documentation.