SharedArrayBuffer Implementation Plan for nginx njs

Overview

Implementation of js_shared_array directive for nginx njs modules (HTTP and Stream) that creates shared memory zones accessible as SharedArrayBuffer objects via ngx.sharedArray.foo dynamic property lookup.

Scope: QuickJS engine only (wrapped in #if (NJS_HAVE_QUICKJS))

Architecture

nginx.conf:  js_shared_array zone=foo:1m;
             ↓
JavaScript:  ngx.sharedArray          ← NGX_QJS_CLASS_ID_SHARED_ARRAY (exotic class)
                         .foo         ← returns built-in SharedArrayBuffer
             ↓
QuickJS:     ngx_qjs_ext_ngx_shared_array() getter
             ↓
Exotic:      NGX_QJS_CLASS_ID_SHARED_ARRAY class (for ngx.sharedArray object)
             ↓
Lookup:      ngx_qjs_shared_array_own_property()
             ↓
Return:      JS_NewArrayBuffer(ctx, data, size, NULL, NULL, TRUE)
                                                    ↑     ↑     ↑
                                                    |     |     is_shared
                                                    |     opaque
                                                    free_func = NULL
             ↓
Result:      Built-in SharedArrayBuffer instance (no custom class needed)

Key Distinction:

• NGX_QJS_CLASS_ID_SHARED_ARRAY is for the exotic container (ngx.sharedArray)
• The SharedArrayBuffer returned by property lookup is a built-in QuickJS type

Key Design Decisions

1. No Custom Allocator

Use QuickJS's JS_NewArrayBuffer() directly instead of setting vm->sab_funcs:

JSValue JS_NewArrayBuffer(JSContext *ctx, uint8_t *buf, size_t len,
                          JSFreeArrayBufferDataFunc *free_func, void *opaque,
                          JS_BOOL is_shared);

• buf: Pointer to pre-allocated shared memory
• len: Size in bytes
• free_func: NULL (nginx owns memory, not QuickJS)
• opaque: Not used
• is_shared: TRUE (creates SharedArrayBuffer)

2. Dynamic Property Lookup

Pattern: ngx.sharedArray.foo (like ngx.shared.foo)

• Uses QuickJS exotic class methods
• Property handler looks up zone by name
• Returns new SharedArrayBuffer wrapping the zone

3. Simple Memory Management

• Allocation: Entire shared zone is the backing store
• Initialization: Zero-filled on first worker
• Cleanup: None (nginx manages lifecycle)
• Reload: Reuse existing memory (preserve data)

4. Separate File Implementation

Create new files for cleaner separation of concerns:

• nginx/ngx_js_shared_array.c - Core implementation (~250 lines)
• nginx/ngx_js_shared_array.h - Public API and structure definitions

Why separate files?

• Distinct features (dict vs array)
• Better maintainability
• ngx_js_shared_dict.c is already 4200+ lines
• Single responsibility principle

Implementation Steps

Step 1: Create Header File

File: nginx/ngx_js_shared_array.h (new file)

/*
 * Copyright (C) Dmitry Volyntsev
 * Copyright (C) NGINX, Inc.
 */


#ifndef _NGX_JS_SHARED_ARRAY_H_INCLUDED_
#define _NGX_JS_SHARED_ARRAY_H_INCLUDED_


#include <ngx_config.h>
#include <ngx_core.h>
#include "ngx_js.h"


typedef struct ngx_js_shared_array_s  ngx_js_shared_array_t;


struct ngx_js_shared_array_s {
    ngx_shm_zone_t        *shm_zone;
    void                  *data;      /* SharedArrayBuffer backing store */
    size_t                 size;      /* Zone size in bytes */

    /* fd for event debug (same position as ngx_connection_t) */
    ngx_socket_t           fd;

    ngx_js_shared_array_t *next;      /* Linked list */
};


/* Directive handler */
char *ngx_js_shared_array_zone(ngx_conf_t *cf, ngx_command_t *cmd,
    void *conf, void *tag);


#endif /* _NGX_JS_SHARED_ARRAY_H_INCLUDED_ */

Step 2: Update ngx_js.h

File: nginx/ngx_js.h

Add class ID (after line 65):

#if (NJS_HAVE_QUICKJS)
enum {
    // ... existing ...
    NGX_QJS_CLASS_ID_SHARED_DICT_ERROR,
    NGX_QJS_CLASS_ID_SHARED_ARRAY,        /* ADD THIS */
    NGX_QJS_CLASS_ID_FETCH_HEADERS,
};
#endif

Modify main config macro (around line 123):

#define NGX_JS_COMMON_MAIN_CONF              \
    ngx_js_dict_t         *dicts;            \
    ngx_js_shared_array_t *shared_arrays;    \  /* ADD THIS */
    ngx_array_t           *periodics

Note: typedef is now in ngx_js_shared_array.h

Step 3: Create Implementation File

File: nginx/ngx_js_shared_array.c (new file)

Pattern: Follow ngx_js_shared_dict_zone() in ngx_js_shared_dict.c (lines 2819-2991)

/*
 * Copyright (C) Dmitry Volyntsev
 * Copyright (C) NGINX, Inc.
 */


#include <ngx_config.h>
#include <ngx_core.h>
#include "ngx_js.h"
#include "ngx_js_shared_array.h"


#if (NJS_HAVE_QUICKJS)

#include <quickjs.h>


/* Forward declarations */
static ngx_int_t ngx_js_shared_array_init_zone(ngx_shm_zone_t *shm_zone,
    void *data);
static int ngx_qjs_shared_array_own_property(JSContext *cx,
    JSPropertyDescriptor *pdesc, JSValueConst obj, JSAtom prop);
static int ngx_qjs_shared_array_own_property_names(JSContext *cx,
    JSPropertyEnum **ptab, uint32_t *plen, JSValueConst obj);
static JSValue ngx_qjs_ext_ngx_shared_array(JSContext *cx,
    JSValueConst this_val);


/* Exotic class definition */
static JSClassDef ngx_qjs_shared_array_class = {
    "SharedArray",
    .finalizer = NULL,
    .exotic = &(JSClassExoticMethods) {
        .get_own_property = ngx_qjs_shared_array_own_property,
        .get_own_property_names = ngx_qjs_shared_array_own_property_names,
    },
};


/* Module definition for registration with ngx object */
static const JSCFunctionListEntry ngx_qjs_ext_ngx_shared_array_props[] = {
    JS_CGETSET_DEF("sharedArray", ngx_qjs_ext_ngx_shared_array, NULL),
};


/* Directive handler */
char *
ngx_js_shared_array_zone(ngx_conf_t *cf, ngx_command_t *cmd, void *conf,
    void *tag)
{
    ssize_t                  size;
    ngx_str_t               *value, name, s;
    ngx_uint_t               i;
    ngx_shm_zone_t          *shm_zone;
    ngx_js_shared_array_t   *array;
    ngx_js_main_conf_t      *jmcf = conf;

    value = cf->args->elts;

    // Parse zone=name:size
    if (ngx_strncmp(value[1].data, "zone=", 5) != 0) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                          "invalid parameter \"%V\"", &value[1]);
        return NGX_CONF_ERROR;
    }

    name.data = value[1].data + 5;
    name.len = 0;

    for (i = 5; i < value[1].len; i++) {
        if (value[1].data[i] == ':') {
            name.len = i - 5;
            break;
        }
    }

    if (name.len == 0) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                          "invalid zone name in \"%V\"", &value[1]);
        return NGX_CONF_ERROR;
    }

    s.data = value[1].data + 5 + name.len + 1;
    s.len = value[1].len - (5 + name.len + 1);

    size = ngx_parse_size(&s);
    if (size == NGX_ERROR) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                          "invalid zone size in \"%V\"", &value[1]);
        return NGX_CONF_ERROR;
    }

    if (size < (ssize_t) (8 * ngx_pagesize)) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                          "zone \"%V\" is too small", &name);
        return NGX_CONF_ERROR;
    }

    // Create shared memory zone
    shm_zone = ngx_shared_memory_add(cf, &name, size, tag);
    if (shm_zone == NULL) {
        return NGX_CONF_ERROR;
    }

    if (shm_zone->data) {
        ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
                          "duplicate zone \"%V\"", &name);
        return NGX_CONF_ERROR;
    }

    // Allocate array structure
    array = ngx_pcalloc(cf->pool, sizeof(ngx_js_shared_array_t));
    if (array == NULL) {
        return NGX_CONF_ERROR;
    }

    array->shm_zone = shm_zone;
    array->size = size;
    array->fd = (ngx_socket_t) -1;

    // Link to main config
    array->next = jmcf->shared_arrays;
    jmcf->shared_arrays = array;

    // Set zone callbacks
    shm_zone->data = array;
    shm_zone->init = ngx_js_shared_array_init_zone;

    return NGX_CONF_OK;
}

#endif /* NJS_HAVE_QUICKJS */

Step 4: Zone Initialization (in same file)

Continue in nginx/ngx_js_shared_array.c:

Pattern: Follow ngx_js_dict_init_zone() in ngx_js_shared_dict.c (lines 2745-2814)

/* Zone initialization callback */
static ngx_int_t
ngx_js_shared_array_init_zone(ngx_shm_zone_t *shm_zone, void *data)
{
    ngx_js_shared_array_t  *array = shm_zone->data;
    ngx_js_shared_array_t  *prev = data;

    // Handle nginx reload - reuse existing data
    if (prev) {
        array->data = prev->data;
        return NGX_OK;
    }

    // First initialization or exists from previous process
    array->data = shm_zone->shm.addr;

    // Zero-initialize on first creation
    if (!shm_zone->shm.exists) {
        ngx_memzero(array->data, array->size);
    }

    return NGX_OK;
}


/* Property handlers */
static int
ngx_qjs_shared_array_own_property(JSContext *cx, JSPropertyDescriptor *pdesc,
    JSValueConst obj, JSAtom prop)
{
    int                      ret;
    ngx_str_t                name;
    ngx_js_shared_array_t   *array;
    ngx_shm_zone_t          *shm_zone;
    ngx_js_main_conf_t      *conf;

    name.data = (u_char *) JS_AtomToCString(cx, prop);
    if (name.data == NULL) {
        return -1;
    }

    name.len = ngx_strlen(name.data);
    ret = 0;
    conf = ngx_qjs_main_conf(cx);

    for (array = conf->shared_arrays; array != NULL; array = array->next) {
        shm_zone = array->shm_zone;

        if (shm_zone->shm.name.len == name.len
            && ngx_strncmp(shm_zone->shm.name.data, name.data, name.len) == 0)
        {
            if (pdesc != NULL) {
                pdesc->flags = JS_PROP_ENUMERABLE;
                pdesc->getter = JS_UNDEFINED;
                pdesc->setter = JS_UNDEFINED;

                /* Create SharedArrayBuffer wrapping shared memory */
                pdesc->value = JS_NewArrayBuffer(cx, array->data, array->size,
                                                 NULL, NULL, TRUE);
                if (JS_IsException(pdesc->value)) {
                    ret = -1;
                    break;
                }
            }

            ret = 1;
            break;
        }
    }

    JS_FreeCString(cx, (char *) name.data);
    return ret;
}


static int
ngx_qjs_shared_array_own_property_names(JSContext *cx, JSPropertyEnum **ptab,
    uint32_t *plen, JSValueConst obj)
{
    int                      ret;
    JSAtom                   key;
    JSValue                  keys;
    ngx_js_shared_array_t   *array;
    ngx_shm_zone_t          *shm_zone;
    ngx_js_main_conf_t      *conf;

    keys = JS_NewObject(cx);
    if (JS_IsException(keys)) {
        return -1;
    }

    conf = ngx_qjs_main_conf(cx);

    for (array = conf->shared_arrays; array != NULL; array = array->next) {
        shm_zone = array->shm_zone;

        key = JS_NewAtomLen(cx, (const char *) shm_zone->shm.name.data,
                            shm_zone->shm.name.len);
        if (key == JS_ATOM_NULL) {
            return -1;
        }

        if (JS_DefinePropertyValue(cx, keys, key, JS_UNDEFINED,
                                   JS_PROP_ENUMERABLE) < 0)
        {
            return -1;
        }
    }

    ret = JS_GetOwnPropertyNames(cx, ptab, plen, keys,
                                 JS_GPN_STRING_MASK | JS_GPN_ENUM_ONLY);

    JS_FreeValue(cx, keys);
    return ret;
}


/* ngx.sharedArray getter */
static JSValue
ngx_qjs_ext_ngx_shared_array(JSContext *cx, JSValueConst this_val)
{
    return JS_NewObjectProtoClass(cx, JS_NULL, NGX_QJS_CLASS_ID_SHARED_ARRAY);
}


/* Initialization function called from ngx_js_shared_dict module */
ngx_int_t
ngx_js_shared_array_init(JSContext *cx)
{
    JSValue  ngx_obj;

    /* Register class */
    if (JS_NewClass(JS_GetRuntime(cx), NGX_QJS_CLASS_ID_SHARED_ARRAY,
                    &ngx_qjs_shared_array_class) < 0)
    {
        return NGX_ERROR;
    }

    /* Add sharedArray getter to ngx object */
    ngx_obj = JS_GetGlobalObject(cx);
    if (JS_IsException(ngx_obj)) {
        return NGX_ERROR;
    }

    ngx_obj = JS_GetPropertyStr(cx, ngx_obj, "ngx");
    if (JS_IsException(ngx_obj)) {
        return NGX_ERROR;
    }

    JS_SetPropertyFunctionList(cx, ngx_obj,
                               ngx_qjs_ext_ngx_shared_array_props,
                               njs_nitems(ngx_qjs_ext_ngx_shared_array_props));

    JS_FreeValue(cx, ngx_obj);

    return NGX_OK;
}

#endif /* NJS_HAVE_QUICKJS */

Step 5: Add Public Init Function to Header

File: nginx/ngx_js_shared_array.h

Add at end before #endif:

#if (NJS_HAVE_QUICKJS)
#include <quickjs.h>

/* Initialize QuickJS class and register with ngx object */
ngx_int_t ngx_js_shared_array_init(JSContext *cx);

#endif

#endif /* _NGX_JS_SHARED_ARRAY_H_INCLUDED_ */

Step 6: Hook into Shared Dict Module Init

File: nginx/ngx_js_shared_dict.c

Add include at top (after line 11):

#include "ngx_js.h"
#include "ngx_js_shared_dict.h"
#include "ngx_js_shared_array.h"   /* ADD THIS */

In ngx_qjs_ngx_shared_dict_init() function (around line 4220), add near the end before return:

static JSModuleDef *
ngx_qjs_ngx_shared_dict_init(JSContext *cx, const char *name)
{
    // ... existing initialization code ...

    JS_SetClassProto(cx, NGX_QJS_CLASS_ID_SHARED_DICT_ERROR, proto);

    // ADD THIS:
    if (ngx_js_shared_array_init(cx) != NGX_OK) {
        return NULL;
    }

    return ngx_js_module(cx, name, ngx_qjs_ext_ngx,
                         njs_nitems(ngx_qjs_ext_ngx));
}

Step 7: Register Directives in nginx Modules

File: nginx/ngx_http_js_module.c (around line 420)

static ngx_command_t  ngx_http_js_commands[] = {
    // ... existing directives ...

#if (NJS_HAVE_QUICKJS)
    { ngx_string("js_shared_array"),
      NGX_HTTP_MAIN_CONF|NGX_CONF_1MORE,
      ngx_js_shared_array_zone,
      NGX_HTTP_MAIN_CONF_OFFSET,
      0,
      &ngx_http_js_module },
#endif

    ngx_null_command
};

File: nginx/ngx_stream_js_module.c (similar location)

static ngx_command_t  ngx_stream_js_commands[] = {
    // ... existing directives ...

#if (NJS_HAVE_QUICKJS)
    { ngx_string("js_shared_array"),
      NGX_STREAM_MAIN_CONF|NGX_CONF_1MORE,
      ngx_js_shared_array_zone,
      NGX_STREAM_MAIN_CONF_OFFSET,
      0,
      &ngx_stream_js_module },
#endif

    ngx_null_command
};

Testing

nginx Configuration

js_engine qjs;
js_shared_array zone=counter:1m;
js_shared_array zone=buffer:256k;

http {
    js_import test.js;

    server {
        location /counter {
            js_content test.incrementCounter;
        }

        location /read {
            js_content test.readBuffer;
        }
    }
}

JavaScript Test Code

function incrementCounter(r) {
    var sab = ngx.sharedArray.counter;
    var view = new Int32Array(sab);

    // Atomic increment
    var oldValue = Atomics.add(view, 0, 1);

    r.return(200, "Counter: " + (oldValue + 1));
}

function readBuffer(r) {
    var sab = ngx.sharedArray.buffer;
    var view = new Uint8Array(sab);

    // Read first 10 bytes
    var data = Array.from(view.slice(0, 10));

    r.return(200, JSON.stringify(data));
}

export default { incrementCounter, readBuffer };

Test Plan

1. Basic allocation: Verify SharedArrayBuffer is created with correct size
2. Multiple zones: Test multiple js_shared_array directives
3. Atomics operations: Test Atomics.add(), Atomics.load(), etc.
4. Multi-worker: Verify data is shared across nginx workers
5. Reload: Verify data persists across nginx -s reload
6. Enumeration: Test Object.keys(ngx.sharedArray)
7. Error handling: Test accessing non-existent zone

Integration Test File

File: nginx/t/js_shared_array.t (new)

#!/usr/bin/perl

use warnings;
use strict;

use Test::More;

BEGIN { use FindBin; chdir($FindBin::Bin); }

use lib 'lib';
use Test::Nginx;

###############################################################################

select STDERR; $| = 1;
select STDOUT; $| = 1;

my $t = Test::Nginx->new()->has(qw/http/)
    ->write_file_expand('nginx.conf', <<'EOF');

%%TEST_GLOBALS%%

daemon off;

events {
}

http {
    %%TEST_GLOBALS_HTTP%%

    js_engine qjs;
    js_shared_array zone=test:1m;

    js_import test.js;

    server {
        listen       127.0.0.1:8080;
        server_name  localhost;

        location /init {
            js_content test.init;
        }

        location /increment {
            js_content test.increment;
        }
    }
}

EOF

$t->write_file('test.js', <<EOF);
function init(r) {
    var sab = ngx.sharedArray.test;
    var view = new Int32Array(sab);
    Atomics.store(view, 0, 0);
    r.return(200, "initialized");
}

function increment(r) {
    var sab = ngx.sharedArray.test;
    var view = new Int32Array(sab);
    var val = Atomics.add(view, 0, 1);
    r.return(200, (val + 1).toString());
}

export default { init, increment };
EOF

$t->try_run('no QuickJS or shared array support')->plan(3);

###############################################################################

like(http_get('/init'), qr/initialized/, 'initialize shared array');
like(http_get('/increment'), qr/1/, 'first increment');
like(http_get('/increment'), qr/2/, 'second increment');

###############################################################################

Files Modified/Created

File	Action	Lines	Purpose
nginx/ngx_js_shared_array.h	CREATE	~35	Structure definitions and public API
nginx/ngx_js_shared_array.c	CREATE	~200	Core implementation (QuickJS only)
nginx/ngx_js.h	MODIFY	+2	Add class ID and main config field
nginx/ngx_js_shared_dict.c	MODIFY	+2	Include header and call init
nginx/ngx_http_js_module.c	MODIFY	+7	HTTP directive registration
nginx/ngx_stream_js_module.c	MODIFY	+7	Stream directive registration
nginx/t/js_shared_array.t	CREATE	~80	Integration tests

Total: ~333 lines of code (235 new + 18 modified + 80 tests)

Critical Implementation Notes

1. All code wrapped in #if (NJS_HAVE_QUICKJS): Not supported in njs engine yet
2. Class ID for exotic container only: NGX_QJS_CLASS_ID_SHARED_ARRAY is for the ngx.sharedArray object, NOT for SharedArrayBuffer (which is a built-in QuickJS type)
3. Separate files for modularity: New ngx_js_shared_array.{c,h} instead of adding to ngx_js_shared_dict.c
4. No cleanup function: free_func = NULL in JS_NewArrayBuffer()
5. Entire zone is backing store: Simple 1:1 mapping
6. No slab pool needed: Unlike dict, no dynamic allocation
7. Zero-initialize: Required for predictable behavior
8. Preserve on reload: Essential for SharedArrayBuffer semantics
9. Atomic access required: JavaScript must use Atomics for safety

Future Enhancements (Out of Scope)

1. Support for njs engine (requires different SAB implementation)
2. Multiple arrays per zone (requires metadata structure)
3. Named TypedArray views (e.g., zone=foo:1m:Int32Array)
4. Integration with existing js_shared_dict zones
5. State persistence across restarts

References

QuickJS API

• JS_NewArrayBuffer(): quickjs/quickjs.h:862
• Exotic class methods: QuickJS documentation

Pattern References (from ngx_js_shared_dict.c)

• Exotic class definition: ngx_qjs_shared_class (line 484)
• Property lookup handler: ngx_qjs_shared_own_property() (line 3030)
• Property enumeration: ngx_qjs_shared_own_property_names() (line 3082)
• Zone initialization: ngx_js_dict_init_zone() (line 2745)
• Directive handler: ngx_js_shared_dict_zone() (line 2819)
• Module initialization: ngx_qjs_ngx_shared_dict_init() (line 4133)

New Files Created

• Implementation: nginx/ngx_js_shared_array.c
• Header: nginx/ngx_js_shared_array.h