NaaN for Experts

This article provides a high density overview of NaaN concepts for expert software developers familiar with JavaScript.

General Concepts

NaaN is a software platform that runs on JavaScript and provides additional capabilities. It is fundamentally a LISP dialect, but with a curly braces surface language to make it more familiar to modern programmers. Key concepts for NaaN are:

• NaaN uses an interpreter implemented in JavaScript, with transparent interoperability. This gives it full access to the JavaScript ecosystem including browsers, NodeJS, NPM, and tooling.

• NaaN is based on a fixed set of primitive data types and operations, where higher-level functionality is composed from these primitive elements.

• A NaaN runtime instance is an environment of code and data. A JavaScript runtime instance can contain multiple NaaN instances that operate in the same address space. Each worker thread in browsers or NodeJS utilizes a separate NaaN instance.

• All code in NaaN is data, and as with LISP the NaaN interpreter evaluates expressions to achieve the desired behavior and return desired results.

• NaaN is designed to support multiple languages, and the language layer is not required for NaaN to operate. NaaN's initial is Lingo with source files having a .nlg extension. An earlier language named Argot was useful for bootstrapping the system but is no longer maintained.

Execution

• NaaN implements an execution environment with a preemptable foreground and multiple background fibers. Foreground code can execute continuously without causing the timeout errors one sees in JavaScript. Background code, e.g. an event handler, runs on fibers that suspend and resume execution as needed for asynchronous operation.

• A NaaN instance may be saved and loaded, which includes the current foreground execution stack but does not include background fibers. When an instance is loaded, foreground execution proceeds as of the point it was saved. Cacheing this runtime state dramatically improves startup performance.

• NaaN's nonce data type is an ephemeral dictionary that is not saved or loaded and is useful for storing state for events and i/o operations.

• Nonces are used as the synchronization mechanism between fibers. For example, one fiber can wait on a nonce until it is signaled by another fiber. A nonce can be repeatedly waited on, signaled, and reset.

• A NaaN future specifies a function call that will be invoked after a designated delay. Futures may also be unscheduled. Futures are a convenient way to initiate asynchronous multiprocessing. Creating a future returns a nonce that can be used to reschedule the future, and that is signaled when the future is triggered.

• Like JavaScript, NaaN employs run-to-block cooperative scheduling. The current execution continues until it returns from an event handler or waits on a nonce. For example, if ten futures are initiated, the current fiber continues to execution, and only after it blocks will the futures begin to execute.

• NaaN interoperates with JavaScript promises and async/await. The NaaN core library function await(promise) pauses the current fiber until the promise resolves, and then returns the result.

• NaaN's core library includes an asyncArray() function that asynchronously applies a procedure to the elements of an array, N at a time. As each one completes another fiber begins until all are complete.

• NaaN's running framework includes anExecutorParallel() function that provisions a specified number of worker threads and evaluates expressions in the context of that worker. Arguments and return values are transparently marshalled and transferred to the worker thread. Along with asyncArray, this is used by the NaaNIDE build system to speed up execution by running in parallel.

Namespaces

• A NaaN instance is divided into multiple namespaces, and all execution occurs within a namespace associated with that code. Different namespaces are independent and can have entirely different configurations.

• At all times one namespace is globally active, with an inheritance chain of parent namespaces. When scanning source code text, symbols are sought first in the active namespace and then up the chain, creating a new symbol in the active namespace if it does not already exist.

• Crucially, code executing in each namespace can only modify its own namespace and the active namespace. This eliminates numerous types of inappropriate coupling, and helps programmers stay disciplined in observing the application structure.

• Creating a new namespace makes it active at the front of the inheritance chain. This allows the code that creates a namespace to modify it, even though it is executing in a different namespace.

• NaaN's standard library includes a sudo function allowing expression evaluation to modify any namespace, but this is intended for development and can be eliminated from production deployments.

• Symbol, dictionary, object, and procedure datatypes are associated with a namespace.

Data types

• NaaN strings and arrays are identical to JavaScript strings and arrays, with all of the same standard member functions, such as .length, .concat, etc.

• NaaN transparently supports the Date, RegExp, Number, Math, String, Array, and Promise global objects. For example, Number.isNaN() recognizes floating point errors, while Math.sqrt() is the expected square root function.

• The default NaaN numeric type is integer, which are bigints. NaaN also supports float which are identical to JavaScript numbers.

• NaaN numeric literals distinguish between integers and floats. The token 1 is an integer, while 1.0 is a float. Literal floats must include a decimal point followed by a digit or use an exponent suffix. Unlike JavaScript, hex constants retain their formatting so #ffff is not converted to decimal.

• Unlike JavaScript, NaaN does not perform implicit type conversion between strings and numbers, or to Booleans. Zero, NaN, and the empty string are not falsey in NaaN. The false values are limited to false, undefined, and null.

• NaaN separates the functionality of JavaScript objects into two different data types. A NaaN dictionary is a generalized key/value data collection type with no inheritance or member functions. A NaaN object is a primitive element for implementing classes and instances, and supports inheritance, wild cards, etc.

• NaaN has a symbol data type that represents a variable. A symbol has a mutable value and an annotation dictionary ("property list" in LISP) for storing metadata.

• Like LISP, NaaN provides a tuple datatype comprising "cons" cells with car and cdr pointers and dotted pairs. Unlike LISP, NaaN tuples are immutable. NaaN tuple literals start with a backquote ` symbol instead of the ' single quote customary in LISP. This is because NaaN needs to stay consistent with JavaScript's use of single quotes to delimit literal strings.

• Like LISP, NaaN considers everything that is not a tuple as an atom, even if it is a collection, like an array.

• NaaN supports weakMap as a non-enumerable NaaN dictionary with the special property of weak references.

Lingo Syntax

• Lingo uses a braces-style syntax similar to JavaScript. You can also enter LISP s-expressions. For example: eval(`(* 6 7)) => 42

• Lingo has no statements. A Lingo program is a series of expressions, separated by newline and/or a terminator such as ;. Because a procedure definition consists of multiple lines, you cannot use a semicolon as a line ending as is optional in JavaScript. If you need a separator, use , (comma).

• NaaN adds an additional indirection operator .* which lists the key names of a dictionary or object: { a:3, b:4 }.* => ("a", "b")

• Symbol annotations use the @ operator for lookup and assignment, e.g. house@color = red for the symbol house.

• Symbol annotations can be listed with the @* operator. For example, it can list the inbuilt functions on a typename: string@* => ("append", "atom", "interned", "length", "string", "tostring", "unicode", "typeof")

• NaaN allows literal references with the . operator. For example, you can use div.classList.'type-check' in addition to JavaScript's div.classList['type-check']. You can also use numeric literals with arrays and tuples: [a, [b, c], d].1.1 => c

• The parentheses around an if or while expression are optional.

• Control structures are: if [...else], loop, while, for...in..., break, continue, return, return (...)

REPL Hints

• The NaaN REPL offers "slash commands" that can be useful. Use /help for a summary.

• The NaaN REPL prompt lists the namespace followed by the language, so Play-lingo> is telling you that you are in the Play namespace using the Lingo language. If you select something like /ch common then the prompt will change to common-lingo to reflect your new namespace.

• If an exception occurs when the GUI debugger is not available the debugger prompt (ndb-1) will display, indicating that you are one level into the debugger. You can type help for a list of debugger commands. To escape the debugger, either type c [return] to continue or q [return] to terminate the current callstack and restart.

• Typing Control-C in the REPL will enter the GUI debugger if available, or will write ^C and enter NDB. NaaN tries hard to respond to ^C, so it's a good thing to try if things become unresponsive.

• Command-K / Control-K clears the REPL screen and scrollback. Up and down arrow traverses the command history.

• The ; terminator prints the result of evaluating the expression in the REPL, which is the default if there is no terminator. The ;; expression does not print the result. The ;> terminator prints the LISP S-expression form without Lingo formatting.

• In the REPL, the $ symbol is assigned the result of evaluating each expression, so that you can easily reference the previous result.

• Adjacent string literals are automatically coalesced in NaaN. To keep strings separate use a comma between them.

• The browser terminal in NaaNIDE currently uses Google hterm. To copy text from the scrollback, select the desired text by dragging or click/shift-click, and the data will be automatically copied to the clipboard.

• In NodeJS, the standard terminal escapes apply, so .exit will quit NaaN and return to the shell.

Error Handling

• The NaaN core library provides a Error() function that creates an error dictionary from a text message, optional arguments, and any underlying error(s). Errors are pure data that is serializable as JSON, and thus can be sent over a network connection or passed around freely.

• NaaN supports multiple value procedure returns and assignment destructuring. This is used for the "standard result tuple" convention, where either an error or data are the return value of a procedure. For example:

	`(error, data) = files.ReadFile(path, mode)

• NaaN supports exceptions with throw and try-catch-finally constructions, however the convention is to use standard result tuple swhenever possible.

• Note that NaaN orders result tuples as error, data instead of the golang convention where the data is first. This is because JavaScript callbacks invariably place the error first.

Procedures

NaaN procedures are available in four common forms:

• function - Lightweight code where parameters and local variables have only local lexical scope. They simply take arguments and return result(s). Every execution of a given function uses the same variables, so a local variable symbol returned by the function will not retain its value.

• closure - Executing a closure creates a new copy of its arguments and local variables, and these persist until garbage collected. All nested functions defined within a closure will refer to the same instance of local variables.

• let - A let or "let block" defines a scope with its own set of local variables, which can be used in the middle of a larger procedure. A let is immediately executed when encountered, unlike the other procedures that can be stored as a reference.

• macro - A macro is a call-by-name function, where the arguments are not evaluated before executing the macro's code. This allows manipulation of the symbols passed by argument. Macros are not usually required but can be very useful for certain tasks.

Additional notes on procedures:

• Each of the procedure forms can be nested within the other forms. Typically, a helper function is nested within a larger function because the helper is called multiple times, and yet would not be useful if shared more broadly. Factory procedures and objects with methods should also use closures.

• Functions are the most efficient procedure but can be problematic with fibers because the local variables end up being shared inappropriately. Procedures implementing async operations usually should be closures instead. Leaf procedures that do not call other procedures, even indirectly, can always be functions.

• Note that in many languages, the term closure refers to a function with persistent access to its enclosing execution environment. In NaaN, the closure is the container for that persistent environment, and anything referenced outside the container is not part of the closure.

• NaaN procedures can define either a list of parameters and local variables, or specify a single symbol to receive a list of the formal arguments for each call. In this case there is usually a nested procedure to defined local variables, and code to parse the list and determine how to structure the variable arguments provided. This can be considered a "programmable calling protocol".

• Symbols have an optional procedure binding that defines code that is executed when the symbol is used in a procedure call context. Procedures can be bound to a symbol or executed inline as a lambda (unnamed) procedure.

• The procedure binding of a symbol can be retrieved with <symbol>.proc

Modules, Components, Libraries, Frameworks, and Plugins

• The standard structure for a NaaN program is a set of modules, each containing components. A module typically correlates to a folder of source code, where each component is a file within that folder.

• Modules are containers for components sharing a single namespace. One component in the module is the init component and is required. The other components are optional and loaded on demand, or when needed as a prerequisite for another component. The init component base filename is the same as the module folder name.

• The standard NaaN JavaScript (NPM) package includes a core runtime library that is always available. This defines the basic functionality of the system, such as the arithmetic operators and the module management primitives.

• NaaN frameworks are inbuilt but optional library functions for NaaN programs. For example, the frequently used common framework provides functions that abstract the differences between browser and NodeJS environments. The less used storage framework provides functions for data persistence and access to local and remote filesystems. Most of the NaaNIDE core functionality below the UI is actually implemented by NaaN frameworks.

• NaaN plugins provide integration features for outside services. For example, the serviceAWS plugin provides access to the AWS cloud.

Packages and Persistence

• NaaN has a broad range of persistence features including NaaN packages, JS packages, and saved state. All of these are always available through the core runtime library.

• The NaaN package manager encodes and decodes NaaN data structures from a running NaaN instance so they can be serialized and later restored in another running NaaN instance. The current interpreted implementation is relatively slow but maintains referential integrity.

• JS packages correspond to source files and encode the structure of NaaN objects into primitive JavaScript types. A package is restored by loading into the interpreter using low-level hooks. JS packages can be loaded before the interpreter is actually live, or in most cases after it has started. They are quite fast.

• NaaN supports saving and loading the entire state of an instance. The state is stored as a single string, which can be quite large--and quite slow, if there is a lot of data involved.

Deployment Structure

• The standard NaaN JavaScript (NPM) package is @naanlang/naan. It is stand-alone, incorporating within it any needed dependencies. The core runtime, including the interpreter, is available in a single JavaScript file. This core runtime is always required to use NaaN and includes support for REPL, the Lingo language, a simple NaaN DeBugger (NDB), etc.

• Outside of the core runtime are a series of environment (env_xxx.js) files that create the appropriate host environment for NaaN's core runtime. These interface with terminals and the runtime environment, and there are versions for AWS lambda functions, for worker threads in browsers and in NodeJS, for web applications, etc.

• Typical startup involves running the environment file that sets up loading the NaaN interpreter and connecting it to the environment, followed by executing a customized NaaN startup file that loads the program's initial module.

Arcana: Dynamic Binding

• NaaN supports dynamic binding with the special forms function& and let&. Under dynamic binding, the parameters and local variables of the function are actually global symbols in the namespace. When the function is invoked, the current values of the parameters and local variables are saved on the runtime stack and the formal arguments bound to these symbols. When the function returns the symbols are unbound and the original values restored.

• Dynamic binding works well for small amounts of code because values can be passed among functions without being explicitly specified as arguments and parameters. Global variables can be safely overridden for the duration of a single function call.

• Dynamic binding does not work well when code gets complex because it becomes difficult or impossible to correlate the value of a variable with the code responsible for setting that value. Still, dynamic binding can be safely used within a containing procedure, when its local variables are being dynamically bound. This restricts the scope over which side effects can occur.