NaaN in Production

NaaN in Production

The largest NaaN-based production system to date is AskHow, a vertical video microcourse educational platform with AI-based video editing. Its browser-based web apps, backend servers, and video processors run NaaN. This article explains how AskHow utilizes NaaN in a commercial software product.

Choosing NaaN for AskHow

There are several reasons why we selected NaaN for AskHow:

• NaaN is has great support for high latency web-oriented workloads using fiber^[1] concurrency. Concurrency is more important than raw execution speed in web software because most processing is dominated by waiting for network requests to complete.
• NaaN supports rapid development cycles with fully integrated tooling. One button builds a project, starts a simulator, and runs your software under a debugger. Typically in less than 5 seconds. Another button can publish a new version (with suitable credentials.) This speed and simplicity is especially helpful fore the frequent iterations needed to build a new platform.
• NaaN has the same language and runtime in both frontend and backend. It saves a lot of time to share the same code and libraries in every environment.
• There is a GUI-based debugger, REPL, and shell access for any running NaaN instance, even inside a Docker container in a VM in a cloud. Appropriate access credentials are required.
• Remote evaluation enables transparent deployment of execution to worker threads, remote VMs, even in different clouds. This is the basis for remote debugging, but is also used to delegate execution to other threads or VMs, and to relocate SSL endpoints within a network fabric.

There were also other reasons for the choice:

• Technical capabilities aren't everything. It's essential that the engineering team be intimately familiar with the language and tools. AskHow and NaaN share the same founding engineer, so familiarity is assured.
• The language and tooling also need to be well supported, with rapid resolution of critical defects. As the most important customer, AskHow could count on top-notch support for NaaN.
• After several years of development NaaN needed a commercial software product to prove its performance and stability under real-world conditions. The development of AskHow exposed design errors and obscure defects in NaaN, helping it evolve into a solid platform.

A note about fibers vs threads

Most programmers believe that high performance systems require multithreading to operate at maximum speed. Intuitively this makes sense because execution must sometimes wait for a network request to complete, or to avoid unexpected delays due to processing a large chunk of data. Preemption promises to avoid these challenges by reallocating the CPU to a waiting thread. That's the theory, but in practice multithreading can be more cost than benefit.

The AskHow API server operates like this:

1. Start by receiving an incoming API request
2. Authenticate the session header or access credentials
3. Parse the JSON to obtain request parameters and data
4. Call the appropriate endpoint code, which will...
5. Make subsidiary requests to cloud services like databases
6. Wait for subsidiary requests to complete
7. Format the JSON response
8. Respond to the API request and End.

NaaN code only executes between Start and Wait, and then after waiting to End. In a multithreading system, if the execution time is shorter than the time slice then it will never be preempted. If the code is never preempted then you can eliminate overhead by not using threads at all.

Even without threads, execution must be suspended for network requests. This is trivial with NaaN concurrency. When a fiber waits its current continuation^[2] is saved, and when it resumes the continuation is restored. Many fibers can be waiting simultaneous with little overhead.

NaaN concurrency is so transparent that you can set a breakpoint in a running server and step through a specific request while it continues to serve other requests. See the NaanIDE Remote Debugging section below for more.

Implementation

This section offers a flavor of how AskHow uses NaaN. Code demographics shows the amount of code proprietary code, which is followed by a couple specific coding examples drawn from the implementation.

Code Demographics

AskHow utilizes a fairly conventional architecture but the NaaN-based implementation is much simpler and more efficient than many alternatives. Here are the code demographics:

This is only NaaN and AskHow code, not including the NPM library dependencies: bcryptjs, body-parser, emailjs, express, jsonwebtoken, leveldown, ms, pouchdb, and ws.

A lightweight, efficient backend server

The backend server handles incoming API requests concurrently on a single thread. Each API request becomes a separate execution fiber in the server.

For many operations stack-based storage is sufficient, but library functions often require fiber-specific global state. For example, logging needs to associate messages with the executing request, but it would be extremely awkward to pass request information as a parameter everywhere that logging is performed. This is better done by a global variable.

NaaN supports Lisp-style dynamically scoped variables which are local to each fiber, so it is easy to define an ID for each request. The following utility function wraps a given procedure invocation to maintain a global requestID having a unique value for each fiber.

/*
 * callRequestID
 *
 * Call a procedure with a new requestID and then return it.
 * Usage: callRequestID(procedure, args...)
 *
 */

 1 closure callRequestID args {
 2    let (proc, arglist) {
 3        let& (requestID) {
 4            try {
 5                apply(proc, arglist)
 6            } catch {
 7                debuglog("request exception:", exception) }
 8            printFlush()
 9            requestID
10        } (UUID())
11    } (pop(args), args) }

Here is a description of the code:

1. The closure parameter args receives a list of arguments specified by the caller.
2. The let block parses the incoming argument list into the proc and the args by initializing at line 11.
3. The special form let& dynamically binds the global symbol requestID during its scope. The initialized value is generated by UUID() on line 10.
4. The try...catch block executes the request using the provided procedure while catching all exceptions.
5. apply simply applies arguments arglist to procedure proc and returns the result.
6. The catch block does not specify a condition, so it catches all exceptions.
7. debuglog() writes a message to the log at the debug log level.
8. The printFlush() call ensures that all buffered output text is flushed to the output so that it is captured by the logs, etc. before the request completes.
9. requestID evaluates to its value, giving the caller the generated value.
10. UUID() is a library function providing a unique ID.
11. NaaN evaluates arguments strictly left-to-right, so pop(args) removes the first item in the list and returns it. Then args evaluates to the rest of the list.

This makes it very easy to execute a request as shown in the following procedure:

/*
 * requestHandler
 *
 *     Process a request received through the RestServer,
 * logging the result.
 *
 */

 1 closure requestHandler(request, response, local rid) {
 2     rid = callRequestID(doRequest, request, response, Date())
 3     bindLogLinesToRequest(rid) }

Here is a description of requestHandler:

1. The closure parameters request and response are provided by the REST server that handles incoming API requests. The rid local variable is the request ID generated within callRequestID()
2. This simply calls doRequest with the REST parameters, along with a timestamp, with the requestID context of course.
3. During request processing, any logs generated are recorded in an array associated with the current requestID. After the request is completed the bindLogLinesToRequest(rid) call emits these accumulated log items to the cloud logging infrastructure.

Pre-signed requests

AskHow accepts random video files uploaded by users, which must be optimized for mobile delivery. For example, some video formats store critical playback information at the end of the file. This causes long delays in playing the video because the entire file must be loaded before playback can begin. This challenge is mitigated by processing all videos using HandBrakeCLI running in a high-performance VM spawned for the purpose. HandBrake is a front-end for ffmpeg that includes presets and certain performance improvements.

When the AskHow server perceives the need for optimizing a video, it enqueues a work item that causes VMs to be spawned as needed to run the video processor. The video processor calls back into the server with status updates, but this requires separate credentials from the original user request. The solution is to generate a pre-signed API request to update status, and include that in the work item.

An alternative to pre-signed requests would be to establish a static trust relationship between the video processor and the server, but that is less secure and would not work for third-party services. Pre-signed requests can be used for a variety of purposes requiring minimal trust.

The implementation uses two simple functions to sign and verify requests. Here is the first:

    // apiMakeReqSig
    //
    // Make a signature for a signed request. Only keys starting with "_"
    // participate so other fields can be changed changed without changing
    // the signature. This allows the requester to set arbitrary "safe"
    // parameters without recomputing signatures outside the server.
    //
 1    function apiMakeReqSig(payload, local fields, key, data) {
 2        fields = []
 3        for `(key, data) in payload
 4            if key.startsWith("_")
 5                fields.push(strcat(key, "-", data))
 6        HashSHA1(strcat(Credentials.internalApi.secret, ",", fields.join(",")))
       }

1. Function apiMakeReqSig accepts a "payload" dictionary and has some local variables.
2. The fields array accumulates those payload fields that should participate in the signature, leaving other fields changeable. This stands in contrast to the AWS approach of signing every field in a request. Some of the information we wish to convey in the request will not be known until later. One could argue that the special naming prefix is a hack, however it is very clear once explained.
3. This is a very standard Lingo (NaaN surface language) iterator that provides both key and data each time through the loop. Just key can be used when data is not also needed.
4. key.startsWith("_") is an example of JavaScript interop, where the methods of String, Number, etc. are available within NaaN.
5. This generates a unique string for each participating field that combines both key and data. The data must "print" as something that changes when the important content is different, such as a string or number. For example data {a:3,b:4} prints as "Dictionary{2}", which would allow the data to be changed without altering the signature. This function should probably generate an error in such cases.
6. HashSHA1 combines a server secret with the entire set of protected fields to form a unique hash that cannot be spoofed externally.

Here is how apiMakeReqSig is used:

    // apiSignRequest
    //
    // Return a signed request that can be invoked on us without
    // separate authentication. It must have an _endpoint field that
    // matches the endpoint used, or it will fail.
    //
 1    function apiSignRequest(request) {
 2        merge(request, {
 3            signature: apiMakeReqSig(request)
          })
      }

    // apiVerifyRequest
    //
    // Verify a signed request, returning true if valid.
    //
 4    function apiVerifyRequest(reqbody, path) {
 5        reqbody._endpoint == event.path
              && apiMakeReqSig(reqbody) == reqbody.signature
      }

1. The server uses apiSignRequest to generate a pre-signed request, given a request dictionary as the argument.
2. The merge library function merges each dictionary argument in turn from left to right, returning a new dictionary of the resulting keys.
3. Add the signature key using the previously-defined function.
4. Incoming requests with the pre-signed header are tested with apiVerifyRequest to ensure they are valid. The path is the API endpoint.
5. If the endpoint matches and the protected fields are not changed then the request is verified.

NaanIDE Remote Debugging

NaanIDE is the NaaN Integrated Development Environment, and the image above shows it running in the Chrome browser. The Run tab manages executing NaaN instances. This is particularly suited to managing a cloud deployment because it connects to each instance for real-time monitoring and source-level debugging in a GUI.

Here is a description of the callouts on the screenshot:

• 🅐 is a list of every known NaaN instance, organized by location:
  Local contains the web browser and worker thread instances for NaanIDE itself. Play workers are there for on-the-fly experiments.
  Host contains instances for NaanIDE's NodeJS backend. The TutorBackend 1.4.0-aws-dev instance is a REPL console for managing the server.
  TutorbeServer is a VM with several NaaN instances but only the dev server is connected here. Remote connectivity is configured in NaanIDE's Accounts & Locations on the Home tab.
• 🅑 is the list of NaaN modules and component that are active in the selected NaaN instance, which is the aws-tutorbe-dev server running inside a Docker container. AskHow's dev server is dedicated to the development team so they can experiment freely without damaging anything important.
• 🅒 is the source code display for the current selection in Components to the left. Source code with comments is decompiled on the fly so that matching source code need not be deployed with code to the running Docker containers. Line 3023 shows a breakpoint set to interrupt execution.
• 🅓 are the execution controls for the debugger, allowing for single-stepping into and over procedure calls, and stepping out of procedures or loops. Breakpoints and break-on-exception may be disabled here.
• 🅔 shows execution status when a fiber is paused in the debugger, or when a waiting future is selected. Local variables in the current continuation can be examined and modified here.
• 🅕 displays the current virtual CPU load of this NaaN instance, along with the peak effective time slice. NaaN scheduling is cooperative so fiber execution is not preempted. If the peak time slice is too long then the code needs to be refactored. If a fiber executes for more than 10 seconds then an exception is thrown to recover.
• 🅖 A NaaN REPL allows normal CLI interactivity. Here the App global variable was evaluated, showing important server state. Note App.watchTimeout with a value of 30 seconds. A watchdog timer fails requests that take too long to complete.
  The !uptime expression is a shell escape that executes inside the server's Docker container.

Summary

The AskHow microcourse video platform demonstrates the strengths of the NaaN platform. The entire implementation is about 100K lines of proprietary code along with a small set of NPM libraries, which is quite small for a full-featured video platform. This shows the benefit of a simple, lightweight approach to web-based systems.

AskHow allows you to upload and share videos, organize them into a course, or use an AI video editing tool to segment a long video into a course of small lessons. Please try it out and see what you think!