# matdbg

1. [User Instructions](#user-instructions)
1. [Architecture Overview](#architecture-overview)
1. [C++ Server](#c-server)
1. [JavaScript Client](#javascript-client)
1. [HTTP Requests](#http-requests)
1. [WebSocket Messages](#websocket-messages)
1. [Wish List](#wish-list)
1. [Screenshot](#screenshot)
1. [Material Chunks](#material-chunks)

## Setup for Desktop

First set an environment variable as follows. In Windows, use `set` instead of `export`.

    export FILAMENT_MATDBG_PORT=8080

Next, launch any app that links against a debug build of a Filament and point your web browser to
http://localhost:8080. Skip ahead to **Debugger Usage**.

## Setup for Android

Rebuild Filament for Android after enabling a CMake option called FILAMENT_ENABLE_MATDBG. Note that
CMake is invoked from several places for Android (both gradle and our easy build script), so one
pragmatic and reliable way of doing this is to simply hack `CMakeLists.txt` and
`filament-android/CMakeLists.txt` by unconditionally setting FILAMENT_ENABLE_MATDBG to ON.

After rebuilding Filament with the option enabled, ensure that internet permissions are enabled in
your app by adding the following into your manifest as a child of the `<manifest>` element.

    <uses-permission android:name="android.permission.INTERNET" />

Now launch your app as usual. The Filament Engine sets up a server that is hardcoded to listen to
port 8081. Next, you will need to forward your device's TCP port 8081 to your host port of choice.
For example, to forward the matdbg server on your device to port 8081 on your host machine, do the
following:

    adb forward tcp:8081 tcp:8081

This lets you go to http://localhost:8081 in Chrome on your host machine.

## Debugger Usage

After opening the matdbg page in your browser, the usual first step is to select a material in the
upper-left pane. Sometimes you might need force your app to redraw (e.g. by resizing the window) in
order make the materials selectable.

The next step is to select an active (boldface) shader variant in the lower-left pane. This allows
you to view the GLSL, MSL, and SPIR-V code that was generated by `matc` or `filamat`.

In the sidebar, inactive shader variants have a disabled appearance, but they can still be examined
in the shader editor. The active status of each shader program is refreshed every second.

You can also make modifications to GLSL or MSL, so long as the shader inputs and uniforms remain
intact. After making an edit, click the `[rebuild]` button in the header. Note that your edits will
be lost after closing the web page.

## Keyboard Shortcuts

To save an edit, press **Cmd+S** (**Ctrl+S** on Linux/Windows) as an alternative to clicking
`[rebuild]`.

If the editor has focus, you can navigate between materials by holding **Shift+Ctrl** while
pressing the up or down arrow. Navigation between variants is similar, just use left / right instead
of up / down.

## Architecture Overview

The matdbg library has two parts: a C++ server and a JavaScript client. The C++ server is
responsible for instancing a [civetweb][1] context that handles HTTP and WebSocket requests. The
JavaScript client is a small web app that contains a view into an in-browser database of materials.

The WebSocket server receives push-style notifications from the client (such as edits) while
the HTTP server responds to material queries using simple JSON messages.

When a new WebSocket connection is established, the client asks the server for a list of materials
in order to populate its in-browser database. If the connection is lost (e.g. if the app crashes),
then the database stays intact and the web app is still functional. If a new Filament app is
launched, the client inserts entries into its database rather than replacing the existing set.

The material database is cleared only when the web page is manually refreshed by the user.

## C++ Server

The civetweb server is wrapped by our `DebugServer` class, whose public interface is comprised of a
couple methods that are called from the Filament engine:

- **addMaterial** Notifies the debugger that the given material package is being loaded into the
  engine.
- **setEditCallback** Sets up a callback that allows the Filament engine to listen for shader edits.
- **setQueryCallback** Sets up a callback that allows the debugger to ask for current information.

## JavaScript Client

The web app is written in simple, modern JavaScript and avoids frameworks like React or Angular. It
uses the following third-party libraries which are fetched from a CDN using `<script>`. This allows
us to avoid adding them to our git repo, and leads to good caching behavior.

- **mustache** Popular tiny library that converts template strings into HTML.
- **monaco** The engine behind Visual Studio Code.
    - We've configured this for C++ for somewhat reasonable syntax highlighting.
    - If desired we could extend the editor to better handle GLSL and SPIR-V.

All the source code for our web app is contained in a single file (`script.js`) and the mustache
template strings are specified using `<template>` tags in `index.html`.

The web app basically provides a view over a pseudo-database which is a just a global variable
that holds a dictionary that maps from material id's to objects that conform to the JSON described
below.

## HTTP requests

The server responds to the following GET requests by returning a JSON blob. The `{id}` in these
requests is a concept specific to matdbg (not Filament) which is an 8-digit hex string that hashes
the entire binary content of the material package.

---

`/api/matids`

Returns an array containing the id for each known material. Example:

```json
["e4c41141", "44ae2b62", "9dab8a03"]
```

---

`/api/materials`

Returns an array with all information (except shader source) for all known materials. Example:

```json
[{
    "matid": "e4c41141",
    "name": "uiBlit",
    "version": 4,
    "shading": { "model": "unlit", "vertex_domain": "object", ... },
    "raster":  { "blending": "transparent", "color_write": "true", ... },
    "opengl": [
        { "index": " 0", "shaderModel": "gl41", "pipelineStage": "vertex  ", "variantString": "", "variant": "0" },
        { "index": " 1", "shaderModel": "gl41", "pipelineStage": "fragment", "variantString": "", "variant": "0" },
    ],
    "vulkan": [],
    "metal": [],
    "required_attributes": ["position", "color", "uv0"]
},
{
    "matid": "44ae2b62",
    ...
}]
```

Some of the returned data may seem redundant (e.g. the `index` and `variantString` fields) but
these allow the client to be very simple by passing the raw JSON into [mustache][4] templates.
Moreover it helps prevent duplication of knowledge between C++ and JavaScript.

---

`/api/material?matid={id}`

Returns all information (except shader source) for a specific known material. The JSON response
is equivalent to one of the items in the top-level array in `/api/materials`.

---

`/api/active`

Returns an object that maps from material ids to their active shader variants. Example:

```json
{"b38d4ad0": ["opengl", 5] , "44ae2b62": ["opengl", 1, 4] }
```

---

`/api/shader?matid={id}&type=[glsl|spirv]&[glindex|vkindex|metalindex]={index}`

Returns the entire shader code for the given variant. This is the only HTTP request that returns
text instead of JSON.

The `type` field in the request selects the desired shading language, not the backend. For example,
for Vulkan it can select between SPIR-V or decompiled GLSL. Note that the original GLSL that was
used to create the SPIR-V is not available.

---

## WebSocket messages

Unlike HTTP requests, WebSocket messages can be pushed at any time and can travel in either
direction. In our homegrown protocol, every WebSocket message starts with a command that matches
\[A-Z\_]+ followed by a space character. Command arguments are delimited with spaces.

Currently we support only one command. It travels from client to server.

    EDIT [material id] [api index] [shader index] [shader length] [entire shader source....]

The `material id` is 8 hex digits.

The `api index` chooses between GL/VK/Metal and matches the values of the Backend enum (except that
zero is invalid).

The `shader index` is a zero-based index into the list of variants using the order that
they appear in the package, where each API (GL / VK / Metal) has its own list.

The `shader length` is the number of bytes required for UTF-8 encoding of the shader source string,
not including the terminating null.

## Wish List

- Allow SPIR-V edits.
- Allow viewing GLSL transpiled from SPIR-V.
    - Also stop piggybacking on `type=glsl` for Metal Shading Language.
- Expose the entire `engine.debug` struct in the web UI.
- When shader errors occur, send them back over the wire to the web client.
- Resizing the Chrome window causes layout issues.
- The sidebar in the web app is not resizeable.
- Refactor shader selection stuff to have "index" and "stage" attributes instead of glindex/vkindex/metalindex.
    - Alternatively do something similar to makeKey in `MaterialChunk`.
- For the material ids, SHA-1 would be better than murmur since the latter can easily have collisions.
- It would be easy to add diff decorations to the editor in our `onEdit` function:
     1. Examine "changes" (IModelContentChange) to get a set of line numbers.
     2. `shader.decorations = gEditor.deltaDecorations(shader.decorations, ...)`
     3. See [these monaco docs](https://microsoft.github.io/monaco-editor/playground.html#interacting-with-the-editor-line-and-inline-decorations).

## Screenshot

<img width="600px" src="https://user-images.githubusercontent.com/1288904/63553241-b043ba80-c4ee-11e9-816c-c6acb1d6cdf7.png">

## Material Chunks

This section exists only to provide a reference for the `ShaderExtractor` and `ShaderReplacer`
features.

The relevant chunk types are listed here. These types are defined in the `filabridge` lib, in
the `filamat` namespace.

```c++
enum UTILS_PUBLIC ChunkType : uint64_t {
    ...
    MaterialGlsl = charTo64bitNum("MAT_GLSL"),    // MaterialTextChunk
    MaterialSpirv = charTo64bitNum("MAT_SPIR"),   // MaterialSpirvChunk
    MaterialMetal = charTo64bitNum("MAT_METL"),   // MaterialTextChunk
    ...
    DictionaryGlsl = charTo64bitNum("DIC_GLSL"),  // DictionaryTextChunk
    DictionarySpirv = charTo64bitNum("DIC_SPIR"), // DictionarySpirvChunk
    DictionaryMetal = charTo64bitNum("DIC_METL"), // DictionaryTextChunk
    ...
}
```

### MaterialTextChunk

These chunks have the following layout.

    [u64] ChunkType magic string
    [u32] Remaining chunk size in bytes
    [u64] Shader count
    for each shader:
        [u8]  Shader model
        [u8]  Shader variant
        [u8]  Shader stage
        [u32] Offset in bytes from (and including) "Shader count" up to "Total string size"
    for each unique shader:
        [u32] Total string size (including null terminator)
        [u32] Number of line indices
        [u16 u16 u16...] Line indices

### MaterialSpirvChunk

These chunks have the following layout.

    [u64] ChunkType magic string
    [u32] Remaining chunk size in bytes
    [u64] Shader count
    for each shader:
        [u8]  Shader model
        [u8]  Shader variant
        [u8]  Shader stage
        [u32] Index into the blob list in DictionarySpirvChunk

### DictionaryTextChunk

These chunks have the following layout.

    [u64] ChunkType magic string
    [u32] Remaining chunk size in bytes
    [u32] Number of strings
    for each string:
        [u8 u8 u8 u8...] include null terminator after each string

### DictionarySpirvChunk

These chunks have the following layout.

    [u64] ChunkType magic string
    [u32] Remaining chunk size in bytes
    [u32] Compression
    [u32] Blob count
    for each blob:
        [u64] Byte count
        [u8 u8 u8 ...]

[1]: https://github.com/civetweb/civetweb
[2]: https://microsoft.github.io/monaco-editor/
[3]: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/template
[4]: https://mustache.github.io/
