How could an embeddable visual scripting language look like?

Once, during my work on the sadly never published game Reptoads – a round-based deterministic multiplayer cooperative card game, I got asked by different members of the art and design team: why can’t we have a visual scripting language such as Blueprints or Shader Graph for either the gameplay code or the visuals? My answer was always the same: we don’t have enough time to implement such a tool. This question, however, got stuck in my mind and I started searching for a “Lua” of the visual scripting languages just to find… nothing. Roughly three years later, the same question kept coming up in different projects, while my answer and my research results stayed the same. This made me curious, why is there no such thing as an embeddable visual scripting language that can be used?

Asking myself why is there no such language and how would one look like, led me to write my Master thesis with the title: “Investigation into the criteria of embeddability of visual scripting languages within the domain of game development”. In my research paper I composed a list of potential criteria that classify a visual scripting language (VSL) as embeddable. In this article I will go through these criteria and design a hypothetical embeddable VSL, which I called Noodle. This will be Part 1 of a series of articles focused on creating a usable VSL prototype.

You might think now: hold on for a second! Aren’t there lots of tools doing similar things? Do we really need to reinvent the wheel?

Yes, your instinct is right, there are some well-known solutions out there like Unreal Engine’s Blueprints or Unity’s Visual Scripting, but you simply cannot take their scripting runtime out and put this into your specific 3D retro shooter game engine. Now you might want to say: okay then, I will just use scripting, there are a lot of scripting languages out there (Lua, luau, wren, daScript, anglescript etc.). That argument is also valid but what happens when you look for a designer to help you with your indie game and all you can find are designers that have only worked with Unity or Unreal before. In those cases, it would be better to tell them that there is a language that works like Blueprints or is less complicated than C#, especially since it is easier to learn a VSL than a normal scripting language, as literature suggests [1]–[3].

Some thoughts about visual scripting

First of all, I need to acknowledge that visual scripting is not the silver bullet for all problems in game development and often does more harm than good, especially when it comes to performance and maintainability (Blueprints from Hell: Share The Horror). I want to stress that this article is not pro or against visual scripting languages but rather sees it as a tool, just as with any other programming language (yes, a VSL is a programming language). Of course, there might be other tools out there that do a specific job better but it always depends on the use case and the requirements. During my Masters in Game Technology, I had the chance to interview industry professionals and one of my participants emphasized that the requirements of the game are the most important factor when deciding what tech to choose when working on a game.

Another aspect to consider is what are we intending to use the visual scripting language for. During my research, all my interviewees stated that they see the benefits of a VSL in being able to quickly prototype features and its quick iteration times overall. Also, a VSL is perfectly made for certain tasks such as processing sequences. While for others it might be not the right tool such as performance-critical code passes, unless the graph is compiled to native code and heavily optimized. Moreover, VSLs are used for gluing systems together, as industry experts and literature suggest [$$]. In addition, VSLs are often used to handle UI events, for example in the UI workflow, but also for the game logic in general. They are also used as a domain specific language for workflows such as Shader / Material editing or state machines.

All these different aspects of possible use cases make it quite difficult to define the best-fit solution. Since it is not easy to determine the right requirements for each of those fields, the idea is to create a generalized model of a language that can be used for these different aspects. I am fully aware that this might not be the ultimate solution that fits all cases.

The basic language concepts

I will discuss the core concepts of Noodle by following the result structure of my paper. My paper found five major aspects that are potentially important when designing a language framework with embeddability in mind. My thesis defines them as follows: 1.performance and the identified subcategories, 2.mechanical aspects of embedding software and their respective subcategories, 3.license, 4.documentation, 5.tooling and workflow*.* While we look into each of these aspects, we define what consequences it has for Noodle.

General design overview

Before we dive deep into these findings, I will briefly describe the main characteristics of Noodle. Noodle will be diagram-based [1], [4] which means that the language is graph-based and the execution can be followed by traversing nodes. As my research shows, this is the most commonly used VSL form in game engines, e.g., in Unreal Engine, Unity and Godot [5]–[8]. A popular alternative is the block-based design. A block-based design is used for Scratch and Google’s Blockly [9]–[11] but it is not often used in games. Moreover, Noodle will offer pure nodes, nodes without side effects and nodes that may have side effects [12]. They are differently identified by either having an “execution” wire/connection or just a data flow connection. This allows the dialect designer to decide if they need support for one of these concepts or both, depending on the use case requirements. On its own, the language is not event-based, so a graph can be triggered at any node at any point in time by the runtime. However, the dialect designer can specify nodes that can be triggered. These nodes can be used as entry points which would allow an event-based structure if needed. All these decisions are based on what is common in tools used in the game industry, be it Godot’s visual scripting or Unreal Engine’s Material Editor [8], [13]–[17].

Noodle’s design idea of being a composable language that one can shape depending on the users’ needs, or how literature would call it “extending” by being an extension language, is greatly inspired by the works of Hisham Muhammad and Roberto Ierusalimschy in their paper about the used API design of the Lua programming language [18]–[21]. Since we are talking about a visual scripting language which logic is expressed in composing visual elements [22], we need some form of specification. In Noodle the dialect is described via a protocol: what nodes it supports, what kind of types are supported, etc. Moreover, Noodle supports internal modules, i.e., subgraphs that can help organizing your graph, as well as external modules, if a module resolve function is provided. If not, only internal modules will work. However, the defined protocol might differ from dialect to dialect, depending on the use of the language. We will talk about the protocol an its uses later.

The next few paragraphs will now take the findings of my study and describe the impact they might have on Noodle.

Performance

Development performance One of the most important factors for the game industry is the ability to perform quick iteration cycles and to be able to quickly prototype a feature [23, pp. 956–957] [20], [24]. Many interviewees have expressed the opinion that script hot reloading is a crucial feature they would expect from a VSL. It has been shown also in literature [23] that script hot reloading is a great feature, that supports quick iteration and rapid prototyping. Based on these observations, Noodle will support script hot reloading to empower users with quick prototyping.

Runtime performance When talking to any game programmer, they will most likely make a similar statement as many of my study participants, that memory is the number one bottleneck in games. Although, it greatly depends on the requirements of the game and how dramatic the runtime performance is influenced by the right memory access patterns or the right allocation strategies. Therefore, it is not surprising that most of the participants expressed their opinions on how memory management should work. The major conclusion is that in a VSL, we know from early on what kind of data we will handle since all inputs and outputs of all nodes are known at translation time, therefore the graph could either allocate a large chunk of memory and manage that like in WebAssembly where “…opcodes accessing memory are not given addresses, but offsets relative to the beginning of a linear memory segment whose size is always known.” The memory model of Noodle will follow the same mentality and will at the beginning allocate the needed memory through a memory allocator interface that the user can modify, if needed. If memory needs to be accessed in a form of a pointer, Noodle will not be able to do anything with these pointers, so it will just pass them down to native functions that are able to understand them.

When it comes to a scripting language and a VSL, which is nothing else than a subcategory [1], [25], the execution method is important. Robert Nystrom states in both of his books “Crafting interpreters” and “Game programming patterns” [26, p. 17], [27, pp. 155–179] that a tree-walking interpreter, a form of an interpreter that traverses the graph by recursively calling the nodes, is slower than compiling the graph down to bytecode that is executed in a virtual machine (VM) or transpiled to a different source code [26, pp. 16–20], [28], [27, pp. 155–179] ,[23, pp. 52,954-958]. Based on these findings, Noodle will compile to bytecode with the intention that the user can compile a Noodle graph representation to C and compile to native code depending on their platform as a last shipping step (if providing a backend). This transpiling or compiling to native code has been mentioned in the interviews as an important feature. It can also be considered to allow the language to enable hot patching of the native generated code. This means that if something is wrong with the compiled C code, a content update of a bytecode compiled graph can be used to hot patch this part of the code.

Mechanical aspects of embedding software

API design As described in the above-mentioned papers from H.Muhammad and R.Ierusalimschy about the API design of Lua and other scripting languages such as Perl or Python, it can be concluded that an API should be flexible to provide the ability to extend the underlaying language not in a verbose (like Perl) manner, but more in a concise declarative manner. This view is supported by my interviewees who describe that an API should be able to bind, for example, an external editor but should also be small enough to be easily manageable. Moreover, academia as well as the interviewed industry professionals argue that the API should be written in the C programming language or at least provide C foreign function interface, since C is considered as the lingua franca of programming languages [18]–[21]. To provide maximal portability from an API point of view, the header files of Noodle will be written in C99 and the implementation will be done in C11.

Dependencies The game industry is notorious for reinventing the wheel [29], [30] and this problem might stem from the platforms we are catering to and the software we are working with. Therefore, having many dependencies that we need to maintain and maybe port to different platforms is not desired. This is what my research suggests and what an industry professional states in an interview: “When you develop a commercial product, you also need to consider two things. Since dependencies might be taken offline at any time, it is very important to have your own copy of them. Also, for various certification on platforms you need to keep in mind that they perform security audits on those dependencies.”. These insights led to the decision for Noodle to have no external dependencies besides the OS dependencies on the platforms it supports.

License & documentation

The game industry caters for many different platforms with different requirements, some are open source, some are not. Therefore, the license needs to be permissive since it is not always possible to open-source certain aspects of the entire codebase due to NDA regulations. This is the reason why Noodle will use the MIT license or the Apache License, Version 2.0, depending on the user’s need.

Besides the license, a good documentation on how to embed the language into your game framework is needed. My thesis research has shown that it is expected by users for some form of online documentation and samples to exist. Therefore, Noodle should provide online documentation in form of a GitHub Page but also examples on how to bind the language.

Workflow & tools

So far, none of the categories of embeddability are really different from what one would expect from a regular scripting language. In fact, what literature says is that they are nearly matching, and Software Engineering Stack Exchange confirms it. The major difference explained in literature by B.Myers already in 1989 in “Taxonomies of Visual Programming” [1] and in the works of Nystrom and J.Gregory [23], [27], the biggest challenge for a VSL is the UI/UX aspect and mainly the visual scripting environment (VSE). A VSE can be seen as the integrated development environment (IDE) for VSLs. My thesis concluded that among all aspects, workflow and tooling are the most important.

Unfortunately, there is not much existing research on what qualifies a good visual scripting environment within the domain of game development. If we look outside of the game industry, there are a few papers on this topic, but they mostly describe the design of visual scripting environments for block-based languages, since they are mainly used for educational purposes. Although the industry professionals participating in my study shared some important insights, proper academic research would be needed to make academic claims. However, for this prototype the statements of industry professionals and the scarce UI / UX research on visual scripting environments will base the foundation for Noodle’s UI / UX.

The industry professionals stated that one cannot just use Notepad or Visual Studio to edit the “source code” of a VSL, therefore they would expect either a fully-fletched editor to come with the VSL or a flexible API to allow them to bind an editor themselves. One of the participants stated that they would use the provided editor to get acquainted with the language and then use the API to build their own that matches the paradigms and needs of their own ecosystem. Hence, Noodle will come with a prototype of an editor and a flexible API that allows the user to bind a custom editor.

For the first prototype, the language will come with a Visual Studio Code extension that enables the editor to understand Noodle files, while the editor will communicate with the Noodle runtime via the WebSocket API by using the protocol described later. Whenever a change occurs in the editor, the updated graph will be sent to the runtime and the runtime user can decide if it is needed to recompile the current graph and swap it with the new changes. The goal is to provide an embeddable editor view written in C or C++ as an external tool on top of the Noodle APIs.

The mentioned WebSocket-based API allows network communication to other software that might be the editor. The VS Code extension will be part of the first prototype to demonstrate this communication feature. A network-based communication might be more suitable for games that separate game runtime and editor runtime, while the API communication via the C interface might be better suited for custom game engines that have a built-in editor. Important to mention is that the WebSocket API is purely optional.

Since debugging and visualizing what is happening within the VSL has been classified as very important, Noodle should come with an ability to attach a debugger to it. In the first prototype we can try to make use of the VS Code Debugger framework and provide the ability to connect to it.

A Noodle document will be in JSON format that can be parsed with any JSON parser and can be checked in source control like a normal text file. The main reason why JSON has been chosen is that it provides a better source control management support. Also, it can be diffed without big issues, while for example a binary file format would cause source control issues (google ‘Blueprints and source control’).

The Noodle Protocol

As mentioned above, Noodle runs on a protocol that gives an overview of what nodes are available and what types can the runtime understand. This indicates that Noodle can be either statically or more dynamically typed, depending on the needs of the dialect designer. The Noodle protocol can be generated via the API and will return either a protocol struct that can be used via the C interface or a JSON representation that can be used otherwise in tools or in the WebSocket API.

Besides the Noodle protocol that describes the language, there is the noodle file or noodle document – the script file. One can see the protocol like a header file in C or C++ and a noodle file as the source file. The noodle document describes the current file the VM is processing. As mentioned above, each noodle file may contain subgraphs (internal modules) or if enabled, external modules. In principle, every noodle document contains three regions:

Protocol – this region describes the meta data of the current language dialect. The VM will check this region and verify that the name of the dialect, as well as the version, are matching. If not, the file cannot be passed. Migration can be implemented with this approach.

Graph – this region describes the actual data of the graph, which includes the modules, nodes, connections, and the data.

Editor – implementation defined region that can be used to define editor specific data such as position of nodes, etc. This region is not used by the VM in any way.

The graph region is the only region where the VM interpreter needs to understand the data of the file, while the protocol region is there to make sure that the noodle file matches this dialect.

For more in depth information checkout the Noodle Language Specifications

Next steps

This article has described so far what one could expect from a visual scripting language to define the VM as potentially embeddable and has illustrated this by giving concrete examples per category on the prototype language Noodle. Along the series of articles, we will take this further and go through each step of implementing the language, which will function as practical application of my theoretical study.

In the next article we will discuss the protocol and the document design a little bit more but will mainly focus on the API design and the editor implementation in VS code. This will also be the first practical test of the protocol and the API.

For more information, follow me on Twitter: Kazum93 or check the Noodle page on GitHub. Also, if you are interested in reading my research paper that I have been referring to all the time, you can find it on ResearchGate or my website.

[1] B. A. Myers, “Taxonomies of Visual Programming,” 1989.

[2] M. F. Msiska and L. Van Zijl, “From visual scripting to Lua,” ACM Int. Conf. Proceeding Ser., pp. 94–99, 2012, doi: 10.1145/2389836.2389848.

[3] M. M. Burnett, “Visual object-oriented programming,” Proc. Conf. Object-Oriented Program. Syst. Lang. Appl. OOPSLA, vol. Part F1296, no. April 1994, pp. 127–129, 1993, doi: 10.1145/260303.261240.

[4] M. M. Burnett and M. J. Baker, “A Classification System for Visual Programming Languages,” J. Vis. Lang. Comput., vol. 5, no. 3, pp. 287–300, 1994, doi: https://doi.org/10.1006/jvlc.1994.1015.

[5] E. Games, “Blueprints,” 2021. https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/

[6] E. Games, “Kismet Visual Scripting.” https://docs.unrealengine.com/udk/Three/KismetHome.html

[7] Unity Technology, “Bolt Documentation.” https://docs.unity3d.com/bolt/1.4/manual/index.html

[8] G. Team, “Godot Visual Scripting.” https://docs.godotengine.org/en/stable/getting_started/scripting/visual_script/getting_started.html

[9] Google, “Blocky.” https://developers.google.com/blockly (accessed Apr. 14, 2022).

[10] J. Maloney, M. Resnick, N. Rusk, B. Silverman, and E. Eastmond, “The scratch programming language and environment,” ACM Trans. Comput. Educ., vol. 10, no. 4, pp. 1–15, 2010, doi: 10.1145/1868358.1868363.

[11] MIT, “Scratch.” https://scratch.mit.edu/

[12] P. Wadler, “The essence of functional ( Invited programming talk ) recursive a compiler language,” Proc. 19th ACM SIGPLANSIGACT Symp. Princ. Program. Lang., pp. 1–14, 1992.

[13] G. Team, “VisualShaders.” https://docs.godotengine.org/en/stable/tutorials/shading/visual_shaders.html

[14] “Unreal Engine 4 Material Editor.” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/Editor/

[15] Unity Technology, “Shader Graph”, [Online]. Available: https://unity.com/shader-graph

[16] M. Autodesk, “Maya Node Editor.” [Online]. Available: https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2019/ENU/Maya-Basics/files/GUID-23277302-6665-465F-8579-9BC734228F69-htm.html

[17] B. F. Blender 3D, “Shader Editor.” https://docs.blender.org/manual/en/latest/editors/shader_editor.html

[18] H. Muhammad and R. Ierusalimschy, “C APIs in extension and extensible languages,” J. Univers. Comput. Sci., vol. 13, no. 6, pp. 839–853, 2007.

[19] H. H. Muhammad, “A study on scripting language APIs,” 2006.

[20] R. Ierusalimschy, L. H. De Figueiredo, and W. C. Filho, “SPE paper Lua – an extensible extension language,” vol. 6, no. 1996, pp. 635–652, 2015.

[21] R. Ierusalimschy, L. De Figueiredo, and W. Celes, “The evolution of an extension language: A history of Lua,” Proc. V Brazilian Symp. Program. Lang., vol. 1, no. 1, pp. 1–16, 2001, [Online]. Available: http://www.lua.org/history.html%0Ahttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.9210&rep=rep1&type=pdf

[22] M. Idrees, F. Aslam, K. Shahzad, and S. M. Sarwar, “Towards a Universal Framework for Visual Programming Languages,” Pak. J. Engg. Appl. Sci., vol. 23, no. July, pp. 55–65, 2018, [Online]. Available: https://www.researchgate.net/publication/328191862_Towards_a_Universal_Framework_for_Visual_Programming_Languages

[23] J. Gregory, Game Engine Architecture, Second Edition, 2nd ed. USA: A. K. Peters, Ltd., 2014.

[24] R. Ierusalimschy, L. H. de Figueiredo, and W. Celes, “Passing a Language through the Eye of a Needle,” Queue, vol. 9, no. 5, pp. 20–29, 2011, doi: 10.1145/1978862.1983083.

[25] J. K. Ousterhout, “Scripting: Higher-level programming for the 21st century,” Computer (Long. Beach. Calif)., vol. 31, no. 3, pp. 23–30, 1998, doi: 10.1109/2.660187.

[26] R. Nystrom, Crafting Interpreters, 1st ed. ‎ Genever Benning. [Online]. Available: https://www.craftinginterpreters.com/

[27] R. Nystrom, “Game Programming Patterns,” in Game Programming Patterns, Genever Benning, 2014. [Online]. Available: https://books.google.nl/books?id=AnvVrQEACAAJ

[28] R. Nystrom, “A Virtual Machine,” 2021. https://craftinginterpreters.com/a-virtual-machine.html

[29] J. G. Guerrero, “Reinventing the wheel,” 2014.

[30] C. O’Toole-Bateman, “The History of the Game Engine: Part 5 – Reinventing the Wheel.” https://ultimategamingparadise.com/features/series/history-of-the-game-engine/part-5-reinventing-the-wheel/