custom VM

Why It Matters

For developers working in environments where code security is paramount, a custom VM provides a strong deterrent against casual reverse engineering. It adds a layer of abstraction that makes it significantly harder for an attacker to understand the original logic, especially when combined with other obfuscation techniques such as string encoding, control flow flattening, or dead code insertion.

In production, the use of a custom VM can increase the complexity of debugging and maintenance. However, it also provides a level of protection that is essential in scenarios where the source code contains proprietary algorithms or sensitive logic. The trade-off between security and maintainability must be carefully weighed, especially in large applications where the obfuscation overhead could affect performance.

How It Works

The core mechanism of a custom VM involves translating or encoding the original source code into a form that can be interpreted by a custom execution engine. This engine typically operates on a set of instructions or bytecode that are not directly executable by the standard JavaScript engine. The process includes:

• Compilation or transformation of source code into an intermediate representation, often bytecode or a custom instruction set.
• Designing a custom interpreter that understands and executes this intermediate format.
• Implementing a runtime environment that supports the execution of obfuscated code, including memory management, stack handling, and function call resolution.
• Ensuring the interpreter can handle dynamic code execution, variable scoping, and control flow constructs.
• Optimizing performance to minimize the overhead introduced by the custom execution layer.

Custom VMs often use a stack-based architecture or a register-based model to manage execution. They may also implement custom garbage collection, exception handling, and optimization strategies. The design of the VM must align with the specific obfuscation techniques being used, such as instruction substitution or control flow obfuscation.

Quick Reference

Basic Example

This example illustrates a simplified custom VM that executes a basic stack-based bytecode. The VM supports operations like ADD and PUSH, which are typical in minimal VM implementations.

function CustomVM() {
  this.stack = [];
  this.ip = 0;
  this.code = [];
}

CustomVM.prototype.execute = function() {
  while (this.ip < this.code.length) {
    const instruction = this.code[this.ip];
    switch (instruction.op) {
      case 'PUSH':
        this.stack.push(instruction.value);
        break;
      case 'ADD':
        const b = this.stack.pop();
        const a = this.stack.pop();
        this.stack.push(a + b);
        break;
    }
    this.ip++;
  }
  return this.stack.pop();
};

const vm = new CustomVM();
vm.code = [
  { op: 'PUSH', value: 5 },
  { op: 'PUSH', value: 3 },
  { op: 'ADD' }
];
console.log(vm.execute()); // Outputs: 8

This example demonstrates how a basic VM can interpret simple instructions. The PUSH instruction adds a value to the stack, and ADD pops two values, adds them, and pushes the result back onto the stack. This is a foundational concept for more complex obfuscation engines.

Production Example

In a production-grade scenario, a custom VM may include support for function calls, dynamic scoping, and more complex control structures. The following example shows a more robust VM with support for functions and variable scoping.

function ProductionVM() {
  this.stack = [];
  this.globals = {};
  this.functions = {};
  this.ip = 0;
  this.code = [];
}

ProductionVM.prototype.execute = function() {
  while (this.ip < this.code.length) {
    const instruction = this.code[this.ip];
    switch (instruction.op) {
      case 'PUSH':
        this.stack.push(instruction.value);
        break;
      case 'CALL':
        const func = this.functions[instruction.name];
        this.stack.push(this.executeFunction(func));
        break;
      case 'SET':
        this.globals[instruction.name] = this.stack.pop();
        break;
      case 'GET':
        this.stack.push(this.globals[instruction.name]);
        break;
    }
    this.ip++;
  }
};

ProductionVM.prototype.executeFunction = function(func) {
  const oldStack = this.stack;
  this.stack = [];
  this.ip = 0;
  this.code = func.instructions;
  const result = this.execute();
  this.stack = oldStack;
  return result;
};

const vm = new ProductionVM();
vm.functions['add'] = {
  instructions: [
    { op: 'PUSH', value: 10 },
    { op: 'PUSH', value: 20 },
    { op: 'ADD' }
  ]
};
vm.code = [
  { op: 'CALL', name: 'add' },
  { op: 'SET', name: 'result' }
];
vm.execute();
console.log(vm.globals.result); // Outputs: 30

This version is more suitable for production because it handles function calls, variable scoping, and modular code execution. It also separates concerns between the main execution context and function execution, making it more maintainable and scalable.

Common Mistakes

• Not validating bytecode input, which can lead to runtime errors or security vulnerabilities.
• Implementing a VM with insufficient memory management, resulting in memory leaks or crashes.
• Using hardcoded values or predictable patterns, which can make obfuscation easier to reverse.
• Failing to handle control flow correctly, leading to incorrect execution paths in obfuscated code.
• Overlooking performance implications of the VM, especially in high-frequency operations.
• Not securing the VM against tampering, which can allow attackers to bypass obfuscation.

Security And Production Notes

• Always validate and sanitize input bytecode to prevent injection attacks or unintended execution.
• Implement robust error handling to avoid exposing internal VM state or stack traces.
• Use secure memory management practices to prevent memory leaks or access violations.
• Ensure the VM is not easily modifiable or debuggable, as this can undermine obfuscation.
• Consider performance overhead when integrating a custom VM, especially in resource-constrained environments.

Related Concepts

Custom VMs are closely related to several core concepts in software engineering and security:

• Bytecode: The intermediate representation used by VMs to execute code.
• Interpreters: Systems that execute code without compilation, often used in VMs.
• Obfuscation: The process of making code harder to understand, often using custom VMs.
• Virtual Machines: General-purpose execution environments, including custom implementations.
• Control Flow Obfuscation: Techniques that alter program execution paths to hinder reverse engineering.

Further Reading

• MDN Web Docs
• WHATWG HTML Standard