Section 3

Methodology

How 26 Lua execution environments and a C baseline were measured under identical, thermally calibrated conditions.

3.1

Experimental equipment

All experiments ran on a laptop with Ubuntu 24.04.2 LTS (Linux 6.8.0-90-generic, x86-64) and an Intel Core i7-8550U CPU @ 1.80 GHz. Only the minimal necessary operating-system processes were active during measurements, and every execution started from a calibrated CPU temperature so that all programs share comparable starting conditions.

3.2

Energy and runtime measurements

Energy was measured with Intel's Running Average Power Limit (RAPL) framework, reading the Model-Specific Registers to capture CPU Package and DRAM energy in Joules. Execution time was taken from the operating system's wall-clock mechanism, in seconds.

To keep the comparison fair across execution models, measurements cover strictly the runtime execution phase: for LuaAOT, the offline C code synthesis and gcc compilation are excluded. Every benchmark program was executed 10 times, and statistical anomalies were filtered with the Interquartile Range (IQR) method before aggregation.

3.3

Evaluated Lua execution environments

Standard interpreter (PUC-Rio)

17 releases

5.3.0–5.3.6, 5.4.0–5.4.8 and 5.5.0. The official reference implementation, written entirely in C.

Just-In-Time compiler (LuaJIT)

8 variants

LuaJIT 2.0.0–2.0.5 and 2.1 — the latter a rolling release, built from the latest source of the official repository at the time of the experiments (luajit -v reports 2.1.1764593432) — plus the LuaJIT Remake project. Trace-based compilation of hot code paths.

Ahead-Of-Time compiler (LuaAOT)

1 release

Uses partial evaluation to translate Lua bytecode into standalone C, then compiles it to a native binary.

Reference baseline (C)

gcc 12.2.0

Equivalent optimized C implementations, used as the absolute efficiency reference.

3.4

Benchmark suites

The LuaAOT suite alone is not representative of everyday Lua code — its programs are written by language experts for maximum speed. It is therefore complemented with four suites built on widely used Lua libraries, covering data structures, higher-order functions, linear algebra and object orientation.

LuaAOT suite

Computer Language Benchmarks Game

binary-trees, fannkuch-redux, fasta, k-nucleotide, mandelbrot, n-body, spectral-norm

Computationally intensive programs written for maximum speed. LuaJIT Remake could not run k-nucleotide and mandelbrot, which rely on unsupported functions of the standard io library. For binary-trees, fasta and mandelbrot — which ship JIT-optimized variants — identical input parameters are used across all execution models.

LuaMatrix

Pure-Lua matrix library

construct, add, mul, scalar_mul (3.14159), transpose over dense 1000 × 1000 matrices

Numerical computation and memory-access efficiency over nested Lua tables.

LuaFun

Functional programming library

imperative_loop, filter_map, map_filter, filterWithConjunction, conjunctionOfFilters, fusionByHand over 10⁷ integers

Paired programs computing identical results, designed to test whether implementations perform program fusion. Written with LuaJIT's trace compiler in mind.

Collections (Penlight)

Penlight data structures

list_construct (10⁶), list_map_filter (10⁶), list_sort (6×10⁵), map_insert_lookup (6×10⁵), set_algebra (5×10⁵), deepcopy (4×10⁵)

Higher-level lists, maps and sets; dominated by allocation and table machinery.

OO Classes (Penlight)

Metatable-based OOP

deep_inheritance (10⁶), instantiation (5×10⁵), method_dispatch (10⁶), multiple_inheritance (10⁶), operator_overload (5×10⁵), polymorphic_dispatch (10⁶)

Object-oriented abstractions built on Lua metatables: inheritance chains, dynamic dispatch and operator overloading.

3.5

Experimental setup

Thermal conditions were calibrated once by idling for 60 seconds and recording the resulting average CPU temperature as the baseline, T_baseline = 32.2 °C. Each subsequent execution only started once the current average core temperature satisfied T_start ≤ T_baseline + 5 °C. A larger tolerance would speed up data collection at the cost of thermal variability.

A battery endurance study complements the RAPL measurements: with display brightness at minimum, background synchronization disabled and wireless interfaces off, the number of continuous benchmark iterations was counted from a full charge to complete depletion.

3.6

The GPS-UP sustainability metric suite

Improvements are reported with the GPS-UP metrics, each computed against a baseline implementation:

Speedup

T_baseline / T_optimized

Performance gain; above 1 means faster.

Greenup

E_baseline / E_optimized

Energy efficiency gain; above 1 means less energy.

Powerup

Speedup / Greenup

Change in average power; below 1 means lower power draw.

To classify the impact of an optimization, the GPS-UP Software Energy Efficiency Quadrant Graph (Figure 1) divides results into eight categories. Categories 1–4 are the green outcomes, where the optimization saves energy; categories 5–8 are the red outcomes, where the optimized version consumes more energy than the baseline.

Table 1

Study cost

The full experimental pipeline involved executing 30 programs ten times across 26 Lua implementations, requiring 11 hours and 10 minutes of active execution time and consuming about 653 kJ of Package energy and 21 kJ of DRAM energy.

The total data-collection time, however, substantially exceeded 24 hours: the thermal regulation protocol enforced cooldown periods between runs until the CPU temperature dropped back below the 32.2 °C baseline threshold.

Energy consumption and execution time for each benchmark
BenchmarkProgramsPackage energy (J)DRAM energy (J)Execution time
LuaAOT7463 253.5 J14 914.1 J07:09:19
LuaMatrix5125 257.5 J3 532.5 J02:30:55
LuaFun630 417.3 J599.7 J00:32:30
Collections (Penlight)612 394.8 J807.4 J00:21:44
OO Classes (Penlight)621 336.0 J1 531.5 J00:36:24
Global Total30652 659.1 J21 385.2 J11:10:52