gsub object emplacement

in an ideal world, we'd be able to do something like this:

local a = "abcd 1234 defg 8 hijk"
a:gsub("%d+",function (s) return {id = s} end)
--> { "abcd ", {id="1234"}, " defg ", {id="8"}, " hijk" }  

but that won't work, because the replacement value we pass to gsub must be a string (and gsub itself must always return a single string).

you might think that in order to circumvent this, we'd need to implement a new pattern matching system ourselves, one without this limitation. but here's the trick:

local replacements = {}
local function emplace(obj)
	table.insert(replacements, obj)
	return "\x0e" .. #replacements .. "\x0f"
end
local b = a:gsub("%d+",function(s) return emplace{id = s} end)
--> b: "abcd \x0e1\x0f defg \x0e2\x0f hijk"
--> replacements: { {id="1234"}, {id="8"} }

we stow away our replacement object into a separate table, and return its index back to gsub, delimited by some escape characters⁽¹⁾. next, we need to do some post-processing:

local output = {""}
while b ~= "" do
	local repl_idx, new_pos = b:match "^\x0e(%d+)\x0f()"
	if repl_idx then
		table.insert(output, replacements[tonumber(repl_idx)])
		table.insert(output, "")
		b = b:sub(new_pos)
	else
		output[#output] = output[#output] .. b:sub(1,1)
		b = b:sub(2)
	end
end
return output
--> { "abcd ", {id="1234"}, " defg ", {id="8"}, " hijk" } as desired

we scan through the string, replacing those escape sequences with the replacement objects we stored earlier. notice that we can't use gsub for this part, because of the aforementioned limitation that gsub must return a single string!

i've only been using a very simple pattern in this example, but we have access to the full power of the lua pattern matching system here. and we didn't need to reimplement any of that ourself.

unlike my last stupid lua trick, i'm actually using this one in production. in garkup, the homemade markup language system that powers the very blog you're reading right now, i need to replace certain stretches of input text with html elements. in my system, i model html elements as tables that store children and attributes.

1. we're assuming that those characters don't appear anywhere in the original input. if you can't make such a guarantee, you could dynamically compute a suitable escape sequence ↩︎