Lua Patterns, Red Parse, and Regex

In my Learning Lua project, I needed to extract a four character code from a string. Many languages use a builtin Regex implementation to provide the capability to match and extract patterns from strings. I am also familiar with Red, which like Rebol before it, provides the Parse Dialect (a Domain Specific Language (DSL)) which not only has excellent string matching and extraction features but makes it easy to write further DSLs. 

I found that Lua has another approach. It has a built-in lightweight pattern language, known as Patterns. From what I understand, it was designed to be simpler than Regex both in its "grammar" and its capability. (Lua does have Regex modules available. It also has the very interesting LPeg pattern-matching library, based on Parsing Expression Grammars (PEGs).)

I thought it would be interesting to compare the three approaches, patterns, parse and regex. Here are three versions of a short function to extract the first four consecutive alphabetic characters from a string. The functions return an empty string if there is no match.

First Regex, courtesy of Ruby:

    def parse_line line

      ticker = line.match(/[A-Za-z]{4,4}/)

      if ticker then

        ticker[0]

      else

        ''

      end

    end

Note: 

    I'm no Regex expert, there may well  be a better way, and probably a more Rubyish way too.

Second Parse, courtesy of Red

parse-line: function [line [string!] /local ticker][

alpha: charset [#"A"-#"Z" #"a"-#"z"] parse line [to copy ticker 4 alpha] any [ticker ""] ]

Notes: 

    1. The charset allows you to efficiently define set of characters to be matched.

    2. The parse function apply the rules provided between the []s to the input.

    3. A parse rule can have any number of sub-rules.

    4. Red Parse details.

Finally Patterns, courtesy of Lua

    function stockfetch.parseLine (line)

      local match = string.match(line, '(%a%a%a%a)')

      if match then 

        return match

      else

        return ''

      end

    end

There doesn't seem much to chose between the approaches for such a simple task. The Lua pattern is very readable but you do have to know (or lookup) that %a matches an alphabetic character. The Regex is self-explanatory once you understand Regex syntax. The Red parse rule may seem a little more complicated because it also specifies to where the data are extracted (copy ticker) and has to ensure that the rules cover the whole supplied string (to end).

It will be interesting to see how the three approaches compare with more challenging requirements. Perhaps, I'll be able to add LPeg into the equation too.

Note: I am a strong supporter of and contributor to Red and am highly biased towards its Parse DSL. 

Learning Lua - Stubbing, Mocking and Spying on Functions in Testing

I was very pleased to find that it is fully possible to overwrite existing functions in Lua.  It even applies to those built-in functions that can feel as though they are part of the language. I guess that they are actually functions that are defined in the global scope.

Here's a quick example:

    > a = 1

    > sp = print

    > sp(a)

    1

    > print = function () a = 2 end

    > print()

    > sp(a)

    2

    > print = sp

    > print(a)

    2

This allows full stubbing, mocking and spying on functions during testing, albeit with a little work. You simply save the function you want to stub, mock or spy on and write a function to act as a stub, mock or spy. You can make use of Lua's upvalues to be able to count how many times the function was called in a test.

Here's an example of mocking a function in a test using LuaUnit:

    function testMocking ()

        saveParseLine = t.parseLine

        local parseLineCalled = 0

        t.parseLine = function (line)

            parseLineCalled = parseLineCalled + 1

            luaunit.assertEquals(line, 'AAPL')

            return 'AAPL' 

        end

        luaunit.assertEquals(funcThatCallsParseLine(data), 0);

        luaunit.assertEquals(parseLineCalled, 1) 

        t.parseLine =  saveParseLine

    end

Learning Lua - Test Doubles

I was able to spend some time learning Lua this week and made a little progress with my stockfetch project which am using to answer Busted or LuaUnit?. 

This week I needed to mock some functions to write some tests. Lua, as I had expected, makes it easy to replace functions with test "doubles". I was expecting that Busted would make it easy to "mock" a function. What I found is that Busted definitions of "stub" and "mock" are not the ones that I understand. 

My understanding was based on four types of test "doubles" - fakes, stubs, mocks and spies. Fakes are functions that provide an implementation just for testing that would not be used in production.  Stubs return a known value or perform a known side effect. Mocks are like stubs with the added functionality of being able to query interaction with them. (Such as what arguments they were called with, how often they were called and what they returned.) Spies could be considered as mocks which additionally call the actual function.

From my current understanding, in Busted spies are like the spies that I understand. It seems that Busted stubs appear to be like the mocks with which I am familiar but without the ability to specify a return value (or values). From the docs, Busted mocks are tables whose functions have been wrapped in spies or stubs.

The different terminology isn't a problem but not being able to specify a return value from a test "double" leads to additional work. (It might actually be possible to do so in Busted but I haven't been able to find out how just yet.)

I had expected to "roll my own" test doubles with LuaUnit but having to do part of the job myself with Busted means that there still isn't much to choose between them.

In one test, that I wrote I wanted to check that a function was called five times and have it return 0 each time it was called.

This is how I achieved that using Busted:

        saveProcessTicker = stockfetch.processTicker

    stockfetch.processTicker = function () 

      return 0 

    end 

    spy.on(stockfetch, "processTicker")

    assert.equal(0, stockfetch.run('fixtures/mixedTickers.txt'))

    assert.spy(stockfetch.processTicker).was.called(5)

    stockfetch.processTicker:revert()

    stockfetch.processTicker = saveProcessTicker

This is the LuaUnit equivalent:

    saveProcessTicker = stockfetch.processTicker

  local processTickerCalled = 0

  stockfetch.processTicker = function ()

    processTickerCalled = processTickerCalled + 1 

    return 0 

  end 

  luaunit.assertEquals(stockfetch.run('fixtures/mixedTickers.txt'), 0);

  luaunit.assertEquals(processTickerCalled, 5)

  stockfetch.processTicker = saveProcessTicker

The main difference being needing to "manually" count the number of times that the function was called with LuaUnit.

I will be looking at using test doubles with builtin modules such as IO next.

Lua inf, -inf and nan

The Lua number datatype has the conventional three special values inf (positive infinity), -inf (negative infinity) and nan (not a number). It seems not to be so conventional when it comes to comparing these values. It returns true when comparing inf with inf, -inf with -inf, and nan with nan. I believe that mathematically this is not quite correct. However, I can see the advantages of being able to compare them from a programming perspective.

Preparing to Learn Lua

I took my second step in preparing to learn Lua. I've set up a simple training project through which I can get a feeling of the workflow and practices that will work for me when programming in Lua. I've based the project on the StockFetch example from Test Driving Javascript Applications by Dr. Venkat Subramaniam. (A very worthwhile read).

Before I set up the project, I wanted to see how easy it would be to read the contents of a web page in Lua. It turned out to be very easy once I had installed luasocket.

    sudo luarocks-jit install luasocket

Reading a webpage highlighted, for me, the ability to return multiple values from a Lua function:

   http = require 'socket.http'

    body,code,headers,status = http.request('http://bbc.co.uk')

I also noticed that much Lua is written defensively in the sense that care is taken to minimise opportunities for crashes through the use of the assert function and other such techniques. Clearly, being able to return a status code and a value from a function will help to write robust code.

Knowing how to read websites in Lua, I proceeded to set up the project. I adopted a directory structure that will be familiar to many people but I don't know how common it is in Lua circles.

The basic structure is:

    stockfetch/                           -- main directory

        stockfetch-cli.lua                -- command line interface

        src/                              -- lua modules

            stockfetch.lua                -- main app module

        test/                             -- tests

            stockfetch-busted.lua         -- busted tests

            stockfetch-unit.lua           -- Lua unit tests

After setting up the project, I wanted to work out how best to test its main program, stock watch-cli.lua. It's basically a three line script to initiate the main stockfetch module.

      local stockfetch = require('src/stockfetch');

      local rc = stockfetch.run()

      os.exit(rc)

I'm glad I did as I found out a few things about file paths in Lua that were useful to know from the off:

• Lua retains the current working director setting from the OS. So relative paths calculated in a script can change depending on the directory from which the script was launched
• Another way of looking at that is the Lua does not change the current working directory to that from which a script is loaded
• The dofile function only accepts absolute file paths
• There is no builtin function to "clean" a file path limiting the use of setting a base-file path and a relative path including ../

It soon became apparent that I wasn't going to be able to test stock watch-cli by using docile. (I would be delighted to learn of such a way if there is one.) Instead, I used Lua's command line interfaces to test it.

I found two different ways to run command line programs from Lua - os.execute and io.popen. The first returns the return code from the command executed, the second returns the output from the command. I wanted both, so I chose to use os.execute and redirect the command output to a file. That way I'm able to get both the output and the return code.

I successfully wrote tests in both Busted and LuaUnit using this approach. 

Finally, I loaded the project into a GitHub repository.

Next, I will complete the stockfetch app following a test-driven approach and start to learn how to code in Lua.