Pure Lua Mode - tested Core¶

For situations where Lua code can be directly executed, the standard tested framework and CLI from LuaRocks should work great, but in instances where tested needs to be embedded and/or LuaRocks (or equivalent) is not available, we offer tested_core.lua - a single file, zero external dependency, pure Lua module that can be used to run tests. Only the CLI based test runner requires additional dependencies to function (mostly around OS calls) and most functionality is available in both.

The actual tests themselves are written the same whether running via tested_core or via the tested CLI, so everything from the Unit Testing page works fine. tested = require("tested_core") is a drop in replacement for tested = require("tested").

In theory all of this allows tested_core to work with Neovim, LÖVE, OpenResty, and many other embedded environments! We haven't fully explored all the opportunities yet, but if you've had success, please let us know!. We want to one day add guides on how to set tested up for various use cases!

All of the Lua API methods are fairly new and could change, if you need help with something or feels like something is missing, feel free to open up an issue!.

Install¶

The latest released version of tested_core.lua can be found on the GitHub Releases page. It can also be installed via LuaRocks:

luarocks install tested-core

Running tests via the API¶

Below are examples of using tested_core to run tests, and at the bottom of the page, we also have the full Teal type definitions for all of the values that get returned.

Fair warning, we are still pre-v1 and while a lot of the returned values haven't changed much, it is always possible. After v1, we will follow semver and any major change (removal or restructuring) will result in a MAJOR version change to tested and additions will be considered MINOR.

Run a single test¶

After writing a test tile, you can directly call run on the tested object

local tested = require("tested_core")

local t = tested.new()

t:test("hello this is my test name", function()
    tested.assert({
        given = "the most basic of tests (4 == 4)",
        should = "be true",
        expected = 4,
        actual = 4,
    })
end)

--- ... other tests

-- results will include all test results, and assertion results and such
local results = t:run()

t:run can take an optional parameter options:

local test_options = {random=true} -- will randomize order of tests
local results = t:run(test_options)

Tested Options¶

The options are the same as the ones from the CLI, and include an additional clock_s and sleep_s for setting timing functions:

• random: boolean - randomizes the order of tests and test instances before running
• filter: string - filter a set of tests down by name
• tags: string - use test tags combined with and, or, and not to run specifically tagged tests
• clock_s: function(): number - a function that returns a number of seconds (by default uses os.clock()). This is used to determine the amount of time a test has been run. Overridable in situations where you don't want CPU time but rather wall time and/or more precision.
• sleep_s: function(seconds: number) - For test retries with a retry_delay, the sleep_s method is used. By default it is a blocking sleep method using os.clock().

Run multiple tests¶

We also provide a convenient way to run a bunch of tests:

local tested = require("tested_core")

-- simple example, but you should be able do test discovery however!
-- these files simply return the tested object `t`
local first_test = require("tests.first")
local second_test = require("tests.second")
local all_tests = {first_test, second_test}

local options = {random=true}
-- This will call `run` on all the individual tests
-- and return the total counts/times of all tests run as well as all the individual test results
local test_runner_output = tested.run_tests(all_tests, options)

It follows all the same options as above.

Formatting results¶

Unlike standard tested, the results of the tests do not get printed out to stdout, but we do offer a few helper methods for formatting the Lua table output as a string to match the regular output.

local tested = require("tested_core")

local test_runner_output = tested.run_tests(all_tests, options)
local results_string = tested.format_results(test_runner_output, "terminal", "all")
print(results_string)

The optional second and third arguments line up with the CLI's --display-format and --show options respectively:

• The second argument "display_format" can either be plain, terminal, or tap (and defaults to plain)
• The third argument "show" is a list of strings that can either be: pass fail skip exception unknown expected unexpected valid invalid all
  • It defaults to tests that need to be addressed: {"fail", "exception", "unknown", "unexpected"}

There are also helper methods for combining individual t:run()s:

local tested = require("tested_core")
local first_test = require("tests.first")
local second_test = require("tests.second")

local first_test_result = first_test:run()
local second_test_result = second_test:run()

-- combine and display all results
local combined_results = tested.combine_results({first_test_result, second_test_result})
print(tested.format_results(combined_results))

Feature Comparison¶

Type Definition Reference¶

The tested_core API methods for test running:

interface TestedClass
  run_tests: function(tests: {Tested}, options?: TestedAPIOptions): TestRunnerOutput
  combine_results: function(tests_output: {TestedOutput}): TestRunnerOutput
  format_results: function(output: TestRunnerOutput, display_type?: DisplayFormat, show?: {DisplayOptions}): string
end

interface TestedObject
  run: function(self: Tested, options?: TestedAPIOptions): TestedOutput
end

interface TestedOutput
   tests: {TestOutput}
   filename: string
   counts: TestCounts
   fully_tested: boolean
   total_time: number
end

interface TestOutput
   name: string
   result: TestResult
   message: string
   time: number
   assertion_results: {AssertionResult}
   options: TestedOptions
   retry_attempts: {TestOutput}
end

interface TestCounts
   passed: integer
   failed: integer
   expected: integer
   skipped: integer
   filtered: integer
   invalid: integer
end

enum TestResult
   "PASS"
   "FAIL"
   "SKIP"
   "FILTERED"
   "EXCEPTION"
   "TIMEOUT"
   "UNKNOWN"
   "EXPECTED_FAIL"
   "EXPECTED_EXCEPTION"
   "EXPECTED_UNKNOWN"
   "UNEXPECTED"
end

interface AssertionResult
   filename: string
   line_number: integer
   result: TestResult
   given: string
   should: string
   error_message: string
end

interface TestedOptions
   expected: ExpectedTestResult
   run_when: boolean
   tags: {string}
   retries: integer
   retry_delay: number
end

enum ExpectedTestResult
   "FAIL"
   "EXCEPTION"
   "UNKNOWN"
end