Unit Testing

Base.runtests(tests=["all"]; ncores=ceil(Int, Sys.CPU_THREADS / 2),
              exit_on_error=false, revise=false, [seed])

Run the Julia unit tests listed in tests, which can be either a string or an array of strings, using ncores processors. If exit_on_error is false, when one test fails, all remaining tests in other files will still be run; they are otherwise discarded, when exit_on_error == true. If revise is true, the Revise package is used to load any modifications to Base or to the standard libraries before running the tests. If a seed is provided via the keyword argument, it is used to seed the global RNG in the context where the tests are run; otherwise the seed is chosen randomly.

@test ex
@test f(args...) key=val ...
@test ex broken=true
@test ex skip=true

Tests that the expression ex evaluates to true. Returns a Pass Result if it does, a Fail Result if it is false, and an Error Result if it could not be evaluated.

Examples

julia> @test true
Test Passed

julia> @test [1, 2] + [2, 1] == [3, 3]
Test Passed

The @test f(args...) key=val... form is equivalent to writing @test f(args..., key=val...) which can be useful when the expression is a call using infix syntax such as approximate comparisons:

julia> @test π ≈ 3.14 atol=0.01
Test Passed

This is equivalent to the uglier test @test ≈(π, 3.14, atol=0.01). It is an error to supply more than one expression unless the first is a call expression and the rest are assignments (k=v).

You can use any key for the key=val arguments, except for broken and skip, which have special meanings in the context of @test:

• broken=cond indicates a test that should pass but currently consistently fails when cond==true. Tests that the expression ex evaluates to false or causes an exception. Returns a Broken Result if it does, or an Error Result if the expression evaluates to true. Regular @test ex is evaluated when cond==false.
• skip=cond marks a test that should not be executed but should be included in test summary reporting as Broken, when cond==true. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. Regular @test ex is evaluated when cond==false.

Examples

julia> @test 2 + 2 ≈ 6 atol=1 broken=true
Test Broken
  Expression: ≈(2 + 2, 6, atol = 1)

julia> @test 2 + 2 ≈ 5 atol=1 broken=false
Test Passed

julia> @test 2 + 2 == 5 skip=true
Test Broken
  Skipped: 2 + 2 == 5

julia> @test 2 + 2 == 4 skip=false
Test Passed

@test_throws exception expr

Tests that the expression expr throws exception. The exception may specify either a type, or a value (which will be tested for equality by comparing fields). Note that @test_throws does not support a trailing keyword form.

Examples

julia> @test_throws BoundsError [1, 2, 3][4]
Test Passed
      Thrown: BoundsError

julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
Test Passed
      Thrown: DimensionMismatch

@testset [CustomTestSet] [option=val  ...] ["description"] begin ... end
@testset [CustomTestSet] [option=val  ...] ["description $v"] for v in (...) ... end
@testset [CustomTestSet] [option=val  ...] ["description $v, $w"] for v in (...), w in (...) ... end

Starts a new test set, or multiple test sets if a for loop is provided.

If no custom testset type is given it defaults to creating a DefaultTestSet. DefaultTestSet records all the results and, if there are any Fails or Errors, throws an exception at the end of the top-level (non-nested) test set, along with a summary of the test results.

Any custom testset type (subtype of AbstractTestSet) can be given and it will also be used for any nested @testset invocations. The given options are only applied to the test set where they are given. The default test set type accepts the verbose boolean option: if true, the result summary of the nested testsets is shown even when they all pass (the default is false).

The description string accepts interpolation from the loop indices. If no description is provided, one is constructed based on the variables.

By default the @testset macro will return the testset object itself, though this behavior can be customized in other testset types. If a for loop is used then the macro collects and returns a list of the return values of the finish method, which by default will return a list of the testset objects used in each iteration.

Before the execution of the body of a @testset, there is an implicit call to Random.seed!(seed) where seed is the current seed of the global RNG. Moreover, after the execution of the body, the state of the global RNG is restored to what it was before the @testset. This is meant to ease reproducibility in case of failure, and to allow seamless re-arrangements of @testsets regardless of their side-effect on the global RNG state.

Examples

julia> @testset "trigonometric identities" begin
           θ = 2/3*π
           @test sin(-θ) ≈ -sin(θ)
           @test cos(-θ) ≈ cos(θ)
           @test sin(2θ) ≈ 2*sin(θ)*cos(θ)
           @test cos(2θ) ≈ cos(θ)^2 - sin(θ)^2
       end;
Test Summary:            | Pass  Total
trigonometric identities |    4      4

@inferred [AllowedType] f(x)

Tests that the call expression f(x) returns a value of the same type inferred by the compiler. It is useful to check for type stability.

f(x) can be any call expression. Returns the result of f(x) if the types match, and an Error Result if it finds different types.

Optionally, AllowedType relaxes the test, by making it pass when either the type of f(x) matches the inferred type modulo AllowedType, or when the return type is a subtype of AllowedType. This is useful when testing type stability of functions returning a small union such as Union{Nothing, T} or Union{Missing, T}.

julia> f(a) = a > 1 ? 1 : 1.0
f (generic function with 1 method)

julia> typeof(f(2))
Int64

julia> @code_warntype f(2)
MethodInstance for f(::Int64)
  from f(a) in Main at none:1
Arguments
  #self#::Core.Const(f)
  a::Int64
Body::UNION{FLOAT64, INT64}
1 ─ %1 = (a > 1)::Bool
└──      goto #3 if not %1
2 ─      return 1
3 ─      return 1.0

julia> @inferred f(2)
ERROR: return type Int64 does not match inferred return type Union{Float64, Int64}
[...]

julia> @inferred max(1, 2)
2

julia> g(a) = a < 10 ? missing : 1.0
g (generic function with 1 method)

julia> @inferred g(20)
ERROR: return type Float64 does not match inferred return type Union{Missing, Float64}
[...]

julia> @inferred Missing g(20)
1.0

julia> h(a) = a < 10 ? missing : f(a)
h (generic function with 1 method)

julia> @inferred Missing h(20)
ERROR: return type Int64 does not match inferred return type Union{Missing, Float64, Int64}
[...]

@test_logs [log_patterns...] [keywords] expression

Collect a list of log records generated by expression using collect_test_logs, check that they match the sequence log_patterns, and return the value of expression. The keywords provide some simple filtering of log records: the min_level keyword controls the minimum log level which will be collected for the test, the match_mode keyword defines how matching will be performed (the default :all checks that all logs and patterns match pairwise; use :any to check that the pattern matches at least once somewhere in the sequence.)

The most useful log pattern is a simple tuple of the form (level,message). A different number of tuple elements may be used to match other log metadata, corresponding to the arguments to passed to AbstractLogger via the handle_message function: (level,message,module,group,id,file,line). Elements which are present will be matched pairwise with the log record fields using == by default, with the special cases that Symbols may be used for the standard log levels, and Regexs in the pattern will match string or Symbol fields using occursin.

Examples

Consider a function which logs a warning, and several debug messages:

function foo(n)
    @info "Doing foo with n=$n"
    for i=1:n
        @debug "Iteration $i"
    end
    42
end

We can test the info message using

@test_logs (:info,"Doing foo with n=2") foo(2)

If we also wanted to test the debug messages, these need to be enabled with the min_level keyword:

@test_logs (:info,"Doing foo with n=2") (:debug,"Iteration 1") (:debug,"Iteration 2") min_level=Logging.Debug foo(2)

If you want to test that some particular messages are generated while ignoring the rest, you can set the keyword match_mode=:any:

@test_logs (:info,) (:debug,"Iteration 42") min_level=Logging.Debug match_mode=:any foo(100)

The macro may be chained with @test to also test the returned value:

@test (@test_logs (:info,"Doing foo with n=2") foo(2)) == 42

If you want to test for the absence of warnings, you can omit specifying log patterns and set the min_level accordingly:

# test that the expression logs no messages when the logger level is warn:
@test_logs min_level=Logging.Warn @info("Some information") # passes
@test_logs min_level=Logging.Warn @warn("Some information") # fails

If you want to test the absence of warnings (or error messages) in stderr which are not generated by @warn, see @test_nowarn.

@test_deprecated [pattern] expression

When --depwarn=yes, test that expression emits a deprecation warning and return the value of expression. The log message string will be matched against pattern which defaults to r"deprecated"i.

When --depwarn=no, simply return the result of executing expression. When --depwarn=error, check that an ErrorException is thrown.

Examples

# Deprecated in julia 0.7
@test_deprecated num2hex(1)

# The returned value can be tested by chaining with @test:
@test (@test_deprecated num2hex(1)) == "0000000000000001"

@test_warn msg expr

Test whether evaluating expr results in stderr output that contains the msg string or matches the msg regular expression. If msg is a boolean function, tests whether msg(output) returns true. If msg is a tuple or array, checks that the error output contains/matches each item in msg. Returns the result of evaluating expr.

See also @test_nowarn to check for the absence of error output.

Note: Warnings generated by @warn cannot be tested with this macro. Use @test_logs instead.

@test_nowarn expr

Test whether evaluating expr results in empty stderr output (no warnings or other messages). Returns the result of evaluating expr.

Note: The absence of warnings generated by @warn cannot be tested with this macro. Use @test_logs instead.

@test_broken ex
@test_broken f(args...) key=val ...

Indicates a test that should pass but currently consistently fails. Tests that the expression ex evaluates to false or causes an exception. Returns a Broken Result if it does, or an Error Result if the expression evaluates to true. This is equivalent to @test ex broken=true.

The @test_broken f(args...) key=val... form works as for the @test macro.

Examples

julia> @test_broken 1 == 2
Test Broken
  Expression: 1 == 2

julia> @test_broken 1 == 2 atol=0.1
Test Broken
  Expression: ==(1, 2, atol = 0.1)

@test_skip ex
@test_skip f(args...) key=val ...

Marks a test that should not be executed but should be included in test summary reporting as Broken. This can be useful for tests that intermittently fail, or tests of not-yet-implemented functionality. This is equivalent to @test ex skip=true.

The @test_skip f(args...) key=val... form works as for the @test macro.

Examples

julia> @test_skip 1 == 2
Test Broken
  Skipped: 1 == 2

julia> @test_skip 1 == 2 atol=0.1
Test Broken
  Skipped: ==(1, 2, atol = 0.1)

record(ts::AbstractTestSet, res::Result)

Record a result to a testset. This function is called by the @testset infrastructure each time a contained @test macro completes, and is given the test result (which could be an Error). This will also be called with an Error if an exception is thrown inside the test block but outside of a @test context.

finish(ts::AbstractTestSet)

Do any final processing necessary for the given testset. This is called by the @testset infrastructure after a test block executes. One common use for this function is to record the testset to the parent's results list, using get_testset.

get_testset()

Retrieve the active test set from the task's local storage. If no test set is active, use the fallback default test set.

get_testset_depth()

Returns the number of active test sets, not including the default test set