Stream: helpdesk (published)

Topic: Multiple test environments?

view this post on Zulip Daniel Wennberg (Jul 22 2024 at 21:28):

Is there an ergonomic way to run a subset of tests in a separate environment from the rest of the test suite? The context is that one of my test dependencies has very restrictive compat bounds and ends up downgrading several low-level packages compared to how the main environment is resolved. I'd prefer to isolate the tests that require this dependency in their own environment.

view this post on Zulip Daniel Wennberg (Jul 22 2024 at 21:32):

I know this normally wouldn't matter if everyone follows semver, however, I'm testing things that are not guaranteed by semver such as inference and allocations.

view this post on Zulip Mason Protter (Jul 22 2024 at 21:36):

SafeTestsets.jl does that if I recall correctly

view this post on Zulip Daniel Wennberg (Jul 22 2024 at 21:39):

Looks like SafeTestsets puts each testset in a separate module, which is not quite the same as a separate environment

view this post on Zulip Mason Protter (Jul 22 2024 at 23:02):

oh, whoops yeah I misread the question

view this post on Zulip Daniel Wennberg (Jul 22 2024 at 23:07):

I tried to hack it together using Pkg operations inside a safetestset, but precompilation fails for reasons I don't understand.

view this post on Zulip Fredrik Ekre (Jul 22 2024 at 23:12):

Use a separate Project.toml + Manifest.toml and spawn a new process is probably the easiest.

view this post on Zulip Daniel Wennberg (Jul 22 2024 at 23:17):

Yeah, that's where I'm heading. Is it possible to start a Distributed.jl worker in a separate environment, or is it better to just use run(`julia --project="extratests" "extratests/extratests.jl"`)?

view this post on Zulip Fredrik Ekre (Jul 22 2024 at 23:28):

The latter seems much easier (but you can pass exeflags to Distributed.addprocs if you want to go down that route).

view this post on Zulip Daniel Wennberg (Jul 23 2024 at 05:43):

For posterity, here's how I ended up configuring the separate test process to mimic the launch configuration for normal tests as closely as possible:

@testset "SpecialTests" begin
    code = "using Pkg; Pkg.update(); include(\"SpecialTests/runtests.jl\")"
    cmd = addenv(
        Cmd(`$(Base.julia_cmd()) --eval "$code"`),
        Dict("JULIA_LOAD_PATH" => "@:SpecialTests", "JULIA_PROJECT" => nothing),
    )
    @test success(run(cmd))
end

view this post on Zulip Christopher Rackauckas (Jul 28 2024 at 13:10):

See how OrdinaryDiffEq does it

view this post on Zulip Christopher Rackauckas (Jul 28 2024 at 13:10):

We have different safetestsets which have if statements over them based on environment variables, and then CI is spawned over many environment variables

view this post on Zulip Christopher Rackauckas (Jul 28 2024 at 13:11):

That allows for separate CI runs to run different parts of the test, parallelizing the tests as well

view this post on Zulip Christopher Rackauckas (Jul 28 2024 at 13:11):

Distributed for that parallelism doesn't work too well because then you're limited by the core size you get on the free machine, which isn't much

view this post on Zulip Nathan Zimmerberg (Jul 28 2024 at 14:52):

I'm using Malt.jl for a similare case where I need to test that the data produced byZipArchives.jl can be read by ZipStreams.jl even if they may have incompatible dependancies. https://github.com/JuliaIO/ZipArchives.jl/blob/479436a31c01b2378861f456d67474205795776b/test/test_writer.jl#L126-L168

I found Malt.jl is much nicer than Distributed.jl for the case where the worker is in a different environment.

view this post on Zulip Daniel Wennberg (Jul 28 2024 at 20:00):

Christopher Rackauckas said:

We have different safetestsets which have if statements over them based on environment variables, and then CI is spawned over many environment variables

This is local small-potatoes exploratory development, I haven't set up CI on a hosting service for now (and I wouldn't want to push every dead-end attempt I make to the cloud anyway), so I'd like to be able to run the entire test suite with a simple ]test MyPackage in the REPL and no faffing with environment variables.

But what I take from OrdinaryDiffEq's setup is that I can of course just call Pkg.activate in the main test process rather than spawning a separate process. If I make sure the SpecialTests testset is the last one to run, that's really all I need.

Now to figure out if there's a safe way to do Pkg.activate when using TestItems and TestItemRunner, given that they run the testitems in unspecified order and in parallel.

view this post on Zulip Daniel Wennberg (Jul 28 2024 at 20:31):

Daniel Wennberg said:

I can of course just call Pkg.activate in the main test process rather than spawning a separate process

Hm, no, it's probably not a good idea to do this in a process where other versions of packages are already loaded. I'll stick to my solution with run from above, it's working great. But I'll follow OrdinaryDiffEq and put Pkg.develop in the code I'm evaluating rather than having the package already dev-ed in the SpecialTests environment---that way I don't have to check in the Manifest.

view this post on Zulip Daniel Wennberg (Jul 28 2024 at 20:42):

Updated solution:

@testset "SpecialTests" begin
    code = """
        using Pkg
        Pkg.develop(path="..")
        Pkg.instantiate()
        try
            include("SpecialTests/runtests.jl")
        finally
            redirect_stderr(devnull) do
                Pkg.rm("MyPackage")
                rm("SpecialTests/Manifest.toml")
            end
        end
    """
    cmd = addenv(
        Cmd(`$(Base.julia_cmd()) --eval "$code"`),
        Dict("JULIA_LOAD_PATH" => "@:SpecialTests", "JULIA_PROJECT" => nothing),
    )
    @test success(run(cmd))
end

Last updated: Nov 06 2024 at 04:40 UTC