Stream: helpdesk (published)

Topic: Thread-safe file IO in Julia?

Robbie Rosati (Sep 24 2021 at 19:01):

I have a multi-threaded loop running some simulations each iteration. I'd like to be able to save the results intermediately, something like

using Base.Threads
using JLD2

@Threads.threads for i in 1:100
    simulation_data = run_simulation(i)
    # something blocking here, a mutex or atomic block??
    jldopen("output.jld2","a")  do f
        write(f, "hdf/path/to/data/$i",  simulation_data)
    end
end

Is something like this possible? I couldn't find anything other than atomic operations on numeric types in the docs.
In my particular application, it's not too bad to wait until after the loop and do all the IO single-threaded, but I thought I'd ask anyway.

Jameson Nash (Sep 24 2021 at 20:23):

yes, you'd need to hold a lock around that bit of IO code

Last updated: Oct 02 2023 at 04:34 UTC