Timothy (Jun 13 2022 at 05:14): Timothy (Jun 13 2022 at 05:14):If I see:
double free or corruption (out)
double free or corruption (!prev)signal (6): Aborted
in expression starting at REPL[2]:1
[1] 104817 IOT instruction (core dumped) julia --project=@. --threads=32
is this a me-problem or a julialang-problem?
Timothy (Jun 13 2022 at 05:16):For context, this seems to be the code triggering it:
rows = fill(DataFrameRow[], threads())
@threads for r in axes(gnomad1, 1)
# stuff
push!(rows[threadid()], rowd[1, :])
end
Tried running again and I see
corrupted size vs. prev_size while consolidating
signal (6): Aborted
in expression starting at REPL[2]:1
__pthread_kill_implementation at /lib64/libc.so.6 (unknown line)
raise at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__libc_message at /lib64/libc.so.6 (unknown line)
malloc_printerr at /lib64/libc.so.6 (unknown line)
_int_free at /lib64/libc.so.6 (unknown line)
free at /lib64/libc.so.6 (unknown line)
jl_realloc_aligned at /buildworker/worker/package_linux64/build/src/gc.c:249 [inlined]
gc_managed_realloc_ at /buildworker/worker/package_linux64/build/src/gc.c:3540 [inlined]
jl_gc_managed_realloc at /buildworker/worker/package_linux64/build/src/gc.c:3557
array_resize_buffer at /buildworker/worker/package_linux64/build/src/array.c:681
jl_array_grow_at_end at /buildworker/worker/package_linux64/build/src/array.c:904 [inlined]
jl_array_grow_end at /buildworker/worker/package_linux64/build/src/array.c:968
_growend! at ./array.jl:948 [inlined]
push! at ./array.jl:995
...
Sukera (Jun 13 2022 at 06:06):threadid() is not a stable way to query this
Sukera (Jun 13 2022 at 06:07):also, your rows holds the exact same array multiple times
Sukera (Jun 13 2022 at 06:08):julia> mutable struct Example
x
end
julia> rows = fill(Example(rand(UInt)), 3)
3-element Vector{Example}:
Example(0x32bdaaec78c24bc5)
Example(0x32bdaaec78c24bc5)
Example(0x32bdaaec78c24bc5)
julia> rows[1] === rows[2]
true
either fill in a loop explicitly or use a comprehension, otherwise you're possibly trying to grow the exact same array from multiple threads
Timothy (Jun 13 2022 at 07:00):Ah, thanks for that hint. I'll try [DataFrameRow[] for _ in 1:nthreads()]. Do you know if this could also cause an UndefRefError? I ask because it will take me ~1.5h to see if the fix worked.
Sukera (Jun 13 2022 at 08:26):don't think so? I'd have to take a look at the stacktrace
Timothy (Jun 13 2022 at 08:27):Well, I'll find out in a few minutes :sweat_smile:
Sukera (Jun 13 2022 at 08:27):1.5h seems like a very long time, do you not have a small-ish check program to run for testing purposes?
Sukera (Jun 13 2022 at 08:27):ah, yeah :joy:
Sukera (Jun 13 2022 at 08:27):or is that already your small-ish check?
Timothy (Jun 13 2022 at 08:28):I've stumbled across a bunch of issues that don't happen with small data sets, I'm guessing they're friendlier on race conditions etc.
Timothy (Jun 13 2022 at 08:32):Wooo! I think it worked
Sukera (Jun 13 2022 at 08:34):haha yeah, I can see that
Sukera (Jun 13 2022 at 08:34):resiliency against data races is hard (unless you go for Rust, then it's a little easier)
Timothy (Jun 13 2022 at 08:34):The moments of anticipation when the ETA counted down to 0s and it was processing were killing me :laughing: worse than a horror movie
Timothy (Jun 13 2022 at 08:35):
Sukera (Jun 13 2022 at 09:28):nice
Last updated: Oct 02 2023 at 04:34 UTC