✔ Is "double free or corruption" my fault? · helpdesk (published)

Stream: helpdesk (published)

Topic: ✔ Is "double free or corruption" my fault?

Timothy (Jun 13 2022 at 05:14):

If I see:

double free or corruption (out)
double free or corruption (!prev)

signal (6): Aborted
in expression starting at REPL[2]:1
[1] 104817 IOT instruction (core dumped) julia --project=@. --threads=32

is this a me-problem or a julialang-problem?

Timothy (Jun 13 2022 at 05:16):

For context, this seems to be the code triggering it:

rows = fill(DataFrameRow[], threads())
@threads for r in axes(gnomad1, 1)
   # stuff
  push!(rows[threadid()], rowd[1, :])
end

Timothy (Jun 13 2022 at 05:23):

Tried running again and I see

corrupted size vs. prev_size while consolidating

signal (6): Aborted
in expression starting at REPL[2]:1
__pthread_kill_implementation at /lib64/libc.so.6 (unknown line)
raise at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__libc_message at /lib64/libc.so.6 (unknown line)
malloc_printerr at /lib64/libc.so.6 (unknown line)
_int_free at /lib64/libc.so.6 (unknown line)
free at /lib64/libc.so.6 (unknown line)
jl_realloc_aligned at /buildworker/worker/package_linux64/build/src/gc.c:249 [inlined]
gc_managed_realloc_ at /buildworker/worker/package_linux64/build/src/gc.c:3540 [inlined]
jl_gc_managed_realloc at /buildworker/worker/package_linux64/build/src/gc.c:3557
array_resize_buffer at /buildworker/worker/package_linux64/build/src/array.c:681
jl_array_grow_at_end at /buildworker/worker/package_linux64/build/src/array.c:904 [inlined]
jl_array_grow_end at /buildworker/worker/package_linux64/build/src/array.c:968
_growend! at ./array.jl:948 [inlined]
push! at ./array.jl:995
...

Sukera (Jun 13 2022 at 06:06):

threadid() is not a stable way to query this

Sukera (Jun 13 2022 at 06:07):

also, your rows holds the exact same array multiple times

Sukera (Jun 13 2022 at 06:08):

julia> mutable struct Example
         x
       end

julia> rows = fill(Example(rand(UInt)), 3)
3-element Vector{Example}:
 Example(0x32bdaaec78c24bc5)
 Example(0x32bdaaec78c24bc5)
 Example(0x32bdaaec78c24bc5)

julia> rows[1] === rows[2]
true

Sukera (Jun 13 2022 at 06:09):

either fill in a loop explicitly or use a comprehension, otherwise you're possibly trying to grow the exact same array from multiple threads

Timothy (Jun 13 2022 at 07:00):

Ah, thanks for that hint. I'll try [DataFrameRow[] for _ in 1:nthreads()]. Do you know if this could also cause an UndefRefError? I ask because it will take me ~1.5h to see if the fix worked.