Stream: helpdesk (published)

Topic: Reinterpreting buffers

Júlio Hoffimann (Apr 15 2022 at 13:08):

Is there an idiom in Julia for taking a chunk of memory and reinterpreting it as a given struct?

For example, suppose I have a parametric type:

struct Foo{T}
  a::T
  b::Int
end

foo = Foo(1.0, 2)

How can I "dump" this object into a memory chunk and then later reinterpret this memory chunk as a Foo{T}?

Chad Scherrer (Apr 15 2022 at 13:16):

Do you mean like this?

julia> buf = reinterpret(reshape, Float64, [foo])
2×1 reinterpret(reshape, Float64, ::Vector{Foo{Float64}}) with eltype Float64:
 1.0
 1.0e-323

julia> reinterpret(Foo{Float64}, buf)
1×1 reinterpret(Foo{Float64}, reinterpret(reshape, Float64, ::Vector{Foo{Float64}})):
 Foo{Float64}(1.0, 2)

Júlio Hoffimann (Apr 15 2022 at 13:25):

Probably @Chad Scherrer :smile: I wonder why we need the reshape and Float64 in the first reinterpret call, will read the docstring...

chriselrod (Apr 16 2022 at 03:10):

reshape is nice so that you have 1 column per foo, with rows corresponding to the data of foo.

Júlio Hoffimann (Apr 16 2022 at 10:47):

To be honest I find it a bit strange that reshape is an argument to the function. Maybe it would be more intuitive to introduce a keyword reshape=true or ascolumns=true in future releases? Can reinterpret accept other functions that are not reshape?

Chad Scherrer (Apr 16 2022 at 13:34):

Not sure. Yeah, the reshape argument is weird to me too

Mason Protter (Apr 16 2022 at 15:36):

If you don't like the reshape you don't have to use it

Mason Protter (Apr 16 2022 at 15:36):

You can just reinterpret it as bools or UInt8s or whatever you like

Mason Protter (Apr 16 2022 at 15:37):

julia> buf = reinterpret(UInt8, [Foo(1.0, 2), Foo(3.0, 4)])
32-element reinterpret(UInt8, ::Vector{Foo{Float64}}):
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0xf0
 0x3f
 0x02
 0x00
 0x00
 0x00
 0x00
    ⋮
 0x00
 0x00
 0x00
 0x08
 0x40
 0x04
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00

julia> buf[1] = 0x01
0x01

julia> reinterpret(Foo{Float64}, buf)
2-element Vector{Foo{Float64}}:
 Foo{Float64}(1.0000000000000002, 2)
 Foo{Float64}(3.0, 4)

Jakob Nybo Nissen (Apr 16 2022 at 18:06):

Another option is to load a pointer of the given type from the memory. This is more efficient than creating a reinterpreted buffer, but it's not exactly idiomatic.
Something like:

julia> struct Foo
           x::UInt32
           y::Char
           z::Int64
       end

julia> x = (55, 1.0)
(55, 1.0)

julia> unsafe_load(Ptr{Foo}(pointer_from_objref(Ref(x))))
Foo(0x00000037, '\0', 4607182418800017408)

If you use Ref + unsafe_load, the allocation will optimize away and it will be a no-op.
Of course you need to be a bit careful with this:

• Use GC.@preserve judiciously to prevent the GC from messing up your pointers
• This technique can circumvent inner constructors, which can make your code unsound if invalid instances are being created
• Any padding in your structs will also be loaded, which can also create invalid instances

Last updated: Oct 02 2023 at 04:34 UTC