Stream: helpdesk (published)

Topic: Lazy array concatenation

DrChainsaw (Jun 09 2023 at 11:55):

I have asked this before but got no response, so now I'm trying again with a slightly different formulation:

Is there some goto package for concatenating arrays without creating a new array (or modifying one of the arrays to concatenate)?

Use case is to avoid that things like FillArrays.jl and Arrow columns end up as an Array when concatenated. I know this is not exactly what one might call lazy concatenation, but I couldn't come up with a short name for it.

ComponentArrays.jl seems to do this, but it also seems to do other things (e.g. index components by name) so maybe there is something more tailor made.

Adam non-jedi Beckmeyer (Jun 09 2023 at 13:28):

Do you need the full array interface or just iteration? Base.Iterators.flatten gets you the iteration interface but not the array interface.

Adam non-jedi Beckmeyer (Jun 09 2023 at 13:31):

If you do need the array interface, I'm pretty sure that Lazy.jl's lists satisfy at least most of it, so you're looking for Lazy.flatten.

Adam non-jedi Beckmeyer (Jun 09 2023 at 13:36):

Nevermind. I forgot that Lazy's functions all assume that you're working with lists rather than letting you bring your own type.

Adam non-jedi Beckmeyer (Jun 09 2023 at 13:51):

Found it. You're looking for RecursiveArrayTools.ArrayPartition.

Michael Fiano (Jun 09 2023 at 15:24):

While not strictly what you are looking for, transducers might help for transformation pipelines where you don't want to allocate intermediates.

DrChainsaw (Jun 09 2023 at 15:39):

Thanks!

Unfortunately I forgot to state that I want it to be an indexable AbstractArray or whatever the requirements are for having it as columns in a DataFrame.

DrChainsaw (Jun 09 2023 at 15:40):

RecursiveArrayTools looks like it could work though at first glance.

Last updated: Oct 02 2023 at 04:34 UTC