Help with `Cat` and Transducers · helpdesk (published)

I am working on some funcitonality for FinanceModels.jl and running into an issue that I'm not able to debug effectively as I'm getting lost in the Transducers internals. Here's a MWE version. The issue is that I'd like to be able to concatenate a negated version of a normal object. I can collect the Foo, a Negated Foo, a Cated Foo and Foo but can't Cat a Negated `Foo.

using Transducers

struct Foo
    n
end

struct Negate{T}
    x::T
end

Base.:-(x::Foo) = Negate(x)
function Transducers.__foldl__(rf, val, x::Foo)
    for i in 1:x.n
        val = Transducers.@next(rf, val, i)
    end
    return Transducers.complete(rf, val)
end

function Transducers.asfoldable(p::Negate{T}) where {T}
    p.x|> Map(-)
end

See how the components and tuple of Foos work, but once I introduce the Negate into the tuple it fails.

julia> Negate(Foo(5)) |> Map(identity) |> collect
5-element Vector{Int64}:
 -1
 -2
 -3
 -4
 -5

julia> (Foo(5),Foo(3)) |> Cat() |> Map(identity) |> collect
8-element Vector{Int64}:
 1
 2
 3
 4
 5
 1
 2
 3

julia> (Foo(5),Negate(Foo(3))) |> Cat() |> Map(identity) |> collect
ERROR: MethodError: no method matching iterate(::Negate{Foo})

Closest candidates are:
  iterate(::Union{LinRange, StepRangeLen})
   @ Base range.jl:880
  iterate(::Union{LinRange, StepRangeLen}, ::Integer)
   @ Base range.jl:880
  iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}}
   @ Base dict.jl:698
  ...

Stacktrace:
  [1] __foldl__(rf::Transducers.Reduction{Transducers.NoComplete, Transducers.Reduction{Map{typeof(identity)}, Transducers.Reduction{Map{Type{BangBang.NoBang.SingletonVector}}, Transducers.BottomRF{Transducers.AdHocRF{typeof(BangBang.collector), typeof(identity), typeof(append!!), typeof(identity), typeof(identity), Nothing}}}}}, init::BangBang.SafeCollector{Vector{Int64}}, coll::Negate{Foo})
    @ Transducers ~/.julia/packages/Transducers/xbs8O/src/processes.jl:154
  [2] foldl_nocomplete
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:352 [inlined]
  [3] next
    @ ~/.julia/packages/Transducers/xbs8O/src/library.jl:157 [inlined]
  [4] foldlargs
    @ ~/.julia/packages/Transducers/xbs8O/src/core.jl:181 [inlined]
  [5] foldlargs
    @ ~/.julia/packages/Transducers/xbs8O/src/core.jl:183 [inlined]
  [6] __foldl__
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:176 [inlined]
  [7] #transduce#142
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:519 [inlined]
  [8] transduce
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:508 [inlined]
  [9] transduce(xform::Transducers.Composition{Cat, Transducers.Composition{Map{typeof(identity)}, Map{Type{BangBang.NoBang.SingletonVector}}}}, f::Transducers.AdHocRF{typeof(BangBang.collector), typeof(identity), typeof(append!!), typeof(identity), typeof(identity), Nothing}, init::BangBang.SafeCollector{Empty{Vector{Union{}}}}, coll::Tuple{Foo, Negate{Foo}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Transducers ~/.julia/packages/Transducers/xbs8O/src/processes.jl:502
 [10] transduce
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:500 [inlined]
 [11] _collect(xf::Transducers.Composition{Cat, Map{typeof(identity)}}, coll::Tuple{Foo, Negate{Foo}}, #unused#::Transducers.SizeChanging, #unused#::Base.HasLength)
    @ Transducers ~/.julia/packages/Transducers/xbs8O/src/processes.jl:806
 [12] collect
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:802 [inlined]
 [13] collect
    @ ~/.julia/packages/Transducers/xbs8O/src/processes.jl:803 [inlined]
 [14] |>(x::Transducers.Eduction{Transducers.Reduction{Cat, Transducers.Reduction{Map{typeof(identity)}, Transducers.BottomRF{Completing{typeof(push!!)}}}}, Tuple{Foo, Negate{Foo}}}, f::typeof(collect))
    @ Base ./operators.jl:907
 [15] top-level scope
    @ REPL[46]:1

My real use case is simply being able to model an obligation/liability as the negated version of a contract in FinanceModels.

Alec (Nov 01 2023 at 19:42):

It appears that a reducible is defined via asfoldable instead of __foldl__ then it will error with (x,x) |> Cat(). See, for example, this example taken from the Transducers documentation:

struct VecOfVec{T}
    vectors::Vector{Vector{T}}
end
function Transducers.__foldl__(rf, val, vov::VecOfVec)
    for vector in vov.vectors
        for x in vector
            val = Transducers.@next(rf, val, x)
        end
    end
    return Transducers.complete(rf, val)
end


struct VecOfVec2{T}
    vectors::Vector{T}
end
Transducers.asfoldable(vov::VecOfVec2) = vov.vectors |> Cat()

vov = VecOfVec(collect.([1:n for n in 1:3]))
vov2 = VecOfVec2(collect.([1:n for n in 1:3]))

(vov,vov).  |> Cat() |> Map(identity) |> collect # this does not error
(vov2,vov2) |> Cat() |> Map(identity) |> collect # this errors

Alec (Nov 04 2023 at 03:04):

I think that the prior message may be worthy of a Transducers issue? Curious what other think.

Alec (Nov 04 2023 at 03:05):

For my particular use case where I had a reducible wrapper type, I was able to define a __foldl__ which unwrapped the Eduction of the wrapped type and then applied the intended Eduction to the wrapper type:

Mason Protter (Nov 04 2023 at 08:25):

Hm yeah that's probebly just a missing application of asfoldable in the implementation of Cat somewhere, can you open an issue?

Mason Protter (Nov 05 2023 at 18:51):

In about 15 or 20 minutes v0.4.79 should be available on the general registry with the fix.

Stream: helpdesk (published)

Topic: Help with `Cat` and Transducers

Alec (Nov 01 2023 at 03:55):

Alec (Nov 01 2023 at 19:42):

Alec (Nov 04 2023 at 03:04):

Alec (Nov 04 2023 at 03:05):

Mason Protter (Nov 04 2023 at 08:25):

Mason Protter (Nov 05 2023 at 18:51):