Alec (Feb 06 2024 at 02:13): Alec (Feb 06 2024 at 02:13):Two questions which may be related:
[2,-2,4] for the input array with autodiff(Forward)? julia> using Enzyme
julia> function sum_of_squares(x)
sum(x.^2)
end
sum_of_squares (generic function with 1 method)
julia> gradient(Reverse,sum_of_squares,[1.,-1.,2.]),
gradient(Forward,sum_of_squares,[1.,-1.,2.])
([2.0, -2.0, 4.0], (2.0, -2.0, 4.0))
julia> let
d1 = zeros(3)
d2 = zeros(3)
a = Enzyme.autodiff(Reverse,sum_of_squares,Duplicated([1.,-1.,2.],d1))
b = Enzyme.autodiff(Forward,sum_of_squares,Duplicated,Duplicated([1.,-1.,2.],d2))
d1,d2, a, b
end
([2.0, -2.0, 4.0], [0.0, 0.0, 0.0], ((nothing,),), (6.0, 0.0))
When you use forward mode you inject a tangent at the input (in your case a zero array) and get out the value and velocity of the output. To get the gradient you have to repeat that for each unit vector.
Alec (Mar 06 2024 at 04:56):Thank you, that helped enough to get the answers I was looking for.
To get the right derivative in b, d2 needs to be seeded with the initial dual number components of one:
julia> let
d1 = zeros(3)
d2 = [1.0, 1.0, 1.0] # Seed vector for forward mode
a = Enzyme.autodiff(Reverse, sum_of_squares, Duplicated([1.0, -1.0, 2.0], d1))
b = Enzyme.autodiff(Forward, sum_of_squares, Duplicated, Duplicated([1.0, -1.0, 2.0], d2))
d1, d2, a, b
end
([2.0, -2.0, 4.0], [1.0, 1.0, 1.0], ((nothing,),), (6.0, 4.0))
And to get the components of the gradient, I need to get the component from each unit vector:
julia> let
d1 = zeros(3)
a = Enzyme.autodiff(Reverse, sum_of_squares, Duplicated([1.,-1.,2.], d1))
b1 = Enzyme.autodiff(Forward, sum_of_squares, Duplicated([1.,-1.,2.], [1.0, 0.0, 0.0]))
b2 = Enzyme.autodiff(Forward, sum_of_squares, Duplicated([1.,-1.,2.], [0.0, 1.0, 0.0]))
b3 = Enzyme.autodiff(Forward, sum_of_squares, Duplicated([1.,-1.,2.], [0.0, 0.0, 1.0]))
d1, a, [b1, b2, b3]
end
([2.0, -2.0, 4.0], ((nothing,),), [(2.0,), (-2.0,), (4.0,)])
Alec has marked this topic as resolved.
Last updated: Dec 28 2024 at 04:38 UTC