Stream: helpdesk (published)

Topic: Is there a package to visualize a data pipeline?


view this post on Zulip Davi Sales Barreira (May 02 2022 at 13:27):

Hey fellas, is there a package that creates DAGs in order to visualize a data processing pipeline? I mean, suppose I have a database, and I extract some data and apply a series of transformations and joins and so on, and produce a final dataframe. I wish to visualize this pipe in a diagram...

view this post on Zulip TheCedarPrince (May 02 2022 at 14:43):

@Andrew Baas, do we have something like this in AlgebraicJulia?
Does Semagrams.jl fit here?

view this post on Zulip Andrew Baas (May 02 2022 at 14:52):

Semagrams wouldn't really work here, as that's more of a graphical editor for these.
Catlab.jl does this kind of visualization of wiring diagrams through Graphviz and includes an @program macro for defining a wiring diagram (see this page of the documentation https://algebraicjulia.github.io/Catlab.jl/dev/generated/wiring_diagrams/wd_cset/). This requires you to define the generators (component boxes with input/output signatures) as a presentation (using @present) and then describe the data pipeline as a series of functions in the @program macro.

There is also an imperative API for constructing these programmatically, which could be more useful in this case (https://algebraicjulia.github.io/Catlab.jl/dev/generated/wiring_diagrams/wiring_diagram_basics/)

view this post on Zulip Davi Sales Barreira (May 03 2022 at 12:38):

Thanks for the answers. I actually am looking for something more in the graphical... I guess I'll write some functions in Luxor to do what I want.

view this post on Zulip Braden (May 04 2022 at 15:18):

If one of the diagrams in Kroki is suitable (Pikchr for example), it might be easier to use Kroki.jl instead.


Last updated: Nov 06 2024 at 04:40 UTC