Stream: helpdesk (published)

Topic: Documenter.jl parallelize running of separate pages

Mason Protter (Mar 10 2025 at 16:30):

I have a Documenter.jl workflow that needs to expand a bunch of Literate.jl tutorial scripts, expand them (i.e. run all the code inside), and then generate the docs from that, and do this in Github Actions CI.

Running all the code in these tutorials is very time consuming, but each one should be independent. Is there a way to have multiple Github Actions processes, and then have each one work on a subset of the .md, and the collect them all together at the end?

Cédric Belmant (Mar 10 2025 at 16:36):

Have you considered using multithreading to process the .jl files, or is that not an option?

Cédric Belmant (Mar 10 2025 at 16:37):

I have been doing it here for example: https://github.com/JuliaGPU/Vulkan.jl/blob/53fad947e1c5bab36cc5f6d4e1945227bede4b3e/docs/make.jl#L21-L31 and never ran into any issues.

Cédric Belmant (Mar 10 2025 at 16:39):

Although I don't think code runs at this stage, so were you talking about processing the generated .md files during makedocs?

Mason Protter (Mar 10 2025 at 16:40):

So I thought Literate.markdown there was just generating a markdown file from the .jl file, but not actually running the code inside of it. It seemed to me that it was makedocs that actually ran the code, and I don't see how to parallelize that

Mason Protter (Mar 10 2025 at 16:40):

Cédric Belmant said:

Although I don't think code runs at this stage, so were you talking about processing the generated .md files during makedocs?

yeah exactly

Cédric Belmant (Mar 10 2025 at 16:41):

Right, I read Literate.jl and thought it would be the previous stage (because I think once processed from .jl into .md it doesn't have anything to do with Literate anymore)

Mason Protter (Mar 10 2025 at 16:43):

Hm, so looking at the Literate.jl docs though, it looks like there is a execute kwarg you can set to tell Literate.jl to execute the code. Interesting, I wonder if it'll get re-run by makedocs after that if I set it to true

Cédric Belmant (Mar 10 2025 at 16:46):

Otherwise, having Documenter allow executing a subset of files then merging the results seems fairly involved, unless it already has the infrastructure to selectively run stages and has appropriate data structures to hold & merge the results of evaluation.

Cédric Belmant (Mar 10 2025 at 16:46):

That would be pretty neat to have though, if it doesn't make it overly complex

Last updated: Apr 25 2025 at 04:43 UTC