Stream: helpdesk (published)

Topic: PrecompileTools with HTTP.jl

DrChainsaw (Dec 11 2023 at 15:56):

I'm getting the following warnings when precompiling my package:

│  [pid 26608] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   tcp[864]           000002281cc4f140->0000022821ce50a0
│   tcp[884]           000002281cc4e430->00000228206c4a60
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.

The source is that the precompile workload uses HTTP.jl get to download the data to work on. This is an integral part of what the package does, not something done just for precompiling and I really don't want to ship the data with the package.

I can't see anything in HTTP which keeps TCP sockets open after requests have been received. Is it 'just' a matter of waiting for the handles to be garbage collected or something? I have tried to add (both busy and non-busy) waits for up to 20 seconds in the precompile statement but it only seems to be able to get rid of one of the handles from the warning.

Are there anything bad that can happen if the warnings are not resolved (like leaking resources)? The precompile stuff helps alot with initial latency so I don't want to just remove it.

Felix Kastner (Dec 11 2023 at 18:14):

But why do you have to do it during precompile?

DrChainsaw (Dec 11 2023 at 18:36):

Felix Kastner said:

But why do you have to do it during precompile?

Because I'm using PrecompileTools.jl to reduce ttfx.

Fredrik Ekre (Dec 11 2023 at 22:58):

Probably because HTTP keeps the connection open for faster subsequent connections. Perhaps try with HTTP.Connections.closeall()?

Eric Hanson (Dec 12 2023 at 00:21):

Once you fix it, Aqua provides a regression test for this FYI: https://juliatesting.github.io/Aqua.jl/stable/persistent_tasks/

Felix Kastner (Dec 12 2023 at 06:06):

Couldn't you download the data during Pkg.build and not during precompilation? I seem to recall having seen some packages do this.

DrChainsaw (Dec 12 2023 at 07:35):

Fredrik Ekre said:

Probably because HTTP keeps the connection open for faster subsequent connections. Perhaps try with HTTP.Connections.closeall()?

Close, but no cigar for me :(. However, this worked:

foreach(vs -> foreach(close, vs), values(HTTP.Connections.OPENSSL_POOL[].keyedvalues))

It seems a bit too reliant on internal details to be comfortable, but it looks like it is possible to supply an own pool through the get call (not sure how the kwargs propagate here). However, could it be a bug in HTTP that the above does not seem to happen when calling Connections.closeall()?

Once you fix it, Aqua provides a regression test for this FYI:
Good tip, although I'm a bit scared of what Aqua will think of this package :)

Couldn't you download the data during Pkg.build and not during precompilation? I seem to recall having seen some packages do this.

Ah, now I understand what you meant. This might be some kind of last resort option in case all else fails. The problem is that the code I want to precompile does quite a few calculations to find out what data to download, so it is not like the data which HTTP gets is some kind of input stimuli. I might be able to work around that though.

Last updated: Nov 06 2024 at 04:40 UTC