I'm getting the following warnings when precompiling my package:
│ [pid 26608] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ tcp[864] 000002281cc4f140->0000022821ce50a0
│ tcp[884] 000002281cc4e430->00000228206c4a60
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
The source is that the precompile workload uses HTTP.jl get
to download the data to work on. This is an integral part of what the package does, not something done just for precompiling and I really don't want to ship the data with the package.
I can't see anything in HTTP which keeps TCP sockets open after requests have been received. Is it 'just' a matter of waiting for the handles to be garbage collected or something? I have tried to add (both busy and non-busy) waits for up to 20 seconds in the precompile statement but it only seems to be able to get rid of one of the handles from the warning.
Are there anything bad that can happen if the warnings are not resolved (like leaking resources)? The precompile stuff helps alot with initial latency so I don't want to just remove it.
But why do you have to do it during precompile?
Felix Kastner said:
But why do you have to do it during precompile?
Because I'm using PrecompileTools.jl to reduce ttfx.
Probably because HTTP keeps the connection open for faster subsequent connections. Perhaps try with HTTP.Connections.closeall()
?
Once you fix it, Aqua provides a regression test for this FYI: https://juliatesting.github.io/Aqua.jl/stable/persistent_tasks/
Couldn't you download the data during Pkg.build
and not during precompilation? I seem to recall having seen some packages do this.
Fredrik Ekre said:
Probably because HTTP keeps the connection open for faster subsequent connections. Perhaps try with
HTTP.Connections.closeall()
?
Close, but no cigar for me :(. However, this worked:
foreach(vs -> foreach(close, vs), values(HTTP.Connections.OPENSSL_POOL[].keyedvalues))
It seems a bit too reliant on internal details to be comfortable, but it looks like it is possible to supply an own pool through the get
call (not sure how the kwargs propagate here). However, could it be a bug in HTTP that the above does not seem to happen when calling Connections.closeall()
?
Once you fix it, Aqua provides a regression test for this FYI:
Good tip, although I'm a bit scared of what Aqua will think of this package :)
Couldn't you download the data during Pkg.build and not during precompilation? I seem to recall having seen some packages do this.
Ah, now I understand what you meant. This might be some kind of last resort option in case all else fails. The problem is that the code I want to precompile does quite a few calculations to find out what data to download, so it is not like the data which HTTP
gets is some kind of input stimuli. I might be able to work around that though.
Last updated: Nov 22 2024 at 04:41 UTC