So I want to intercept everything that's printed to stdout, and each time something is printed, I want to first put that output into a file, and then also continue printing to stdout
.
I know I can do
open(filepath, "w+") do io
redirect_stdout(myio) do
f() # some code I have no control over which prints
end
end
but this will only log to the file. I want to send the output to the file concurrently with stuff also being printed in the usual way. Anyone have any ideas as to how I can do this?
Not exactly what you're asking, but in BinaryBuilder.jl we use https://github.com/JuliaPackaging/OutputCollectors.jl to collect stdout and stderr from subprocesses and print to screen and log to file at the same time
I wonder if logging can be used for this: https://julialogging.github.io/reference/loggingextras/#LoggingExtras.TeeLogger-Tuple{Vararg{Base.CoreLogging.AbstractLogger}}
Perhaps there's a package with some TeeIO
type, but if not https://discourse.julialang.org/t/write-to-file-and-stdout/35042/3 looks pretty short.
Ah thank you Brian, that’s nice because I’m trying to avoid dependencies where possible here.
Is there any documentation out there on what the expected interface for an IO
type is?
I wrote TeeStreams.jl, but doesn't work with redirect_stdio
so you would need to pass the io
to any write functions. Perhaps there is a way to fix that though, not sure.
Hm yeah, rewriting the functions is not an option unfortunately
@Fredrik Ekre okay, I was screwing around with the internals of Logging2 by @Chris Foster and I found a way to get redirect_stdio
working on your TeeStreams
.
using TeeStreams
using Logging2: LineBufferedIO
# adapted from Logging2.jl
function (redirect_func::Base.RedirectStdStream)(f::Function, io::TeeStream)
prev_stream, stream_name =
redirect_func.unix_fd == 0 ? (stdin, :stdin) :
redirect_func.unix_fd == 1 ? (stdout, :stdout) :
redirect_func.unix_fd == 2 ? (stderr, :stderr) :
throw(ArgumentError("Not implemented to get old handle of fd except for stdio"))
result = nothing
output = LineBufferedIO(io)
rd, rw = redirect_func()
try
@sync begin
try
Threads.@spawn write(output, rd) # loops while !eof(rd)
result = f()
finally
# To close the read side of the pipe, we must close *all*
# writers. This includes `rw`, but *also* the dup'd fd
# created behind the scenes by redirect_func(). (To close
# that, must call redirect_func() here with the prev stream.)
close(rw)
redirect_func(prev_stream)
end
end
finally
close(rd)
close(output)
end
return result
end
(f::Base.RedirectStdStream)(io::TeeStream) = f(() -> nothing, io)
Now
julia> file = tempname()
"/var/folders/8j/wwv2d08x00v_5ss1g4dq6f2r0000gn/T/jl_3gEtIl"
julia> open(file, "w+") do io
tio = TeeStream(io, stdout)
redirect_stderr(tio) do
redirect_stdout(tio) do
println("hi")
@warn "bye"
end
end
end
hi
┌ Warning: bye
└ @ Main REPL[21]:6
julia> read(file, String)
"hi\n┌ Warning: bye\n└ @ Main REPL[21]:6\n"
For some reason I couldn't get redirect_stdio
to work and instead had to nest redirect_stdout
with redirect_stderr
, but it seems to work.
I wonder if there's a way we can get this into TeeStreams?
Nice. Why is LineBufferedIO
needed?
I couldn’t find any other io construct that worked unfortunately
Maybe @Chris Foster knows why?
I guess it’s maybe a thread safety issue?
Your example works for me without it though
Really? It crashes for me
What did you replace LineBufferIO(io)
with? just io
?
Yea, and dropped the close of it.
Yep, that would have been smart to drop
Cool you found that code in Logging2.jl and were able to reuse it :+1: That stuff I did with testing redirect_func.unix_fd
is super ugly, but I think there's a missing abstraction in Base
which forced me into it. (I didn't figure out exactly what Base should be doing though.) BTW, this will also fail on Julia 1.6 as Base appears to have had unintentional breaking changes in 1.7 (oh, I see Fredrik Ekre noticed that and fixed it already :) )
Yeah, I also encountered that code in base and didn't even understand it well enough to copy it the way you did. I had to copy your copy :laughing:
It definitely feels like a missing abstraction that there's no clean way to implement redirect
functions without resorting to this stuff
@Chris Foster do you know what it would take with this to support color and other IOContext stuff in the redirects? I've looked through the Base code and I don't see what it's doing that yours doesn't in terms of color stuff
E.g. this loses color info
julia> using Logging, Logging2
julia> logger = current_logger();
julia> redirect_stdout(logger) do
printstyled("Hello\n"; color = :blue)
end
[ Info: Hello
That might be tricky.
For printstyled to work like that, we'd need to have get(stdout, :color, nothing) == true
within the redirect_stdout
block. So ideally we'd have stdout
be an IOContext
or something to capture the settings of the original stdout.
But the internal function Logging2._redirect_to_logger
uses a PipeEndpoint
to capture stdout in a reliable way. And I think it's necessary to use an operating system construct here (with an underlying file descriptor) for it to be possible to really capture stdout from the whole process, including from libc and not just from the Julia runtime.
stdout
per se has no notion of color - that's always a property of where the resulting text is then displayed after all
all ANSI color codes are a in-band code to whatever is interpreting the bytestream, having stdout
switch which it is requires knowing where it will end up
I was more thinking in the coxtext of TeeStreams.jl but trying to adapt the question to logging parlance
Basically, this works for embedding colors into a file:
julia> open(file, "w+") do io
io = IOContext(io, :color => true)
redirect_stderr(io) do
redirect_stdout(io) do
println("hi")
@warn "bye"
end
end
end
julia> read(file, String)
"hi\n\e[33m\e[1m┌ \e[22m\e[39m\e[33m\e[1mWarning: \e[22m\e[39mbye\n\e[33m\e[1m└ \e[22m\e[39m\e[90m@ Main REPL[46]:6\e[39m\n"
And I was trying to get IOContext
to work with a TeeStream
like this:
@eval TeeStreams function (redirect_func::Base.RedirectStdStream)(f::Function, io::IOContext{<:TeeStream})
prev_stream =
redirect_func.unix_fd == 1 ? stdout :
redirect_func.unix_fd == 2 ? stderr :
throw(ArgumentError("Can only redirect stdout and stderr to TeeStream."))
return _redirect_internal(f, io, redirect_func, prev_stream)
end
but the colors just get ignored:
julia> open(file, "w+") do io
tio = IOContext(TeeStream(io, stdout), :color => true)
redirect_stderr(tio) do
redirect_stdout(tio) do
println("hi")
@warn "bye"
end
end
end
┌ Warning: bye
└ @ Main REPL[52]:6
hi
julia> read(file, String)
"hi┌ Warning: bye\n└ @ Main REPL[52]:6\n\n"
TeeStream(IOContext(io, :color => true), stdout)
also didn't work
Ah, I didn't realize that IOContext
already works with redirect_stdout
(because IOContext <: AbstractPipe
... what??)
In that case, you can probably get this to work.
Last updated: Nov 22 2024 at 04:41 UTC