Mason Protter (Apr 20 2022 at 23:21): Mason Protter (Apr 20 2022 at 23:21):So I want to intercept everything that's printed to stdout, and each time something is printed, I want to first put that output into a file, and then also continue printing to stdout.
I know I can do
open(filepath, "w+") do io
redirect_stdout(myio) do
f() # some code I have no control over which prints
end
end
but this will only log to the file. I want to send the output to the file concurrently with stuff also being printed in the usual way. Anyone have any ideas as to how I can do this?
Mosè Giordano (Apr 21 2022 at 00:23):Not exactly what you're asking, but in BinaryBuilder.jl we use https://github.com/JuliaPackaging/OutputCollectors.jl to collect stdout and stderr from subprocesses and print to screen and log to file at the same time
mbaz (Apr 21 2022 at 00:25):I wonder if logging can be used for this: https://julialogging.github.io/reference/loggingextras/#LoggingExtras.TeeLogger-Tuple{Vararg{Base.CoreLogging.AbstractLogger}}
Brian Chen (Apr 21 2022 at 00:49):Perhaps there's a package with some TeeIO type, but if not https://discourse.julialang.org/t/write-to-file-and-stdout/35042/3 looks pretty short.
Mason Protter (Apr 21 2022 at 16:11):Ah thank you Brian, that’s nice because I’m trying to avoid dependencies where possible here.
Mason Protter (Apr 21 2022 at 16:12):Is there any documentation out there on what the expected interface for an IO type is?
Fredrik Ekre (Apr 21 2022 at 16:27):I wrote TeeStreams.jl, but doesn't work with redirect_stdio so you would need to pass the io to any write functions. Perhaps there is a way to fix that though, not sure.
Mason Protter (Apr 21 2022 at 19:03):Hm yeah, rewriting the functions is not an option unfortunately
Mason Protter (Apr 21 2022 at 20:39):@Fredrik Ekre okay, I was screwing around with the internals of Logging2 by @Chris Foster and I found a way to get redirect_stdio working on your TeeStreams.
using TeeStreams
using Logging2: LineBufferedIO
# adapted from Logging2.jl
function (redirect_func::Base.RedirectStdStream)(f::Function, io::TeeStream)
prev_stream, stream_name =
redirect_func.unix_fd == 0 ? (stdin, :stdin) :
redirect_func.unix_fd == 1 ? (stdout, :stdout) :
redirect_func.unix_fd == 2 ? (stderr, :stderr) :
throw(ArgumentError("Not implemented to get old handle of fd except for stdio"))
result = nothing
output = LineBufferedIO(io)
rd, rw = redirect_func()
try
@sync begin
try
Threads.@spawn write(output, rd) # loops while !eof(rd)
result = f()
finally
# To close the read side of the pipe, we must close *all*
# writers. This includes `rw`, but *also* the dup'd fd
# created behind the scenes by redirect_func(). (To close
# that, must call redirect_func() here with the prev stream.)
close(rw)
redirect_func(prev_stream)
end
end
finally
close(rd)
close(output)
end
return result
end
(f::Base.RedirectStdStream)(io::TeeStream) = f(() -> nothing, io)
Now
julia> file = tempname()
"/var/folders/8j/wwv2d08x00v_5ss1g4dq6f2r0000gn/T/jl_3gEtIl"
julia> open(file, "w+") do io
tio = TeeStream(io, stdout)
redirect_stderr(tio) do
redirect_stdout(tio) do
println("hi")
@warn "bye"
end
end
end
hi
┌ Warning: bye
└ @ Main REPL[21]:6
julia> read(file, String)
"hi\n┌ Warning: bye\n└ @ Main REPL[21]:6\n"
For some reason I couldn't get redirect_stdio to work and instead had to nest redirect_stdout with redirect_stderr, but it seems to work.
I wonder if there's a way we can get this into TeeStreams?
Fredrik Ekre (Apr 21 2022 at 21:24):Nice. Why is LineBufferedIO needed?
Mason Protter (Apr 21 2022 at 22:11):I couldn’t find any other io construct that worked unfortunately
Mason Protter (Apr 21 2022 at 22:11):Maybe @Chris Foster knows why?
Mason Protter (Apr 21 2022 at 22:11):I guess it’s maybe a thread safety issue?
Fredrik Ekre (Apr 21 2022 at 22:17):Your example works for me without it though
Mason Protter (Apr 21 2022 at 22:41):Really? It crashes for me
Mason Protter (Apr 21 2022 at 22:41):What did you replace LineBufferIO(io) with? just io?
Fredrik Ekre (Apr 21 2022 at 23:10):Yea, and dropped the close of it.
Mason Protter (Apr 22 2022 at 00:18):Yep, that would have been smart to drop
Chris Foster (May 06 2022 at 07:30):Cool you found that code in Logging2.jl and were able to reuse it :+1: That stuff I did with testing redirect_func.unix_fd is super ugly, but I think there's a missing abstraction in Base which forced me into it. (I didn't figure out exactly what Base should be doing though.) BTW, this will also fail on Julia 1.6 as Base appears to have had unintentional breaking changes in 1.7 (oh, I see Fredrik Ekre noticed that and fixed it already :) )
Mason Protter (May 06 2022 at 17:47):Yeah, I also encountered that code in base and didn't even understand it well enough to copy it the way you did. I had to copy your copy :laughing:
Mason Protter (May 06 2022 at 17:48):It definitely feels like a missing abstraction that there's no clean way to implement redirect functions without resorting to this stuff
Mason Protter (May 10 2022 at 23:34):@Chris Foster do you know what it would take with this to support color and other IOContext stuff in the redirects? I've looked through the Base code and I don't see what it's doing that yours doesn't in terms of color stuff
Mason Protter (May 10 2022 at 23:35):E.g. this loses color info
julia> using Logging, Logging2
julia> logger = current_logger();
julia> redirect_stdout(logger) do
printstyled("Hello\n"; color = :blue)
end
[ Info: Hello
That might be tricky.
For printstyled to work like that, we'd need to have get(stdout, :color, nothing) == true within the redirect_stdout block. So ideally we'd have stdout be an IOContext or something to capture the settings of the original stdout.
But the internal function Logging2._redirect_to_logger uses a PipeEndpoint to capture stdout in a reliable way. And I think it's necessary to use an operating system construct here (with an underlying file descriptor) for it to be possible to really capture stdout from the whole process, including from libc and not just from the Julia runtime.
Sukera (May 11 2022 at 08:51):stdout per se has no notion of color - that's always a property of where the resulting text is then displayed after all
Sukera (May 11 2022 at 08:52):all ANSI color codes are a in-band code to whatever is interpreting the bytestream, having stdout switch which it is requires knowing where it will end up
Mason Protter (May 11 2022 at 17:33):I was more thinking in the coxtext of TeeStreams.jl but trying to adapt the question to logging parlance
Mason Protter (May 11 2022 at 17:39):Basically, this works for embedding colors into a file:
julia> open(file, "w+") do io
io = IOContext(io, :color => true)
redirect_stderr(io) do
redirect_stdout(io) do
println("hi")
@warn "bye"
end
end
end
julia> read(file, String)
"hi\n\e[33m\e[1m┌ \e[22m\e[39m\e[33m\e[1mWarning: \e[22m\e[39mbye\n\e[33m\e[1m└ \e[22m\e[39m\e[90m@ Main REPL[46]:6\e[39m\n"
And I was trying to get IOContext to work with a TeeStream like this:
@eval TeeStreams function (redirect_func::Base.RedirectStdStream)(f::Function, io::IOContext{<:TeeStream})
prev_stream =
redirect_func.unix_fd == 1 ? stdout :
redirect_func.unix_fd == 2 ? stderr :
throw(ArgumentError("Can only redirect stdout and stderr to TeeStream."))
return _redirect_internal(f, io, redirect_func, prev_stream)
end
but the colors just get ignored:
julia> open(file, "w+") do io
tio = IOContext(TeeStream(io, stdout), :color => true)
redirect_stderr(tio) do
redirect_stdout(tio) do
println("hi")
@warn "bye"
end
end
end
┌ Warning: bye
└ @ Main REPL[52]:6
hi
julia> read(file, String)
"hi┌ Warning: bye\n└ @ Main REPL[52]:6\n\n"
TeeStream(IOContext(io, :color => true), stdout) also didn't work
Chris Foster (May 12 2022 at 08:45):Ah, I didn't realize that IOContext already works with redirect_stdout (because IOContext <: AbstractPipe... what??)
In that case, you can probably get this to work.
Last updated: Oct 02 2023 at 04:34 UTC