Stream: helpdesk (published)

Topic: send `stdout` to multiple places


view this post on Zulip Mason Protter (Apr 20 2022 at 23:21):

So I want to intercept everything that's printed to stdout, and each time something is printed, I want to first put that output into a file, and then also continue printing to stdout.

I know I can do

open(filepath, "w+") do io
    redirect_stdout(myio) do
        f() # some code I have no control over which prints
    end
end

but this will only log to the file. I want to send the output to the file concurrently with stuff also being printed in the usual way. Anyone have any ideas as to how I can do this?

view this post on Zulip Mosè Giordano (Apr 21 2022 at 00:23):

Not exactly what you're asking, but in BinaryBuilder.jl we use https://github.com/JuliaPackaging/OutputCollectors.jl to collect stdout and stderr from subprocesses and print to screen and log to file at the same time

view this post on Zulip mbaz (Apr 21 2022 at 00:25):

I wonder if logging can be used for this: https://julialogging.github.io/reference/loggingextras/#LoggingExtras.TeeLogger-Tuple{Vararg{Base.CoreLogging.AbstractLogger}}

view this post on Zulip Brian Chen (Apr 21 2022 at 00:49):

Perhaps there's a package with some TeeIO type, but if not https://discourse.julialang.org/t/write-to-file-and-stdout/35042/3 looks pretty short.

view this post on Zulip Mason Protter (Apr 21 2022 at 16:11):

Ah thank you Brian, that’s nice because I’m trying to avoid dependencies where possible here.

view this post on Zulip Mason Protter (Apr 21 2022 at 16:12):

Is there any documentation out there on what the expected interface for an IO type is?

view this post on Zulip Fredrik Ekre (Apr 21 2022 at 16:27):

I wrote TeeStreams.jl, but doesn't work with redirect_stdio so you would need to pass the io to any write functions. Perhaps there is a way to fix that though, not sure.

view this post on Zulip Mason Protter (Apr 21 2022 at 19:03):

Hm yeah, rewriting the functions is not an option unfortunately

view this post on Zulip Mason Protter (Apr 21 2022 at 20:39):

@Fredrik Ekre okay, I was screwing around with the internals of Logging2 by @Chris Foster and I found a way to get redirect_stdio working on your TeeStreams.

using TeeStreams
using Logging2: LineBufferedIO

# adapted from Logging2.jl
function (redirect_func::Base.RedirectStdStream)(f::Function, io::TeeStream)
    prev_stream, stream_name =
        redirect_func.unix_fd == 0 ? (stdin, :stdin) :
        redirect_func.unix_fd == 1 ? (stdout, :stdout) :
        redirect_func.unix_fd == 2 ? (stderr, :stderr) :
        throw(ArgumentError("Not implemented to get old handle of fd except for stdio"))

    result = nothing

    output = LineBufferedIO(io)
    rd, rw = redirect_func()
    try
        @sync begin
            try
                Threads.@spawn write(output, rd) # loops while !eof(rd)
                result = f()
            finally
                # To close the read side of the pipe, we must close *all*
                # writers. This includes `rw`, but *also* the dup'd fd
                # created behind the scenes by redirect_func(). (To close
                # that, must call redirect_func() here with the prev stream.)
                close(rw)
                redirect_func(prev_stream)
            end
        end
    finally
        close(rd)
        close(output)
    end
    return result
end

(f::Base.RedirectStdStream)(io::TeeStream) = f(() -> nothing, io)

Now

julia> file = tempname()
"/var/folders/8j/wwv2d08x00v_5ss1g4dq6f2r0000gn/T/jl_3gEtIl"

julia> open(file, "w+") do io
           tio = TeeStream(io, stdout)
           redirect_stderr(tio) do
               redirect_stdout(tio) do
                   println("hi")
                   @warn "bye"
               end
           end
       end
hi
 Warning: bye
 @ Main REPL[21]:6

julia> read(file, String)
"hi\n┌ Warning: bye\n└ @ Main REPL[21]:6\n"

view this post on Zulip Mason Protter (Apr 21 2022 at 20:40):

For some reason I couldn't get redirect_stdio to work and instead had to nest redirect_stdout with redirect_stderr, but it seems to work.

I wonder if there's a way we can get this into TeeStreams?

view this post on Zulip Fredrik Ekre (Apr 21 2022 at 21:24):

Nice. Why is LineBufferedIO needed?

view this post on Zulip Mason Protter (Apr 21 2022 at 22:11):

I couldn’t find any other io construct that worked unfortunately

view this post on Zulip Mason Protter (Apr 21 2022 at 22:11):

Maybe @Chris Foster knows why?

view this post on Zulip Mason Protter (Apr 21 2022 at 22:11):

I guess it’s maybe a thread safety issue?

view this post on Zulip Fredrik Ekre (Apr 21 2022 at 22:17):

Your example works for me without it though

view this post on Zulip Mason Protter (Apr 21 2022 at 22:41):

Really? It crashes for me

view this post on Zulip Mason Protter (Apr 21 2022 at 22:41):

What did you replace LineBufferIO(io) with? just io?

view this post on Zulip Fredrik Ekre (Apr 21 2022 at 23:10):

Yea, and dropped the close of it.

view this post on Zulip Mason Protter (Apr 22 2022 at 00:18):

Yep, that would have been smart to drop

view this post on Zulip c42f (May 06 2022 at 07:30):

Cool you found that code in Logging2.jl and were able to reuse it :+1: That stuff I did with testing redirect_func.unix_fd is super ugly, but I think there's a missing abstraction in Base which forced me into it. (I didn't figure out exactly what Base should be doing though.) BTW, this will also fail on Julia 1.6 as Base appears to have had unintentional breaking changes in 1.7 (oh, I see Fredrik Ekre noticed that and fixed it already :) )

view this post on Zulip Mason Protter (May 06 2022 at 17:47):

Yeah, I also encountered that code in base and didn't even understand it well enough to copy it the way you did. I had to copy your copy :laughing:

view this post on Zulip Mason Protter (May 06 2022 at 17:48):

It definitely feels like a missing abstraction that there's no clean way to implement redirect functions without resorting to this stuff

view this post on Zulip Mason Protter (May 10 2022 at 23:34):

@Chris Foster do you know what it would take with this to support color and other IOContext stuff in the redirects? I've looked through the Base code and I don't see what it's doing that yours doesn't in terms of color stuff

view this post on Zulip Mason Protter (May 10 2022 at 23:35):

E.g. this loses color info

julia> using Logging, Logging2

julia> logger = current_logger();

julia> redirect_stdout(logger) do
           printstyled("Hello\n"; color = :blue)
       end
[ Info: Hello

view this post on Zulip c42f (May 11 2022 at 08:49):

That might be tricky.

For printstyled to work like that, we'd need to have get(stdout, :color, nothing) == true within the redirect_stdout block. So ideally we'd have stdout be an IOContext or something to capture the settings of the original stdout.

But the internal function Logging2._redirect_to_logger uses a PipeEndpoint to capture stdout in a reliable way. And I think it's necessary to use an operating system construct here (with an underlying file descriptor) for it to be possible to really capture stdout from the whole process, including from libc and not just from the Julia runtime.

view this post on Zulip Sukera (May 11 2022 at 08:51):

stdout per se has no notion of color - that's always a property of where the resulting text is then displayed after all

view this post on Zulip Sukera (May 11 2022 at 08:52):

all ANSI color codes are a in-band code to whatever is interpreting the bytestream, having stdout switch which it is requires knowing where it will end up

view this post on Zulip Mason Protter (May 11 2022 at 17:33):

I was more thinking in the coxtext of TeeStreams.jl but trying to adapt the question to logging parlance

view this post on Zulip Mason Protter (May 11 2022 at 17:39):

Basically, this works for embedding colors into a file:

julia> open(file, "w+") do io
           io = IOContext(io, :color => true)
           redirect_stderr(io) do
               redirect_stdout(io) do
                   println("hi")
                   @warn "bye"
               end
           end
       end

julia> read(file, String)
"hi\n\e[33m\e[1m┌ \e[22m\e[39m\e[33m\e[1mWarning: \e[22m\e[39mbye\n\e[33m\e[1m└ \e[22m\e[39m\e[90m@ Main REPL[46]:6\e[39m\n"

And I was trying to get IOContext to work with a TeeStream like this:

@eval TeeStreams function (redirect_func::Base.RedirectStdStream)(f::Function, io::IOContext{<:TeeStream})
    prev_stream =
        redirect_func.unix_fd == 1 ? stdout :
        redirect_func.unix_fd == 2 ? stderr :
        throw(ArgumentError("Can only redirect stdout and stderr to TeeStream."))
    return _redirect_internal(f, io, redirect_func, prev_stream)
end

but the colors just get ignored:

julia> open(file, "w+") do io
           tio = IOContext(TeeStream(io, stdout), :color => true)
           redirect_stderr(tio) do
               redirect_stdout(tio) do
                   println("hi")
                   @warn "bye"
               end
           end
       end
 Warning: bye
 @ Main REPL[52]:6
hi

julia> read(file, String)
"hi┌ Warning: bye\n└ @ Main REPL[52]:6\n\n"

view this post on Zulip Mason Protter (May 11 2022 at 17:40):

TeeStream(IOContext(io, :color => true), stdout) also didn't work

view this post on Zulip c42f (May 12 2022 at 08:45):

Ah, I didn't realize that IOContext already works with redirect_stdout (because IOContext <: AbstractPipe... what??)

In that case, you can probably get this to work.


Last updated: Nov 06 2024 at 04:40 UTC