Stream: helpdesk (published)

Topic: Meta.parse with LineNumberNodes

Fredrik Ekre (Jul 05 2021 at 08:31):

Is there a way to get LineNumberNodes inserted into the expression returned by Meta.parse similar to Meta.parseall?

julia> Meta.parse("""
           a = 1 + 1
           b = a + 3
       """, 1)
(:(a = 1 + 1), 15)

julia> Meta.parseall("""
           a = 1 + 1
           b = a + 3
       """; filename="REPL")
:($(Expr(:toplevel, :(#= REPL:1 =#), :(a = 1 + 1), :(#= REPL:2 =#), :(b = a + 3))))

Simeon Schaub (Jul 05 2021 at 08:40):

There is the undocumented Meta._parse_string(str, filename, 1, :statement)

Fredrik Ekre (Jul 05 2021 at 08:42):

Yea I played around with that, but it doesn't result in any LineNumberNodes.

Fredrik Ekre (Jul 05 2021 at 08:45):

That works for linenumbernodes for macros though:

julia> Meta._parse_string("""
       @info "hello"
       """, "REPL", 1, :statement)
(:(#= REPL:1 =# @info "hello"), 15)

Fredrik Ekre (Jul 05 2021 at 08:46):

But how about e.g.

julia> e = Meta._parse_string("""
       error()
       """, "REPL", 1, :statement)[1]
:(error())

julia> eval(e)
ERROR:
Stacktrace:
 [1] error()
   @ Base ./error.jl:42
 [2] top-level scope
   @ none:1

Is there a way to replace the none there for example?

Simeon Schaub (Jul 05 2021 at 08:46):

I think you will need an explicit begin ... end to get the LineNumberNodes in between the statements.

Simeon Schaub (Jul 05 2021 at 08:48):

Otherwise you won't get a block expression, so there is nowhere for those nodes to go.

Fredrik Ekre (Jul 05 2021 at 08:49):

Yea makes sense. I will have to parse twice then I think, first to get the single statement, then wrap that into begin ... end and parse again, and then decrement linenumbers with 1 or something.

Sebastian Pfitzner (Jul 05 2021 at 08:57):

Can't you unconditionally wrap your string into "begin;$str;end"? begin/end blocks never change your code's semantics, right?

Fredrik Ekre (Jul 05 2021 at 08:58):

But I want to parse one expression at a time, and begin end would make it all a single expression.

Fredrik Ekre (Jul 05 2021 at 08:59):

This is the result I want:

str = """
a = 1 + 2
b = a + 3
error()
@info "hello"
"""

exprs = []
start = 1

while true
    ex, next_start = Meta.parse(str, start)
    ex === nothing && break

    stop = prevind(str, next_start)

    ex, _ = Meta._parse_string("begin; " * str[start:stop] * "; end", "REPL", 1, :statement)
    push!(exprs, ex)

    start = next_start
end

exprs

giving

julia> exprs
4-element Vector{Any}:
 quote
    #= REPL:1 =#
    a = 1 + 2
end
 quote
    #= REPL:1 =#
    b = a + 3
end
 quote
    #= REPL:1 =#
    error()
end
 quote
    #= REPL:1 =#
    #= REPL:1 =# @info "hello"
end

Now I just need to adjust the line number.

Sebastian Pfitzner (Jul 05 2021 at 09:02):

Right, but that looks equivalent to

julia> Meta.parse(string("begin;", str, ";end")).args
8-element Vector{Any}:
 :(#= none:1 =#)
 :(a = 1 + 2)
 :(#= none:2 =#)
 :(b = a + 3)
 :(#= none:3 =#)
 :(error())
 :(#= none:4 =#)
 :(#= none:4 =# @info "hello")

Fredrik Ekre (Jul 05 2021 at 09:02):

Yea was just gonna post :)

Fredrik Ekre (Jul 05 2021 at 09:02):

Great, thanks.

Fredrik Ekre (Jul 05 2021 at 09:06):

So then I will just use Meta.parseall (which lets me specify the filename) and reach into the :toplevel expr, and then the :block expr

julia> Meta.parseall("begin;" * str * ";end"; filename="test.jl").args[2].args
8-element Vector{Any}:
 :(#= test.jl:1 =#)
 :(a = 1 + 2)
 :(#= test.jl:2 =#)
 :(b = a + 3)
 :(#= test.jl:3 =#)
 :(error())
 :(#= test.jl:4 =#)
 :(#= test.jl:4 =# @info "hello")

Sebastian Pfitzner (Jul 05 2021 at 09:14):

You don't need to wrap the string then, no?

Fredrik Ekre (Jul 05 2021 at 09:16):

Yea of course. Looks like I had my answer in my first post :D

Sebastian Pfitzner (Jul 05 2021 at 09:16):

Mason Protter (Jul 05 2021 at 15:34):

Sebastian Pfitzner said:

Can't you unconditionally wrap your string into "begin;$str;end"? begin/end blocks never change your code's semantics, right?

This can actually change the semantics ocassionally. There are differences between Expr(:block, ...) and Expr(:toplevel, ...).

Fredrik Ekre (Jul 06 2021 at 10:42):

Ah, forgot about the requirement to also know the string index where each expression starts/ends, so I think I can use https://julialang.zulipchat.com/#narrow/stream/274208-helpdesk-.28published.29/topic/Meta.2Eparse.20with.20LineNumberNodes/near/244910605 with some adjustments.

Sebastian Pfitzner (Jul 06 2021 at 12:41):

Theoretically you could use CSTParser for that :stuck_out_tongue:

Fredrik Ekre (Jul 08 2021 at 00:21):

To close out this thread, here is the result: https://github.com/JuliaDocs/Documenter.jl/pull/1634.

Last updated: Oct 02 2023 at 04:34 UTC