Stream: helpdesk (published)

Topic: Reading content of URL as a table


view this post on Zulip Júlio Hoffimann (Jul 31 2021 at 12:19):

I don't have much experience with webdev and appreciate if someone can give a hand. Suppose I want to read the contents of this URL into a JSON or Tables.jl table:

https://apitempo.inmet.gov.br/estacoes/T

I can get the page with HTTP.jl:

using HTTP

r = HTTP.get("https://apitempo.inmet.gov.br/estacoes/T")

r.body
233853-element Vector{UInt8}:
 0x5b
 0x7b
 0x22
 0x43
    
 0x30
 0x22
 0x7d
 0x5d

How to reinterpret these bits as a JSON or DataFrame directly?

view this post on Zulip Júlio Hoffimann (Jul 31 2021 at 12:20):

Would you recommend using Gumbo.jl for this task or HTTP.jl suffices?

view this post on Zulip Júlio Hoffimann (Jul 31 2021 at 12:32):

I can convert the body to a string with String(r.body), but how to convert the string into a JSON or table?

view this post on Zulip Júlio Hoffimann (Jul 31 2021 at 12:34):

Oh nevermind, JSON.parse on the resulting string just works :tada:

view this post on Zulip Eric Hanson (Aug 08 2021 at 09:42):

If you use JSON3, you can also do JSON3.read(bytes) to avoid constructing the string first. JSON3 also does clever things to reuse the memory of the bytes themselves and expose nice Julia objects on top of that memory. (But this probably only matters for massive files or if you are processing many of them).

view this post on Zulip Mosè Giordano (Aug 08 2021 at 21:45):

for what is worth, you can download files in-memory also with Downloads.jl (which is now a stdlib, so don't even need to use third-party packages):

julia> using Downloads

julia> take!(Downloads.download("https://apitempo.inmet.gov.br/estacoes/T", IOBuffer()))
233857-element Vector{UInt8}:
 0x5b
 0x7b
 0x22
 0x43
 0x44
 0x5f
 0x4f
 0x53
 0x43
 0x41
 0x52
 0x22
 0x3a
 0x22
 0x30
    
 0x30
 0x30
 0x2e
 0x30
 0x30
 0x30
 0x2d
 0x30
 0x33
 0x3a
 0x30
 0x30
 0x22
 0x7d
 0x5d

Last updated: Nov 22 2024 at 04:41 UTC