Stream: helpdesk (published)

Topic: Identifying what is causing increase in memory.

Davi Sales Barreira (May 04 2022 at 16:40):

Friends, I've encapsulated a series of data transformations in functions and created a pipe to perform the transformations. By the end of the pipe, I get the final dataframe. Now, I'm running this code in a notebook (for testing and tweaking). For some reason, everytime I run the pipe, the memory used by the notebook is increasing. I don't understand why is that so. From what I thought, since everything was in functions, it should be taking space in the memory once the code ran. Here is the code:

function importmovements()
    movements = CSV.read(datadir("exp_pro") * "/csvs/movements.csv", DataFrame, header = false, ntasks = ntasks)
    colsmovement = ["number", "date", "title", "text"]
    rename!(movements, colsmovement)
    return movements
end
function adicionarid!(movements)
    movements[!,:id] = 1:size(movements)[1]
    return movements
end

function adicionaretapasiniciofim!(movements)
    movinit = combine(groupby(movements, :number),
        [:date, :title, :text, :id] .=> last,
        renamecols = false)
    movinit[!, :etapa] .= "inicio"
    movinit[!, :subetapa] .= "inicio"


    combine(groupby(movinit, :title), nrow => :count)
    movements = antijoin(movements, movinit, on = [:number, :date, :title])
    movements = vcat(movements, movinit)
    filter!(row -> !ismissing(row[:etapa]), movements)
    sort!(movements, [:id, :number, :date], rev = true)
    return movements
end


function runpipemovement()
    @pipe importmovements() |> unique!(_) |> adicionarid!(_) |> formatardata!(_) |> adicionaretapasiniciofim!(_) ;
end

runpipemovement()

Giovanni (May 04 2022 at 16:42):

Are you using Pluto notebooks?

Davi Sales Barreira (May 04 2022 at 16:42):

Jupyter

Giovanni (May 04 2022 at 16:49):

Then I have no clue :)

Mason Protter (May 04 2022 at 16:58):

It’s probably because you’re printing the data frame each time

Mason Protter (May 04 2022 at 16:58):

jupyter stores the output of each evaluation in a dictionary called Out

Mason Protter (May 04 2022 at 16:59):

you can do empty!(Out) to free up that memory

Davi Sales Barreira (May 04 2022 at 17:09):

@Mason Protter , I just run empty!(Out) in any cell? I've tried here, but the memory is still high.

Mason Protter (May 04 2022 at 17:09):

You’ll probably need to wait for the GC to run

Davi Sales Barreira (May 04 2022 at 17:10):

I've checked the notebook file. An it's not storing the dataframe.

Giovanni (May 04 2022 at 17:18):

just an idea but did you check without using @pipe?

Davi Sales Barreira (May 04 2022 at 17:19):

No. I'll try.

Giovanni (May 04 2022 at 17:25):

The very last thing I would try is copying movements at end of adicionaretasinitiofim and see if that solves the problem the only other thing you seem to create is movinit, in case it is carried out of the function somehow but essentially brainstorming here :)

Mason Protter (May 04 2022 at 17:32):

Davi Sales Barreira said:

I've checked the notebook file. An it's not storing the dataframe.

It’s not in the notebook file itself

Mason Protter (May 04 2022 at 17:32):

It’s an object in the jupyter kernel session

Davi Sales Barreira (May 04 2022 at 17:49):

Oh, ok. Thanks, I'll try again. BTW, any tools to inspect his sort of thing? I've tried ProfileSVG.jl, but didn't quite understood the output.

Last updated: Oct 02 2023 at 04:34 UTC