Stream: helpdesk (published)

Topic: Package integration dilemma

DrChainsaw (Oct 31 2025 at 10:48):

Not exactly a dilemma, but more of an open-ended question around package extensions (or maybe just package interaction in general) and how to think about APIs then.

Suppose I have a package CarSimulation.jl (actual project is not for cars, but this hopefully makes it easier to understand what I’m after) which comes with a few built-in cars so that users can do things like

crashtest(CarModels.TypicalSemiTruck(), 50kmh)

Where the whole point of CarModels is that users can type it in the REPL and then use tab to autocomplete and list available models.

I now learned that there is a database where I can get accurate parameters for real car models and I’d like to integrate that CarSimulation.jl.

I could of course do something like this

 crashtest(fetch_from_database(model = “Honda Civic”, year=…), 50kmh)

But that would lose the nice autocomplete properties. I would much rather have something like

crashtest(CarModels.HondaCivic(year=…), 50kmh)

This would make the integration seamless and the autocomplete workflow will still work. Question is basically how best to accomplish this.

The first things that some to mind is to generate structs based on entries in the database into a module called CarModels.

One complication is that since database is optional (maybe users don’t even have access) I don’t want to add the database stuff directly to CarSimulations.jl. The nicest would be to have an extension since I will anyway be creating a package for just the interaction with the database since this is needed in other contexts.

However, extensions tend to be quite insulated from the main package (and probably for good reasons) and seem to mostly want to work with types owned by the extended package. I guess trying to eval stuff into the parent package is asking for trouble (if it even works at all)?

Another option could be to make CarModels an constant instance of some struct (which might need some pretty convoluted implementations of getproperty to make the syntax above work) which the extension modifies when initialized. Or is this almost as bad as the above?

Last resort option I can think of is to compromise a bit and have something like DataBaseCarModels.HondaCivic in the database package as some struct which just wraps the database output and then let the extension (in CarSimulation.jl) translate the database output into some generic CarModel which the CarSimulations.jl code can work with.

This is probably perfectly fine from the user perspective, but I’m worried about the maintenance since CarSimulation.jl exports a quite large number of functions which take a car model as input which all would need to be implemented for the wrapped database output.

Having a function to convert a database struct to a CarModel could alleviate this maintenance burden at the cost of user inconvenience. This also create the dilemma of where the conversion function shall live, but I suppose it could be owned by CarSimulations.jl and just default to an error.

A to me much less attractive last resort option I can think of is to have a combination package (DataBaseCarSimulation.jl) which depends on both CarSimulation and the database package. I guess this would allow DataBaseCarModels to have autocompletable car models on the format which CarSimulation.jl expects. Maybe this is actually the idiomatic solution to this problem? I’d still like to avoid it if possible, but I won’t die on that hill.

Sundar R (Oct 31 2025 at 19:41):

Can you show the base setup of this in code - is CarModels a module? What type is TypicalSemiTruck? (Maybe you're rethinking those in the face of this database idea, but understanding the original structure you were thinking of without that would still help understand the context better.)

Btw, dictionaries also get auto-complete in the REPL, so that might be a convenient option here.

DrChainsaw (Oct 31 2025 at 19:58):

Can you show the base setup of this in code - is CarModels a module? What type is TypicalSemiTruck?

Yeah, at the moment CarModels is a baremodule (just so it does not show anything except the things I have decalared in it) and TypicalSemiTruck is a type (which just wraps a few generic parameter containing other structs), but the only reason for this design is to get the auto-complete discovery aspect.

I am indeed rethinking the design as I would assume that eval-ing stuff into CarModels from a package extension is not a workable design.

I didn't know dicts auto-complete. It would make a simpler version of the second option I thought ot. That could very well be the simplest option then.

cschen (Nov 06 2025 at 17:33):

In my view it would be better to have your own CarModels and the DatabaseCarModels as separate objects instead of having DatabaseCarModels optionally hide inside CarModels. This gives you the freedom of not having to maintain a possible abstraction layer between the two if they ever diverge in their api, which sounds as a decent headache.

You could define DatabaseCarModels as a struct and play around with Base.propertynames for the tab completion and Base.getproperty for retrieving the models.

If you want “callable” properties, it might be a bit tricky, it would probably need you to generate functor stubs of all the database models in the extension, which could be done with, admittedly, a probably rather involved macro.
But at that point you can have the translation happen automatically in the functor.

P.S.: apologies for the unformatted code, Zulip on mobile is a disgrace

DrChainsaw (Nov 06 2025 at 18:45):

Thanks,

I think I've come to the same conclusion w.r.t the namespace separation.

It was perhaps a bit obscured in the question, but a big part of what I'm struggling with is a good way to cross the mainpackage - extension border. For example, it seems as if DatabaseCarModels would either need to be owned by the main package and created as a global empty instance or returned through a function call which the main package owns (and the extension implements a method which takes some type owned by the extended package).

For now I have used the latter method so there is an empty generate_carmodels function in the main package which the extension implements. I made the database package just wrap all the downloaded data in structs so that it has own types which can be used for this dispatch.

All in all I'm not too unhappy about this.

I couldn't really let go of how nice it would be (from the end users perspective) to have the database models as types, so for now this function returns an expression which the main package evals to create DatabaseCarModels as a submodule to the main package, although I do have a feeling I will regret this choice soon. The logic to handle the database output is quite generic though so it should not be too much effort to change it so it just returns some collection of non-unique-type instances.

Last updated: Nov 07 2025 at 04:42 UTC