[D] Alternatives to fb Hydra?

Submitted by alyflex t3_125nf35 in MachineLearning

I have been trying to find a nice tech stack I like for designing and running machine learning models, and currently I'm trying out mlflow, hydra, and optuna.

However, hydra seems to have several limitations that are really annoying and are making me reconsider my choice. Most problematic is the inability to group parameters together in a multirun. Hydra only supports trying all combinations of parameters, as described in https://github.com/facebookresearch/hydra/issues/1258, which does not seem to be a priority for hydra. Furthermore, hydras optuna optimizer implementation does not allow for early pruning of bad runs, which while not a deal breaker is definitely a nice to have feature.

What I do like about hydra is their ability to combine config yaml, using defaults. So does anyone have any good alternatives or suggestions for how to fix this or what to switch to?

34

Comments

You must log in or register to comment.

RicketyCricket t1_je50mbi wrote

https://github.com/fidelity/spock

8

_Arsenie_Boca_ t1_je5d04j wrote

Looks interesting, a bit more lightweight than hydra. But also misses a lot of cool features like composing multiple yaml configs

5

RicketyCricket t1_je5j2n9 wrote

Most of the cool stuff is buried in the docs under advanced features :-)

https://fidelity.github.io/spock/advanced_features/Composition

(full transparency I'm the author/maintainer/core-developer. I know the docs need a re-org to surface more of the useful features)

7

RicketyCricket t1_je5jchr wrote

This being my favorite hidden one:

https://fidelity.github.io/spock/advanced_features/Evolve#maintaining-cli-and-python-api-configuration-parity

3

RicketyCricket t1_je5kgy4 wrote

second favorite:

https://fidelity.github.io/spock/advanced_features/Post-Hooks

Basically lets you do any validation necessary on your configs. Spock provides some basics (greater than, within bounds, etc) but it's totally up to the user via any simple asserts or validation functions a user wants to write.

3

_Arsenie_Boca_ t1_je6ayl7 wrote

Thanks, looks like your library isn't far behind hydra in terms of functionality. Will definitely look into it more closely the next time I set up a project.

What would you say are the pros and cons between hydra and spock?

1

RicketyCricket t1_je9lp7z wrote

Mainly that Spock is much lighter weight and really focuses on just configuration management and stateful ness. Hydra has all these crazy bells and whistles (Ray integration etc) that could be useful for certain things but kinda starts meandering from the original purpose of configuration management imo. Hydra is great and if it works for you then use it. We built Spock internally when I was at Fidelity because Hydra didn’t exist… just so happens that FB/Meta was doing the same thing at the same time so both libraries end up covering a very similar usage space

1

_Arsenie_Boca_ t1_je9n0ea wrote

Thanks, I basically use only the config part of hydra and am regularly annoyed that its so clunky, so spock might be a good alternative. Gonna check it out :)

1

alyflex OP t1_je5y5gh wrote

> https://github.com/fidelity/spock

This looks quite promising, and I like the Post hooks you linked below, but I do not see any way of running a series of experiment in a non-combinatoric way? There is Optuna api (though I can't tell whether early pruning is supported in this?), but I don't see any way of grouping parameters for a set of experiments.

2

DigThatData t1_je5kfoc wrote

go closer to the metal and use omegaconf directly.

7

RicketyCricket t1_je5ky7l wrote

As the developer of Spock (posted in another comment) -- OmegaConf is also an awesome choice and super useful. I'd suggest checking it out too!

You can go even closer to metal and use the attrs library as well (https://www.attrs.org/en/stable/)

1

alyflex OP t1_je5yia4 wrote

That is certainly an option that I was considering, but then I would have to make my own job planner / multirunner, (which I actually already have done for my current project, but this whole refactoring was to try and move away from my own custom functions and try to use some more standardized methods)

1

fmindme t1_je5ty5k wrote

I'm also using Omega conf. It's a great lib: full of feature, not opiniated, perfect for Mlops !

1

pseeth t1_je8mold wrote

I have a lightweight package that I use that has all the main things I wanted from hydra or gin-config. It's here and it's pretty tiny in terms of lines of code: https://github.com/pseeth/argbind

1

fnordstar t1_je8rhex wrote

Isn't just using python flexible enough for you?

0