Skip to content

Multiple hidden layers for generic feature embedding. - #1748

Open
williamdavie wants to merge 3 commits into
ACEsuit:developfrom
williamdavie:embedding-depth
Open

williamdavie wants to merge 3 commits into
ACEsuit:developfrom
williamdavie:embedding-depth

Conversation

@williamdavie

Copy link
Copy Markdown
Contributor

It is known that depth as well as width matters for MLP expressivity. In my current area of research the dependence of energy on a generic feature (at ~ fixed geometry) is complex enough whereby the addition of hidden layers for this MLP is relevant.

This pull request adds the option to add hidden layers to the embedding MLP via:

    generic_feature:
        type: continuous
        per: graph
        in_dim: 1
        emb_dim: 32
        num_hidden_layers: 2

An optional feature that defaults to the original code (num_hidden_layers = 1) if not specified.

Opus 5.5 went ahead and updated the finetuning_utils.py to cope with this change, here are its comments:

  1. New helper, _with_embedding_defaults(spec). It returns a copy of the spec with num_hidden_layers set to 1 for continuous features if the key is missing. It works on a copy, so the specs stored on the model are left alone. Categorical features don't use this key, so it isn't added to them.
  2. Defaults filled in once for the foundation. Right after foundation_specs is read, I build foundation_specs_full, the same specs with defaults filled in. The original foundation_specs is still used to work out column offsets, since those depend only on emb_dim.
  3. One comparison instead of two. The loop now unpacks spec_name, spec directly. At the top of each pass it computes matches once: does the foundation's filled-in spec equal the model's filled-in spec? That single value then decides both whether to copy the embedding's weights and whether its head columns come from the foundation. Before, those were two separate exact checks. If a name exists only in the new model, .get returns None, so matches is false, as before.

Tests have been performed, but not yet added .

accepts hidden_layer option for generic embedding small MLP
@ilyes319

ilyes319 commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@williamdavie Thank you for that looks good! Can you further tests of the finetuning with embedding?

This branch is waiting to be deployed

1 waiting deployment
gpu-external — 60af8a1b Waiting Sep 30, 2026 by williamdavie via push #623
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants