Multilab Plugin to run parallel integration Tests

[sdargoeuves]

hi @ipspace,
When working on ASAv I got tired of being able to only run one integration_test at a time and wanted to run them in parallel. I was trying to remember your Autocon5 presentation (I had to look it up to confirm) that you are indeed running those integration tests in parallel.
What I didn't realise, is that you are using the branch integration_tests, as it says here: https://netlab.tools/dev/integration-tests/

Before this realisation, I vibe coded an heavy, clunky piece of shcode, which works! My idea is to use the multilab plugin, so I can run 10 or 20 labs at the same time, all using a different lab ID, to go through the tests. I haven't tested how many I can try to run at the same time.

It took 15min to perform all tests for asav (but only ~28, iirc, are supported), and 20min for frr. For my ASAv use case, this is brilliant, because I can now run multiple tests, come back 5 or 10min later and see where I stand.

Now, knowing about this integration_tests branch, I feel like I've reinvented the wheel (🥳). I'm wondering if I should carry on using it because it works on my machine and maybe share it to any interested soul, or if I should investigate the magic behind the integration_tests branch and use this instead.

[ipspace]

The integration_tests branch is just a wrapper around device-module-test that organizes the log files and test results in subdirectories to help me generate test reports (https://tests.netlab.tools/ and https://release.netlab.tools/)

I tried running tests in parallel for different devices, but Vagrant kept crashing and messing up the system. Things should be much better with containerlab and locking we already implemented, so I should probably start running multi-device tests in parallel with different multilab IDs. However, what would be really interesting (and help you get your tests finished faster) would be a multi-threaded implementation of device-module-test.

[sdargoeuves]

Although I didn't look at the device-module-test that's pretty much what Claude created for me, so I can run as many labs as reasonable, with a command like this:

./parallel-test --settings settings/asav.yml -j 10 --all
# or you can be specific:
# ./parallel-test --settings settings/asav.yml -j 10 ../netlab/tests/integration/ospf/ospfv2 ../netlab/tests/integration/bgp 

It would create 10 folders in /tmp/, all using a different multilab ID, it will do the up, initial, validate and down for each one, and move on to the next.

I also had to take into account the fact that some of the defaults won't work for me, for example, I don't have the eos version used as the default, so I asked to be able to use a settings/local.yml to overwrite those values. It's also useful to pass config.inline when testing ASAv, otherwise, it denies everything!

If you are curious, I've pushed the code here: https://github.com/sdargoeuves/netlab-integration-tests, disclaimer: it has been "proper" vibe coded, i.e. I haven't looked at the code at all 🙈

[ipspace]

Ah, so you vibe-coded the whole test harness ;) And yes, looking at the code makes one's head hurt 🤷‍♂️

[sdargoeuves]

Indeed, I didn't go for pretty & readable, I just wanted to get those tests started, and couldn't wait ~6min per single test!