Store test/benchmark suites from stack (or cabal) for use in a separate build job

Learn
#1

We want our build environment to:

  1. Build libraries, executables, test and benchmarks in a single job
  2. Run test suites in a separate job
  3. Run benchmark suites in a separate job

Is it possible with either stack (preference as we use it already) or cabal to easily pull the built tests/benchmarks out of the artifacts?

Currently with stack we have to download both of:

  1. Dependency cache (~/.stack) [global dependency cache]
  2. Local build artifacts (repo/.stack-work) [generated from the build job]

in order to run the tests/benchmarks without rebuilding anything. If (1) isn’t present stack --test ... breaks and if (2) isn’t present we have to rebuild everything again. I’d like to simply make the tests and benchmarks available to the next job rather than everything from the previous build.

Is there a better way? I don’t have a problem switching to cabal if there’s a better story there but I do hope there’s a way to separate out building and running tests/benchmarks with minimal caching overheads.

#2

From the context of your question I guess you are talking about some CI/CD pipeline isn’t it?

In principle tests and benchmarks are just regular executables. I guess you can copy those already-compiled binaries, upload them into your CI/CD artifact folder and pass to the next job. In order to run them you can’t simply stack test but rather your-test-binary-name.

How to do it? I don’t know. stack offers a stack install --copy-bins to move executables out of .stack-work folder into ~/.local/bin, I don’t know if there is something similar for test and benchmarks. worst case, I guess you can dive into .stack-work and simply move the binaries.

1 Like
#3

cabal has a --list-bins command that gives the path to various components including tests and benchmarks.

1 Like
#4

Thanks - that’s a great help.

cabal only appears to have cabal list-bin at the moment which targets a single binary but there is an active discussion about getting it to work better with cabal list-bin all or cabal list-bin:tests etc.

However, cabal-plan does have a list-bins option.

I also discovered via the cabal issue tracker on list-bin a useful stack variant

stack ide targets [--benchmarks] [-tests] [--exes]

1 Like
#5

In the case of Stack, for test-suite foo-test, the built ‘test’ executable foo-test.exe (on Windows; foo-test on Unix-like operating systems) will be in the directory <dist-dir>\build\foo-test, where <dist-dir> is the relative directory reported by stack path --dist-dir (relative to the project root directory, reported by stack path --project-root).

2 Likes
#6

Thanks - is there anyway to get the information on a per project basis?

e.g. if stack ide targets returns

package1:test:integration
package1:test:regression
package1:exe:run-package-1
package2:test:unit-tests

I still need to know where to look for package1. stack must know that but I’m not sure I can get it to tell me? And then it must also know where the given test, bench and exes are as well to run them?

I could rename everything to make it simpler to find/grep them out but would be nice to have something that feels a bit more solid if possible.

#7

Stack can report the absolute file paths to the root directory of each local/mutable package in a project. See the documentation for the stack query command: query command - The Haskell Tool Stack.

2 Likes
#8

Excellent. I think that gives me all the pieces of the puzzle. :partying_face: