Separation of Building and Testing in CI - any known best practices?

Learn
10

I gave a try to cache-based approach inspired by @hellwolf here.

TL;DR: it is surely possible, but it is far more complicated compared to the artifact-based approach (at least with GitHub cache alone).

The basic strategy is to include commit hash in the final cache key and requires exact matching in Testing Phase.
However, there are several points to be carefully considered to get everything as expected:

  • If you include hash of files into the cache keys to share the common cache across the builds (e.g. to share the cache of ~/.cache/store based on the contents of cabal.project.freeze and package.yamls), , we must consistently use the one computed at appropriate point.
    • For example, if you use hashFiles('**/*.hs') in cache key, this hash must be computed right after checking-out because some new .hs files can be generated under dist-newstyle and/or some source-repository-package can introduce extra hs-files.
    • We also must be pay attention to use exactly the same key in Testing and Building Phases - I used workflow output mechanism to ensure Testing Phase to use the same key.
  • We must call cabal with the same configuration both in Testing and Building phase - otherwise, cabal test will recompile some parts needlessly.

That being said, as we must keep the consistency between Building and Testing Phases, it is far more lightweight to use artifacts mechanism than caches. This story is of course about GitHub Caching mechanism, though. Perhaps Nix can offer more clever caching or haskell-ci could do this more well. So please let me know if anyone is doing this kinda thing with caching mechanism.

1 Like