Supporting GHC Continuous Integration Infrastructure

[bgamari]

Hello Haskellers,

For the past several years GHC’s continuous integration infrastructure has been supported in-kind by the generosity of a handful of individuals, and a few major corporate sponsors (thanks to Packet, Rackspace, ARM, SimSpace, and Google X). Thanks to these contributors, GHC has had access to CI capacity which is mostly adequate for our typical needs.

However, recently X has declined to continue their support, leaving us with a hole in this capacity. Windows support poses a particular challenge as the loss of all five of our Windows runners leaves us with the unfortunate choice of either substantially reducing CI coverage or asking our sponsors to shoulder considerable licensing fees on our behalf.

Given that GHC’s CI infrastructure is now quite central to the project’s releases, contributor experience, and continued correctness, support of this infrastructure seems quite in line with the Foundation’s mandate. Furthermore, compared to the human resources that HF will support, the cost of replacing the capacity lost from our previous X sponsorship is quite small, likely only a few hundred USD per month. Assuming we think that this is within the scope of the Foundation I would be happy to put together some concrete proposals describing possible paths forward.

Thoughts?

Cheers,

• Ben

[bgamari]

To make this proposal a bit more concrete, @davean and I worked through a couple of possible options for replacing the existing Google Compute Engine (GCE) configuration.

Executive summary

I would propose the following:

• 1x Hetzner ax101 instance (120 EUR/month)
• 1x Windows Standard 16-core license (around 800 USD one-time-cost)
• 1x Windows 2-core license pack (120 USD one-time cost)

This would allow us to virtualise four Windows runners on faster hardware than our GCE resources currently provide for a small fraction of the cost.

Options evaluated

For reference, I describe the options I evaluated below.

Funding the existing GCE setup

The scenario below is presented only for reference; the scenario that I would propose is described in the following section

The current Windows capacity consists of five Google Compute Engine n1-standard-4 instances. These are four-core (Skylake-class) hardware with sufficient memory and disk to accommodate one GHC build. The hardware and license costs of each of these instances sums to roughly 250 USD/month. This totals to around 1250 USD/month.

However, this is considerably higher than strictly necessary as GCE only offers Windows Server Datacenter edition licensing, which is nearly an order of magnitude more than Standard edition. Consequently, over half of total cost is being eaten by licensing.

Moving to Hetzner

In my experience Hezner has considerably lower prices than traditional cloud providers. Moreover, it would give us access to Windows Server Standard edition, which reduces licensing costs substantially. For instance, we might opt for a machine type like the ax101 (16 cores/32 threads, 128 GB RAM, 3.8 TB disk) which at 120 EUR/month is roughly one-fifth the price of one of the previous GCE instances. This hardware could easily run four or five concurrent builds.

Licensing Windows Server Standard through Hetzner would add roughly 55 EUR/month. Alternatively, we can provide our own license, which would come at an up-front cost of roughly 800 USD. The break-even point for providing our own license is roughly 14 month; as I expect this configuration will be used for at least two years it may be worthwhile going this rough.

[rae]

Thanks for writing this up, especially with the details including costs. I’ve put this proposal on the agenda for tomorrow (Thursday)'s HF Board meeting, but I don’t expect an answer then – instead, we’ll use this to figure out how to respond to such requests, and I hope to get an answer to you in a few weeks.

Bear with us – we’re new at this!

[bgamari]

Indeed I expected this sort of decision to take some time and thankfully we have a bit of time before we lose our GCE resources. Thanks for the consideration!