# Haskell CI through Staking

**URL:** <https://discourse.haskell.org/t/haskell-ci-through-staking/1949>\
**Category:** Show and Tell\
**Created:** [February 24, 2021, 12:24pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949 "2021-02-24T12:24:08Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [February 24, 2021, 12:24pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/1 "2021-02-24T12:24:08Z")

</div>

_Full disclosure: I work for a cryptocurrency company_

I’m running an experiment: can I cover the costs for running build machines for GHC’s CI by running a stakepool? I think this could work out if enough people feel compelled to stake their ADA with the [zw3rk stakepool](https://pooltool.io/pool/e2c17915148f698723cb234f3cd89e9325f40b89af9fd6e1f9d1701a), if they have some.

## Backstory

I’ve been running three **x86\_64-linux** builders as well as one **aarch64-darwin** builder, and have been wondering how to make this more sustainable, I’d also like to add some fast windows builders, however that would mean higher monthly expenses.

## What is staking

The Cardano network builds is consensus using a Proof of Stake algorithm. Similarly to running mining nodes for bitcoin one can run a stake pool for cardano. The effort put into running a stake pool is rewarded by the network. Pools rewards depend on the amount of ADA staked with them. Earned rewards are then distributed among those that put their ADA with the stake pool. Currently Staking ADA with stake pools will produce a ROI of up to ~5% p.a. on the staked ADA.

## Idea

The idea is to run a stake pool (separate from the build machines!), with competitive parameters (1% margin). And use the rewards the pool operator obtains to pay for CI builders for GHC.

## Risks involved

For anyone staking with ZW3RK, the risks are minimal. At worst you’ll miss out on better rewards due to the currently small stake. This will increase as more people stake with the pool. At best you’ll obtain the same rewards as with any other 1% pool, but also help pay for CI resources for GHC.

For me the risk is that this doesn’t work out and I’ll loose a bunch of money on build machines, and trying to run a stake pool.

## Plan forward

If the pool ends up making its block, I’ll pledge to add a windows build machine. This is what we need right now the most. Spare build capacity will be put towards smoke testing GHCs by building all of stackage or other package sets against compilers.

## My plea

If you hold any ADA, please consider staking it with ZW3RK and thus help pay and run CI builders for GHC.

_Edit_  
You can see the load of the current builders on [this dashboard](https://mon.zw3rk.com/). The current [pending](https://gitlab.haskell.org/ghc/ghc/-/jobs?scope=pending) and [running](https://gitlab.haskell.org/ghc/ghc/-/jobs?scope=running) GHC CI jobs on [GHC’s gitlab instance](https://gitlab.haskell.org/ghc/ghc).

---

<div class="post-metadata">

**Author:** ![ocramz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ocramz/32/3713_2.png) [@ocramz](https://discourse.haskell.org/u/ocramz)\
**Post date:** [February 24, 2021, 9:00pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/2 "2021-02-24T21:00:42Z")

</div>

Where would the money to pay the electricity bill for the CI servers emerge from?

Disclaimer: I don’t understand cryptocurrencies at all, and in the specific case cannot see how this plan makes any economic sense.

---

<div class="post-metadata">

**Author:** ![jaror](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/jaror/32/3271_2.png) [@jaror](https://discourse.haskell.org/u/jaror)\
**Post date:** [February 24, 2021, 9:50pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/3 "2021-02-24T21:50:41Z")

</div>

My interpretation is that it is analogous to lending him money which he can invest for higher returns than the interest and him spending the difference on CI servers while still paying interest to the lenders. So, everyone benefits, except perhaps that the lenders could have invested the money themselves and earn more. And the more people invest the smaller the relative cost of the CI servers will be, so the more interest he can pay back to the lenders. Is that a good interpretation?

---

<div class="post-metadata">

**Author:** ![ocramz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ocramz/32/3713_2.png) [@ocramz](https://discourse.haskell.org/u/ocramz)\
**Post date:** [February 24, 2021, 10:26pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/4 "2021-02-24T22:26:42Z")

</div>

I understand how investments work in general, but I fail to see how do cryptographic keys translate into real money. But I guess this isn’t the best place for this discussion.

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [February 24, 2021, 11:17pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/5 "2021-02-24T23:17:57Z")

</div>

These questions are perfectly valid, in fact I get them asked often by people who heard about cryptocurrencies and hope I can answer all their questions. I’m not always completely successful 😅

If we assume that we do want a distributed ledger to record events on, we need some form of consensus mechanism. The two prominent approaches are Proof of Work (e.g. Bitcoin, Doge, …) and Proof of Stake (Tezos, Cardano, …). I’m not going to go too much into the technical details, IOHK has written lots of public research papers on Ouroborus. There is even a [dedicated, animated website](https://ouroboros.iohk.io/).

While you need massive energy to run a Bitcoin mining node, you can (currently) run a Cardano stake _pool_ even on a raspberry pi. (This might get a bit tricky soon, as the onboard memory of the raspberry pi might become an issue.)

Similar to how there are rewards for each block produced in PoW there are rewards for each block produced in PoS. The amount of blocks a pool produces are proportional to the delegated stake a pool has (up to a certain limit).

Thus if you hold say, ADA, you can stake your ADA with a stake pool, similarly how you can take your USD, EUR, GBP, JPY, … to a bank, and they may offer you an interest on the deposit, a stake pool will provide you with a proportional share of the rewards generated from the pool. _A crucial difference here is that there is zero risk involved in delegating your stake. Unlike a bank that could go bankrupt and you’d loose your deposits above insurance, you are at zero risk to loose your stake (unless there is a **serious** bug in Cardano) You are just delegating it, not really depositing it with someone else_.

The final idea is then that by running a pool with competitive parameters, it should make next to no difference for someone to stake with zw3rk or any other pool with respect to the rewards produced. However the rewards the pool generates for the pool operator are not used to pay for (part of) my living, but rather to offset the costs for CI builders for GHC.

_Example_: Say the pool generates 10340 ada in rewards, 340 (fixed minimum fee) gets subtracted, leaving 10000 ada, 1% margin means the pool get 100 ada, and 9900 ada are distributed among those who stake with the pool. Thus there would be 440 ada to offset costs the costs for running the stake pool and GHC CI Servers.

Let me know if you want to know more @ocramz!

Edit: added clarification about delegating and lending.

---

<div class="post-metadata">

**Author:** ![ocharles](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/ocharles/32/102_2.png) [@ocharles](https://discourse.haskell.org/u/ocharles)\
**Post date:** [February 25, 2021, 2:13pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/6 "2021-02-25T14:13:20Z")

</div>

Could you tell us a bit more about ZW3RK? Your post seems to say “stake your ADA here”, but doesn’t tell me anything about who is running it, or - as the assumption is that they will use the rewards to invest in GHC CI resources - that I can trust them to do what I expect.

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [February 26, 2021, 1:39am UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/7 "2021-02-26T01:39:24Z")

</div>

Of course I can tell you a bit more about ZW3RK! I _am_ zw3rk, and I run the stake pool, as three x86\_64-linux builders and one m1 builder right now. Now whether or not you trust me is clearly up to you. I hope I’m trustworthy 😉

If you look at the [dashboard](https://mon.zw3rk.com/) I posted above you’ll see all the linux runners currently at work, and if you go through the running x86\_64-linux jobs on the [running](https://gitlab.haskell.org/ghc/ghc/-/jobs?scope=running) gitlab page, you’ll see the x86\_64-linux-{1,2,3}.zw3rk runners picking up jobs.

You might have seen posts in the past [on cross compilation](https://medium.com/@zw3rk), which also used the zw3rk handle; and ultimately lead [haskell.nix](https://github.com/input-output-hk/haskell.nix).

You might also have heard of the [aarch64 native code gen](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3641) I’ve been working on or seen my [work on ghc-8.10.5](https://gitlab.haskell.org/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=all&author_username=angerman&milestone_title=8.10.5). This is all just to say that I’m quite involved in GHC development and **very** interested in fixing the CI story.

You will also find a few emails from me on [ghc-devs](https://mail.haskell.org/pipermail/ghc-devs), where I bring up CI as an issue.

Finally I’ve got a [google sheets pivot table](https://docs.google.com/spreadsheets/d/1DavxohJRJ6S1JiEuvz-un6rCUdQ57z1usMt0QVTm6WA/edit?usp=sharing) where I track current (successful) builds and their timings.

In the end it should make no difference to stake with ZW3RK _or_ say 1PCT, the parameters are the same, and once ZW3RK has accumulated enough stake the rewards for those who stake will also be identical. The only difference is that I’ll put the rewards ZW3RK generates towards towards GHC’s CI and running the pool.

In summary: I’m already running a few GHC builders to improve CI, I’m now running ZW3RK to make this sustainable to me, and hopefully be even able to add more builders. I’m taking on the risk to run the stake pool, and deal with expenses in fiat, and covering them with ada somehow. For those who stake with ZW3RK there should be no difference than staking with any other 1% pool, in rewards paid out to them. The difference is the warm fuzzy feeling of also helping GHC’s CI build times 🙂

---

<div class="post-metadata">

**Author:** ![wiz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/wiz/32/2408_2.png) [@wiz](https://discourse.haskell.org/u/wiz)\
**Post date:** [March 4, 2021, 3:03pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/8 "2021-03-04T15:03:24Z")

</div>

> [@angerman](#):
>
> For me the risk is that this doesn’t work out

What is the “structure” of this risk? What could go wrong and what mitigations are in place already?

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [March 4, 2021, 3:36pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/9 "2021-03-04T15:36:08Z")

</div>

Let’s try to itemise the risks:

- volatility: expenses are in EUR, rewards in ADA. If ADA goes down, I might not be able to cover and will have to pay out of my own pocket. I’m willing to take this risk for a duration for ~1-2 years to see if this works out.

- lack of stake: unless I can attract enough stake with the pool, the pool will not make any blocks, and thus never produce any rewards, but will cost time and money to run.

- reputational risk: if this doesn’t work out, I’ve made a clown of myself. And potentially angered those who put their stake with the pool but didn’t get any rewards as the pool never made any blocks. (As we now know, this is unlikely, as the pool has already made a block).

- having the pool compromised: while you can be careful about your wallet and keys, by operating a pool you effectively have to come out of anonymity to some degree. This also puts you at increased risk of fishing or hacking attacks. While the delegated stake is never at risk, as it is always only delegated and never handed over, the pool operator might get their keys stolen and wallets emptied.

So what mitigations are in place: for volatility I can try to hedge a little with liquidating ada ahead of time at relatively high prices as well as using futures and options. Lack of stake is a bit harder, but mostly means advertising the pool. For reputational risk, I can’t really do more than giving my best 😅 And finally for security I can rely on years of experience running servers, as well as best industry practices (hardware wallets, offline keys, …).

---

<div class="post-metadata">

**Author:** ![chak](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/chak/32/1274_2.png) [@chak](https://discourse.haskell.org/u/chak)\
**Post date:** [March 12, 2021, 12:47pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/10 "2021-03-12T12:47:04Z")

</div>

I think, this is a fantastic idea. Thanks, Moritz, for all your effort.

BTW, I would suggest to mention somewhere at the top in bold letters that the choice of Cardano as the underlying blockchain here ties in with the goals of the pool: Cardano is implemented in Haskell (and hence, of course, built with GHC).

Consequently, even Ada holders who are not per se interested in Haskell are still interested in a high quality Haskell compiler, and hence, GHC CI.

---

<div class="post-metadata">

**Author:** ![wiz](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/wiz/32/2408_2.png) [@wiz](https://discourse.haskell.org/u/wiz)\
**Post date:** [August 25, 2021, 3:19pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/11 "2021-08-25T15:19:58Z")

</div>

So, did the plan work?

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [August 25, 2021, 3:36pm UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/12 "2021-08-25T15:36:39Z")

</div>

Thanks for asking!

So far, yes! The stake pool right now provides:

- 3x Mac mini’s (M1)
- 3x x86\_64-linux
- 2x x86\_64-windows  
builders as well as the management of those.

We’ve been able to significantly reduce build pipeline times. I still think we can do a bit better. I hope to find the time to do a proper write up soon.

---

<div class="post-metadata">

**Author:** ![tmcgilchrist](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/tmcgilchrist/32/1145_2.png) [@tmcgilchrist](https://discourse.haskell.org/u/tmcgilchrist)\
**Post date:** [April 21, 2022, 6:59am UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/13 "2022-04-21T06:59:51Z")

</div>

How is the plan working out in 2022? @angerman

FYI The dashboard link from the original post shows that the grafana deploy isn’t working anymore.

---

<div class="post-metadata">

**Author:** ![angerman](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.haskell.org/angerman/32/892_2.png) [@angerman](https://discourse.haskell.org/u/angerman)\
**Post date:** [April 21, 2022, 7:39am UTC](https://discourse.haskell.org/t/haskell-ci-through-staking/1949/14 "2022-04-21T07:39:10Z")

</div>

Thanks for the heads up. The monitor just ran out of disk space. Should be back up.