Hackage failure?

#1

Starting to see these failures in CI:

Selected mirror https://hackage.haskell.org/
Downloading root
Could not deserialize <repo>/root.json: Unknown key: b0107b7b3ccbda342459fc9ef98e6ea9bec672a5046bd310ea70915a91d846a7

Is this an upstream issue?

1 Like
#2

I can confirm, apparently something during the recent key rotation. Ping @david-christiansen.

cabal-install is currently rejecting hackage.haskell.org and falling back to hackage.fpcomplete.com:

āÆ cabal update -v
...
Downloading root
Running: /usr/bin/curl 'http://hackage.haskell.org/root.json' --output /tmp/transportAdapterGet27870-7 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.10.1.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers27870-8.txt
Verification error: Could not deserialize <repo>/root.json: Unknown key:
b0107b7b3ccbda342459fc9ef98e6ea9bec672a5046bd310ea70915a91d846a7
Downloading root
Running: /usr/bin/curl 'http://hackage.haskell.org/root.json' --output /tmp/transportAdapterGet27870-10 --location --write-out '%{http_code}' --user-agent 'cabal-install/3.10.1.0 (linux; x86_64)' --silent --show-error --dump-header /tmp/curl-headers27870-11.txt
Exception Could not deserialize <repo>/root.json: Unknown key:
b0107b7b3ccbda342459fc9ef98e6ea9bec672a5046bd310ea70915a91d846a7 when using
mirror http://hackage.haskell.org/
Selected mirror http://hackage.fpcomplete.com/
...
Package list of hackage.haskell.org is up to date.
...

From what I can tell the key id

b0107b7b3ccbda342459fc9ef98e6ea9bec672a5046bd310ea70915a91d846a7

is missing its public key.

āÆ curl --silent http://hackage.haskell.org/root.json | jq -r '.signatures[].keyid as $keyid | "key id: \($keyid) -> public key \(.signed.keys[$keyid].keyval.public)"'
key id: 1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42 -> public key bYoUXXQ9TtX10UriaMiQtTccuXPGnmldP68djzZ7cLo=
key id: 0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d -> public key zazm5w480r+zPO6Z0+8fjGuxZtb9pAuoVmQ+VkuCvgU=
key id: d26e46f3b631aae1433b89379a6c68bd417eb5d1c408f0643dcc07757fece522 -> public key 5iUgwqZCWrCJktqMx0bBMIuoIyT4A1RYGozzchRN9rA=
key id: b0107b7b3ccbda342459fc9ef98e6ea9bec672a5046bd310ea70915a91d846a7 -> public key null
key id: fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0 -> public key uRPdSiL3/MNsk50z6NB55ABo0OrrNDXigtCul4vtzmw=
key id: be75553f3c7ba1dbe298da81f1d1b05c9d39dd8ed2616c9bddf1525ca8c03e48 -> public key ydN1nGGQ79K1Q0nN+ul+Ln8MxikTB95w0YdGd3v3kmg=

while for fpcompleteā€™s mirror

āÆ curl --silent http://hackage.fpcomplete.com/root.json | jq -r '.signatures[].keyid as $keyid | "key id: \($keyid) -> public key \(.signed.keys[$keyid].keyval.public)"'
key id: 1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42 -> public key bYoUXXQ9TtX10UriaMiQtTccuXPGnmldP68djzZ7cLo=
key id: fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0 -> public key uRPdSiL3/MNsk50z6NB55ABo0OrrNDXigtCul4vtzmw=
key id: 0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d -> public key zazm5w480r+zPO6Z0+8fjGuxZtb9pAuoVmQ+VkuCvgU=
1 Like
#3

I cannot find that key-id anywhere, I have no idea where it is from.

āÆ sha256sum root-keys/*.public
fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0  root-keys/adam-gundry.public
1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42  root-keys/gershom-bazerman.public
d26e46f3b631aae1433b89379a6c68bd417eb5d1c408f0643dcc07757fece522  root-keys/joachim-breitner.public
0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d  root-keys/john-wiegley.public
a45dd02dbde9e83651c632c06d887da8e76fd73a8c593e5d4aea6ca3150b5a0a  root-keys/lennart.public
be75553f3c7ba1dbe298da81f1d1b05c9d39dd8ed2616c9bddf1525ca8c03e48  root-keys/mathieu-boespflug.public
89d2f075421b46de184d49dfd159dabeeccec9870a081afde8f5596e12a10bd7  root-keys/norman-ramsey.public
ea706b80239a486972d14126f85b04f977f72c4a25059543391a83fa34e82d54  root-keys/tom-schrijvers.public

One this is: this root.json is entirely rejected by the hackage-security framework.
Could we add an automated CI to make sure this does not repeat?

#4

Iā€™m looking into this and trying to figure out what happened - Iā€™m getting in contact with Hackage admins and will post something ASAP.

3 Likes
#5

I think that the problem has been diagnosed - a signature from the wrong key made it through multiple layers of review, and the client is very conservative and rejects unknown keys.

Weā€™re well above the threshold for valid signatures, so deleting it should fix the problem :crossed_fingers: . Iā€™ll post here as things develop.

3 Likes
#6

I note that according to https://status.haskell.org/, Hackage is ā€˜operationalā€™. Surely this canā€™t be correct?

#7

The Hackage admins have fixed the issue outstandingly quickly - I just confirmed from my end.

Thank you all for a constructive report!

8 Likes