Stack script fails to find pandoc

Learn
#1

This silly script:

#!/usr/bin/env stack
{- stack script --resolver lts-21.25 --package pandoc -}

import Prelude
import Text.Pandoc

main :: IO ()
main = putStrLn "Pandoc detected"

is failing to execute with error:

❯ ./example.hs

/home/cebrian/borralodedentro/example.hs:5:1: error:
    Could not load module ‘Text.Pandoc’
    It is a member of the hidden package ‘pandoc-3.0.1’.
    You can run ‘:set -package pandoc’ to expose it.
    (Note: this unloads all the modules in the current scope.)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
5 | import Text.Pandoc
  | ^^^^^^^^^^^^^^^^^^

and I don’t know why because this is a trimmed down version of another script importing a bunch of libraries with same procedure, only pandoc is failing.

Any ideas?

#2

No idea, haven’t used stack for scripts yet.
Does it do the same if you run stack your-script.hs instead of ./your-script.hs?

#3

Same result here, and you’re right this failure is silly. If I change the script command to runghc it works.

#4

Reproducible. Verbosity uncovers a different error message, “cannot satisfy -package pandoc”.

$ stack --verbose example.hs
...
 [debug] Run process: /.../.ghcup/ghc/9.4.8/bin/runghc-9.4.8 -i -i/.../ -hide-all-packages -fdiagnostics-color=always -packagebase -packagepandoc /.../example.hs

/.../example.hs:5:1: error:
    Could not load module ‘Text.Pandoc’
    It is a member of the hidden package ‘pandoc-3.0.1’.
    You can run ‘:set -package pandoc’ to expose it.
    (Note: this unloads all the modules in the current scope.)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
5 | import Text.Pandoc
  | ^^^^^^^^^^^^^^^^^^

Running that command in isolation shows that something is missing (another dependency?):

$ runghc-9.4.8 -v -hide-all-packages -packagebase -packagepandoc example.hs
Glasgow Haskell Compiler, Version 9.4.8, stage 2 booted by GHC version 9.2.2
<command line>: cannot satisfy -package pandoc
    (use -v for more information)
#5

I’ve filed a bug report Stack script fails to use Pandoc library · Issue #6616 · commercialhaskell/stack · GitHub since this doesn’t seem like normal behaviour.

1 Like
#6

Does your script do the same if you change s/pandoc/aeson/ and s/Text.Pandoc/Data.Aeson/?

#7

I could not reproduce this on Windows. With example.hs:

{- stack script --resolver lts-21.25 --package pandoc -}

import Prelude
import Text.Pandoc

main :: IO ()
main = putStrLn "Pandoc detected"

stack example.hs behaves as expected.

(stack --version - Version 2.15.7, Git revision f590e0165b5ab92f5f7f87f8aa55852e1972ee25 x86_64 hpack-0.36.0)

However, I can reproduce it on Ubuntu (via WSL2). In both cases, the final command run by Stack ‘under the hood’ is identical:

Windows:
Run process: D:\sr\programs\x86_64-windows\ghc-9.4.8\bin\runghc-9.4.8.exe -i -i...\ -hide-all-packages -fdiagnostics-color=always -packagebase -packagepandoc ...\example.hs

Linux:
Run process: /home/mpilgrem/.stack/programs/x86_64-linux/ghc-tinfo6-9.4.8/bin/runghc-9.4.8 -i -i.../ -hide-all-packages -fdiagnostics-color=always -packagebase -packagepandoc .../example.hs

So, I am wondering if this is a bug in runghc-9.4.8 on Linux.

#8

If I use GHC 9.4.7 instead (lts-21.21 - which specifies the same version of pandoc), I can’t reproduce the problem on Ubuntu (via WSL2):

{- stack --verbose script --resolver lts-21.21 --package pandoc -}

import Prelude
import Text.Pandoc

main :: IO ()
main = putStrLn "Pandoc detected"

EDT: lts-21.25 has (extract):

- hackage: pandoc-3.0.1@sha256:b86cff1afae695247ee180ac66769881a4900ea4643c87820a9f653dd06d4cf2,37767
  pantry-tree:
    sha256: 4ceae05f232ec6e0a062b36d89da99ad071ef8f498b141ac748c219136b789d3
    size: 141318

lts-21.21 has (extract), which is identical:

- hackage: pandoc-3.0.1@sha256:b86cff1afae695247ee180ac66769881a4900ea4643c87820a9f653dd06d4cf2,37767
  pantry-tree:
    sha256: 4ceae05f232ec6e0a062b36d89da99ad071ef8f498b141ac748c219136b789d3
    size: 141318

EDIT: If I explore with --ghc-options -v, the one that works has:

Loading unit pandoc-3.0.1 ... linking ... done.

and the one that does not work does not. (The one that works also loads lots of other units that the one that does not work does not.)

Both start with (where x is the GHC minor version):

*** initializing unit database:
package flags [-package base{package base True ([])},
               -package pandoc{package pandoc True ([])}]
loading package database /home/mpilgrem/.stack/programs/x86_64-linux/ghc-tinfo6-9.4.x/lib/ghc-9.4.x/lib/package.conf.d
loading package database /home/mpilgrem/.stack/snapshots/x86_64-linux-tinfo6/<hash>/9.4.x/pkgdb
loading package database /home/mpilgrem/.stack/.stack-work/install/x86_64-linux-tinfo6/<hash>/9.4.x/pkgdb

and stack --resolver lts-21.nn exec -- ghc-pkg list pandoc (where nn is the relevant index) shows that that package database does expose pandoc-3.0.1.

EDIT2: It is something about the Linux/GHC 9.4.8/pandoc combination. If I pick, say, ansi-terminal, there is no problem with Linux/GHC 9.4.8. So, this fails on Linux:

{- stack script
   --verbose
   --snapshot lts-21.25
-}

import Text.Pandoc

main :: IO ()
main = putStrLn "Pandoc detected"

but this works on Linux:

{- stack script
   --verbose
   --snapshot lts-21.25
-}

import System.Console.ANSI

main :: IO ()
main = putStrLn "ansi-terminal detected"

as does this (stack is quite a large package, in terms of modules and dependencies):

{- stack script
   --verbose
   --snapshot lts-21.25
-}

import Paths_stack -- From stack package

main :: IO ()
main = print version

EDIT: There does not seem to be a problem with ‘normal’ builds of pandoc; it seems to be something with runghc which (I believe) uses the interactive mode of GHC ‘under the hood’. My tests of ‘normal’ builds have used:

package.yaml:

name: example
version: 1.0.0

dependencies:
- base
- pandoc

executables:
  example:
    main: example.hs

example.hs:

module Main ( main ) where

import Text.Pandoc

main :: IO ()
main = print pandocVersion

stack.yaml

snapshot: lts-21.25 # GHC 9.4.8

and varying the snapshot at the command line.

1 Like
#9

So, this is the nub of the paradox on Linux with runghc from GHC 9.4.8:

$ stack --snapshot lts-21.25 exec -- /home/mpilgrem/.stack/programs/x86_64-linux/ghc-tinfo6-9.4.8/bin/ghc-pkg list pandoc --simple-output

pandoc-3.0.1

$ stack --snapshot lts-21.25 exec -- /home/mpilgrem/.stack/programs/x86_64-linux/ghc-tinfo6-9.4.8/bin/runghc-9.4.8 -hide-all-packages -package=base -package=pandoc /home/mpilgrem/code/has
kell/script-test/example1.hs

/home/mpilgrem/code/haskell/script-test/example1.hs:8:1: error:
    Could not load module ‘Text.Pandoc’
    It is a member of the hidden package ‘pandoc-3.0.1’.
    You can run ‘:set -package pandoc’ to expose it.
    (Note: this unloads all the modules in the current scope.)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
8 | import Text.Pandoc
  | ^^^^^^^^^^^^^^^^^^

I have raised: #25035: `-package=pandoc` ignored · Issues · Glasgow Haskell Compiler / GHC · GitLab

EDIT2: The reason I have raised this as a GHC issue is becuase I understand (a) -package=pandoc passed to runghc means ‘pass this same option on to GHC’ and (b) -package=pandoc passed to GHC means ‘cause the identified package to be exposed’. So, it should not be open to runghc to warn that pandoc-3.0.1 is hidden if it has been passed -package=pandoc.

EDIT: I am continuing to experiment with this, and getting some really odd outcomes. I am going to explore if older versions of Stack behave the same way.

EDIT3: I can reduce the nub further, having looked at the source code for runghc. On Ubuntu (via WSL2), with example.hs:

import Text.Pandoc

main :: IO ()
main = print pandocVersion

$ # GHC 9.4.8 ... does not work ...
$ stack --snapshot lts-21.25 exec -- ghc -hide-all-packages -package=base -package=pandoc example.hs

example.hs:1:1: error:
    Could not load module ‘Text.Pandoc’
    It is a member of the hidden package ‘pandoc-3.0.1’.
    You can run ‘:set -package pandoc’ to expose it.
    (Note: this unloads all the modules in the current scope.)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
  |
1 | import Text.Pandoc
  | ^^^^^^^^^^^^^^^^^^

$ # GHC 9.4.7 ... does work ...
$ stack --snapshot lts-21.21 exec -- ghc -hide-all-packages -package=base -package=pandoc example.hs

[1 of 2] Compiling Main             ( example.hs, example.o )
[2 of 2] Linking example

(To confirm, it is something to do with -package=pandoc not exposing pandoc-3.0.1, because:

$ stack --snapshot lts-21.25 exec -- ghc example.hs
$ stack --snapshot lts-21.21 exec -- ghc example.hs

are both fine.)

EDIT: Resolution

The cause of this has been identified. It is, essentially, a bug in GHC, as follows:

GHC has the concept of installed packages in package databases, and an installed package has a name. The names of installed packages in package databases are provided by command ghc-pkg list.

For the Cabal package pandoc, the name of the installed package corresponding to the main (unnamed) library is pandoc. The name of the installed package corresponding to the internal library xml-light is z-pandoc-z-xml-light. Those names are what the tool ghc-pkg uses and reports.

However, the installed package corresponding to a named sub-library of a Cabal package also has a package-name, which corresponds to the name of the Cabal package. For installed package z-pandoc-z-xml-light it is pandoc.

Undocumented, GHC’s -package option treats package-name (if it exists) as if it were also the name of the installed package. That inconsistency with ghc-pkg means that GHC option -package=pandoc can’t distinguish between (a) the installed package pandoc and (b) the installed package z-pandoc-z-xml-light - so GHC picks one installed package ‘at random’ (that is, what GHC actually does is indeterminate).

What the original poster was experiencing is GHC picking internal package z-pandoc-z-xml-light to expose when passed -package=pandoc, and continuing to treat internal package pandoc as hidden.

6 Likes
#10

It seems that I’ve met the same bug, but with --extra-dep ?

> ./scripts/ReadmeReplace.hs        

/Users/roger/dev/haskell/schedule-maker/scripts/ReadmeReplace.hs:22:1: error:
    Could not find module ‘OptEnvConf’
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
22 | import OptEnvConf
   | ^^^^^^^^^^^^^^^^^


#!/usr/bin/env stack
{- stack script
   --snapshot lts-22.23
   --package regex-tdfa
   --package text
   --extra-dep opt-env-conf-0.0.0.0
 -}

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE NamedFieldPuns #-}

import Data.Text (Text)
import OptEnvConf
import Text.Regex.TDFA (getAllTextMatches, (=~))


main :: IO ()
main = do
  pure ()
#11

I think that may be something different.

As you are specifying --package, stack script will not deduce all the packages needed from the import statements alone. You need package opt-env-conf (as it exposes module OptEnvConf) but you do not specify that with a --package. All you do is add opt-env-conf-0.0.0.0 to the snapshot (the snapshot specifying the set of package versions that needed packages are to be selected from, not specifying what packages are needed).

#12

Thanks! that solves it!

Reading again the docs has clarified the problem. I didn’t realize that if you specify a single --package, either inlcuded or not in the snapshot, stack then requires you to explicitly specify your dependencies. The problem arose for me because I had to specify where did Data.Text came from:

Module Data.Text appears in multiple packages: 
incipit-base incipit-core relude text
1 Like
#13

Interesting discussion. For disambiguating your import, the PackageImports language extension might have been another solution ?

Though, I don’t see how the incipit and relude packages would get involved, in the example you showed…

#14

It is not documented online (to fix), but stack script does indeed recognise GHC’s PackageImports when it parses import statements.

1 Like
#15

I didn’t remember the PackageImports extension, thanks!

Answering your second point, my understading is that in a stack script, stack infers the packages needed from the imported modules, and that’s what I initially did with something like this:

#!/usr/bin/env stack
{- stack script
   --snapshot lts-22.23
   --extra-dep opt-env-conf-0.0.0.0
 -}

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE NamedFieldPuns #-}

import Data.Text (Text)
import OptEnvConf
import Text.Regex.TDFA (getAllTextMatches, (=~))

...

However, stack couldn’t infer which Data.Text I wanted to use (from the modules available in the snapshot), because it could be from incipit-base, text, etc. Then, I explicitly added all explicit packages (except opt-env-conf), and that’s where my confusion stemmed from: I thought that it was not needed because I only had in mind how the documentation examples looked like, and thought that with only specifying --extra-dep, I was fine.

1 Like
#16

Ah I see, stack script’s auto package detection going wrong. That functionality could use more docs.

1 Like
#17

Actually, Stack’s auto-detection works. If the import in a script is ambiguous, Stack detects that and reports the two or more packages that could be intended. The online documentation has been extended.

1 Like