How to stream object from S3 with amazonka-s3 through servant Stream endpoint?

Learn
#1

I’m having difficulties implementing streaming of s3 object through servant endpoint using amazonka-s3.

I boiled down my code to this standalone single-module cabal project:

The currently working solution is inspired by this (5 years old) example from Domen Kožar.

The issue is that the “intuitively looking” implementation

    runResourceT $ do
        resp <- send env req
        pure $ toSourceIO $ respBodyConduit resp

which just unwraps the conduit from amazonka’s GetObjectResponse’s body and converts it to servant’s SourceIO seems to close HTTP connection even before streaming starts.

I suspect (though not sure) that the issue stems from the fact that runResourceT which is required to run send from amazonka somehow finalizes the response body’s conduit before this runResourceT from servant-contuit has a chance to run the stream.

The workaround that seems to work creates resourcet’s InternalState explicitly, and adds the closeInternalState as last “step” of the conduit, which makes sure that resources are finalized after the conduit finishes streaming.

But I must say this feels very non-idiomatic and somewhat unsatisfactory.
Isn’t this solution exception unsafe (e.g. what is the conduit from the response throws an exception? Wouldn’t this mean that finalization of resources is not performed?)

I tried using bracketP in an attempt to make this code more exception safe, but this feels hacky (because the resourcet’s InternalState needs to be initialized outside of bracketP to make it possible to run amazonka’s send…)

    st <- createInternalState
    resp <- runInternalState (send env req) st
    pure $ toSourceIO (bracketP (pure () ) (\() -> closeInternalState st) (\() -> respBodyConduit resp))

Has anyone implemented something similar in nicer way? Is there more “canonical” looking solution to this? Or do you think that workaround is safe (as in “no resources will be leaked in case that response conduit throws an exception”)?

3 Likes
#2

Don’t exit the runResourceT block until you’re ready to close the connection (i.e. until the source is fully consumed). That’s the point of the block.

1 Like
#3

I thought so too, but not sure if this is possible without defining a custom servant combinator or something like that.

You see, to get at the conduit withing S3 response (to be able to call toSourceIO on it), I first have to send the request.
I would have to do that without using runResourceT somehow (which I think I need, because send has MonadResource m within its constraints). But doing that finalizes the conduit prematurely.

Maybe I could use my custom “App” monad (alternative to Handler) which would allow my handler implementation to have unsolved MonadResource constraint and then discharge that constraint by using runResourceT within hoisServer?

Well, I tried that much (see Use ResourceT instead of handler by jhrcek · Pull Request #1 · jhrcek/servant-amazonka-s3-stream · GitHub) but it has the same issue as the original “naive” implementation with runResourceT within handler - streaming ends prematurely :thinking:

If you have better idea, could you please give me further hint?

#4

Oh, nevermind, I think I found the solution.
I just needed to call lift from MonadTrans to make amazonka’s send part of ConduitM :man_facepalming:

solution: send as part of conduit by jhrcek · Pull Request #2 · jhrcek/servant-amazonka-s3-stream · GitHub

1 Like
#5

Someone else asked about this a while back, so you have another example to crib from.

1 Like
#6

Thanks for sharing! It seems that person went for the same workaround as Domen, but I find the solution with lift (and not relying on lower-level resourcet stuff) cleaner and more exception safe.

3 Likes