@mxgr, I believe the root of the problem is a very confusing distinction between local and non-local packages for cabal. Put shortly, command line flags (such as --extra-include-dirs) only apply to local packages.
In your case, when you do cabal install hw-kafka-client, the target hw-kafka-client is considered local, so the command line flags apply.
When you use a cabal file and list hw-kafka-client as a dependency, it is not local. In that case, core is a local package so the flags apply to core, which is not very useful.
What you want is to specify flags for non-local packages. You can do that by creating a cabal.project file in the root of your package with the contents similar to this:
packages: .
package hw-kafka-client
extra-include-dirs: /opt/homebrew/Cellar/librdkafka/2.6.1/include
extra-lib-dirs: /opt/homebrew/Cellar/librdkafka/2.6.1/lib
It says to apply those flags to that package.
The ticket in Cabal where people with this problem end up: --extra-lib-dirs and --extra-include-dirs ignored? · Issue #2997 · haskell/cabal · GitHub