Hi,
My minimalistic N-Order Markov Chain project is on air.
9 Likes
Internally, strict maps on keys are used whose WHNF does not force much of the data. My biggest concern using the library would be space leaks. Have you profiled the model generation on a very large input sequence? Be careful when using thunk-building functions such as Map.unionWith (+).
Perhaps you could abstract the steps necessary to build the model, then experiment with different representations of TransitionProbabilites, E.g. hash maps.
If the input sequence is a linked list or vector, you could use the space-saving trick employed in suffix tree: Each key in your TransitionProbabilites a, each context in your Transition type, is a sub-sequence of the observed sequence. Hence a pointer into the original sequence would suffice.
(Actually, starting from a suffix tree of the observed sequence would make the transition count easier. There exist linear-time algorithms for suffix tree construction.)
Hi, thanks for sharing your valuable insights .
I’ll think and work on them.
I have a few idea about possible problems on big data sets. I’ll share here.