I am working on the same type of project. What distribution are you using? I tried it with ubuntu 7.10 server and CentOS 5 both running into a problem of libconntrack not being able to find libnfnetlink.
I will look into my notes about configuring the L7 Filter and post back later.