hello roaling
lots of books treat fins of variable cross section area, and most of them walk through the derivation of one example geometry.
for example Bejan ("Heat Transfer") derives the tapered fin equation....i think the "serated" design you are looking at is his "tapered" design rotated 90 degrees......it might not be hard to take your geometry and walk through his step by step derivation.
actually you might be able to use the basic tapered fin equations (fin of linearly varying cross section) as ithey stand....havent really studied that possiblity.
daveleo