you are using FEA (linear elastic material) incorrectly ... or rather extracting invalid information from the model. Any stress above yield is invalid (for a linear elastic material). You may be able to use this with a "Neuber" notch analysis ... but I haven't.
So one fix is to use non-linear material (elasto-plastic).
Also, I surmise, your notch has a sharp (point) radius, which is easy to model but impractical.
Also where you are trying to get the stress at the base of a Kt, you need a lot of mesh refinement to have any chance of a real value.
But I won't trust any model result for this.
Normally in fatigue analysis, we use gross section stress (which we might get from FEA), and the Kt of the notch is applied separately, and then some analysis with S/N diagrams and some spectrum of loads.
If you want to do fatigue analysis, there are plenty of texts out there. Petersen is a good source for Kt, but there are many.
You may want to investigate "ncode" which is an FEA based fatigue analysis.
"Hoffen wir mal, dass alles gut geht !"
General Paulus, Nov 1942, outside Stalingrad after the launch of Operation Uranus.