keeping in mind that you're concerned more with the average velocity for the 100 foot profile, i'd be a little concerned from the geotechnical side too. however, being that deep in the profile, i'm not sure what that does to you foundation support. i'd probably want additional borings (or another shear wave velocity assessment) to verify or discount the lower velocity layer. keep liquefaction in mind too. it would be beneficial to make sure you have soil samples and test data to back up your final decision. check the moistures and PI's and go from there...if nothing other than the low velocity layer forces it to class E, i suppose you'll just have to make an engineering judgement one way or the other. are you in a geology known to have soft layers that deep below relatively hard soils?
what method are you using to evaluate the shear wave velocity? you might double check and fine tune your analysis as best you can to make sure the 3m layer isn't more or less thick. this may help you sleep better at night once you make a final decision.
read the entire chapter(s) in asce7 to make sure you haven't overlooked anything and rely on the information you have in front of you. good luck.