No, NT4 does not have a RAM limit of 256, to my knowledge. But what Service Pack is it on? Some of the older SP's are a little touchy, but my memory fades since I run Win2000 now.
How is your RAM configured?
Does it have to correct FSB speed to match your mother board?
Are you using 2 slots (most likely) or 3? If 3, I have heard of problems accessing the third board unless you get registered memory. (That topic is out of my league; someone else may be able to elaborate on registered memory.)
Are the memory sticks the same brand? I had problems with conflicts between brands of RAM.
Did you verify that Windows is seeing 512MB RAM? You may have a bad stick.
Try swapping the sticks between the slots. That may help verify if you have a bad stick.
Give us some more specifics on your other hardware in order to troubleshoot your problem. You may also want to visit eng-tips sister site
They handle more computer related problems such as this and you will get advice from a computer guru rather than a bunch of engineers that play with computers as a hobby.
--Scott