I do not believe it is referred to as 'AASHTO 57 stone', however, #57 stone as defined by quarries, state agencies, etc. is an aggregate blend of size 5, 6, & 7 stone. This material cannot be 'compacted' in a true sense, but can be properly oriented with compaction equipment. This is particularly important when using stone as a replacement material in undercut footing excavations where sidewalls are soft. I have used 57 stone in this application where sidewalls are as soft as 0.25 tsf at total stone depths of 10 feet. As the stone is placed, we use a trackhoe bucket to beat on the stone and press it into the sidewalls - helps to reduce the potential for future stone loss into footing sidewalls, and subsequent loss of contact between poured footings and in-place stone.
As far as testing w/ a nuclear gauge, it is impossible. I have never seen a proper means of doing this, but have often seen specifiers state that it should be compacted to 95% of Proctor values. We simply state in our reports that it should be properly oriented per the directions of the geotechnical engineer.