You need to consult the actual test method used for obtaining the fracture energy results. The test method will explain what the sample size is supposed to be, and how to calculate the fracture energy. Charpy testing is most commonly used for testing metals, and Izod testing is most commonly used for plastics. Differences exist in the details of specimen size, notch geometry (size, radius, etc.), so they may not be exactly compatible.
In general, for a given volume of material, steel will absorb more energy than aluminum or plastic, which is indicated by the general impact test results. Remember that for a given volume, though, aluminum weighs only 1/3 as much as steel, and plastic will be only ~ 1/7 the weight. This has significant implications for anything other than the standard impact test.