Was the source of his powers soley kryptonite?
Yes, but there's more to it than that. Ultraman of Earth-3 was a Kryptonian who was sent to earth (his Krypton didn't explode). Kryptonians in the Earth-3 dimension gain powers when exposed to Kryptonite, and in this way, he gained his powers one by one (in addition to a few others that Superman doesn't have, such as Dimension Vision). Presumably, Ultraman only gains one power per Kryptonite lump, otherwise he'd just keep one and keep exposing himself to it until he gets nearly every power around. What Kryptonite IS, in the Earth-3 dimension is not clear, because Krypton-3 didn't explode.
If you mean the second Ultraman that lives in the antimatter universe of Qward that Grant Morrison wrote about and that Kurt Busiek used recently, Qward-Ultraman gains his powers from Anti-Kryptonite, which "powers" his superpowers like a battery. In the Busiek story, Ultraman kept some Anti-Kryptonite in micro-capsules beneath his skin, meaning he could fight for nearly weeks.