According to Slime, Orbits are caused by the struggle between the acceleration of gravity pulling the object inwards, and inertia (the object's natural tendancy to go forwards.) When these two things work together, the object tries to move to the side, but then is pulled in a little bit, then moves to the side, and is pulled in again, causing it to go in a circle.
Note that the "pulling" is only done on the velocity, which in turn changes the way that the object moves. The object's position itself isn't changed directly by acceleration. (acceleration changes velocity, velocity changes position.)
And all that's necessary to cause this effect is to continually pull the velocity vector towards the gravitational source. This is done by adding (the direction to the gravitational source times some constant) to the velocity vector.