If you want to see how loud a pair of stereo speakers can get, you turn the volume knob up as far as you can. Similarly, if you want to see how quickly your car can speed up, you might push the gas pedal down as hard as you can. These are both examples of "testing the limits" of something.

The "limit" of something is be largest, strongest, farthest, hottest, etc. possible. When you "test the limits" of something, you're trying to find out what the limit is.

Here are some expressions with "test the limits":

test the limits of my endurance

test the limits of human potential

test the limits of the law

test the limits of credulity