This refers to swinging the cat-o-nine-tails.

If the room was too small to swing the whip and lash a poor soul,  it was consdered "to small to swing a cat."