I believe it is 32,768, which is the maximum number that can be represented in a signed 16-bit integer, which Rockstar uses to store ammo.
I will make it so players can't buy more ammo than this, or possibly even 10,000 (because even carrying that much is just plain silly..).