The real issue that we're dealing with here is the muscle that you're essentially trying to develop isn't that big to begin with and probably has a certain limit to how much you can increase the cross-sectional area. The deltoid (anterior, medial, & posterior) is just not that big of a muscle. So, yes, a lot of this is going to have to do with genetics.
Here is my first question - how is your posture? Are you rounded forward much? Just being in a more stand-up posture may help the appearance of your shoulders.
Plus, one thing that people often never think about is how posture and good biomechanical alignment can affect the muscles. One of the "laws" of muscle is that to generate adequate amounts of force, they must maintain a good length-tension relationship. In other words, if a person has poor posture and bad alignment, a muscle may be stretched out or shortened up due to physiological adaptations (soft tissue "creep") and this may cause the actin-myosin cross-bridges to be poorly aligned to generate adequate force. If your muscle cannot generate adequate force and tension, then how can it be effectively trained and grow? What happens is that other muscles will be called into action to compensate and pick up the slack.
Just something to think about.
Also, any chance of posting a picture? Let's see what we're really working with for starters.
Take care and good luck!
And, thanks for the "plug" Rockdawg21.