I've been able to reach 2.8 on an e4400 with just the stock cooler, but the difference between you and me is that I did know what I was doing. I suggest you read up well on things before you do anything drastic.
The first step to take is to install temperature monitoring software, I use Speedfan, even though not very dependable, it allows me to pick up the difference between Stock temperatures and when it's overclocked.
Second, find out what kind of motherboard you have, have a look at the BIOS just to get familiar with it. Because you will be looking at that screen for a long time. (k, dinner time, can't explain more right now, but the others will be able to help you
)