There's some free stuff here
that could get you started. Keep on the Shadowfell (intro rules/adventure) + the free version of Character Builder is more than enough to run a few games.
For advice on how to be a DM, the Dungeon Master's Guide and Dungeon Master's Guide 2 are both pretty good. (Something called The DM's Kit just came out, too, but I haven't gotten a look at it yet.)
I'd also suggest investing in a "battle mat" and some dry erase markers, since the latest version of D&D relies a lot on having a map/game board during play. You'll also need either minatures or some sort of tokens to keep track of where the players' characters and monsters are. I'd probably hold off on buying a ton of minatures until you're sure you like the game. Either make your own tokens or buy some. I bought this token set
a while back and it was pretty good.
If you find you like this stuff, a subscription to D&D Insider is a pretty good investment. It basically gives you online access to nearly all the rules that have ever been published for the 4th edition of the game, plus the full version of the Character & Monster builders.