Network Neutrality is a sticky issue, commingling separate issues. Let me handle them one by one.
Far and away the biggest "first mile" providers are the telephone and cable companies. Why? Because they had legislated monopolies, guaranteed to be profitable. They don't anymore, but then again, they already have cables in place. Any overbuilders are often prohibited from installing their own poles by law, and the incumbents are allowed to charge large "make ready" costs to overbuilders. Disincentives for foot-dragging? Nonexistent. So we're not talking about a free market for providers here. Expect competition to work poorly.
Linear infrastructure is hard. Think of multiple kinds of pipes, roads, and wires. The infrastructure MUST be 100% contiguous to have any value at all. There are huge up-front costs to create this. A right-of-way must be negotiated, and the infrastructure must be built. Once built, it is a sunk cost. It's hard for today's prices to pay for yesterday's sunk costs. Expect competition to work poorly.
Network providers are selling a commodity. All Internet access is the same, by definition. Prices are easy to discover. Not much room for price differentiation (except by bandwidth). Acme Networks might try to, say, give their customers better bandwidth to Youtube, but Beta Networks can buy the same type of faster pipe to Youtube, and for the same price. Youtube has no interest in giving any network an exclusive deal, since their value is in uniform accessibility all over the network. Expect ruinous competition.
On top of all of that, you have the fact that a lot of the Internet bandwidth is consumed in copyright violations. The copyright holders get grumpy at that, and keep pressuring the ISPs to take steps to stop file sharing. Naturally, the ISPs don't get any money from the copyright holders, and have little interest in providing worse service for their customers. On the other hand, cable networks aren't designed for symmetric use (such as you need for many peer-to-peer applications). Such connections are often shared (see next paragraph) and the more file sharing the less bandwidth.
Network providers oversell. When you buy a "5 mbps" connection from an Internet provider, they know full well that you will only be using a fraction of that bandwidth at any moment. When you want it, you need it, but you don't often want it. Consequently, the network providers oversell their connection. They can sell 10 "5 mbps" connections over the same 5 mbps pipe, and everybody will feel that they have 5 mbps. Works fine for Internet browsing. Doesn't work so well for downloading music and video.
This market will usually have one, sometimes two, and rarely three providers (phone, cable, and WiFi/WiMax). We have more than one provider for technological reasons: phone wire was for phone and cable wire was for video. But fiber to the home is coming. Once one company runs fiber to the home, nobody has any reason to overbuild. That one fiber can bring all three types of connectivity: phone, video, and Internet). Not likely that an overbuilder will get enough take-up to justify the cost of installation.
Price differentiation, and indeed, service differentiation only makes sense in a market with choices. Corporate-owned fiber to the home has no provision for choices. Don't look at the past to justify network neutrality. Look to the future, where every communication technology into someone's home is owned by a monopoly.
Having said all that, a better future is available by neighbors pooling their resources and building out their own fiber. Bring it to a central standard connection point to which providers bring wholesale services. Rather than merely fighting against network neutrality we should be convincing people to solve their problems through the creation of a market for connectivity. Coercion bad, markets good. You don't have to be a pacifist to believe that.