Help with Internet speeds

[P-Dub]

Problem: At work our connection to the outside internet has been very slow for the past 2 months. Prior to this there was not any problem or noticable slowdowns. Starting in Dec, how we connect to the net has been really slow. Response time from calling up a website and downloading. I have been tracking using Speedtest and at the best of times we get 1.2M down, 500k up and a latency of 50ms. At the worst of times, which seems to be a lot these days, we get maybe 500k down, 250k up and a latency as much as 500ms. This is during normal business hours. On top of this, during nonbusiness hours we have remote access to our desktop PC's and this response is terrible! I believe it is a result of the poor up speeds of our ADSL service. Now when I started, Sept - Nov, I used the remote desktop extensively and the speed was acceptable. I have used remote desktop in other companies, so I know what can and can't be done.

Another example is that our Exchange is offsite and at times it just takes forever for it to sync. Never used to do that.

We have had construction near by, across the lane, but this has been going on since I started. That is one excuse our IT guys have been saying.

So now I am really at wits end. Will a dedicated T1 line give us the performance we need? or is there another issue that we have to look at?

Is the increase in upload speed going to solve the issue, or is it the decrease latency? What is latency anyway? I just know that lower is better. but when I look at 50ms to 500ms, is that like multiplied by the number of hops to a site?

Me very :?:

[Kevin P]

Do you know any other sites that use the same ISP/service as you? Ask them if they're having slowness issues as well.

Try doing a traceroute to various sites. That should tell you where your bottleneck is. If it's in the first hop from your internal router to the ISP, then it could be a line issue which the telco can address (noise on the line, etc). If those hops are fast but it's slow a few hops downstream, then it could be a problem at the ISP or their upstream provider.

What download and upload speed are you supposed to be getting (provisioned)?

[P-Dub]

Our current contract should be 1.5M down and I think 500k up.

Let me see what kind of results I am getting when I am the only one here. I'll also try during the day tomorrow.

[P-Dub]

This is what I get right now.

Tracing route to www.l.google.com [74.125.19.147]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.76.45.1
2 20 ms 20 ms 19 ms 10.18.3.254
3 18 ms 19 ms 19 ms 172.33.34.38
4 20 ms 27 ms 19 ms vancbc01dr01.bb.telus.com [208.181.250.110]
5 20 ms 19 ms 23 ms nwmrbc01gr01.bb.telus.com [154.11.4.66]
6 22 ms 24 ms 23 ms 204.225.243.18
7 32 ms 36 ms 38 ms 72.14.197.217
8 27 ms 27 ms 27 ms 216.239.46.50
9 42 ms 41 ms 42 ms 216.239.47.185
10 47 ms 55 ms 54 ms 209.85.251.94
11 43 ms 41 ms 42 ms cf-in-f147.google.com [74.125.19.147]

This is what I get a few minutes later:


Tracing route to www.l.google.com [74.125.19.104]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.76.45.1
2 23 ms 19 ms 19 ms 10.18.3.254
3 18 ms 19 ms 19 ms 172.33.34.38
4 22 ms 19 ms 19 ms vancbc01dr01.bb.telus.com [208.181.250.110]
5 19 ms 19 ms 20 ms nwmrbc01gr01.bb.telus.com [154.11.4.66]
6 23 ms 23 ms 24 ms 204.225.243.18
7 81 ms 23 ms 24 ms 72.14.197.217
8 46 ms 47 ms 57 ms 216.239.46.50
9 153 ms 160 ms 171 ms 216.239.47.185
10 170 ms 159 ms 171 ms 209.85.251.94
11 169 ms 170 ms 163 ms cf-in-f104.google.com [74.125.19.104]

I notice that the last few took longer.

I did it again and the results look worse.

Tracing route to www.l.google.com [74.125.19.104]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.76.45.1
2 134 ms 143 ms 135 ms 10.18.3.254
3 152 ms 134 ms 141 ms 172.33.34.38
4 138 ms 137 ms 141 ms vancbc01dr01.bb.telus.com [208.181.250.110]
5 139 ms 141 ms 147 ms nwmrbc01gr01.bb.telus.com [154.11.4.66]
6 148 ms 141 ms 135 ms 204.225.243.18
7 152 ms 142 ms 135 ms 72.14.197.217
8 157 ms 183 ms 142 ms 216.239.46.50
9 77 ms 80 ms 89 ms 216.239.47.185
10 126 ms 132 ms 125 ms 209.85.251.94
11 145 ms 144 ms 159 ms cf-in-f104.google.com [74.125.19.104]

[littlesaint]

Changing to a different technology only helps if it is the technology that is the problem. A dedicated telco connection (T1, frame relay, MPLS) will certainly improve upstream speeds, and is more reliable overall, but if the problem is something inherent to your provider (bad route tables, misconfigured or unoptimized equipment) or beyond your ISP, changing your connection isn't going to much.

Throwing more bandwidth at a problem is an easy fix, but eventually the real problem will always creep back. I'd work with the provider to try and figure out what is really going on.

[Kevin P]

1 <1 ms <1 ms <1 ms 10.76.45.1
2 134 ms 143 ms 135 ms 10.18.3.254
3 152 ms 134 ms 141 ms 172.33.34.38

Do you know if that middle hop (10.18.3.254) is in your building or at the ISP?

If it's at the ISP, I would say there's an issue with your line, causing noise/transmit errors which cause delays.

[P-Dub]

Kevin, I think that is in our building, if not in our office. I have put in the call to find out.

Here's a typical workday result.

Tracing route to www.l.google.com [74.125.19.104]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.76.45.1
2 671 ms 624 ms 629 ms 10.18.3.254
3 480 ms 507 ms 504 ms 172.33.34.38
4 441 ms 485 ms 421 ms vancbc01dr01.bb.telus.com [208.181.250.110]
5 389 ms 330 ms 296 ms nwmrbc01gr01.bb.telus.com [154.11.4.66]
6 231 ms 225 ms 227 ms 204.225.243.18
7 248 ms 243 ms 245 ms 72.14.197.217
8 315 ms 348 ms 291 ms 216.239.46.50
9 211 ms 137 ms 125 ms 216.239.47.185
10 326 ms 365 ms 340 ms 209.85.251.94
11 180 ms 183 ms 190 ms cf-in-f104.google.com [74.125.19.104]

Now I notice that when everything is working well, it goes well. But when it hits the second point, it slows and everything else slows.

[P-Dub]

Changing to a different technology only helps if it is the technology that is the problem. A dedicated telco connection (T1, frame relay, MPLS) will certainly improve upstream speeds, and is more reliable overall, but if the problem is something inherent to your provider (bad route tables, misconfigured or unoptimized equipment) or beyond your ISP, changing your connection isn't going to much.

Throwing more bandwidth at a problem is an easy fix, but eventually the real problem will always creep back. I'd work with the provider to try and figure out what is really going on.

I agree. That is one reason I am reluctant to just sign a contract and hope that it solves the problem. Once that contract is signed, we will be stuck for at least a year. I really want to be able to determine the route cause so we can find the right solution.

Here is another issue. In a previous company I worked at, we had the same connection package ADSL 1.5M etc. We had a terminal server on site, a real old machine, but it was able to handle our loads without any problem. We had multiple workday users accessing the terminal server, plus staff at the main office accessing the net. Never any slowdowns. So that is why it leads me to believe that throwing more bandwidth at the problem is not the best solution. It is just the easiest.

It is funny how I wasn't supposed to get involved in the IT issues.

[Kevin P]

Hop 2 is where your problem is. Find out where that hop is. If it's in your office/building, then the problem lies in your office/building (perhaps someone/something is hogging bandwidth). Do you know anything about your network's infrastructure? Is the first hop a router, a firewall box, or the ADSL modem with a built-in router? Unplug the line from the modem and then see how many hops you get. If you still get to the 2nd hop, it's before the modem. If not, then it's at the ISP.

Doesn't your company have an IT person? Or are you wearing that hat along with the rest?

[P-Dub]

I don't really know too much about the infrastructure. The 2nd hop IP is a question being asked back to me from our 'IT' guys.

I will have to look into this more. This is not really in my job description, but the degredation is affecting everybody. As much as I'd like to get a fasterline, I have to be able to justify the cost.

I'm not the IT guy, but it seems like I have taken on that responsibility.

[littlesaint]

Hop 3 is the first Internet routed address, so I'm going to guess that hop 2 is still behind your WAN connection. Probably the inside interface on the WAN router. I would look at the connection between your PCs gateway router and the WAN router. You mentioned Exchange being offsite. Do have a VPN running to another site before hitting the WAN router?

[P-Dub]

Hop 3 is the first Internet routed address, so I'm going to guess that hop 2 is still behind your WAN connection. Probably the inside interface on the WAN router. I would look at the connection between your PCs gateway router and the WAN router. You mentioned Exchange being offsite. Do have a VPN running to another site before hitting the WAN router?

I think we have some kind of VPN to another site. We are supposed to go through this login page to connected remotely to our physical desktop at work.

Home->web access->work pc.

Where the web access is definitely at another physical location. Supposed to be very well connected to the net. At another job, the web access device was in house.

I will have to see what our IT guys will do about this.

Thanks for the help guys.

[1Michael]

If I was a betting man I would bet that people in your company are streaming radio which is killing your bandwidth. The IT people can stop that by changing the group policies for the network.

[P-Dub]

Oh thanks for moving this one back up.

Now I haven't gotten the "IT" company to admit it on paper. But there was one PC that was infected and was hogging the bandwidth. How did we find out? Well the switchover to the T1 did not immediately solve the problem. They then started searching and found the trouble PC. So, now we have a premium line at an added cost. There are other business reasons for us to move to a dedicated line. I just found it funny that my initial assessment was correct. They just didn't do the work to identify the problem up front.

Yes there are also some that were streaming audio and using skype.

[here.now]

Oh thanks for moving this one back up.

Now I haven't gotten the "IT" company to admit it on paper. But there was one PC that was infected and was hogging the bandwidth. How did we find out? Well the switchover to the T1 did not immediately solve the problem. They then started searching and found the trouble PC. So, now we have a premium line at an added cost. There are other business reasons for us to move to a dedicated line. I just found it funny that my initial assessment was correct. They just didn't do the work to identify the problem up front.

Yes there are also some that were streaming audio and using skype.

Sounds like you could use an "IT" upgrade as well...