| Posted by DjiXas, 05-21-2008, 08:57 AM | |
Anyoen experiencing issues? All our sites are very slow as well as
https://www.liquidweb.com itself.
|
| Posted by blowouthosts, 05-21-2008, 08:59 AM | |
Yes, it is. That is the slowest i have seen a hosts site load for in a long while. Maybe it is a DDOS attack?
|
| Posted by net, 05-21-2008, 09:03 AM | |
Loading fine here but a little bit slow. Maybe they updated something in their site that is causing a slow in surfing...
|
| Posted by FuNKyMIkE, 05-21-2008, 09:38 AM | |
Level3 OC-48 Outage
At 1150 UTC the Level3 OC-48 circuit to Liquid Web went into a down/down state. The Liquid Web Inc. Network Engineering Team is presently working on resolving this issue with our provider as quickly as possible.
Until Level3 corrects this issue with the circuit customers will notice increased latencies to the data centers. Global connectivity to the data centers is still intact but in a degraded state presently. We apologize for the inconveniences that this introduces and are pushing the provider as quickly as possible to correct this issue.
--
Liquid Web Inc.
Network Engineering Team
Update - 1300 UTC
The Liquid Web Inc. Network Engineering Team has received notification that the Level3 outage is due to a fiber cut on the transport circuit from our Lansing Michigan based data centers to the Level3 POP in Southfield Michigan. We are awaiting an ETR from our transport provider and once there is one we will provide an update here.
Again we would like to thank everyone for the patience and understanding regarding this issue.
--
Liquid Web Inc.
Network Engineering Team
Source : http://liquidweb.com/support/
|
| Posted by Floridasan, 05-21-2008, 10:35 AM | |
I can't even get on my site. But that is nothing new for Liquid Web. I am so tired of dealing with these issues with LW.
|
| Posted by Floridasan, 05-21-2008, 10:39 AM | |
I am nowhere near Michigan.
|
| Posted by DjiXas, 05-21-2008, 10:49 AM | |
You've more issues as well?
|
| Posted by FuNKyMIkE, 05-21-2008, 10:52 AM | |
Personnaly it is my first issue in a year with Liquidweb. I am a very happy customer.
Hopefully it will be solved rapidly, but anyway, my sites are still online so it's not a big deal. They're just slower than usually.
|
| Posted by Floridasan, 05-21-2008, 10:53 AM | |
I just called Liquid Web. They said "something caused your load to go real high and your server needs to be rebooted." Well, my site has been off line for 30 minutes. So I guess if I didn't call them my site would not be rebooted? This is ridiculous. My site's downtime with Liquid Web has been horrendous since I purchased the dedicated box. I need to move to another host but what a headache that is.
|
| Posted by Floridasan, 05-21-2008, 10:55 AM | |
I guess you don't have a high traffic or PHP intensive site like mine.
|
| Posted by DjiXas, 05-21-2008, 10:58 AM | |
I'm in the same boat with you then. Still have 1.5 month left with this one. Not sure if I'll move if won't experiene any more issues.
|
| Posted by Floridasan, 05-21-2008, 11:08 AM | |
I hosted my site on Yahoo and was never offline once. I outgrew their biggest bandwidth package (500 gigs/month) so I moved to Liquid Web, which, as I said, has been a nightmare.
|
| Posted by FuNKyMIkE, 05-21-2008, 11:35 AM | |
Well you're not guessing right.
If your server is slow with your application you can :
- tune your server (should be liquidweb's job, but also your's)
- optimize your site/scripts -> your job
- choose and pay an adequate hardware for your site -> your job, helped by liquidweb for the choice.
|
| Posted by Floridasan, 05-21-2008, 12:13 PM | |
I'm a female with a small Wordpress blog that gets a lot of traffic. I don't know about optimizing scripts. I pay almost $300 a month for my hardware.
|
| Posted by cristibighea, 05-21-2008, 12:19 PM | |
In that case you should spend some extra cash to get someone to optimize it for you, as FuNKyMIkE is right.
|
| Posted by CretaForce, 05-21-2008, 12:25 PM | |
How many hits does the blog gets? Do you use wp-cache?
|
| Posted by jnguyen13, 05-21-2008, 12:40 PM | |
We are having high latency issues as well. Pinging our dedicated server results in 300-400ms response times. The website is taking super long to load. Even if you are not experiencing the latency, some of the users trying to connect to your website definitely are.
|
| Posted by Hastings, 05-21-2008, 12:54 PM | |
I'm with LiquidWeb as well,s just called and found out that the level 3 fiber connection was cut - supposedly the level 3 company (whatever their real company name is called) has dispatched a repair crew, but as of now LiquidWeb doesn't know how big the cut was, they hope it'll be "back by the end of the day".
|
| Posted by Floridasan, 05-21-2008, 01:29 PM | |
My blog gets over 200,000 page views a day. I know that's not crazy traffic but it's enough to drive the server bonkers. I will look into wp-cache if that helps.
|
| Posted by magnalight, 05-21-2008, 02:26 PM | |
The reality is that Liquidweb should have 2 distinct providers with a cutover plan when one fails. We have 2 distinct providers in house to keep office staff and customer service operating with internet access in case one provider fails. This is real basic redundancy 101. Given that hosting is their business, why would Liquidweb rely on a single provider? It doesn't make sense to me. The acquisition costs of new customers are too high, only to forfeit the sale due to poor site speed. I am baffled. Their internet access should never be in my thoughts...ever.
|
| Posted by CretaForce, 05-21-2008, 02:42 PM | |
Try to use wp-cache, also install xcache to boost php performance. If LiquidWeb can support it ask them to setup lighttpd instead of apache.
But I still don't understand why you have problems with only 200.000 page views. We have customers that do more than 500.000 page views (wordpress) on our smaller shared hosting plans and they using only 2-5% of server resources.
Last edited by CretaForce; 05-21-2008 at 02:46 PM.
|
| Posted by RWMedia, 05-21-2008, 02:43 PM | |
I hope it comes back up alright!
|
| Posted by Jedito, 05-21-2008, 02:44 PM | |
If you have it correctly installed/set , you'll see a big improvement.
We have a customer using the 75% of the CPU with his WP blog, after install WP-Cache, lowered to 7-12%.
About LW, this is the 4 network issue in less than a month.
|
| Posted by nina111, 05-21-2008, 03:03 PM | |
Everything is slow with Liquidweb - not only the sites but support also. I have lost the whole day and still don't have response to my ticket.
|
| Posted by wbpro, 05-21-2008, 03:03 PM | |
well and again same issue as always, one of their providers have problems and their network gets congested. There is no redundancy I believe.
All our 5 servers are super slow, so much that there is people that can't even access the servers.
in one month we have had already 4 issues like this? at least today we are not 100% down.
|
| Posted by LiquidWebTravis, 05-21-2008, 03:08 PM | |
-Status Update-
Level3 OC-48 Outage
At 1150 UTC the Level3 OC-48 circuit to Liquid Web went down. We have received notification that the Level3 outage is due to a fiber cut caused by a construction crew in downtown Lansing which affected the transport circuit from our Lansing Michigan based data centers to the Level3 POP in Southfield Michigan. We are awaiting an ETR from our transport provider and once there is one we will provide an update. They are actively working to repair the circuit. They assure us it should be repaired very shortly.
This has NOT affected all of our customers and has not affected all connections. Our MCI and Savvis connections are working perfectly with no latency. This fiber cut is causing increased wait times only on our ATT connections.
For up to date information, please check
www.liquidweb.com/support
If anyone needs to be provided an SLA Credit for this event, please submit an SLA credit through our ticketing system and once this event is corrected you will be receiving your compensation for this event.
As always, we take these events very seriously. As many of you know, we are in the process of adding a significant amount of new connectivity which will prevent this from happening in the future.
If anyone would like to discuss this even with me I would be happy to call you personally. You can find my email and phone numbers below.
We sincerely apologize to the customers that are being affected by this and we will be sure to compensate you for this unfortunate fiber disruption.
|
| Posted by Floridasan, 05-21-2008, 03:08 PM | |
That sounds great! I tried to install wp-cache on my server but I got stuck at:
I have no idea what "executing -s" means.
|
| Posted by vibrokatana, 05-21-2008, 03:13 PM | |
"ln -s" creates a symbolic link. Think shortcut for windows or alias for OSX.
|
| Posted by Floridasan, 05-21-2008, 03:23 PM | |
Can you advise me how to do it on my server?
|
| Posted by vibrokatana, 05-21-2008, 03:29 PM | |
cd into wp-content/plugins/wp-cache/
(ssh into the server and use "ls" to list where you are and "cd nameofdir" to change to that directory. Use tab completion to keep yourself sane)
$ ln -s ./wp-cache-phase1.php ../../advanced-cache.php
|
| Posted by DjiXas, 05-21-2008, 03:34 PM | |
Hi,
Can you answer to magnalight's question? I've the same one actually.
|
| Posted by Floridasan, 05-21-2008, 03:41 PM | |
Thank you! I found a cool plugin that does this for me.
|
| Posted by MaB, 05-21-2008, 03:50 PM | |
Travis, this would be our third SLA request. As much as I'd love another refund for this seemingly endless series of network issues, perhaps Liquidweb should hang on to some of it to ensure that this doesn't continue to happen in the future
|
| Posted by LiquidWebTravis, 05-21-2008, 04:01 PM | |
- This event should be resolved now. We will be updating with more information in a few moments.
Thank you,
|
| Posted by LiquidWebTravis, 05-21-2008, 04:31 PM | |
Our transport provider has completed the repairs to the fiber between our Lansing Michigan based data centers to the Level3 POP in Southfield Michigan. At this time traffic is routing over the Level3 circuit and latency issues should have subsided at this point. The Network Engineering Team is continuing to monitor this circuit to ensure that it will continue to operate in an acceptable manner.
We would again like to extend our gratefulness to everyone regarding their patience and understanding regarding this issue. If you have any concerns or questions regarding this outage which are not addressed here please open a ticket and one of our Systems or Sales engineers will be back with you as soon as possible to address them for you.
|
| Posted by webmarksllc, 05-21-2008, 04:32 PM | |
Of all the hosts with whom I've dealt, LW is by far the best for customer service (for dedi customers at least).
The recent network disruptions (DDoS attacks, then this fiber cut) bely the robustness of LWs connection to the rest of the world, and I think I am not alone in my desire to see them be a thing of the past - which I would assume is the purpose of adding more (varied?) capacity to LW's worldly interface.
Thanks for all your hard work, and thanks in advance for letting us know when we won't have to answer to angry customers.
|
| Posted by LiquidWebTravis, 05-21-2008, 04:38 PM | |
I appreciate your support and kind words during this very unfortunate event. We are definitely working to add this capacity to prevent any future problems. I'd be happy to discuss this with you personally. Take care,
|
| Posted by sirius, 05-21-2008, 04:45 PM | |
LiquidWeb has no control over a fiber cut impacting Level 3's network.
Sirius
|
| Posted by MaB, 05-21-2008, 04:48 PM | |
Of course not, but they *do* have control over their spare capacity. Their phone tech explained that the additional latency and speed issues were caused because all the Level3 traffic shifted onto their other circuits (as they should have) but some of the other links were getting saturated. A properly engineered datacenter *should* have enough spare capacity to handle a circuit failure with no impact on performance. This is obviously not the case at LiquidWeb based on the events today.
|
| Posted by bqinternet, 05-21-2008, 04:52 PM | |
No, but Liquidweb is!
|
| Posted by lood, 05-21-2008, 04:53 PM | |
I'm with them for almost 2 years, and my blog gets 400,000 views/day w/o problems after some sql/apache tunings.
Today's issue is the first serious issue for me, so compared to others hosts, it's a good average
|
| Posted by toddt, 05-21-2008, 05:01 PM | |
[quote=LiquidWebTravis;5120789]-Status Update-
Level3 OC-48 Outage
This has NOT affected all of our customers and has not affected all connections. Our MCI and Savvis connections are working perfectly with no latency. This fiber cut is causing increased wait times only on our ATT connections.
Granted I don't know "how things work" in terms of hosting but I'm just curious as to why some customers didn't experience any slowdown and some of us had basically unusuable websites for the better part of today. Isn't the purpose of having redundency to spread the load out in case of a problem? I'm not trying to be a jerk, but when I see multiple bandwidth providers advertised I would not assume a problem with one could cause a meltdown for those unlucky enough to be associated with a particular connection.
Todd
|
| Posted by LiquidWebTravis, 05-21-2008, 05:10 PM | |
This is not accurate, as I stated before, our MCI and Savvis connections were not affected and customers received fine connections of those providers.
I completely understand your frustration and I am not trying to make excuses, just wanted to clarify your post.
|
| Posted by MaB, 05-21-2008, 05:12 PM | |
Thank you Travis. Can you describe what the problem was with the AT&T circuit?
|
| Posted by LiquidWebTravis, 05-21-2008, 05:13 PM | |
[QUOTE=toddt;5121000]You are partially correct and I understand exactly your concern. However if we were to blindly fail over all bandwidth to the other providers, that would have transfered the latency issue to those connections.
I've been assured that this will NOT be a problem once our new connections arrive.
|
| Posted by MaB, 05-21-2008, 05:15 PM | |
[QUOTE=LiquidWebTravis;5121022]Travis, can you please describe what happened exactly with the AT&T line? Your post makes it sound like all the Level3 traffic shifted onto your AT&T links which didn't have enough capacity, while MCI & SAVVIS links remained unsaturated.
Thank you in advance for clarifying
|
| Posted by LiquidWebTravis, 05-21-2008, 05:15 PM | |
The only problem with ATT for some people was the saturated line due to level3's fiber cut which caused slow connections or long wait times.
I'm really sorry.
|
| Posted by MaB, 05-21-2008, 05:17 PM | |
So then my original statement was in fact accurate. LiquidWeb didn't have enough capacity to handle a Level3 circuit being cut without causing latency and other performance issues.
"A properly engineered datacenter *should* have enough spare capacity to handle a circuit failure with no impact on performance. This is obviously not the case at LiquidWeb based on the events today."
|
| Posted by hostmaniac, 05-21-2008, 05:31 PM | |
Travis,
Thank you for keeping us informed in an honest and communicative manner. I was scratching my head all day today trying to figure out what might be causing the sluggish speed. Liquidweb has been so fast and reliable for the past year (since we moved to LW) that I suspected everything else before that.
These things happen and I just hope that your data centers are well prepared to deal with such incidents in the future. The last data center we used had a serious incident and not a good redundancy plan which resulted in 4 days of downtime and very slow connectivity. That caused at least 30% of our clients to move away. Now we are selling streaming services and network speed is of utmost importance. That's why we moved to LW
|
| Posted by hostmaniac, 05-22-2008, 03:11 PM | |
Travis,
The problems seem to be persisting today even though you guys announced it was fixed last night. It did look fixed indeed but today again our customers are complaining that their videos are not playing well. It's starting to affect our business since we're selling media streaming services. What's going on?
|
| Posted by LiquidWebTravis, 05-22-2008, 03:31 PM | |
That is not correct at all, the problem has not persisted.
Please submit a ticket, our network is working at peak performance. You can also email me if you have any issues what so ever.
|
| Posted by hostmaniac, 05-22-2008, 04:39 PM | |
Travis,
I sent some traceroute's to your support and waiting to hear back (Ticket #1083236). I try downloading a file from either of my servers and I get a very slow download speed. Try for yourself:
http://www.registerforless.com/flash..._720p_1500.flv
|
| Posted by CretaForce, 05-22-2008, 04:47 PM | |
Very slow speeds from my 4mbit ADSL in Greece, 15kb/s. And from Leaseweb server I get 215kb/s. Both Leaseweb and my home connection go through Level3.
|
| Posted by HarriL, 05-22-2008, 04:49 PM | |
I have Pingdom monitoring my servers. US connections seems to be all right now but international connections are still very slow.
22/05/2008 23:41:15 169 ms Dallas 3, TX, USA
22/05/2008 23:36:15 275 ms Houston, TX, USA
22/05/2008 23:31:15 10 567 ms Reading, England
22/05/2008 23:26:15 262 ms Dallas 2, TX, USA
22/05/2008 23:21:15 10 627 ms Amsterdam, Holland
22/05/2008 23:17:02 294 ms Houston, TX, USA
22/05/2008 23:16:23 Timeout (> 30s) Vasteras, Sweden
22/05/2008 23:11:15 644 ms Berkeley, CA, USA
22/05/2008 23:06:15 200 ms Dallas 3, TX, USA
22/05/2008 23:01:15 570 ms Houston, TX, USA
22/05/2008 22:56:15 10 545 ms Reading, England
22/05/2008 22:51:15 280 ms Dallas 2, TX, USA
22/05/2008 22:46:15 10 652 ms Amsterdam, Holland
22/05/2008 22:42:05 10 552 ms Reading, England
22/05/2008 22:41:26 Timeout (> 30s) Vasteras, Sweden
|
| Posted by DjiXas, 05-22-2008, 06:15 PM | |
So when will you fix everything? I'm tired of your problems.
|
| Posted by hostmaniac, 05-22-2008, 06:27 PM | |
DjiXas, the problem appears to be related to Level3 and not just Liquidweb. I'm sure the guys at LW are doing their best as always, please be respectful.
Seems to be working OK now. Most of the complaints we had from users were from our Canadian customers (using Rogers, Sympatico, & Shaw ISP's). Also one complaint from Texas.
I'm in Toronto by the way behind Rogers ISP.
|
| Posted by Jedito, 05-22-2008, 07:28 PM | |
I got reports of people still having slow connections to our servers at LW.
And yesterday I have to made a +$400 refund due the 4 downtimes in this month. Not happy at all.
|
| Posted by LiquidWebTravis, 05-22-2008, 07:52 PM | |
The problems that people are seeing with international carriers, are just that, problems with international carriers. It is completely and wholly unrelated to the Liquid Web network or the Level 3 fiber cut.
If you continue having problems, open a ticket and our team will help trouble shoot it with the problem providers.
Again, this is unrelated to the liquidweb network or the level3 fiber cut.
Additionally, we HAVE already added additional connections since yesterday. Another connection with SPRINT was added yesterday 5/21.
|
| Posted by hostmaniac, 05-22-2008, 08:07 PM | |
Anyone come across any web site or news clips covering this issue? Forwarding a link to our customers will help us show them the problem is not related to our service.
|
| Posted by EEssam, 05-23-2008, 09:45 AM | |
The problem is still there, many clients are complaining.
|
| Posted by HarriL, 05-23-2008, 10:01 AM | |
My servers are back to normal.
|
| Posted by LiquidWebTravis, 05-23-2008, 11:35 AM | |
What is your ticket number?
This problem is NOT still there.
|
| Posted by hostmaniac, 05-23-2008, 01:36 PM | |
I too am noticing this problem again today with both my LW servers. Download speed is ~35 kb/sec which completely kills the media streaming service. It resolved yesterday around 5:30pm EST.
I opened up a thread asking people to test the download speed so we can try to figure out who's affected:
http://www.webhostingtalk.com/showthread.php?t=695349
|
| Posted by hostmaniac, 05-23-2008, 02:36 PM | |
traceroute from my server to my home IP:
Hop 2 & 5 are problematic!
Travis: Ticket #1083236, waiting to hear back...
|
| Posted by MaB, 05-23-2008, 02:39 PM | |
Hi,
It actually looks like your traceroute is fine. Hops 2 & 5 are simply slower to respond to ICMP packets which is normal for heavily used routers. The facts that hops 3 and 6 have normal ping times (as doe the destination) shows that hops 2 & 5 are operating normally.
|
| Posted by FastServ, 05-23-2008, 03:22 PM | |
As of today, I'm getting terribly slow speeds from that link and I'm on the eastern US coast. Either your server has problems, or major issues still exist in the network.
|
| Posted by hostmaniac, 05-23-2008, 03:23 PM | |
It's odd, I'm getting 30 KB/sec downloads from either of my two servers (e.g. http://kimiauranus.com/test.zip) but download from Liquidweb's own web site (http://www.liquidweb.com/data/10mb.bin) is 1100 KB/sec.
Traceroute to Liquidweb.com:
Traceroute to my server:
Are these 'Request timed out' notices indicative of a problem or normal?
|
| Posted by hostmaniac, 05-23-2008, 03:28 PM | |
Both my servers have low loads. One is a new server with load near 0. Not a server issue.
|
| Posted by hycloud, 05-23-2008, 04:15 PM | |
No slow down here in the US west coast.
|
| Posted by LiquidWebTravis, 05-23-2008, 04:24 PM | |
Hostmaniac,
As you know, our Lead Network Engineer, Darrel is responding to your tickets and investigating the issue that you are seeing. He's in the process of responding to your ticket. You should receive an email shortly.
Again, this issue is wholly unrelated to the Level3 fiber cut.
|
| Posted by vibrokatana, 05-23-2008, 04:44 PM | |
I wonder if it is a faulty NIC or switch? Hopefully they can solve the issue [fast].
|
| Posted by LiquidWebTravis, 05-23-2008, 04:46 PM | |
Hi Hostmaniac,
As I'm sure you're aware by now, since your last conversation with Darrel, this is a server specific issue, which is why you get 1100kb/s from liquidweb.com.
Our system engineering team is going to be making some tweaks to optimize your server performance. Our engineers will be following up with you shortly.
Thank you,
|
| Posted by Jedito, 05-26-2008, 11:11 AM | |
Is anybody else still having speed problems?
I keep getting reports every day from people downloading files at 10 kbs on servers with load of 0.1. When I report it to LW they keep saying that's there's no problem at the network, but it can't be just casuallity that many people from many parts of the world (including the USA), are complaining about the speed.
|
| Posted by CretaForce, 05-26-2008, 11:43 AM | |
Level3 seems ok. I get 350kb/s from a 4mbit ADSL connection from Greece.
|
| Posted by hostmaniac, 05-26-2008, 02:18 PM | |
Yes, we're still having problems. There's two good clues I have on this problem:
The problem occurs with some IP's on my servers while not others. This is probably the best clue. For example, right now a file from 67.227.133.225 is downloading at 25 kb p/sec and a file from 67.227.132.25 is download at 700 kb p/sec. They're both IP's on the same server!
The problem resolves itself usually around 6pm every day.
|
| Posted by FastServ, 05-26-2008, 05:41 PM | |
Try doing a traceroute FROM each IP on the server. Specify the IP on the traceroute command so it sources from the correct IP.
Do you see a different outbound path for each IP? If so, there is an unusual form of traffic shaping (source routing) going on and that is well known to cause issues.
If you don't see different source routes, then I don't know what to say.
|
| Posted by Jedito, 05-26-2008, 09:00 PM | |
Yes, that's exactly what I was told to by the customers reporting the problem.
|
| Posted by hostmaniac, 05-26-2008, 09:48 PM | |
Do you also find that the problem exists with some IP's and not all? The bulk of the server IP's including the main server IP's are affected but I've confirmed some IP's are not affected. It's strange because they are all on the same subnet.
|
| Posted by DjiXas, 05-28-2008, 02:52 AM | |
Can anyone confirm that when pinging via http://just-ping.com it gives you a packet lost for Shanghai?
Shanghai, China Packets lost (10%) 299.0 303.4 311.2
yesterday was 100% lost
|
|
 Add to Favourites
 Print this Article
|
