Network layer/routing issue causing poor hitreg on DOD:S servers :(

R

Rufio

Active Member
Joined
Mar 23, 2013
Messages
263
**Please read entire original post before commenting to allow for efficient follow-on discussion**

BACKGROUND: Writing a research paper for my Telecommunications class around network protocols for multiplayer game developers to determine the best protocols to use in FPS games and how servers and clients can optimize their local configuration files to reduce hit registration errors. The first step was looking for a test server that I had a consistent and reliable connection to. I started by using PingPlotter Pro to conduct traceroutes to the most popular DOD:S servers that are still active. While tracing the network path from my client to our server, I noticed that one of the routers/access points along the path may have significant congestion control issues which is causing VERY high latency and some packet loss for packets traveling to their final destination - our server. Below are my observations, data/conclusions, and recommendations for correcting this significant issue. If you have networking experience, please feel free to add any information you think is relevant. I recommend watching this video (https://www.youtube.com/watch?v=sLCC1Wu2Cmc) to develop a simple understanding of how packets are forwarded from your computer to a server. Essentially your packets are pushed over the network from a series of connected routers to the end destination. Thank you.

OBSERVATION: Bad hit-registration caused by latency, packet loss, or some other anomaly.

HYPOTHESIS: Network layer issue causing latency and packet loss. Specifically the second to last access point the server provider is routing our packets to. This means that there is nothing wrong with the server, but a network link layer issue on a different access point/router. This might not make sense to you if you think that the ping you are seeing in the server is actually how long it takes for your packets to reach that server. The ping you are seeing in-game is simply the time from your router directly to the server ignoring the total network path that your packets take. The analysis below reveals that your packets to our server are taking significantly longer to reach the server than you may previously have thought.

DATA ANALYSIS: The first 2/3 of the hops traced are irrelevant to you as they are mostly routers in my ISP. The hops of particular interest are the the final two or three hops of packets heading into the server provider's network. High average latency and high jitter in the final few hops indicates a server network provider routing issue which will negatively affect your performance on the server.

eGO (173.234.28.20) Yikes...over 700ms latency at the second to last hop
o3vGNoG.png


6th Rangers (72.5.195.106) OH very nice!!!! No packet loss and low latency at each router (hop).
8pGXZdG.png


1st Marine Raider BN (208.167.243.207)
PmB9rSy.png


Snow Bridge (173.199.99.5)
DOpMtGd.png


Blitzkrieg W. (162.248.94.53)
hMcExAI.png


CONCLUSION: The actual server IP address indicates that there isn't anything wrong with our server. However you can clearly see, our server path is the only one with significant delay caused by the router located at 216.105.170.2 (the second to last stop for your packets before arriving at our server). Bottom line, ours is bad, others' are much better.

Looks like this has been an issue for a while:
https://www.edge-gamers.com/forums/showthread.php?304747-Whiskey-Tango-Foxtrot-!


I think that the receive buffer at 216.105.170.2 may be at max capacity as it seems that other gaming communities are also sending a lot of traffic through this access point. By googling this access point I found that a CS:GO community called SG is also experiencing terrible connectivity issues by routing traffic through this router.
http://www.joinsg.net/forums/topic/72563-lag-issues-on-sg-servers/

RECOMMENDATION: This may be something that we want to discuss with our server provider to negotiate a better deal or reroute our packets to a different access point before reaching 173.234.28.20. The current configuration is causing unnecessary latency and packet loss and therefore our clients are seeing poor performance on our servers. This may not be apparent to you if you use a support or assault class as you are already conceding to a rather large and random cone of fire. However, if you use a rifle, with a very narrow and precise cone of fire, you have probably noticed these issues as the hits that your client process registers are often times never registered on the server process.

Our commandant has given us a call to action (https://www.edge-gamers.com/forums/showthread.php?309132-DOD-S-State-of-the-Union-2017) in his state of the union address. I think that if we can nip this problem in the butt by switching to a new virtual machine provider to bypass our current network path, it would certainly result in more attendance. I'm sure that many of you prefer playing on other servers that you get a better connection and that is honestly an unacceptable solution. Our players ought to play on our servers.
 
Last edited:
Basically the second to last stop (router) before your packets reach the server is causing some of your packets to be lost and increasing your latency (both of these are bad things). We can fix this by changing the network path - the path our packets take to reach the server. Unfortunately our server provider may not have the ability to reroute (they only have one router for all of their incoming packets before sending the packets to the appropriate game server process), or they may be unwilling to reroute us. If the above is true, we would have to find a new server provider which may be too costly. Given the solid network path that the 6th RB is experiencing, maybe we should choose their server provider instead of our current provider if its at the same cost to eGO. They currently use a virtual server on nfoservers.com. Quickly looking up virtual server prices, choosing chicago as the location, it would cost eGO $31.99 a month. Now I have no idea if that's better or worse than what we are currently paying, but if its the same then maybe its worth a switch. Our current provider is US dedicated servers and it looks like the cost for the same setup is also $32.
https://www.nfoservers.com/order-virtual-dedicated-server.php
https://usdedicated.com/vps-servers
 
Last edited:
I'll look over this in a second but one thing to note is that traceroutes are not proof of anything except getting the IP/Hostname in the path. These rely on "Time Exceeded ICMP" messages to be returned from the device which decrements the TTL to zero. These are done by the processor, and are almost certainly given the lowest priority and often rate limited by CoPP. I believe our servers are provided by US Dedicated who does DDoS protection so they almost certainly have CoPP.

Part of the problem is desyncs between what your client sees, and what the server sees. Another issue is that setup MG players just don't have hitboxes (joking, but not really). The better the tickrate of the server and client, the less likely these desyncs are to occur. Generally for me this only seems to occur with hipfired weapons, and my guess is that the raycast for the blood splat and the actual raycast for the shot sent to the server are different angles.

If the server is under duress, like when playing on a map that throws the tickrate out the window, the packets may be getting flushed out of the buffer on the server. I notice problems on other servers that run map rotations, such as 6th RB, when they switch to a map like strand/charlie and the server tickrate starts to dip.

1ST MR also has server rate issues which are server enforced, so you get choke like crazy there. Everyone jitters around.
 
Last edited:
Good point. I thought about ICMP deprioritization and sent TCP packets instead and found the same results.
 
TTL value remaining is one thing that CoPP matches against, and all traffic regardless of transport protocol has an ICMP type 11 (exceeded) returned by the router when TTL expires. TTL attacks are quite good at CPU starvation with no CoPP or similar.

We may have a problem, but we would need a UDL jitter IP SLA/RPM to extrapolate any network issues. This would require some software on the server VM that I doubt the tech team would be chill with.

Im more inclined to believe its just the HL2 engine showing its age, despite my incessant whining about reg when it happens to me (its a coping mechanism!).

Here's a bit of additional information about the traceroute utility, and using it for troubleshooting:
https://www.nanog.org/sites/default/files/tuesday_steenbergen_troublshootingtraceroute_62.49.pdf
 
Last edited:
my hit reg has seemed off for some time I mean I know I am not the best player in the game but I was above in kills at one time and I notice I am getting way less hits yet seeing blood splatter. I hope whatever is done will help, cause I like doing better than I have been.
 
GoT HiTReG? said:
TTL value remaining is one thing that CoPP matches against, and all traffic regardless of transport protocol has an ICMP type 11 (exceeded) returned by the router when TTL expires. TTL attacks are quite good at CPU starvation with no CoPP or similar.

We may have a problem, but we would need a UDL jitter IP SLA/RPM to extrapolate any network issues. This would require some software on the server VM that I doubt the tech team would be chill with.

Im more inclined to believe its just the HL2 engine showing its age, despite my incessant whining about reg when it happens to me (its a coping mechanism!).

Here's a bit of additional information about the traceroute utility, and using it for troubleshooting:
https://www.nanog.org/sites/default/files/tuesday_steenbergen_troublshootingtraceroute_62.49.pdf
Click to expand...



Have they (the tech team) examined non low-latency kernels, virtualization running on the server etc.
Servers overloaded, plugins on the game server etc?? If the routing isn't the issue, it very well could be that our plugins are causing issues on the server. Might be worth trimming the fat on some of the unnecessary or unpopular plugins and running straight vanilla admin stuff.

I'm glad this discussion isn't about whether or not there is an issue but rather how to fix the issues at hand.
 
Last edited:
Rufio said:
Looks like this has been an issue for a while:
https://www.edge-gamers.com/forums/showthread.php?304747-Whiskey-Tango-Foxtrot-!
Click to expand...

As far as this particular piece of your post - Although I can see what you're getting at due to the address relation to the graphs, to be fair, this particular evening/night, if I recall correctly, was DDoS heavy (hence my reply to to the thread).

But I will admit I have noticed the issue you've brought up when I occasionally played on other servers in comparison to ours.
 
Ticket has been submitted to our provider. I've noticed this on other servers prior to your post.

I'm waiting to hear back.
 
I followed up with Darnell, tech support at US Dedicated (our server service provider) and he provided some solid insights into what is happening. He said that they use ICMP depriortization to prevent 'denial-of-service' or massive ping attacks on their routers. Ping or Echo requests are given the very least priority when multiple packets arrive (there are always packets arriving so they are always given last priority) so that the UDP game packets can flow through quickly. Essentially, our game packets are flowing just fine to our server.

I saved the conversation if anyone would like to see it. He also mentioned that its probably our plugins that have been causing issues with our application process on the server. He suggested maybe try replacing the current mods/plugins with a basic sourcemod/admin only plugin and testing the server for a few hours that way.

Would any of the admins be opposed to hosting an event on the test server (or donner) with a lighter plugin package active?
 
As much of a nerd that I am and might I say... holy crap well done on this investigation. Although this is insanely interesting and I want to see what can be done to fix it. Are we getting enough complaints to justify changing anything? Is it worth fixing a small bug by taking down a server for testing or anything of that matter if it affects the public. I mean I'd concur if this was affecting a clan ranked match server but, for just the fun eGO servers that many people play is it worth it?

Again, if its something someone can fix with a minimal or no outage id say go for it. It seems like there are enough netsys people here that are addicted to what they do that the resources to get the work done isnt the problem :) I guess while we still have this test server it could be a cool DOD event that we could all goof around while Rufio checks the stats and gets his data.

If that does happen sign me up


Edit: If it does fix it, I am going to miss the raging I have when the kill cam of my opponent shows an obvious headshot wound.
 
Jake, I cant follow the problem, but I just want to point out that taking one of our servers off line for a period of time shouldnt be a big issue. Some of our servers are empty very often, such as Kalt and of course, the event server is used sparingly as well. If the end result is a better server that more people will play on, I think it is worth a shot.
 
We really probably only have

Sourcemod and SMAC
Metamod
Gameme
Stripper Mod
Medic
Advertisements plugin for scrolling chat messages


That's pretty light (there may be a couple more, but those are what I am thinking off the top of my head)

We don't have a ping checker, team balance, AFK manager, spawn protection, parachute, clusterbombs, extra grenades and ammo, etc that a lot of other servers are running. Which is probably for stability reasons, it ain't no fun having everything break and checking things one by one.
 
Last edited:
You must log in or register to reply here.

Latest posts

Back
Top