Troubleshooting Docker UDP Traffic

TODO: collapasable, copyable, download, etc

I am trying to troubleshoot why my server stage cannot send UDP traffic to a dockerized service on my host jvb2.

stage has the public IP address of 54.160.180.240 jvb2 has the public IP address of 52.3.233.11

jvb2 has the dockerized service

docker ps
CONTAINER ID   IMAGE                     COMMAND   CREATED          STATUS         PORTS                                                                                                                                           NAMES
03489e1fa3bd   jitsi/jvb:stable-7648-4   "/init"   10 seconds ago   Up 8 seconds   172.31.0.9:4096->4096/udp, 0.0.0.0:4443->4443/tcp, :::4443->4443/tcp, 0.0.0.0:10000->10000/udp, :::10000->10000/udp, 127.0.0.1:8081->8080/tcp   docker-jitsi-meet-stable-7648-4-plauz-stage-jvb2-1

on stage I executed nc -vzu 52.3.233.11 4096

and on jvb2 I executed ``` sudo tcpdump -i any -nn -s 0 -v udp port 4096 tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 17:56:33.633307 IP (tos 0x0, ttl 254, id 47907, offset 0, flags [DF], proto UDP (17), length 29) 54.160.180.240.36639 > 172.31.0.9.4096: UDP, length 1 17:56:33.633349 IP (tos 0x0, ttl 253, id 47907, offset 0, flags [DF], proto UDP (17), length 29) 54.160.180.240.36639 > 172.26.0.2.4096: UDP, length 1 17:56:33.633355 IP (tos 0x0, ttl 253, id 47907, offset 0, flags [DF], proto UDP (17), length 29) 54.160.180.240.36639 > 172.26.0.2.4096: UDP, length 1

What is wrong? Why doesn't docker forward the UDP traffic?Based on the information you provided, it seems that the UDP traffic is being sent to the internal IP address (172.31.0.9) of the Docker container instead of the public IP address (52.3.233.11) of the host `jvb2`.

This could be because the Docker container is not correctly forwarding the traffic from the public IP to the internal IP. Check the following:

1. Make sure that the Docker container is exposing the correct ports. In your case, the container should expose port 4096/udp. From the `docker ps` output, it seems that the port is exposed correctly, but double-check to ensure there are no typos or other issues.

2. Make sure that the host `jvb2` is forwarding traffic from the public IP address (52.3.233.11) to the Docker container's internal IP address (172.31.0.9). You can check this by examining the `iptables` or `firewalld` rules on the host. If you're using `iptables`, run the command:

sudo iptables -t nat -L -n -v

Look for a rule that forwards traffic from the public IP to the container's IP address for the specific port. If it's missing, you can add the rule with:

sudo iptables -t nat -A PREROUTING -p udp --dport 4096 -j DNAT --to-destination 172.31.0.9:4096

3. Ensure that there are no firewalls or security groups blocking UDP traffic between the `stage` and `jvb2` hosts. Check the security group rules in your cloud provider's management console or any additional firewalls on your hosts.

If you have checked all these steps and the issue persists, consider restarting the Docker container and checking the logs for any errors. You can view the logs with:

docker logs

Replace `<container_id>` with the ID of your container (e.g., `03489e1fa3bd`).What does this output tell you?
```sudo iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 216 packets, 10244 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 265K   14M DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
 286K   15M DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            169.254.170.2        tcp dpt:80 to:127.0.0.1:51679

Chain INPUT (policy ACCEPT 209 packets, 9779 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1126 packets, 68057 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   33  1639 DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
   10   352 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            169.254.170.2        tcp dpt:80 redir ports 51679

Chain POSTROUTING (policy ACCEPT 1128 packets, 68115 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   81  5355 MASQUERADE  all  --  *      !br-6d6fb1731cbe  172.26.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !br-774060a9f30d  172.22.0.0/16        0.0.0.0/0           
    2   120 MASQUERADE  all  --  *      docker_gwbridge  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match src-type LOCAL
 3381  205K MASQUERADE  all  --  *      !docker_gwbridge  172.18.0.0/16        0.0.0.0/0           
    2    58 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  udp  --  *      *       172.26.0.2           172.26.0.2           udp dpt:10000
    0     0 MASQUERADE  tcp  --  *      *       172.26.0.2           172.26.0.2           tcp dpt:8080
    0     0 MASQUERADE  tcp  --  *      *       172.26.0.2           172.26.0.2           tcp dpt:4443
    0     0 MASQUERADE  udp  --  *      *       172.26.0.2           172.26.0.2           udp dpt:4096

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  br-6d6fb1731cbe *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  br-774060a9f30d *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker_gwbridge *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           
    0     0 DNAT       udp  --  !br-6d6fb1731cbe *       0.0.0.0/0            0.0.0.0/0            udp dpt:10000 to:172.26.0.2:10000
    0     0 DNAT       tcp  --  !br-6d6fb1731cbe *       0.0.0.0/0            127.0.0.1            tcp dpt:8081 to:172.26.0.2:8080
    0     0 DNAT       tcp  --  !br-6d6fb1731cbe *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4443 to:172.26.0.2:4443
    2    58 DNAT       udp  --  !br-6d6fb1731cbe *       0.0.0.0/0            0.0.0.0/0            udp dpt:4096 to:172.26.0.2:4096

Chain DOCKER-INGRESS (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 265K   14M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0 ```The output shows the NAT table of the `iptables` rules. Specifically, it shows the PREROUTING, INPUT, OUTPUT, POSTROUTING, DOCKER, and DOCKER-INGRESS chains.

From the output, it looks like the Docker container's network traffic is being handled correctly. The rules show that the host is forwarding traffic from the public IP to the container's internal IP (172.26.0.2) for the relevant ports, including 4096/udp:

Chain DOCKER (2 references) ... 2 58 DNAT udp -- !br-6d6fb1731cbe * 0.0.0.0/0 0.0.0.0/0 udp dpt:4096 to:172.26.0.2:4096

This rule is forwarding UDP traffic targeting port 4096 to the container's IP address (172.26.0.2) and the same port.

However, the tcpdump output in your first message showed traffic directed to 172.31.0.9, not 172.26.0.2. This seems to be a mismatch between the actual container IP address and the destination IP address shown in the tcpdump.

Double-check the IP addresses in your setup, and make sure the traffic is being sent to the correct IP address. You can get the IP address of the Docker container by running:

docker inspect | grep IPAddress

Replace `<container_id>` with the ID of your container (e.g., `03489e1fa3bd`).

If you find that the IP address is incorrect, you will need to update your iptables rules to forward traffic to the correct container IP address.docker inspect 332f4d7c76c4 | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.26.0.2",It looks like the correct IP address for the Docker container is indeed 172.26.0.2, as shown in the `iptables` rules.

Considering that the `iptables` rules appear to be set up correctly, the issue might be related to a firewall or security group blocking UDP traffic between the `stage` and `jvb2` hosts. Double-check the security group rules in your cloud provider's management console or any additional firewalls on your hosts to ensure that they allow UDP traffic on port 4096.

If you still encounter the issue, try running the `tcpdump` command again to capture more information about the incoming traffic:

sudo tcpdump -i any -nn -s 0 -v udp port 4096

Pay attention to the destination IP addresses in the captured packets. If the packets are still being sent to the incorrect IP address (172.31.0.9), you should investigate why the sender (`stage` server) is using the wrong IP address. Check if there are any misconfigurations on the sender side or if any intermediate devices (e.g., routers or switches) are modifying the packets.I ran the test again and I see ```18:09:46.221204 IP (tos 0x0, ttl 253, id 11497, offset 0, flags [DF], proto UDP (17), length 29)
    54.160.180.240.40513 > 172.26.0.2.4096: UDP, length 1
18:09:46.221222 IP (tos 0x0, ttl 253, id 11497, offset 0, flags [DF], proto UDP (17), length 29)
    54.160.180.240.40513 > 172.26.0.2.4096: UDP, length 1
```It looks like the UDP traffic is now being sent to the correct container IP address (172.26.0.2) on port 4096.

However, if the Docker container is still not processing the traffic, there might be an issue with the application running inside the container. Check the logs of the application inside the container to see if there are any errors or unexpected behavior:

docker logs

Replace `<container_id>` with the ID of your container (e.g., `03489e1fa3bd`).

Also, verify that the application is listening for UDP traffic on port 4096. You can do this by running the following command inside the container:

netstat -tuln | grep 4096

If the application is not listening on the correct port, you may need to check the application configuration or settings to ensure it is set up to listen for UDP traffic on port 4096.

Additionally, ensure there are no firewalls or security groups blocking UDP traffic between the `stage` and `jvb2` hosts. Check the security group rules in your cloud provider's management console or any additional firewalls on your hosts.

Troubleshooting Docker UDP Traffic created at Tue Jan 20 1970 11:25:05 GMT+0000 (Coordinated Universal Time)