Ghost of an IP past...driving us NUTS!

[OldManEmu]

Once upon a time in the recent past I was running a Ubuntu web server inside my network for dev purposes. At that time I assigned the server an internal IP of 10.0.1.151. Port 80 was briefly opened on the firewall to allow beta testing.

I have since moved this server to an offsite location for deployment. However we can now not reach our website from inside our network because the domain is resolving internally to the old IP of 10.0.1.151. Outside our network there is no problem, but inside we can only reach the site using it's new IP address.

The hostname must be resolving to a hosts or known_hosts file on one of the machines in our network, but we can't find it anywhere!

Anyone who wishes to play "IP Ghostbuster" with us please offer your suggestions.

NOTE: I run 4 mail servers inside my network, all for different domains, none for the domain in question. Notice that under the second "ANSWER SECTION" it's referencing one of my internal mail server domains as "mail.differentdomain.com" and not "mail.mydomain.com". This may be a clue but I can find no reference to "mydomain" on that mail server.

Here is the output of a "host -v mydomain.com" lookup from terminal inside the network:

Trying "mydomain.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2829
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;mydomain.com. IN A

;; ANSWER SECTION:
mydomain.com. 10800 IN A 10.0.1.151

;; AUTHORITY SECTION:
mydomain.com. 10800 IN NS mail.differentdomain.com.

;; ADDITIONAL SECTION:
mail.differentdomain.com. 10800 IN A 10.0.1.3

Received 92 bytes from 10.0.1.3#53 in 13 ms
Trying "mydomain.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61898
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;mydomain.com. IN AAAA

;; AUTHORITY SECTION:
mydomain.com. 10800 IN SOA mydomain.com. admin.mydomain.com. 2013042701 86400 3600 604800 10800

Received 74 bytes from 10.0.1.3#53 in 3 ms
Trying "mydomain.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12488
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;mydomain.com. IN MX

;; AUTHORITY SECTION:
mydomain.com. 10800 IN SOA mydomain.com. admin.mydomain.com. 2013042701 86400 3600 604800 10800

Received 74 bytes from 10.0.1.3#53 in 2 ms

 

[OldManEmu]

450 reads and no replies?

Update:

Testing a hypothesis last week I unplugged the ethernet from each internal mail server one by one doing a host lookup each time. I have found that when one of the mail servers is offline mydomain resolves to the correct external IP.

Now that I know where the issue is coming from, does anyone have a suggestion where on the server I should look to remove this old IP and why my network is resolving mydomain to this one internal server? I've tried all the usual places...it's gotta be in one of the .conf files but I'm at a loss.

 

[satcomer]

I would start looking in the DNS or Hostnames file on the server.

 

[chown33]

Look carefully at who is supplying the DNS reply. Your DNS server at 10.0.1.3 is providing the posted information. Whether you realize it or not, you're running a DNS server at that IP address, and your originating host is asking that server for lookups.

These are in the output you posted:

Received 92 bytes from 10.0.1.3#53 in 13 ms
Received 74 bytes from 10.0.1.3#53 in 3 ms
Received 74 bytes from 10.0.1.3#53 in 2 ms
​

The #53 is port 53, which is the well-known port for DNS.

You can change the DNS server for the host doing the lookup, so it uses a server with correct info, or you can correct the server at 10.0.1.3 providing the reply.

When you take the mail server hosts down, that will also take down a DNS server running on the same host. Maybe the DNS server running on the mail-server host is unintended. If so, figure out what's starting the DNS server and stop the DNS server from starting up.