Mac OS X Lion, /etc/hosts Bugs, and DNS Resolution

Final Update: I’m working on a follow-up post, but thanks to all the comments, it seems that somehow during the Snow Leopard to Lion upgrade, the /etc/hosts file was encoded incorrectly (likely the type of newlines used), and that Lion would fail to read it properly. Please try to copy the contents of the /etc/hosts file, recreate the file, and paste to fix your problem.

Update: I was still having problems with using .local for my development sites. After some extensive debugging and research I’ve found out some interesting things. It appears that Multicast DNS (mDNS), which is used in Apple’s Bonjuor for network discovery without a tradition DNS Server, has the Top Level Domain .local reserved for special mDNS functionality. Mac OS X will treat domains ending with .local differently! Use another TLD, like .dev, and it will work correctly.

This article will still help you fix problems with /etc/hosts abnormalities with it not having the correct priority.

Update 2: Thanks for all the comments! A few updates: First off, per the comments, if you declare both a IPv4 and IPv6 address for a .local domain like so:

27.0.0.1 domain.local
::1 domain.local

Then it will work correctly. Second, I ended up doing a fresh install of Lion and this problem has gone away for me. So it appears that this issue would affect some people who upgraded from Snow Leopard. So as Chris suggested below, I don’t think it has to do with the resolve order. I haven’t been able to figure out the difference between Lion installs that work correctly, and those that don’t. So if you are have the following problems, the suggestions below will work to fix it.

This morning I was up a few hours earlier than normal today, and was looking forward to getting in some solid programming hours earlier so I wouldn’t work late tonight. Instead, I spent about four hours fighting a new change (I’d call it a bug) in Mac OS X Lion.

The /etc/hosts File

Now, instead of running a full blown DNS server on my machine, which is overkill and more work to edit and maintain, I would make manual adjustments to my /etc/hosts file. For those not familiar with the /etc/hosts file, it allows you to add entries that traditionally your computer will look up first before trying your server DNS. This is where your computer would set “localhost” is the IP “127.0.0.1″.

I use this a lot with local development. For example, when I work on joind.in, I use MAMP and create a VirtualHost for the code under the server name local.joind.in (i.e. http://local.joind.in/). I don’t have access to the joind.in DNS, so I create an entry in my /etc/hosts file. You have to have admin permissions to edit the file, so I normally run the command “sudo nano /etc/hosts” and make my changes and then save.

The /etc/hosts file looks something like this:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost
fe80::1%lo0    localhost

127.0.0.1       local.joind.in
209.114.40.229  www.example.com

Now, my actual /etc/hosts file contains 83 entries. Some are just short cuts to machines I often connect to. Sometimes during a migration when I’m making DNS entries, I can make the change in my /etc/hosts file as well, so I don’t have to wait for the DNS changes to propagate to test. All in all, it is a very handle little file that I use a great deal. Even windows has their own hosts file that works well.

Problem #1: Bugs with Multiple Domains per Line

With the new Lion Update comes better support for IPv6, however this seems to have broken the /etc/hosts file from allowing multiple domains per line. Before you could do something like this, which is works in several other operating systems:

127.0.0.1 local.joind.in local.example.com

However, this now seems to break and not work. You must keep each line to one ip and one domain. So your configuration should look like this:

127.0.0.1 local.joind.in
127.0.0.1 local.example.com

There was also reports about the order of IPv4 and IPv6 entries (blog post about it). I personally did not have this problem, but others seem to have had it. The problem was that OS X Lion seemed to ignore IPv4 entries after a IPv6 entry was made.

Problem #2: DNS Resolution Orders has DNS Servers Before the Hosts File

This is the problem that really, really ticked me off and caused me to waste hours and hours trying to debug and find a work-around. I was working on my local copy of Clipish, and because we have different domains for it that work together, I replace .com with .local. So I would have something like this in my /etc/hosts file:

127.0.0.1 staff.example.local
127.0.0.1 images.example.local
127.0.0.1 www.example.local

However, anytime I made a request in Chrome to any of those domains, it would take 5.01 seconds. At first I thought it might be a bug with the new version of MAMP for OS X Lion. But I found using Chrome’s & FireFox’s developer tools that it was taking 5 seconds for the DNS lookup, and 0.01 seconds for the rest. So I would go into my terminal and ping the domain. It worked just fine without the delay.

Apparently, the terminal and some of the BSD Unix tools correctly use /etc/resolv.conf and the correct order of /etc/hosts first and then DNS servers. However, everything else on OS X Lion, including all of your Applications, do it backwards! This wasn’t the case in Tiger, Leopard, or Snow Leopard.

You can test this by running the command “scutil –dns”:

Justin-Carmonys-MacBook-Pro-2:etc justin$ scutil --dns
DNS configuration

resolver #1
  nameserver[0] : 8.8.8.8
  nameserver[1] : 192.168.1.1

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 127.0.0.1
  if_index : 5 (en1)
  flags    : Scoped

I spent hours and hours googling, finding a bunch of people with this same problem, but no solution on how to fix the ordering. I have my suspicions why Apple made this change, which I’ll address later on. But after I couldn’t figure out a way to change the resolve order myself, I had to find another alternative. (Of course, OS X removed /etc/nsswitch.conf so I can’t change it there, grrr!)

Solution: DNSMasq

The only solution we have, that I could find, is to use your own DNS Server locally on your own machine. I’ve heard of others setting up BIND on OS X. Now, the problem with this solution is that it isn’t nearly as quick and easy to manage DNS entries.

Then, I found out about DNSMasq. It is a lightweight DNS server that is easy to configure. However, it had one key feature we needed: it will read the /etc/hosts file and serve it’s entries. This means we can have our OS check the DNSMasq instance first, which will check the /etc/hosts file, restoring the proper order.

Installing DNSMasq

After some Googling around, I think installing dnsmasq using MacPorts is the easiest. First, you’ll need to install Xcode so your machine can compile with the tools. It is free on the Mac App Store (warning, it is a kind of big download). Then, you’ll need to download and install the MacPorts for Lion. If you already have MacPorts installed, make sure it is the current version of Lion.

Once it is installed, open up the terminal and then execute these commands. First, run selfupdate to make sure your MacPorts repositories are up to date:

sudo port selfupdate

Then, install dnsmasq:

sudo port install dnsmasq

That should execute with no problems. Now, it is installed, but we need to have OS X run it:

sudo port load dnsmasq

Now, if you want to make some changes to your dnsmasq configurations, you can do that in this file:

/opt/local/etc/dnsmasq.conf

One thing I like to do is control which DNS Server’s dnsmasq will use. By default it will use /etc/resolv.conf, but OS X will overwrite that file on reboot and anytime you change internet connections (like switching wifi hotspots). So I would copy /etc/resolv.conf to /etc/resolv.dnsmasq.conf. When dealing with files in /etc/ and /opt/ you’ll need to run these commands using sudo.

sudo cp /etc/resolv.conf /etc/resolv.dnsmasq.conf
sudo nano /etc/resolv.dnsmasq.conf

Then I set dnsmasq to read my new resolve.dnsmasq.conf by setting “resolv-file=/etc/resolv.dnsmasq.conf” in /opt/local/etc/dnsmasq.conf.

Now, we need to configure OS X to add 127.0.0.1 to our DNS Servers. Go to System Preferences > Network > (Select Connection, i.e. Wi-Fi) > Advanced > DNS.

You can have other DNS entries, like Google’s DNS Entries (8.8.8.8, 8.8.4.4). Just make sure 127.0.0.1 is on top.

One thing I haven’t been able to figure out is how to reboot dnsmasq, but if you kill the process in the Activity Monitor OS X will restart it:

This is a lot of work to setup (especially if you don’t have Xcode install). However, it will work, and give you a lot more control over your DNS. So in a way it is an improvement! Just wish it wasn’t necessary.

If you would like to do wildcard dns entries, something you couldn’t do in /etc/hosts, then you can add this to your dnsmasq config:

address=/.local/127.0.0.1

Now any domains ending in .local will route to 127.0.0.1

Why is this necessary Apple?

What I don’t get, is why this change Apple? No other operating system that I know if, including Windows for crying out loud, does it this way. This had to be a deliberate change by Apple. I can only see one of the following reasons:

Security – Maybe, just maybe, it was for security. Technically, a virus can overwrite the /etc/hosts file if it the virus can prompt the user for their password to run a command as root. So you could maybe have a user visit www.google.com when in reality it isn’t a google server. But even then, HTTPS wouldn’t work without ugly errors. So this might be a reason, but out of all the things that can go wrong with a virus, it is a poor excuse. The virus could just hack mDNSresponder to use /etc/hosts.
Prevent User Hacks – Most likely, I think this is why Apple made this change. I know with jailbreaking iOS devices and getting hacked apps, iTunes will try to validate an App by “phoning home” to Apple. However, if you change the entries in your /etc/hosts file so iTunes can’t reach the correct Apple Servers. I know people who have circumvented other software activation systems do the same technique. I think this is why Apple made the change, though at the expensive of all the professionals who actually use /etc/hosts for work.
An Honest Bug – It is possible this was a bug, but seeing as how you would have to deliberately change the resolve orders for the operating system, I doubt this is the case.

If this was an intended change by Apple, it is a frustrating one. One of my favorite things is that underneath the OS X’s nice UI and Applications is a BSD Unix system. I love how I can compile and install so many things just like I would on Ubuntu. But if they continue to change that Unix underbelly and make it less and less like it is everywhere else, then I think they’ll be frustrating many, many professionals.

Hopefully in the future I’ll find a way to change the resolve order in Mac OS X, but until this, this is how I worked around the problem.