Under Yosemite (and El Capitan), the launch launchctl utility changed.

For dnsmasq:

1. brew install dnsmasq or brew upgrade dnsmasq
2. sudo cp -fv /usr/local/opt/dnsmasq/*.plist /Library/LaunchDaemons
3. In /usr/local/etc/dnsmasq.conf add the line address=/devel/127.0.0.1 (do NOT add dev as this is now a top-level TLD)
4. sudo launchctl bootstrap system /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
5. sudo launchctl enable system/homebrew.mxcl.dnsmasq
6. sudo launchctl kickstart -kp system/homebrew.mxcl.dnsmasq
7. launchctl print | grep dnsmasq, you should see an entry for DNSMasq with a pid number (not a -, which means it isn't running)
8. In System Preferences -> Network -> Advanced -> DNS, add an entry for 127.0.01 and move it to the top of the list
9. dig fancydomain.devel, you should get an immediate response from the local host

NOTE: You can also create /etc/resolver/devel with contents nameserver 127.0.0.1 to avoid using System Preferences. NOTE: If you use VPN, this setup will likely change every time you connect/disconnect. You'll have to add the search domain manually to your connections.

For nginx:

1. Install and configure Nginx
2. sudo cp /usr/local/opt/nginx/*.plist /Library/LaunchAgents
3. sudo launchctl bootstrap system /Library/LaunchAgents/homebrew.mxcl.nginx.plist
4. sudo launchctl enable system/homebrew.mxcl.nginx
5. sudo launchctl kickstart -kp system/homebrew.mxcl.nginx
6. launchctl print | grep nginx, you should see an entry for Nginx with a pid number (not a -, which means it isn't running)

Note that for both services, you need to bootstrap and enable the service before starting them via kickstart. If you receive errors that launchctl cannot find a service in the specified domain, it's probably because it was not bootstrapped. If you see a pid number of zero, then Nginx did not successfully start and likely has a configuration error.

UPDATE 2019:

Do not use load or unload anymore, as they are deprecated. Additionally, some of the semantics of starting services were updated to the following (for a per-user daemon):

1. Get your user identifier with id -u.
2. launchctl bootstrap user/<your-uid> file.name.of.service.plist
3. launchctl kickstart -k system/<your-uid>/label.name.of.service
4. To stop service: launchctl bootout user/<your-uid> file.name.of.service.plist