This article actually started with a question I had about search volume for two different HTTP servers: Apache and Nginx. So I loaded up Google Trends, started checking volume with a bunch of terms and found a fairly steep decline in search interest for the term 'Apache'.

But when I look at the modern web, that makes a lot of sense. That base knowledge in HTTP servers that we needed back in the day when I got started is still important, but it has been diffused into other layers. So instead of digging right into the underlying HTTP server, sometimes we have diffused that knowledge into things like Kubernetes access or a myriad of other tools by major vendors.

However, the basic debugging has stayed the same.

The Ancient Ways

Back in the day, commands like dig, curl and openssl were very important parts of a normal debugging workflow for a web application that wasn't behaving. If one of those commands actually showed you the problem, the problem was usually sitting at a pretty fundamental layer. But mostly, they showed you where to look.

Sometimes they might point to an entire LAMP stack on a box that runs WordPress. Other times they would point to an Nginx or Apache reverse proxy. And sometimes when you would dig into those addresses you would find the dragons.

But armed with a decent sense of how the overall application was constructed, armed with a few basic commands and a myriad of flags, a good developer could figure out where to look fairly quickly. From there, normal debugging would take over.

Understanding DNS

DNS (Domain Name System) is like the internet's phonebook. When I explained that to my child, they replied "what's a phonebook?" So crap, I had to rethink that all.

You know how if you say something like "Siri text dad Can you get me a new app" it looks me up and sends me a message? That's how DNS works. It takes human-readable addresses like hluska.ca and translates them into machine-readable IP addresses.

You can have a lot of different services running at a domain like hluska.ca. It's sometimes easier to visualize how this all works together if you think of how a farm works.

Pretend that you and your sibling own a farm. You have a large mixed operation with cattle and you farm a variety of crops. Within your farm you will have all the infrastructure for the crops you grow and the livestock you raise. You will also have things like storage, grain bins, feed bins, a mechanics shop to maintain your equipment or a welding shop.

Depending on a variety of factors, you may monetize the land in different ways. Maybe you are a great welder and your sibling can fix any engine that has ever run? You can run those services within the farm in buildings you have to provide services to your farm. And maybe you have signed contracts with oil companies providing access, exploration and some variation of mineral rights in exchange for royalties.

A domain is a lot like that. You can run email on it, host internal chats, host files or host any myriad of applications imaginable. The important thing is that DNS is not really an access control system. If I decide that supersecret-donotcomehereplease.hluska.ca should be accessible from anywhere on the public internet, I would need to publish a public record and point it at something that can be reached from the public internet. At that point, the name and the route are public enough that I should assume other people can find them.

So heck, maybe I'd put a tersely worded banner, like "Hey jerk, you're not allowed here unless you're authorized. In that case, you're not a jerk sorry" on the page. Or heck, even a login system. Or maybe I would have the presence of mind to keep it somewhere that is not reachable from the public internet at all because the internet is not the only network there is. That might mean putting it on a private LAN, behind a VPN, behind something like WireGuard or Tailscale, inside a private cloud network, or using internal DNS where the name only resolves from the networks that are supposed to know about it.

And so while DNS does not really account for the gate around a property, it can show you the published entrances. It does not show you the whole farm, but it gives you enough directions to start figuring out what operation sits behind the entrance you are using.

Modern debugging

Even though the surface has changed and it's gotten increasingly rare to actually SSH into a running Apache or Nginx server, the toolkit still applies. Basic DNS commands still provide insight into what is there, what should be there and how it is responding versus how it should be responding. Armed with those basic concepts, a developer with appropriate knowledge of how a system is put together can figure out where to look.

That's when stuff gets a little topsy turvy, but it always has been. As an example, a modern request might start at DNS, hit Cloudflare or another CDN, terminate TLS at the edge, pass through a WAF rule, forward to a load balancer, route through a Kubernetes ingress, hit an Nginx reverse proxy, land in an application container, call a database, enqueue a job and fetch media from object storage. If each layer has its own idea of logging, tracing, headers, IDs and failure messages, you no longer have one problem to solve, you have 31.

But while that sounds like a critique of a microservices based architecture, it really isn't. If I changed the scale slightly, I could describe a LAMP stack running WordPress on a single server in very similar terms. DNS points at the box, Apache handles the request, .htaccess rewrites it, PHP executes it, WordPress loads plugins and themes, the database answers some questions, the filesystem answers others, cron jobs may be involved and permissions may quietly ruin your afternoon. Lots of stuff can go wrong in a lot of different places and the logs may not necessarily correlate to each other.

Structured logging is a solution... but it's one with a significant implementation cost even if it's implemented from the very beginning. And it contains a significant code review and training burden because the moment one developer mixes up operation_id and op_id you're in a heap of trouble. Or rather, you were in that heap of trouble the moment that those two variables diverged. That technical debt will need to be collected. But the same arguments were made a long long time ago so this isn't anything new either.

Instead, while the tooling has changed dramatically and in many cases developers work with abstractions built upon technology, all the same debugging lessons about digging into requests and figuring out where the errors come from still work.

Conclusion

I have a feeling that I'll do more work in this area because I find it quite interesting. There's a real history to debugging web applications and while the abstractions have changed a lot, the basic tools to find which abstraction to look into really haven't.

It's also interesting because while the tooling has changed a lot, the same problems remain. Observability is always a minefield to the point that I should technically make this entire sentence invisible to get the point across. Structured logging is only as good as adherence to basic rules of the format, especially when it comes down to correlation.

But DNS and basic tools will still get you a long ways towards resolution because once you learn a little bit about them and relate that knowledge to what you know of your application, you can quickly locate where the problem happens.

The ancient ways still work because they were never really about Apache, Nginx or any specific server. They were about learning how to follow a request until reality stops matching the diagram.