How to Configure Custom Nameservers for Your Domain and VPS

To configure custom nameservers for your domain and VPS, you need to register private nameservers (such as ns1.yourdomain.com) with your domain registrar, map them to your VPS IP address in your DNS zone file, and ensure your server runs a reliable DNS daemon like BIND or PowerDNS. This setup replaces generic host nameservers with your own branded infrastructure.

For website owners, agencies, and developers, relying on default hosting nameservers can limit branding and flexibility. By configuring custom nameservers for your VPS, you take full control of your Domain Name System (DNS) layer. This comprehensive guide explains what custom nameservers are, why you should use them, and how to configure them step-by-step.

Understanding Nameservers and DNS Basics

The Domain Name System functions as the phonebook of the internet. It translates human-readable domain names (like bukabyte.me) into numerical IP addresses that computers use to identify each other on a network. Nameservers are specialized servers that store these DNS records and answer queries from visitors’ browsers.

n

n

When you buy a domain name, your registrar assigns default nameservers to it. If you purchase web hosting, your provider typically asks you to point your domain to their shared nameservers. However, when you upgrade to a Virtual Private Server (VPS), you often manage your own software stack. Setting up custom nameservers allows your domain to use your VPS as its primary authority for DNS queries.

Why Set Up Custom Nameservers for Your VPS?

Transitioning to private nameservers offers several operational and strategic advantages:

n

n

• Brand Consistency: Instead of showing a hosting provider’s branding in your nameservers (e.g., ns1.somehostingprovider.com), your clients and visitors see your own domain (e.g., ns1.yourdomain.com). This is especially valuable for web agencies and resellers.
• Centralized Control: Managing your DNS directly on your VPS lets you handle zone files programmatically or through a control panel without logging into third-party registrar dashboards.
• Seamless Migrations: If you change your underlying hosting infrastructure or move your VPS to a new provider, you only need to update the IP addresses attached to your custom nameservers at the registrar level, rather than updating every single domain’s nameservers individually.

Prerequisites Before You Begin

Before diving into the configuration steps, ensure you have the following:

• A registered domain name.
• Root or administrative access to your VPS running a Linux distribution (such as Ubuntu, Debian, or AlmaLinux).
• A static public IPv4 address assigned to your VPS. (IPv6 addresses can also be added, but starting with IPv4 is standard).
• A DNS server package installed on your VPS, such as BIND9, or a hosting control panel (like cPanel, CyberPanel, or HestiaCP) that manages DNS out of the box.

Step 1: Create Glue Records at Your Domain Registrar

A glue record is a combination of a hostname (like ns1.yourdomain.com) and an IP address (like 192.0.2.1) registered directly with your domain’s top-level registry. Because your nameservers use your own domain name, the internet needs to know the direct IP address of the nameserver before it can resolve the domain itself. This prevents a classic “chicken-and-egg” DNS loop.

1. Log into the dashboard of the domain registrar where you purchased your domain.
2. Navigate to the domain management section and look for options labeled “Private Nameservers”, “Register Nameserver”, or “Glue Records”.
3. Add your first nameserver (e.g., ns1.yourdomain.com) and assign your VPS public IP address to it.
4. Add your second nameserver (e.g., ns2.yourdomain.com) and assign the same or a secondary VPS IP address to it. (Note: Many registrars require at least two distinct nameservers for redundancy, even if they point to the same VPS IP address).

Step 2: Configure Your VPS Firewall and DNS Software

With your glue records registered, your VPS must be configured to listen for and respond to incoming DNS queries. DNS queries primarily use UDP and TCP port 53.

Configuring the Firewall

Ensure that your server firewall permits traffic on port 53. If you are using UFW on Ubuntu, run:

sudo ufw allow 53/tcp
sudo ufw allow 53/udp
sudo ufw reload

Installing and Configuring BIND9 (Manual Setup)

If you manage your VPS manually without a control panel, BIND9 is the most widely used DNS server on Linux.

1. Install BIND9 using your package manager: sudo apt update && sudo apt install bind9 bind9utils -y
2. Edit the main configuration file (/etc/bind/named.conf.options) to allow queries from external networks and configure forwarders if necessary.
3. Define your domain zone in /etc/bind/named.conf.local:

zone "yourdomain.com" {
    type master;
    file "/etc/bind/db.yourdomain.com";
};

4. Create your zone file at /etc/bind/db.yourdomain.com, ensuring you include A records for your custom nameservers:

$TTL 86400
@   IN  SOA ns1.yourdomain.com. admin.yourdomain.com. (
        2023102401 ; Serial
        3600       ; Refresh
        1800       ; Retry
        604800     ; Expire
        86400 )    ; Minimum TTL

@   IN  NS  ns1.yourdomain.com.
@   IN  NS  ns2.yourdomain.com.

ns1 IN  A   YOUR_VPS_IP
ns2 IN  A   YOUR_VPS_IP

@   IN  A   YOUR_VPS_IP
www IN  A   YOUR_VPS_IP

5. Test your configuration for syntax errors using named-checkconf and restart the service: sudo systemctl restart bind9.

Tip: If you use a control panel like cPanel, CyberPanel, or aaPanel, you can skip editing configuration files manually. These platforms provide automated interfaces to create nameservers and generate zone files instantly.

Step 3: Update Your Domain’s Active Nameservers

Now that your registrar knows about your glue records and your VPS is ready to answer DNS requests, you need to tell your domain to use your new custom nameservers.

1. Return to your domain registrar’s control panel.
2. Find the “Nameservers” or “Manage Nameservers” setting for your domain.
3. Change the setting from default registrar nameservers to “Custom Nameservers”.
4. Enter your newly created hostnames: ns1.yourdomain.com and ns2.yourdomain.com.
5. Save your changes.

Step 4: Verify Your Configuration

DNS changes do not happen instantly worldwide due to global caching (known as DNS propagation). Propagation typically takes between 2 to 24 hours, though it often resolves much faster.

You can verify your setup using command-line tools like dig or online lookup utilities:

dig ns yourdomain.com +trace

This command queries the root servers and traces the path down to your custom nameservers, confirming whether your VPS is successfully recognized as the authoritative source.

Best Practices and Troubleshooting

• Maintain Redundancy: Whenever possible, assign your custom nameservers to two different VPS instances or utilize a secondary external DNS provider to ensure high availability if one server goes offline.
• Check PTR Records (Reverse DNS): Ensure your VPS provider has configured a valid Reverse DNS (PTR) record pointing your IP address back to your domain. This improves email deliverability and server reputation.
• Double-Check Syntax: A single missing period in a zone file can break DNS resolution. Always run validation checks before restarting your DNS service.

Conclusion

Configuring custom nameservers for your domain and VPS gives you ultimate control over your web infrastructure, strengthens your professional branding, and simplifies multi-domain management. By carefully registering your glue records, setting up your server’s DNS daemon, and updating your registrar settings, you establish an independent and robust hosting environment.

Explore reliable hosting and domain services from Bukabyte.

Securing Your Custom DNS Server Against Amplification Attacks

Running your own DNS server on a Virtual Private Server introduces distinct security responsibilities. Open public resolvers are frequently targeted by malicious actors seeking to launch Distributed Denial of Service (DDoS) attacks, specifically DNS amplification and reflection attacks. If your BIND9 or PowerDNS instance is misconfigured to accept recursive queries from anyone on the internet, your server can be hijacked to flood third-party networks.

To prevent your VPS from becoming an open resolver, you must strictly limit recursion permissions in your DNS configuration. In BIND9, navigate to your /etc/bind/named.conf.options file and locate the recursion directive. You should set this to no for public queries, or explicitly define trusted internal subnets if your applications require local recursion:

options {
    directory "/var/cache/bind";

    recursion no;
    allow-query { any; };
    allow-recursion { localhost; };

    dnssec-validation auto;

    listen-on { any; };
    listen-on-v6 { any; };
};

By setting allow-query { any; };, your custom nameservers remain publicly accessible to answer authoritative queries for your specific domains (like yourdomain.com), while allow-recursion { localhost; }; blocks external clients from using your server to query external websites like google.com. Always restart your DNS daemon after applying security updates.

Automating DNS Record Management with Control Panels

While manual text-file editing in BIND teaches you the foundational mechanics of the Domain Name System, maintaining dozens or hundreds of zones manually becomes cumbersome. Many webmasters and small agencies prefer deploying lightweight hosting control panels on their VPS to streamline both web hosting and custom nameserver administration.

Modern open-source and commercial control panels—such as HestiaCP, CyberPanel, or aaPanel—automate the creation of glue records, BIND zone files, and SSL certificates for your hostname services. When you add a new client domain inside these panels, the system automatically appends the correct SOA, NS, A, and MX records pointing back to your designated ns1 and ns2 prefixes.

If you choose to use a control panel for your custom nameservers:

• Ensure the panel’s internal IP settings match your VPS public IPv4 address precisely during initial installation.
• Verify that the automated firewall rules created by the panel do not inadvertently block TCP and UDP port 53.
• Back up your panel’s configuration databases regularly so your DNS zone maps are preserved during server restoration tasks.