Bind fails to start due to GeoIP

13 posts / 0 new
Last post
#1 Mon, 03/27/2017 - 15:29
havfrue
havfrue's picture

Bind fails to start due to GeoIP

We have a pretty standard Webmin running on Centos 7. I haven't checked when it began as we don't really use the BIND server (for now), but it can't start at the moment due to a weird GeoIP error - this error comes up in Webmin as well when trying to config BIND - it throws it from named --help as well.

named: symbol lookup error: named: undefined symbol: GeoIP_cleanup

We're not even using GeoIP in named - there's no references to it in the named.conf file.

I haven't been able to find anything about this anywhere. How do I debug this?

Mon, 03/27/2017 - 18:39
Diabolico
Diabolico's picture

Try this:
1. yum clean all
2. yum update
3. yum update bind-libs
4. restart bind and see if it works

If this doesnt help try with "yum upgrade bind-libs" and then try to restart bind.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Tue, 03/28/2017 - 02:41
havfrue
havfrue's picture

Thanks for the suggestions @Diabolico but that didn't change anything. Still getting the same error. I have no clue how to debug this.

Tue, 03/28/2017 - 13:22
Diabolico
Diabolico's picture

Hm ok, then before you go nuclear on Bind try this:
1. yum clean all
2. yum update bind-utils
3. yum update bind-libs
4. yum update
5. restart bind and see if it works

If this doesnt work then the problem could be inside named.conf file or something else:
1. Check the content of named.conf and see if everything is in order. Especially pay attention if you see any line containing geoip, if there is one try to comment out and restart Bind. If you really dont know anything about Bind copy the content of named.conf here so someone can take a look (please use forum code tags).
2. Check /var/log/yum.log(-*) and try to find Bind and what happened during the last update. If you see any errors during the update, revert back to previous version, restart Bind and then try to update again. Its advisable to run yum clean all before you attempt to update Bind.

Based on what you described i would say some libraries didnt properly update, but it could be something else.

EDIT: Please make local copy of any file you intend to modify BEFORE you proceed with modifications!

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Wed, 03/29/2017 - 16:11
havfrue
havfrue's picture

Tried all your suggestions, but nothing. Even tried reinstalling bind - nothing.

It's got nothing to do with the bind.conf - it doesn't even read it:

[{root} 13:30:22 /etc]# named -c 123
named: symbol lookup error: named: undefined symbol: GeoIP_cleanup
[{root} 13:48:27 /etc/yum.repos.d]# named -v
named: symbol lookup error: named: undefined symbol: GeoIP_cleanup

There's nothing in the yum log, but I only have it 5 weeks back, and it may have happened before then.

I'm perplexed. I looked at uninstalling Geo-IP, but it's listing bind, awstats, dhclient, webalizer and proftpd as dependencies, and I'm not going to uninstall those. I'll leave it for now - see if the problem solves itself down the road.

Thu, 03/30/2017 - 01:44
Diabolico
Diabolico's picture

Could you post the content of /etc/named.conf?

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Thu, 03/30/2017 - 12:26
havfrue
havfrue's picture

Sure. But it doesn't look like it's even reading it - the -c option should specify another .conf, but it still fails. It's a pretty standard setup. I removed the 3 slave zones in there earlier to see if that was the problem, so it's pretty clean now (IP address removed below on purpose).

//
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
        listen-on port 53 {
                any;
                };
        listen-on-v6 port 53 {
                any;
                };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
        allow-recursion { 127.0.0.1; 54.183.xxx.yyy; };
        allow-recursion-on { 127.0.0.1; 54.183.xxx.yyy; };
       
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";

        query-source address * port *;
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
    channel security_file {
        file "/var/log/named/security.log" versions 3 size 30m;
        severity dynamic;
        print-time yes;
    };
    category security {
        security_file;
    };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

I can see in /var/named/data that the last time it's worked was on Dec 12.

-rw-r--r--. 1 named named 277589702 Dec 12 13:23 named.run

This looks like it's coinciding with the Centos 7.3 upgrade which updated more-or-less everything.

Thu, 03/30/2017 - 19:08
Diabolico
Diabolico's picture

This is just a suggestion to make things much cleaner
Instead of using allow-recursion { 127.0.0.1; 54.183.xxx.yyy; }; ... try with:

acl "trusted" {
localhost;
127.0.0.1;
54.183.xxx.yyy;
...(list all IP's you need)...
};

options {
listen-on port 53 { any; };

/snipped/

allow-recursion { trusted; };
/snipped/


Did you mask this part query-source address * port *;? I'm asking because Bind uses ports bigger than 1024 to query other nameservers and this line have meaning only if some service will expect this queries on port 53 or some other uncommon ports lover than 1024, but usually if something requires port lower than 1024 it will be port 53.

For the rest i cant see anything wrong with your named.conf and now i suspect even more that last update could break some Bind libraries. Could you try to downgrade your Bind to previous version and see if it works? If the last update was the culprit for Bind to stop working downgrading could repair things. Other solution could be to force reinstall Bind yum reinstall bind. Use both suggestions with caution.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Thu, 03/30/2017 - 20:23
havfrue
havfrue's picture

I didn't mask it. Not sure where it came from - I've removed it. I don't think anyone have edited it - the server is ~3 years old now.

Anyway - I tried downgrading bind - as far back as I could, and I'm still getting the same problem: named: symbol lookup error: named: undefined symbol: GeoIP_cleanup It doesn't read a single parameter before exiting.

This is the bind that's now installed with the same problem. It's a Nov 11 version, which did work at the time, so it indicates that bind it not even the problem.

bind.x86_64                                                     32:9.9.4-37.el7                                             @base 
bind-libs.x86_64                                                32:9.9.4-37.el7                                             @base 
bind-libs-lite.x86_64                                           32:9.9.4-37.el7                                             @base 
bind-license.noarch                                             32:9.9.4-37.el7                                             @base

Something else has fubar'ed it. Oddly enough GeoIP itself was not part of the Dec 12 update, so that isn't what's causing it.

Thu, 03/30/2017 - 21:40
Diabolico
Diabolico's picture

This line query-source address * port *; was never part of the default Bind settings so someone put it there.

For the rest: I'm not sure Vmin is the cause here and looks like this is different than already reported bug, even that bug is pretty old and probably already patched. To be honest i'm out of ideas what could cause this problem. At this point i can only suggest to pay Vmin or SysAdmin to check your server, because someone should really take deeper look at what is going on. Other solution would be to go nuclear on Bind (and GeoIP), remove them and try to install again. To remove any package without dependencies use rpm -e --nodeps /package_name/. Again this isnt 100% safe and it could cause additional problems to pop out. You should backup anything you can before this attempt, especially your website(s).

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Fri, 03/31/2017 - 16:56
cruiskeen

You sure you're using the standard BIND executables from CentOS? This sounds for all the world like you have a copy of named that was built with GeoIP support Which implies that it's bind 9.10, which isn't part of CentOS or RedHat --- so my suspicion is that you have an oddball copy of bind. What does rpm -q -a |grep bind say?

Oh -- never mind, you more or less answered that above. I'm outta ideas.

Do you have GeoIP installed?

Sat, 04/01/2017 - 00:23
Diabolico
Diabolico's picture

If he didnt manually install/compile some software i'm pretty sure it would not install by accident, even with automatic updates. In case he did then this is something the person would remember. I think something went wrong during the last (?) Bind update, but he doesnt have anymore the log files containing Bind update and if he had enabled automatic updates (what should be big No) who knows what happened.

- I often come to the conclusion that my brain has too many tabs open. -
Failing at desktop publishing & graphic design since 1994.

Sat, 04/01/2017 - 13:45
havfrue
havfrue's picture
[{root} 11:38:51 /var/log]# rpm -q -a | grep bind
bind-9.9.4-38.el7_3.2.x86_64
bind-license-9.9.4-38.el7_3.2.noarch
rpcbind-0.2.0-38.el7.x86_64
bind-libs-9.9.4-38.el7_3.2.x86_64
bind-libs-lite-9.9.4-38.el7_3.2.x86_64

I've tried downgrading them as mentioned above - no difference. It should be bog-standard bind - especially since I've reinstalled it.

We don't have automatic updates but do run them frequently manually. :)

I'm not going to bother spending more time on it - I give up - I'll reinstall on a fresh VM later instead. Easier. I'll increase our log storage time in the future though.

Topic locked