bind/SOURCES/bind-9.11-serve-stale.patch

From d55a57427ee696dec51149950478394e43019607 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Petr=20Men=C5=A1=C3=ADk?= <pemensik@redhat.com>
Date: Thu, 7 Nov 2019 14:31:03 +0100
Subject: [PATCH] Implement serve-stale in 9.11
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Squashed commit of the following:

commit 32f47f36e545223b2a4757588d7bd4af8c5f5760
Author: Petr Menšík <pemensik@redhat.com>
Date:   Tue Sep 3 18:45:54 2019 +0200

    convert serve_stale to db_test

    Manual checkout from commit e8f61dd315c5d1c88915bb79361182241e42e47a.
    Use test modified for cmocka, including serve-stale tests.

commit 071eb1fb0786f6d614955813d99c3caabff33383
Author: Michał Kępień <michal@isc.org>
Date:   Fri Apr 27 09:13:26 2018 +0200

    Detect recursion loops during query processing

    Interrupt query processing when query_recurse() attempts to ask the same
    name servers for the same QNAME/QTYPE tuple for two times in a row as
    this indicates that query processing may be stuck for an indeterminate
    period of time, e.g. due to interactions between features able to
    restart query_lookup().

    (cherry picked from commit 46bb4dd124ed031d4c219d1e37a3c6322092e30c)

commit c12090bc361c7fa4522ace73899e778e44e9b295
Author: Petr Menšík <pemensik@redhat.com>
Date:   Mon Sep 2 11:12:32 2019 +0200

    Fix test name used in whole test-suite

    Correct name is serve-stale

commit ff4d826f295d268a248ca06941d65c903e1b405c
Author: Petr Menšík <pemensik@redhat.com>
Date:   Fri Aug 30 17:43:28 2019 +0200

    Clean files in more generic rules

commit 8d81ed15eda9a2a11e1433d1fdddacfc772708b6
Author: Petr Menšík <pemensik@redhat.com>
Date:   Thu Aug 29 21:27:57 2019 +0200

    [rt46602] Pass port numbers to tests via environment variables

    Manually applied commit f5d8f079008b648d2e343543e66dd728054c6101

commit 94fafa477891576286def8c4041ad127734af2d1
Author: Tony Finch <dot@dotat.at>
Date:   Tue Apr 10 16:17:57 2018 +0100

    Move serve-stale logging to its own category, so that its verbosity can be curtailed.

    (cherry picked from commit 4b442c309dfb2c8880b19af4133047655bb734df)

commit e0c884bee98c3d2533dfaa667f58c6a80d8a3a00
Author: Michał Kępień <michal@isc.org>
Date:   Fri Apr 27 09:13:26 2018 +0200

    Prevent check_stale_header() from leaking rdataset headers

    check_stale_header() fails to update the pointer to the previous header
    while processing rdataset headers eligible for serve-stale, thus
    enabling rdataset headers to be leaked (i.e. disassociated from a node
    and left on the relevant TTL heap) while iterating through a node.  This
    can lead to several different assertion failures.  Add the missing
    pointer update.

    (cherry picked from commit 391fac1fc8d2e470287b5cc4344b3adb90c6f54a)

commit d724cc1d80ee8d46113eaf82549d49636739b67c
Author: Matthijs Mekking <matthijs@isc.org>
Date:   Thu Jan 24 10:24:44 2019 +0100

    Print in dump-file stale ttl

    This change makes rndc dumpdb correctly print the "; stale" line.
    It also provides extra information on how long this data may still
    be served to clients (in other words how long the stale RRset may
    still be used).

    (cherry picked from commit 924ebc605db798e2a383ee5eaaebad739e7c789c)

commit 625da4bd4590ac6108bb30eddd23ceffb245ae49
Author: Michał Kępień <michal@isc.org>
Date:   Mon Oct 22 15:26:45 2018 +0200

    Check serve-stale behavior with a cold cache

    Ensure that serve-stale works as expected when returning stale answers
    is enabled, the authoritative server does not respond, and there is no
    cached answer available.

    (cherry picked from commit 27cfe83a388147edfa0451b28c06c746912ea684)

commit d67ae10461c409fdafdbbe64f857db2552b71059
Author: Michał Kępień <michal@isc.org>
Date:   Mon Oct 22 15:26:45 2018 +0200

    Check TTL of stale answers

    Make sure that stale answers returned when the serve-stale feature is
    enabled have a TTL matching the value of the stale-answer-ttl setting.

    (cherry picked from commit 893ab37ce78c658215bd3a019f25afe795b37d5a)

commit 50459107805e68e4a63a8e497bf58ef3ce013ddb
Author: Michał Kępień <michal@isc.org>
Date:   Mon Jul 9 14:35:12 2018 +0200

    Do not use Net::DNS::Nameserver in the "serve-stale" system test

    Net::DNS versions older than 0.67 respond to queries sent to a
    Net::DNS::Nameserver even if its ReplyHandler returns undef.  This makes
    the "serve-stale" system test fail as it takes advantage of the newer
    behavior.  Since the latest Net::DNS version available with stock
    RHEL/CentOS 6 packages is 0.65 and we officially support that operating
    system, bin/tests/system/serve-stale/ans2/ans.pl should behave
    consistently for various Net::DNS versions.  Ensure that by reworking it
    so that it does not use Net::DNS::Nameserver.

    (cherry picked from commit c4209418a50c09142375f7edadca731c526f3d3a)

commit 4b5befc714bb386bd245b1c14ce3bce5ae6fb5fa
Author: Petr Menšík <pemensik@redhat.com>
Date:   Tue Jun 5 21:38:29 2018 +0200

    Fix server-stale requirement, skip without Time::HiRes

    (cherry picked from commit 7a0c7bf9c8e6a724e52635eed213ad25b9504e66)

commit 5ce51a3a7e5ef3087c4d022e3fca42fb2fd0c996
Author: Ondřej Surý <ondrej@sury.org>
Date:   Wed Oct 18 13:01:14 2017 +0200

    [rt46602] Update server-stale test to run on port passed from run.sh script

    (cherry picked from commit f83ebd34b9555a5a834c58146035173bcbd01dda)

commit 3954a9bf3437f6fab050294a7f2f954a23d161ec
Author: Ondřej Surý <ondrej@sury.org>
Date:   Wed Oct 18 14:18:59 2017 +0200

    [rt46602] Add serve-stale working files to .gitignore

    (cherry picked from commit cba162e70e7fac43435a606106841a69ce468526)

commit 112aa21f5fa875494820e4d1eb70e41e10e1aae7
Author: Mark Andrews <marka@isc.org>
Date:   Thu Oct 12 15:33:47 2017 +1100

    test for Net::DNS::Nameserver

    (cherry picked from commit 5b60d0608ac2852753180b762d1917163f9dc315)

commit 9d610e46af8a636f44914cee4cf8b2016054db1e
Author: Mark Andrews <marka@isc.org>
Date:   Thu Oct 12 15:19:45 2017 +1100

    add Net::DNS prerequiste test

    (cherry picked from commit fa644181f51559da3e3913acd72dbc3f6d916e71)

commit e4ea7ba88d9a9a0c79579400c68a5dabe03e8572
Author: Mark Andrews <marka@isc.org>
Date:   Wed Sep 6 19:26:10 2017 +1000

    add quotes arount $send_response

    (cherry picked from commit 023ab19634b287543169e9b7b5259f3126cd60ff)

commit 0af0c5d33c2de34da164571288b650282c6be10a
Author: Mark Andrews <marka@isc.org>
Date:   Thu Nov 23 16:11:49 2017 +1100

    initalise serve_stale_ttl

    (cherry picked from commit 2f4e0e5a81278f59037bf06ae99ff52245cd57e9)

commit fbadd90ee81863d617c4c319d5f0079b877fe102
Author: Evan Hunt <each@isc.org>
Date:   Thu Sep 14 11:48:21 2017 -0700

    [master] add thanks to APNIC and add missing note for serve-stale

commit deb8adaa59955970b9d2f2fe58060a3cbf08312b
Author: Mark Andrews <marka@isc.org>
Date:   Wed Sep 6 12:16:10 2017 +1000

    silence 'staleanswersok' may be used uninitialized in this function warning. [RT #14147

commit 0e2d03823768dc545015e6ce309777210f4a9f85
Author: Petr Menšík <pemensik@redhat.com>
Date:   Thu Aug 29 19:57:58 2019 +0200

    More fixes to merge

commit 360e25ffe7623ea0a2eec49395001f4940967776
Author: Mark Andrews <marka@isc.org>
Date:   Wed Sep 6 09:58:29 2017 +1000

    4700.   [func]          Serving of stale answers is now supported. This
                            allows named to provide stale cached answers when
                            the authoritative server is under attack.
                            See max-stale-ttl, stale-answer-enable,
                            stale-answer-ttl. [RT #44790]

Signed-off-by: Petr Menšík <pemensik@redhat.com>
---
 bin/named/config.c                            |   9 +-
 bin/named/control.c                           |   2 +
 bin/named/include/named/control.h             |   1 +
 bin/named/include/named/log.h                 |   1 +
 bin/named/include/named/query.h               |  15 +
 bin/named/include/named/server.h              |  13 +-
 bin/named/log.c                               |   1 +
 bin/named/query.c                             | 164 +++++-
 bin/named/server.c                            | 177 +++++-
 bin/named/statschannel.c                      |   6 +
 bin/rndc/rndc.c                               |   2 +
 bin/rndc/rndc.docbook                         |  19 +
 bin/tests/system/chain/prereq.sh              |   7 +
 bin/tests/system/conf.sh.in                   |   2 +-
 bin/tests/system/dyndb/driver/db.c            |   2 +
 bin/tests/system/serve-stale/.gitignore       |  11 +
 bin/tests/system/serve-stale/ans2/ans.pl.in   | 178 ++++++
 bin/tests/system/serve-stale/clean.sh         |  15 +
 .../system/serve-stale/ns1/named1.conf.in     |  35 ++
 .../system/serve-stale/ns1/named2.conf.in     |  35 ++
 bin/tests/system/serve-stale/ns1/root.db      |   5 +
 .../system/serve-stale/ns3/named.conf.in      |  35 ++
 bin/tests/system/serve-stale/prereq.sh        |  38 ++
 bin/tests/system/serve-stale/setup.sh         |  13 +
 bin/tests/system/serve-stale/tests.sh         | 536 ++++++++++++++++++
 doc/arm/Bv9ARM-book.xml                       |  77 ++-
 doc/arm/logging-categories.xml                |  11 +
 doc/arm/notes-rh-changes.xml                  |  14 +-
 doc/misc/options                              |  10 +
 lib/bind9/check.c                             |  78 ++-
 lib/dns/cache.c                               |  38 +-
 lib/dns/db.c                                  |  22 +
 lib/dns/ecdb.c                                |   4 +-
 lib/dns/include/dns/cache.h                   |  21 +
 lib/dns/include/dns/db.h                      |  35 ++
 lib/dns/include/dns/rdataset.h                |  11 +
 lib/dns/include/dns/resolver.h                |  43 +-
 lib/dns/include/dns/types.h                   |   6 +
 lib/dns/include/dns/view.h                    |   3 +
 lib/dns/master.c                              |  14 +-
 lib/dns/masterdump.c                          |  23 +
 lib/dns/rbtdb.c                               | 207 ++++++-
 lib/dns/resolver.c                            |  79 ++-
 lib/dns/sdb.c                                 |   4 +-
 lib/dns/sdlz.c                                |   4 +-
 lib/dns/tests/db_test.c                       | 198 ++++++-
 lib/dns/view.c                                |   3 +
 lib/isccfg/namedconf.c                        |   5 +
 48 files changed, 2126 insertions(+), 106 deletions(-)
 create mode 100644 bin/tests/system/serve-stale/.gitignore
 create mode 100644 bin/tests/system/serve-stale/ans2/ans.pl.in
 create mode 100644 bin/tests/system/serve-stale/clean.sh
 create mode 100644 bin/tests/system/serve-stale/ns1/named1.conf.in
 create mode 100644 bin/tests/system/serve-stale/ns1/named2.conf.in
 create mode 100644 bin/tests/system/serve-stale/ns1/root.db
 create mode 100644 bin/tests/system/serve-stale/ns3/named.conf.in
 create mode 100644 bin/tests/system/serve-stale/prereq.sh
 create mode 100644 bin/tests/system/serve-stale/setup.sh
 create mode 100755 bin/tests/system/serve-stale/tests.sh

diff --git a/bin/named/config.c b/bin/named/config.c
index 9e071bb..d2cd3bc 100644
--- a/bin/named/config.c
+++ b/bin/named/config.c
@@ -182,13 +182,14 @@ options {\n\
 #ifdef HAVE_LMDB
 "	lmdb-mapsize 32M;\n"
 #endif
-"	max-acache-size 16M;\n\
-	max-cache-size 90%;\n\
+"	max-cache-size 90%;\n\
+	max-acache-size 16M;\n\
 	max-cache-ttl 604800; /* 1 week */\n\
 	max-clients-per-query 100;\n\
 	max-ncache-ttl 10800; /* 3 hours */\n\
 	max-recursion-depth 7;\n\
 	max-recursion-queries 100;\n\
+	max-stale-ttl 604800; /* 1 week */\n\
 	message-compression yes;\n\
 #	min-roots <obsolete>;\n\
 	minimal-any false;\n\
@@ -203,10 +204,14 @@ options {\n\
 	request-expire true;\n\
 	request-ixfr true;\n\
 	require-server-cookie no;\n\
+	resolver-nonbackoff-tries 3;\n\
+	resolver-retry-interval 800; /* in milliseconds */\n\
 #	rfc2308-type1 <obsolete>;\n\
 	root-key-sentinel yes;\n\
 	servfail-ttl 1;\n\
 #	sortlist <none>\n\
+	stale-answer-enable false;\n\
+	stale-answer-ttl 1; /* 1 second */\n\
 #	topology <none>\n\
 	transfer-format many-answers;\n\
 	v6-bias 50;\n\
diff --git a/bin/named/control.c b/bin/named/control.c
index 23620b4..0756c73 100644
--- a/bin/named/control.c
+++ b/bin/named/control.c
@@ -282,6 +282,8 @@ ns_control_docommand(isccc_sexpr_t *message, bool readonly,
 		result = ns_server_validation(ns_g_server, lex, text);
 	} else if (command_compare(command, NS_COMMAND_ZONESTATUS)) {
 		result = ns_server_zonestatus(ns_g_server, lex, text);
+	} else if (command_compare(command, NS_COMMAND_SERVESTALE)) {
+		result = ns_server_servestale(ns_g_server, lex, text);
 	} else {
 		isc_log_write(ns_g_lctx, NS_LOGCATEGORY_GENERAL,
 			      NS_LOGMODULE_CONTROL, ISC_LOG_WARNING,
diff --git a/bin/named/include/named/control.h b/bin/named/include/named/control.h
index 56bad8d..37403f1 100644
--- a/bin/named/include/named/control.h
+++ b/bin/named/include/named/control.h
@@ -67,6 +67,7 @@
 #define NS_COMMAND_MKEYS	"managed-keys"
 #define NS_COMMAND_DNSTAPREOPEN	"dnstap-reopen"
 #define NS_COMMAND_DNSTAP	"dnstap"
+#define NS_COMMAND_SERVESTALE	"serve-stale"
 
 isc_result_t
 ns_controls_create(ns_server_t *server, ns_controls_t **ctrlsp);
diff --git a/bin/named/include/named/log.h b/bin/named/include/named/log.h
index 76e3a51..0d1d985 100644
--- a/bin/named/include/named/log.h
+++ b/bin/named/include/named/log.h
@@ -30,6 +30,7 @@
 #define NS_LOGCATEGORY_UPDATE_SECURITY	(&ns_g_categories[6])
 #define NS_LOGCATEGORY_QUERY_ERRORS	(&ns_g_categories[7])
 #define NS_LOGCATEGORY_TAT		(&ns_g_categories[8])
+#define NS_LOGCATEGORY_SERVE_STALE	(&ns_g_categories[9])
 
 /*
  * Backwards compatibility.
diff --git a/bin/named/include/named/query.h b/bin/named/include/named/query.h
index ef1b172..53c052b 100644
--- a/bin/named/include/named/query.h
+++ b/bin/named/include/named/query.h
@@ -35,6 +35,18 @@ typedef struct ns_dbversion {
 	ISC_LINK(struct ns_dbversion)	link;
 } ns_dbversion_t;
 
+/*%
+ * nameserver recursion parameters, to uniquely identify a recursion
+ * query; this is used to detect a recursion loop
+ */
+typedef struct ns_query_recparam {
+	dns_rdatatype_t			qtype;
+	dns_name_t *			qname;
+	dns_fixedname_t			fqname;
+	dns_name_t *			qdomain;
+	dns_fixedname_t			fqdomain;
+} ns_query_recparam_t;
+
 /*% nameserver query structure */
 struct ns_query {
 	unsigned int			attributes;
@@ -63,6 +75,7 @@ struct ns_query {
 	unsigned int			dns64_aaaaoklen;
 	unsigned int			dns64_options;
 	unsigned int			dns64_ttl;
+
 	struct {
 		dns_db_t *      	db;
 		dns_zone_t *      	zone;
@@ -76,6 +89,8 @@ struct ns_query {
 		bool		authoritative;
 		bool		is_zone;
 	} redirect;
+
+	ns_query_recparam_t		recparam;
 	dns_keytag_t root_key_sentinel_keyid;
 	bool root_key_sentinel_is_ta;
 	bool root_key_sentinel_not_ta;
diff --git a/bin/named/include/named/server.h b/bin/named/include/named/server.h
index 0ba2627..08a02dc 100644
--- a/bin/named/include/named/server.h
+++ b/bin/named/include/named/server.h
@@ -227,7 +227,10 @@ enum {
 
 	dns_nsstatscounter_reclimitdropped = 58,
 
-	dns_nsstatscounter_max = 59
+	dns_nsstatscounter_trystale = 59,
+	dns_nsstatscounter_usedstale = 60,
+
+	dns_nsstatscounter_max = 61
 };
 
 /*%
@@ -766,4 +769,12 @@ ns_server_mkeys(ns_server_t *server, isc_lex_t *lex, isc_buffer_t **text);
 isc_result_t
 ns_server_dnstap(ns_server_t *server, isc_lex_t *lex, isc_buffer_t **text);
 
+
+/*%
+ * Control whether stale answers are served or not when configured in
+ * named.conf.
+ */
+isc_result_t
+ns_server_servestale(ns_server_t *server, isc_lex_t *lex,
+		     isc_buffer_t **text);
 #endif /* NAMED_SERVER_H */
diff --git a/bin/named/log.c b/bin/named/log.c
index acfa766..ea6f114 100644
--- a/bin/named/log.c
+++ b/bin/named/log.c
@@ -38,6 +38,7 @@ static isc_logcategory_t categories[] = {
 	{ "update-security",		0 },
 	{ "query-errors",		0 },
 	{ "trust-anchor-telemetry",	0 },
+	{ "serve-stale",                0 },
 	{ NULL, 			0 }
 };
 
diff --git a/bin/named/query.c b/bin/named/query.c
index b14f081..a95f5ad 100644
--- a/bin/named/query.c
+++ b/bin/named/query.c
@@ -149,10 +149,14 @@ last_cmpxchg(isc_stdtime_t *x, isc_stdtime_t *e, isc_stdtime_t r) {
 #define REDIRECT(c)		(((c)->query.attributes & \
 				  NS_QUERYATTR_REDIRECT) != 0)
 
-/*% No QNAME Proof? */
+/*% Does the rdataset 'r' have an attached 'No QNAME Proof'? */
 #define NOQNAME(r)		(((r)->attributes & \
 				  DNS_RDATASETATTR_NOQNAME) != 0)
 
+/*% Does the rdataset 'r' contain a stale answer? */
+#define STALE(r)		(((r)->attributes & \
+				  DNS_RDATASETATTR_STALE) != 0)
+
 #ifdef WANT_QUERYTRACE
 static inline void
 client_trace(ns_client_t *client, int level, const char *message) {
@@ -241,6 +245,10 @@ static bool
 rpz_ck_dnssec(ns_client_t *client, isc_result_t qresult,
 	      dns_rdataset_t *rdataset, dns_rdataset_t *sigrdataset);
 
+static void
+recparam_update(ns_query_recparam_t *param, dns_rdatatype_t qtype,
+		const dns_name_t *qname, const dns_name_t *qdomain);
+
 /*%
  * Increment query statistics counters.
  */
@@ -494,6 +502,7 @@ query_reset(ns_client_t *client, bool everything) {
 	client->query.isreferral = false;
 	client->query.dns64_options = 0;
 	client->query.dns64_ttl = UINT32_MAX;
+	recparam_update(&client->query.recparam, 0, NULL, NULL);
 	client->query.root_key_sentinel_keyid = 0;
 	client->query.root_key_sentinel_is_ta = false;
 	client->query.root_key_sentinel_not_ta = false;
@@ -4305,6 +4314,54 @@ log_quota(ns_client_t *client, isc_stdtime_t *last, isc_stdtime_t now,
 	}
 }
 
+/*%
+ * Check whether the recursion parameters in 'param' match the current query's
+ * recursion parameters provided in 'qtype', 'qname', and 'qdomain'.
+ */
+static bool
+recparam_match(const ns_query_recparam_t *param, dns_rdatatype_t qtype,
+	       const dns_name_t *qname, const dns_name_t *qdomain)
+{
+	REQUIRE(param != NULL);
+
+	return (param->qtype == qtype &&
+	        param->qname != NULL && qname != NULL &&
+	        param->qdomain != NULL && qdomain != NULL &&
+	        dns_name_equal(param->qname, qname) &&
+	        dns_name_equal(param->qdomain, qdomain));
+}
+
+/*%
+ * Update 'param' with current query's recursion parameters provided in
+ * 'qtype', 'qname', and 'qdomain'.
+ */
+static void
+recparam_update(ns_query_recparam_t *param, dns_rdatatype_t qtype,
+		const dns_name_t *qname, const dns_name_t *qdomain)
+{
+	isc_result_t result;
+
+	REQUIRE(param != NULL);
+
+	param->qtype = qtype;
+
+	if (qname == NULL) {
+		param->qname = NULL;
+	} else {
+		param->qname = dns_fixedname_initname(&param->fqname);
+		result = dns_name_copy(qname, param->qname, NULL);
+		RUNTIME_CHECK(result == ISC_R_SUCCESS);
+	}
+
+	if (qdomain == NULL) {
+		param->qdomain = NULL;
+	} else {
+		param->qdomain = dns_fixedname_initname(&param->fqdomain);
+		result = dns_name_copy(qdomain, param->qdomain, NULL);
+		RUNTIME_CHECK(result == ISC_R_SUCCESS);
+	}
+}
+
 static isc_result_t
 query_recurse(ns_client_t *client, dns_rdatatype_t qtype, dns_name_t *qname,
 	      dns_name_t *qdomain, dns_rdataset_t *nameservers,
@@ -4314,6 +4371,19 @@ query_recurse(ns_client_t *client, dns_rdatatype_t qtype, dns_name_t *qname,
 	dns_rdataset_t *rdataset, *sigrdataset;
 	isc_sockaddr_t *peeraddr;
 
+	/*
+	 * Check recursion parameters from the previous query to see if they
+	 * match.  If not, update recursion parameters and proceed.
+	 */
+	if (recparam_match(&client->query.recparam, qtype, qname, qdomain)) {
+		ns_client_log(client, NS_LOGCATEGORY_CLIENT,
+			      NS_LOGMODULE_QUERY, ISC_LOG_INFO,
+			      "recursion loop detected");
+		return (ISC_R_FAILURE);
+	}
+
+	recparam_update(&client->query.recparam, qtype, qname, qdomain);
+
 	if (!resuming)
 		inc_stats(client, dns_nsstatscounter_recursion);
 
@@ -6821,6 +6891,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 	int line = -1;
 	bool dns64_exclude, dns64, rpz;
 	bool nxrewrite = false;
+	bool want_stale = false;
 	bool redirected = false;
 	dns_clientinfomethods_t cm;
 	dns_clientinfo_t ci;
@@ -7130,6 +7201,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		type = qtype;
 
  restart:
+	// query_start
 	CTRACE(ISC_LOG_DEBUG(3), "query_find: restart");
 	want_restart = false;
 	authoritative = false;
@@ -7274,6 +7346,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 	}
 
  db_find:
+	// query_lookup
 	CTRACE(ISC_LOG_DEBUG(3), "query_find: db_find");
 	/*
 	 * We'll need some resources...
@@ -7331,6 +7404,35 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 	if (!is_zone)
 		dns_cache_updatestats(client->view->cache, result);
 
+	if (want_stale) {
+		char namebuf[DNS_NAME_FORMATSIZE];
+		bool success;
+
+		client->query.dboptions &= ~DNS_DBFIND_STALEOK;
+		want_stale = false;
+
+		if (dns_rdataset_isassociated(rdataset) &&
+		    dns_rdataset_count(rdataset) > 0 &&
+		    STALE(rdataset)) {
+			rdataset->ttl = client->view->staleanswerttl;
+			success = true;
+		} else {
+			success = false;
+		}
+
+		dns_name_format(client->query.qname,
+				namebuf, sizeof(namebuf));
+		isc_log_write(ns_g_lctx, NS_LOGCATEGORY_SERVE_STALE,
+			      NS_LOGMODULE_QUERY, ISC_LOG_INFO,
+			      "%s resolver failure, stale answer %s",
+			      namebuf, success ? "used" : "unavailable");
+
+		if (!success) {
+			QUERY_ERROR(DNS_R_SERVFAIL);
+			goto cleanup;
+		}
+	}
+
  resume:
 	CTRACE(ISC_LOG_DEBUG(3), "query_find: resume");
 
@@ -7676,6 +7778,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		 * The cache doesn't even have the root NS.  Get them from
 		 * the hints DB.
 		 */
+		// query_notfound
 		INSIST(!is_zone);
 		if (db != NULL)
 			dns_db_detach(&db);
@@ -7738,12 +7841,14 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		 */
 		/* FALLTHROUGH */
 	case DNS_R_DELEGATION:
+		// query_delegation
 		authoritative = false;
 		if (is_zone) {
 			/*
 			 * Look to see if we are authoritative for the
 			 * child zone if the query type is DS.
 			 */
+			// query_zone_delegation
 			if (!RECURSIONOK(client) &&
 			    (options & DNS_GETDB_NOEXACT) != 0 &&
 			    qtype == dns_rdatatype_ds) {
@@ -8130,6 +8235,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 						       false, true);
 			}
 		}
+		// query_nxdomain
 		if (dns_rdataset_isassociated(rdataset)) {
 			/*
 			 * If we've got a NSEC record, we need to save the
@@ -8450,7 +8556,8 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		/*
 		 * If we have a zero ttl from the cache refetch it.
 		 */
-		if (!is_zone && !resuming && rdataset->ttl == 0 &&
+		// query_cname
+		if (!is_zone && !resuming && !STALE(rdataset) && rdataset->ttl == 0 &&
 		    RECURSIONOK(client))
 		{
 			if (dns_rdataset_isassociated(rdataset))
@@ -8676,7 +8783,11 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 			 "query_find: unexpected error after resuming: %s",
 			 isc_result_totext(result));
 		CTRACE(ISC_LOG_ERROR, errmsg);
-		QUERY_ERROR(DNS_R_SERVFAIL);
+		if (resuming) {
+			want_stale = true;
+		} else {
+			QUERY_ERROR(DNS_R_SERVFAIL);
+		}
 		goto cleanup;
 	}
 
@@ -8932,7 +9043,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		/*
 		 * If we have a zero ttl from the cache refetch it.
 		 */
-		if (!is_zone && !resuming && rdataset->ttl == 0 &&
+		if (!is_zone && !resuming && !STALE(rdataset) && rdataset->ttl == 0 &&
 		    RECURSIONOK(client))
 		{
 			if (dns_rdataset_isassociated(rdataset))
@@ -8943,6 +9054,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 			if (node != NULL)
 				dns_db_detachnode(db, &node);
 
+			// query_respond
 			INSIST(!REDIRECT(client));
 			result = query_recurse(client, qtype,
 					       client->query.qname,
@@ -9223,6 +9335,7 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 				       dns_fixedname_name(&wildcardname),
 				       true, false);
  cleanup:
+	// query_done
 	CTRACE(ISC_LOG_DEBUG(3), "query_find: cleanup");
 	/*
 	 * General cleanup.
@@ -9279,6 +9392,49 @@ query_find(ns_client_t *client, dns_fetchevent_t *event, dns_rdatatype_t qtype)
 		goto restart;
 	}
 
+	if (want_stale) {
+		dns_ttl_t stale_ttl = 0;
+		isc_result_t result;
+		bool staleanswersok = false;
+
+		/*
+		 * Stale answers only make sense if stale_ttl > 0 but
+		 * we want rndc to be able to control returning stale
+		 * answers if they are configured.
+		 */
+		dns_db_attach(client->view->cachedb, &db);
+		result = dns_db_getservestalettl(db, &stale_ttl);
+		if (result == ISC_R_SUCCESS && stale_ttl > 0)  {
+			switch (client->view->staleanswersok) {
+			case dns_stale_answer_yes:
+				staleanswersok = true;
+				break;
+			case dns_stale_answer_conf:
+				staleanswersok =
+					client->view->staleanswersenable;
+				break;
+			case dns_stale_answer_no:
+				staleanswersok = false;
+				break;
+			}
+		} else {
+			staleanswersok = false;
+		}
+
+		if (staleanswersok) {
+			client->query.dboptions |= DNS_DBFIND_STALEOK;
+			inc_stats(client, dns_nsstatscounter_trystale);
+			if (client->query.fetch != NULL)
+				dns_resolver_destroyfetch(
+						   &client->query.fetch);
+			goto db_find;
+		}
+		dns_db_detach(&db);
+		want_stale = false;
+		QUERY_ERROR(DNS_R_SERVFAIL);
+		goto cleanup;
+	}
+
 	if (eresult != ISC_R_SUCCESS &&
 	    (!PARTIALANSWER(client) || WANTRECURSION(client)
 	     || eresult == DNS_R_DROP)) {
diff --git a/bin/named/server.c b/bin/named/server.c
index 2bdf690..3a5ba91 100644
--- a/bin/named/server.c
+++ b/bin/named/server.c
@@ -1720,7 +1720,8 @@ static bool
 cache_sharable(dns_view_t *originview, dns_view_t *view,
 	       bool new_zero_no_soattl,
 	       unsigned int new_cleaning_interval,
-	       uint64_t new_max_cache_size)
+	       uint64_t new_max_cache_size,
+	       uint32_t new_stale_ttl)
 {
 	/*
 	 * If the cache cannot even reused for the same view, it cannot be
@@ -1735,6 +1736,7 @@ cache_sharable(dns_view_t *originview, dns_view_t *view,
 	 */
 	if (dns_cache_getcleaninginterval(originview->cache) !=
 	    new_cleaning_interval ||
+	    dns_cache_getservestalettl(originview->cache) != new_stale_ttl ||
 	    dns_cache_getcachesize(originview->cache) != new_max_cache_size) {
 		return (false);
 	}
@@ -3290,6 +3292,7 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 	size_t max_acache_size;
 	size_t max_adb_size;
 	uint32_t lame_ttl, fail_ttl;
+	uint32_t max_stale_ttl;
 	dns_tsig_keyring_t *ring = NULL;
 	dns_view_t *pview = NULL;	/* Production view */
 	isc_mem_t *cmctx = NULL, *hmctx = NULL;
@@ -3318,6 +3321,7 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 	bool old_rpz_ok = false;
 	isc_dscp_t dscp4 = -1, dscp6 = -1;
 	dns_dyndbctx_t *dctx = NULL;
+	unsigned int resolver_param;
 
 	REQUIRE(DNS_VIEW_VALID(view));
 
@@ -3732,6 +3736,24 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 	if (view->maxncachettl > 7 * 24 * 3600)
 		view->maxncachettl = 7 * 24 * 3600;
 
+	obj = NULL;
+	result = ns_config_get(maps, "max-stale-ttl", &obj);
+	INSIST(result == ISC_R_SUCCESS);
+	max_stale_ttl = cfg_obj_asuint32(obj);
+
+	obj = NULL;
+	result = ns_config_get(maps, "stale-answer-enable", &obj);
+	INSIST(result == ISC_R_SUCCESS);
+	view->staleanswersenable = cfg_obj_asboolean(obj);
+
+	result = dns_viewlist_find(&ns_g_server->viewlist, view->name,
+				   view->rdclass, &pview);
+	if (result == ISC_R_SUCCESS) {
+		view->staleanswersok = pview->staleanswersok;
+		dns_view_detach(&pview);
+	} else
+		view->staleanswersok = dns_stale_answer_conf;
+
 	/*
 	 * Configure the view's cache.
 	 *
@@ -3765,7 +3787,8 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 	nsc = cachelist_find(cachelist, cachename, view->rdclass);
 	if (nsc != NULL) {
 		if (!cache_sharable(nsc->primaryview, view, zero_no_soattl,
-				    cleaning_interval, max_cache_size)) {
+				    cleaning_interval, max_cache_size,
+				    max_stale_ttl)) {
 			isc_log_write(ns_g_lctx, NS_LOGCATEGORY_GENERAL,
 				      NS_LOGMODULE_SERVER, ISC_LOG_ERROR,
 				      "views %s and %s can't share the cache "
@@ -3864,9 +3887,15 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 
 	dns_cache_setcleaninginterval(cache, cleaning_interval);
 	dns_cache_setcachesize(cache, max_cache_size);
+	dns_cache_setservestalettl(cache, max_stale_ttl);
 
 	dns_cache_detach(&cache);
 
+	obj = NULL;
+	result = ns_config_get(maps, "stale-answer-ttl", &obj);
+	INSIST(result == ISC_R_SUCCESS);
+	view->staleanswerttl = ISC_MAX(cfg_obj_asuint32(obj), 1);
+
 	/*
 	 * Resolver.
 	 *
@@ -4055,6 +4084,21 @@ configure_view(dns_view_t *view, dns_viewlist_t *viewlist,
 		maxbits = 4096;
 	view->maxbits = maxbits;
 
+	/*
+	 * Set resolver retry parameters.
+	 */
+	obj = NULL;
+	CHECK(ns_config_get(maps, "resolver-retry-interval", &obj));
+	resolver_param = cfg_obj_asuint32(obj);
+	if (resolver_param > 0)
+		dns_resolver_setretryinterval(view->resolver, resolver_param);
+
+	obj = NULL;
+	CHECK(ns_config_get(maps, "resolver-nonbackoff-tries", &obj));
+	resolver_param = cfg_obj_asuint32(obj);
+	if (resolver_param > 0)
+		dns_resolver_setnonbackofftries(view->resolver, resolver_param);
+
 	/*
 	 * Set supported DNSSEC algorithms.
 	 */
@@ -14559,3 +14603,132 @@ ns_server_dnstap(ns_server_t *server, isc_lex_t *lex, isc_buffer_t **text) {
 	return (ISC_R_NOTIMPLEMENTED);
 #endif
 }
+
+isc_result_t
+ns_server_servestale(ns_server_t *server, isc_lex_t *lex,
+		     isc_buffer_t **text)
+{
+	char *ptr, *classtxt, *viewtxt = NULL;
+	char msg[128];
+	dns_rdataclass_t rdclass = dns_rdataclass_in;
+	dns_view_t *view;
+	bool found = false;
+	dns_stale_answer_t staleanswersok = dns_stale_answer_conf;
+	bool wantstatus = false;
+	isc_result_t result = ISC_R_SUCCESS;
+
+	/* Skip the command name. */
+	ptr = next_token(lex, text);
+	if (ptr == NULL)
+		return (ISC_R_UNEXPECTEDEND);
+
+	ptr = next_token(lex, NULL);
+	if (ptr == NULL)
+		return (ISC_R_UNEXPECTEDEND);
+
+	if (strcasecmp(ptr, "on") == 0 || strcasecmp(ptr, "yes") == 0) {
+		staleanswersok = dns_stale_answer_yes;
+	} else if (strcasecmp(ptr, "off") == 0 || strcasecmp(ptr, "no") == 0) {
+		staleanswersok = dns_stale_answer_no;
+	} else if (strcasecmp(ptr, "reset") == 0) {
+		staleanswersok = dns_stale_answer_conf;
+	} else if (strcasecmp(ptr, "status") == 0) {
+		wantstatus = true;
+	} else
+		return (DNS_R_SYNTAX);
+
+	/* Look for the optional class name. */
+	classtxt = next_token(lex, text);
+	if (classtxt != NULL) {
+		/* Look for the optional view name. */
+		viewtxt = next_token(lex, text);
+	}
+
+	if (classtxt != NULL) {
+		isc_textregion_t r;
+
+		r.base = classtxt;
+		r.length = strlen(classtxt);
+		result = dns_rdataclass_fromtext(&rdclass, &r);
+		if (result != ISC_R_SUCCESS) {
+			if (viewtxt == NULL) {
+				viewtxt = classtxt;
+				classtxt = NULL;
+				result = ISC_R_SUCCESS;
+			} else {
+				snprintf(msg, sizeof(msg),
+					 "unknown class '%s'", classtxt);
+				(void) putstr(text, msg);
+				goto cleanup;
+			}
+		}
+	}
+
+	result = isc_task_beginexclusive(server->task);
+	RUNTIME_CHECK(result == ISC_R_SUCCESS);
+
+	for (view = ISC_LIST_HEAD(server->viewlist);
+	     view != NULL;
+	     view = ISC_LIST_NEXT(view, link))
+	{
+		dns_ttl_t stale_ttl = 0;
+		dns_db_t *db = NULL;
+
+		if (classtxt != NULL && rdclass != view->rdclass)
+			continue;
+
+		if (viewtxt != NULL && strcmp(view->name, viewtxt) != 0)
+			continue;
+
+		if (!wantstatus) {
+			view->staleanswersok = staleanswersok;
+			found = true;
+			continue;
+		}
+
+		db = NULL;
+		dns_db_attach(view->cachedb, &db);
+		(void)dns_db_getservestalettl(db, &stale_ttl);
+		dns_db_detach(&db);
+		if (found)
+			CHECK(putstr(text, "\n"));
+		CHECK(putstr(text, view->name));
+		CHECK(putstr(text, ": "));
+		switch (view->staleanswersok) {
+		case dns_stale_answer_yes:
+			if (stale_ttl > 0)
+				CHECK(putstr(text, "on (rndc)"));
+			else
+				CHECK(putstr(text, "off (not-cached)"));
+			break;
+		case dns_stale_answer_no:
+			CHECK(putstr(text, "off (rndc)"));
+			break;
+		case dns_stale_answer_conf:
+			if (view->staleanswersenable && stale_ttl > 0)
+				CHECK(putstr(text, "on"));
+			else if (view->staleanswersenable)
+				CHECK(putstr(text, "off (not-cached)"));
+			else
+				CHECK(putstr(text, "off"));
+			break;
+		}
+		if (stale_ttl > 0) {
+			snprintf(msg, sizeof(msg),
+				 " (stale-answer-ttl=%u max-stale-ttl=%u)",
+				 view->staleanswerttl, stale_ttl);
+			CHECK(putstr(text, msg));
+		}
+		found = true;
+	}
+	isc_task_endexclusive(ns_g_server->task);
+
+	if (!found)
+		result = ISC_R_NOTFOUND;
+
+cleanup:
+	if (isc_buffer_usedlength(*text) > 0)
+		(void) putnull(text);
+
+	return (result);
+}
diff --git a/bin/named/statschannel.c b/bin/named/statschannel.c
index 12ab048..4938c03 100644
--- a/bin/named/statschannel.c
+++ b/bin/named/statschannel.c
@@ -300,6 +300,12 @@ init_desc(void) {
 	SET_NSSTATDESC(reclimitdropped,
 		       "queries dropped due to recursive client limit",
 		       "RecLimitDropped");
+	SET_NSSTATDESC(trystale,
+		       "attempts to use stale cache data after lookup failure",
+		       "QryTryStale");
+	SET_NSSTATDESC(usedstale,
+		       "successful uses of stale cache data after lookup failure",
+		       "QryUsedStale");
 	INSIST(i == dns_nsstatscounter_max);
 
 	/* Initialize resolver statistics */
diff --git a/bin/rndc/rndc.c b/bin/rndc/rndc.c
index 0acfe3a..2c21c1d 100644
--- a/bin/rndc/rndc.c
+++ b/bin/rndc/rndc.c
@@ -160,6 +160,8 @@ command is one of the following:\n\
   scan		Scan available network interfaces for changes.\n\
   secroots [view ...]\n\
 		Write security roots to the secroots file.\n\
+  serve-stale	( yes | no | reset ) [class [view]]\n\
+		Control whether stale answers are returned\n\
   showzone zone [class [view]]\n\
 		Print a zone's configuration.\n\
   sign zone [class [view]]\n\
diff --git a/bin/rndc/rndc.docbook b/bin/rndc/rndc.docbook
index 159ded9..12a7208 100644
--- a/bin/rndc/rndc.docbook
+++ b/bin/rndc/rndc.docbook
@@ -689,6 +689,25 @@
 	</listitem>
       </varlistentry>
 
+      <varlistentry>
+	<term><userinput>serve-stale ( on | off | reset | status) <optional><replaceable>class</replaceable> <optional><replaceable>view</replaceable></optional></optional></userinput></term>
+	<listitem>
+	  <para>
+	    Enable, disable, or reset the serving of stale answers
+	    as configured in named.conf. Serving of stale answers
+	    will remain disabled across <filename>named.conf</filename>
+	    reloads if disabled via rndc until it is reset via rndc.
+	  </para>
+	  <para>
+	    Status will report whether serving of stale answers is
+	    currently enabled, disabled or not configured for a
+	    view.  If serving of stale records is configured then
+	    the values of stale-answer-ttl and max-stale-ttl are
+	    reported.
+	  </para>
+	</listitem>
+      </varlistentry>
+
       <varlistentry>
 	<term><userinput>secroots <optional>-</optional> <optional><replaceable>view ...</replaceable></optional></userinput></term>
 	<listitem>
diff --git a/bin/tests/system/chain/prereq.sh b/bin/tests/system/chain/prereq.sh
index 23bedcd..43385de 100644
--- a/bin/tests/system/chain/prereq.sh
+++ b/bin/tests/system/chain/prereq.sh
@@ -48,3 +48,10 @@ else
     echo_i "This test requires the Net::DNS::Nameserver library." >&2
     exit 1
 fi
+if $PERL -e 'use Net::DNS::Nameserver;' 2>/dev/null
+then
+	:
+else
+    echo "I:This test requires the Net::DNS::Nameserver library." >&2
+    exit 1
+fi
diff --git a/bin/tests/system/conf.sh.in b/bin/tests/system/conf.sh.in
index f6412f6..26c8901 100644
--- a/bin/tests/system/conf.sh.in
+++ b/bin/tests/system/conf.sh.in
@@ -128,7 +128,7 @@ PARALLELDIRS="dnssec rpzrecurse \
 	reclimit redirect resolver rndc rootkeysentinel rpz \
 	rrchecker rrl rrsetorder rsabigexponent runtime \
 	sfcache smartsign sortlist \
-	spf staticstub statistics statschannel stub \
+	spf serve-stale staticstub statistics statschannel stub \
 	tcp tsig tsiggss \
 	unknown upforwd verify views wildcard \
 	xfer xferquota zero zonechecks"
diff --git a/bin/tests/system/dyndb/driver/db.c b/bin/tests/system/dyndb/driver/db.c
index 02aa6ab..a77c7de 100644
--- a/bin/tests/system/dyndb/driver/db.c
+++ b/bin/tests/system/dyndb/driver/db.c
@@ -629,6 +629,8 @@ static dns_dbmethods_t sampledb_methods = {
 	hashsize,
 	NULL,
 	NULL,
+	NULL,
+	NULL,
 };
 
 /* Auxiliary driver functions. */
diff --git a/bin/tests/system/serve-stale/.gitignore b/bin/tests/system/serve-stale/.gitignore
new file mode 100644
index 0000000..2272eef
--- /dev/null
+++ b/bin/tests/system/serve-stale/.gitignore
@@ -0,0 +1,11 @@
+/ans2/ans.pid
+/ans2/ans.pl
+/dig.out*
+/ns1/named.conf
+/ns3/named.conf
+/ns3/root.bk
+/rndc.out*
+named.lock
+named.pid
+named.port
+named.run
diff --git a/bin/tests/system/serve-stale/ans2/ans.pl.in b/bin/tests/system/serve-stale/ans2/ans.pl.in
new file mode 100644
index 0000000..2b39eca
--- /dev/null
+++ b/bin/tests/system/serve-stale/ans2/ans.pl.in
@@ -0,0 +1,178 @@
+#!/usr/bin/env perl
+#
+# Copyright (C) 2014-2016  Internet Systems Consortium, Inc. ("ISC")
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+use strict;
+use warnings;
+
+use IO::File;
+use IO::Socket;
+use Getopt::Long;
+use Net::DNS;
+use Time::HiRes qw(usleep nanosleep);
+
+my $pidf = new IO::File "ans.pid", "w" or die "cannot open pid file: $!";
+print $pidf "$$\n" or die "cannot write pid file: $!";
+$pidf->close or die "cannot close pid file: $!";
+sub rmpid { unlink "ans.pid"; exit 1; };
+
+$SIG{INT} = \&rmpid;
+$SIG{TERM} = \&rmpid;
+
+my $send_response = 1;
+
+my $localaddr = "10.53.0.2";
+my $localport = @PORT@;
+my $udpsock = IO::Socket::INET->new(LocalAddr => "$localaddr",
+   LocalPort => $localport, Proto => "udp", Reuse => 1) or die "$!";
+
+#
+# Delegation
+#
+my $SOA = "example 300 IN SOA . . 0 0 0 0 300";
+my $NS = "example 300 IN NS ns.example";
+my $A = "ns.example 300 IN A $localaddr";
+#
+# Records to be TTL stretched
+#
+my $TXT = "data.example 1 IN TXT \"A text record with a 1 second ttl\"";
+my $negSOA = "example 1 IN SOA . . 0 0 0 0 300";
+
+sub reply_handler {
+    my ($qname, $qclass, $qtype) = @_;
+    my ($rcode, @ans, @auth, @add);
+
+    print ("request: $qname/$qtype\n");
+    STDOUT->flush();
+
+    # Control whether we send a response or not.
+    # We always respond to control commands.