diff --git a/SOURCES/0001-Break-long-store_client-call-chains-with-async-calls.patch b/SOURCES/0001-Break-long-store_client-call-chains-with-async-calls.patch new file mode 100644 index 0000000..0b63af6 --- /dev/null +++ b/SOURCES/0001-Break-long-store_client-call-chains-with-async-calls.patch @@ -0,0 +1,599 @@ +From 4896d07bf753683a3dbba4210384b0d862ff2d11 Mon Sep 17 00:00:00 2001 +From: Eduard Bagdasaryan +Date: Thu, 7 Dec 2023 16:47:08 +0000 +Subject: [PATCH 1/7] Break long store_client call chains with async calls + (#1056) + +The store_client class design created very long call chains spanning +Squid-client and Squid-server processing and multiple transactions. +These call chains also create ideal conditions for dangerous recursive +relationships between communicating classes (a.k.a. "reentrancy" among +Squid developers). For example, storeClientCopy() enters store_client +and triggers disk I/O that triggers invokeHandlers() that re-enters the +same store_client object and starts competing with the original +storeClientCopy() processing state. + +The official code prevented the worst recursion cases with three(!) +boolean flags and time-based events abused to break some of the call +chains, but that approach did not solve all of the problems while also +losing transaction context information across time-based events. + +This change effectively makes STCB storeClientCopy() callbacks +asynchronous, eliminating the need for time-based events and one of the +flags. It shortens many call chains and preserves transaction context. +The remaining problems can and should be eliminated by converting +store_client into AsyncJob, but those changes deserve a dedicated PR. + +store_client orchestrates cooperation of multiple asynchronous players: + +* Sink: A Store client requests a STCB callback via a + storeClientCopy()/copy() call. A set _callback.callback_handler + implies that the client is waiting for this callback. + +* Source1: A Store disk reading subsystem activated by the storeRead() + call "spontaneously" delivers response bytes via storeClientRead*() + callbacks. The disk_io_pending flag implies waiting for them. + +* Source2: Store memory subsystem activated by storeClientListAdd() + "spontaneously" delivers response bytes via invokeHandlers(). + +* Source3: Store disk subsystem activated by storeSwapInStart() + "spontaneously" notifies of EOF/error by calling noteSwapInDone(). + +* Source4: A store_client object owner may delete the object by + "spontaneously" calling storeUnregister(). The official code was + converting this event into an error-notifying callback. + +We continue to answer each storeClientCopy() request with the first +available information even though several SourceN calls are possible +while we are waiting to complete the STCB callback. The StoreIOBuffer +API and STCB recipients do not support data+error/eof combinations, and +future code will move this wait to the main event loop anyway. This +first-available approach means that the creation of the notifier call +effectively ends answer processing -- store_client just waits for that +call to fire so that it can relay the answer to still-synchronous STCB. +When STCB itself becomes asynchronous, this logic will continue to work. + +Also stopped calling STCB from storeUnregister(). Conceptually, the +storeUnregister() and storeClientCopy() callers ought to represent the +same get-content-from-Store task; there should be no need to notify that +task about what it is doing. Technically, analysis of STCB callbacks +showed that many such notifications would be dangerous (if they are or +become reachable). At the time of the storeUnregister() call, the STCB +callbacks are usually unset (e.g., when storeUnregister() is called from +the destructor, after that object has finished copying -- a very common +case) or do not do anything (useful). + +Also removed callback_data from the Callback::pending() condition. It is +conceptually wrong to require non-nil callback parameter, and it is +never cleared separately from the callback_handler data member anyway. + +Also hid copyInto into the private store_client section to make sure it +is not modified while we are waiting to complete the STCB callback. This +move required adding a couple of read-only wrapper methods like +bytesWanted() and noteSwapInDone(). + +Also simplified error/EOF/bytes handling on copy()-STCB path using +dedicated methods (e.g., store_client::callback() API is no longer +mixing EOF and error signals). + +Modified-by: Alex Burmashev +Signed-off-by: Alex Burmashev +--- + src/MemObject.cc | 6 +- + src/StoreClient.h | 64 ++++++++++-- + src/store_client.cc | 177 ++++++++++++++++++++++----------- + src/store_swapin.cc | 2 +- + src/tests/stub_store_client.cc | 5 +- + 5 files changed, 186 insertions(+), 68 deletions(-) + +diff --git a/src/MemObject.cc b/src/MemObject.cc +index df7791f..4ba63cc 100644 +--- a/src/MemObject.cc ++++ b/src/MemObject.cc +@@ -196,8 +196,8 @@ struct LowestMemReader : public unary_function { + LowestMemReader(int64_t seed):current(seed) {} + + void operator() (store_client const &x) { +- if (x.memReaderHasLowerOffset(current)) +- current = x.copyInto.offset; ++ if (x.getType() == STORE_MEM_CLIENT) ++ current = std::min(current, x.readOffset()); + } + + int64_t current; +@@ -492,7 +492,7 @@ MemObject::mostBytesAllowed() const + + #endif + +- j = sc->delayId.bytesWanted(0, sc->copyInto.length); ++ j = sc->bytesWanted(); + + if (j > jmax) { + jmax = j; +diff --git a/src/StoreClient.h b/src/StoreClient.h +index 65472d8..457844a 100644 +--- a/src/StoreClient.h ++++ b/src/StoreClient.h +@@ -12,6 +12,7 @@ + #include "dlink.h" + #include "StoreIOBuffer.h" + #include "StoreIOState.h" ++#include "base/AsyncCall.h" + + typedef void STCB(void *, StoreIOBuffer); /* store callback */ + +@@ -39,14 +40,32 @@ class store_client + public: + store_client(StoreEntry *); + ~store_client(); +- bool memReaderHasLowerOffset(int64_t) const; ++ ++ /// An offset into the stored response bytes, including the HTTP response ++ /// headers (if any). Note that this offset does not include Store entry ++ /// metadata, because it is not a part of the stored response. ++ /// \retval 0 means the client wants to read HTTP response headers. ++ /// \retval +N the response byte that the client wants to read next. ++ /// \retval -N should not occur. ++ // TODO: Callers do not expect negative offset. Verify that the return ++ // value cannot be negative and convert to unsigned in this case. ++ int64_t readOffset() const { return copyInto.offset; } ++ + int getType() const; +- void fail(); +- void callback(ssize_t len, bool error = false); ++ ++ /// React to the end of reading the response from disk. There will be no ++ /// more readHeader() and readBody() callbacks for the current storeRead() ++ /// swapin after this notification. ++ void noteSwapInDone(bool error); ++ + void doCopy (StoreEntry *e); + void readHeader(const char *buf, ssize_t len); + void readBody(const char *buf, ssize_t len); ++ ++ /// Request StoreIOBuffer-described response data via an asynchronous STCB ++ /// callback. At most one outstanding request is allowed per store_client. + void copy(StoreEntry *, StoreIOBuffer, STCB *, void *); ++ + void dumpStats(MemBuf * output, int clientNumber) const; + + int64_t cmp_offset; +@@ -59,19 +78,29 @@ public: + StoreIOState::Pointer swapin_sio; + + struct { ++ /// whether we are expecting a response to be swapped in from disk ++ /// (i.e. whether async storeRead() is currently in progress) ++ // TODO: a better name reflecting the 'in' scope of the flag + bool disk_io_pending; ++ ++ /// whether the store_client::doCopy()-initiated STCB sequence is ++ /// currently in progress + bool store_copying; +- bool copy_event_pending; + } flags; + + #if USE_DELAY_POOLS + DelayId delayId; ++ ++ /// The maximum number of bytes the Store client can read/copy next without ++ /// overflowing its buffer and without violating delay pool limits. Store ++ /// I/O is not rate-limited, but we assume that the same number of bytes may ++ /// be read from the Squid-to-server connection that may be rate-limited. ++ int bytesWanted() const; ++ + void setDelayId(DelayId delay_id); + #endif + + dlink_node node; +- /* Below here is private - do no alter outside storeClient calls */ +- StoreIOBuffer copyInto; + + private: + bool moreToSend() const; +@@ -83,9 +112,25 @@ private: + bool startSwapin(); + bool unpackHeader(char const *buf, ssize_t len); + ++ void fail(); ++ void callback(ssize_t); ++ void noteCopiedBytes(size_t); ++ void noteEof(); ++ void noteNews(); ++ void finishCallback(); ++ static void FinishCallback(store_client *); ++ + int type; + bool object_ok; + ++ /// Storage and metadata associated with the current copy() request. Ought ++ /// to be ignored when not answering a copy() request. ++ StoreIOBuffer copyInto; ++ ++ /// The number of bytes loaded from Store into copyInto while answering the ++ /// current copy() request. Ought to be ignored when not answering. ++ size_t copiedSize; ++ + /* Until we finish stuffing code into store_client */ + + public: +@@ -94,9 +139,16 @@ public: + Callback ():callback_handler(NULL), callback_data(NULL) {} + + Callback (STCB *, void *); ++ ++ /// Whether the copy() answer is needed/expected (by the client) and has ++ /// not been computed (by us). False during (asynchronous) answer ++ /// delivery to the STCB callback_handler. + bool pending() const; + STCB *callback_handler; + void *callback_data; ++ ++ /// a scheduled asynchronous finishCallback() call (or nil) ++ AsyncCall::Pointer notifier; + } _callback; + }; + +diff --git a/src/store_client.cc b/src/store_client.cc +index 1b54f04..207c96b 100644 +--- a/src/store_client.cc ++++ b/src/store_client.cc +@@ -9,6 +9,7 @@ + /* DEBUG: section 90 Storage Manager Client-Side Interface */ + + #include "squid.h" ++#include "base/AsyncCbdataCalls.h" + #include "event.h" + #include "globals.h" + #include "HttpReply.h" +@@ -39,17 +40,10 @@ + static StoreIOState::STRCB storeClientReadBody; + static StoreIOState::STRCB storeClientReadHeader; + static void storeClientCopy2(StoreEntry * e, store_client * sc); +-static EVH storeClientCopyEvent; + static bool CheckQuickAbortIsReasonable(StoreEntry * entry); + + CBDATA_CLASS_INIT(store_client); + +-bool +-store_client::memReaderHasLowerOffset(int64_t anOffset) const +-{ +- return getType() == STORE_MEM_CLIENT && copyInto.offset < anOffset; +-} +- + int + store_client::getType() const + { +@@ -104,22 +98,41 @@ storeClientListAdd(StoreEntry * e, void *data) + return sc; + } + ++/// schedules asynchronous STCB call to relay disk or memory read results ++/// \param outcome an error signal (if negative), an EOF signal (if zero), or the number of bytes read ++void ++store_client::callback(const ssize_t outcome) ++{ ++ if (outcome > 0) ++ return noteCopiedBytes(outcome); ++ ++ if (outcome < 0) ++ return fail(); ++ ++ noteEof(); ++} ++/// finishCallback() wrapper; TODO: Add NullaryMemFunT for non-jobs. + void +-store_client::callback(ssize_t sz, bool error) ++store_client::FinishCallback(store_client * const sc) + { +- size_t bSz = 0; ++ sc->finishCallback(); ++} + +- if (sz >= 0 && !error) +- bSz = sz; ++/// finishes a copy()-STCB sequence by synchronously calling STCB ++void ++store_client::finishCallback() ++{ ++ Assure(_callback.callback_handler); ++ Assure(_callback.notifier); + +- StoreIOBuffer result(bSz, 0 ,copyInto.data); ++ // callers are not ready to handle a content+error combination ++ Assure(object_ok || !copiedSize); + +- if (sz < 0 || error) +- result.flags.error = 1; ++ StoreIOBuffer result(copiedSize, copyInto.offset, copyInto.data); ++ result.flags.error = object_ok ? 0 : 1; ++ copiedSize = 0; + +- result.offset = cmp_offset; +- assert(_callback.pending()); +- cmp_offset = copyInto.offset + bSz; ++ cmp_offset = result.offset + result.length; + STCB *temphandler = _callback.callback_handler; + void *cbdata = _callback.callback_data; + _callback = Callback(NULL, NULL); +@@ -131,18 +144,24 @@ store_client::callback(ssize_t sz, bool error) + cbdataReferenceDone(cbdata); + } + +-static void +-storeClientCopyEvent(void *data) ++/// schedules asynchronous STCB call to relay a successful disk or memory read ++/// \param bytesCopied the number of response bytes copied into copyInto ++void ++store_client::noteCopiedBytes(const size_t bytesCopied) + { +- store_client *sc = (store_client *)data; +- debugs(90, 3, "storeClientCopyEvent: Running"); +- assert (sc->flags.copy_event_pending); +- sc->flags.copy_event_pending = false; +- +- if (!sc->_callback.pending()) +- return; ++ debugs(90, 5, bytesCopied); ++ Assure(bytesCopied > 0); ++ Assure(!copiedSize); ++ copiedSize = bytesCopied; ++ noteNews(); ++} + +- storeClientCopy2(sc->entry, sc); ++void ++store_client::noteEof() ++{ ++ debugs(90, 5, copiedSize); ++ Assure(!copiedSize); ++ noteNews(); + } + + store_client::store_client(StoreEntry *e) : +@@ -152,11 +171,11 @@ store_client::store_client(StoreEntry *e) : + #endif + entry(e), + type(e->storeClientType()), +- object_ok(true) ++ object_ok(true), ++ copiedSize(0) + { + flags.disk_io_pending = false; + flags.store_copying = false; +- flags.copy_event_pending = false; + ++ entry->refcount; + + if (getType() == STORE_DISK_CLIENT) { +@@ -272,17 +291,11 @@ static void + storeClientCopy2(StoreEntry * e, store_client * sc) + { + /* reentrancy not allowed - note this could lead to +- * dropped events ++ * dropped notifications about response data availability + */ + +- if (sc->flags.copy_event_pending) { +- return; +- } +- + if (sc->flags.store_copying) { +- sc->flags.copy_event_pending = true; +- debugs(90, 3, "storeClientCopy2: Queueing storeClientCopyEvent()"); +- eventAdd("storeClientCopyEvent", storeClientCopyEvent, sc, 0.0, 0); ++ debugs(90, 3, "prevented recursive copying for " << *e); + return; + } + +@@ -295,21 +308,16 @@ storeClientCopy2(StoreEntry * e, store_client * sc) + * if the peer aborts, we want to give the client(s) + * everything we got before the abort condition occurred. + */ +- /* Warning: doCopy may indirectly free itself in callbacks, +- * hence the lock to keep it active for the duration of +- * this function +- * XXX: Locking does not prevent calling sc destructor (it only prevents +- * freeing sc memory) so sc may become invalid from C++ p.o.v. +- */ +- CbcPointer tmpLock = sc; +- assert (!sc->flags.store_copying); + sc->doCopy(e); +- assert(!sc->flags.store_copying); + } + + void + store_client::doCopy(StoreEntry *anEntry) + { ++ Assure(_callback.pending()); ++ Assure(!flags.disk_io_pending); ++ Assure(!flags.store_copying); ++ + assert (anEntry == entry); + flags.store_copying = true; + MemObject *mem = entry->mem_obj; +@@ -321,7 +329,7 @@ store_client::doCopy(StoreEntry *anEntry) + if (!moreToSend()) { + /* There is no more to send! */ + debugs(33, 3, HERE << "There is no more to send!"); +- callback(0); ++ noteEof(); + flags.store_copying = false; + return; + } +@@ -382,6 +390,16 @@ store_client::startSwapin() + } + } + ++void ++store_client::noteSwapInDone(const bool error) ++{ ++ Assure(_callback.pending()); ++ if (error) ++ fail(); ++ else ++ noteEof(); ++} ++ + void + store_client::scheduleRead() + { +@@ -421,7 +439,7 @@ store_client::scheduleMemRead() + /* What the client wants is in memory */ + /* Old style */ + debugs(90, 3, "store_client::doCopy: Copying normal from memory"); +- size_t sz = entry->mem_obj->data_hdr.copy(copyInto); ++ const auto sz = entry->mem_obj->data_hdr.copy(copyInto); // may be <= 0 per copy() API + callback(sz); + flags.store_copying = false; + } +@@ -493,7 +511,19 @@ store_client::readBody(const char *, ssize_t len) + void + store_client::fail() + { ++ debugs(90, 3, (object_ok ? "once" : "again")); ++ if (!object_ok) ++ return; // we failed earlier; nothing to do now ++ + object_ok = false; ++ ++ noteNews(); ++} ++ ++/// if necessary and possible, informs the Store reader about copy() result ++void ++store_client::noteNews() ++{ + /* synchronous open failures callback from the store, + * before startSwapin detects the failure. + * TODO: fix this inconsistent behaviour - probably by +@@ -501,8 +531,20 @@ store_client::fail() + * not synchronous + */ + +- if (_callback.pending()) +- callback(0, true); ++ if (!_callback.callback_handler) { ++ debugs(90, 5, "client lost interest"); ++ return; ++ } ++ ++ if (_callback.notifier) { ++ debugs(90, 5, "earlier news is being delivered by " << _callback.notifier); ++ return; ++ } ++ ++ _callback.notifier = asyncCall(90, 4, "store_client::FinishCallback", cbdataDialer(store_client::FinishCallback, this)); ++ ScheduleCallHere(_callback.notifier); ++ ++ Assure(!_callback.pending()); + } + + static void +@@ -673,10 +715,12 @@ storeUnregister(store_client * sc, StoreEntry * e, void *data) + ++statCounter.swap.ins; + } + +- if (sc->_callback.pending()) { +- /* callback with ssize = -1 to indicate unexpected termination */ +- debugs(90, 3, "store_client for " << *e << " has a callback"); +- sc->fail(); ++ if (sc->_callback.callback_handler || sc->_callback.notifier) { ++ debugs(90, 3, "forgetting store_client callback for " << *e); ++ // Do not notify: Callers want to stop copying and forget about this ++ // pending copy request. Some would mishandle a notification from here. ++ if (sc->_callback.notifier) ++ sc->_callback.notifier->cancel("storeUnregister"); + } + + #if STORE_CLIENT_LIST_DEBUG +@@ -684,6 +728,8 @@ storeUnregister(store_client * sc, StoreEntry * e, void *data) + + #endif + ++ // XXX: We might be inside sc store_client method somewhere up the call ++ // stack. TODO: Convert store_client to AsyncJob to make destruction async. + delete sc; + + assert(e->locked()); +@@ -740,6 +786,16 @@ StoreEntry::invokeHandlers() + + if (sc->flags.disk_io_pending) + continue; ++ if (sc->flags.store_copying) ++ continue; ++ ++ // XXX: If invokeHandlers() is (indirectly) called from a store_client ++ // method, then the above three conditions may not be sufficient to ++ // prevent us from reentering the same store_client object! This ++ // probably does not happen in the current code, but no observed ++ // invariant prevents this from (accidentally) happening in the future. ++ ++ // TODO: Convert store_client into AsyncJob; make this call asynchronous + + storeClientCopy2(this, sc); + } +@@ -864,8 +920,8 @@ store_client::dumpStats(MemBuf * output, int clientNumber) const + if (flags.store_copying) + output->append(" store_copying", 14); + +- if (flags.copy_event_pending) +- output->append(" copy_event_pending", 19); ++ if (_callback.notifier) ++ output->append(" notifying", 10); + + output->append("\n",1); + } +@@ -873,12 +929,19 @@ store_client::dumpStats(MemBuf * output, int clientNumber) const + bool + store_client::Callback::pending() const + { +- return callback_handler && callback_data; ++ return callback_handler && !notifier; + } + + store_client::Callback::Callback(STCB *function, void *data) : callback_handler(function), callback_data (data) {} + + #if USE_DELAY_POOLS ++int ++store_client::bytesWanted() const ++{ ++ // TODO: To avoid using stale copyInto, return zero if !_callback.pending()? ++ return delayId.bytesWanted(0, copyInto.length); ++} ++ + void + store_client::setDelayId(DelayId delay_id) + { +diff --git a/src/store_swapin.cc b/src/store_swapin.cc +index a05d7e3..cd32e94 100644 +--- a/src/store_swapin.cc ++++ b/src/store_swapin.cc +@@ -56,7 +56,7 @@ storeSwapInFileClosed(void *data, int errflag, StoreIOState::Pointer) + + if (sc->_callback.pending()) { + assert (errflag <= 0); +- sc->callback(0, errflag ? true : false); ++ sc->noteSwapInDone(errflag); + } + + ++statCounter.swap.ins; +diff --git a/src/tests/stub_store_client.cc b/src/tests/stub_store_client.cc +index 2a13874..4a73863 100644 +--- a/src/tests/stub_store_client.cc ++++ b/src/tests/stub_store_client.cc +@@ -34,7 +34,10 @@ void storeLogOpen(void) STUB + void storeDigestInit(void) STUB + void storeRebuildStart(void) STUB + void storeReplSetup(void) STUB +-bool store_client::memReaderHasLowerOffset(int64_t anOffset) const STUB_RETVAL(false) ++void store_client::noteSwapInDone(bool) STUB ++#if USE_DELAY_POOLS ++int store_client::bytesWanted() const STUB_RETVAL(0) ++#endif + void store_client::dumpStats(MemBuf * output, int clientNumber) const STUB + int store_client::getType() const STUB_RETVAL(0) + +-- +2.39.3 + diff --git a/SOURCES/0001-Fix-incremental-parsing-of-chunked-quoted-extensions.patch b/SOURCES/0001-Fix-incremental-parsing-of-chunked-quoted-extensions.patch deleted file mode 100644 index 7a65628..0000000 --- a/SOURCES/0001-Fix-incremental-parsing-of-chunked-quoted-extensions.patch +++ /dev/null @@ -1,1190 +0,0 @@ -From 96d95b036e28c863c810b334f17d0ec619bf421c Mon Sep 17 00:00:00 2001 -From: Eduard Bagdasaryan -Date: Sun, 5 Nov 2023 11:20:35 +0000 -Subject: [PATCH 1/2] Fix incremental parsing of chunked quoted extensions - (#310) - -Before this change, incremental parsing of quoted chunked extensions -was broken for two reasons: - -* Http::One::Parser::skipLineTerminator() unexpectedly threw after - partially received quoted chunk extension value. - -* When Http::One::Tokenizer was unable to parse a quoted extension, - it incorrectly restored the input buffer to the beginning of the - extension value (instead of the extension itself), thus making - further incremental parsing iterations impossible. - -IMO, the reason for this problem was that Http::One::Tokenizer::qdText() -could not distinguish two cases (returning false in both): - -* the end of the quoted string not yet reached - -* an input error, e.g., wrong/unexpected character - -A possible approach could be to improve Http::One::Tokenizer, making it -aware about "needs more data" state. However, to be acceptable, -these improvements should be done in the base Parser::Tokenizer -class instead. These changes seem to be non-trivial and could be -done separately and later. - -Another approach, used here, is to simplify the complex and error-prone -chunked extensions parsing algorithm, fixing incremental parsing bugs -and still parse incrementally in almost all cases. The performance -regression could be expected only in relatively rare cases of partially -received or malformed extensions. - -Also: -* fixed parsing of partial use-original-body extension values -* do not treat an invalid use-original-body as an unknown extension -* optimization: parse use-original-body extension only in ICAP context - (i.e., where it is expected) -* improvement: added a new API to TeChunkedParser to specify known - chunked extensions list - -Modified-by: Alex Burmashev -Signed-off-by: Alex Burmashev ---- - src/adaptation/icap/ModXact.cc | 22 ++++- - src/adaptation/icap/ModXact.h | 20 +++++ - src/http/one/Parser.cc | 35 ++++---- - src/http/one/Parser.h | 10 ++- - src/http/one/RequestParser.cc | 16 ++-- - src/http/one/RequestParser.h | 8 +- - src/http/one/ResponseParser.cc | 17 ++-- - src/http/one/ResponseParser.h | 2 +- - src/http/one/TeChunkedParser.cc | 139 ++++++++++++++++++-------------- - src/http/one/TeChunkedParser.h | 41 ++++++++-- - src/http/one/Tokenizer.cc | 104 ++++++++++++------------ - src/http/one/Tokenizer.h | 89 ++++++++------------ - src/http/one/forward.h | 3 + - src/parser/BinaryTokenizer.h | 3 +- - src/parser/Makefile.am | 1 + - src/parser/Tokenizer.cc | 40 +++++++++ - src/parser/Tokenizer.h | 13 +++ - src/parser/forward.h | 22 +++++ - 18 files changed, 364 insertions(+), 221 deletions(-) - create mode 100644 src/parser/forward.h - -diff --git a/src/adaptation/icap/ModXact.cc b/src/adaptation/icap/ModXact.cc -index 2db0a68..22a87f5 100644 ---- a/src/adaptation/icap/ModXact.cc -+++ b/src/adaptation/icap/ModXact.cc -@@ -25,12 +25,13 @@ - #include "comm.h" - #include "comm/Connection.h" - #include "err_detail_type.h" --#include "http/one/TeChunkedParser.h" - #include "HttpHeaderTools.h" - #include "HttpMsg.h" - #include "HttpReply.h" - #include "HttpRequest.h" - #include "MasterXaction.h" -+#include "parser/Tokenizer.h" -+#include "sbuf/Stream.h" - #include "SquidTime.h" - - // flow and terminology: -@@ -44,6 +45,8 @@ CBDATA_NAMESPACED_CLASS_INIT(Adaptation::Icap, ModXactLauncher); - - static const size_t TheBackupLimit = BodyPipe::MaxCapacity; - -+const SBuf Adaptation::Icap::ChunkExtensionValueParser::UseOriginalBodyName("use-original-body"); -+ - Adaptation::Icap::ModXact::State::State() - { - memset(this, 0, sizeof(*this)); -@@ -1108,6 +1111,7 @@ void Adaptation::Icap::ModXact::decideOnParsingBody() - state.parsing = State::psBody; - replyHttpBodySize = 0; - bodyParser = new Http1::TeChunkedParser; -+ bodyParser->parseExtensionValuesWith(&extensionParser); - makeAdaptedBodyPipe("adapted response from the ICAP server"); - Must(state.sending == State::sendingAdapted); - } else { -@@ -1142,9 +1146,8 @@ void Adaptation::Icap::ModXact::parseBody() - } - - if (parsed) { -- if (state.readyForUob && bodyParser->useOriginBody >= 0) { -- prepPartialBodyEchoing( -- static_cast(bodyParser->useOriginBody)); -+ if (state.readyForUob && extensionParser.sawUseOriginalBody()) { -+ prepPartialBodyEchoing(extensionParser.useOriginalBody()); - stopParsing(); - return; - } -@@ -2014,3 +2017,14 @@ void Adaptation::Icap::ModXactLauncher::updateHistory(bool doStart) - } - } - -+void -+Adaptation::Icap::ChunkExtensionValueParser::parse(Tokenizer &tok, const SBuf &extName) -+{ -+ if (extName == UseOriginalBodyName) { -+ useOriginalBody_ = tok.udec64("use-original-body"); -+ assert(useOriginalBody_ >= 0); -+ } else { -+ Ignore(tok, extName); -+ } -+} -+ -diff --git a/src/adaptation/icap/ModXact.h b/src/adaptation/icap/ModXact.h -index f7afa69..fb4dec0 100644 ---- a/src/adaptation/icap/ModXact.h -+++ b/src/adaptation/icap/ModXact.h -@@ -15,6 +15,7 @@ - #include "adaptation/icap/Xaction.h" - #include "BodyPipe.h" - #include "http/one/forward.h" -+#include "http/one/TeChunkedParser.h" - - /* - * ICAPModXact implements ICAP REQMOD and RESPMOD transaction using -@@ -105,6 +106,23 @@ private: - enum State { stDisabled, stWriting, stIeof, stDone } theState; - }; - -+/// handles ICAP-specific chunk extensions supported by Squid -+class ChunkExtensionValueParser: public Http1::ChunkExtensionValueParser -+{ -+public: -+ /* Http1::ChunkExtensionValueParser API */ -+ virtual void parse(Tokenizer &tok, const SBuf &extName) override; -+ -+ bool sawUseOriginalBody() const { return useOriginalBody_ >= 0; } -+ uint64_t useOriginalBody() const { assert(sawUseOriginalBody()); return static_cast(useOriginalBody_); } -+ -+private: -+ static const SBuf UseOriginalBodyName; -+ -+ /// the value of the parsed use-original-body chunk extension (or -1) -+ int64_t useOriginalBody_ = -1; -+}; -+ - class ModXact: public Xaction, public BodyProducer, public BodyConsumer - { - CBDATA_CLASS(ModXact); -@@ -270,6 +288,8 @@ private: - - int adaptHistoryId; ///< adaptation history slot reservation - -+ ChunkExtensionValueParser extensionParser; -+ - class State - { - -diff --git a/src/http/one/Parser.cc b/src/http/one/Parser.cc -index 0c86733..affe0b1 100644 ---- a/src/http/one/Parser.cc -+++ b/src/http/one/Parser.cc -@@ -7,10 +7,11 @@ - */ - - #include "squid.h" -+#include "base/CharacterSet.h" - #include "Debug.h" - #include "http/one/Parser.h" --#include "http/one/Tokenizer.h" - #include "mime_header.h" -+#include "parser/Tokenizer.h" - #include "SquidConfig.h" - - /// RFC 7230 section 2.6 - 7 magic octets -@@ -61,20 +62,19 @@ Http::One::Parser::DelimiterCharacters() - RelaxedDelimiterCharacters() : CharacterSet::SP; - } - --bool --Http::One::Parser::skipLineTerminator(Http1::Tokenizer &tok) const -+void -+Http::One::Parser::skipLineTerminator(Tokenizer &tok) const - { - if (tok.skip(Http1::CrLf())) -- return true; -+ return; - - if (Config.onoff.relaxed_header_parser && tok.skipOne(CharacterSet::LF)) -- return true; -+ return; - - if (tok.atEnd() || (tok.remaining().length() == 1 && tok.remaining().at(0) == '\r')) -- return false; // need more data -+ throw InsufficientInput(); - - throw TexcHere("garbage instead of CRLF line terminator"); -- return false; // unreachable, but make naive compilers happy - } - - /// all characters except the LF line terminator -@@ -102,7 +102,7 @@ LineCharacters() - void - Http::One::Parser::cleanMimePrefix() - { -- Http1::Tokenizer tok(mimeHeaderBlock_); -+ Tokenizer tok(mimeHeaderBlock_); - while (tok.skipOne(RelaxedDelimiterCharacters())) { - (void)tok.skipAll(LineCharacters()); // optional line content - // LF terminator is required. -@@ -137,7 +137,7 @@ Http::One::Parser::cleanMimePrefix() - void - Http::One::Parser::unfoldMime() - { -- Http1::Tokenizer tok(mimeHeaderBlock_); -+ Tokenizer tok(mimeHeaderBlock_); - const auto szLimit = mimeHeaderBlock_.length(); - mimeHeaderBlock_.clear(); - // prevent the mime sender being able to make append() realloc/grow multiple times. -@@ -228,7 +228,7 @@ Http::One::Parser::getHostHeaderField() - debugs(25, 5, "looking for " << name); - - // while we can find more LF in the SBuf -- Http1::Tokenizer tok(mimeHeaderBlock_); -+ Tokenizer tok(mimeHeaderBlock_); - SBuf p; - - while (tok.prefix(p, LineCharacters())) { -@@ -250,7 +250,7 @@ Http::One::Parser::getHostHeaderField() - p.consume(namelen + 1); - - // TODO: optimize SBuf::trim to take CharacterSet directly -- Http1::Tokenizer t(p); -+ Tokenizer t(p); - t.skipAll(CharacterSet::WSP); - p = t.remaining(); - -@@ -278,10 +278,15 @@ Http::One::ErrorLevel() - } - - // BWS = *( SP / HTAB ) ; WhitespaceCharacters() may relax this RFC 7230 rule --bool --Http::One::ParseBws(Tokenizer &tok) -+void -+Http::One::ParseBws(Parser::Tokenizer &tok) - { -- if (const auto count = tok.skipAll(Parser::WhitespaceCharacters())) { -+ const auto count = tok.skipAll(Parser::WhitespaceCharacters()); -+ -+ if (tok.atEnd()) -+ throw InsufficientInput(); // even if count is positive -+ -+ if (count) { - // Generating BWS is a MUST-level violation so warn about it as needed. - debugs(33, ErrorLevel(), "found " << count << " BWS octets"); - // RFC 7230 says we MUST parse BWS, so we fall through even if -@@ -289,6 +294,6 @@ Http::One::ParseBws(Tokenizer &tok) - } - // else we successfully "parsed" an empty BWS sequence - -- return true; -+ // success: no more BWS characters expected - } - -diff --git a/src/http/one/Parser.h b/src/http/one/Parser.h -index 58a5cae..40e281b 100644 ---- a/src/http/one/Parser.h -+++ b/src/http/one/Parser.h -@@ -12,6 +12,7 @@ - #include "anyp/ProtocolVersion.h" - #include "http/one/forward.h" - #include "http/StatusCode.h" -+#include "parser/forward.h" - #include "sbuf/SBuf.h" - - namespace Http { -@@ -40,6 +41,7 @@ class Parser : public RefCountable - { - public: - typedef SBuf::size_type size_type; -+ typedef ::Parser::Tokenizer Tokenizer; - - Parser() : parseStatusCode(Http::scNone), parsingStage_(HTTP_PARSE_NONE), hackExpectsMime_(false) {} - virtual ~Parser() {} -@@ -118,11 +120,11 @@ protected: - * detect and skip the CRLF or (if tolerant) LF line terminator - * consume from the tokenizer. - * -- * throws if non-terminator is detected. -+ * \throws exception on bad or InsuffientInput. - * \retval true only if line terminator found. - * \retval false incomplete or missing line terminator, need more data. - */ -- bool skipLineTerminator(Http1::Tokenizer &tok) const; -+ void skipLineTerminator(Tokenizer &) const; - - /** - * Scan to find the mime headers block for current message. -@@ -159,8 +161,8 @@ private: - }; - - /// skips and, if needed, warns about RFC 7230 BWS ("bad" whitespace) --/// \returns true (always; unlike all the skip*() functions) --bool ParseBws(Tokenizer &tok); -+/// \throws InsufficientInput when the end of BWS cannot be confirmed -+void ParseBws(Parser::Tokenizer &); - - /// the right debugs() level for logging HTTP violation messages - int ErrorLevel(); -diff --git a/src/http/one/RequestParser.cc b/src/http/one/RequestParser.cc -index a325f7d..0f13c92 100644 ---- a/src/http/one/RequestParser.cc -+++ b/src/http/one/RequestParser.cc -@@ -9,8 +9,8 @@ - #include "squid.h" - #include "Debug.h" - #include "http/one/RequestParser.h" --#include "http/one/Tokenizer.h" - #include "http/ProtocolVersion.h" -+#include "parser/Tokenizer.h" - #include "profiler/Profiler.h" - #include "SquidConfig.h" - -@@ -64,7 +64,7 @@ Http::One::RequestParser::skipGarbageLines() - * RFC 7230 section 2.6, 3.1 and 3.5 - */ - bool --Http::One::RequestParser::parseMethodField(Http1::Tokenizer &tok) -+Http::One::RequestParser::parseMethodField(Tokenizer &tok) - { - // method field is a sequence of TCHAR. - // Limit to 32 characters to prevent overly long sequences of non-HTTP -@@ -145,7 +145,7 @@ Http::One::RequestParser::RequestTargetCharacters() - } - - bool --Http::One::RequestParser::parseUriField(Http1::Tokenizer &tok) -+Http::One::RequestParser::parseUriField(Tokenizer &tok) - { - /* Arbitrary 64KB URI upper length limit. - * -@@ -178,7 +178,7 @@ Http::One::RequestParser::parseUriField(Http1::Tokenizer &tok) - } - - bool --Http::One::RequestParser::parseHttpVersionField(Http1::Tokenizer &tok) -+Http::One::RequestParser::parseHttpVersionField(Tokenizer &tok) - { - static const SBuf http1p0("HTTP/1.0"); - static const SBuf http1p1("HTTP/1.1"); -@@ -253,7 +253,7 @@ Http::One::RequestParser::skipDelimiter(const size_t count, const char *where) - - /// Parse CRs at the end of request-line, just before the terminating LF. - bool --Http::One::RequestParser::skipTrailingCrs(Http1::Tokenizer &tok) -+Http::One::RequestParser::skipTrailingCrs(Tokenizer &tok) - { - if (Config.onoff.relaxed_header_parser) { - (void)tok.skipAllTrailing(CharacterSet::CR); // optional; multiple OK -@@ -289,12 +289,12 @@ Http::One::RequestParser::parseRequestFirstLine() - // Earlier, skipGarbageLines() took care of any leading LFs (if allowed). - // Now, the request line has to end at the first LF. - static const CharacterSet lineChars = CharacterSet::LF.complement("notLF"); -- ::Parser::Tokenizer lineTok(buf_); -+ Tokenizer lineTok(buf_); - if (!lineTok.prefix(line, lineChars) || !lineTok.skip('\n')) { - if (buf_.length() >= Config.maxRequestHeaderSize) { - /* who should we blame for our failure to parse this line? */ - -- Http1::Tokenizer methodTok(buf_); -+ Tokenizer methodTok(buf_); - if (!parseMethodField(methodTok)) - return -1; // blame a bad method (or its delimiter) - -@@ -308,7 +308,7 @@ Http::One::RequestParser::parseRequestFirstLine() - return 0; - } - -- Http1::Tokenizer tok(line); -+ Tokenizer tok(line); - - if (!parseMethodField(tok)) - return -1; -diff --git a/src/http/one/RequestParser.h b/src/http/one/RequestParser.h -index 7086548..26697cd 100644 ---- a/src/http/one/RequestParser.h -+++ b/src/http/one/RequestParser.h -@@ -54,11 +54,11 @@ private: - bool doParse(const SBuf &aBuf); - - /* all these return false and set parseStatusCode on parsing failures */ -- bool parseMethodField(Http1::Tokenizer &); -- bool parseUriField(Http1::Tokenizer &); -- bool parseHttpVersionField(Http1::Tokenizer &); -+ bool parseMethodField(Tokenizer &); -+ bool parseUriField(Tokenizer &); -+ bool parseHttpVersionField(Tokenizer &); - bool skipDelimiter(const size_t count, const char *where); -- bool skipTrailingCrs(Http1::Tokenizer &tok); -+ bool skipTrailingCrs(Tokenizer &tok); - - bool http0() const {return !msgProtocol_.major;} - static const CharacterSet &RequestTargetCharacters(); -diff --git a/src/http/one/ResponseParser.cc b/src/http/one/ResponseParser.cc -index 24af849..65baf09 100644 ---- a/src/http/one/ResponseParser.cc -+++ b/src/http/one/ResponseParser.cc -@@ -9,8 +9,8 @@ - #include "squid.h" - #include "Debug.h" - #include "http/one/ResponseParser.h" --#include "http/one/Tokenizer.h" - #include "http/ProtocolVersion.h" -+#include "parser/Tokenizer.h" - #include "profiler/Profiler.h" - #include "SquidConfig.h" - -@@ -47,7 +47,7 @@ Http::One::ResponseParser::firstLineSize() const - // NP: we found the protocol version and consumed it already. - // just need the status code and reason phrase - int --Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, const CharacterSet &WspDelim) -+Http::One::ResponseParser::parseResponseStatusAndReason(Tokenizer &tok, const CharacterSet &WspDelim) - { - if (!completedStatus_) { - debugs(74, 9, "seek status-code in: " << tok.remaining().substr(0,10) << "..."); -@@ -87,14 +87,13 @@ Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, c - static const CharacterSet phraseChars = CharacterSet::WSP + CharacterSet::VCHAR + CharacterSet::OBSTEXT; - (void)tok.prefix(reasonPhrase_, phraseChars); // optional, no error if missing - try { -- if (skipLineTerminator(tok)) { -- debugs(74, DBG_DATA, "parse remaining buf={length=" << tok.remaining().length() << ", data='" << tok.remaining() << "'}"); -- buf_ = tok.remaining(); // resume checkpoint -- return 1; -- } -+ skipLineTerminator(tok); -+ buf_ = tok.remaining(); // resume checkpoint -+ debugs(74, DBG_DATA, Raw("leftovers", buf_.rawContent(), buf_.length())); -+ return 1; -+ } catch (const InsufficientInput &) { - reasonPhrase_.clear(); - return 0; // need more to be sure we have it all -- - } catch (const std::exception &ex) { - debugs(74, 6, "invalid status-line: " << ex.what()); - } -@@ -119,7 +118,7 @@ Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, c - int - Http::One::ResponseParser::parseResponseFirstLine() - { -- Http1::Tokenizer tok(buf_); -+ Tokenizer tok(buf_); - - const CharacterSet &WspDelim = DelimiterCharacters(); - -diff --git a/src/http/one/ResponseParser.h b/src/http/one/ResponseParser.h -index 15db4a0..cf13b4d 100644 ---- a/src/http/one/ResponseParser.h -+++ b/src/http/one/ResponseParser.h -@@ -43,7 +43,7 @@ public: - - private: - int parseResponseFirstLine(); -- int parseResponseStatusAndReason(Http1::Tokenizer&, const CharacterSet &); -+ int parseResponseStatusAndReason(Tokenizer&, const CharacterSet &); - - /// magic prefix for identifying ICY response messages - static const SBuf IcyMagic; -diff --git a/src/http/one/TeChunkedParser.cc b/src/http/one/TeChunkedParser.cc -index 754086e..6d2f8ea 100644 ---- a/src/http/one/TeChunkedParser.cc -+++ b/src/http/one/TeChunkedParser.cc -@@ -13,10 +13,13 @@ - #include "http/one/Tokenizer.h" - #include "http/ProtocolVersion.h" - #include "MemBuf.h" -+#include "parser/Tokenizer.h" - #include "Parsing.h" -+#include "sbuf/Stream.h" - #include "SquidConfig.h" - --Http::One::TeChunkedParser::TeChunkedParser() -+Http::One::TeChunkedParser::TeChunkedParser(): -+ customExtensionValueParser(nullptr) - { - // chunked encoding only exists in HTTP/1.1 - Http1::Parser::msgProtocol_ = Http::ProtocolVersion(1,1); -@@ -31,7 +34,11 @@ Http::One::TeChunkedParser::clear() - buf_.clear(); - theChunkSize = theLeftBodySize = 0; - theOut = NULL; -- useOriginBody = -1; -+ // XXX: We do not reset customExtensionValueParser here. Based on the -+ // clear() API description, we must, but it makes little sense and could -+ // break method callers if they appear because some of them may forget to -+ // reset customExtensionValueParser. TODO: Remove Http1::Parser as our -+ // parent class and this unnecessary method with it. - } - - bool -@@ -49,14 +56,14 @@ Http::One::TeChunkedParser::parse(const SBuf &aBuf) - if (parsingStage_ == Http1::HTTP_PARSE_NONE) - parsingStage_ = Http1::HTTP_PARSE_CHUNK_SZ; - -- Http1::Tokenizer tok(buf_); -+ Tokenizer tok(buf_); - - // loop for as many chunks as we can - // use do-while instead of while so that we can incrementally - // restart in the middle of a chunk/frame - do { - -- if (parsingStage_ == Http1::HTTP_PARSE_CHUNK_EXT && !parseChunkExtension(tok, theChunkSize)) -+ if (parsingStage_ == Http1::HTTP_PARSE_CHUNK_EXT && !parseChunkMetadataSuffix(tok)) - return false; - - if (parsingStage_ == Http1::HTTP_PARSE_CHUNK && !parseChunkBody(tok)) -@@ -80,7 +87,7 @@ Http::One::TeChunkedParser::needsMoreSpace() const - - /// RFC 7230 section 4.1 chunk-size - bool --Http::One::TeChunkedParser::parseChunkSize(Http1::Tokenizer &tok) -+Http::One::TeChunkedParser::parseChunkSize(Tokenizer &tok) - { - Must(theChunkSize <= 0); // Should(), really - -@@ -104,66 +111,75 @@ Http::One::TeChunkedParser::parseChunkSize(Http1::Tokenizer &tok) - return false; // should not be reachable - } - --/** -- * Parses chunk metadata suffix, looking for interesting extensions and/or -- * getting to the line terminator. RFC 7230 section 4.1.1 and its Errata #4667: -- * -- * chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] ) -- * chunk-ext-name = token -- * chunk-ext-val = token / quoted-string -- * -- * ICAP 'use-original-body=N' extension is supported. -- */ -+/// Parses "[chunk-ext] CRLF" from RFC 7230 section 4.1.1: -+/// chunk = chunk-size [ chunk-ext ] CRLF chunk-data CRLF -+/// last-chunk = 1*"0" [ chunk-ext ] CRLF - bool --Http::One::TeChunkedParser::parseChunkExtension(Http1::Tokenizer &tok, bool skipKnown) -+Http::One::TeChunkedParser::parseChunkMetadataSuffix(Tokenizer &tok) - { -- SBuf ext; -- SBuf value; -- while ( -- ParseBws(tok) && // Bug 4492: IBM_HTTP_Server sends SP after chunk-size -- tok.skip(';') && -- ParseBws(tok) && // Bug 4492: ICAP servers send SP before chunk-ext-name -- tok.prefix(ext, CharacterSet::TCHAR)) { // chunk-ext-name -- -- // whole value part is optional. if no '=' expect next chunk-ext -- if (ParseBws(tok) && tok.skip('=') && ParseBws(tok)) { -- -- if (!skipKnown) { -- if (ext.cmp("use-original-body",17) == 0 && tok.int64(useOriginBody, 10)) { -- debugs(94, 3, "Found chunk extension " << ext << "=" << useOriginBody); -- buf_ = tok.remaining(); // parse checkpoint -- continue; -- } -- } -- -- debugs(94, 5, "skipping unknown chunk extension " << ext); -- -- // unknown might have a value token or quoted-string -- if (tok.quotedStringOrToken(value) && !tok.atEnd()) { -- buf_ = tok.remaining(); // parse checkpoint -- continue; -- } -- -- // otherwise need more data OR corrupt syntax -- break; -- } -- -- if (!tok.atEnd()) -- buf_ = tok.remaining(); // parse checkpoint (unless there might be more token name) -- } -- -- if (skipLineTerminator(tok)) { -- buf_ = tok.remaining(); // checkpoint -- // non-0 chunk means data, 0-size means optional Trailer follows -+ // Code becomes much simpler when incremental parsing functions throw on -+ // bad or insufficient input, like in the code below. TODO: Expand up. -+ try { -+ parseChunkExtensions(tok); // a possibly empty chunk-ext list -+ skipLineTerminator(tok); -+ buf_ = tok.remaining(); - parsingStage_ = theChunkSize ? Http1::HTTP_PARSE_CHUNK : Http1::HTTP_PARSE_MIME; - return true; -+ } catch (const InsufficientInput &) { -+ tok.reset(buf_); // backtrack to the last commit point -+ return false; - } -+ // other exceptions bubble up to kill message parsing -+} - -- return false; -+/// Parses the chunk-ext list (RFC 7230 section 4.1.1 and its Errata #4667): -+/// chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] ) -+void -+Http::One::TeChunkedParser::parseChunkExtensions(Tokenizer &tok) -+{ -+ do { -+ ParseBws(tok); // Bug 4492: IBM_HTTP_Server sends SP after chunk-size -+ -+ if (!tok.skip(';')) -+ return; // reached the end of extensions (if any) -+ -+ parseOneChunkExtension(tok); -+ buf_ = tok.remaining(); // got one extension -+ } while (true); -+} -+ -+void -+Http::One::ChunkExtensionValueParser::Ignore(Tokenizer &tok, const SBuf &extName) -+{ -+ const auto ignoredValue = tokenOrQuotedString(tok); -+ debugs(94, 5, extName << " with value " << ignoredValue); -+} -+ -+/// Parses a single chunk-ext list element: -+/// chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] ) -+void -+Http::One::TeChunkedParser::parseOneChunkExtension(Tokenizer &tok) -+{ -+ ParseBws(tok); // Bug 4492: ICAP servers send SP before chunk-ext-name -+ -+ const auto extName = tok.prefix("chunk-ext-name", CharacterSet::TCHAR); -+ -+ ParseBws(tok); -+ -+ if (!tok.skip('=')) -+ return; // parsed a valueless chunk-ext -+ -+ ParseBws(tok); -+ -+ // optimization: the only currently supported extension needs last-chunk -+ if (!theChunkSize && customExtensionValueParser) -+ customExtensionValueParser->parse(tok, extName); -+ else -+ ChunkExtensionValueParser::Ignore(tok, extName); - } - - bool --Http::One::TeChunkedParser::parseChunkBody(Http1::Tokenizer &tok) -+Http::One::TeChunkedParser::parseChunkBody(Tokenizer &tok) - { - if (theLeftBodySize > 0) { - buf_ = tok.remaining(); // sync buffers before buf_ use -@@ -188,17 +204,20 @@ Http::One::TeChunkedParser::parseChunkBody(Http1::Tokenizer &tok) - } - - bool --Http::One::TeChunkedParser::parseChunkEnd(Http1::Tokenizer &tok) -+Http::One::TeChunkedParser::parseChunkEnd(Tokenizer &tok) - { - Must(theLeftBodySize == 0); // Should(), really - -- if (skipLineTerminator(tok)) { -+ try { -+ skipLineTerminator(tok); - buf_ = tok.remaining(); // parse checkpoint - theChunkSize = 0; // done with the current chunk - parsingStage_ = Http1::HTTP_PARSE_CHUNK_SZ; - return true; - } -- -- return false; -+ catch (const InsufficientInput &) { -+ return false; -+ } -+ // other exceptions bubble up to kill message parsing - } - -diff --git a/src/http/one/TeChunkedParser.h b/src/http/one/TeChunkedParser.h -index 1b0319e..2ca8988 100644 ---- a/src/http/one/TeChunkedParser.h -+++ b/src/http/one/TeChunkedParser.h -@@ -18,6 +18,26 @@ namespace Http - namespace One - { - -+using ::Parser::InsufficientInput; -+ -+// TODO: Move this class into http/one/ChunkExtensionValueParser.* -+/// A customizable parser of a single chunk extension value (chunk-ext-val). -+/// From RFC 7230 section 4.1.1 and its Errata #4667: -+/// chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] ) -+/// chunk-ext-name = token -+/// chunk-ext-val = token / quoted-string -+class ChunkExtensionValueParser -+{ -+public: -+ typedef ::Parser::Tokenizer Tokenizer; -+ -+ /// extracts and ignores the value of a named extension -+ static void Ignore(Tokenizer &tok, const SBuf &extName); -+ -+ /// extracts and then interprets (or ignores) the extension value -+ virtual void parse(Tokenizer &tok, const SBuf &extName) = 0; -+}; -+ - /** - * An incremental parser for chunked transfer coding - * defined in RFC 7230 section 4.1. -@@ -25,7 +45,7 @@ namespace One - * - * The parser shovels content bytes from the raw - * input buffer into the content output buffer, both caller-supplied. -- * Ignores chunk extensions except for ICAP's ieof. -+ * Chunk extensions like use-original-body are handled via parseExtensionValuesWith(). - * Trailers are available via mimeHeader() if wanted. - */ - class TeChunkedParser : public Http1::Parser -@@ -37,6 +57,10 @@ public: - /// set the buffer to be used to store decoded chunk data - void setPayloadBuffer(MemBuf *parsedContent) {theOut = parsedContent;} - -+ /// Instead of ignoring all chunk extension values, give the supplied -+ /// parser a chance to handle them. Only applied to last-chunk (for now). -+ void parseExtensionValuesWith(ChunkExtensionValueParser *parser) { customExtensionValueParser = parser; } -+ - bool needsMoreSpace() const; - - /* Http1::Parser API */ -@@ -45,17 +69,20 @@ public: - virtual Parser::size_type firstLineSize() const {return 0;} // has no meaning with multiple chunks - - private: -- bool parseChunkSize(Http1::Tokenizer &tok); -- bool parseChunkExtension(Http1::Tokenizer &tok, bool skipKnown); -- bool parseChunkBody(Http1::Tokenizer &tok); -- bool parseChunkEnd(Http1::Tokenizer &tok); -+ bool parseChunkSize(Tokenizer &tok); -+ bool parseChunkMetadataSuffix(Tokenizer &); -+ void parseChunkExtensions(Tokenizer &); -+ void parseOneChunkExtension(Tokenizer &); -+ bool parseChunkBody(Tokenizer &tok); -+ bool parseChunkEnd(Tokenizer &tok); - - MemBuf *theOut; - uint64_t theChunkSize; - uint64_t theLeftBodySize; - --public: -- int64_t useOriginBody; -+ /// An optional plugin for parsing and interpreting custom chunk-ext-val. -+ /// This "visitor" object is owned by our creator. -+ ChunkExtensionValueParser *customExtensionValueParser; - }; - - } // namespace One -diff --git a/src/http/one/Tokenizer.cc b/src/http/one/Tokenizer.cc -index 804b8e1..3a6bef3 100644 ---- a/src/http/one/Tokenizer.cc -+++ b/src/http/one/Tokenizer.cc -@@ -8,35 +8,18 @@ - - #include "squid.h" - #include "Debug.h" -+#include "http/one/Parser.h" - #include "http/one/Tokenizer.h" -- --bool --Http::One::Tokenizer::quotedString(SBuf &returnedToken, const bool http1p0) --{ -- checkpoint(); -- -- if (!skip('"')) -- return false; -- -- return qdText(returnedToken, http1p0); --} -- --bool --Http::One::Tokenizer::quotedStringOrToken(SBuf &returnedToken, const bool http1p0) -+#include "parser/Tokenizer.h" -+#include "sbuf/Stream.h" -+ -+/// Extracts quoted-string after the caller removes the initial '"'. -+/// \param http1p0 whether to prohibit \-escaped characters in quoted strings -+/// \throws InsufficientInput when input can be a token _prefix_ -+/// \returns extracted quoted string (without quotes and with chars unescaped) -+static SBuf -+parseQuotedStringSuffix(Parser::Tokenizer &tok, const bool http1p0) - { -- checkpoint(); -- -- if (!skip('"')) -- return prefix(returnedToken, CharacterSet::TCHAR); -- -- return qdText(returnedToken, http1p0); --} -- --bool --Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0) --{ -- // the initial DQUOTE has been skipped by the caller -- - /* - * RFC 1945 - defines qdtext: - * inclusive of LWS (which includes CR and LF) -@@ -61,12 +44,17 @@ Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0) - // best we can do is a conditional reference since http1p0 value may change per-client - const CharacterSet &tokenChars = (http1p0 ? qdtext1p0 : qdtext1p1); - -- for (;;) { -- SBuf::size_type prefixLen = buf().findFirstNotOf(tokenChars); -- returnedToken.append(consume(prefixLen)); -+ SBuf parsedToken; -+ -+ while (!tok.atEnd()) { -+ SBuf qdText; -+ if (tok.prefix(qdText, tokenChars)) -+ parsedToken.append(qdText); -+ -+ if (!http1p0 && tok.skip('\\')) { // HTTP/1.1 allows quoted-pair, HTTP/1.0 does not -+ if (tok.atEnd()) -+ break; - -- // HTTP/1.1 allows quoted-pair, HTTP/1.0 does not -- if (!http1p0 && skip('\\')) { - /* RFC 7230 section 3.2.6 - * - * The backslash octet ("\") can be used as a single-octet quoting -@@ -78,32 +66,42 @@ Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0) - */ - static const CharacterSet qPairChars = CharacterSet::HTAB + CharacterSet::SP + CharacterSet::VCHAR + CharacterSet::OBSTEXT; - SBuf escaped; -- if (!prefix(escaped, qPairChars, 1)) { -- returnedToken.clear(); -- restoreLastCheckpoint(); -- return false; -- } -- returnedToken.append(escaped); -+ if (!tok.prefix(escaped, qPairChars, 1)) -+ throw TexcHere("invalid escaped character in quoted-pair"); -+ -+ parsedToken.append(escaped); - continue; -+ } - -- } else if (skip('"')) { -- break; // done -+ if (tok.skip('"')) -+ return parsedToken; // may be empty - -- } else if (atEnd()) { -- // need more data -- returnedToken.clear(); -- restoreLastCheckpoint(); -- return false; -- } -+ if (tok.atEnd()) -+ break; - -- // else, we have an error -- debugs(24, 8, "invalid bytes for set " << tokenChars.name); -- returnedToken.clear(); -- restoreLastCheckpoint(); -- return false; -+ throw TexcHere(ToSBuf("invalid bytes for set ", tokenChars.name)); - } - -- // found the whole string -- return true; -+ throw Http::One::InsufficientInput(); -+} -+ -+SBuf -+Http::One::tokenOrQuotedString(Parser::Tokenizer &tok, const bool http1p0) -+{ -+ if (tok.skip('"')) -+ return parseQuotedStringSuffix(tok, http1p0); -+ -+ if (tok.atEnd()) -+ throw InsufficientInput(); -+ -+ SBuf parsedToken; -+ if (!tok.prefix(parsedToken, CharacterSet::TCHAR)) -+ throw TexcHere("invalid input while expecting an HTTP token"); -+ -+ if (tok.atEnd()) -+ throw InsufficientInput(); -+ -+ // got the complete token -+ return parsedToken; - } - -diff --git a/src/http/one/Tokenizer.h b/src/http/one/Tokenizer.h -index 658875f..2d40574 100644 ---- a/src/http/one/Tokenizer.h -+++ b/src/http/one/Tokenizer.h -@@ -9,68 +9,47 @@ - #ifndef SQUID_SRC_HTTP_ONE_TOKENIZER_H - #define SQUID_SRC_HTTP_ONE_TOKENIZER_H - --#include "parser/Tokenizer.h" -+#include "parser/forward.h" -+#include "sbuf/forward.h" - - namespace Http { - namespace One { - - /** -- * Lexical processor extended to tokenize HTTP/1.x syntax. -+ * Extracts either an HTTP/1 token or quoted-string while dealing with -+ * possibly incomplete input typical for incremental text parsers. -+ * Unescapes escaped characters in HTTP/1.1 quoted strings. - * -- * \see ::Parser::Tokenizer for more detail -+ * \param http1p0 whether to prohibit \-escaped characters in quoted strings -+ * \throws InsufficientInput as appropriate, including on unterminated tokens -+ * \returns extracted token or quoted string (without quotes) -+ * -+ * Governed by: -+ * - RFC 1945 section 2.1 -+ * " -+ * A string of text is parsed as a single word if it is quoted using -+ * double-quote marks. -+ * -+ * quoted-string = ( <"> *(qdtext) <"> ) -+ * -+ * qdtext = and CTLs, -+ * but including LWS> -+ * -+ * Single-character quoting using the backslash ("\") character is not -+ * permitted in HTTP/1.0. -+ * " -+ * -+ * - RFC 7230 section 3.2.6 -+ * " -+ * A string of text is parsed as a single value if it is quoted using -+ * double-quote marks. -+ * -+ * quoted-string = DQUOTE *( qdtext / quoted-pair ) DQUOTE -+ * qdtext = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text -+ * obs-text = %x80-FF -+ * " - */ --class Tokenizer : public ::Parser::Tokenizer --{ --public: -- Tokenizer(SBuf &s) : ::Parser::Tokenizer(s), savedStats_(0) {} -- -- /** -- * Attempt to parse a quoted-string lexical construct. -- * -- * Governed by: -- * - RFC 1945 section 2.1 -- * " -- * A string of text is parsed as a single word if it is quoted using -- * double-quote marks. -- * -- * quoted-string = ( <"> *(qdtext) <"> ) -- * -- * qdtext = and CTLs, -- * but including LWS> -- * -- * Single-character quoting using the backslash ("\") character is not -- * permitted in HTTP/1.0. -- * " -- * -- * - RFC 7230 section 3.2.6 -- * " -- * A string of text is parsed as a single value if it is quoted using -- * double-quote marks. -- * -- * quoted-string = DQUOTE *( qdtext / quoted-pair ) DQUOTE -- * qdtext = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text -- * obs-text = %x80-FF -- * " -- * -- * \param escaped HTTP/1.0 does not permit \-escaped characters -- */ -- bool quotedString(SBuf &value, const bool http1p0 = false); -- -- /** -- * Attempt to parse a (token / quoted-string ) lexical construct. -- */ -- bool quotedStringOrToken(SBuf &value, const bool http1p0 = false); -- --private: -- /// parse the internal component of a quote-string, and terminal DQUOTE -- bool qdText(SBuf &value, const bool http1p0); -- -- void checkpoint() { savedCheckpoint_ = buf(); savedStats_ = parsedSize(); } -- void restoreLastCheckpoint() { undoParse(savedCheckpoint_, savedStats_); } -- -- SBuf savedCheckpoint_; -- SBuf::size_type savedStats_; --}; -+SBuf tokenOrQuotedString(Parser::Tokenizer &tok, const bool http1p0 = false); - - } // namespace One - } // namespace Http -diff --git a/src/http/one/forward.h b/src/http/one/forward.h -index c90dc34..2b4ad28 100644 ---- a/src/http/one/forward.h -+++ b/src/http/one/forward.h -@@ -10,6 +10,7 @@ - #define SQUID_SRC_HTTP_ONE_FORWARD_H - - #include "base/RefCount.h" -+#include "parser/forward.h" - #include "sbuf/forward.h" - - namespace Http { -@@ -31,6 +32,8 @@ typedef RefCount ResponseParserPointer; - /// CRLF textual representation - const SBuf &CrLf(); - -+using ::Parser::InsufficientInput; -+ - } // namespace One - } // namespace Http - -diff --git a/src/parser/BinaryTokenizer.h b/src/parser/BinaryTokenizer.h -index acebd4d..24042d4 100644 ---- a/src/parser/BinaryTokenizer.h -+++ b/src/parser/BinaryTokenizer.h -@@ -9,6 +9,7 @@ - #ifndef SQUID_SRC_PARSER_BINARYTOKENIZER_H - #define SQUID_SRC_PARSER_BINARYTOKENIZER_H - -+#include "parser/forward.h" - #include "sbuf/SBuf.h" - - namespace Parser -@@ -44,7 +45,7 @@ public: - class BinaryTokenizer - { - public: -- class InsufficientInput {}; // thrown when a method runs out of data -+ typedef ::Parser::InsufficientInput InsufficientInput; - typedef uint64_t size_type; // enough for the largest supported offset - - BinaryTokenizer(); -diff --git a/src/parser/Makefile.am b/src/parser/Makefile.am -index af2b759..0daa5a8 100644 ---- a/src/parser/Makefile.am -+++ b/src/parser/Makefile.am -@@ -13,6 +13,7 @@ noinst_LTLIBRARIES = libparser.la - libparser_la_SOURCES = \ - BinaryTokenizer.h \ - BinaryTokenizer.cc \ -+ forward.h \ - Tokenizer.h \ - Tokenizer.cc - -diff --git a/src/parser/Tokenizer.cc b/src/parser/Tokenizer.cc -index 7e73e04..68f4aec 100644 ---- a/src/parser/Tokenizer.cc -+++ b/src/parser/Tokenizer.cc -@@ -10,7 +10,9 @@ - - #include "squid.h" - #include "Debug.h" -+#include "parser/forward.h" - #include "parser/Tokenizer.h" -+#include "sbuf/Stream.h" - - #include - #if HAVE_CTYPE_H -@@ -96,6 +98,23 @@ Parser::Tokenizer::prefix(SBuf &returnedToken, const CharacterSet &tokenChars, c - return true; - } - -+SBuf -+Parser::Tokenizer::prefix(const char *description, const CharacterSet &tokenChars, const SBuf::size_type limit) -+{ -+ if (atEnd()) -+ throw InsufficientInput(); -+ -+ SBuf result; -+ -+ if (!prefix(result, tokenChars, limit)) -+ throw TexcHere(ToSBuf("cannot parse ", description)); -+ -+ if (atEnd()) -+ throw InsufficientInput(); -+ -+ return result; -+} -+ - bool - Parser::Tokenizer::suffix(SBuf &returnedToken, const CharacterSet &tokenChars, const SBuf::size_type limit) - { -@@ -283,3 +302,24 @@ Parser::Tokenizer::int64(int64_t & result, int base, bool allowSign, const SBuf: - return success(s - range.rawContent()); - } - -+int64_t -+Parser::Tokenizer::udec64(const char *description, const SBuf::size_type limit) -+{ -+ if (atEnd()) -+ throw InsufficientInput(); -+ -+ int64_t result = 0; -+ -+ // Since we only support unsigned decimals, a parsing failure with a -+ // non-empty input always implies invalid/malformed input (or a buggy -+ // limit=0 caller). TODO: Support signed and non-decimal integers by -+ // refactoring int64() to detect insufficient input. -+ if (!int64(result, 10, false, limit)) -+ throw TexcHere(ToSBuf("cannot parse ", description)); -+ -+ if (atEnd()) -+ throw InsufficientInput(); // more digits may be coming -+ -+ return result; -+} -+ -diff --git a/src/parser/Tokenizer.h b/src/parser/Tokenizer.h -index 54414be..03a8388 100644 ---- a/src/parser/Tokenizer.h -+++ b/src/parser/Tokenizer.h -@@ -143,6 +143,19 @@ public: - */ - bool int64(int64_t &result, int base = 0, bool allowSign = true, SBuf::size_type limit = SBuf::npos); - -+ /* -+ * The methods below mimic their counterparts documented above, but they -+ * throw on errors, including InsufficientInput. The field description -+ * parameter is used for error reporting and debugging. -+ */ -+ -+ /// prefix() wrapper but throws InsufficientInput if input contains -+ /// nothing but the prefix (i.e. if the prefix is not "terminated") -+ SBuf prefix(const char *description, const CharacterSet &tokenChars, SBuf::size_type limit = SBuf::npos); -+ -+ /// int64() wrapper but limited to unsigned decimal integers (for now) -+ int64_t udec64(const char *description, SBuf::size_type limit = SBuf::npos); -+ - protected: - SBuf consume(const SBuf::size_type n); - SBuf::size_type success(const SBuf::size_type n); -diff --git a/src/parser/forward.h b/src/parser/forward.h -new file mode 100644 -index 0000000..5a95b7a ---- /dev/null -+++ b/src/parser/forward.h -@@ -0,0 +1,22 @@ -+/* -+ * Copyright (C) 1996-2019 The Squid Software Foundation and contributors -+ * -+ * Squid software is distributed under GPLv2+ license and includes -+ * contributions from numerous individuals and organizations. -+ * Please see the COPYING and CONTRIBUTORS files for details. -+ */ -+ -+#ifndef SQUID_PARSER_FORWARD_H -+#define SQUID_PARSER_FORWARD_H -+ -+namespace Parser { -+class Tokenizer; -+class BinaryTokenizer; -+ -+// TODO: Move this declaration (to parser/Elements.h) if we need more like it. -+/// thrown by modern "incremental" parsers when they need more data -+class InsufficientInput {}; -+} // namespace Parser -+ -+#endif /* SQUID_PARSER_FORWARD_H */ -+ --- -2.39.3 - diff --git a/SOURCES/0002-Remove-serialized-HTTP-headers-from-storeClientCopy.patch b/SOURCES/0002-Remove-serialized-HTTP-headers-from-storeClientCopy.patch new file mode 100644 index 0000000..76eac17 --- /dev/null +++ b/SOURCES/0002-Remove-serialized-HTTP-headers-from-storeClientCopy.patch @@ -0,0 +1,3388 @@ +From e4e1a48a6d53cad77c8bab561addb1ed48abba4f Mon Sep 17 00:00:00 2001 +From: Alex Rousskov +Date: Thu, 7 Dec 2023 17:58:49 +0000 +Subject: [PATCH 2/7] Remove serialized HTTP headers from storeClientCopy() + (#1335) + +Do not send serialized HTTP response header bytes in storeClientCopy() +answers. Ignore serialized header size when calling storeClientCopy(). + +This complex change adjusts storeClientCopy() API to addresses several +related problems with storeClientCopy() and its callers. The sections +below summarize storeClientCopy() changes and then move on to callers. + +Squid incorrectly assumed that serialized HTTP response headers are read +from disk in a single storeRead() request. In reality, many situations +lead to store_client::readBody() receiving partial HTTP headers, +resulting in parseCharBuf() failure and a level-0 cache.log message: + + Could not parse headers from on disk object + +Inadequate handling of this failure resulted in a variety of problems. +Squid now accumulates storeRead() results to parse larger headers and +also handles parsing failures better, but we could not just stop there. + +With the storeRead() accumulation in place, it is no longer possible to +send parsed serialized HTTP headers to storeClientCopy() callers because +those callers do not provide enough buffer space to fit larger headers. +Increasing caller buffer capacity does not work well because the actual +size of the serialized header is unknown in advance and may be quite +large. Always allocating large buffers "just in case" is bad for +performance. Finally, larger buffers may jeopardize hard-to-find code +that uses hard-coded 4KB buffers without using HTTP_REQBUF_SZ macro. + +Fortunately, storeClientCopy() callers either do not care about +serialized HTTP response headers or should not care about them! The API +forced callers to deal with serialized headers, but callers could (and +some did) just use the parsed headers available in the corresponding +MemObject. With this API change, storeClientCopy() callers no longer +receive serialized headers and do not need to parse or skip them. +Consequently, callers also do not need to account for response headers +size when computing offsets for subsequent storeClientCopy() requests. + +Restricting storeClientCopy() API to HTTP _body_ bytes removed a lot of +problematic caller code. Caller changes are summarized further below. + +A similar HTTP response header parsing problem existed in shared memory +cache code. That code was actually aware that headers may span multiple +cache slices but incorrectly assumed that httpMsgParseStep() accumulates +input as needed (to make another parsing "step"). It does not. Large +response headers cached in shared memory triggered a level-1 message: + + Corrupted mem-cached headers: e:... + +Fixed MemStore code now accumulates serialized HTTP response headers as +needed to parse them, sharing high-level parsing code with store_client. + +Old clientReplyContext methods worked hard to skip received serialized +HTTP headers. The code contained dangerous and often complex/unreadable +manipulation of various raw offsets and buffer pointers, aggravated by +the perceived need to save/restore those offsets across asynchronous +checks (see below). That header skipping code is gone now. Several stale +and misleading comments related to Store buffers management were also +removed or updated. + +We replaced reqofs/reqsize with simpler/safer lastStreamBufferedBytes, +while becoming more consistent with that "cached" info invalidation. We +still need this info to resume HTTP body processing after asynchronous +http_reply_access checks and cache hit validations, but we no longer +save/restore this info for hit validation: No need to save/restore +information about the buffer that hit validation does not use and must +never touch! + +The API change also moved from-Store StoreIOBuffer usage closer to +StoreIOBuffers manipulated by Clients Streams code. Buffers in both +categories now contain just the body bytes, and both now treat zero +length as EOF only _after_ processing the response headers. + +These changes improve overall code quality, but this code path and these +changes still suffer from utterly unsafe legacy interfaces like +StoreIOBuffer and clientStreamNode. We cannot rely on the compiler to +check our work. The risk of these changes exposing/causing bugs is high. + +asHandleReply() expected WHOIS response body bytes where serialized HTTP +headers were! The code also had multiple problems typical for manually +written C parsers dealing with raw input buffers. Now replaced with a +Tokenizer-based code. + +To skip received HTTP response headers, peerDigestHandleReply() helper +functions called headersEnd() on the received buffer. Twice. We have now +merged those two parsing helper functions into one (that just checks the +already parsed headers). This merger preserved "304s must come with +fetch->pd->cd" logic that was hidden/spread across those two functions. + +urnHandleReply() re-parsed received HTTP response headers. We left its +HTTP body parsing code unchanged except for polishing NUL-termination. + +netdbExchangeHandleReply() re-parsed received HTTP response headers to +find where they end (via headersEnd()). We improved handing of corner +cases and replaced some "tricky bits" code, reusing the new +Store::ParsingBuffer class. The net_db record parsing code is unchanged. +[root@ol8-gcc squid]# cat 0002-Remove-serialized-HTTP-headers-from-storeClientCopy-.patch |head -n 140 +From 8d50a09c3a0e9500becd21624ea62eb02660cc6d Mon Sep 17 00:00:00 2001 +From: Alex Rousskov +Date: Thu, 23 Nov 2023 18:26:33 +0000 +Subject: [PATCH 2/6] Remove serialized HTTP headers from storeClientCopy() + (#1335) + +Do not send serialized HTTP response header bytes in storeClientCopy() +answers. Ignore serialized header size when calling storeClientCopy(). + +This complex change adjusts storeClientCopy() API to addresses several +related problems with storeClientCopy() and its callers. The sections +below summarize storeClientCopy() changes and then move on to callers. + +Squid incorrectly assumed that serialized HTTP response headers are read +from disk in a single storeRead() request. In reality, many situations +lead to store_client::readBody() receiving partial HTTP headers, +resulting in parseCharBuf() failure and a level-0 cache.log message: + + Could not parse headers from on disk object + +Inadequate handling of this failure resulted in a variety of problems. +Squid now accumulates storeRead() results to parse larger headers and +also handles parsing failures better, but we could not just stop there. + +With the storeRead() accumulation in place, it is no longer possible to +send parsed serialized HTTP headers to storeClientCopy() callers because +those callers do not provide enough buffer space to fit larger headers. +Increasing caller buffer capacity does not work well because the actual +size of the serialized header is unknown in advance and may be quite +large. Always allocating large buffers "just in case" is bad for +performance. Finally, larger buffers may jeopardize hard-to-find code +that uses hard-coded 4KB buffers without using HTTP_REQBUF_SZ macro. + +Fortunately, storeClientCopy() callers either do not care about +serialized HTTP response headers or should not care about them! The API +forced callers to deal with serialized headers, but callers could (and +some did) just use the parsed headers available in the corresponding +MemObject. With this API change, storeClientCopy() callers no longer +receive serialized headers and do not need to parse or skip them. +Consequently, callers also do not need to account for response headers +size when computing offsets for subsequent storeClientCopy() requests. + +Restricting storeClientCopy() API to HTTP _body_ bytes removed a lot of +problematic caller code. Caller changes are summarized further below. + +A similar HTTP response header parsing problem existed in shared memory +cache code. That code was actually aware that headers may span multiple +cache slices but incorrectly assumed that httpMsgParseStep() accumulates +input as needed (to make another parsing "step"). It does not. Large +response headers cached in shared memory triggered a level-1 message: + + Corrupted mem-cached headers: e:... + +Fixed MemStore code now accumulates serialized HTTP response headers as +needed to parse them, sharing high-level parsing code with store_client. + +Old clientReplyContext methods worked hard to skip received serialized +HTTP headers. The code contained dangerous and often complex/unreadable +manipulation of various raw offsets and buffer pointers, aggravated by +the perceived need to save/restore those offsets across asynchronous +checks (see below). That header skipping code is gone now. Several stale +and misleading comments related to Store buffers management were also +removed or updated. + +We replaced reqofs/reqsize with simpler/safer lastStreamBufferedBytes, +while becoming more consistent with that "cached" info invalidation. We +still need this info to resume HTTP body processing after asynchronous +http_reply_access checks and cache hit validations, but we no longer +save/restore this info for hit validation: No need to save/restore +information about the buffer that hit validation does not use and must +never touch! + +The API change also moved from-Store StoreIOBuffer usage closer to +StoreIOBuffers manipulated by Clients Streams code. Buffers in both +categories now contain just the body bytes, and both now treat zero +length as EOF only _after_ processing the response headers. + +These changes improve overall code quality, but this code path and these +changes still suffer from utterly unsafe legacy interfaces like +StoreIOBuffer and clientStreamNode. We cannot rely on the compiler to +check our work. The risk of these changes exposing/causing bugs is high. + +asHandleReply() expected WHOIS response body bytes where serialized HTTP +headers were! The code also had multiple problems typical for manually +written C parsers dealing with raw input buffers. Now replaced with a +Tokenizer-based code. + +To skip received HTTP response headers, peerDigestHandleReply() helper +functions called headersEnd() on the received buffer. Twice. We have now +merged those two parsing helper functions into one (that just checks the +already parsed headers). This merger preserved "304s must come with +fetch->pd->cd" logic that was hidden/spread across those two functions. + +urnHandleReply() re-parsed received HTTP response headers. We left its +HTTP body parsing code unchanged except for polishing NUL-termination. + +netdbExchangeHandleReply() re-parsed received HTTP response headers to +find where they end (via headersEnd()). We improved handing of corner +cases and replaced some "tricky bits" code, reusing the new +Store::ParsingBuffer class. The net_db record parsing code is unchanged. + +Mgr::StoreToCommWriter::noteStoreCopied() is a very special case. It +actually worked OK because, unlike all other storeClientCopy() callers, +this code does not get serialized HTTP headers from Store: The code +adding bytes to the corresponding StoreEntry does not write serialized +HTTP headers at all. StoreToCommWriter is used to deliver kid-specific +pieces of an HTTP body of an SMP cache manager response. The HTTP +headers of that response are handled elsewhere. We left this code +unchanged, but the existence of the special no-headers case does +complicate storeClientCopy() API documentation, implementation, and +understanding. + +Co-authored-by: Eduard Bagdasaryan + +Modified-by: Alex Burmashev +Signed-off-by: Alex Burmashev +--- + src/HttpReply.cc | 34 +++ + src/HttpReply.h | 7 + + src/MemObject.cc | 6 + + src/MemObject.h | 9 + + src/MemStore.cc | 75 ++++--- + src/MemStore.h | 2 +- + src/StoreClient.h | 65 +++++- + src/StoreIOBuffer.h | 3 + + src/acl/Asn.cc | 163 +++++--------- + src/clientStream.cc | 3 +- + src/client_side_reply.cc | 322 +++++++++++---------------- + src/client_side_reply.h | 38 +++- + src/enums.h | 1 - + src/icmp/net_db.cc | 144 ++++-------- + src/peer_digest.cc | 96 ++------ + src/store.cc | 11 + + src/store/Makefile.am | 2 + + src/store/Makefile.in | 9 +- + src/store/ParsingBuffer.cc | 198 +++++++++++++++++ + src/store/ParsingBuffer.h | 128 +++++++++++ + src/store/forward.h | 1 + + src/store_client.cc | 429 ++++++++++++++++++++++++------------ + src/tests/stub_HttpReply.cc | 1 + + src/urn.cc | 89 +++----- + 24 files changed, 1094 insertions(+), 742 deletions(-) + create mode 100644 src/store/ParsingBuffer.cc + create mode 100644 src/store/ParsingBuffer.h + +diff --git a/src/HttpReply.cc b/src/HttpReply.cc +index 6feb262..af2bd4d 100644 +--- a/src/HttpReply.cc ++++ b/src/HttpReply.cc +@@ -20,7 +20,9 @@ + #include "HttpReply.h" + #include "HttpRequest.h" + #include "MemBuf.h" ++#include "sbuf/Stream.h" + #include "SquidConfig.h" ++#include "SquidMath.h" + #include "SquidTime.h" + #include "Store.h" + #include "StrList.h" +@@ -524,6 +526,38 @@ HttpReply::expectedBodyTooLarge(HttpRequest& request) + return expectedSize > bodySizeMax; + } + ++size_t ++HttpReply::parseTerminatedPrefix(const char * const terminatedBuf, const size_t bufSize) ++{ ++ auto error = Http::scNone; ++ const bool eof = false; // TODO: Remove after removing atEnd from HttpHeader::parse() ++ if (parse(terminatedBuf, bufSize, eof, &error)) { ++ debugs(58, 7, "success after accumulating " << bufSize << " bytes and parsing " << hdr_sz); ++ Assure(pstate == Http::Message::psParsed); ++ Assure(hdr_sz > 0); ++ Assure(!Less(bufSize, hdr_sz)); // cannot parse more bytes than we have ++ return hdr_sz; // success ++ } ++ ++ Assure(pstate != Http::Message::psParsed); ++ hdr_sz = 0; ++ ++ if (error) { ++ throw TextException(ToSBuf("failed to parse HTTP headers", ++ Debug::Extra, "parser error code: ", error, ++ Debug::Extra, "accumulated unparsed bytes: ", bufSize, ++ Debug::Extra, "reply_header_max_size: ", Config.maxReplyHeaderSize), ++ Here()); ++ } ++ ++ debugs(58, 3, "need more bytes after accumulating " << bufSize << " out of " << Config.maxReplyHeaderSize); ++ ++ // the parse() call above enforces Config.maxReplyHeaderSize limit ++ // XXX: Make this a strict comparison after fixing Http::Message::parse() enforcement ++ Assure(bufSize <= Config.maxReplyHeaderSize); ++ return 0; // parsed nothing, need more data ++} ++ + void + HttpReply::calcMaxBodySize(HttpRequest& request) const + { +diff --git a/src/HttpReply.h b/src/HttpReply.h +index 6c90e20..4301cfd 100644 +--- a/src/HttpReply.h ++++ b/src/HttpReply.h +@@ -121,6 +121,13 @@ public: + /// \returns false if any information is missing + bool olderThan(const HttpReply *them) const; + ++ /// Parses response status line and headers at the start of the given ++ /// NUL-terminated buffer of the given size. Respects reply_header_max_size. ++ /// Assures pstate becomes Http::Message::psParsed on (and only on) success. ++ /// \returns the number of bytes in a successfully parsed prefix (or zero) ++ /// \retval 0 implies that more data is needed to parse the response prefix ++ size_t parseTerminatedPrefix(const char *, size_t); ++ + private: + /** initialize */ + void init(); +diff --git a/src/MemObject.cc b/src/MemObject.cc +index 4ba63cc..d7aaf5e 100644 +--- a/src/MemObject.cc ++++ b/src/MemObject.cc +@@ -369,6 +369,12 @@ MemObject::policyLowestOffsetToKeep(bool swap) const + */ + int64_t lowest_offset = lowestMemReaderOffset(); + ++ // XXX: Remove the last (Config.onoff.memory_cache_first-based) condition ++ // and update keepForLocalMemoryCache() accordingly. The caller wants to ++ // remove all local memory that is safe to remove. Honoring caching ++ // preferences is its responsibility. Our responsibility is safety. The ++ // situation was different when ff4b33f added that condition -- there was no ++ // keepInLocalMemory/keepForLocalMemoryCache() call guard back then. + if (endOffset() < lowest_offset || + endOffset() - inmem_lo > (int64_t)Config.Store.maxInMemObjSize || + (swap && !Config.onoff.memory_cache_first)) +diff --git a/src/MemObject.h b/src/MemObject.h +index 711966d..ba6646f 100644 +--- a/src/MemObject.h ++++ b/src/MemObject.h +@@ -59,6 +59,15 @@ public: + HttpReply const *getReply() const; + void replaceHttpReply(HttpReply *newrep); + void stat (MemBuf * mb) const; ++ ++ /// The offset of the last memory-stored HTTP response byte plus one. ++ /// * HTTP response headers (if any) are stored at offset zero. ++ /// * HTTP response body byte[n] usually has offset (hdr_sz + n), where ++ /// hdr_sz is the size of stored HTTP response headers (zero if none); and ++ /// n is the corresponding byte offset in the whole resource body. ++ /// However, some 206 (Partial Content) response bodies are stored (and ++ /// retrieved) as regular 200 response bodies, disregarding offsets of ++ /// their body parts. \sa HttpStateData::decideIfWeDoRanges(). + int64_t endOffset () const; + void markEndOfReplyHeaders(); ///< sets _reply->hdr_sz to endOffset() + /// negative if unknown; otherwise, expected object_sz, expected endOffset +diff --git a/src/MemStore.cc b/src/MemStore.cc +index a4a6ab2..fe7af2f 100644 +--- a/src/MemStore.cc ++++ b/src/MemStore.cc +@@ -17,6 +17,8 @@ + #include "MemObject.h" + #include "MemStore.h" + #include "mime_header.h" ++#include "sbuf/SBuf.h" ++#include "sbuf/Stream.h" + #include "SquidConfig.h" + #include "SquidMath.h" + #include "StoreStats.h" +@@ -316,19 +318,25 @@ MemStore::get(const cache_key *key) + // create a brand new store entry and initialize it with stored info + StoreEntry *e = new StoreEntry(); + +- // XXX: We do not know the URLs yet, only the key, but we need to parse and +- // store the response for the Root().find() callers to be happy because they +- // expect IN_MEMORY entries to already have the response headers and body. +- e->createMemObject(); +- +- anchorEntry(*e, index, *slot); +- +- const bool copied = copyFromShm(*e, index, *slot); +- +- if (copied) +- return e; ++ try { ++ // XXX: We do not know the URLs yet, only the key, but we need to parse and ++ // store the response for the Root().find() callers to be happy because they ++ // expect IN_MEMORY entries to already have the response headers and body. ++ e->createMemObject(); ++ ++ anchorEntry(*e, index, *slot); ++ ++ // TODO: make copyFromShm() throw on all failures, simplifying this code ++ if (copyFromShm(*e, index, *slot)) ++ return e; ++ debugs(20, 3, "failed for " << *e); ++ } catch (...) { ++ // see store_client::parseHttpHeadersFromDisk() for problems this may log ++ debugs(20, DBG_IMPORTANT, "ERROR: Cannot load a cache hit from shared memory" << ++ Debug::Extra << "exception: " << CurrentException << ++ Debug::Extra << "cache_mem entry: " << *e); ++ } + +- debugs(20, 3, "failed for " << *e); + map->freeEntry(index); // do not let others into the same trap + destroyStoreEntry(static_cast(e)); + return NULL; +@@ -473,6 +481,8 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + Ipc::StoreMapSliceId sid = anchor.start; // optimize: remember the last sid + bool wasEof = anchor.complete() && sid < 0; + int64_t sliceOffset = 0; ++ ++ SBuf httpHeaderParsingBuffer; + while (sid >= 0) { + const Ipc::StoreMapSlice &slice = map->readableSlice(index, sid); + // slice state may change during copying; take snapshots now +@@ -495,10 +505,18 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + const StoreIOBuffer sliceBuf(wasSize - prefixSize, + e.mem_obj->endOffset(), + page + prefixSize); +- if (!copyFromShmSlice(e, sliceBuf, wasEof)) +- return false; ++ ++ copyFromShmSlice(e, sliceBuf); + debugs(20, 8, "entry " << index << " copied slice " << sid << + " from " << extra.page << '+' << prefixSize); ++ ++ // parse headers if needed; they might span multiple slices! ++ auto &reply = e.mem().adjustableBaseReply(); ++ if (reply.pstate != Http::Message::psParsed) { ++ httpHeaderParsingBuffer.append(sliceBuf.data, sliceBuf.length); ++ if (reply.parseTerminatedPrefix(httpHeaderParsingBuffer.c_str(), httpHeaderParsingBuffer.length())) ++ httpHeaderParsingBuffer = SBuf(); // we do not need these bytes anymore ++ } + } + // else skip a [possibly incomplete] slice that we copied earlier + +@@ -524,6 +542,9 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + debugs(20, 5, "mem-loaded all " << e.mem_obj->endOffset() << '/' << + anchor.basics.swap_file_sz << " bytes of " << e); + ++ if (e.mem().adjustableBaseReply().pstate != Http::Message::psParsed) ++ throw TextException(ToSBuf("truncated mem-cached headers; accumulated: ", httpHeaderParsingBuffer.length()), Here()); ++ + // from StoreEntry::complete() + e.mem_obj->object_sz = e.mem_obj->endOffset(); + e.store_status = STORE_OK; +@@ -539,32 +560,11 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + } + + /// imports one shared memory slice into local memory +-bool +-MemStore::copyFromShmSlice(StoreEntry &e, const StoreIOBuffer &buf, bool eof) ++void ++MemStore::copyFromShmSlice(StoreEntry &e, const StoreIOBuffer &buf) + { + debugs(20, 7, "buf: " << buf.offset << " + " << buf.length); + +- // from store_client::readBody() +- // parse headers if needed; they might span multiple slices! +- HttpReply *rep = (HttpReply *)e.getReply(); +- if (rep->pstate < psParsed) { +- // XXX: have to copy because httpMsgParseStep() requires 0-termination +- MemBuf mb; +- mb.init(buf.length+1, buf.length+1); +- mb.append(buf.data, buf.length); +- mb.terminate(); +- const int result = rep->httpMsgParseStep(mb.buf, buf.length, eof); +- if (result > 0) { +- assert(rep->pstate == psParsed); +- } else if (result < 0) { +- debugs(20, DBG_IMPORTANT, "Corrupted mem-cached headers: " << e); +- return false; +- } else { // more slices are needed +- assert(!eof); +- } +- } +- debugs(20, 7, "rep pstate: " << rep->pstate); +- + // local memory stores both headers and body so copy regardless of pstate + const int64_t offBefore = e.mem_obj->endOffset(); + assert(e.mem_obj->data_hdr.write(buf)); // from MemObject::write() +@@ -572,7 +572,6 @@ MemStore::copyFromShmSlice(StoreEntry &e, const StoreIOBuffer &buf, bool eof) + // expect to write the entire buf because StoreEntry::write() never fails + assert(offAfter >= 0 && offBefore <= offAfter && + static_cast(offAfter - offBefore) == buf.length); +- return true; + } + + /// whether we should cache the entry +diff --git a/src/MemStore.h b/src/MemStore.h +index 516da3c..31a2015 100644 +--- a/src/MemStore.h ++++ b/src/MemStore.h +@@ -76,7 +76,7 @@ protected: + void copyToShm(StoreEntry &e); + void copyToShmSlice(StoreEntry &e, Ipc::StoreMapAnchor &anchor, Ipc::StoreMap::Slice &slice); + bool copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnchor &anchor); +- bool copyFromShmSlice(StoreEntry &e, const StoreIOBuffer &buf, bool eof); ++ void copyFromShmSlice(StoreEntry &, const StoreIOBuffer &); + + void updateHeadersOrThrow(Ipc::StoreMapUpdate &update); + +diff --git a/src/StoreClient.h b/src/StoreClient.h +index 457844a..1d90e5a 100644 +--- a/src/StoreClient.h ++++ b/src/StoreClient.h +@@ -10,11 +10,24 @@ + #define SQUID_STORECLIENT_H + + #include "dlink.h" ++#include "store/ParsingBuffer.h" + #include "StoreIOBuffer.h" + #include "StoreIOState.h" + #include "base/AsyncCall.h" + +-typedef void STCB(void *, StoreIOBuffer); /* store callback */ ++/// A storeClientCopy() callback function. ++/// ++/// Upon storeClientCopy() success, StoreIOBuffer::flags.error is zero, and ++/// * HTTP response headers (if any) are available via MemObject::freshestReply(); ++/// * HTTP response body bytes (if any) are available via StoreIOBuffer. ++/// ++/// STCB callbacks may use response semantics to detect certain EOF conditions. ++/// Callbacks that expect HTTP headers may call store_client::atEof(). Similar ++/// to clientStreamCallback() callbacks, callbacks dedicated to receiving HTTP ++/// bodies may use zero StoreIOBuffer::length as an EOF condition. ++/// ++/// Errors are indicated by setting StoreIOBuffer flags.error. ++using STCB = void (void *, StoreIOBuffer); + + class StoreEntry; + +@@ -68,7 +81,13 @@ public: + + void dumpStats(MemBuf * output, int clientNumber) const; + +- int64_t cmp_offset; ++ // TODO: When STCB gets a dedicated Answer type, move this info there. ++ /// Whether the last successful storeClientCopy() answer was known to ++ /// contain the last body bytes of the HTTP response ++ /// \retval true requesting bytes at higher offsets is futile ++ /// \sa STCB ++ bool atEof() const { return atEof_; } ++ + #if STORE_CLIENT_LIST_DEBUG + + void *owner; +@@ -103,19 +122,28 @@ public: + dlink_node node; + + private: +- bool moreToSend() const; ++ bool moreToRead() const; ++ bool canReadFromMemory() const; ++ bool answeredOnce() const { return answers >= 1; } ++ bool sendingHttpHeaders() const; ++ int64_t nextHttpReadOffset() const; + + void fileRead(); + void scheduleDiskRead(); +- void scheduleMemRead(); ++ void readFromMemory(); + void scheduleRead(); + bool startSwapin(); + bool unpackHeader(char const *buf, ssize_t len); ++ void handleBodyFromDisk(); ++ void maybeWriteFromDiskToMemory(const StoreIOBuffer &); ++ ++ bool parseHttpHeadersFromDisk(); ++ bool tryParsingHttpHeaders(); ++ void skipHttpHeadersFromDisk(); + + void fail(); + void callback(ssize_t); + void noteCopiedBytes(size_t); +- void noteEof(); + void noteNews(); + void finishCallback(); + static void FinishCallback(store_client *); +@@ -123,13 +151,23 @@ private: + int type; + bool object_ok; + ++ /// \copydoc atEof() ++ bool atEof_; ++ + /// Storage and metadata associated with the current copy() request. Ought + /// to be ignored when not answering a copy() request. + StoreIOBuffer copyInto; + +- /// The number of bytes loaded from Store into copyInto while answering the +- /// current copy() request. Ought to be ignored when not answering. +- size_t copiedSize; ++ /// the total number of finishCallback() calls ++ uint64_t answers; ++ ++ /// Accumulates raw bytes read from Store while answering the current copy() ++ /// request. Buffer contents depends on the source and parsing stage; it may ++ /// hold (parts of) swap metadata, HTTP response headers, and/or HTTP ++ /// response body bytes. ++ std::optional parsingBuffer; ++ ++ StoreIOBuffer lastDiskRead; ///< buffer used for the last storeRead() call + + /* Until we finish stuffing code into store_client */ + +@@ -152,7 +190,18 @@ public: + } _callback; + }; + ++/// Asynchronously read HTTP response headers and/or body bytes from Store. ++/// ++/// The requested zero-based HTTP body offset is specified via the ++/// StoreIOBuffer::offset field. The first call (for a given store_client ++/// object) must specify zero offset. ++/// ++/// The requested HTTP body portion size is specified via the ++/// StoreIOBuffer::length field. The function may return fewer body bytes. ++/// ++/// See STCB for result delivery details. + void storeClientCopy(store_client *, StoreEntry *, StoreIOBuffer, STCB *, void *); ++ + store_client* storeClientListAdd(StoreEntry * e, void *data); + int storeClientCopyPending(store_client *, StoreEntry * e, void *data); + int storeUnregister(store_client * sc, StoreEntry * e, void *data); +diff --git a/src/StoreIOBuffer.h b/src/StoreIOBuffer.h +index 009aafe..ad1c491 100644 +--- a/src/StoreIOBuffer.h ++++ b/src/StoreIOBuffer.h +@@ -43,6 +43,9 @@ public: + return Range(offset, offset + length); + } + ++ /// convenience method for changing the offset of a being-configured buffer ++ StoreIOBuffer &positionAt(const int64_t newOffset) { offset = newOffset; return *this; } ++ + void dump() const { + if (fwrite(data, length, 1, stderr)) {} + if (fwrite("\n", 1, 1, stderr)) {} +diff --git a/src/acl/Asn.cc b/src/acl/Asn.cc +index 94ec862..ad450c0 100644 +--- a/src/acl/Asn.cc ++++ b/src/acl/Asn.cc +@@ -16,20 +16,22 @@ + #include "acl/DestinationIp.h" + #include "acl/SourceAsn.h" + #include "acl/Strategised.h" ++#include "base/CharacterSet.h" + #include "FwdState.h" + #include "HttpReply.h" + #include "HttpRequest.h" + #include "ipcache.h" + #include "MasterXaction.h" + #include "mgr/Registration.h" ++#include "parser/Tokenizer.h" + #include "radix.h" + #include "RequestFlags.h" ++#include "sbuf/SBuf.h" + #include "SquidConfig.h" + #include "Store.h" + #include "StoreClient.h" + + #define WHOIS_PORT 43 +-#define AS_REQBUF_SZ 4096 + + /* BEGIN of definitions for radix tree entries */ + +@@ -77,10 +79,9 @@ public: + store_client *sc; + HttpRequest::Pointer request; + int as_number; +- int64_t offset; +- int reqofs; +- char reqbuf[AS_REQBUF_SZ]; +- bool dataRead; ++ ++ /// for receiving a WHOIS reply body from Store and interpreting it ++ Store::ParsingBuffer parsingBuffer; + }; + + CBDATA_CLASS_INIT(ASState); +@@ -112,7 +113,7 @@ struct rtentry_t { + m_ADDR e_mask; + }; + +-static int asnAddNet(char *, int); ++static int asnAddNet(const SBuf &, int); + + static void asnCacheStart(int as); + +@@ -256,8 +257,7 @@ asnCacheStart(int as) + } + + asState->entry = e; +- StoreIOBuffer readBuffer (AS_REQBUF_SZ, asState->offset, asState->reqbuf); +- storeClientCopy(asState->sc, e, readBuffer, asHandleReply, asState); ++ storeClientCopy(asState->sc, e, asState->parsingBuffer.makeInitialSpace(), asHandleReply, asState); + } + + static void +@@ -265,13 +265,8 @@ asHandleReply(void *data, StoreIOBuffer result) + { + ASState *asState = (ASState *)data; + StoreEntry *e = asState->entry; +- char *s; +- char *t; +- char *buf = asState->reqbuf; +- int leftoversz = -1; + +- debugs(53, 3, "asHandleReply: Called with size=" << (unsigned int)result.length); +- debugs(53, 3, "asHandleReply: buffer='" << buf << "'"); ++ debugs(53, 3, result << " for " << asState->as_number << " with " << *e); + + /* First figure out whether we should abort the request */ + +@@ -280,11 +275,7 @@ asHandleReply(void *data, StoreIOBuffer result) + return; + } + +- if (result.length == 0 && asState->dataRead) { +- debugs(53, 3, "asHandleReply: Done: " << e->url()); +- delete asState; +- return; +- } else if (result.flags.error) { ++ if (result.flags.error) { + debugs(53, DBG_IMPORTANT, "asHandleReply: Called with Error set and size=" << (unsigned int) result.length); + delete asState; + return; +@@ -294,78 +285,39 @@ asHandleReply(void *data, StoreIOBuffer result) + return; + } + +- /* +- * Next, attempt to parse our request +- * Remembering that the actual buffer size is retsize + reqofs! +- */ +- s = buf; +- +- while ((size_t)(s - buf) < result.length + asState->reqofs && *s != '\0') { +- while (*s && xisspace(*s)) +- ++s; +- +- for (t = s; *t; ++t) { +- if (xisspace(*t)) +- break; +- } +- +- if (*t == '\0') { +- /* oof, word should continue on next block */ +- break; +- } +- +- *t = '\0'; +- debugs(53, 3, "asHandleReply: AS# " << s << " (" << asState->as_number << ")"); +- asnAddNet(s, asState->as_number); +- s = t + 1; +- asState->dataRead = true; ++ asState->parsingBuffer.appended(result.data, result.length); ++ Parser::Tokenizer tok(SBuf(asState->parsingBuffer.content().data, asState->parsingBuffer.contentSize())); ++ SBuf address; ++ // Word delimiters in WHOIS ASN replies. RFC 3912 mentions SP, CR, and LF. ++ // Others are added to mimic an earlier isspace()-based implementation. ++ static const auto WhoisSpaces = CharacterSet("ASCII_spaces", " \f\r\n\t\v"); ++ while (tok.token(address, WhoisSpaces)) { ++ (void)asnAddNet(address, asState->as_number); + } +- +- /* +- * Next, grab the end of the 'valid data' in the buffer, and figure +- * out how much data is left in our buffer, which we need to keep +- * around for the next request +- */ +- leftoversz = (asState->reqofs + result.length) - (s - buf); +- +- assert(leftoversz >= 0); +- +- /* +- * Next, copy the left over data, from s to s + leftoversz to the +- * beginning of the buffer +- */ +- memmove(buf, s, leftoversz); +- +- /* +- * Next, update our offset and reqofs, and kick off a copy if required +- */ +- asState->offset += result.length; +- +- asState->reqofs = leftoversz; +- +- debugs(53, 3, "asState->offset = " << asState->offset); +- +- if (e->store_status == STORE_PENDING) { +- debugs(53, 3, "asHandleReply: store_status == STORE_PENDING: " << e->url() ); +- StoreIOBuffer tempBuffer (AS_REQBUF_SZ - asState->reqofs, +- asState->offset, +- asState->reqbuf + asState->reqofs); +- storeClientCopy(asState->sc, +- e, +- tempBuffer, +- asHandleReply, +- asState); +- } else { +- StoreIOBuffer tempBuffer; +- debugs(53, 3, "asHandleReply: store complete, but data received " << e->url() ); +- tempBuffer.offset = asState->offset; +- tempBuffer.length = AS_REQBUF_SZ - asState->reqofs; +- tempBuffer.data = asState->reqbuf + asState->reqofs; +- storeClientCopy(asState->sc, +- e, +- tempBuffer, +- asHandleReply, +- asState); ++ asState->parsingBuffer.consume(tok.parsedSize()); ++ const auto leftoverBytes = asState->parsingBuffer.contentSize(); ++ if (asState->sc->atEof()) { ++ if (leftoverBytes) ++ debugs(53, 2, "WHOIS: Discarding the last " << leftoverBytes << " received bytes of a truncated AS response"); ++ delete asState; ++ return; ++ } ++ const auto remainingSpace = asState->parsingBuffer.space().positionAt(result.offset + result.length); ++ if (!remainingSpace.length) { ++ Assure(leftoverBytes); ++ debugs(53, DBG_IMPORTANT, "WARNING: Ignoring the tail of a WHOIS AS response" << ++ " with an unparsable section of " << leftoverBytes << ++ " bytes ending at offset " << remainingSpace.offset); ++ delete asState; ++ return; ++ } ++ const decltype(StoreIOBuffer::offset) stillReasonableOffset = 100000; // an arbitrary limit in bytes ++ if (remainingSpace.offset > stillReasonableOffset) { ++ // stop suspicious accumulation of parsed addresses and/or work ++ debugs(53, DBG_IMPORTANT, "WARNING: Ignoring the tail of a suspiciously large WHOIS AS response" << ++ " exceeding " << stillReasonableOffset << " bytes"); ++ delete asState; ++ return; + } + } + +@@ -373,38 +325,29 @@ asHandleReply(void *data, StoreIOBuffer result) + * add a network (addr, mask) to the radix tree, with matching AS number + */ + static int +-asnAddNet(char *as_string, int as_number) ++asnAddNet(const SBuf &addressAndMask, const int as_number) + { + struct squid_radix_node *rn; + CbDataList **Tail = NULL; + CbDataList *q = NULL; + as_info *asinfo = NULL; + +- Ip::Address mask; +- Ip::Address addr; +- char *t; +- int bitl; +- +- t = strchr(as_string, '/'); +- +- if (t == NULL) { ++ static const CharacterSet NonSlashSet = CharacterSet("slash", "/").complement("non-slash"); ++ Parser::Tokenizer tok(addressAndMask); ++ SBuf addressToken; ++ if (!(tok.prefix(addressToken, NonSlashSet) && tok.skip('/'))) { + debugs(53, 3, "asnAddNet: failed, invalid response from whois server."); + return 0; + } +- +- *t = '\0'; +- addr = as_string; +- bitl = atoi(t + 1); +- +- if (bitl < 0) +- bitl = 0; +- ++ const Ip::Address addr = addressToken.c_str(); + // INET6 TODO : find a better way of identifying the base IPA family for mask than this. +- t = strchr(as_string, '.'); +- ++ const auto addrFamily = (addressToken.find('.') != SBuf::npos) ? AF_INET : AF_INET6; + // generate Netbits Format Mask ++ Ip::Address mask; + mask.setNoAddr(); +- mask.applyMask(bitl, (t!=NULL?AF_INET:AF_INET6) ); ++ int64_t bitl = 0; ++ if (tok.int64(bitl, 10, false)) ++ mask.applyMask(bitl, addrFamily); + + debugs(53, 3, "asnAddNet: called for " << addr << "/" << mask ); + +diff --git a/src/clientStream.cc b/src/clientStream.cc +index 04d89c0..bd5dd09 100644 +--- a/src/clientStream.cc ++++ b/src/clientStream.cc +@@ -154,8 +154,7 @@ clientStreamCallback(clientStreamNode * thisObject, ClientHttpRequest * http, + assert(thisObject && http && thisObject->node.next); + next = thisObject->next(); + +- debugs(87, 3, "clientStreamCallback: Calling " << next->callback << " with cbdata " << +- next->data.getRaw() << " from node " << thisObject); ++ debugs(87, 3, thisObject << " gives " << next->data << ' ' << replyBuffer); + next->callback(next, http, rep, replyBuffer); + } + +diff --git a/src/client_side_reply.cc b/src/client_side_reply.cc +index c919af4..861f4b4 100644 +--- a/src/client_side_reply.cc ++++ b/src/client_side_reply.cc +@@ -33,6 +33,7 @@ + #include "refresh.h" + #include "RequestFlags.h" + #include "SquidConfig.h" ++#include "SquidMath.h" + #include "SquidTime.h" + #include "Store.h" + #include "StrList.h" +@@ -76,11 +77,7 @@ clientReplyContext::clientReplyContext(ClientHttpRequest *clientContext) : + purgeStatus(Http::scNone), + lookingforstore(0), + http(cbdataReference(clientContext)), +- headers_sz(0), + sc(NULL), +- old_reqsize(0), +- reqsize(0), +- reqofs(0), + #if USE_CACHE_DIGESTS + lookup_type(NULL), + #endif +@@ -166,8 +163,6 @@ void clientReplyContext::setReplyToStoreEntry(StoreEntry *entry, const char *rea + #if USE_DELAY_POOLS + sc->setDelayId(DelayId::DelayClient(http)); + #endif +- reqofs = 0; +- reqsize = 0; + if (http->request) + http->request->ignoreRange(reason); + flags.storelogiccomplete = 1; +@@ -206,13 +201,9 @@ clientReplyContext::saveState() + old_sc = sc; + old_lastmod = http->request->lastmod; + old_etag = http->request->etag; +- old_reqsize = reqsize; +- tempBuffer.offset = reqofs; + /* Prevent accessing the now saved entries */ + http->storeEntry(NULL); + sc = NULL; +- reqsize = 0; +- reqofs = 0; + } + + void +@@ -223,8 +214,6 @@ clientReplyContext::restoreState() + removeClientStoreReference(&sc, http); + http->storeEntry(old_entry); + sc = old_sc; +- reqsize = old_reqsize; +- reqofs = tempBuffer.offset; + http->request->lastmod = old_lastmod; + http->request->etag = old_etag; + /* Prevent accessed the old saved entries */ +@@ -232,7 +221,6 @@ clientReplyContext::restoreState() + old_sc = NULL; + old_lastmod = -1; + old_etag.clean(); +- old_reqsize = 0; + tempBuffer.offset = 0; + } + +@@ -250,18 +238,27 @@ clientReplyContext::getNextNode() const + return (clientStreamNode *)ourNode->node.next->data; + } + +-/* This function is wrong - the client parameters don't include the +- * header offset +- */ ++/// Request HTTP response headers from Store, to be sent to the given recipient. ++/// That recipient also gets zero, some, or all HTTP response body bytes (into ++/// next()->readBuffer). + void +-clientReplyContext::triggerInitialStoreRead() ++clientReplyContext::triggerInitialStoreRead(STCB recipient) + { +- /* when confident, 0 becomes reqofs, and then this factors into +- * startSendProcess +- */ +- assert(reqofs == 0); ++ Assure(recipient != HandleIMSReply); ++ lastStreamBufferedBytes = StoreIOBuffer(); // storeClientCopy(next()->readBuffer) invalidates + StoreIOBuffer localTempBuffer (next()->readBuffer.length, 0, next()->readBuffer.data); +- storeClientCopy(sc, http->storeEntry(), localTempBuffer, SendMoreData, this); ++ ::storeClientCopy(sc, http->storeEntry(), localTempBuffer, recipient, this); ++} ++ ++/// Request HTTP response body bytes from Store into next()->readBuffer. This ++/// method requests body bytes at readerBuffer.offset and, hence, it should only ++/// be called after we triggerInitialStoreRead() and get the requested HTTP ++/// response headers (using zero offset). ++void ++clientReplyContext::requestMoreBodyFromStore() ++{ ++ lastStreamBufferedBytes = StoreIOBuffer(); // storeClientCopy(next()->readBuffer) invalidates ++ ::storeClientCopy(sc, http->storeEntry(), next()->readBuffer, SendMoreData, this); + } + + /* there is an expired entry in the store. +@@ -358,30 +355,22 @@ clientReplyContext::processExpired() + { + /* start counting the length from 0 */ + StoreIOBuffer localTempBuffer(HTTP_REQBUF_SZ, 0, tempbuf); +- storeClientCopy(sc, entry, localTempBuffer, HandleIMSReply, this); ++ // keep lastStreamBufferedBytes: tempbuf is not a Client Stream buffer ++ ::storeClientCopy(sc, entry, localTempBuffer, HandleIMSReply, this); + } + } + + void +-clientReplyContext::sendClientUpstreamResponse() ++clientReplyContext::sendClientUpstreamResponse(const StoreIOBuffer &upstreamResponse) + { +- StoreIOBuffer tempresult; + removeStoreReference(&old_sc, &old_entry); + + if (collapsedRevalidation) + http->storeEntry()->clearPublicKeyScope(); + + /* here the data to send is the data we just received */ +- tempBuffer.offset = 0; +- old_reqsize = 0; +- /* sendMoreData tracks the offset as well. +- * Force it back to zero */ +- reqofs = 0; + assert(!EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)); +- /* TODO: provide sendMoreData with the ready parsed reply */ +- tempresult.length = reqsize; +- tempresult.data = tempbuf; +- sendMoreData(tempresult); ++ sendMoreData(upstreamResponse); + } + + void +@@ -398,11 +387,9 @@ clientReplyContext::sendClientOldEntry() + restoreState(); + /* here the data to send is in the next nodes buffers already */ + assert(!EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)); +- /* sendMoreData tracks the offset as well. +- * Force it back to zero */ +- reqofs = 0; +- StoreIOBuffer tempresult (reqsize, reqofs, next()->readBuffer.data); +- sendMoreData(tempresult); ++ Assure(matchesStreamBodyBuffer(lastStreamBufferedBytes)); ++ Assure(!lastStreamBufferedBytes.offset); ++ sendMoreData(lastStreamBufferedBytes); + } + + /* This is the workhorse of the HandleIMSReply callback. +@@ -411,16 +398,16 @@ clientReplyContext::sendClientOldEntry() + * IMS request to revalidate a stale entry. + */ + void +-clientReplyContext::handleIMSReply(StoreIOBuffer result) ++clientReplyContext::handleIMSReply(const StoreIOBuffer result) + { + if (deleting) + return; + +- debugs(88, 3, http->storeEntry()->url() << ", " << (long unsigned) result.length << " bytes"); +- + if (http->storeEntry() == NULL) + return; + ++ debugs(88, 3, http->storeEntry()->url() << " got " << result); ++ + if (result.flags.error && !EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)) + return; + +@@ -433,9 +420,6 @@ clientReplyContext::handleIMSReply(StoreIOBuffer result) + return; + } + +- /* update size of the request */ +- reqsize = result.length + reqofs; +- + const Http::StatusCode status = http->storeEntry()->getReply()->sline.status(); + + // request to origin was aborted +@@ -460,7 +444,7 @@ clientReplyContext::handleIMSReply(StoreIOBuffer result) + if (http->request->flags.ims && !old_entry->modifiedSince(http->request->ims, http->request->imslen)) { + // forward the 304 from origin + debugs(88, 3, "origin replied 304, revalidating existing entry and forwarding 304 to client"); +- sendClientUpstreamResponse(); ++ sendClientUpstreamResponse(result); + } else { + // send existing entry, it's still valid + debugs(88, 3, "origin replied 304, revalidating existing entry and sending " << +@@ -484,7 +468,7 @@ clientReplyContext::handleIMSReply(StoreIOBuffer result) + http->logType = LOG_TCP_REFRESH_MODIFIED; + debugs(88, 3, "origin replied " << status << + ", replacing existing entry and forwarding to client"); +- sendClientUpstreamResponse(); ++ sendClientUpstreamResponse(result); + } + } + +@@ -493,7 +477,7 @@ clientReplyContext::handleIMSReply(StoreIOBuffer result) + http->logType = LOG_TCP_REFRESH_FAIL_ERR; + debugs(88, 3, "origin replied with error " << status << + ", forwarding to client due to fail_on_validation_err"); +- sendClientUpstreamResponse(); ++ sendClientUpstreamResponse(result); + } else { + // ignore and let client have old entry + http->logType = LOG_TCP_REFRESH_FAIL_OLD; +@@ -506,13 +490,7 @@ clientReplyContext::handleIMSReply(StoreIOBuffer result) + SQUIDCEXTERN CSR clientGetMoreData; + SQUIDCEXTERN CSD clientReplyDetach; + +-/** +- * clientReplyContext::cacheHit Should only be called until the HTTP reply headers +- * have been parsed. Normally this should be a single call, but +- * it might take more than one. As soon as we have the headers, +- * we hand off to clientSendMoreData, processExpired, or +- * processMiss. +- */ ++/// \copydoc clientReplyContext::cacheHit() + void + clientReplyContext::CacheHit(void *data, StoreIOBuffer result) + { +@@ -520,11 +498,11 @@ clientReplyContext::CacheHit(void *data, StoreIOBuffer result) + context->cacheHit(result); + } + +-/** +- * Process a possible cache HIT. +- */ ++/// Processes HTTP response headers received from Store on a suspected cache hit ++/// path. May be called several times (e.g., a Vary marker object hit followed ++/// by the corresponding variant hit). + void +-clientReplyContext::cacheHit(StoreIOBuffer result) ++clientReplyContext::cacheHit(const StoreIOBuffer result) + { + /** Ignore if the HIT object is being deleted. */ + if (deleting) { +@@ -536,7 +514,7 @@ clientReplyContext::cacheHit(StoreIOBuffer result) + + HttpRequest *r = http->request; + +- debugs(88, 3, "clientCacheHit: " << http->uri << ", " << result.length << " bytes"); ++ debugs(88, 3, http->uri << " got " << result); + + if (http->storeEntry() == NULL) { + debugs(88, 3, "clientCacheHit: request aborted"); +@@ -560,20 +538,7 @@ clientReplyContext::cacheHit(StoreIOBuffer result) + return; + } + +- if (result.length == 0) { +- debugs(88, 5, "store IO buffer has no content. MISS"); +- /* the store couldn't get enough data from the file for us to id the +- * object +- */ +- /* treat as a miss */ +- http->logType = LOG_TCP_MISS; +- processMiss(); +- return; +- } +- + assert(!EBIT_TEST(e->flags, ENTRY_ABORTED)); +- /* update size of the request */ +- reqsize = result.length + reqofs; + + /* + * Got the headers, now grok them +@@ -587,6 +552,8 @@ clientReplyContext::cacheHit(StoreIOBuffer result) + return; + } + ++ noteStreamBufferredBytes(result); ++ + switch (varyEvaluateMatch(e, r)) { + + case VARY_NONE: +@@ -687,7 +654,7 @@ clientReplyContext::cacheHit(StoreIOBuffer result) + return; + } else if (r->conditional()) { + debugs(88, 5, "conditional HIT"); +- if (processConditional(result)) ++ if (processConditional()) + return; + } + +@@ -806,7 +773,7 @@ clientReplyContext::processOnlyIfCachedMiss() + + /// process conditional request from client + bool +-clientReplyContext::processConditional(StoreIOBuffer &result) ++clientReplyContext::processConditional() + { + StoreEntry *const e = http->storeEntry(); + +@@ -984,16 +951,7 @@ clientReplyContext::purgeFoundObject(StoreEntry *entry) + + http->logType = LOG_TCP_HIT; + +- reqofs = 0; +- +- localTempBuffer.offset = http->out.offset; +- +- localTempBuffer.length = next()->readBuffer.length; +- +- localTempBuffer.data = next()->readBuffer.data; +- +- storeClientCopy(sc, http->storeEntry(), +- localTempBuffer, CacheHit, this); ++ triggerInitialStoreRead(CacheHit); + } + + void +@@ -1111,16 +1069,10 @@ clientReplyContext::purgeDoPurgeHead(StoreEntry *newEntry) + } + + void +-clientReplyContext::traceReply(clientStreamNode * node) ++clientReplyContext::traceReply() + { +- clientStreamNode *nextNode = (clientStreamNode *)node->node.next->data; +- StoreIOBuffer localTempBuffer; + createStoreEntry(http->request->method, RequestFlags()); +- localTempBuffer.offset = nextNode->readBuffer.offset + headers_sz; +- localTempBuffer.length = nextNode->readBuffer.length; +- localTempBuffer.data = nextNode->readBuffer.data; +- storeClientCopy(sc, http->storeEntry(), +- localTempBuffer, SendMoreData, this); ++ triggerInitialStoreRead(); + http->storeEntry()->releaseRequest(); + http->storeEntry()->buffer(); + HttpReply *rep = new HttpReply; +@@ -1169,16 +1121,15 @@ int + clientReplyContext::storeOKTransferDone() const + { + assert(http->storeEntry()->objectLen() >= 0); ++ const auto headers_sz = http->storeEntry()->mem().baseReply().hdr_sz; + assert(http->storeEntry()->objectLen() >= headers_sz); +- if (http->out.offset >= http->storeEntry()->objectLen() - headers_sz) { +- debugs(88,3,HERE << "storeOKTransferDone " << +- " out.offset=" << http->out.offset << +- " objectLen()=" << http->storeEntry()->objectLen() << +- " headers_sz=" << headers_sz); +- return 1; +- } +- +- return 0; ++ const auto done = http->out.offset >= http->storeEntry()->objectLen() - headers_sz; ++ const auto debugLevel = done ? 3 : 5; ++ debugs(88, debugLevel, done << ++ " out.offset=" << http->out.offset << ++ " objectLen()=" << http->storeEntry()->objectLen() << ++ " headers_sz=" << headers_sz); ++ return done ? 1 : 0; + } + + int +@@ -1190,11 +1141,8 @@ clientReplyContext::storeNotOKTransferDone() const + MemObject *mem = http->storeEntry()->mem_obj; + assert(mem != NULL); + assert(http->request != NULL); +- /* mem->reply was wrong because it uses the UPSTREAM header length!!! */ +- HttpReply const *curReply = mem->getReply(); + +- if (headers_sz == 0) +- /* haven't found end of headers yet */ ++ if (mem->baseReply().pstate != Http::Message::psParsed) + return 0; + + /* +@@ -1202,19 +1150,12 @@ clientReplyContext::storeNotOKTransferDone() const + * If we are sending a body and we don't have a content-length, + * then we must wait for the object to become STORE_OK. + */ +- if (curReply->content_length < 0) +- return 0; +- +- uint64_t expectedLength = curReply->content_length + http->out.headers_sz; +- +- if (http->out.size < expectedLength) +- return 0; +- else { +- debugs(88,3,HERE << "storeNotOKTransferDone " << +- " out.size=" << http->out.size << +- " expectedLength=" << expectedLength); +- return 1; +- } ++ const auto done = http->out.offset >= expectedBodySize; ++ const auto debugLevel = done ? 3 : 5; ++ debugs(88, debugLevel, done << ++ " out.offset=" << http->out.offset << ++ " expectedBodySize=" << expectedBodySize); ++ return done ? 1 : 0; + } + + /* A write has completed, what is the next status based on the +@@ -1778,20 +1719,12 @@ clientGetMoreData(clientStreamNode * aNode, ClientHttpRequest * http) + assert (context); + assert(context->http == http); + +- clientStreamNode *next = ( clientStreamNode *)aNode->node.next->data; +- + if (!context->ourNode) + context->ourNode = aNode; + + /* no cbdatareference, this is only used once, and safely */ + if (context->flags.storelogiccomplete) { +- StoreIOBuffer tempBuffer; +- tempBuffer.offset = next->readBuffer.offset + context->headers_sz; +- tempBuffer.length = next->readBuffer.length; +- tempBuffer.data = next->readBuffer.data; +- +- storeClientCopy(context->sc, http->storeEntry(), +- tempBuffer, clientReplyContext::SendMoreData, context); ++ context->requestMoreBodyFromStore(); + return; + } + +@@ -1804,7 +1737,7 @@ clientGetMoreData(clientStreamNode * aNode, ClientHttpRequest * http) + + if (context->http->request->method == Http::METHOD_TRACE) { + if (context->http->request->header.getInt64(Http::HdrType::MAX_FORWARDS) == 0) { +- context->traceReply(aNode); ++ context->traceReply(); + return; + } + +@@ -1834,7 +1767,6 @@ clientReplyContext::doGetMoreData() + #endif + + assert(http->logType.oldType == LOG_TCP_HIT); +- reqofs = 0; + /* guarantee nothing has been sent yet! */ + assert(http->out.size == 0); + assert(http->out.offset == 0); +@@ -1849,10 +1781,7 @@ clientReplyContext::doGetMoreData() + } + } + +- localTempBuffer.offset = reqofs; +- localTempBuffer.length = getNextNode()->readBuffer.length; +- localTempBuffer.data = getNextNode()->readBuffer.data; +- storeClientCopy(sc, http->storeEntry(), localTempBuffer, CacheHit, this); ++ triggerInitialStoreRead(CacheHit); + } else { + /* MISS CASE, http->logType is already set! */ + processMiss(); +@@ -1878,6 +1807,32 @@ clientReplyContext::SendMoreData(void *data, StoreIOBuffer result) + context->sendMoreData (result); + } + ++/// Whether the given body area describes the start of our Client Stream buffer. ++/// An empty area does. ++bool ++clientReplyContext::matchesStreamBodyBuffer(const StoreIOBuffer &their) const ++{ ++ // the answer is undefined for errors; they are not really "body buffers" ++ Assure(!their.flags.error); ++ ++ if (!their.length) ++ return true; // an empty body area always matches our body area ++ ++ if (their.data != next()->readBuffer.data) { ++ debugs(88, 7, "no: " << their << " vs. " << next()->readBuffer); ++ return false; ++ } ++ ++ return true; ++} ++ ++void ++clientReplyContext::noteStreamBufferredBytes(const StoreIOBuffer &result) ++{ ++ Assure(matchesStreamBodyBuffer(result)); ++ lastStreamBufferedBytes = result; // may be unchanged and/or zero-length ++} ++ + void + clientReplyContext::makeThisHead() + { +@@ -1887,12 +1842,11 @@ clientReplyContext::makeThisHead() + } + + bool +-clientReplyContext::errorInStream(StoreIOBuffer const &result, size_t const &sizeToProcess)const ++clientReplyContext::errorInStream(const StoreIOBuffer &result) const + { + return /* aborted request */ + (http->storeEntry() && EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)) || +- /* Upstream read error */ (result.flags.error) || +- /* Upstream EOF */ (sizeToProcess == 0); ++ /* Upstream read error */ (result.flags.error); + } + + void +@@ -1913,24 +1867,16 @@ clientReplyContext::sendStreamError(StoreIOBuffer const &result) + } + + void +-clientReplyContext::pushStreamData(StoreIOBuffer const &result, char *source) ++clientReplyContext::pushStreamData(const StoreIOBuffer &result) + { +- StoreIOBuffer localTempBuffer; +- + if (result.length == 0) { + debugs(88, 5, "clientReplyContext::pushStreamData: marking request as complete due to 0 length store result"); + flags.complete = 1; + } + +- assert(result.offset - headers_sz == next()->readBuffer.offset); +- localTempBuffer.offset = result.offset - headers_sz; +- localTempBuffer.length = result.length; +- +- if (localTempBuffer.length) +- localTempBuffer.data = source; +- ++ assert(!result.length || result.offset == next()->readBuffer.offset); + clientStreamCallback((clientStreamNode*)http->client_stream.head->data, http, NULL, +- localTempBuffer); ++ result); + } + + clientStreamNode * +@@ -2022,7 +1968,6 @@ clientReplyContext::processReplyAccess () + if (http->logType.oldType == LOG_TCP_DENIED || + http->logType.oldType == LOG_TCP_DENIED_REPLY || + alwaysAllowResponse(reply->sline.status())) { +- headers_sz = reply->hdr_sz; + processReplyAccessResult(ACCESS_ALLOWED); + return; + } +@@ -2033,8 +1978,6 @@ clientReplyContext::processReplyAccess () + return; + } + +- headers_sz = reply->hdr_sz; +- + /** check for absent access controls (permit by default) */ + if (!Config.accessList.reply) { + processReplyAccessResult(ACCESS_ALLOWED); +@@ -2091,11 +2034,9 @@ clientReplyContext::processReplyAccessResult(const allow_t &accessAllowed) + /* Ok, the reply is allowed, */ + http->loggingEntry(http->storeEntry()); + +- ssize_t body_size = reqofs - reply->hdr_sz; +- if (body_size < 0) { +- reqofs = reply->hdr_sz; +- body_size = 0; +- } ++ Assure(matchesStreamBodyBuffer(lastStreamBufferedBytes)); ++ Assure(!lastStreamBufferedBytes.offset); ++ auto body_size = lastStreamBufferedBytes.length; // may be zero + + debugs(88, 3, "clientReplyContext::sendMoreData: Appending " << + (int) body_size << " bytes after " << reply->hdr_sz << +@@ -2123,19 +2064,27 @@ clientReplyContext::processReplyAccessResult(const allow_t &accessAllowed) + assert (!flags.headersSent); + flags.headersSent = true; + ++ // next()->readBuffer.offset may be positive for Range requests, but our ++ // localTempBuffer initialization code assumes that next()->readBuffer.data ++ // points to the response body at offset 0 because the first ++ // storeClientCopy() request always has offset 0 (i.e. our first Store ++ // request ignores next()->readBuffer.offset). ++ // ++ // XXX: We cannot fully check that assumption: readBuffer.offset field is ++ // often out of sync with the buffer content, and if some buggy code updates ++ // the buffer while we were waiting for the processReplyAccessResult() ++ // callback, we may not notice. ++ + StoreIOBuffer localTempBuffer; +- char *buf = next()->readBuffer.data; +- char *body_buf = buf + reply->hdr_sz; ++ const auto body_buf = next()->readBuffer.data; + + //Server side may disable ranges under some circumstances. + + if ((!http->request->range)) + next()->readBuffer.offset = 0; + +- body_buf -= next()->readBuffer.offset; +- +- if (next()->readBuffer.offset != 0) { +- if (next()->readBuffer.offset > body_size) { ++ if (next()->readBuffer.offset > 0) { ++ if (Less(body_size, next()->readBuffer.offset)) { + /* Can't use any of the body we received. send nothing */ + localTempBuffer.length = 0; + localTempBuffer.data = NULL; +@@ -2148,7 +2097,6 @@ clientReplyContext::processReplyAccessResult(const allow_t &accessAllowed) + localTempBuffer.data = body_buf; + } + +- /* TODO??: move the data in the buffer back by the request header size */ + clientStreamCallback((clientStreamNode *)http->client_stream.head->data, + http, reply, localTempBuffer); + +@@ -2161,6 +2109,8 @@ clientReplyContext::sendMoreData (StoreIOBuffer result) + if (deleting) + return; + ++ debugs(88, 5, http->uri << " got " << result); ++ + StoreEntry *entry = http->storeEntry(); + + if (ConnStateData * conn = http->getConn()) { +@@ -2173,7 +2123,9 @@ clientReplyContext::sendMoreData (StoreIOBuffer result) + return; + } + +- if (reqofs==0 && !http->logType.isTcpHit()) { ++ if (!flags.headersSent && !http->logType.isTcpHit()) { ++ // We get here twice if processReplyAccessResult() calls startError(). ++ // TODO: Revise when we check/change QoS markings to reduce syscalls. + if (Ip::Qos::TheConfig.isHitTosActive()) { + Ip::Qos::doTosLocalMiss(conn->clientConnection, http->request->hier.code); + } +@@ -2187,21 +2139,9 @@ clientReplyContext::sendMoreData (StoreIOBuffer result) + " out.offset=" << http->out.offset); + } + +- char *buf = next()->readBuffer.data; +- +- if (buf != result.data) { +- /* we've got to copy some data */ +- assert(result.length <= next()->readBuffer.length); +- memcpy(buf, result.data, result.length); +- } +- + /* We've got the final data to start pushing... */ + flags.storelogiccomplete = 1; + +- reqofs += result.length; +- +- assert(reqofs <= HTTP_REQBUF_SZ || flags.headersSent); +- + assert(http->request != NULL); + + /* ESI TODO: remove this assert once everything is stable */ +@@ -2210,20 +2150,25 @@ clientReplyContext::sendMoreData (StoreIOBuffer result) + + makeThisHead(); + +- debugs(88, 5, "clientReplyContext::sendMoreData: " << http->uri << ", " << +- reqofs << " bytes (" << result.length << +- " new bytes)"); +- +- /* update size of the request */ +- reqsize = reqofs; +- +- if (errorInStream(result, reqofs)) { ++ if (errorInStream(result)) { + sendStreamError(result); + return; + } + ++ if (!matchesStreamBodyBuffer(result)) { ++ // Subsequent processing expects response body bytes to be at the start ++ // of our Client Stream buffer. When given something else (e.g., bytes ++ // in our tempbuf), we copy and adjust to meet those expectations. ++ const auto &ourClientStreamsBuffer = next()->readBuffer; ++ assert(result.length <= ourClientStreamsBuffer.length); ++ memcpy(ourClientStreamsBuffer.data, result.data, result.length); ++ result.data = ourClientStreamsBuffer.data; ++ } ++ ++ noteStreamBufferredBytes(result); ++ + if (flags.headersSent) { +- pushStreamData (result, buf); ++ pushStreamData(result); + return; + } + +@@ -2289,13 +2234,6 @@ clientReplyContext::createStoreEntry(const HttpRequestMethod& m, RequestFlags re + sc->setDelayId(DelayId::DelayClient(http)); + #endif + +- reqofs = 0; +- +- reqsize = 0; +- +- /* I don't think this is actually needed! -- adrian */ +- /* http->reqbuf = http->norm_reqbuf; */ +- // assert(http->reqbuf == http->norm_reqbuf); + /* The next line is illegal because we don't know if the client stream + * buffers have been set up + */ +diff --git a/src/client_side_reply.h b/src/client_side_reply.h +index dddab1a..bc702e3 100644 +--- a/src/client_side_reply.h ++++ b/src/client_side_reply.h +@@ -39,7 +39,6 @@ public: + void purgeFoundGet(StoreEntry *newEntry); + void purgeFoundHead(StoreEntry *newEntry); + void purgeFoundObject(StoreEntry *entry); +- void sendClientUpstreamResponse(); + void purgeDoPurgeGet(StoreEntry *entry); + void purgeDoPurgeHead(StoreEntry *entry); + void doGetMoreData(); +@@ -67,7 +66,7 @@ public: + void processExpired(); + clientStream_status_t replyStatus(); + void processMiss(); +- void traceReply(clientStreamNode * node); ++ void traceReply(); + const char *storeId() const { return (http->store_id.size() > 0 ? http->store_id.termedBuf() : http->uri); } + + Http::StatusCode purgeStatus; +@@ -77,13 +76,14 @@ public: + virtual void created (StoreEntry *newEntry); + + ClientHttpRequest *http; +- int headers_sz; + store_client *sc; /* The store_client we're using */ + StoreIOBuffer tempBuffer; /* For use in validating requests via IMS */ +- int old_reqsize; /* ... again, for the buffer */ +- size_t reqsize; +- size_t reqofs; +- char tempbuf[HTTP_REQBUF_SZ]; ///< a temporary buffer if we need working storage ++ ++ /// Buffer dedicated to receiving storeClientCopy() responses to generated ++ /// revalidation requests. These requests cannot use next()->readBuffer ++ /// because the latter keeps the contents of the stale HTTP response during ++ /// revalidation. sendClientOldEntry() uses that contents. ++ char tempbuf[HTTP_REQBUF_SZ]; + #if USE_CACHE_DIGESTS + + const char *lookup_type; /* temporary hack: storeGet() result: HIT/MISS/NONE */ +@@ -101,9 +101,10 @@ public: + private: + clientStreamNode *getNextNode() const; + void makeThisHead(); +- bool errorInStream(StoreIOBuffer const &result, size_t const &sizeToProcess)const ; ++ bool errorInStream(const StoreIOBuffer &result) const; ++ bool matchesStreamBodyBuffer(const StoreIOBuffer &) const; + void sendStreamError(StoreIOBuffer const &result); +- void pushStreamData(StoreIOBuffer const &result, char *source); ++ void pushStreamData(const StoreIOBuffer &); + clientStreamNode * next() const; + StoreIOBuffer holdingBuffer; + HttpReply *reply; +@@ -115,11 +116,13 @@ private: + bool alwaysAllowResponse(Http::StatusCode sline) const; + int checkTransferDone(); + void processOnlyIfCachedMiss(); +- bool processConditional(StoreIOBuffer &result); ++ bool processConditional(); ++ void noteStreamBufferredBytes(const StoreIOBuffer &); + void cacheHit(StoreIOBuffer result); + void handleIMSReply(StoreIOBuffer result); + void sendMoreData(StoreIOBuffer result); +- void triggerInitialStoreRead(); ++ void triggerInitialStoreRead(STCB = SendMoreData); ++ void requestMoreBodyFromStore(); + void sendClientOldEntry(); + void purgeAllCached(); + void forgetHit(); +@@ -129,6 +132,13 @@ private: + void sendPreconditionFailedError(); + void sendNotModified(); + void sendNotModifiedOrPreconditionFailedError(); ++ void sendClientUpstreamResponse(const StoreIOBuffer &upstreamResponse); ++ ++ /// Reduces a chance of an accidental direct storeClientCopy() call that ++ /// (should but) forgets to invalidate our lastStreamBufferedBytes. This ++ /// function is not defined; decltype() syntax prohibits "= delete", but ++ /// function usage will trigger deprecation warnings and linking errors. ++ static decltype(::storeClientCopy) storeClientCopy [[deprecated]]; + + StoreEntry *old_entry; + /* ... for entry to be validated */ +@@ -145,6 +155,12 @@ private: + } CollapsedRevalidation; + + CollapsedRevalidation collapsedRevalidation; ++ ++ /// HTTP response body bytes stored in our Client Stream buffer (if any) ++ StoreIOBuffer lastStreamBufferedBytes; ++ ++ // TODO: Remove after moving the meat of this function into a method. ++ friend CSR clientGetMoreData; + }; + + #endif /* SQUID_CLIENTSIDEREPLY_H */ +diff --git a/src/enums.h b/src/enums.h +index 4a860d8..262d62c 100644 +--- a/src/enums.h ++++ b/src/enums.h +@@ -203,7 +203,6 @@ enum { + typedef enum { + DIGEST_READ_NONE, + DIGEST_READ_REPLY, +- DIGEST_READ_HEADERS, + DIGEST_READ_CBLOCK, + DIGEST_READ_MASK, + DIGEST_READ_DONE +diff --git a/src/icmp/net_db.cc b/src/icmp/net_db.cc +index 7dc42a2..ce8067a 100644 +--- a/src/icmp/net_db.cc ++++ b/src/icmp/net_db.cc +@@ -33,6 +33,7 @@ + #include "mgr/Registration.h" + #include "mime_header.h" + #include "neighbors.h" ++#include "sbuf/SBuf.h" + #include "SquidConfig.h" + #include "SquidTime.h" + #include "Store.h" +@@ -49,8 +50,6 @@ + #include "ipcache.h" + #include "StoreClient.h" + +-#define NETDB_REQBUF_SZ 4096 +- + typedef enum { + STATE_NONE, + STATE_HEADER, +@@ -72,7 +71,6 @@ public: + buf_ofs(0), + connstate(STATE_HEADER) + { +- *buf = 0; + + assert(NULL != r); + HTTPMSGLOCK(r); +@@ -92,10 +90,10 @@ public: + StoreEntry *e; + store_client *sc; + HttpRequest *r; +- int64_t used; +- size_t buf_sz; +- char buf[NETDB_REQBUF_SZ]; +- int buf_ofs; ++ ++ /// for receiving a NetDB reply body from Store and interpreting it ++ Store::ParsingBuffer parsingBuffer; ++ + netdb_conn_state_t connstate; + }; + +@@ -698,24 +696,20 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + Ip::Address addr; + + netdbExchangeState *ex = (netdbExchangeState *)data; +- int rec_sz = 0; +- int o; + + struct in_addr line_addr; + double rtt; + double hops; +- char *p; + int j; + HttpReply const *rep; +- size_t hdr_sz; + int nused = 0; +- int size; +- int oldbufofs = ex->buf_ofs; + +- rec_sz = 0; ++ size_t rec_sz = 0; // received record size (TODO: make const) + rec_sz += 1 + sizeof(struct in_addr); + rec_sz += 1 + sizeof(int); + rec_sz += 1 + sizeof(int); ++ // to make progress without growing buffer space, we must parse at least one record per call ++ Assure(rec_sz <= ex->parsingBuffer.capacity()); + debugs(38, 3, "netdbExchangeHandleReply: " << receivedData.length << " read bytes"); + + if (!cbdataReferenceValid(ex->p)) { +@@ -726,64 +720,28 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + + debugs(38, 3, "netdbExchangeHandleReply: for '" << ex->p->host << ":" << ex->p->http_port << "'"); + +- if (receivedData.length == 0 && !receivedData.flags.error) { +- debugs(38, 3, "netdbExchangeHandleReply: Done"); ++ if (receivedData.flags.error) { + delete ex; + return; + } + +- p = ex->buf; +- +- /* Get the size of the buffer now */ +- size = ex->buf_ofs + receivedData.length; +- debugs(38, 3, "netdbExchangeHandleReply: " << size << " bytes buf"); +- +- /* Check if we're still doing headers */ +- + if (ex->connstate == STATE_HEADER) { +- +- ex->buf_ofs += receivedData.length; +- +- /* skip reply headers */ +- +- if ((hdr_sz = headersEnd(p, ex->buf_ofs))) { +- debugs(38, 5, "netdbExchangeHandleReply: hdr_sz = " << hdr_sz); +- rep = ex->e->getReply(); +- assert(rep->sline.status() != Http::scNone); +- debugs(38, 3, "netdbExchangeHandleReply: reply status " << rep->sline.status()); +- +- if (rep->sline.status() != Http::scOkay) { +- delete ex; +- return; +- } +- +- assert((size_t)ex->buf_ofs >= hdr_sz); +- +- /* +- * Now, point p to the part of the buffer where the data +- * starts, and update the size accordingly +- */ +- assert(ex->used == 0); +- ex->used = hdr_sz; +- size = ex->buf_ofs - hdr_sz; +- p += hdr_sz; +- +- /* Finally, set the conn state mode to STATE_BODY */ +- ex->connstate = STATE_BODY; +- } else { +- StoreIOBuffer tempBuffer; +- tempBuffer.offset = ex->buf_ofs; +- tempBuffer.length = ex->buf_sz - ex->buf_ofs; +- tempBuffer.data = ex->buf + ex->buf_ofs; +- /* Have more headers .. */ +- storeClientCopy(ex->sc, ex->e, tempBuffer, +- netdbExchangeHandleReply, ex); ++ const auto scode = ex->e->mem().baseReply().sline.status(); ++ assert(scode != Http::scNone); ++ debugs(38, 3, "reply status " << scode); ++ if (scode != Http::scOkay) { ++ delete ex; + return; + } ++ ex->connstate = STATE_BODY; + } + + assert(ex->connstate == STATE_BODY); + ++ ex->parsingBuffer.appended(receivedData.data, receivedData.length); ++ auto p = ex->parsingBuffer.c_str(); // current parsing position ++ auto size = ex->parsingBuffer.contentSize(); // bytes we still need to parse ++ + /* If we get here, we have some body to parse .. */ + debugs(38, 5, "netdbExchangeHandleReply: start parsing loop, size = " << size); + +@@ -792,6 +750,7 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + addr.setAnyAddr(); + hops = rtt = 0.0; + ++ size_t o; // current record parsing offset + for (o = 0; o < rec_sz;) { + switch ((int) *(p + o)) { + +@@ -829,8 +788,6 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + + assert(o == rec_sz); + +- ex->used += rec_sz; +- + size -= rec_sz; + + p += rec_sz; +@@ -838,32 +795,8 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + ++nused; + } + +- /* +- * Copy anything that is left over to the beginning of the buffer, +- * and adjust buf_ofs accordingly +- */ +- +- /* +- * Evilly, size refers to the buf size left now, +- * ex->buf_ofs is the original buffer size, so just copy that +- * much data over +- */ +- memmove(ex->buf, ex->buf + (ex->buf_ofs - size), size); +- +- ex->buf_ofs = size; +- +- /* +- * And don't re-copy the remaining data .. +- */ +- ex->used += size; +- +- /* +- * Now the tricky bit - size _included_ the leftover bit from the _last_ +- * storeClientCopy. We don't want to include that, or our offset will be wrong. +- * So, don't count the size of the leftover buffer we began with. +- * This can _disappear_ when we're not tracking offsets .. +- */ +- ex->used -= oldbufofs; ++ const auto parsedSize = ex->parsingBuffer.contentSize() - size; ++ ex->parsingBuffer.consume(parsedSize); + + debugs(38, 3, "netdbExchangeHandleReply: size left over in this buffer: " << size << " bytes"); + +@@ -871,20 +804,26 @@ netdbExchangeHandleReply(void *data, StoreIOBuffer receivedData) + " entries, (x " << rec_sz << " bytes) == " << nused * rec_sz << + " bytes total"); + +- debugs(38, 3, "netdbExchangeHandleReply: used " << ex->used); +- + if (EBIT_TEST(ex->e->flags, ENTRY_ABORTED)) { + debugs(38, 3, "netdbExchangeHandleReply: ENTRY_ABORTED"); + delete ex; +- } else if (ex->e->store_status == STORE_PENDING) { +- StoreIOBuffer tempBuffer; +- tempBuffer.offset = ex->used; +- tempBuffer.length = ex->buf_sz - ex->buf_ofs; +- tempBuffer.data = ex->buf + ex->buf_ofs; +- debugs(38, 3, "netdbExchangeHandleReply: EOF not received"); +- storeClientCopy(ex->sc, ex->e, tempBuffer, +- netdbExchangeHandleReply, ex); ++ return; + } ++ ++ if (ex->sc->atEof()) { ++ if (const auto leftoverBytes = ex->parsingBuffer.contentSize()) ++ debugs(38, 2, "discarding a partially received record due to Store EOF: " << leftoverBytes); ++ delete ex; ++ return; ++ } ++ ++ // TODO: To protect us from a broken peer sending an "infinite" stream of ++ // new addresses, limit the cumulative number of received bytes or records? ++ ++ const auto remainingSpace = ex->parsingBuffer.space().positionAt(receivedData.offset + receivedData.length); ++ // rec_sz is at most buffer capacity, and we consume all fully loaded records ++ Assure(remainingSpace.length); ++ storeClientCopy(ex->sc, ex->e, remainingSpace, netdbExchangeHandleReply, ex); + } + + #endif /* USE_ICMP */ +@@ -1296,14 +1235,9 @@ netdbExchangeStart(void *data) + ex->e = storeCreateEntry(uri, uri, RequestFlags(), Http::METHOD_GET); + assert(NULL != ex->e); + +- StoreIOBuffer tempBuffer; +- tempBuffer.length = ex->buf_sz; +- tempBuffer.data = ex->buf; +- + ex->sc = storeClientListAdd(ex->e, ex); ++ storeClientCopy(ex->sc, ex->e, ex->parsingBuffer.makeInitialSpace(), netdbExchangeHandleReply, ex); + +- storeClientCopy(ex->sc, ex->e, tempBuffer, +- netdbExchangeHandleReply, ex); + ex->r->flags.loopDetected = true; /* cheat! -- force direct */ + + // XXX: send as Proxy-Authenticate instead +diff --git a/src/peer_digest.cc b/src/peer_digest.cc +index 7b6314d..abfea4a 100644 +--- a/src/peer_digest.cc ++++ b/src/peer_digest.cc +@@ -39,7 +39,6 @@ static EVH peerDigestCheck; + static void peerDigestRequest(PeerDigest * pd); + static STCB peerDigestHandleReply; + static int peerDigestFetchReply(void *, char *, ssize_t); +-int peerDigestSwapInHeaders(void *, char *, ssize_t); + int peerDigestSwapInCBlock(void *, char *, ssize_t); + int peerDigestSwapInMask(void *, char *, ssize_t); + static int peerDigestFetchedEnough(DigestFetchState * fetch, char *buf, ssize_t size, const char *step_name); +@@ -374,6 +373,9 @@ peerDigestRequest(PeerDigest * pd) + fetch->sc = storeClientListAdd(e, fetch); + /* set lastmod to trigger IMS request if possible */ + ++ // TODO: Also check for fetch->pd->cd presence as a precondition for sending ++ // IMS requests because peerDigestFetchReply() does not accept 304 responses ++ // without an in-memory cache digest. + if (old_e) + e->lastModified(old_e->lastModified()); + +@@ -408,6 +410,11 @@ peerDigestHandleReply(void *data, StoreIOBuffer receivedData) + digest_read_state_t prevstate; + int newsize; + ++ if (receivedData.flags.error) { ++ peerDigestFetchAbort(fetch, fetch->buf, "failure loading digest reply from Store"); ++ return; ++ } ++ + assert(fetch->pd && receivedData.data); + /* The existing code assumes that the received pointer is + * where we asked the data to be put +@@ -444,10 +451,6 @@ peerDigestHandleReply(void *data, StoreIOBuffer receivedData) + retsize = peerDigestFetchReply(fetch, fetch->buf, fetch->bufofs); + break; + +- case DIGEST_READ_HEADERS: +- retsize = peerDigestSwapInHeaders(fetch, fetch->buf, fetch->bufofs); +- break; +- + case DIGEST_READ_CBLOCK: + retsize = peerDigestSwapInCBlock(fetch, fetch->buf, fetch->bufofs); + break; +@@ -487,7 +490,7 @@ peerDigestHandleReply(void *data, StoreIOBuffer receivedData) + // checking at the beginning of this function. However, in this case, we would have to require + // that the parser does not regard EOF as a special condition (it is true now but may change + // in the future). +- if (!receivedData.length) { // EOF ++ if (fetch->sc->atEof()) { + peerDigestFetchAbort(fetch, fetch->buf, "premature end of digest reply"); + return; + } +@@ -506,19 +509,12 @@ peerDigestHandleReply(void *data, StoreIOBuffer receivedData) + } + } + +-/* wait for full http headers to be received then parse them */ +-/* +- * This routine handles parsing the reply line. +- * If the reply line indicates an OK, the same data is thrown +- * to SwapInHeaders(). If the reply line is a NOT_MODIFIED, +- * we simply stop parsing. +- */ ++/// handle HTTP response headers in the initial storeClientCopy() response + static int + peerDigestFetchReply(void *data, char *buf, ssize_t size) + { + DigestFetchState *fetch = (DigestFetchState *)data; + PeerDigest *pd = fetch->pd; +- size_t hdr_size; + assert(pd && buf); + assert(!fetch->offset); + +@@ -527,7 +523,7 @@ peerDigestFetchReply(void *data, char *buf, ssize_t size) + if (peerDigestFetchedEnough(fetch, buf, size, "peerDigestFetchReply")) + return -1; + +- if ((hdr_size = headersEnd(buf, size))) { ++ { + HttpReply const *reply = fetch->entry->getReply(); + assert(reply); + assert(reply->sline.status() != Http::scNone); +@@ -563,6 +559,15 @@ peerDigestFetchReply(void *data, char *buf, ssize_t size) + /* preserve request -- we need its size to update counters */ + /* requestUnlink(r); */ + /* fetch->entry->mem_obj->request = NULL; */ ++ ++ if (!fetch->pd->cd) { ++ peerDigestFetchAbort(fetch, buf, "304 without the old in-memory digest"); ++ return -1; ++ } ++ ++ // stay with the old in-memory digest ++ peerDigestFetchStop(fetch, buf, "Not modified"); ++ fetch->state = DIGEST_READ_DONE; + } else if (status == Http::scOkay) { + /* get rid of old entry if any */ + +@@ -573,70 +578,15 @@ peerDigestFetchReply(void *data, char *buf, ssize_t size) + fetch->old_entry->unlock("peerDigestFetchReply 200"); + fetch->old_entry = NULL; + } ++ ++ fetch->state = DIGEST_READ_CBLOCK; + } else { + /* some kind of a bug */ + peerDigestFetchAbort(fetch, buf, reply->sline.reason()); + return -1; /* XXX -1 will abort stuff in ReadReply! */ + } + +- /* must have a ready-to-use store entry if we got here */ +- /* can we stay with the old in-memory digest? */ +- if (status == Http::scNotModified && fetch->pd->cd) { +- peerDigestFetchStop(fetch, buf, "Not modified"); +- fetch->state = DIGEST_READ_DONE; +- } else { +- fetch->state = DIGEST_READ_HEADERS; +- } +- } else { +- /* need more data, do we have space? */ +- +- if (size >= SM_PAGE_SIZE) +- peerDigestFetchAbort(fetch, buf, "reply header too big"); +- } +- +- /* We don't want to actually ack that we've handled anything, +- * otherwise SwapInHeaders() won't get the reply line .. */ +- return 0; +-} +- +-/* fetch headers from disk, pass on to SwapInCBlock */ +-int +-peerDigestSwapInHeaders(void *data, char *buf, ssize_t size) +-{ +- DigestFetchState *fetch = (DigestFetchState *)data; +- size_t hdr_size; +- +- assert(fetch->state == DIGEST_READ_HEADERS); +- +- if (peerDigestFetchedEnough(fetch, buf, size, "peerDigestSwapInHeaders")) +- return -1; +- +- assert(!fetch->offset); +- +- if ((hdr_size = headersEnd(buf, size))) { +- assert(fetch->entry->getReply()); +- assert(fetch->entry->getReply()->sline.status() != Http::scNone); +- +- if (fetch->entry->getReply()->sline.status() != Http::scOkay) { +- debugs(72, DBG_IMPORTANT, "peerDigestSwapInHeaders: " << fetch->pd->host << +- " status " << fetch->entry->getReply()->sline.status() << +- " got cached!"); +- +- peerDigestFetchAbort(fetch, buf, "internal status error"); +- return -1; +- } +- +- fetch->state = DIGEST_READ_CBLOCK; +- return hdr_size; /* Say how much data we read */ +- } +- +- /* need more data, do we have space? */ +- if (size >= SM_PAGE_SIZE) { +- peerDigestFetchAbort(fetch, buf, "stored header too big"); +- return -1; +- } +- +- return 0; /* We need to read more to parse .. */ ++ return 0; // we consumed/used no buffered bytes + } + + int +diff --git a/src/store.cc b/src/store.cc +index 1948447..b4c7f82 100644 +--- a/src/store.cc ++++ b/src/store.cc +@@ -273,6 +273,8 @@ StoreEntry::storeClientType() const + + assert(mem_obj); + ++ debugs(20, 7, *this << " inmem_lo=" << mem_obj->inmem_lo); ++ + if (mem_obj->inmem_lo) + return STORE_DISK_CLIENT; + +@@ -300,6 +302,7 @@ StoreEntry::storeClientType() const + return STORE_MEM_CLIENT; + } + } ++ debugs(20, 7, "STORE_OK STORE_DISK_CLIENT"); + return STORE_DISK_CLIENT; + } + +@@ -319,10 +322,18 @@ StoreEntry::storeClientType() const + if (swap_status == SWAPOUT_NONE) + return STORE_MEM_CLIENT; + ++ // TODO: The above "must make this a mem client" logic contradicts "Slight ++ // weirdness" logic in store_client::doCopy() that converts hits to misses ++ // on startSwapin() failures. We should probably attempt to open a swapin ++ // file _here_ instead (and avoid STORE_DISK_CLIENT designation for clients ++ // that fail to do so). That would also address a similar problem with Rock ++ // store that does not yet support swapin during SWAPOUT_WRITING. ++ + /* + * otherwise, make subsequent clients read from disk so they + * can not delay the first, and vice-versa. + */ ++ debugs(20, 7, "STORE_PENDING STORE_DISK_CLIENT"); + return STORE_DISK_CLIENT; + } + +diff --git a/src/store/Makefile.am b/src/store/Makefile.am +index be177d8..ccfc2dd 100644 +--- a/src/store/Makefile.am ++++ b/src/store/Makefile.am +@@ -23,4 +23,6 @@ libstore_la_SOURCES= \ + forward.h \ + LocalSearch.cc \ + LocalSearch.h \ ++ ParsingBuffer.cc \ ++ ParsingBuffer.h \ + Storage.h +diff --git a/src/store/Makefile.in b/src/store/Makefile.in +index bb4387d..1ea6c45 100644 +--- a/src/store/Makefile.in ++++ b/src/store/Makefile.in +@@ -163,7 +163,7 @@ CONFIG_CLEAN_FILES = + CONFIG_CLEAN_VPATH_FILES = + LTLIBRARIES = $(noinst_LTLIBRARIES) + libstore_la_LIBADD = +-am_libstore_la_OBJECTS = Controller.lo Disk.lo Disks.lo LocalSearch.lo ++am_libstore_la_OBJECTS = Controller.lo Disk.lo Disks.lo LocalSearch.lo ParsingBuffer.lo + libstore_la_OBJECTS = $(am_libstore_la_OBJECTS) + AM_V_lt = $(am__v_lt_@AM_V@) + am__v_lt_ = $(am__v_lt_@AM_DEFAULT_V@) +@@ -184,7 +184,7 @@ am__v_at_1 = + DEFAULT_INCLUDES = + depcomp = $(SHELL) $(top_srcdir)/cfgaux/depcomp + am__maybe_remake_depfiles = depfiles +-am__depfiles_remade = ./$(DEPDIR)/Controller.Plo ./$(DEPDIR)/Disk.Plo \ ++am__depfiles_remade = ./$(DEPDIR)/Controller.Plo ./$(DEPDIR)/Disk.Plo ./$(DEPDIR)/ParsingBuffer.Plo \ + ./$(DEPDIR)/Disks.Plo ./$(DEPDIR)/LocalSearch.Plo + am__mv = mv -f + CXXCOMPILE = $(CXX) $(DEFS) $(DEFAULT_INCLUDES) $(INCLUDES) \ +@@ -776,6 +776,8 @@ libstore_la_SOURCES = \ + forward.h \ + LocalSearch.cc \ + LocalSearch.h \ ++ ParsingBuffer.cc \ ++ ParsingBuffer.h \ + Storage.h + + all: all-recursive +@@ -846,6 +848,7 @@ distclean-compile: + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/Disk.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/Disks.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/LocalSearch.Plo@am__quote@ # am--include-marker ++@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/ParsingBuffer.Plo@am__quote@ # am--include-marker + + $(am__depfiles_remade): + @$(MKDIR_P) $(@D) +@@ -1254,6 +1257,7 @@ distclean: distclean-recursive + -rm -f ./$(DEPDIR)/Disk.Plo + -rm -f ./$(DEPDIR)/Disks.Plo + -rm -f ./$(DEPDIR)/LocalSearch.Plo ++ -rm -f ./$(DEPDIR)/ParsingBuffer.Plo + -rm -f Makefile + distclean-am: clean-am distclean-compile distclean-generic \ + distclean-tags +@@ -1303,6 +1307,7 @@ maintainer-clean: maintainer-clean-recursive + -rm -f ./$(DEPDIR)/Disk.Plo + -rm -f ./$(DEPDIR)/Disks.Plo + -rm -f ./$(DEPDIR)/LocalSearch.Plo ++ -rm -f ./$(DEPDIR)/ParsingBuffer.Plo + -rm -f Makefile + maintainer-clean-am: distclean-am maintainer-clean-generic + +diff --git a/src/store/ParsingBuffer.cc b/src/store/ParsingBuffer.cc +new file mode 100644 +index 0000000..e948fe2 +--- /dev/null ++++ b/src/store/ParsingBuffer.cc +@@ -0,0 +1,198 @@ ++/* ++ * Copyright (C) 1996-2023 The Squid Software Foundation and contributors ++ * ++ * Squid software is distributed under GPLv2+ license and includes ++ * contributions from numerous individuals and organizations. ++ * Please see the COPYING and CONTRIBUTORS files for details. ++ */ ++ ++#include "squid.h" ++#include "sbuf/Stream.h" ++#include "SquidMath.h" ++#include "store/ParsingBuffer.h" ++#include "base/Assure.h" ++#include ++ ++// Several Store::ParsingBuffer() methods use assert() because the corresponding ++// failure means there is a good chance that somebody have already read from (or ++// written to) the wrong memory location. Since this buffer is used for storing ++// HTTP response bytes, such failures may corrupt traffic. No Assure() handling ++// code can safely recover from such failures. ++ ++Store::ParsingBuffer::ParsingBuffer(StoreIOBuffer &initialSpace): ++ readerSuppliedMemory_(initialSpace) ++{ ++} ++ ++/// a read-only content start (or nil for some zero-size buffers) ++const char * ++Store::ParsingBuffer::memory() const ++{ ++ return extraMemory_ ? extraMemory_->rawContent() : readerSuppliedMemory_.data; ++} ++ ++size_t ++Store::ParsingBuffer::capacity() const ++{ ++ return extraMemory_ ? (extraMemory_->length() + extraMemory_->spaceSize()) : readerSuppliedMemory_.length; ++} ++ ++size_t ++Store::ParsingBuffer::contentSize() const ++{ ++ return extraMemory_ ? extraMemory_->length() : readerSuppliedMemoryContentSize_; ++} ++ ++void ++Store::ParsingBuffer::appended(const char * const newBytes, const size_t newByteCount) ++{ ++ // a positive newByteCount guarantees that, after the first assertion below ++ // succeeds, the second assertion will not increment a nil memory() pointer ++ if (!newByteCount) ++ return; ++ ++ // these checks order guarantees that memory() is not nil in the second assertion ++ assert(newByteCount <= spaceSize()); // the new bytes end in our space ++ assert(memory() + contentSize() == newBytes); // the new bytes start in our space ++ // and now we know that newBytes is not nil either ++ ++ if (extraMemory_) ++ extraMemory_->rawAppendFinish(newBytes, newByteCount); ++ else ++ readerSuppliedMemoryContentSize_ = *IncreaseSum(readerSuppliedMemoryContentSize_, newByteCount); ++ ++ assert(contentSize() <= capacity()); // paranoid ++} ++ ++void ++Store::ParsingBuffer::consume(const size_t parsedBytes) ++{ ++ Assure(contentSize() >= parsedBytes); // more conservative than extraMemory_->consume() ++ if (extraMemory_) { ++ extraMemory_->consume(parsedBytes); ++ } else { ++ readerSuppliedMemoryContentSize_ -= parsedBytes; ++ if (parsedBytes && readerSuppliedMemoryContentSize_) ++ memmove(readerSuppliedMemory_.data, memory() + parsedBytes, readerSuppliedMemoryContentSize_); ++ } ++} ++ ++StoreIOBuffer ++Store::ParsingBuffer::space() ++{ ++ const auto size = spaceSize(); ++ const auto start = extraMemory_ ? ++ extraMemory_->rawAppendStart(size) : ++ (readerSuppliedMemory_.data + readerSuppliedMemoryContentSize_); ++ return StoreIOBuffer(spaceSize(), 0, start); ++} ++ ++StoreIOBuffer ++Store::ParsingBuffer::makeSpace(const size_t pageSize) ++{ ++ growSpace(pageSize); ++ auto result = space(); ++ Assure(result.length >= pageSize); ++ result.length = pageSize; ++ return result; ++} ++ ++StoreIOBuffer ++Store::ParsingBuffer::content() const ++{ ++ // This const_cast is a StoreIOBuffer API limitation: That class does not ++ // support a "constant content view", even though it is used as such a view. ++ return StoreIOBuffer(contentSize(), 0, const_cast(memory())); ++} ++ ++/// makes sure we have the requested number of bytes, allocates enough memory if needed ++void ++Store::ParsingBuffer::growSpace(const size_t minimumSpaceSize) ++{ ++ const auto capacityIncreaseAttempt = IncreaseSum(contentSize(), minimumSpaceSize); ++ if (!capacityIncreaseAttempt) ++ throw TextException(ToSBuf("no support for a single memory block of ", contentSize(), '+', minimumSpaceSize, " bytes"), Here()); ++ const auto newCapacity = *capacityIncreaseAttempt; ++ ++ if (newCapacity <= capacity()) ++ return; // already have enough space; no reallocation is needed ++ ++ debugs(90, 7, "growing to provide " << minimumSpaceSize << " in " << *this); ++ ++ if (extraMemory_) { ++ extraMemory_->reserveCapacity(newCapacity); ++ } else { ++ SBuf newStorage; ++ newStorage.reserveCapacity(newCapacity); ++ newStorage.append(readerSuppliedMemory_.data, readerSuppliedMemoryContentSize_); ++ extraMemory_ = std::move(newStorage); ++ } ++ Assure(spaceSize() >= minimumSpaceSize); ++} ++ ++SBuf ++Store::ParsingBuffer::toSBuf() const ++{ ++ return extraMemory_ ? *extraMemory_ : SBuf(content().data, content().length); ++} ++ ++size_t ++Store::ParsingBuffer::spaceSize() const ++{ ++ if (extraMemory_) ++ return extraMemory_->spaceSize(); ++ ++ assert(readerSuppliedMemoryContentSize_ <= readerSuppliedMemory_.length); ++ return readerSuppliedMemory_.length - readerSuppliedMemoryContentSize_; ++} ++ ++/// 0-terminates stored byte sequence, allocating more memory if needed, but ++/// without increasing the number of stored content bytes ++void ++Store::ParsingBuffer::terminate() ++{ ++ *makeSpace(1).data = 0; ++} ++ ++StoreIOBuffer ++Store::ParsingBuffer::packBack() ++{ ++ const auto bytesToPack = contentSize(); ++ // until our callers do not have to work around legacy code expectations ++ Assure(bytesToPack); ++ ++ // if we accumulated more bytes at some point, any extra metadata should ++ // have been consume()d by now, allowing readerSuppliedMemory_.data reuse ++ Assure(bytesToPack <= readerSuppliedMemory_.length); ++ ++ auto result = readerSuppliedMemory_; ++ result.length = bytesToPack; ++ Assure(result.data); ++ ++ if (!extraMemory_) { ++ // no accumulated bytes copying because they are in readerSuppliedMemory_ ++ debugs(90, 7, "quickly exporting " << result.length << " bytes via " << readerSuppliedMemory_); ++ } else { ++ debugs(90, 7, "slowly exporting " << result.length << " bytes from " << extraMemory_->id << " back into " << readerSuppliedMemory_); ++ memmove(result.data, extraMemory_->rawContent(), result.length); ++ } ++ ++ return result; ++} ++ ++void ++Store::ParsingBuffer::print(std::ostream &os) const ++{ ++ os << "size=" << contentSize(); ++ ++ if (extraMemory_) { ++ os << " capacity=" << capacity(); ++ os << " extra=" << extraMemory_->id; ++ } ++ ++ // report readerSuppliedMemory_ (if any) even if we are no longer using it ++ // for content storage; it affects packBack() and related parsing logic ++ if (readerSuppliedMemory_.length) ++ os << ' ' << readerSuppliedMemory_; ++} ++ +diff --git a/src/store/ParsingBuffer.h b/src/store/ParsingBuffer.h +new file mode 100644 +index 0000000..b8aa957 +--- /dev/null ++++ b/src/store/ParsingBuffer.h +@@ -0,0 +1,128 @@ ++/* ++ * Copyright (C) 1996-2023 The Squid Software Foundation and contributors ++ * ++ * Squid software is distributed under GPLv2+ license and includes ++ * contributions from numerous individuals and organizations. ++ * Please see the COPYING and CONTRIBUTORS files for details. ++ */ ++ ++#ifndef SQUID_SRC_STORE_PARSINGBUFFER_H ++#define SQUID_SRC_STORE_PARSINGBUFFER_H ++ ++#include "sbuf/SBuf.h" ++#include "StoreIOBuffer.h" ++ ++#include ++ ++namespace Store ++{ ++ ++/// A continuous buffer for efficient accumulation and NUL-termination of ++/// Store-read bytes. The buffer accumulates two kinds of Store readers: ++/// ++/// * Readers that do not have any external buffer to worry about but need to ++/// accumulate, terminate, and/or consume buffered content read by Store. ++/// These readers use the default constructor and then allocate the initial ++/// buffer space for their first read (if any). ++/// ++/// * Readers that supply their StoreIOBuffer at construction time. That buffer ++/// is enough to handle the majority of use cases. However, the supplied ++/// StoreIOBuffer capacity may be exceeded when parsing requires accumulating ++/// multiple Store read results and/or NUL-termination of a full buffer. ++/// ++/// This buffer seamlessly grows as needed, reducing memory over-allocation and, ++/// in case of StoreIOBuffer-seeded construction, memory copies. ++class ParsingBuffer ++{ ++public: ++ /// creates buffer without any space or content ++ ParsingBuffer() = default; ++ ++ /// seeds this buffer with the caller-supplied buffer space ++ explicit ParsingBuffer(StoreIOBuffer &); ++ ++ /// a NUL-terminated version of content(); same lifetime as content() ++ const char *c_str() { terminate(); return memory(); } ++ ++ /// export content() into SBuf, avoiding content copying when possible ++ SBuf toSBuf() const; ++ ++ /// the total number of append()ed bytes that were not consume()d ++ size_t contentSize() const; ++ ++ /// the number of bytes in the space() buffer ++ size_t spaceSize() const; ++ ++ /// the maximum number of bytes we can store without allocating more space ++ size_t capacity() const; ++ ++ /// Stored append()ed bytes that have not been consume()d. The returned ++ /// buffer offset is set to zero; the caller is responsible for adjusting ++ /// the offset if needed (TODO: Add/return a no-offset Mem::View instead). ++ /// The returned buffer is invalidated by calling a non-constant method or ++ /// by changing the StoreIOBuffer contents given to our constructor. ++ StoreIOBuffer content() const; ++ ++ /// A (possibly empty) buffer for reading the next byte(s). The returned ++ /// buffer offset is set to zero; the caller is responsible for adjusting ++ /// the offset if needed (TODO: Add/return a no-offset Mem::Area instead). ++ /// The returned buffer is invalidated by calling a non-constant method or ++ /// by changing the StoreIOBuffer contents given to our constructor. ++ StoreIOBuffer space(); ++ ++ /// A buffer for reading the exact number of next byte(s). The method may ++ /// allocate new memory and copy previously appended() bytes as needed. ++ /// \param pageSize the exact number of bytes the caller wants to read ++ /// \returns space() after any necessary allocations ++ StoreIOBuffer makeSpace(size_t pageSize); ++ ++ /// A buffer suitable for the first storeClientCopy() call. The method may ++ /// allocate new memory and copy previously appended() bytes as needed. ++ /// \returns space() after any necessary allocations ++ /// \deprecated New clients should call makeSpace() with client-specific ++ /// pageSize instead of this one-size-fits-all legacy method. ++ StoreIOBuffer makeInitialSpace() { return makeSpace(4096); } ++ ++ /// remember the new bytes received into the previously provided space() ++ void appended(const char *, size_t); ++ ++ /// get rid of previously appended() prefix of a given size ++ void consume(size_t); ++ ++ /// Returns stored content, reusing the StoreIOBuffer given at the ++ /// construction time. Copying is avoided if we did not allocate extra ++ /// memory since construction. Not meant for default-constructed buffers. ++ /// \prec positive contentSize() (\sa store_client::finishCallback()) ++ StoreIOBuffer packBack(); ++ ++ /// summarizes object state (for debugging) ++ void print(std::ostream &) const; ++ ++private: ++ const char *memory() const; ++ void terminate(); ++ void growSpace(size_t); ++ ++private: ++ /// externally allocated buffer we were seeded with (or a zero-size one) ++ StoreIOBuffer readerSuppliedMemory_; ++ ++ /// append()ed to readerSuppliedMemory_ bytes that were not consume()d ++ size_t readerSuppliedMemoryContentSize_ = 0; ++ ++ /// our internal buffer that takes over readerSuppliedMemory_ when the ++ /// latter becomes full and more memory is needed ++ std::optional extraMemory_; ++}; ++ ++inline std::ostream & ++operator <<(std::ostream &os, const ParsingBuffer &b) ++{ ++ b.print(os); ++ return os; ++} ++ ++} // namespace Store ++ ++#endif /* SQUID_SRC_STORE_PARSINGBUFFER_H */ ++ +diff --git a/src/store/forward.h b/src/store/forward.h +index 1422a85..db5ee1c 100644 +--- a/src/store/forward.h ++++ b/src/store/forward.h +@@ -46,6 +46,7 @@ class Disks; + class Disk; + class DiskConfig; + class EntryGuard; ++class ParsingBuffer; + + typedef ::StoreEntry Entry; + typedef ::MemStore Memory; +diff --git a/src/store_client.cc b/src/store_client.cc +index 207c96b..1731c4c 100644 +--- a/src/store_client.cc ++++ b/src/store_client.cc +@@ -16,9 +16,11 @@ + #include "HttpRequest.h" + #include "MemBuf.h" + #include "MemObject.h" ++#include "sbuf/Stream.h" + #include "mime_header.h" + #include "profiler/Profiler.h" + #include "SquidConfig.h" ++#include "SquidMath.h" + #include "StatCounters.h" + #include "Store.h" + #include "store_swapin.h" +@@ -98,19 +100,6 @@ storeClientListAdd(StoreEntry * e, void *data) + return sc; + } + +-/// schedules asynchronous STCB call to relay disk or memory read results +-/// \param outcome an error signal (if negative), an EOF signal (if zero), or the number of bytes read +-void +-store_client::callback(const ssize_t outcome) +-{ +- if (outcome > 0) +- return noteCopiedBytes(outcome); +- +- if (outcome < 0) +- return fail(); +- +- noteEof(); +-} + /// finishCallback() wrapper; TODO: Add NullaryMemFunT for non-jobs. + void + store_client::FinishCallback(store_client * const sc) +@@ -125,14 +114,20 @@ store_client::finishCallback() + Assure(_callback.callback_handler); + Assure(_callback.notifier); + +- // callers are not ready to handle a content+error combination +- Assure(object_ok || !copiedSize); +- +- StoreIOBuffer result(copiedSize, copyInto.offset, copyInto.data); ++ // XXX: Some legacy code relies on zero-length buffers having nil data ++ // pointers. Some other legacy code expects "correct" result.offset even ++ // when there is no body to return. Accommodate all those expectations. ++ auto result = StoreIOBuffer(0, copyInto.offset, nullptr); ++ if (object_ok && parsingBuffer && parsingBuffer->contentSize()) ++ result = parsingBuffer->packBack(); + result.flags.error = object_ok ? 0 : 1; +- copiedSize = 0; + +- cmp_offset = result.offset + result.length; ++ // no HTTP headers and no body bytes (but not because there was no space) ++ atEof_ = !sendingHttpHeaders() && !result.length && copyInto.length; ++ ++ parsingBuffer.reset(); ++ ++answers; ++ + STCB *temphandler = _callback.callback_handler; + void *cbdata = _callback.callback_data; + _callback = Callback(NULL, NULL); +@@ -144,35 +139,15 @@ store_client::finishCallback() + cbdataReferenceDone(cbdata); + } + +-/// schedules asynchronous STCB call to relay a successful disk or memory read +-/// \param bytesCopied the number of response bytes copied into copyInto +-void +-store_client::noteCopiedBytes(const size_t bytesCopied) +-{ +- debugs(90, 5, bytesCopied); +- Assure(bytesCopied > 0); +- Assure(!copiedSize); +- copiedSize = bytesCopied; +- noteNews(); +-} +- +-void +-store_client::noteEof() +-{ +- debugs(90, 5, copiedSize); +- Assure(!copiedSize); +- noteNews(); +-} +- + store_client::store_client(StoreEntry *e) : +- cmp_offset(0), + #if STORE_CLIENT_LIST_DEBUG + owner(cbdataReference(data)), + #endif + entry(e), + type(e->storeClientType()), + object_ok(true), +- copiedSize(0) ++ atEof_(false), ++ answers(0) + { + flags.disk_io_pending = false; + flags.store_copying = false; +@@ -221,16 +196,29 @@ store_client::copy(StoreEntry * anEntry, + #endif + + assert(!_callback.pending()); +-#if ONLYCONTIGUOUSREQUESTS +- +- assert(cmp_offset == copyRequest.offset); +-#endif +- /* range requests will skip into the body */ +- cmp_offset = copyRequest.offset; + _callback = Callback (callback_fn, cbdataReference(data)); + copyInto.data = copyRequest.data; + copyInto.length = copyRequest.length; + copyInto.offset = copyRequest.offset; ++ Assure(copyInto.offset >= 0); ++ ++ if (!copyInto.length) { ++ // During the first storeClientCopy() call, a zero-size buffer means ++ // that we will have to drop any HTTP response body bytes we read (with ++ // the HTTP headers from disk). After that, it means we cannot return ++ // anything to the caller at all. ++ debugs(90, 2, "WARNING: zero-size storeClientCopy() buffer: " << copyInto); ++ // keep going; moreToRead() should prevent any from-Store reading ++ } ++ ++ // Our nextHttpReadOffset() expects the first copy() call to have zero ++ // offset. More complex code could handle a positive first offset, but it ++ // would only be useful when reading responses from memory: We would not ++ // _delay_ the response (to read the requested HTTP body bytes from disk) ++ // when we already can respond with HTTP headers. ++ Assure(!copyInto.offset || answeredOnce()); ++ ++ parsingBuffer.emplace(copyInto); + + static bool copying (false); + assert (!copying); +@@ -258,33 +246,30 @@ store_client::copy(StoreEntry * anEntry, + // Add no code here. This object may no longer exist. + } + +-/// Whether there is (or will be) more entry data for us. ++/// Whether Store has (or possibly will have) more entry data for us. + bool +-store_client::moreToSend() const ++store_client::moreToRead() const + { ++ if (!copyInto.length) ++ return false; // the client supplied a zero-size buffer ++ + if (entry->store_status == STORE_PENDING) + return true; // there may be more coming + + /* STORE_OK, including aborted entries: no more data is coming */ + +- const int64_t len = entry->objectLen(); ++ if (canReadFromMemory()) ++ return true; // memory has the first byte wanted by the client + +- // If we do not know the entry length, then we have to open the swap file. +- const bool canSwapIn = entry->hasDisk(); +- if (len < 0) +- return canSwapIn; +- +- if (copyInto.offset >= len) +- return false; // sent everything there is ++ if (!entry->hasDisk()) ++ return false; // cannot read anything from disk either + +- if (canSwapIn) +- return true; // if we lack prefix, we can swap it in ++ if (entry->objectLen() >= 0 && copyInto.offset >= entry->contentLen()) ++ return false; // the disk cannot have byte(s) wanted by the client + +- // If we cannot swap in, make sure we have what we want in RAM. Otherwise, +- // scheduleRead calls scheduleDiskRead which asserts without a swap file. +- const MemObject *mem = entry->mem_obj; +- return mem && +- mem->inmem_lo <= copyInto.offset && copyInto.offset < mem->endOffset(); ++ // we cannot be sure until we swap in metadata and learn contentLen(), ++ // but the disk may have the byte(s) wanted by the client ++ return true; + } + + static void +@@ -311,6 +296,14 @@ storeClientCopy2(StoreEntry * e, store_client * sc) + sc->doCopy(e); + } + ++/// Whether our answer, if sent right now, will announce the availability of ++/// HTTP response headers (to the STCB callback) for the first time. ++bool ++store_client::sendingHttpHeaders() const ++{ ++ return !answeredOnce() && entry->mem().baseReply().hdr_sz > 0; ++} ++ + void + store_client::doCopy(StoreEntry *anEntry) + { +@@ -322,20 +315,22 @@ store_client::doCopy(StoreEntry *anEntry) + flags.store_copying = true; + MemObject *mem = entry->mem_obj; + +- debugs(33, 5, "store_client::doCopy: co: " << +- copyInto.offset << ", hi: " << +- mem->endOffset()); ++ debugs(33, 5, this << " into " << copyInto << ++ " hi: " << mem->endOffset() << ++ " objectLen: " << entry->objectLen() << ++ " past_answers: " << answers); + +- if (!moreToSend()) { ++ const auto sendHttpHeaders = sendingHttpHeaders(); ++ ++ if (!sendHttpHeaders && !moreToRead()) { + /* There is no more to send! */ + debugs(33, 3, HERE << "There is no more to send!"); +- noteEof(); ++ noteNews(); + flags.store_copying = false; + return; + } + +- /* Check that we actually have data */ +- if (anEntry->store_status == STORE_PENDING && copyInto.offset >= mem->endOffset()) { ++ if (!sendHttpHeaders && anEntry->store_status == STORE_PENDING && nextHttpReadOffset() >= mem->endOffset()) { + debugs(90, 3, "store_client::doCopy: Waiting for more"); + flags.store_copying = false; + return; +@@ -357,7 +352,24 @@ store_client::doCopy(StoreEntry *anEntry) + if (!startSwapin()) + return; // failure + } +- scheduleRead(); ++ ++ // send any immediately available body bytes even if we also sendHttpHeaders ++ if (canReadFromMemory()) { ++ readFromMemory(); ++ noteNews(); // will sendHttpHeaders (if needed) as well ++ flags.store_copying = false; ++ return; ++ } ++ ++ if (sendHttpHeaders) { ++ debugs(33, 5, "just send HTTP headers: " << mem->baseReply().hdr_sz); ++ noteNews(); ++ flags.store_copying = false; ++ return; ++ } ++ ++ // no information that the client needs is available immediately ++ scheduleDiskRead(); + } + + /// opens the swapin "file" if possible; otherwise, fail()s and returns false +@@ -397,18 +409,7 @@ store_client::noteSwapInDone(const bool error) + if (error) + fail(); + else +- noteEof(); +-} +- +-void +-store_client::scheduleRead() +-{ +- MemObject *mem = entry->mem_obj; +- +- if (copyInto.offset >= mem->inmem_lo && copyInto.offset < mem->endOffset()) +- scheduleMemRead(); +- else +- scheduleDiskRead(); ++ noteNews(); + } + + void +@@ -433,15 +434,44 @@ store_client::scheduleDiskRead() + flags.store_copying = false; + } + ++/// whether at least one byte wanted by the client is in memory ++bool ++store_client::canReadFromMemory() const ++{ ++ const auto &mem = entry->mem(); ++ const auto memReadOffset = nextHttpReadOffset(); ++ return mem.inmem_lo <= memReadOffset && memReadOffset < mem.endOffset() && ++ parsingBuffer->spaceSize(); ++} ++ ++/// The offset of the next stored HTTP response byte wanted by the client. ++int64_t ++store_client::nextHttpReadOffset() const ++{ ++ Assure(parsingBuffer); ++ const auto &mem = entry->mem(); ++ const auto hdr_sz = mem.baseReply().hdr_sz; ++ // Certain SMP cache manager transactions do not store HTTP headers in ++ // mem_hdr; they store just a kid-specific piece of the future report body. ++ // In such cases, hdr_sz ought to be zero. In all other (known) cases, ++ // mem_hdr contains HTTP response headers (positive hdr_sz if parsed) ++ // followed by HTTP response body. This code math accommodates all cases. ++ return NaturalSum(hdr_sz, copyInto.offset, parsingBuffer->contentSize()).value(); ++} ++ ++/// Copies at least some of the requested body bytes from MemObject memory, ++/// satisfying the copy() request. ++/// \pre canReadFromMemory() is true + void +-store_client::scheduleMemRead() ++store_client::readFromMemory() + { +- /* What the client wants is in memory */ +- /* Old style */ +- debugs(90, 3, "store_client::doCopy: Copying normal from memory"); +- const auto sz = entry->mem_obj->data_hdr.copy(copyInto); // may be <= 0 per copy() API +- callback(sz); +- flags.store_copying = false; ++ Assure(parsingBuffer); ++ const auto readInto = parsingBuffer->space().positionAt(nextHttpReadOffset()); ++ ++ debugs(90, 3, "copying HTTP body bytes from memory into " << readInto); ++ const auto sz = entry->mem_obj->data_hdr.copy(readInto); ++ Assure(sz > 0); // our canReadFromMemory() precondition guarantees that ++ parsingBuffer->appended(readInto.data, sz); + } + + void +@@ -453,59 +483,136 @@ store_client::fileRead() + assert(!flags.disk_io_pending); + flags.disk_io_pending = true; + ++ // mem->swap_hdr_sz is zero here during initial read(s) ++ const auto nextStoreReadOffset = NaturalSum(mem->swap_hdr_sz, nextHttpReadOffset()).value(); ++ ++ // XXX: If fileRead() is called when we do not yet know mem->swap_hdr_sz, ++ // then we must start reading from disk offset zero to learn it: we cannot ++ // compute correct HTTP response start offset on disk without it. However, ++ // late startSwapin() calls imply that the assertion below might fail. ++ Assure(mem->swap_hdr_sz > 0 || !nextStoreReadOffset); ++ ++ // TODO: Remove this assertion. Introduced in 1998 commit 3157c72, it ++ // assumes that swapped out memory is freed unconditionally, but we no ++ // longer do that because trimMemory() path checks lowestMemReaderOffset(). ++ // It is also misplaced: We are not swapping out anything here and should ++ // not care about any swapout invariants. ++ + if (mem->swap_hdr_sz != 0) + if (entry->swappingOut()) +- assert(mem->swapout.sio->offset() > copyInto.offset + (int64_t)mem->swap_hdr_sz); ++ assert(mem->swapout.sio->offset() > nextStoreReadOffset); ++ ++ // XXX: We should let individual cache_dirs limit the read size instead, but ++ // we cannot do that without more fixes and research because: ++ // * larger reads corrupt responses when cache_dir uses SharedMemory::get(); ++ // * we do not know how to find all I/O code that assumes this limit; ++ // * performance effects of larger disk reads may be negative somewhere. ++ const decltype(StoreIOBuffer::length) maxReadSize = SM_PAGE_SIZE; ++ ++ Assure(parsingBuffer); ++ // also, do not read more than we can return (via a copyInto.length buffer) ++ const auto readSize = std::min(copyInto.length, maxReadSize); ++ lastDiskRead = parsingBuffer->makeSpace(readSize).positionAt(nextStoreReadOffset); ++ debugs(90, 5, "into " << lastDiskRead); + + storeRead(swapin_sio, +- copyInto.data, +- copyInto.length, +- copyInto.offset + mem->swap_hdr_sz, ++ lastDiskRead.data, ++ lastDiskRead.length, ++ lastDiskRead.offset, + mem->swap_hdr_sz == 0 ? storeClientReadHeader + : storeClientReadBody, + this); + } + + void +-store_client::readBody(const char *, ssize_t len) ++store_client::readBody(const char * const buf, const ssize_t lastIoResult) + { + int parsed_header = 0; + +- // Don't assert disk_io_pending here.. may be called by read_header ++ Assure(flags.disk_io_pending); + flags.disk_io_pending = false; + assert(_callback.pending()); +- debugs(90, 3, "storeClientReadBody: len " << len << ""); ++ Assure(parsingBuffer); ++ debugs(90, 3, "got " << lastIoResult << " using " << *parsingBuffer); ++ if (lastIoResult < 0) ++ return fail(); + +- if (len < 0) ++ if (!lastIoResult) { ++ if (answeredOnce()) ++ return noteNews(); ++ ++ debugs(90, DBG_CRITICAL, "ERROR: Truncated HTTP headers in on-disk object"); + return fail(); ++ } ++ assert(lastDiskRead.data == buf); ++ lastDiskRead.length = lastIoResult; + +- if (copyInto.offset == 0 && len > 0 && entry->getReply()->sline.status() == Http::scNone) { +- /* Our structure ! */ +- HttpReply *rep = (HttpReply *) entry->getReply(); // bypass const ++ parsingBuffer->appended(buf, lastIoResult); + +- if (!rep->parseCharBuf(copyInto.data, headersEnd(copyInto.data, len))) { +- debugs(90, DBG_CRITICAL, "Could not parse headers from on disk object"); +- } else { +- parsed_header = 1; +- } ++ // we know swap_hdr_sz by now and were reading beyond swap metadata because ++ // readHead() would have been called otherwise (to read swap metadata) ++ const auto swap_hdr_sz = entry->mem().swap_hdr_sz; ++ Assure(swap_hdr_sz > 0); ++ Assure(!Less(lastDiskRead.offset, swap_hdr_sz)); ++ ++ // Map lastDiskRead (i.e. the disk area we just read) to an HTTP reply part. ++ // The bytes are the same, but disk and HTTP offsets differ by swap_hdr_sz. ++ const auto httpOffset = lastDiskRead.offset - swap_hdr_sz; ++ const auto httpPart = StoreIOBuffer(lastDiskRead).positionAt(httpOffset); ++ ++ maybeWriteFromDiskToMemory(httpPart); ++ handleBodyFromDisk(); ++} ++ ++/// de-serializes HTTP response (partially) read from disk storage ++void ++store_client::handleBodyFromDisk() ++{ ++ // We cannot de-serialize on-disk HTTP response without MemObject because ++ // without MemObject::swap_hdr_sz we cannot know where that response starts. ++ Assure(entry->mem_obj); ++ Assure(entry->mem_obj->swap_hdr_sz > 0); ++ ++ if (!answeredOnce()) { ++ // All on-disk responses have HTTP headers. First disk body read(s) ++ // include HTTP headers that we must parse (if needed) and skip. ++ const auto haveHttpHeaders = entry->mem_obj->baseReply().pstate == Http::Message::psParsed; ++ if (!haveHttpHeaders && !parseHttpHeadersFromDisk()) ++ return; ++ skipHttpHeadersFromDisk(); + } + + const HttpReply *rep = entry->getReply(); +- if (len > 0 && rep && entry->mem_obj->inmem_lo == 0 && entry->objectLen() <= (int64_t)Config.Store.maxInMemObjSize && Config.onoff.memory_cache_disk) { +- storeGetMemSpace(len); ++ noteNews(); ++} ++ ++/// Adds HTTP response data loaded from disk to the memory cache (if ++/// needed/possible). The given part may contain portions of HTTP response ++/// headers and/or HTTP response body. ++void ++store_client::maybeWriteFromDiskToMemory(const StoreIOBuffer &httpResponsePart) ++{ ++ // XXX: Reject [memory-]uncachable/unshareable responses instead of assuming ++ // that an HTTP response should be written to MemObject's data_hdr (and that ++ // it may purge already cached entries) just because it "fits" and was ++ // loaded from disk. For example, this response may already be marked for ++ // release. The (complex) cachability decision(s) should be made outside ++ // (and obeyed by) this low-level code. ++ if (httpResponsePart.length && entry->mem_obj->inmem_lo == 0 && entry->objectLen() <= (int64_t)Config.Store.maxInMemObjSize && Config.onoff.memory_cache_disk) { ++ storeGetMemSpace(httpResponsePart.length); ++ // XXX: This "recheck" is not needed because storeGetMemSpace() cannot ++ // purge mem_hdr bytes of a locked entry, and we do lock ours. And ++ // inmem_lo offset itself should not be relevant to appending new bytes. ++ // + // The above may start to free our object so we need to check again + if (entry->mem_obj->inmem_lo == 0) { +- /* Copy read data back into memory. +- * copyInto.offset includes headers, which is what mem cache needs +- */ +- int64_t mem_offset = entry->mem_obj->endOffset(); +- if ((copyInto.offset == mem_offset) || (parsed_header && mem_offset == rep->hdr_sz)) { +- entry->mem_obj->write(StoreIOBuffer(len, copyInto.offset, copyInto.data)); ++ // XXX: This code assumes a non-shared memory cache. ++ if (httpResponsePart.offset == entry->mem_obj->endOffset()) ++ entry->mem_obj->write(httpResponsePart); + } + } + } + +- callback(len); + } + + void +@@ -615,38 +722,21 @@ store_client::readHeader(char const *buf, ssize_t len) + if (!object_ok) + return; + ++ Assure(parsingBuffer); ++ debugs(90, 3, "got " << len << " using " << *parsingBuffer); ++ + if (len < 0) + return fail(); + ++ Assure(!parsingBuffer->contentSize()); ++ parsingBuffer->appended(buf, len); + if (!unpackHeader(buf, len)) { + fail(); + return; + } +- +- /* +- * If our last read got some data the client wants, then give +- * it to them, otherwise schedule another read. +- */ +- size_t body_sz = len - mem->swap_hdr_sz; +- +- if (copyInto.offset < static_cast(body_sz)) { +- /* +- * we have (part of) what they want +- */ +- size_t copy_sz = min(copyInto.length, body_sz); +- debugs(90, 3, "storeClientReadHeader: copying " << copy_sz << " bytes of body"); +- memmove(copyInto.data, copyInto.data + mem->swap_hdr_sz, copy_sz); +- +- readBody(copyInto.data, copy_sz); +- +- return; +- } +- +- /* +- * we don't have what the client wants, but at least we now +- * know the swap header size. +- */ +- fileRead(); ++ parsingBuffer->consume(mem->swap_hdr_sz); ++ maybeWriteFromDiskToMemory(parsingBuffer->content()); ++ handleBodyFromDisk(); + } + + int +@@ -903,6 +993,63 @@ CheckQuickAbortIsReasonable(StoreEntry * entry) + return true; + } + ++/// parses HTTP header bytes loaded from disk ++/// \returns false if fail() or scheduleDiskRead() has been called and, hence, ++/// the caller should just quit without any further action ++bool ++store_client::parseHttpHeadersFromDisk() ++{ ++ try { ++ return tryParsingHttpHeaders(); ++ } catch (...) { ++ // XXX: Our parser enforces Config.maxReplyHeaderSize limit, but our ++ // packer does not. Since packing might increase header size, we may ++ // cache a header that we cannot parse and get here. Same for MemStore. ++ debugs(90, DBG_CRITICAL, "ERROR: Cannot parse on-disk HTTP headers" << ++ Debug::Extra << "exception: " << CurrentException << ++ Debug::Extra << "raw input size: " << parsingBuffer->contentSize() << " bytes" << ++ Debug::Extra << "current buffer capacity: " << parsingBuffer->capacity() << " bytes"); ++ fail(); ++ return false; ++ } ++} ++ ++/// parseHttpHeadersFromDisk() helper ++/// \copydoc parseHttpHeaders() ++bool ++store_client::tryParsingHttpHeaders() ++{ ++ Assure(parsingBuffer); ++ Assure(!copyInto.offset); // otherwise, parsingBuffer cannot have HTTP response headers ++ auto &adjustableReply = entry->mem().adjustableBaseReply(); ++ if (adjustableReply.parseTerminatedPrefix(parsingBuffer->c_str(), parsingBuffer->contentSize())) ++ return true; ++ ++ // TODO: Optimize by checking memory as well. For simplicity sake, we ++ // continue on the disk-reading path, but readFromMemory() can give us the ++ // missing header bytes immediately if a concurrent request put those bytes ++ // into memory while we were waiting for our disk response. ++ scheduleDiskRead(); ++ return false; ++} ++ ++/// skips HTTP header bytes previously loaded from disk ++void ++store_client::skipHttpHeadersFromDisk() ++{ ++ const auto hdr_sz = entry->mem_obj->baseReply().hdr_sz; ++ Assure(hdr_sz > 0); // all on-disk responses have HTTP headers ++ if (Less(parsingBuffer->contentSize(), hdr_sz)) { ++ debugs(90, 5, "discovered " << hdr_sz << "-byte HTTP headers in memory after reading some of them from disk: " << *parsingBuffer); ++ parsingBuffer->consume(parsingBuffer->contentSize()); // skip loaded HTTP header prefix ++ } else { ++ parsingBuffer->consume(hdr_sz); // skip loaded HTTP headers ++ const auto httpBodyBytesAfterHeader = parsingBuffer->contentSize(); // may be zero ++ Assure(httpBodyBytesAfterHeader <= copyInto.length); ++ debugs(90, 5, "read HTTP body prefix: " << httpBodyBytesAfterHeader); ++ } ++} ++ + void + store_client::dumpStats(MemBuf * output, int clientNumber) const + { +diff --git a/src/tests/stub_HttpReply.cc b/src/tests/stub_HttpReply.cc +index 8ca7f9e..5cde8e6 100644 +--- a/src/tests/stub_HttpReply.cc ++++ b/src/tests/stub_HttpReply.cc +@@ -25,6 +25,7 @@ void httpBodyPackInto(const HttpBody *, Packable *) STUB + bool HttpReply::sanityCheckStartLine(const char *buf, const size_t hdr_len, Http::StatusCode *error) STUB_RETVAL(false) + int HttpReply::httpMsgParseError() STUB_RETVAL(0) + bool HttpReply::expectingBody(const HttpRequestMethod&, int64_t&) const STUB_RETVAL(false) ++size_t HttpReply::parseTerminatedPrefix(const char *, size_t) STUB_RETVAL(0) + bool HttpReply::parseFirstLine(const char *start, const char *end) STUB_RETVAL(false) + void HttpReply::hdrCacheInit() STUB + HttpReply * HttpReply::clone() const STUB_RETVAL(NULL) +diff --git a/src/urn.cc b/src/urn.cc +index 74453e1..9f5e89d 100644 +--- a/src/urn.cc ++++ b/src/urn.cc +@@ -26,8 +26,6 @@ + #include "tools.h" + #include "urn.h" + +-#define URN_REQBUF_SZ 4096 +- + class UrnState : public StoreClient + { + CBDATA_CLASS(UrnState); +@@ -45,8 +43,8 @@ public: + HttpRequest::Pointer request; + HttpRequest::Pointer urlres_r; + +- char reqbuf[URN_REQBUF_SZ] = { '\0' }; +- int reqofs = 0; ++ /// for receiving a URN resolver reply body from Store and interpreting it ++ Store::ParsingBuffer parsingBuffer; + + private: + char *urlres; +@@ -63,7 +61,7 @@ typedef struct { + } url_entry; + + static STCB urnHandleReply; +-static url_entry *urnParseReply(const char *inbuf, const HttpRequestMethod&); ++static url_entry *urnParseReply(const SBuf &, const HttpRequestMethod &); + static const char *const crlf = "\r\n"; + + CBDATA_CLASS_INIT(UrnState); +@@ -183,13 +181,8 @@ UrnState::created(StoreEntry *newEntry) + sc = storeClientListAdd(urlres_e, this); + } + +- reqofs = 0; +- StoreIOBuffer tempBuffer; +- tempBuffer.offset = reqofs; +- tempBuffer.length = URN_REQBUF_SZ; +- tempBuffer.data = reqbuf; + storeClientCopy(sc, urlres_e, +- tempBuffer, ++ parsingBuffer.makeInitialSpace(), + urnHandleReply, + this); + } +@@ -224,9 +217,6 @@ urnHandleReply(void *data, StoreIOBuffer result) + UrnState *urnState = static_cast(data); + StoreEntry *e = urnState->entry; + StoreEntry *urlres_e = urnState->urlres_e; +- char *s = NULL; +- size_t k; +- HttpReply *rep; + url_entry *urls; + url_entry *u; + url_entry *min_u; +@@ -234,10 +224,8 @@ urnHandleReply(void *data, StoreIOBuffer result) + ErrorState *err; + int i; + int urlcnt = 0; +- char *buf = urnState->reqbuf; +- StoreIOBuffer tempBuffer; + +- debugs(52, 3, "urnHandleReply: Called with size=" << result.length << "."); ++ debugs(52, 3, result << " with " << *e); + + if (EBIT_TEST(urlres_e->flags, ENTRY_ABORTED) || result.flags.error) { + delete urnState; +@@ -250,59 +238,38 @@ urnHandleReply(void *data, StoreIOBuffer result) + return; + } + +- /* Update reqofs to point to where in the buffer we'd be */ +- urnState->reqofs += result.length; +- +- /* Handle reqofs being bigger than normal */ +- if (urnState->reqofs >= URN_REQBUF_SZ) { +- delete urnState; +- return; +- } +++ urnState->parsingBuffer.appended(result.data, result.length); + + /* If we haven't received the entire object (urn), copy more */ +- if (urlres_e->store_status == STORE_PENDING) { +- Must(result.length > 0); // zero length ought to imply STORE_OK +- tempBuffer.offset = urnState->reqofs; +- tempBuffer.length = URN_REQBUF_SZ - urnState->reqofs; +- tempBuffer.data = urnState->reqbuf + urnState->reqofs; ++ if (!urnState->sc->atEof()) { ++ const auto bufferedBytes = urnState->parsingBuffer.contentSize(); ++ const auto remainingSpace = urnState->parsingBuffer.space().positionAt(bufferedBytes); ++ ++ if (!remainingSpace.length) { ++ debugs(52, 3, "ran out of buffer space after " << bufferedBytes << " bytes"); ++ // TODO: Here and in other error cases, send ERR_URN_RESOLVE to client. ++ delete urnState; ++ return; ++ } + storeClientCopy(urnState->sc, urlres_e, +- tempBuffer, ++ remainingSpace, + urnHandleReply, + urnState); + return; + } + +- /* we know its STORE_OK */ +- k = headersEnd(buf, urnState->reqofs); +- +- if (0 == k) { +- debugs(52, DBG_IMPORTANT, "urnHandleReply: didn't find end-of-headers for " << e->url() ); +- delete urnState; +- return; +- } +- +- s = buf + k; +- assert(urlres_e->getReply()); +- rep = new HttpReply; +- rep->parseCharBuf(buf, k); +- debugs(52, 3, "reply exists, code=" << rep->sline.status() << "."); +- +- if (rep->sline.status() != Http::scOkay) { ++ const auto &peerReply = urlres_e->mem().baseReply(); ++ debugs(52, 3, "got reply, code=" << peerReply.sline.status()); ++ if (peerReply.sline.status() != Http::scOkay) { + debugs(52, 3, "urnHandleReply: failed."); + err = new ErrorState(ERR_URN_RESOLVE, Http::scNotFound, urnState->request.getRaw()); + err->url = xstrdup(e->url()); + errorAppendEntry(e, err); +- delete rep; + delete urnState; + return; + } + +- delete rep; +- +- while (xisspace(*s)) +- ++s; +- +- urls = urnParseReply(s, urnState->request->method); ++ urls = urnParseReply(urnState->parsingBuffer.toSBuf(), urnState->request->method); + + if (!urls) { /* unknown URN error */ + debugs(52, 3, "urnTranslateDone: unknown URN " << e->url()); +@@ -350,7 +317,7 @@ urnHandleReply(void *data, StoreIOBuffer result) + "Generated by %s@%s\n" + "\n", + APP_FULLNAME, getMyHostname()); +- rep = new HttpReply; ++ const auto rep = new HttpReply; + rep->setHeaders(Http::scFound, NULL, "text/html", mb->contentSize(), 0, squid_curtime); + + if (min_u) { +@@ -372,9 +339,8 @@ urnHandleReply(void *data, StoreIOBuffer result) + } + + static url_entry * +-urnParseReply(const char *inbuf, const HttpRequestMethod& m) ++urnParseReply(const SBuf &inBuf, const HttpRequestMethod &m) + { +- char *buf = xstrdup(inbuf); + char *token; + url_entry *list; + url_entry *old; +@@ -383,6 +349,13 @@ urnParseReply(const char *inbuf, const HttpRequestMethod& m) + debugs(52, 3, "urnParseReply"); + list = (url_entry *)xcalloc(n + 1, sizeof(*list)); + ++ // XXX: Switch to tokenizer-based parsing. ++ const auto allocated = SBufToCstring(inBuf); ++ ++ auto buf = allocated; ++ while (xisspace(*buf)) ++ ++buf; ++ + for (token = strtok(buf, crlf); token; token = strtok(NULL, crlf)) { + debugs(52, 3, "urnParseReply: got '" << token << "'"); + +@@ -418,7 +391,7 @@ urnParseReply(const char *inbuf, const HttpRequestMethod& m) + } + + debugs(52, 3, "urnParseReply: Found " << i << " URLs"); +- xfree(buf); ++ xfree(allocated); + return list; + } + +-- +2.39.3 + diff --git a/SOURCES/0003-Bug-5309-frequent-lowestOffset-target_offset-asserti.patch b/SOURCES/0003-Bug-5309-frequent-lowestOffset-target_offset-asserti.patch new file mode 100644 index 0000000..8ba5bd3 --- /dev/null +++ b/SOURCES/0003-Bug-5309-frequent-lowestOffset-target_offset-asserti.patch @@ -0,0 +1,119 @@ +From af18cb04f07555f49daef982c8c21459bfbe388c Mon Sep 17 00:00:00 2001 +From: Alex Rousskov +Date: Thu, 23 Nov 2023 18:27:24 +0000 +Subject: [PATCH 3/7] Bug 5309: frequent "lowestOffset () <= target_offset" + assertion (#1561) + + Recent commit 122a6e3 left store_client::readOffset() unchanged but + should have adjusted it to match changed copyInto.offset semantics: + Starting with that commit, storeClientCopy() callers supply HTTP + response _body_ offset rather than HTTP response offset. +.... + This bug decreased readOffset() values (by the size of stored HTTP + response headers), effectively telling Store that we are not yet done + with some of the MemObject/mem_hdr bytes. This bug could cause slightly + higher transaction memory usage because the same response bytes are + trimmed later. This bug should not have caused any assertions. +.... + However, the old mem_hdr::freeDataUpto() code that uses readOffset() is + also broken -- the assertion in that method only "works" when + readOffset() returns values matching a memory node boundary. The smaller + values returned by buggy readOffset() triggered buggy assertions. +.... + This minimal fix removes the recent store_client::readOffset() bug + described above. We will address old mem_hdr problems separately. + +Modified-by: Alex Burmashev +Signed-off-by: Alex Burmashev +--- + src/MemObject.cc | 2 +- + src/StoreClient.h | 19 ++++++++++--------- + src/store_client.cc | 13 +++++++++++++ + 3 files changed, 24 insertions(+), 10 deletions(-) + +diff --git a/src/MemObject.cc b/src/MemObject.cc +index d7aaf5e..650d3fd 100644 +--- a/src/MemObject.cc ++++ b/src/MemObject.cc +@@ -197,7 +197,7 @@ struct LowestMemReader : public unary_function { + + void operator() (store_client const &x) { + if (x.getType() == STORE_MEM_CLIENT) +- current = std::min(current, x.readOffset()); ++ current = std::min(current, x.discardableHttpEnd()); + } + + int64_t current; +diff --git a/src/StoreClient.h b/src/StoreClient.h +index 1d90e5a..0524776 100644 +--- a/src/StoreClient.h ++++ b/src/StoreClient.h +@@ -54,15 +54,8 @@ public: + store_client(StoreEntry *); + ~store_client(); + +- /// An offset into the stored response bytes, including the HTTP response +- /// headers (if any). Note that this offset does not include Store entry +- /// metadata, because it is not a part of the stored response. +- /// \retval 0 means the client wants to read HTTP response headers. +- /// \retval +N the response byte that the client wants to read next. +- /// \retval -N should not occur. +- // TODO: Callers do not expect negative offset. Verify that the return +- // value cannot be negative and convert to unsigned in this case. +- int64_t readOffset() const { return copyInto.offset; } ++ /// the client will not use HTTP response bytes with lower offsets (if any) ++ auto discardableHttpEnd() const { return discardableHttpEnd_; } + + int getType() const; + +@@ -156,8 +149,16 @@ private: + + /// Storage and metadata associated with the current copy() request. Ought + /// to be ignored when not answering a copy() request. ++ /// * copyInto.offset is the requested HTTP response body offset; ++ /// * copyInto.data is the client-owned, client-provided result buffer; ++ /// * copyInto.length is the size of the .data result buffer; ++ /// * copyInto.flags are unused by this class. + StoreIOBuffer copyInto; + ++ // TODO: Convert to uint64_t after fixing mem_hdr::endOffset() and friends. ++ /// \copydoc discardableHttpEnd() ++ int64_t discardableHttpEnd_ = 0; ++ + /// the total number of finishCallback() calls + uint64_t answers; + +diff --git a/src/store_client.cc b/src/store_client.cc +index 1731c4c..383aac8 100644 +--- a/src/store_client.cc ++++ b/src/store_client.cc +@@ -122,6 +122,16 @@ store_client::finishCallback() + result = parsingBuffer->packBack(); + result.flags.error = object_ok ? 0 : 1; + ++ // TODO: Move object_ok handling above into this `if` statement. ++ if (object_ok) { ++ // works for zero hdr_sz cases as well; see also: nextHttpReadOffset() ++ discardableHttpEnd_ = NaturalSum(entry->mem().baseReply().hdr_sz, result.offset, result.length).value(); ++ } else { ++ // object_ok is sticky, so we will not be able to use any response bytes ++ discardableHttpEnd_ = entry->mem().endOffset(); ++ } ++ debugs(90, 7, "with " << result << "; discardableHttpEnd_=" << discardableHttpEnd_); ++ + // no HTTP headers and no body bytes (but not because there was no space) + atEof_ = !sendingHttpHeaders() && !result.length && copyInto.length; + +@@ -220,6 +230,9 @@ store_client::copy(StoreEntry * anEntry, + + parsingBuffer.emplace(copyInto); + ++ discardableHttpEnd_ = nextHttpReadOffset(); ++ debugs(90, 7, "discardableHttpEnd_=" << discardableHttpEnd_); ++ + static bool copying (false); + assert (!copying); + copying = true; +-- +2.39.3 + diff --git a/SOURCES/0004-Remove-mem_hdr-freeDataUpto-assertion-1562.patch b/SOURCES/0004-Remove-mem_hdr-freeDataUpto-assertion-1562.patch new file mode 100644 index 0000000..202823b --- /dev/null +++ b/SOURCES/0004-Remove-mem_hdr-freeDataUpto-assertion-1562.patch @@ -0,0 +1,67 @@ +From 422272d78399d5fb2fc340281611961fc7c528e7 Mon Sep 17 00:00:00 2001 +From: Alex Rousskov +Date: Thu, 23 Nov 2023 18:27:45 +0000 +Subject: [PATCH 4/7] Remove mem_hdr::freeDataUpto() assertion (#1562) + + stmem.cc:98: "lowestOffset () <= target_offset" +.... + The assertion is conceptually wrong: The given target_offset parameter + may have any value; that value does not have to correlate with mem_hdr + state in any way. It is freeDataUpto() job to preserve nodes at or above + the given offset and (arguably optionally) remove nodes below it, but + the assertion does not actually validate that freeDataUpdo() did that. +.... + The old mem_hdr::freeDataUpto() assertion incorrectly assumed that, + after zero or more unneeded memory nodes were freed, the remaining + memory area never starts after the given target_offset parameter. That + assumption fails in at least two use cases, both using target_offset + values that do not belong to any existing or future mem_hdr node: +.... + 1. target_offset is points to the left of the first node. freeDataUpto() + correctly keeps all memory nodes in such calls, but then asserts. For + example, calling freeDataUpto(0) when mem_hdr has bytes [100,199) + triggers this incorrect assertion. +.... + 2. target_offset is in the gap between two nodes. For example, calling + freeDataUpto(2000) when mem_hdr contains two nodes: [0,1000) and + [3000,3003) will trigger this assertion (as happened in Bug 5309). + Such gaps are very common for HTTP 206 responses with a Content-Range + header because such responses often specify a range that does not + start with zero and create a gap after the node(s) with HTTP headers. +.... + Bugs notwithstanding, it is unlikely that relevant calls exist today, + but they certainly could be added, especially when freeDataUpto() stops + preserving the last unused node. The current "avoid change to [some + unidentified] part of code" hoarding excuse should not last forever. +.... + Prior to commit 122a6e3, Squid did not (frequently) assert in gap cases: + Callers first give target_offset 0 (which results in freeDataUpto() + doing nothing, keeping the header node(s)) and then they give + target_offset matching the beginning of the first body node (which + results in freeDataUpto() freeing the header nodes(s) and increasing + lowerOffset() from zero to target_offset). A bug in commit 122a6e3 + lowered target_offset a bit, placing target_offset in the gap and + triggering frequent (and incorrect) assertions (Bug 5309). + +Modified-by: Alex Burmashev +Signed-off-by: Alex Burmashev +--- + src/stmem.cc | 2 -- + 1 file changed, 2 deletions(-) + +diff --git a/src/stmem.cc b/src/stmem.cc +index d117c15..b627005 100644 +--- a/src/stmem.cc ++++ b/src/stmem.cc +@@ -95,8 +95,6 @@ mem_hdr::freeDataUpto(int64_t target_offset) + break; + } + +- assert (lowestOffset () <= target_offset); +- + return lowestOffset (); + } + +-- +2.39.3 + diff --git a/SOURCES/0005-Backport-Add-Assure-as-a-replacement-for-problematic.patch b/SOURCES/0005-Backport-Add-Assure-as-a-replacement-for-problematic.patch new file mode 100644 index 0000000..1953209 --- /dev/null +++ b/SOURCES/0005-Backport-Add-Assure-as-a-replacement-for-problematic.patch @@ -0,0 +1,210 @@ +From 5df95b5923de244eaf2ddccf980d5f28d7114b1f Mon Sep 17 00:00:00 2001 +From: Alex Burmashev +Date: Thu, 7 Dec 2023 18:01:47 +0000 +Subject: [PATCH 5/7] Backport Add Assure() as a replacement for problematic + Must() + +This is a partial backport of +b9a1bbfbc531359a87647271a282edff9ccdd206 +b8ae064d94784934b3402e5db015246d1b1ca658 + +Needed for CVE CVE-2023-5824 fix + +Signed-off-by: Alex Burmashev +--- + src/HttpReply.cc | 1 + + src/acl/Asn.cc | 1 + + src/base/Assure.cc | 23 ++++++++++++++++++ + src/base/Assure.h | 51 ++++++++++++++++++++++++++++++++++++++++ + src/base/Makefile.am | 2 ++ + src/base/Makefile.in | 8 +++++-- + src/client_side_reply.cc | 1 + + 7 files changed, 85 insertions(+), 2 deletions(-) + create mode 100644 src/base/Assure.cc + create mode 100644 src/base/Assure.h + +diff --git a/src/HttpReply.cc b/src/HttpReply.cc +index af2bd4d..df5bcef 100644 +--- a/src/HttpReply.cc ++++ b/src/HttpReply.cc +@@ -10,6 +10,7 @@ + + #include "squid.h" + #include "acl/AclSizeLimit.h" ++#include "base/Assure.h" + #include "acl/FilledChecklist.h" + #include "base/EnumIterator.h" + #include "globals.h" +diff --git a/src/acl/Asn.cc b/src/acl/Asn.cc +index ad450c0..bcedc82 100644 +--- a/src/acl/Asn.cc ++++ b/src/acl/Asn.cc +@@ -17,6 +17,7 @@ + #include "acl/SourceAsn.h" + #include "acl/Strategised.h" + #include "base/CharacterSet.h" ++#include "base/Assure.h" + #include "FwdState.h" + #include "HttpReply.h" + #include "HttpRequest.h" +diff --git a/src/base/Assure.cc b/src/base/Assure.cc +new file mode 100644 +index 0000000..b09b848 +--- /dev/null ++++ b/src/base/Assure.cc +@@ -0,0 +1,23 @@ ++/* ++ * Copyright (C) 1996-2023 The Squid Software Foundation and contributors ++ * ++ * Squid software is distributed under GPLv2+ license and includes ++ * contributions from numerous individuals and organizations. ++ * Please see the COPYING and CONTRIBUTORS files for details. ++ */ ++ ++#include "squid.h" ++#include "base/Assure.h" ++#include "base/TextException.h" ++#include "sbuf/Stream.h" ++ ++[[ noreturn ]] void ++ReportAndThrow_(const int debugLevel, const char *description, const SourceLocation &location) ++{ ++ const TextException ex(description, location); ++ const auto label = debugLevel <= DBG_IMPORTANT ? "ERROR: Squid BUG: " : ""; ++ // TODO: Consider also printing the number of BUGs reported so far. It would ++ // require GC, but we could even print the number of same-location reports. ++ debugs(0, debugLevel, label << ex); ++ throw ex; ++} +diff --git a/src/base/Assure.h b/src/base/Assure.h +new file mode 100644 +index 0000000..650c204 +--- /dev/null ++++ b/src/base/Assure.h +@@ -0,0 +1,51 @@ ++/* ++ * Copyright (C) 1996-2023 The Squid Software Foundation and contributors ++ * ++ * Squid software is distributed under GPLv2+ license and includes ++ * contributions from numerous individuals and organizations. ++ * Please see the COPYING and CONTRIBUTORS files for details. ++ */ ++ ++#ifndef SQUID_SRC_BASE_ASSURE_H ++#define SQUID_SRC_BASE_ASSURE_H ++ ++#include "base/Here.h" ++ ++/// Reports the description (at the given debugging level) and throws ++/// the corresponding exception. Reduces compiled code size of Assure() and ++/// Must() callers. Do not call directly; use Assure() instead. ++/// \param description explains the condition (i.e. what MUST happen) ++[[ noreturn ]] void ReportAndThrow_(int debugLevel, const char *description, const SourceLocation &); ++ ++/// Calls ReportAndThrow() if needed. Reduces caller code duplication. ++/// Do not call directly; use Assure() instead. ++/// \param description c-string explaining the condition (i.e. what MUST happen) ++#define Assure_(debugLevel, condition, description, location) \ ++ while (!(condition)) \ ++ ReportAndThrow_((debugLevel), (description), (location)) ++ ++#if !defined(NDEBUG) ++ ++/// Like assert() but throws an exception instead of aborting the process. Use ++/// this macro to detect code logic mistakes (i.e. bugs) where aborting the ++/// current AsyncJob or a similar task is unlikely to jeopardize Squid service ++/// integrity. For example, this macro is _not_ appropriate for detecting bugs ++/// that indicate a dangerous global state corruption which may go unnoticed by ++/// other jobs after the current job or task is aborted. ++#define Assure(condition) \ ++ Assure2((condition), #condition) ++ ++/// Like Assure() but allows the caller to customize the exception message. ++/// \param description string literal describing the condition (i.e. what MUST happen) ++#define Assure2(condition, description) \ ++ Assure_(0, (condition), ("assurance failed: " description), Here()) ++ ++#else ++ ++/* do-nothing implementations for NDEBUG builds */ ++#define Assure(condition) ((void)0) ++#define Assure2(condition, description) ((void)0) ++ ++#endif /* NDEBUG */ ++ ++#endif /* SQUID_SRC_BASE_ASSURE_H */ +diff --git a/src/base/Makefile.am b/src/base/Makefile.am +index 9b0f4cf..c22dd0e 100644 +--- a/src/base/Makefile.am ++++ b/src/base/Makefile.am +@@ -19,6 +19,8 @@ libbase_la_SOURCES = \ + AsyncJob.cc \ + AsyncJob.h \ + AsyncJobCalls.h \ ++ Assure.cc \ ++ Assure.h \ + ByteCounter.h \ + CbcPointer.h \ + CbDataList.h \ +diff --git a/src/base/Makefile.in b/src/base/Makefile.in +index 90a4f5b..f43e098 100644 +--- a/src/base/Makefile.in ++++ b/src/base/Makefile.in +@@ -163,7 +163,7 @@ CONFIG_CLEAN_FILES = + CONFIG_CLEAN_VPATH_FILES = + LTLIBRARIES = $(noinst_LTLIBRARIES) + libbase_la_LIBADD = +-am_libbase_la_OBJECTS = AsyncCall.lo AsyncCallQueue.lo AsyncJob.lo \ ++am_libbase_la_OBJECTS = AsyncCall.lo AsyncCallQueue.lo AsyncJob.lo Assure.lo \ + CharacterSet.lo File.lo Here.lo RegexPattern.lo \ + RunnersRegistry.lo TextException.lo + libbase_la_OBJECTS = $(am_libbase_la_OBJECTS) +@@ -187,7 +187,7 @@ DEFAULT_INCLUDES = + depcomp = $(SHELL) $(top_srcdir)/cfgaux/depcomp + am__maybe_remake_depfiles = depfiles + am__depfiles_remade = ./$(DEPDIR)/AsyncCall.Plo \ +- ./$(DEPDIR)/AsyncCallQueue.Plo ./$(DEPDIR)/AsyncJob.Plo \ ++ ./$(DEPDIR)/AsyncCallQueue.Plo ./$(DEPDIR)/AsyncJob.Plo ./$(DEPDIR)/Assure.Plo \ + ./$(DEPDIR)/CharacterSet.Plo ./$(DEPDIR)/File.Plo \ + ./$(DEPDIR)/Here.Plo ./$(DEPDIR)/RegexPattern.Plo \ + ./$(DEPDIR)/RunnersRegistry.Plo ./$(DEPDIR)/TextException.Plo +@@ -737,6 +737,8 @@ libbase_la_SOURCES = \ + AsyncJob.cc \ + AsyncJob.h \ + AsyncJobCalls.h \ ++ Assure.cc \ ++ Assure.h \ + ByteCounter.h \ + CbcPointer.h \ + CbDataList.h \ +@@ -830,6 +832,7 @@ distclean-compile: + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/AsyncCall.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/AsyncCallQueue.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/AsyncJob.Plo@am__quote@ # am--include-marker ++@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/Assure.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/CharacterSet.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/File.Plo@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/Here.Plo@am__quote@ # am--include-marker +@@ -1224,6 +1227,7 @@ maintainer-clean: maintainer-clean-am + -rm -f ./$(DEPDIR)/AsyncCall.Plo + -rm -f ./$(DEPDIR)/AsyncCallQueue.Plo + -rm -f ./$(DEPDIR)/AsyncJob.Plo ++ -rm -f ./$(DEPDIR)/Assure.Plo + -rm -f ./$(DEPDIR)/CharacterSet.Plo + -rm -f ./$(DEPDIR)/File.Plo + -rm -f ./$(DEPDIR)/Here.Plo +diff --git a/src/client_side_reply.cc b/src/client_side_reply.cc +index 861f4b4..470f4bc 100644 +--- a/src/client_side_reply.cc ++++ b/src/client_side_reply.cc +@@ -12,6 +12,7 @@ + #include "acl/FilledChecklist.h" + #include "acl/Gadgets.h" + #include "anyp/PortCfg.h" ++#include "base/Assure.h" + #include "client_side_reply.h" + #include "errorpage.h" + #include "ETag.h" +-- +2.39.3 + diff --git a/SOURCES/0006-Backport-additional-functions-for-SquidMath.patch b/SOURCES/0006-Backport-additional-functions-for-SquidMath.patch new file mode 100644 index 0000000..54143bc --- /dev/null +++ b/SOURCES/0006-Backport-additional-functions-for-SquidMath.patch @@ -0,0 +1,200 @@ +From c24b9507e35fa43ddb40211a50fae9d58a0381bc Mon Sep 17 00:00:00 2001 +From: Alex Burmashev +Date: Mon, 27 Nov 2023 11:47:40 +0000 +Subject: [PATCH 6/7] Backport additional functions for SquidMath + +This includes some cherry-picks from +b308d7e2ad02ae6622f380d94d2303446f5831a9 and later commits. + +This is needed for CVE-2023-5824 fix + +Signed-off-by: Alex Burmashev +--- + src/SquidMath.h | 164 ++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 164 insertions(+) + +diff --git a/src/SquidMath.h b/src/SquidMath.h +index c70acd1..e5b6e58 100644 +--- a/src/SquidMath.h ++++ b/src/SquidMath.h +@@ -8,7 +8,11 @@ + + #ifndef _SQUID_SRC_SQUIDMATH_H + #define _SQUID_SRC_SQUIDMATH_H ++#include "base/forward.h" ++#include "base/TypeTraits.h" + ++#include ++#include + /* Math functions we define locally for Squid */ + namespace Math + { +@@ -21,5 +25,165 @@ double doubleAverage(const double, const double, int, const int); + + } // namespace Math + ++// If Sum() performance becomes important, consider using GCC and clang ++// built-ins like __builtin_add_overflow() instead of manual overflow checks. ++ ++/// detects a pair of unsigned types ++/// reduces code duplication in declarations further below ++template ++using AllUnsigned = typename std::conditional< ++ std::is_unsigned::value && std::is_unsigned::value, ++ std::true_type, ++ std::false_type ++ >::type; ++ ++// TODO: Replace with std::cmp_less() after migrating to C++20. ++/// whether integer a is less than integer b, with correct overflow handling ++template ++constexpr bool ++Less(const A a, const B b) { ++ // The casts below make standard C++ integer conversions explicit. They ++ // quell compiler warnings about signed/unsigned comparison. The first two ++ // lines exclude different-sign a and b, making the casts/comparison safe. ++ using AB = typename std::common_type::type; ++ return ++ (a >= 0 && b < 0) ? false : ++ (a < 0 && b >= 0) ? true : ++ /* (a >= 0) == (b >= 0) */ static_cast(a) < static_cast(b); ++} ++ ++/// ensure that T is supported by NaturalSum() and friends ++template ++constexpr void ++AssertNaturalType() ++{ ++ static_assert(std::numeric_limits::is_bounded, "std::numeric_limits::max() is meaningful"); ++ static_assert(std::numeric_limits::is_exact, "no silent loss of precision"); ++ static_assert(!std::is_enum::value, "no silent creation of non-enumerated values"); ++} ++ ++// TODO: Investigate whether this optimization can be expanded to [signed] types ++// A and B when std::numeric_limits::is_modulo is true. ++/// This IncreaseSumInternal() overload is optimized for speed. ++/// \returns a non-overflowing sum of the two unsigned arguments (or nothing) ++/// \prec both argument types are unsigned ++template ::value, int> = 0> ++std::optional ++IncreaseSumInternal(const A a, const B b) { ++ // paranoid: AllUnsigned precondition established that already ++ static_assert(std::is_unsigned::value, "AllUnsigned dispatch worked for A"); ++ static_assert(std::is_unsigned::value, "AllUnsigned dispatch worked for B"); ++ ++ AssertNaturalType(); ++ AssertNaturalType(); ++ AssertNaturalType(); ++ ++ // we should only be called by IncreaseSum(); it forces integer promotion ++ static_assert(std::is_same::value, "a will not be promoted"); ++ static_assert(std::is_same::value, "b will not be promoted"); ++ // and without integer promotions, a sum of unsigned integers is unsigned ++ static_assert(std::is_unsigned::value, "a+b is unsigned"); ++ ++ // with integer promotions ruled out, a or b can only undergo integer ++ // conversion to the higher rank type (A or B, we do not know which) ++ using AB = typename std::common_type::type; ++ static_assert(std::is_same::value || std::is_same::value, "no unexpected conversions"); ++ static_assert(std::is_same::value, "lossless assignment"); ++ const AB sum = a + b; ++ ++ static_assert(std::numeric_limits::is_modulo, "we can detect overflows"); ++ // 1. modulo math: overflowed sum is smaller than any of its operands ++ // 2. the sum may overflow S (i.e. the return base type) ++ // We do not need Less() here because we compare promoted unsigned types. ++ return (sum >= a && sum <= std::numeric_limits::max()) ? ++ std::optional(sum) : std::optional(); ++} ++ ++/// This IncreaseSumInternal() overload supports a larger variety of types. ++/// \returns a non-overflowing sum of the two arguments (or nothing) ++/// \returns nothing if at least one of the arguments is negative ++/// \prec at least one of the argument types is signed ++template ::value, int> = 0> ++std::optional constexpr ++IncreaseSumInternal(const A a, const B b) { ++ AssertNaturalType(); ++ AssertNaturalType(); ++ AssertNaturalType(); ++ ++ // we should only be called by IncreaseSum() that does integer promotion ++ static_assert(std::is_same::value, "a will not be promoted"); ++ static_assert(std::is_same::value, "b will not be promoted"); ++ ++ return ++ // We could support a non-under/overflowing sum of negative numbers, but ++ // our callers use negative values specially (e.g., for do-not-use or ++ // do-not-limit settings) and are not supposed to do math with them. ++ (a < 0 || b < 0) ? std::optional() : ++ // To avoid undefined behavior of signed overflow, we must not compute ++ // the raw a+b sum if it may overflow. When A is not B, a or b undergoes ++ // (safe for non-negatives) integer conversion in these expressions, so ++ // we do not know the resulting a+b type AB and its maximum. We must ++ // also detect subsequent casting-to-S overflows. ++ // Overflow condition: (a + b > maxAB) or (a + b > maxS). ++ // A is an integer promotion of S, so maxS <= maxA <= maxAB. ++ // Since maxS <= maxAB, it is sufficient to just check: a + b > maxS, ++ // which is the same as the overflow-safe condition here: maxS - a < b. ++ // Finally, (maxS - a) cannot overflow because a is not negative and ++ // cannot underflow because a is a promotion of s: 0 <= a <= maxS. ++ Less(std::numeric_limits::max() - a, b) ? std::optional() : ++ std::optional(a + b); ++} ++ ++/// argument pack expansion termination for IncreaseSum() ++template ++std::optional ++IncreaseSum(const S s, const T t) ++{ ++ // Force (always safe) integer promotions now, to give std::enable_if_t<> ++ // promoted types instead of entering IncreaseSumInternal(s,t) ++ // but getting a _signed_ promoted value of s or t in s + t. ++ return IncreaseSumInternal(+s, +t); ++} ++ ++/// \returns a non-overflowing sum of the arguments (or nothing) ++template ++std::optional ++IncreaseSum(const S sum, const T t, const Args... args) { ++ if (const auto head = IncreaseSum(sum, t)) { ++ return IncreaseSum(head.value(), args...); ++ } else { ++ // std::optional() triggers bogus -Wmaybe-uninitialized warnings in GCC v10.3 ++ return std::nullopt; ++ } ++} ++ ++/// \returns an exact, non-overflowing sum of the arguments (or nothing) ++template ++std::optional ++NaturalSum(const Args... args) { ++ return IncreaseSum(0, args...); ++} ++ ++/// Safely resets the given variable to NaturalSum() of the given arguments. ++/// If the sum overflows, resets to variable's maximum possible value. ++/// \returns the new variable value (like an assignment operator would) ++template ++S ++SetToNaturalSumOrMax(S &var, const Args... args) ++{ ++ var = NaturalSum(args...).value_or(std::numeric_limits::max()); ++ return var; ++} ++ ++/// converts a given non-negative integer into an integer of a given type ++/// without loss of information or undefined behavior ++template ++Result ++NaturalCast(const Source s) ++{ ++ return NaturalSum(s).value(); ++} ++ ++ + #endif /* _SQUID_SRC_SQUIDMATH_H */ + +-- +2.39.3 + diff --git a/SOURCES/0007-Adapt-to-older-gcc-cleanup.patch b/SOURCES/0007-Adapt-to-older-gcc-cleanup.patch new file mode 100644 index 0000000..126a8fd --- /dev/null +++ b/SOURCES/0007-Adapt-to-older-gcc-cleanup.patch @@ -0,0 +1,763 @@ +From 37de4ce82f7f8906606d0625774d856ffd3a9453 Mon Sep 17 00:00:00 2001 +From: Alex Burmashev +Date: Thu, 7 Dec 2023 20:51:39 +0000 +Subject: [PATCH] Adapt to older gcc, cleanup + +Fix code that is not applicable to older codebase of squidv4. +On top do some work to adapt code to older gcc. +most of that is std::optional to std::pair conversion + +Signed-off-by: Alex Burmashev +--- + src/HttpReply.cc | 4 +- + src/MemObject.h | 3 ++ + src/MemStore.cc | 6 +-- + src/SquidMath.h | 27 ++++++------ + src/Store.h | 3 ++ + src/StoreClient.h | 2 +- + src/acl/Asn.cc | 14 +------ + src/base/Assure.cc | 8 ++++ + src/client_side_reply.cc | 64 ++++++++++++----------------- + src/peer_digest.cc | 1 + + src/store/ParsingBuffer.cc | 47 ++++++++++----------- + src/store/ParsingBuffer.h | 2 +- + src/store_client.cc | 84 +++++++++++++++++--------------------- + src/urn.cc | 2 +- + 14 files changed, 123 insertions(+), 144 deletions(-) + +diff --git a/src/HttpReply.cc b/src/HttpReply.cc +index df5bcef..21c62c2 100644 +--- a/src/HttpReply.cc ++++ b/src/HttpReply.cc +@@ -534,13 +534,13 @@ HttpReply::parseTerminatedPrefix(const char * const terminatedBuf, const size_t + const bool eof = false; // TODO: Remove after removing atEnd from HttpHeader::parse() + if (parse(terminatedBuf, bufSize, eof, &error)) { + debugs(58, 7, "success after accumulating " << bufSize << " bytes and parsing " << hdr_sz); +- Assure(pstate == Http::Message::psParsed); ++ Assure(pstate == psParsed); + Assure(hdr_sz > 0); + Assure(!Less(bufSize, hdr_sz)); // cannot parse more bytes than we have + return hdr_sz; // success + } + +- Assure(pstate != Http::Message::psParsed); ++ Assure(pstate != psParsed); + hdr_sz = 0; + + if (error) { +diff --git a/src/MemObject.h b/src/MemObject.h +index ba6646f..5a7590a 100644 +--- a/src/MemObject.h ++++ b/src/MemObject.h +@@ -56,6 +56,9 @@ public: + + void write(const StoreIOBuffer &buf); + void unlinkRequest(); ++ ++ HttpReply &baseReply() const { return *_reply; } ++ + HttpReply const *getReply() const; + void replaceHttpReply(HttpReply *newrep); + void stat (MemBuf * mb) const; +diff --git a/src/MemStore.cc b/src/MemStore.cc +index fe7af2f..6762c4f 100644 +--- a/src/MemStore.cc ++++ b/src/MemStore.cc +@@ -511,8 +511,8 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + " from " << extra.page << '+' << prefixSize); + + // parse headers if needed; they might span multiple slices! +- auto &reply = e.mem().adjustableBaseReply(); +- if (reply.pstate != Http::Message::psParsed) { ++ auto &reply = e.mem().baseReply(); ++ if (reply.pstate != psParsed) { + httpHeaderParsingBuffer.append(sliceBuf.data, sliceBuf.length); + if (reply.parseTerminatedPrefix(httpHeaderParsingBuffer.c_str(), httpHeaderParsingBuffer.length())) + httpHeaderParsingBuffer = SBuf(); // we do not need these bytes anymore +@@ -542,7 +542,7 @@ MemStore::copyFromShm(StoreEntry &e, const sfileno index, const Ipc::StoreMapAnc + debugs(20, 5, "mem-loaded all " << e.mem_obj->endOffset() << '/' << + anchor.basics.swap_file_sz << " bytes of " << e); + +- if (e.mem().adjustableBaseReply().pstate != Http::Message::psParsed) ++ if (e.mem().baseReply().pstate != psParsed) + throw TextException(ToSBuf("truncated mem-cached headers; accumulated: ", httpHeaderParsingBuffer.length()), Here()); + + // from StoreEntry::complete() +diff --git a/src/SquidMath.h b/src/SquidMath.h +index e5b6e58..538833b 100644 +--- a/src/SquidMath.h ++++ b/src/SquidMath.h +@@ -8,8 +8,6 @@ + + #ifndef _SQUID_SRC_SQUIDMATH_H + #define _SQUID_SRC_SQUIDMATH_H +-#include "base/forward.h" +-#include "base/TypeTraits.h" + + #include + #include +@@ -68,7 +66,7 @@ AssertNaturalType() + /// \returns a non-overflowing sum of the two unsigned arguments (or nothing) + /// \prec both argument types are unsigned + template ::value, int> = 0> +-std::optional ++std::pair + IncreaseSumInternal(const A a, const B b) { + // paranoid: AllUnsigned precondition established that already + static_assert(std::is_unsigned::value, "AllUnsigned dispatch worked for A"); +@@ -96,7 +94,7 @@ IncreaseSumInternal(const A a, const B b) { + // 2. the sum may overflow S (i.e. the return base type) + // We do not need Less() here because we compare promoted unsigned types. + return (sum >= a && sum <= std::numeric_limits::max()) ? +- std::optional(sum) : std::optional(); ++ std::make_pair(sum, true) : std::make_pair(S(), false); + } + + /// This IncreaseSumInternal() overload supports a larger variety of types. +@@ -104,7 +102,7 @@ IncreaseSumInternal(const A a, const B b) { + /// \returns nothing if at least one of the arguments is negative + /// \prec at least one of the argument types is signed + template ::value, int> = 0> +-std::optional constexpr ++std::pair + IncreaseSumInternal(const A a, const B b) { + AssertNaturalType(); + AssertNaturalType(); +@@ -118,7 +116,7 @@ IncreaseSumInternal(const A a, const B b) { + // We could support a non-under/overflowing sum of negative numbers, but + // our callers use negative values specially (e.g., for do-not-use or + // do-not-limit settings) and are not supposed to do math with them. +- (a < 0 || b < 0) ? std::optional() : ++ (a < 0 || b < 0) ? std::make_pair(S(), false) : + // To avoid undefined behavior of signed overflow, we must not compute + // the raw a+b sum if it may overflow. When A is not B, a or b undergoes + // (safe for non-negatives) integer conversion in these expressions, so +@@ -130,13 +128,13 @@ IncreaseSumInternal(const A a, const B b) { + // which is the same as the overflow-safe condition here: maxS - a < b. + // Finally, (maxS - a) cannot overflow because a is not negative and + // cannot underflow because a is a promotion of s: 0 <= a <= maxS. +- Less(std::numeric_limits::max() - a, b) ? std::optional() : +- std::optional(a + b); ++ Less(std::numeric_limits::max() - a, b) ? std::make_pair(S(), false) : ++ std::make_pair(S(a + b), true); + } + + /// argument pack expansion termination for IncreaseSum() + template +-std::optional ++std::pair + IncreaseSum(const S s, const T t) + { + // Force (always safe) integer promotions now, to give std::enable_if_t<> +@@ -147,19 +145,20 @@ IncreaseSum(const S s, const T t) + + /// \returns a non-overflowing sum of the arguments (or nothing) + template +-std::optional ++std::pair + IncreaseSum(const S sum, const T t, const Args... args) { +- if (const auto head = IncreaseSum(sum, t)) { +- return IncreaseSum(head.value(), args...); ++ const auto head = IncreaseSum(sum, t); ++ if (head.second) { ++ return IncreaseSum(head.first, args...); + } else { + // std::optional() triggers bogus -Wmaybe-uninitialized warnings in GCC v10.3 +- return std::nullopt; ++ return std::make_pair(S(), false); + } + } + + /// \returns an exact, non-overflowing sum of the arguments (or nothing) + template +-std::optional ++std::pair + NaturalSum(const Args... args) { + return IncreaseSum(0, args...); + } +diff --git a/src/Store.h b/src/Store.h +index 3eb6b84..2475fe0 100644 +--- a/src/Store.h ++++ b/src/Store.h +@@ -49,6 +49,9 @@ public: + StoreEntry(); + virtual ~StoreEntry(); + ++ MemObject &mem() { assert(mem_obj); return *mem_obj; } ++ const MemObject &mem() const { assert(mem_obj); return *mem_obj; } ++ + virtual HttpReply const *getReply() const; + virtual void write (StoreIOBuffer); + +diff --git a/src/StoreClient.h b/src/StoreClient.h +index 0524776..ba5e669 100644 +--- a/src/StoreClient.h ++++ b/src/StoreClient.h +@@ -166,7 +166,7 @@ private: + /// request. Buffer contents depends on the source and parsing stage; it may + /// hold (parts of) swap metadata, HTTP response headers, and/or HTTP + /// response body bytes. +- std::optional parsingBuffer; ++ std::pair parsingBuffer = std::make_pair(Store::ParsingBuffer(), false); + + StoreIOBuffer lastDiskRead; ///< buffer used for the last storeRead() call + +diff --git a/src/acl/Asn.cc b/src/acl/Asn.cc +index bcedc82..67e453f 100644 +--- a/src/acl/Asn.cc ++++ b/src/acl/Asn.cc +@@ -73,7 +73,7 @@ class ASState + CBDATA_CLASS(ASState); + + public: +- ASState(); ++ ASState() = default; + ~ASState(); + + StoreEntry *entry; +@@ -87,18 +87,6 @@ public: + + CBDATA_CLASS_INIT(ASState); + +-ASState::ASState() : +- entry(NULL), +- sc(NULL), +- request(NULL), +- as_number(0), +- offset(0), +- reqofs(0), +- dataRead(false) +-{ +- memset(reqbuf, 0, AS_REQBUF_SZ); +-} +- + ASState::~ASState() + { + debugs(53, 3, entry->url()); +diff --git a/src/base/Assure.cc b/src/base/Assure.cc +index b09b848..b4cf3e5 100644 +--- a/src/base/Assure.cc ++++ b/src/base/Assure.cc +@@ -11,6 +11,14 @@ + #include "base/TextException.h" + #include "sbuf/Stream.h" + ++std::ostream & ++operator <<(std::ostream &os, const TextException &ex) ++{ ++ ex.print(os); ++ return os; ++} ++ ++ + [[ noreturn ]] void + ReportAndThrow_(const int debugLevel, const char *description, const SourceLocation &location) + { +diff --git a/src/client_side_reply.cc b/src/client_side_reply.cc +index 470f4bc..64fd489 100644 +--- a/src/client_side_reply.cc ++++ b/src/client_side_reply.cc +@@ -1142,8 +1142,8 @@ clientReplyContext::storeNotOKTransferDone() const + MemObject *mem = http->storeEntry()->mem_obj; + assert(mem != NULL); + assert(http->request != NULL); +- +- if (mem->baseReply().pstate != Http::Message::psParsed) ++ const auto expectedBodySize = mem->baseReply().content_length; ++ if (mem->baseReply().pstate != psParsed) + return 0; + + /* +@@ -1808,32 +1808,6 @@ clientReplyContext::SendMoreData(void *data, StoreIOBuffer result) + context->sendMoreData (result); + } + +-/// Whether the given body area describes the start of our Client Stream buffer. +-/// An empty area does. +-bool +-clientReplyContext::matchesStreamBodyBuffer(const StoreIOBuffer &their) const +-{ +- // the answer is undefined for errors; they are not really "body buffers" +- Assure(!their.flags.error); +- +- if (!their.length) +- return true; // an empty body area always matches our body area +- +- if (their.data != next()->readBuffer.data) { +- debugs(88, 7, "no: " << their << " vs. " << next()->readBuffer); +- return false; +- } +- +- return true; +-} +- +-void +-clientReplyContext::noteStreamBufferredBytes(const StoreIOBuffer &result) +-{ +- Assure(matchesStreamBodyBuffer(result)); +- lastStreamBufferedBytes = result; // may be unchanged and/or zero-length +-} +- + void + clientReplyContext::makeThisHead() + { +@@ -2180,21 +2154,33 @@ clientReplyContext::sendMoreData (StoreIOBuffer result) + sc->setDelayId(DelayId::DelayClient(http,reply)); + #endif + +- /* handle headers */ ++ holdingBuffer = result; ++ processReplyAccess(); ++ return; ++} + +- if (Config.onoff.log_mime_hdrs) { +- size_t k; ++/// Whether the given body area describes the start of our Client Stream buffer. ++/// An empty area does. ++bool ++clientReplyContext::matchesStreamBodyBuffer(const StoreIOBuffer &their) const ++{ ++ // the answer is undefined for errors; they are not really "body buffers" ++ Assure(!their.flags.error); ++ if (!their.length) ++ return true; // an empty body area always matches our body area ++ if (their.data != next()->readBuffer.data) { ++ debugs(88, 7, "no: " << their << " vs. " << next()->readBuffer); ++ return false; + +- if ((k = headersEnd(buf, reqofs))) { +- safe_free(http->al->headers.reply); +- http->al->headers.reply = (char *)xcalloc(k + 1, 1); +- xstrncpy(http->al->headers.reply, buf, k); +- } + } ++ return true; ++} + +- holdingBuffer = result; +- processReplyAccess(); +- return; ++void ++clientReplyContext::noteStreamBufferredBytes(const StoreIOBuffer &result) ++{ ++ Assure(matchesStreamBodyBuffer(result)); ++ lastStreamBufferedBytes = result; // may be unchanged and/or zero-length + } + + /* Using this breaks the client layering just a little! +diff --git a/src/peer_digest.cc b/src/peer_digest.cc +index abfea4a..89ea73e 100644 +--- a/src/peer_digest.cc ++++ b/src/peer_digest.cc +@@ -588,6 +588,7 @@ peerDigestFetchReply(void *data, char *buf, ssize_t size) + + return 0; // we consumed/used no buffered bytes + } ++} + + int + peerDigestSwapInCBlock(void *data, char *buf, ssize_t size) +diff --git a/src/store/ParsingBuffer.cc b/src/store/ParsingBuffer.cc +index e948fe2..affbe9e 100644 +--- a/src/store/ParsingBuffer.cc ++++ b/src/store/ParsingBuffer.cc +@@ -28,19 +28,19 @@ Store::ParsingBuffer::ParsingBuffer(StoreIOBuffer &initialSpace): + const char * + Store::ParsingBuffer::memory() const + { +- return extraMemory_ ? extraMemory_->rawContent() : readerSuppliedMemory_.data; ++ return extraMemory_.second ? extraMemory_.first.rawContent() : readerSuppliedMemory_.data; + } + + size_t + Store::ParsingBuffer::capacity() const + { +- return extraMemory_ ? (extraMemory_->length() + extraMemory_->spaceSize()) : readerSuppliedMemory_.length; ++ return extraMemory_.second ? (extraMemory_.first.length() + extraMemory_.first.spaceSize()) : readerSuppliedMemory_.length; + } + + size_t + Store::ParsingBuffer::contentSize() const + { +- return extraMemory_ ? extraMemory_->length() : readerSuppliedMemoryContentSize_; ++ return extraMemory_.second ? extraMemory_.first.length() : readerSuppliedMemoryContentSize_; + } + + void +@@ -56,10 +56,10 @@ Store::ParsingBuffer::appended(const char * const newBytes, const size_t newByte + assert(memory() + contentSize() == newBytes); // the new bytes start in our space + // and now we know that newBytes is not nil either + +- if (extraMemory_) +- extraMemory_->rawAppendFinish(newBytes, newByteCount); ++ if (extraMemory_.second) ++ extraMemory_.first.rawAppendFinish(newBytes, newByteCount); + else +- readerSuppliedMemoryContentSize_ = *IncreaseSum(readerSuppliedMemoryContentSize_, newByteCount); ++ readerSuppliedMemoryContentSize_ = IncreaseSum(readerSuppliedMemoryContentSize_, newByteCount).first; + + assert(contentSize() <= capacity()); // paranoid + } +@@ -68,8 +68,8 @@ void + Store::ParsingBuffer::consume(const size_t parsedBytes) + { + Assure(contentSize() >= parsedBytes); // more conservative than extraMemory_->consume() +- if (extraMemory_) { +- extraMemory_->consume(parsedBytes); ++ if (extraMemory_.second) { ++ extraMemory_.first.consume(parsedBytes); + } else { + readerSuppliedMemoryContentSize_ -= parsedBytes; + if (parsedBytes && readerSuppliedMemoryContentSize_) +@@ -81,8 +81,8 @@ StoreIOBuffer + Store::ParsingBuffer::space() + { + const auto size = spaceSize(); +- const auto start = extraMemory_ ? +- extraMemory_->rawAppendStart(size) : ++ const auto start = extraMemory_.second ? ++ extraMemory_.first.rawAppendStart(size) : + (readerSuppliedMemory_.data + readerSuppliedMemoryContentSize_); + return StoreIOBuffer(spaceSize(), 0, start); + } +@@ -110,22 +110,23 @@ void + Store::ParsingBuffer::growSpace(const size_t minimumSpaceSize) + { + const auto capacityIncreaseAttempt = IncreaseSum(contentSize(), minimumSpaceSize); +- if (!capacityIncreaseAttempt) ++ if (!capacityIncreaseAttempt.second) + throw TextException(ToSBuf("no support for a single memory block of ", contentSize(), '+', minimumSpaceSize, " bytes"), Here()); +- const auto newCapacity = *capacityIncreaseAttempt; ++ const auto newCapacity = capacityIncreaseAttempt.first; + + if (newCapacity <= capacity()) + return; // already have enough space; no reallocation is needed + + debugs(90, 7, "growing to provide " << minimumSpaceSize << " in " << *this); + +- if (extraMemory_) { +- extraMemory_->reserveCapacity(newCapacity); ++ if (extraMemory_.second) { ++ extraMemory_.first.reserveCapacity(newCapacity); + } else { + SBuf newStorage; + newStorage.reserveCapacity(newCapacity); + newStorage.append(readerSuppliedMemory_.data, readerSuppliedMemoryContentSize_); +- extraMemory_ = std::move(newStorage); ++ extraMemory_.first = std::move(newStorage); ++ extraMemory_.second = true; + } + Assure(spaceSize() >= minimumSpaceSize); + } +@@ -133,14 +134,14 @@ Store::ParsingBuffer::growSpace(const size_t minimumSpaceSize) + SBuf + Store::ParsingBuffer::toSBuf() const + { +- return extraMemory_ ? *extraMemory_ : SBuf(content().data, content().length); ++ return extraMemory_.second ? extraMemory_.first : SBuf(content().data, content().length); + } + + size_t + Store::ParsingBuffer::spaceSize() const + { +- if (extraMemory_) +- return extraMemory_->spaceSize(); ++ if (extraMemory_.second) ++ return extraMemory_.first.spaceSize(); + + assert(readerSuppliedMemoryContentSize_ <= readerSuppliedMemory_.length); + return readerSuppliedMemory_.length - readerSuppliedMemoryContentSize_; +@@ -169,12 +170,12 @@ Store::ParsingBuffer::packBack() + result.length = bytesToPack; + Assure(result.data); + +- if (!extraMemory_) { ++ if (!extraMemory_.second) { + // no accumulated bytes copying because they are in readerSuppliedMemory_ + debugs(90, 7, "quickly exporting " << result.length << " bytes via " << readerSuppliedMemory_); + } else { +- debugs(90, 7, "slowly exporting " << result.length << " bytes from " << extraMemory_->id << " back into " << readerSuppliedMemory_); +- memmove(result.data, extraMemory_->rawContent(), result.length); ++ debugs(90, 7, "slowly exporting " << result.length << " bytes from " << extraMemory_.first.id << " back into " << readerSuppliedMemory_); ++ memmove(result.data, extraMemory_.first.rawContent(), result.length); + } + + return result; +@@ -185,9 +186,9 @@ Store::ParsingBuffer::print(std::ostream &os) const + { + os << "size=" << contentSize(); + +- if (extraMemory_) { ++ if (extraMemory_.second) { + os << " capacity=" << capacity(); +- os << " extra=" << extraMemory_->id; ++ os << " extra=" << extraMemory_.first.id; + } + + // report readerSuppliedMemory_ (if any) even if we are no longer using it +diff --git a/src/store/ParsingBuffer.h b/src/store/ParsingBuffer.h +index b8aa957..b473ac6 100644 +--- a/src/store/ParsingBuffer.h ++++ b/src/store/ParsingBuffer.h +@@ -112,7 +112,7 @@ private: + + /// our internal buffer that takes over readerSuppliedMemory_ when the + /// latter becomes full and more memory is needed +- std::optional extraMemory_; ++ std::pair extraMemory_ = std::make_pair(SBuf(), false); + }; + + inline std::ostream & +diff --git a/src/store_client.cc b/src/store_client.cc +index 383aac8..0236274 100644 +--- a/src/store_client.cc ++++ b/src/store_client.cc +@@ -10,6 +10,7 @@ + + #include "squid.h" + #include "base/AsyncCbdataCalls.h" ++#include "base/Assure.h" + #include "event.h" + #include "globals.h" + #include "HttpReply.h" +@@ -118,24 +119,14 @@ store_client::finishCallback() + // pointers. Some other legacy code expects "correct" result.offset even + // when there is no body to return. Accommodate all those expectations. + auto result = StoreIOBuffer(0, copyInto.offset, nullptr); +- if (object_ok && parsingBuffer && parsingBuffer->contentSize()) +- result = parsingBuffer->packBack(); ++ if (object_ok && parsingBuffer.second && parsingBuffer.first.contentSize()) ++ result = parsingBuffer.first.packBack(); + result.flags.error = object_ok ? 0 : 1; + +- // TODO: Move object_ok handling above into this `if` statement. +- if (object_ok) { +- // works for zero hdr_sz cases as well; see also: nextHttpReadOffset() +- discardableHttpEnd_ = NaturalSum(entry->mem().baseReply().hdr_sz, result.offset, result.length).value(); +- } else { +- // object_ok is sticky, so we will not be able to use any response bytes +- discardableHttpEnd_ = entry->mem().endOffset(); +- } +- debugs(90, 7, "with " << result << "; discardableHttpEnd_=" << discardableHttpEnd_); +- + // no HTTP headers and no body bytes (but not because there was no space) + atEof_ = !sendingHttpHeaders() && !result.length && copyInto.length; + +- parsingBuffer.reset(); ++ parsingBuffer.second = false; + ++answers; + + STCB *temphandler = _callback.callback_handler; +@@ -228,7 +219,9 @@ store_client::copy(StoreEntry * anEntry, + // when we already can respond with HTTP headers. + Assure(!copyInto.offset || answeredOnce()); + +- parsingBuffer.emplace(copyInto); ++ parsingBuffer.first = Store::ParsingBuffer(copyInto); ++ parsingBuffer.second = true; ++ + + discardableHttpEnd_ = nextHttpReadOffset(); + debugs(90, 7, "discardableHttpEnd_=" << discardableHttpEnd_); +@@ -454,14 +447,14 @@ store_client::canReadFromMemory() const + const auto &mem = entry->mem(); + const auto memReadOffset = nextHttpReadOffset(); + return mem.inmem_lo <= memReadOffset && memReadOffset < mem.endOffset() && +- parsingBuffer->spaceSize(); ++ parsingBuffer.first.spaceSize(); + } + + /// The offset of the next stored HTTP response byte wanted by the client. + int64_t + store_client::nextHttpReadOffset() const + { +- Assure(parsingBuffer); ++ Assure(parsingBuffer.second); + const auto &mem = entry->mem(); + const auto hdr_sz = mem.baseReply().hdr_sz; + // Certain SMP cache manager transactions do not store HTTP headers in +@@ -469,7 +462,7 @@ store_client::nextHttpReadOffset() const + // In such cases, hdr_sz ought to be zero. In all other (known) cases, + // mem_hdr contains HTTP response headers (positive hdr_sz if parsed) + // followed by HTTP response body. This code math accommodates all cases. +- return NaturalSum(hdr_sz, copyInto.offset, parsingBuffer->contentSize()).value(); ++ return NaturalSum(hdr_sz, copyInto.offset, parsingBuffer.first.contentSize()).first; + } + + /// Copies at least some of the requested body bytes from MemObject memory, +@@ -478,13 +471,13 @@ store_client::nextHttpReadOffset() const + void + store_client::readFromMemory() + { +- Assure(parsingBuffer); +- const auto readInto = parsingBuffer->space().positionAt(nextHttpReadOffset()); ++ Assure(parsingBuffer.second); ++ const auto readInto = parsingBuffer.first.space().positionAt(nextHttpReadOffset()); + + debugs(90, 3, "copying HTTP body bytes from memory into " << readInto); + const auto sz = entry->mem_obj->data_hdr.copy(readInto); + Assure(sz > 0); // our canReadFromMemory() precondition guarantees that +- parsingBuffer->appended(readInto.data, sz); ++ parsingBuffer.first.appended(readInto.data, sz); + } + + void +@@ -497,7 +490,7 @@ store_client::fileRead() + flags.disk_io_pending = true; + + // mem->swap_hdr_sz is zero here during initial read(s) +- const auto nextStoreReadOffset = NaturalSum(mem->swap_hdr_sz, nextHttpReadOffset()).value(); ++ const auto nextStoreReadOffset = NaturalSum(mem->swap_hdr_sz, nextHttpReadOffset()).first; + + // XXX: If fileRead() is called when we do not yet know mem->swap_hdr_sz, + // then we must start reading from disk offset zero to learn it: we cannot +@@ -522,10 +515,10 @@ store_client::fileRead() + // * performance effects of larger disk reads may be negative somewhere. + const decltype(StoreIOBuffer::length) maxReadSize = SM_PAGE_SIZE; + +- Assure(parsingBuffer); ++ Assure(parsingBuffer.second); + // also, do not read more than we can return (via a copyInto.length buffer) + const auto readSize = std::min(copyInto.length, maxReadSize); +- lastDiskRead = parsingBuffer->makeSpace(readSize).positionAt(nextStoreReadOffset); ++ lastDiskRead = parsingBuffer.first.makeSpace(readSize).positionAt(nextStoreReadOffset); + debugs(90, 5, "into " << lastDiskRead); + + storeRead(swapin_sio, +@@ -540,13 +533,12 @@ store_client::fileRead() + void + store_client::readBody(const char * const buf, const ssize_t lastIoResult) + { +- int parsed_header = 0; + + Assure(flags.disk_io_pending); + flags.disk_io_pending = false; + assert(_callback.pending()); +- Assure(parsingBuffer); +- debugs(90, 3, "got " << lastIoResult << " using " << *parsingBuffer); ++ Assure(parsingBuffer.second); ++ debugs(90, 3, "got " << lastIoResult << " using " << parsingBuffer.first); + if (lastIoResult < 0) + return fail(); + +@@ -560,7 +552,7 @@ store_client::readBody(const char * const buf, const ssize_t lastIoResult) + assert(lastDiskRead.data == buf); + lastDiskRead.length = lastIoResult; + +- parsingBuffer->appended(buf, lastIoResult); ++ parsingBuffer.first.appended(buf, lastIoResult); + + // we know swap_hdr_sz by now and were reading beyond swap metadata because + // readHead() would have been called otherwise (to read swap metadata) +@@ -589,13 +581,12 @@ store_client::handleBodyFromDisk() + if (!answeredOnce()) { + // All on-disk responses have HTTP headers. First disk body read(s) + // include HTTP headers that we must parse (if needed) and skip. +- const auto haveHttpHeaders = entry->mem_obj->baseReply().pstate == Http::Message::psParsed; ++ const auto haveHttpHeaders = entry->mem_obj->baseReply().pstate == psParsed; + if (!haveHttpHeaders && !parseHttpHeadersFromDisk()) + return; + skipHttpHeadersFromDisk(); + } + +- const HttpReply *rep = entry->getReply(); + noteNews(); + } + +@@ -626,8 +617,6 @@ store_client::maybeWriteFromDiskToMemory(const StoreIOBuffer &httpResponsePart) + } + } + +-} +- + void + store_client::fail() + { +@@ -735,20 +724,20 @@ store_client::readHeader(char const *buf, ssize_t len) + if (!object_ok) + return; + +- Assure(parsingBuffer); +- debugs(90, 3, "got " << len << " using " << *parsingBuffer); ++ Assure(parsingBuffer.second); ++ debugs(90, 3, "got " << len << " using " << parsingBuffer.first); + + if (len < 0) + return fail(); + +- Assure(!parsingBuffer->contentSize()); +- parsingBuffer->appended(buf, len); ++ Assure(!parsingBuffer.first.contentSize()); ++ parsingBuffer.first.appended(buf, len); + if (!unpackHeader(buf, len)) { + fail(); + return; + } +- parsingBuffer->consume(mem->swap_hdr_sz); +- maybeWriteFromDiskToMemory(parsingBuffer->content()); ++ parsingBuffer.first.consume(mem->swap_hdr_sz); ++ maybeWriteFromDiskToMemory(parsingBuffer.first.content()); + handleBodyFromDisk(); + } + +@@ -1020,8 +1009,9 @@ store_client::parseHttpHeadersFromDisk() + // cache a header that we cannot parse and get here. Same for MemStore. + debugs(90, DBG_CRITICAL, "ERROR: Cannot parse on-disk HTTP headers" << + Debug::Extra << "exception: " << CurrentException << +- Debug::Extra << "raw input size: " << parsingBuffer->contentSize() << " bytes" << +- Debug::Extra << "current buffer capacity: " << parsingBuffer->capacity() << " bytes"); ++ Debug::Extra << "raw input size: " << parsingBuffer.first.contentSize() << " bytes" << ++ Debug::Extra << "current buffer capacity: " << parsingBuffer.first.capacity() << " bytes"); ++ + fail(); + return false; + } +@@ -1032,10 +1022,10 @@ store_client::parseHttpHeadersFromDisk() + bool + store_client::tryParsingHttpHeaders() + { +- Assure(parsingBuffer); ++ Assure(parsingBuffer.second); + Assure(!copyInto.offset); // otherwise, parsingBuffer cannot have HTTP response headers +- auto &adjustableReply = entry->mem().adjustableBaseReply(); +- if (adjustableReply.parseTerminatedPrefix(parsingBuffer->c_str(), parsingBuffer->contentSize())) ++ auto &adjustableReply = entry->mem().baseReply(); ++ if (adjustableReply.parseTerminatedPrefix(parsingBuffer.first.c_str(), parsingBuffer.first.contentSize())) + return true; + + // TODO: Optimize by checking memory as well. For simplicity sake, we +@@ -1052,12 +1042,12 @@ store_client::skipHttpHeadersFromDisk() + { + const auto hdr_sz = entry->mem_obj->baseReply().hdr_sz; + Assure(hdr_sz > 0); // all on-disk responses have HTTP headers +- if (Less(parsingBuffer->contentSize(), hdr_sz)) { +- debugs(90, 5, "discovered " << hdr_sz << "-byte HTTP headers in memory after reading some of them from disk: " << *parsingBuffer); +- parsingBuffer->consume(parsingBuffer->contentSize()); // skip loaded HTTP header prefix ++ if (Less(parsingBuffer.first.contentSize(), hdr_sz)) { ++ debugs(90, 5, "discovered " << hdr_sz << "-byte HTTP headers in memory after reading some of them from disk: " << parsingBuffer.first); ++ parsingBuffer.first.consume(parsingBuffer.first.contentSize()); // skip loaded HTTP header prefix + } else { +- parsingBuffer->consume(hdr_sz); // skip loaded HTTP headers +- const auto httpBodyBytesAfterHeader = parsingBuffer->contentSize(); // may be zero ++ parsingBuffer.first.consume(hdr_sz); // skip loaded HTTP headers ++ const auto httpBodyBytesAfterHeader = parsingBuffer.first.contentSize(); // may be zero + Assure(httpBodyBytesAfterHeader <= copyInto.length); + debugs(90, 5, "read HTTP body prefix: " << httpBodyBytesAfterHeader); + } +diff --git a/src/urn.cc b/src/urn.cc +index 9f5e89d..ad42b74 100644 +--- a/src/urn.cc ++++ b/src/urn.cc +@@ -238,7 +238,7 @@ urnHandleReply(void *data, StoreIOBuffer result) + return; + } + +-+ urnState->parsingBuffer.appended(result.data, result.length); ++ urnState->parsingBuffer.appended(result.data, result.length); + + /* If we haven't received the entire object (urn), copy more */ + if (!urnState->sc->atEof()) { +-- +2.39.3 + diff --git a/SOURCES/perl-requires-squid.sh b/SOURCES/perl-requires-squid.sh old mode 100755 new mode 100644 diff --git a/SOURCES/squid-4.15-CVE-2023-46724.patch b/SOURCES/squid-4.15-CVE-2023-46724.patch new file mode 100644 index 0000000..41c30aa --- /dev/null +++ b/SOURCES/squid-4.15-CVE-2023-46724.patch @@ -0,0 +1,38 @@ +From 792ef23e6e1c05780fe17f733859eef6eb8c8be3 Mon Sep 17 00:00:00 2001 +From: Andreas Weigel +Date: Wed, 18 Oct 2023 04:14:31 +0000 +Subject: [PATCH] Fix validation of certificates with CN=* (#1523) + +The bug was discovered and detailed by Joshua Rogers at +https://megamansec.github.io/Squid-Security-Audit/ +where it was filed as "Buffer UnderRead in SSL CN Parsing". +--- + src/anyp/Uri.cc | 6 ++++++ + 1 file changed, 6 insertions(+) + +diff --git a/src/anyp/Uri.cc b/src/anyp/Uri.cc +index 77b6f0c92..a6a5d5d9e 100644 +--- a/src/anyp/Uri.cc ++++ b/src/anyp/Uri.cc +@@ -173,6 +173,10 @@ urlInitialize(void) + assert(0 == matchDomainName("*.foo.com", ".foo.com", mdnHonorWildcards)); + assert(0 != matchDomainName("*.foo.com", "foo.com", mdnHonorWildcards)); + ++ assert(0 != matchDomainName("foo.com", "")); ++ assert(0 != matchDomainName("foo.com", "", mdnHonorWildcards)); ++ assert(0 != matchDomainName("foo.com", "", mdnRejectSubsubDomains)); ++ + /* more cases? */ + } + +@@ -756,6 +760,8 @@ matchDomainName(const char *h, const char *d, MatchDomainNameFlags flags) + return -1; + + dl = strlen(d); ++ if (dl == 0) ++ return 1; + + /* + * Start at the ends of the two strings and work towards the +-- +2.25.1 diff --git a/SOURCES/squid-4.15-CVE-2023-46728.patch b/SOURCES/squid-4.15-CVE-2023-46728.patch new file mode 100644 index 0000000..bb720b0 --- /dev/null +++ b/SOURCES/squid-4.15-CVE-2023-46728.patch @@ -0,0 +1,3381 @@ +From 6ea12e8fb590ac6959e9356a81aa3370576568c3 Mon Sep 17 00:00:00 2001 +From: Alex Rousskov +Date: Tue, 26 Jul 2022 15:05:54 +0000 +Subject: [PATCH] Remove support for Gopher protocol (#1092) + +Gopher code quality remains too low for production use in most +environments. The code is a persistent source of vulnerabilities and +fixing it requires significant effort. We should not be spending scarce +Project resources on improving that code, especially given the lack of +strong demand for Gopher support. + +With this change, Gopher requests will be handled like any other request +with an unknown (to Squid) protocol. For example, HTTP requests with +Gopher URI scheme result in ERR_UNSUP_REQ. + +Default Squid configuration still considers TCP port 70 "safe". The +corresponding Safe_ports ACL rule has not been removed for consistency +sake: We consider WAIS port safe even though Squid refuses to forward +WAIS requests: + + acl Safe_ports port 70 # gopher + acl Safe_ports port 210 # wais + +Back port upstream patch +Signed-Off-By: Tianyue.lan@oracle.com +--- + doc/debug-sections.txt | 1 - + errors/af/ERR_UNSUP_REQ | 2 +- + errors/ar/ERR_UNSUP_REQ | 2 +- + errors/az/ERR_UNSUP_REQ | 2 +- + errors/bg/ERR_UNSUP_REQ | 2 +- + errors/ca/ERR_UNSUP_REQ | 2 +- + errors/cs/ERR_UNSUP_REQ | 2 +- + errors/da/ERR_UNSUP_REQ | 2 +- + errors/de/ERR_UNSUP_REQ | 2 +- + errors/el/ERR_UNSUP_REQ | 2 +- + errors/en/ERR_UNSUP_REQ | 2 +- + errors/errorpage.css | 2 +- + errors/es/ERR_UNSUP_REQ | 2 +- + errors/et/ERR_UNSUP_REQ | 2 +- + errors/fa/ERR_UNSUP_REQ | 2 +- + errors/fi/ERR_UNSUP_REQ | 2 +- + errors/fr/ERR_UNSUP_REQ | 2 +- + errors/he/ERR_UNSUP_REQ | 2 +- + errors/hu/ERR_UNSUP_REQ | 2 +- + errors/hy/ERR_UNSUP_REQ | 2 +- + errors/id/ERR_UNSUP_REQ | 2 +- + errors/it/ERR_UNSUP_REQ | 2 +- + errors/ja/ERR_UNSUP_REQ | 2 +- + errors/ka/ERR_UNSUP_REQ | 2 +- + errors/ko/ERR_UNSUP_REQ | 2 +- + errors/lt/ERR_UNSUP_REQ | 2 +- + errors/lv/ERR_UNSUP_REQ | 2 +- + errors/ms/ERR_UNSUP_REQ | 2 +- + errors/nl/ERR_UNSUP_REQ | 2 +- + errors/oc/ERR_UNSUP_REQ | 2 +- + errors/pl/ERR_UNSUP_REQ | 2 +- + errors/pt-br/ERR_UNSUP_REQ | 2 +- + errors/pt/ERR_UNSUP_REQ | 2 +- + errors/ro/ERR_UNSUP_REQ | 2 +- + errors/ru/ERR_UNSUP_REQ | 2 +- + errors/sk/ERR_UNSUP_REQ | 2 +- + errors/sl/ERR_UNSUP_REQ | 2 +- + errors/sr-cyrl/ERR_UNSUP_REQ | 2 +- + errors/sr-latn/ERR_UNSUP_REQ | 2 +- + errors/sv/ERR_UNSUP_REQ | 2 +- + errors/templates/ERR_UNSUP_REQ | 2 +- + errors/th/ERR_UNSUP_REQ | 2 +- + errors/tr/ERR_UNSUP_REQ | 2 +- + errors/uk/ERR_UNSUP_REQ | 2 +- + errors/uz/ERR_UNSUP_REQ | 2 +- + errors/vi/ERR_UNSUP_REQ | 2 +- + errors/zh-hans/ERR_UNSUP_REQ | 2 +- + errors/zh-hant/ERR_UNSUP_REQ | 2 +- + src/FwdState.cc | 5 - + src/HttpMsg.h | 1 - + src/HttpRequest.cc | 6 - + src/IoStats.h | 2 +- + src/Makefile.am | 14 - + src/Makefile.in | 53 +- + src/adaptation/ecap/Host.cc | 1 - + src/adaptation/ecap/MessageRep.cc | 2 - + src/anyp/ProtocolType.cc | 1 - + src/anyp/ProtocolType.h | 1 - + src/anyp/Uri.cc | 2 - + src/anyp/UriScheme.cc | 3 - + src/cf.data.pre | 6 +- + src/cf.data.pre.config | 6 +- + src/client_side_request.cc | 4 - + src/err_type.h | 2 +- + src/gopher.cc | 977 ---------------------- + src/gopher.cc.CVE-2021-46784 | 982 ----------------------- + src/gopher.h | 29 - + src/mgr/IoAction.cc | 3 - + src/mgr/IoAction.h | 2 - + src/squid.8.in | 2 +- + src/stat.cc | 17 - + test-suite/squidconf/regressions-3.4.0.1 | 1 - + 72 files changed, 73 insertions(+), 2144 deletions(-) + delete mode 100644 src/gopher.cc + delete mode 100644 src/gopher.cc.CVE-2021-46784 + delete mode 100644 src/gopher.h + +diff --git a/doc/debug-sections.txt b/doc/debug-sections.txt +index 8b8b25f..50bd122 100644 +--- a/doc/debug-sections.txt ++++ b/doc/debug-sections.txt +@@ -27,7 +27,6 @@ section 06 Disk I/O Routines + section 07 Multicast + section 08 Swap File Bitmap + section 09 File Transfer Protocol (FTP) +-section 10 Gopher + section 11 Hypertext Transfer Protocol (HTTP) + section 12 Internet Cache Protocol (ICP) + section 13 High Level Memory Pool Management +diff --git a/errors/af/ERR_UNSUP_REQ b/errors/af/ERR_UNSUP_REQ +index c8c3152..d0895e2 100644 +--- a/errors/af/ERR_UNSUP_REQ ++++ b/errors/af/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Niegesteunde versoekmetode en -protokol

+ + +-

Squid ondersteun nie alle navraagmetodes vir alle toegangsprotokolle nie. Mens kan by voorbeeld nie 'n Gopher-navraag POST nie.

++

Squid ondersteun nie alle navraagmetodes vir alle toegangsprotokolle nie.

+ + +
+diff --git a/errors/ar/ERR_UNSUP_REQ b/errors/ar/ERR_UNSUP_REQ +index 909722f..dc8bceb 100644 +--- a/errors/ar/ERR_UNSUP_REQ ++++ b/errors/ar/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/az/ERR_UNSUP_REQ b/errors/az/ERR_UNSUP_REQ +index 50207d8..a1fba06 100644 +--- a/errors/az/ERR_UNSUP_REQ ++++ b/errors/az/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Dəstəklənməyən sorğu metodu və protokol

+ + +-

Squid bütün sorğu metodları və bütün protokollardəstəkləmir. Məsələn, Gopher protokolu üzrə siz POST sorğu metodunu yerinə yetirə bilməzsiniz.

++

Squid bütün sorğu metodları və bütün protokollardəstəkləmir.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/bg/ERR_UNSUP_REQ b/errors/bg/ERR_UNSUP_REQ +index e9130f9..6ff57a3 100644 +--- a/errors/bg/ERR_UNSUP_REQ ++++ b/errors/bg/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Сървърът не поддържа метода и/или протокола, посочен в заявката

+ + +-

Кеш сървърът не поддържа всички методи на заявка за всички протоколи. Например, не можете да заявите метод POST за протокол Gopher.

++

Кеш сървърът не поддържа всички методи на заявка за всички протоколи.

+ +

Вашият кеш администратор е %w.

+
+diff --git a/errors/ca/ERR_UNSUP_REQ b/errors/ca/ERR_UNSUP_REQ +index fe4433b..a62cf03 100644 +--- a/errors/ca/ERR_UNSUP_REQ ++++ b/errors/ca/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Mètode i protocol no admesos

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

L'administrador d'aquesta cache és %w.

+
+diff --git a/errors/cs/ERR_UNSUP_REQ b/errors/cs/ERR_UNSUP_REQ +index cb955f9..42aeb7e 100644 +--- a/errors/cs/ERR_UNSUP_REQ ++++ b/errors/cs/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid nepodporuje všechny typy metod u všech protokolů. Např. není možno použit metodu POST u služby GOPHER.

++

Squid nepodporuje všechny typy metod u všech protokolů.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/da/ERR_UNSUP_REQ b/errors/da/ERR_UNSUP_REQ +index f41d696..0d5d09a 100644 +--- a/errors/da/ERR_UNSUP_REQ ++++ b/errors/da/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Uunderstøttet Forespørgsels Metode og Protokol

+ + +-

Proxy'en Squid understøtter ikke alle forespørgselsmetoder for alle adgangs protokoller. For eksempel kan du ikke POST en Gopher forespørgsel.

++

Proxy'en Squid understøtter ikke alle forespørgselsmetoder for alle adgangs protokoller.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/de/ERR_UNSUP_REQ b/errors/de/ERR_UNSUP_REQ +index f106207..614e675 100644 +--- a/errors/de/ERR_UNSUP_REQ ++++ b/errors/de/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Anfragemethode und Protokoll nicht unterstützt

+ + +-

Squid unterstützt nicht alle Anfragemethoden für alle Protokolle. Sie können zum Beispiel keine POST Anfrage über das Gopher Protokoll senden.

++

Squid unterstützt nicht alle Anfragemethoden für alle Protokolle.

+ +

Ihr Cache Administrator ist %w.

+
+diff --git a/errors/el/ERR_UNSUP_REQ b/errors/el/ERR_UNSUP_REQ +index 0c232a5..5d092a7 100644 +--- a/errors/el/ERR_UNSUP_REQ ++++ b/errors/el/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Μη υποστηριζόμενη μέθοδος αίτησης και πρωτόκολλο

+ + +-

Το Squid δεν υποστηρίζει όλες τις μεθόδους αιτήσεων για όλα τα πρωτόκολλα πρόσβασης. Για παράδειγμα, το POST για Gopher δεν υποστηρίζεται.

++

Το Squid δεν υποστηρίζει όλες τις μεθόδους αιτήσεων για όλα τα πρωτόκολλα πρόσβασης.

+ +

Ο διαχειριστής του μεσολαβητή σας είναι ο %w.

+
+diff --git a/errors/en/ERR_UNSUP_REQ b/errors/en/ERR_UNSUP_REQ +index 352399d..e208043 100644 +--- a/errors/en/ERR_UNSUP_REQ ++++ b/errors/en/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/errorpage.css b/errors/errorpage.css +index 38ba434..facee93 100644 +--- a/errors/errorpage.css ++++ b/errors/errorpage.css +@@ -73,7 +73,7 @@ p { + pre { + } + +-/* special event: FTP / Gopher directory listing */ ++/* special event: FTP directory listing */ + #dirmsg { + font-family: courier, monospace; + color: black; +diff --git a/errors/es/ERR_UNSUP_REQ b/errors/es/ERR_UNSUP_REQ +index eb1e86e..fc1a63f 100644 +--- a/errors/es/ERR_UNSUP_REQ ++++ b/errors/es/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Método de la petición y protocolo no soportados.

+ + +-

Squid no admite todos los métodos para todos los protocolos de acceso. Por ejemplo, no se puede hacer un POST a un servidor Gopher.

++

Squid no admite todos los métodos para todos los protocolos de acceso.

+ +

Su administrador del caché es %w.

+
+diff --git a/errors/et/ERR_UNSUP_REQ b/errors/et/ERR_UNSUP_REQ +index 5488e41..cf6ec2a 100644 +--- a/errors/et/ERR_UNSUP_REQ ++++ b/errors/et/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Tundmatu päringu meetod ja protokoll

+ + +-

Squid ei toeta kõiki päringu meetodeid kõikide protokollidega. Näiteks, te ei saa teha POST operatsiooni Gopher päringus.

++

Squid ei toeta kõiki päringu meetodeid kõikide protokollidega.

+ +

Teie teenusepakkuja aadress on %w.

+
+diff --git a/errors/fa/ERR_UNSUP_REQ b/errors/fa/ERR_UNSUP_REQ +index 065da44..9940bdc 100644 +--- a/errors/fa/ERR_UNSUP_REQ ++++ b/errors/fa/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

روش پشتیبانی‌نشده درخواست و قرارداد

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/fi/ERR_UNSUP_REQ b/errors/fi/ERR_UNSUP_REQ +index 6a99e60..e06ec69 100644 +--- a/errors/fi/ERR_UNSUP_REQ ++++ b/errors/fi/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Hakupyynnon tyyppi ja yhteyskäytäntö ei tuettu

+ + +-

Squid ei tue kaikkia hakupyynnon tyyppejä kaikilla protokollilla. Et voi esimerkiksi käyttää POST-pyyntöä gopherilla.

++

Squid ei tue kaikkia hakupyynnon tyyppejä kaikilla protokollilla.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/fr/ERR_UNSUP_REQ b/errors/fr/ERR_UNSUP_REQ +index 9bccd19..ddb6b85 100644 +--- a/errors/fr/ERR_UNSUP_REQ ++++ b/errors/fr/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

La méthode de requête et le protocole ne sont pas pris en charge.

+ + +-

Squid ne prend pas en charge tous les types de requêtes par rapport à tous les protocoles d'accès. Vous ne pouvez pas par exemple utiliser une requête POST avec le protocole Gopher.

++

Squid ne prend pas en charge tous les types de requêtes par rapport à tous les protocoles d'accès.

+ +

Votre administrateur proxy est %w.

+
+diff --git a/errors/he/ERR_UNSUP_REQ b/errors/he/ERR_UNSUP_REQ +index eaff6f3..8daee1a 100644 +--- a/errors/he/ERR_UNSUP_REQ ++++ b/errors/he/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

שיטת בקשה ופרוטוקול לא נתמכים

+ + +-

שרת ה Squid אינו תומך בכל שיטות הבקשה לכל הפרוטוקולים. לדוגמא אינך יכול לשלוח בקשת POST ב-Gopher.

++

שרת ה Squid אינו תומך בכל שיטות הבקשה לכל הפרוטוקולים. לדוגמא אינך יכול לשלוח בקשת.

+ +

מנהל השרת הוא %w.

+
+diff --git a/errors/hu/ERR_UNSUP_REQ b/errors/hu/ERR_UNSUP_REQ +index a7a6e43..d1602da 100644 +--- a/errors/hu/ERR_UNSUP_REQ ++++ b/errors/hu/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Nem támogatott kéréstípus vagy protokoll

+ + +-

A proxyszerver nem támogat minden létező kéréstípus és protokoll kombinációt, így pl. nem lehet POST kéréstípust használni egy Gopher kérésben.

++

A proxyszerver nem támogat minden létező kéréstípus és protokoll kombinációt.

+ +

A proxyszerver üzemeltetőjének e-mail címe: %w.

+
+diff --git a/errors/hy/ERR_UNSUP_REQ b/errors/hy/ERR_UNSUP_REQ +index 0a3cce7..db82035 100644 +--- a/errors/hy/ERR_UNSUP_REQ ++++ b/errors/hy/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Հարցում իրականացնելու մեթոդը և արձանագրությունը չեն աջակցվում

+ + +-

Squid-ը բոլոր արձանագրությունների համար բոլոր հարցման մեթոդները չի աջակցում. Օրինակ, Gopher արձանագրության համար չեք կարող POST հարցում կատարել.

++

Squid-ը բոլոր արձանագրությունների համար բոլոր հարցման մեթոդները չի աջակցում.

+ +

Ձեր քեշի կառավարիչը %w է.

+
+diff --git a/errors/id/ERR_UNSUP_REQ b/errors/id/ERR_UNSUP_REQ +index 352399d..e208043 100644 +--- a/errors/id/ERR_UNSUP_REQ ++++ b/errors/id/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/it/ERR_UNSUP_REQ b/errors/it/ERR_UNSUP_REQ +index d6ebc13..4f770bb 100644 +--- a/errors/it/ERR_UNSUP_REQ ++++ b/errors/it/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Metodo e protocollo della richiesta non sono supportati.

+ + +-

Squid non consente di utilizzare qualsiasi tipo di richiesta per qualsiasi protocollo (a esempio non consente una richiesta POST su protocollo Gopher).

++

Squid non consente di utilizzare qualsiasi tipo di richiesta per qualsiasi protocollo.

+ +

L'amministratore del proxy è %w.

+
+diff --git a/errors/ja/ERR_UNSUP_REQ b/errors/ja/ERR_UNSUP_REQ +index 67b6cf2..a7b7950 100644 +--- a/errors/ja/ERR_UNSUP_REQ ++++ b/errors/ja/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

サポートしていないリクエストメソッドとプロトコルです。

+ + +-

Squidはすべてのアクセス・プロトコルに対して、すべてのリクエストメソッドをサポートしているわけではありません。例えば、POSTをGopherのリクエストで行うことはできません。

++

Squidはすべてのアクセス・プロトコルに対して、すべてのリクエストメソッドをサポートしているわけではありません。

+ +

Your cache administrator is %w.

+
+diff --git a/errors/ka/ERR_UNSUP_REQ b/errors/ka/ERR_UNSUP_REQ +index 1238302..8d2c62e 100644 +--- a/errors/ka/ERR_UNSUP_REQ ++++ b/errors/ka/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

თქვენი კეშის ადმინისტრატორია %w.

+
+diff --git a/errors/ko/ERR_UNSUP_REQ b/errors/ko/ERR_UNSUP_REQ +index d19ce25..ca7c946 100644 +--- a/errors/ko/ERR_UNSUP_REQ ++++ b/errors/ko/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

지원되지 않는 Request Method와 프로토콜입니다.

+ + +-

Squid는 모든 접속 프로토콜에 대한 request method를 지원하지 않습니다. 한가지 예로, Gopher에서 POST request를 사용할 수 없습니다.

++

Squid는 모든 접속 프로토콜에 대한 request method를 지원하지 않습니다.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/lt/ERR_UNSUP_REQ b/errors/lt/ERR_UNSUP_REQ +index 9e3949b..29af2de 100644 +--- a/errors/lt/ERR_UNSUP_REQ ++++ b/errors/lt/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Nepalaikomas užklausos metodas ir protokolas

+ + +-

Squid palaiko ne visus užklausos metodus daliai protokolų. Pavyzdžiui, jūs negalite vykdyti POST Gopher tipo užklausoje.

++

Squid palaiko ne visus užklausos metodus daliai protokolų.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/lv/ERR_UNSUP_REQ b/errors/lv/ERR_UNSUP_REQ +index 85450e6..88bfc8b 100644 +--- a/errors/lv/ERR_UNSUP_REQ ++++ b/errors/lv/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Neatbalstīta pieprasījuma metode un protokols

+ + +-

Squid neatbalsta visas pieprasījuma metodes visiem protokoliem. Piemēram, Jūs nevarat veikt POST pieprasījumu izmantojot Gopher protokolu.

++

Squid neatbalsta visas pieprasījuma metodes visiem protokoliem.

+ +

Jūsu kešatmiņas administrators ir %w.

+
+diff --git a/errors/ms/ERR_UNSUP_REQ b/errors/ms/ERR_UNSUP_REQ +index 987fe76..20948f5 100644 +--- a/errors/ms/ERR_UNSUP_REQ ++++ b/errors/ms/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Pengurus Proxy anda ialah %w.

+
+diff --git a/errors/nl/ERR_UNSUP_REQ b/errors/nl/ERR_UNSUP_REQ +index a8cb984..c46c47a 100644 +--- a/errors/nl/ERR_UNSUP_REQ ++++ b/errors/nl/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Niet ondersteunde verzoekmethode of protocol

+ + +-

Squid ondersteunt niet alle verzoekmethoden voor alle toegangsprotocollen. U kunt bijvoorbeeld geen Gopher verzoek POSTen.

++

Squid ondersteunt niet alle verzoekmethoden voor alle toegangsprotocollen.

+ +

De beheerder van deze cache is %w.

+
+diff --git a/errors/oc/ERR_UNSUP_REQ b/errors/oc/ERR_UNSUP_REQ +index 617f4a9..4e2ea38 100644 +--- a/errors/oc/ERR_UNSUP_REQ ++++ b/errors/oc/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Vòstre administrator d'amagatal es %w.

+
+diff --git a/errors/pl/ERR_UNSUP_REQ b/errors/pl/ERR_UNSUP_REQ +index 44bc0de..64c594c 100644 +--- a/errors/pl/ERR_UNSUP_REQ ++++ b/errors/pl/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

użyta w żądaniu kombinacja metoda/protokół jest niewłaściwa

+ + +-

Squid nie wspiera wszystkich metod we wszystkich protokołach. Na przykład nie możesz użyć metody POST w żądaniu skierowanym do usługi Gopher.

++

Squid nie wspiera wszystkich metod we wszystkich protokołach.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/pt-br/ERR_UNSUP_REQ b/errors/pt-br/ERR_UNSUP_REQ +index 60e08d3..5fbc882 100644 +--- a/errors/pt-br/ERR_UNSUP_REQ ++++ b/errors/pt-br/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Método e Protocolo de Requisição Não-Suportado

+ + +-

Squid não suporta todos os métodos de requisição para todos os protocolos de acesso. Por exemplo, você não pode emitir uma requisição POST ao protocolo Gopher.

++

Squid não suporta todos os métodos de requisição para todos os protocolos de acesso.

+ +

Seu administrador do cache é %w.

+
+diff --git a/errors/pt/ERR_UNSUP_REQ b/errors/pt/ERR_UNSUP_REQ +index ed3a68b..4b8bbbb 100644 +--- a/errors/pt/ERR_UNSUP_REQ ++++ b/errors/pt/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Método ou protocolo não suportado.

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/ro/ERR_UNSUP_REQ b/errors/ro/ERR_UNSUP_REQ +index f97375f..a237af2 100644 +--- a/errors/ro/ERR_UNSUP_REQ ++++ b/errors/ro/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Metodă de cerere şi protocol nesuportată

+ + +-

Squid nu suportă toate metodele de cerere pentru toate protocoalele de acces. De exemplu, nu puteţi face o cerere de tip POST pentru Gopher.

++

Squid nu suportă toate metodele de cerere pentru toate protocoalele de acces.

+ +

Administratorul cache-ului este %w.

+
+diff --git a/errors/ru/ERR_UNSUP_REQ b/errors/ru/ERR_UNSUP_REQ +index 2a22302..b7fa536 100644 +--- a/errors/ru/ERR_UNSUP_REQ ++++ b/errors/ru/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Неподдерживаемый метод запроса или протокол

+ + +-

Squid не поддерживает все методы запросов для всех протоколов. К примеру, для протокола Gopher Вы не можете выполнить запрос POST.

++

Squid не поддерживает все методы запросов для всех протоколов.

+ +

Администратор Вашего кэша: %w.

+
+diff --git a/errors/sk/ERR_UNSUP_REQ b/errors/sk/ERR_UNSUP_REQ +index 4c37736..aecebc7 100644 +--- a/errors/sk/ERR_UNSUP_REQ ++++ b/errors/sk/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Nepodporovaná metóda a protokol požiadavky

+ + +-

Squid nepodporuje všetky typy metód pri všetkých protokoloch. Napríklad: nie je možné použiť metódu POST pri službe Gopher.

++

Squid nepodporuje všetky typy metód pri všetkých protokoloch.

+ +

Vaším správcom cache je %w.

+
+diff --git a/errors/sl/ERR_UNSUP_REQ b/errors/sl/ERR_UNSUP_REQ +index 3fff99a..7d421a5 100644 +--- a/errors/sl/ERR_UNSUP_REQ ++++ b/errors/sl/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Nepodprta metoda zahteve in protokol

+ + +-

Squid ne podpira vseh metod zahtev za vse protokole dostopa. Tako npr. metode POST ne morete uporabiti za zahtevo Gopher.

++

Squid ne podpira vseh metod zahtev za vse protokole dostopa.

+ +

Skrbnik vašega predpomnilnika je %w.

+
+diff --git a/errors/sr-cyrl/ERR_UNSUP_REQ b/errors/sr-cyrl/ERR_UNSUP_REQ +index 352399d..e208043 100644 +--- a/errors/sr-cyrl/ERR_UNSUP_REQ ++++ b/errors/sr-cyrl/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/sr-latn/ERR_UNSUP_REQ b/errors/sr-latn/ERR_UNSUP_REQ +index 11ba17b..64ee787 100644 +--- a/errors/sr-latn/ERR_UNSUP_REQ ++++ b/errors/sr-latn/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Nepodržan metod ili protokol zahteva (Request)

+ + +-

Squid Proksi server ne podržava sve metode zahteva za sve moguæe pristupne protokole. Na primer ne možete da uradite POST na Gopher zahtev.

++

Squid Proksi server ne podržava sve metode zahteva za sve moguæe pristupne protokole.

+ +

Vaš keš/proksi administrator je: %w.

+
+diff --git a/errors/sv/ERR_UNSUP_REQ b/errors/sv/ERR_UNSUP_REQ +index 0fcb988..d7fdeef 100644 +--- a/errors/sv/ERR_UNSUP_REQ ++++ b/errors/sv/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Ej stöd för begärd Metod och Protokoll

+ + +-

Squid stödjer inte alla frågemetoder för alla protokoll. Till exempel, Ni kan inte POST'a en Gopher förfrågan.

++

Squid stödjer inte alla frågemetoder för alla protokoll.

+ +

Din cacheserver administratör är %w.

+
+diff --git a/errors/templates/ERR_UNSUP_REQ b/errors/templates/ERR_UNSUP_REQ +index e880392..196887d 100644 +--- a/errors/templates/ERR_UNSUP_REQ ++++ b/errors/templates/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/th/ERR_UNSUP_REQ b/errors/th/ERR_UNSUP_REQ +index d34fc2d..9586681 100644 +--- a/errors/th/ERR_UNSUP_REQ ++++ b/errors/th/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

ไม่รองรับโปรโตคอลและวิธีการหรือคำสั่งที่เรียกมา (request method)

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

ผู้ดูแลระบบแคชของคุณคือ %w

+
+diff --git a/errors/tr/ERR_UNSUP_REQ b/errors/tr/ERR_UNSUP_REQ +index 9c00be4..90db4b7 100644 +--- a/errors/tr/ERR_UNSUP_REQ ++++ b/errors/tr/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Desteklenmeyen istek yöntemi ve protokol.

+ + +-

Squid, bazı erişim protokollerin, bazı istek yöntemlerini desteklemiyor. Örneğin Gopher isteğinizde POST yapamazsınız.

++

Squid, bazı erişim protokollerin, bazı istek yöntemlerini desteklemiyor.

+ +

Önbellk yöneticiniz %w.

+
+diff --git a/errors/uk/ERR_UNSUP_REQ b/errors/uk/ERR_UNSUP_REQ +index d92e9e5..4ffb93e 100644 +--- a/errors/uk/ERR_UNSUP_REQ ++++ b/errors/uk/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Метод запиту чи протокол не підтримуються

+ + +-

Squid не підтримує всі методи запитів для всіх наявних протоколів. Як приклад, Ви не можете виконати запит POST для протоколу Gopher.

++

Squid не підтримує всі методи запитів для всіх наявних протоколів.

+ +

Адміністратор даного кешу %w.

+
+diff --git a/errors/uz/ERR_UNSUP_REQ b/errors/uz/ERR_UNSUP_REQ +index 47f5fe9..7c4cfa7 100644 +--- a/errors/uz/ERR_UNSUP_REQ ++++ b/errors/uz/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid does not support all request methods for all access protocols. For example, you can not POST a Gopher request.

++

Squid does not support all request methods for all access protocols.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/vi/ERR_UNSUP_REQ b/errors/vi/ERR_UNSUP_REQ +index 807df9e..f84d447 100644 +--- a/errors/vi/ERR_UNSUP_REQ ++++ b/errors/vi/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

Unsupported Request Method and Protocol

+ + +-

Squid không hỗ trợ tất cả các phương pháp yêu cầu cho mỗi giao thức truy cập. Chẳng hạn, bạn không có khả năng POST một yêu cầu Gopher.

++

Squid không hỗ trợ tất cả các phương pháp yêu cầu cho mỗi giao thức truy cập.

+ +

Your cache administrator is %w.

+
+diff --git a/errors/zh-hans/ERR_UNSUP_REQ b/errors/zh-hans/ERR_UNSUP_REQ +index 056c22b..35b28a3 100644 +--- a/errors/zh-hans/ERR_UNSUP_REQ ++++ b/errors/zh-hans/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

不支持的请求方式和协议

+ + +-

Squid （缓存服务器）不能对所有的存取协议支持所有的请求方式。比如说，你不能对 GOPHER 进行一个 POST 请求。

++

Squid （缓存服务器）不能对所有的存取协议支持所有的请求方式。

+ +

缓存服务器的管理员 %w.

+
+diff --git a/errors/zh-hant/ERR_UNSUP_REQ b/errors/zh-hant/ERR_UNSUP_REQ +index eacf4c4..8023a8b 100644 +--- a/errors/zh-hant/ERR_UNSUP_REQ ++++ b/errors/zh-hant/ERR_UNSUP_REQ +@@ -24,7 +24,7 @@ body +

尚未支援的要求方式或通訊協定

+ + +-

因為 Squid （網路快取程式）並未支援所有的連結要求方式在各式通訊協定上。比如說，你不能要求一個 GOPHER 的 POST 連結要求。

++

因為 Squid （網路快取程式）並未支援所有的連結要求方式在各式通訊協定上。

+ +

Your cache administrator is %w.

+
+diff --git a/src/FwdState.cc b/src/FwdState.cc +index 41a1679..5363572 100644 +--- a/src/FwdState.cc ++++ b/src/FwdState.cc +@@ -28,7 +28,6 @@ + #include "fde.h" + #include "FwdState.h" + #include "globals.h" +-#include "gopher.h" + #include "hier_code.h" + #include "http.h" + #include "http/Stream.h" +@@ -1007,10 +1006,6 @@ FwdState::dispatch() + httpStart(this); + break; + +- case AnyP::PROTO_GOPHER: +- gopherStart(this); +- break; +- + case AnyP::PROTO_FTP: + if (request->flags.ftpNative) + Ftp::StartRelay(this); +diff --git a/src/HttpMsg.h b/src/HttpMsg.h +index 2bf799f..06ef081 100644 +--- a/src/HttpMsg.h ++++ b/src/HttpMsg.h +@@ -38,7 +38,6 @@ public: + srcFtp = 1 << (16 + 1), ///< ftp_port or FTP server + srcIcap = 1 << (16 + 2), ///< traditional ICAP service without encryption + srcEcap = 1 << (16 + 3), ///< eCAP service that uses insecure libraries/daemons +- srcGopher = 1 << (16 + 14), ///< Gopher server + srcWhois = 1 << (16 + 15), ///< Whois server + srcUnsafe = 0xFFFF0000, ///< Unsafe sources mask + srcSafe = 0x0000FFFF ///< Safe sources mask +diff --git a/src/HttpRequest.cc b/src/HttpRequest.cc +index 0c11f5a..38b9307 100644 +--- a/src/HttpRequest.cc ++++ b/src/HttpRequest.cc +@@ -18,7 +18,6 @@ + #include "Downloader.h" + #include "err_detail_type.h" + #include "globals.h" +-#include "gopher.h" + #include "http.h" + #include "http/one/RequestParser.h" + #include "http/Stream.h" +@@ -556,11 +555,6 @@ HttpRequest::maybeCacheable() + return false; + break; + +- case AnyP::PROTO_GOPHER: +- if (!gopherCachable(this)) +- return false; +- break; +- + case AnyP::PROTO_CACHE_OBJECT: + return false; + +diff --git a/src/IoStats.h b/src/IoStats.h +index e04deef..0b69d41 100644 +--- a/src/IoStats.h ++++ b/src/IoStats.h +@@ -22,7 +22,7 @@ public: + int writes; + int write_hist[histSize]; + } +- Http, Ftp, Gopher; ++ Http, Ftp; + }; + + #endif /* SQUID_IOSTATS_H_ */ +diff --git a/src/Makefile.am b/src/Makefile.am +index 7189757..cbce754 100644 +--- a/src/Makefile.am ++++ b/src/Makefile.am +@@ -306,8 +306,6 @@ squid_SOURCES = \ + FwdState.h \ + Generic.h \ + globals.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + helper.h \ + hier_code.h \ +@@ -1260,8 +1258,6 @@ tests_testCacheManager_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + hier_code.h \ + helper.cc \ + $(HTCPSOURCE) \ +@@ -1679,8 +1675,6 @@ tests_testEvent_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -1915,8 +1909,6 @@ tests_testEventLoop_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -2146,8 +2138,6 @@ tests_test_http_range_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -2462,8 +2452,6 @@ tests_testHttpRequest_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -3308,8 +3296,6 @@ tests_testURL_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +diff --git a/src/Makefile.in b/src/Makefile.in +index 53ac16d..d46f49c 100644 +--- a/src/Makefile.in ++++ b/src/Makefile.in +@@ -263,7 +263,7 @@ am__squid_SOURCES_DIST = AclRegs.cc AuthReg.cc AccessLogEntry.cc \ + ExternalACL.h ExternalACLEntry.cc ExternalACLEntry.h \ + FadingCounter.h FadingCounter.cc fatal.h fatal.cc fd.h fd.cc \ + fde.cc fde.h FileMap.h filemap.cc fqdncache.h fqdncache.cc \ +- FwdState.cc FwdState.h Generic.h globals.h gopher.h gopher.cc \ ++ FwdState.cc FwdState.h Generic.h globals.h \ + helper.cc helper.h hier_code.h HierarchyLogEntry.h htcp.cc \ + htcp.h http.cc http.h HttpHeaderFieldStat.h HttpHdrCc.h \ + HttpHdrCc.cc HttpHdrCc.cci HttpHdrRange.cc HttpHdrSc.cc \ +@@ -352,7 +352,7 @@ am_squid_OBJECTS = $(am__objects_1) AccessLogEntry.$(OBJEXT) \ + EventLoop.$(OBJEXT) external_acl.$(OBJEXT) \ + ExternalACLEntry.$(OBJEXT) FadingCounter.$(OBJEXT) \ + fatal.$(OBJEXT) fd.$(OBJEXT) fde.$(OBJEXT) filemap.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpHdrCc.$(OBJEXT) HttpHdrRange.$(OBJEXT) HttpHdrSc.$(OBJEXT) \ + HttpHdrScTarget.$(OBJEXT) HttpHdrContRange.$(OBJEXT) \ +@@ -539,7 +539,7 @@ am__tests_testCacheManager_SOURCES_DIST = AccessLogEntry.cc debug.cc \ + tests/stub_ETag.cc event.cc external_acl.cc \ + ExternalACLEntry.cc fatal.h tests/stub_fatal.cc fd.h fd.cc \ + fde.cc FileMap.h filemap.cc fqdncache.h fqdncache.cc \ +- FwdState.cc FwdState.h gopher.h gopher.cc hier_code.h \ ++ FwdState.cc FwdState.h hier_code.h \ + helper.cc htcp.cc htcp.h http.cc HttpBody.h HttpBody.cc \ + HttpHeader.h HttpHeader.cc HttpHeaderFieldInfo.h \ + HttpHeaderTools.h HttpHeaderTools.cc HttpHeaderFieldStat.h \ +@@ -594,7 +594,7 @@ am_tests_testCacheManager_OBJECTS = AccessLogEntry.$(OBJEXT) \ + event.$(OBJEXT) external_acl.$(OBJEXT) \ + ExternalACLEntry.$(OBJEXT) tests/stub_fatal.$(OBJEXT) \ + fd.$(OBJEXT) fde.$(OBJEXT) filemap.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpBody.$(OBJEXT) HttpHeader.$(OBJEXT) \ + HttpHeaderTools.$(OBJEXT) HttpHdrCc.$(OBJEXT) \ +@@ -838,7 +838,7 @@ am__tests_testEvent_SOURCES_DIST = AccessLogEntry.cc BodyPipe.cc \ + EventLoop.h EventLoop.cc external_acl.cc ExternalACLEntry.cc \ + FadingCounter.cc fatal.h tests/stub_fatal.cc fd.h fd.cc fde.cc \ + FileMap.h filemap.cc fqdncache.h fqdncache.cc FwdState.cc \ +- FwdState.h gopher.h gopher.cc helper.cc hier_code.h htcp.cc \ ++ FwdState.h helper.cc hier_code.h htcp.cc \ + htcp.h http.cc HttpBody.h HttpBody.cc \ + tests/stub_HttpControlMsg.cc HttpHeader.h HttpHeader.cc \ + HttpHeaderFieldInfo.h HttpHeaderTools.h HttpHeaderTools.cc \ +@@ -891,7 +891,7 @@ am_tests_testEvent_OBJECTS = AccessLogEntry.$(OBJEXT) \ + external_acl.$(OBJEXT) ExternalACLEntry.$(OBJEXT) \ + FadingCounter.$(OBJEXT) tests/stub_fatal.$(OBJEXT) \ + fd.$(OBJEXT) fde.$(OBJEXT) filemap.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpBody.$(OBJEXT) tests/stub_HttpControlMsg.$(OBJEXT) \ + HttpHeader.$(OBJEXT) HttpHeaderTools.$(OBJEXT) \ +@@ -975,8 +975,8 @@ am__tests_testEventLoop_SOURCES_DIST = AccessLogEntry.cc BodyPipe.cc \ + tests/stub_ETag.cc EventLoop.h EventLoop.cc event.cc \ + external_acl.cc ExternalACLEntry.cc FadingCounter.cc fatal.h \ + tests/stub_fatal.cc fd.h fd.cc fde.cc FileMap.h filemap.cc \ +- fqdncache.h fqdncache.cc FwdState.cc FwdState.h gopher.h \ +- gopher.cc helper.cc hier_code.h htcp.cc htcp.h http.cc \ ++ fqdncache.h fqdncache.cc FwdState.cc FwdState.h \ ++ helper.cc hier_code.h htcp.cc htcp.h http.cc \ + HttpBody.h HttpBody.cc tests/stub_HttpControlMsg.cc \ + HttpHeader.h HttpHeader.cc HttpHeaderFieldInfo.h \ + HttpHeaderTools.h HttpHeaderTools.cc HttpHeaderFieldStat.h \ +@@ -1029,7 +1029,7 @@ am_tests_testEventLoop_OBJECTS = AccessLogEntry.$(OBJEXT) \ + external_acl.$(OBJEXT) ExternalACLEntry.$(OBJEXT) \ + FadingCounter.$(OBJEXT) tests/stub_fatal.$(OBJEXT) \ + fd.$(OBJEXT) fde.$(OBJEXT) filemap.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpBody.$(OBJEXT) tests/stub_HttpControlMsg.$(OBJEXT) \ + HttpHeader.$(OBJEXT) HttpHeaderTools.$(OBJEXT) \ +@@ -1187,7 +1187,7 @@ am__tests_testHttpRequest_SOURCES_DIST = AccessLogEntry.cc \ + fs_io.cc dlink.h dlink.cc dns_internal.cc errorpage.cc \ + tests/stub_ETag.cc external_acl.cc ExternalACLEntry.cc fatal.h \ + tests/stub_fatal.cc fd.h fd.cc fde.cc fqdncache.h fqdncache.cc \ +- FwdState.cc FwdState.h gopher.h gopher.cc helper.cc \ ++ FwdState.cc FwdState.h helper.cc \ + hier_code.h htcp.cc htcp.h http.cc HttpBody.h HttpBody.cc \ + tests/stub_HttpControlMsg.cc HttpHeader.h HttpHeader.cc \ + HttpHeaderFieldInfo.h HttpHeaderTools.h HttpHeaderTools.cc \ +@@ -1243,7 +1243,7 @@ am_tests_testHttpRequest_OBJECTS = AccessLogEntry.$(OBJEXT) \ + $(am__objects_4) errorpage.$(OBJEXT) tests/stub_ETag.$(OBJEXT) \ + external_acl.$(OBJEXT) ExternalACLEntry.$(OBJEXT) \ + tests/stub_fatal.$(OBJEXT) fd.$(OBJEXT) fde.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpBody.$(OBJEXT) tests/stub_HttpControlMsg.$(OBJEXT) \ + HttpHeader.$(OBJEXT) HttpHeaderTools.$(OBJEXT) \ +@@ -1670,8 +1670,8 @@ am__tests_testURL_SOURCES_DIST = AccessLogEntry.cc BodyPipe.cc \ + fs_io.cc dlink.h dlink.cc dns_internal.cc errorpage.cc ETag.cc \ + event.cc external_acl.cc ExternalACLEntry.cc fatal.h \ + tests/stub_fatal.cc fd.h fd.cc fde.cc FileMap.h filemap.cc \ +- fqdncache.h fqdncache.cc FwdState.cc FwdState.h gopher.h \ +- gopher.cc helper.cc hier_code.h htcp.cc htcp.h http.cc \ ++ fqdncache.h fqdncache.cc FwdState.cc FwdState.h \ ++ helper.cc hier_code.h htcp.cc htcp.h http.cc \ + HttpBody.h HttpBody.cc tests/stub_HttpControlMsg.cc \ + HttpHeaderFieldStat.h HttpHdrCc.h HttpHdrCc.cc HttpHdrCc.cci \ + HttpHdrContRange.cc HttpHdrRange.cc HttpHdrSc.cc \ +@@ -1725,7 +1725,7 @@ am_tests_testURL_OBJECTS = AccessLogEntry.$(OBJEXT) BodyPipe.$(OBJEXT) \ + event.$(OBJEXT) external_acl.$(OBJEXT) \ + ExternalACLEntry.$(OBJEXT) tests/stub_fatal.$(OBJEXT) \ + fd.$(OBJEXT) fde.$(OBJEXT) filemap.$(OBJEXT) \ +- fqdncache.$(OBJEXT) FwdState.$(OBJEXT) gopher.$(OBJEXT) \ ++ fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ + helper.$(OBJEXT) $(am__objects_5) http.$(OBJEXT) \ + HttpBody.$(OBJEXT) tests/stub_HttpControlMsg.$(OBJEXT) \ + HttpHdrCc.$(OBJEXT) HttpHdrContRange.$(OBJEXT) \ +@@ -1925,8 +1925,8 @@ am__tests_test_http_range_SOURCES_DIST = AccessLogEntry.cc BodyPipe.cc \ + dns_internal.cc errorpage.cc tests/stub_ETag.cc event.cc \ + FadingCounter.cc fatal.h tests/stub_libauth.cc \ + tests/stub_fatal.cc fd.h fd.cc fde.cc FileMap.h filemap.cc \ +- fqdncache.h fqdncache.cc FwdState.cc FwdState.h gopher.h \ +- gopher.cc helper.cc hier_code.h htcp.cc htcp.h http.cc \ ++ fqdncache.h fqdncache.cc FwdState.cc FwdState.h \ ++ helper.cc hier_code.h htcp.cc htcp.h http.cc \ + HttpBody.h HttpBody.cc tests/stub_HttpControlMsg.cc \ + HttpHeaderFieldStat.h HttpHdrCc.h HttpHdrCc.cc HttpHdrCc.cci \ + HttpHdrContRange.cc HttpHdrRange.cc HttpHdrSc.cc \ +@@ -1979,7 +1979,7 @@ am_tests_test_http_range_OBJECTS = AccessLogEntry.$(OBJEXT) \ + FadingCounter.$(OBJEXT) tests/stub_libauth.$(OBJEXT) \ + tests/stub_fatal.$(OBJEXT) fd.$(OBJEXT) fde.$(OBJEXT) \ + filemap.$(OBJEXT) fqdncache.$(OBJEXT) FwdState.$(OBJEXT) \ +- gopher.$(OBJEXT) helper.$(OBJEXT) $(am__objects_5) \ ++ helper.$(OBJEXT) $(am__objects_5) \ + http.$(OBJEXT) HttpBody.$(OBJEXT) \ + tests/stub_HttpControlMsg.$(OBJEXT) HttpHdrCc.$(OBJEXT) \ + HttpHdrContRange.$(OBJEXT) HttpHdrRange.$(OBJEXT) \ +@@ -2131,7 +2131,7 @@ am__depfiles_remade = ./$(DEPDIR)/AccessLogEntry.Po \ + ./$(DEPDIR)/external_acl.Po ./$(DEPDIR)/fatal.Po \ + ./$(DEPDIR)/fd.Po ./$(DEPDIR)/fde.Po ./$(DEPDIR)/filemap.Po \ + ./$(DEPDIR)/fqdncache.Po ./$(DEPDIR)/fs_io.Po \ +- ./$(DEPDIR)/globals.Po ./$(DEPDIR)/gopher.Po \ ++ ./$(DEPDIR)/globals.Po \ + ./$(DEPDIR)/helper.Po ./$(DEPDIR)/hier_code.Po \ + ./$(DEPDIR)/htcp.Po ./$(DEPDIR)/http.Po \ + ./$(DEPDIR)/icp_opcode.Po ./$(DEPDIR)/icp_v2.Po \ +@@ -3046,7 +3046,7 @@ squid_SOURCES = $(ACL_REGISTRATION_SOURCES) AccessLogEntry.cc \ + ExternalACL.h ExternalACLEntry.cc ExternalACLEntry.h \ + FadingCounter.h FadingCounter.cc fatal.h fatal.cc fd.h fd.cc \ + fde.cc fde.h FileMap.h filemap.cc fqdncache.h fqdncache.cc \ +- FwdState.cc FwdState.h Generic.h globals.h gopher.h gopher.cc \ ++ FwdState.cc FwdState.h Generic.h globals.h \ + helper.cc helper.h hier_code.h HierarchyLogEntry.h \ + $(HTCPSOURCE) http.cc http.h HttpHeaderFieldStat.h HttpHdrCc.h \ + HttpHdrCc.cc HttpHdrCc.cci HttpHdrRange.cc HttpHdrSc.cc \ +@@ -3711,8 +3711,6 @@ tests_testCacheManager_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + hier_code.h \ + helper.cc \ + $(HTCPSOURCE) \ +@@ -4137,8 +4135,6 @@ tests_testEvent_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -4374,8 +4370,6 @@ tests_testEventLoop_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -4607,8 +4601,6 @@ tests_test_http_range_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -4927,8 +4919,6 @@ tests_testHttpRequest_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -5780,8 +5770,6 @@ tests_testURL_SOURCES = \ + fqdncache.cc \ + FwdState.cc \ + FwdState.h \ +- gopher.h \ +- gopher.cc \ + helper.cc \ + hier_code.h \ + $(HTCPSOURCE) \ +@@ -6826,7 +6814,6 @@ distclean-compile: + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/fqdncache.Po@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/fs_io.Po@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/globals.Po@am__quote@ # am--include-marker +-@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/gopher.Po@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/helper.Po@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/hier_code.Po@am__quote@ # am--include-marker + @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/htcp.Po@am__quote@ # am--include-marker +@@ -7808,7 +7795,6 @@ distclean: distclean-recursive + -rm -f ./$(DEPDIR)/fqdncache.Po + -rm -f ./$(DEPDIR)/fs_io.Po + -rm -f ./$(DEPDIR)/globals.Po +- -rm -f ./$(DEPDIR)/gopher.Po + -rm -f ./$(DEPDIR)/helper.Po + -rm -f ./$(DEPDIR)/hier_code.Po + -rm -f ./$(DEPDIR)/htcp.Po +@@ -8133,7 +8119,6 @@ maintainer-clean: maintainer-clean-recursive + -rm -f ./$(DEPDIR)/fqdncache.Po + -rm -f ./$(DEPDIR)/fs_io.Po + -rm -f ./$(DEPDIR)/globals.Po +- -rm -f ./$(DEPDIR)/gopher.Po + -rm -f ./$(DEPDIR)/helper.Po + -rm -f ./$(DEPDIR)/hier_code.Po + -rm -f ./$(DEPDIR)/htcp.Po +diff --git a/src/adaptation/ecap/Host.cc b/src/adaptation/ecap/Host.cc +index 33fbb5a..5f17f86 100644 +--- a/src/adaptation/ecap/Host.cc ++++ b/src/adaptation/ecap/Host.cc +@@ -49,7 +49,6 @@ Adaptation::Ecap::Host::Host() + libecap::protocolHttp.assignHostId(AnyP::PROTO_HTTP); + libecap::protocolHttps.assignHostId(AnyP::PROTO_HTTPS); + libecap::protocolFtp.assignHostId(AnyP::PROTO_FTP); +- libecap::protocolGopher.assignHostId(AnyP::PROTO_GOPHER); + libecap::protocolWais.assignHostId(AnyP::PROTO_WAIS); + libecap::protocolUrn.assignHostId(AnyP::PROTO_URN); + libecap::protocolWhois.assignHostId(AnyP::PROTO_WHOIS); +diff --git a/src/adaptation/ecap/MessageRep.cc b/src/adaptation/ecap/MessageRep.cc +index a2779e7..94595b6 100644 +--- a/src/adaptation/ecap/MessageRep.cc ++++ b/src/adaptation/ecap/MessageRep.cc +@@ -140,8 +140,6 @@ Adaptation::Ecap::FirstLineRep::protocol() const + return libecap::protocolHttps; + case AnyP::PROTO_FTP: + return libecap::protocolFtp; +- case AnyP::PROTO_GOPHER: +- return libecap::protocolGopher; + case AnyP::PROTO_WAIS: + return libecap::protocolWais; + case AnyP::PROTO_WHOIS: +diff --git a/src/anyp/ProtocolType.cc b/src/anyp/ProtocolType.cc +index 7b8c3ef..9b92c79 100644 +--- a/src/anyp/ProtocolType.cc ++++ b/src/anyp/ProtocolType.cc +@@ -13,7 +13,6 @@ const char * ProtocolType_str[] = { + "HTTPS", + "COAP", + "COAPS", +- "GOPHER", + "WAIS", + "CACHE_OBJECT", + "ICP", +diff --git a/src/anyp/ProtocolType.h b/src/anyp/ProtocolType.h +index 66f7bc2..ef3ab25 100644 +--- a/src/anyp/ProtocolType.h ++++ b/src/anyp/ProtocolType.h +@@ -27,7 +27,6 @@ typedef enum { + PROTO_HTTPS, + PROTO_COAP, + PROTO_COAPS, +- PROTO_GOPHER, + PROTO_WAIS, + PROTO_CACHE_OBJECT, + PROTO_ICP, +diff --git a/src/anyp/Uri.cc b/src/anyp/Uri.cc +index ced3181..b0b60cf 100644 +--- a/src/anyp/Uri.cc ++++ b/src/anyp/Uri.cc +@@ -885,8 +885,6 @@ urlCheckRequest(const HttpRequest * r) + if (r->method == Http::METHOD_PUT) + rc = 1; + +- case AnyP::PROTO_GOPHER: +- + case AnyP::PROTO_WAIS: + + case AnyP::PROTO_WHOIS: +diff --git a/src/anyp/UriScheme.cc b/src/anyp/UriScheme.cc +index bac5435..f96c73f 100644 +--- a/src/anyp/UriScheme.cc ++++ b/src/anyp/UriScheme.cc +@@ -87,9 +87,6 @@ AnyP::UriScheme::defaultPort() const + // Assuming IANA policy of allocating same port for base and TLS protocol versions will occur. + return 5683; + +- case AnyP::PROTO_GOPHER: +- return 70; +- + case AnyP::PROTO_WAIS: + return 210; + +diff --git a/src/cf.data.pre b/src/cf.data.pre +index b5519b2..bc2ddcd 100644 +--- a/src/cf.data.pre ++++ b/src/cf.data.pre +@@ -1513,7 +1513,6 @@ acl SSL_ports port 443 + acl Safe_ports port 80 # http + acl Safe_ports port 21 # ftp + acl Safe_ports port 443 # https +-acl Safe_ports port 70 # gopher + acl Safe_ports port 210 # wais + acl Safe_ports port 1025-65535 # unregistered ports + acl Safe_ports port 280 # http-mgmt +@@ -4563,7 +4562,7 @@ DOC_START + [http::]url.getScheme() == AnyP::PROTO_HTTP) + return method.respMaybeCacheable(); + +- if (request->url.getScheme() == AnyP::PROTO_GOPHER) +- return gopherCachable(request); +- + if (request->url.getScheme() == AnyP::PROTO_CACHE_OBJECT) + return 0; + +diff --git a/src/err_type.h b/src/err_type.h +index 742fc5a..dbb4527 100644 +--- a/src/err_type.h ++++ b/src/err_type.h +@@ -65,7 +65,7 @@ typedef enum { + ERR_GATEWAY_FAILURE, + + /* Special Cases */ +- ERR_DIR_LISTING, /* Display of remote directory (FTP, Gopher) */ ++ ERR_DIR_LISTING, /* Display of remote directory (FTP) */ + ERR_SQUID_SIGNATURE, /* not really an error */ + ERR_SHUTTING_DOWN, + ERR_PROTOCOL_UNKNOWN, +diff --git a/src/gopher.cc b/src/gopher.cc +deleted file mode 100644 +index 6187da1..0000000 +--- a/src/gopher.cc ++++ /dev/null +@@ -1,977 +0,0 @@ +-/* +- * Copyright (C) 1996-2021 The Squid Software Foundation and contributors +- * +- * Squid software is distributed under GPLv2+ license and includes +- * contributions from numerous individuals and organizations. +- * Please see the COPYING and CONTRIBUTORS files for details. +- */ +- +-/* DEBUG: section 10 Gopher */ +- +-#include "squid.h" +-#include "comm.h" +-#include "comm/Read.h" +-#include "comm/Write.h" +-#include "errorpage.h" +-#include "fd.h" +-#include "FwdState.h" +-#include "globals.h" +-#include "html_quote.h" +-#include "HttpReply.h" +-#include "HttpRequest.h" +-#include "MemBuf.h" +-#include "mime.h" +-#include "parser/Tokenizer.h" +-#include "rfc1738.h" +-#include "SquidConfig.h" +-#include "SquidTime.h" +-#include "StatCounters.h" +-#include "Store.h" +-#include "tools.h" +- +-#if USE_DELAY_POOLS +-#include "DelayPools.h" +-#include "MemObject.h" +-#endif +- +-/* gopher type code from rfc. Anawat. */ +-#define GOPHER_FILE '0' +-#define GOPHER_DIRECTORY '1' +-#define GOPHER_CSO '2' +-#define GOPHER_ERROR '3' +-#define GOPHER_MACBINHEX '4' +-#define GOPHER_DOSBIN '5' +-#define GOPHER_UUENCODED '6' +-#define GOPHER_INDEX '7' +-#define GOPHER_TELNET '8' +-#define GOPHER_BIN '9' +-#define GOPHER_REDUNT '+' +-#define GOPHER_3270 'T' +-#define GOPHER_GIF 'g' +-#define GOPHER_IMAGE 'I' +- +-#define GOPHER_HTML 'h' +-#define GOPHER_INFO 'i' +- +-/// W3 address +-#define GOPHER_WWW 'w' +-#define GOPHER_SOUND 's' +- +-#define GOPHER_PLUS_IMAGE ':' +-#define GOPHER_PLUS_MOVIE ';' +-#define GOPHER_PLUS_SOUND '<' +- +-#define GOPHER_PORT 70 +- +-#define TAB '\t' +- +-// TODO CODE: should this be a protocol-specific thing? +-#define TEMP_BUF_SIZE 4096 +- +-#define MAX_CSO_RESULT 1024 +- +-/** +- * Gopher Gateway Internals +- * +- * Gopher is somewhat complex and gross because it must convert from +- * the Gopher protocol to HTTP. +- */ +-class GopherStateData +-{ +- CBDATA_CLASS(GopherStateData); +- +-public: +- GopherStateData(FwdState *aFwd) : +- entry(aFwd->entry), +- conversion(NORMAL), +- HTML_header_added(0), +- HTML_pre(0), +- type_id(GOPHER_FILE /* '0' */), +- cso_recno(0), +- len(0), +- buf(NULL), +- fwd(aFwd) +- { +- *request = 0; +- buf = (char *)memAllocate(MEM_4K_BUF); +- entry->lock("gopherState"); +- *replybuf = 0; +- } +- ~GopherStateData() {if(buf) swanSong();} +- +- /* AsyncJob API emulated */ +- void deleteThis(const char *aReason); +- void swanSong(); +- +-public: +- StoreEntry *entry; +- enum { +- NORMAL, +- HTML_DIR, +- HTML_INDEX_RESULT, +- HTML_CSO_RESULT, +- HTML_INDEX_PAGE, +- HTML_CSO_PAGE +- } conversion; +- int HTML_header_added; +- int HTML_pre; +- char type_id; +- char request[MAX_URL]; +- int cso_recno; +- int len; +- char *buf; /* pts to a 4k page */ +- Comm::ConnectionPointer serverConn; +- FwdState::Pointer fwd; +- HttpReply::Pointer reply_; +- char replybuf[BUFSIZ]; +-}; +- +-CBDATA_CLASS_INIT(GopherStateData); +- +-static CLCB gopherStateFree; +-static void gopherMimeCreate(GopherStateData *); +-static void gopher_request_parse(const HttpRequest * req, +- char *type_id, +- char *request); +-static void gopherEndHTML(GopherStateData *); +-static void gopherToHTML(GopherStateData *, char *inbuf, int len); +-static CTCB gopherTimeout; +-static IOCB gopherReadReply; +-static IOCB gopherSendComplete; +-static PF gopherSendRequest; +- +-static char def_gopher_bin[] = "www/unknown"; +- +-static char def_gopher_text[] = "text/plain"; +- +-static void +-gopherStateFree(const CommCloseCbParams ¶ms) +-{ +- GopherStateData *gopherState = (GopherStateData *)params.data; +- +- if (gopherState == NULL) +- return; +- +- gopherState->deleteThis("gopherStateFree"); +-} +- +-void +-GopherStateData::deleteThis(const char *) +-{ +- swanSong(); +- delete this; +-} +- +-void +-GopherStateData::swanSong() +-{ +- if (entry) +- entry->unlock("gopherState"); +- +- if (buf) { +- memFree(buf, MEM_4K_BUF); +- buf = nullptr; +- } +-} +- +-/** +- * Create MIME Header for Gopher Data +- */ +-static void +-gopherMimeCreate(GopherStateData * gopherState) +-{ +- StoreEntry *entry = gopherState->entry; +- const char *mime_type = NULL; +- const char *mime_enc = NULL; +- +- switch (gopherState->type_id) { +- +- case GOPHER_DIRECTORY: +- +- case GOPHER_INDEX: +- +- case GOPHER_HTML: +- +- case GOPHER_WWW: +- +- case GOPHER_CSO: +- mime_type = "text/html"; +- break; +- +- case GOPHER_GIF: +- +- case GOPHER_IMAGE: +- +- case GOPHER_PLUS_IMAGE: +- mime_type = "image/gif"; +- break; +- +- case GOPHER_SOUND: +- +- case GOPHER_PLUS_SOUND: +- mime_type = "audio/basic"; +- break; +- +- case GOPHER_PLUS_MOVIE: +- mime_type = "video/mpeg"; +- break; +- +- case GOPHER_MACBINHEX: +- +- case GOPHER_DOSBIN: +- +- case GOPHER_UUENCODED: +- +- case GOPHER_BIN: +- /* Rightnow We have no idea what it is. */ +- mime_enc = mimeGetContentEncoding(gopherState->request); +- mime_type = mimeGetContentType(gopherState->request); +- if (!mime_type) +- mime_type = def_gopher_bin; +- break; +- +- case GOPHER_FILE: +- +- default: +- mime_enc = mimeGetContentEncoding(gopherState->request); +- mime_type = mimeGetContentType(gopherState->request); +- if (!mime_type) +- mime_type = def_gopher_text; +- break; +- } +- +- assert(entry->isEmpty()); +- +- HttpReply *reply = new HttpReply; +- entry->buffer(); +- reply->setHeaders(Http::scOkay, "Gatewaying", mime_type, -1, -1, -2); +- if (mime_enc) +- reply->header.putStr(Http::HdrType::CONTENT_ENCODING, mime_enc); +- +- entry->replaceHttpReply(reply); +- gopherState->reply_ = reply; +-} +- +-/** +- * Parse a gopher request into components. By Anawat. +- */ +-static void +-gopher_request_parse(const HttpRequest * req, char *type_id, char *request) +-{ +- ::Parser::Tokenizer tok(req->url.path()); +- +- if (request) +- *request = 0; +- +- tok.skip('/'); // ignore failures? path could be ab-empty +- +- if (tok.atEnd()) { +- *type_id = GOPHER_DIRECTORY; +- return; +- } +- +- static const CharacterSet anyByte("UTF-8",0x00, 0xFF); +- +- SBuf typeId; +- (void)tok.prefix(typeId, anyByte, 1); // never fails since !atEnd() +- *type_id = typeId[0]; +- +- if (request) { +- SBufToCstring(request, tok.remaining().substr(0, MAX_URL-1)); +- /* convert %xx to char */ +- rfc1738_unescape(request); +- } +-} +- +-/** +- * Parse the request to determine whether it is cachable. +- * +- * \param req Request data. +- * \retval 0 Not cachable. +- * \retval 1 Cachable. +- */ +-int +-gopherCachable(const HttpRequest * req) +-{ +- int cachable = 1; +- char type_id; +- /* parse to see type */ +- gopher_request_parse(req, +- &type_id, +- NULL); +- +- switch (type_id) { +- +- case GOPHER_INDEX: +- +- case GOPHER_CSO: +- +- case GOPHER_TELNET: +- +- case GOPHER_3270: +- cachable = 0; +- break; +- +- default: +- cachable = 1; +- } +- +- return cachable; +-} +- +-static void +-gopherHTMLHeader(StoreEntry * e, const char *title, const char *substring) +-{ +- storeAppendPrintf(e, "\n"); +- storeAppendPrintf(e, ""); +- storeAppendPrintf(e, title, substring); +- storeAppendPrintf(e, ""); +- storeAppendPrintf(e, "\n"); +- storeAppendPrintf(e, "\n

"); +- storeAppendPrintf(e, title, substring); +- storeAppendPrintf(e, "

\n"); +-} +- +-static void +-gopherHTMLFooter(StoreEntry * e) +-{ +- storeAppendPrintf(e, "
\n"); +- storeAppendPrintf(e, "
\n"); +- storeAppendPrintf(e, "Generated %s by %s (%s)\n", +- mkrfc1123(squid_curtime), +- getMyHostname(), +- visible_appname_string); +- storeAppendPrintf(e, "
\n"); +-} +- +-static void +-gopherEndHTML(GopherStateData * gopherState) +-{ +- StoreEntry *e = gopherState->entry; +- +- if (!gopherState->HTML_header_added) { +- gopherHTMLHeader(e, "Server Return Nothing", NULL); +- storeAppendPrintf(e, "

The Gopher query resulted in a blank response

"); +- } else if (gopherState->HTML_pre) { +- storeAppendPrintf(e, "\n"); +- } +- +- gopherHTMLFooter(e); +-} +- +-/** +- * Convert Gopher to HTML. +- * +- * Borrow part of code from libwww2 came with Mosaic distribution. +- */ +-static void +-gopherToHTML(GopherStateData * gopherState, char *inbuf, int len) +-{ +- char *pos = inbuf; +- char *lpos = NULL; +- char *tline = NULL; +- LOCAL_ARRAY(char, line, TEMP_BUF_SIZE); +- char *name = NULL; +- char *selector = NULL; +- char *host = NULL; +- char *port = NULL; +- char *escaped_selector = NULL; +- const char *icon_url = NULL; +- char gtype; +- StoreEntry *entry = NULL; +- +- memset(line, '\0', TEMP_BUF_SIZE); +- +- entry = gopherState->entry; +- +- if (gopherState->conversion == GopherStateData::HTML_INDEX_PAGE) { +- char *html_url = html_quote(entry->url()); +- gopherHTMLHeader(entry, "Gopher Index %s", html_url); +- storeAppendPrintf(entry, +- "

This is a searchable Gopher index. Use the search\n" +- "function of your browser to enter search terms.\n" +- "\n"); +- gopherHTMLFooter(entry); +- /* now let start sending stuff to client */ +- entry->flush(); +- gopherState->HTML_header_added = 1; +- +- return; +- } +- +- if (gopherState->conversion == GopherStateData::HTML_CSO_PAGE) { +- char *html_url = html_quote(entry->url()); +- gopherHTMLHeader(entry, "CSO Search of %s", html_url); +- storeAppendPrintf(entry, +- "

A CSO database usually contains a phonebook or\n" +- "directory. Use the search function of your browser to enter\n" +- "search terms.

\n"); +- gopherHTMLFooter(entry); +- /* now let start sending stuff to client */ +- entry->flush(); +- gopherState->HTML_header_added = 1; +- +- return; +- } +- +- SBuf outbuf; +- +- if (!gopherState->HTML_header_added) { +- if (gopherState->conversion == GopherStateData::HTML_CSO_RESULT) +- gopherHTMLHeader(entry, "CSO Search Result", NULL); +- else +- gopherHTMLHeader(entry, "Gopher Menu", NULL); +- +- outbuf.append ("
");
+-
+-        gopherState->HTML_header_added = 1;
+-
+-        gopherState->HTML_pre = 1;
+-    }
+-
+-    while (pos < inbuf + len) {
+-        int llen;
+-        int left = len - (pos - inbuf);
+-        lpos = (char *)memchr(pos, '\n', left);
+-        if (lpos) {
+-            ++lpos;             /* Next line is after \n */
+-            llen = lpos - pos;
+-        } else {
+-            llen = left;
+-        }
+-        if (gopherState->len + llen >= TEMP_BUF_SIZE) {
+-            debugs(10, DBG_IMPORTANT, "GopherHTML: Buffer overflow. Lost some data on URL: " << entry->url()  );
+-            llen = TEMP_BUF_SIZE - gopherState->len - 1;
+-        }
+-        if (!lpos) {
+-            /* there is no complete line in inbuf */
+-            /* copy it to temp buffer */
+-            /* note: llen is adjusted above */
+-            memcpy(gopherState->buf + gopherState->len, pos, llen);
+-            gopherState->len += llen;
+-            break;
+-        }
+-        if (gopherState->len != 0) {
+-            /* there is something left from last tx. */
+-            memcpy(line, gopherState->buf, gopherState->len);
+-            memcpy(line + gopherState->len, pos, llen);
+-            llen += gopherState->len;
+-            gopherState->len = 0;
+-        } else {
+-            memcpy(line, pos, llen);
+-        }
+-        line[llen + 1] = '\0';
+-        /* move input to next line */
+-        pos = lpos;
+-
+-        /* at this point. We should have one line in buffer to process */
+-
+-        if (*line == '.') {
+-            /* skip it */
+-            memset(line, '\0', TEMP_BUF_SIZE);
+-            continue;
+-        }
+-
+-        switch (gopherState->conversion) {
+-
+-        case GopherStateData::HTML_INDEX_RESULT:
+-
+-        case GopherStateData::HTML_DIR: {
+-            tline = line;
+-            gtype = *tline;
+-            ++tline;
+-            name = tline;
+-            selector = strchr(tline, TAB);
+-
+-            if (selector) {
+-                *selector = '\0';
+-                ++selector;
+-                host = strchr(selector, TAB);
+-
+-                if (host) {
+-                    *host = '\0';
+-                    ++host;
+-                    port = strchr(host, TAB);
+-
+-                    if (port) {
+-                        char *junk;
+-                        port[0] = ':';
+-                        junk = strchr(host, TAB);
+-
+-                        if (junk)
+-                            *junk++ = 0;    /* Chop port */
+-                        else {
+-                            junk = strchr(host, '\r');
+-
+-                            if (junk)
+-                                *junk++ = 0;    /* Chop port */
+-                            else {
+-                                junk = strchr(host, '\n');
+-
+-                                if (junk)
+-                                    *junk++ = 0;    /* Chop port */
+-                            }
+-                        }
+-
+-                        if ((port[1] == '0') && (!port[2]))
+-                            port[0] = 0;    /* 0 means none */
+-                    }
+-
+-                    /* escape a selector here */
+-                    escaped_selector = xstrdup(rfc1738_escape_part(selector));
+-
+-                    switch (gtype) {
+-
+-                    case GOPHER_DIRECTORY:
+-                        icon_url = mimeGetIconURL("internal-menu");
+-                        break;
+-
+-                    case GOPHER_HTML:
+-
+-                    case GOPHER_FILE:
+-                        icon_url = mimeGetIconURL("internal-text");
+-                        break;
+-
+-                    case GOPHER_INDEX:
+-
+-                    case GOPHER_CSO:
+-                        icon_url = mimeGetIconURL("internal-index");
+-                        break;
+-
+-                    case GOPHER_IMAGE:
+-
+-                    case GOPHER_GIF:
+-
+-                    case GOPHER_PLUS_IMAGE:
+-                        icon_url = mimeGetIconURL("internal-image");
+-                        break;
+-
+-                    case GOPHER_SOUND:
+-
+-                    case GOPHER_PLUS_SOUND:
+-                        icon_url = mimeGetIconURL("internal-sound");
+-                        break;
+-
+-                    case GOPHER_PLUS_MOVIE:
+-                        icon_url = mimeGetIconURL("internal-movie");
+-                        break;
+-
+-                    case GOPHER_TELNET:
+-
+-                    case GOPHER_3270:
+-                        icon_url = mimeGetIconURL("internal-telnet");
+-                        break;
+-
+-                    case GOPHER_BIN:
+-
+-                    case GOPHER_MACBINHEX:
+-
+-                    case GOPHER_DOSBIN:
+-
+-                    case GOPHER_UUENCODED:
+-                        icon_url = mimeGetIconURL("internal-binary");
+-                        break;
+-
+-                    case GOPHER_INFO:
+-                        icon_url = NULL;
+-                        break;
+-
+-                    default:
+-                        icon_url = mimeGetIconURL("internal-unknown");
+-                        break;
+-                    }
+-
+-                    if ((gtype == GOPHER_TELNET) || (gtype == GOPHER_3270)) {
+-                        if (strlen(escaped_selector) != 0)
+-                            outbuf.appendf(" %s\n",
+-                                           icon_url, escaped_selector, rfc1738_escape_part(host),
+-                                           *port ? ":" : "", port, html_quote(name));
+-                        else
+-                            outbuf.appendf(" %s\n",
+-                                           icon_url, rfc1738_escape_part(host), *port ? ":" : "",
+-                                           port, html_quote(name));
+-
+-                    } else if (gtype == GOPHER_INFO) {
+-                        outbuf.appendf("\t%s\n", html_quote(name));
+-                    } else {
+-                        if (strncmp(selector, "GET /", 5) == 0) {
+-                            /* WWW link */
+-                            outbuf.appendf(" %s\n",
+-                                           icon_url, host, rfc1738_escape_unescaped(selector + 5), html_quote(name));
+-                        } else if (gtype == GOPHER_WWW) {
+-                            outbuf.appendf(" %s\n",
+-                                           icon_url, rfc1738_escape_unescaped(selector), html_quote(name));
+-                        } else {
+-                            /* Standard link */
+-                            outbuf.appendf(" %s\n",
+-                                           icon_url, host, gtype, escaped_selector, html_quote(name));
+-                        }
+-                    }
+-
+-                    safe_free(escaped_selector);
+-                } else {
+-                    memset(line, '\0', TEMP_BUF_SIZE);
+-                    continue;
+-                }
+-            } else {
+-                memset(line, '\0', TEMP_BUF_SIZE);
+-                continue;
+-            }
+-
+-            break;
+-            }           /* HTML_DIR, HTML_INDEX_RESULT */
+-
+-        case GopherStateData::HTML_CSO_RESULT: {
+-            if (line[0] == '-') {
+-                int code, recno;
+-                char *s_code, *s_recno, *result;
+-
+-                s_code = strtok(line + 1, ":\n");
+-                s_recno = strtok(NULL, ":\n");
+-                result = strtok(NULL, "\n");
+-
+-                if (!result)
+-                    break;
+-
+-                code = atoi(s_code);
+-
+-                recno = atoi(s_recno);
+-
+-                if (code != 200)
+-                    break;
+-
+-                if (gopherState->cso_recno != recno) {
+-                    outbuf.appendf("

Record# %d
%s

\n
", recno, html_quote(result));
+-                    gopherState->cso_recno = recno;
+-                } else {
+-                    outbuf.appendf("%s\n", html_quote(result));
+-                }
+-
+-                break;
+-            } else {
+-                int code;
+-                char *s_code, *result;
+-
+-                s_code = strtok(line, ":");
+-                result = strtok(NULL, "\n");
+-
+-                if (!result)
+-                    break;
+-
+-                code = atoi(s_code);
+-
+-                switch (code) {
+-
+-                case 200: {
+-                    /* OK */
+-                    /* Do nothing here */
+-                    break;
+-                }
+-
+-                case 102:   /* Number of matches */
+-
+-                case 501:   /* No Match */
+-
+-                case 502: { /* Too Many Matches */
+-                    /* Print the message the server returns */
+-                    outbuf.appendf("

%s

\n
", html_quote(result));
+-                    break;
+-                }
+-
+-                }
+-            }
+-
+-            }           /* HTML_CSO_RESULT */
+-
+-        default:
+-            break;      /* do nothing */
+-
+-        }           /* switch */
+-
+-    }               /* while loop */
+-
+-    if (outbuf.length() > 0) {
+-        entry->append(outbuf.rawContent(), outbuf.length());
+-        /* now let start sending stuff to client */
+-        entry->flush();
+-    }
+-
+-    return;
+-}
+-
+-static void
+-gopherTimeout(const CommTimeoutCbParams &io)
+-{
+-    GopherStateData *gopherState = static_cast(io.data);
+-    debugs(10, 4, HERE << io.conn << ": '" << gopherState->entry->url() << "'" );
+-
+-    gopherState->fwd->fail(new ErrorState(ERR_READ_TIMEOUT, Http::scGatewayTimeout, gopherState->fwd->request));
+-
+-    if (Comm::IsConnOpen(io.conn))
+-        io.conn->close();
+-}
+-
+-/**
+- * This will be called when data is ready to be read from fd.
+- * Read until error or connection closed.
+- */
+-static void
+-gopherReadReply(const Comm::ConnectionPointer &conn, char *buf, size_t len, Comm::Flag flag, int xerrno, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *)data;
+-    StoreEntry *entry = gopherState->entry;
+-    int clen;
+-    int bin;
+-    size_t read_sz = BUFSIZ;
+-#if USE_DELAY_POOLS
+-    DelayId delayId = entry->mem_obj->mostBytesAllowed();
+-#endif
+-
+-    /* Bail out early on Comm::ERR_CLOSING - close handlers will tidy up for us */
+-
+-    if (flag == Comm::ERR_CLOSING) {
+-        return;
+-    }
+-
+-    assert(buf == gopherState->replybuf);
+-
+-    // XXX: Should update delayId, statCounter, etc. before bailing
+-    if (!entry->isAccepting()) {
+-        debugs(10, 3, "terminating due to bad " << *entry);
+-        // TODO: Do not abuse connection for triggering cleanup.
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-#if USE_DELAY_POOLS
+-    read_sz = delayId.bytesWanted(1, read_sz);
+-#endif
+-
+-    /* leave one space for \0 in gopherToHTML */
+-
+-    if (flag == Comm::OK && len > 0) {
+-#if USE_DELAY_POOLS
+-        delayId.bytesIn(len);
+-#endif
+-
+-        statCounter.server.all.kbytes_in += len;
+-        statCounter.server.other.kbytes_in += len;
+-    }
+-
+-    debugs(10, 5, HERE << conn << " read len=" << len);
+-
+-    if (flag == Comm::OK && len > 0) {
+-        AsyncCall::Pointer nil;
+-        commSetConnTimeout(conn, Config.Timeout.read, nil);
+-        ++IOStats.Gopher.reads;
+-
+-        for (clen = len - 1, bin = 0; clen; ++bin)
+-            clen >>= 1;
+-
+-        ++IOStats.Gopher.read_hist[bin];
+-
+-        HttpRequest *req = gopherState->fwd->request;
+-        if (req->hier.bodyBytesRead < 0) {
+-            req->hier.bodyBytesRead = 0;
+-            // first bytes read, update Reply flags:
+-            gopherState->reply_->sources |= HttpMsg::srcGopher;
+-        }
+-
+-        req->hier.bodyBytesRead += len;
+-    }
+-
+-    if (flag != Comm::OK) {
+-        debugs(50, DBG_IMPORTANT, MYNAME << "error reading: " << xstrerr(xerrno));
+-
+-        if (ignoreErrno(xerrno)) {
+-            AsyncCall::Pointer call = commCbCall(5,4, "gopherReadReply",
+-                                                 CommIoCbPtrFun(gopherReadReply, gopherState));
+-            comm_read(conn, buf, read_sz, call);
+-        } else {
+-            ErrorState *err = new ErrorState(ERR_READ_ERROR, Http::scInternalServerError, gopherState->fwd->request);
+-            err->xerrno = xerrno;
+-            gopherState->fwd->fail(err);
+-            gopherState->serverConn->close();
+-        }
+-    } else if (len == 0 && entry->isEmpty()) {
+-        gopherState->fwd->fail(new ErrorState(ERR_ZERO_SIZE_OBJECT, Http::scServiceUnavailable, gopherState->fwd->request));
+-        gopherState->serverConn->close();
+-    } else if (len == 0) {
+-        /* Connection closed; retrieval done. */
+-        /* flush the rest of data in temp buf if there is one. */
+-
+-        if (gopherState->conversion != GopherStateData::NORMAL)
+-            gopherEndHTML(gopherState);
+-
+-        entry->timestampsSet();
+-        entry->flush();
+-        gopherState->fwd->complete();
+-        gopherState->serverConn->close();
+-    } else {
+-        if (gopherState->conversion != GopherStateData::NORMAL) {
+-            gopherToHTML(gopherState, buf, len);
+-        } else {
+-            entry->append(buf, len);
+-        }
+-        AsyncCall::Pointer call = commCbCall(5,4, "gopherReadReply",
+-                                             CommIoCbPtrFun(gopherReadReply, gopherState));
+-        comm_read(conn, buf, read_sz, call);
+-    }
+-}
+-
+-/**
+- * This will be called when request write is complete. Schedule read of reply.
+- */
+-static void
+-gopherSendComplete(const Comm::ConnectionPointer &conn, char *, size_t size, Comm::Flag errflag, int xerrno, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *) data;
+-    StoreEntry *entry = gopherState->entry;
+-    debugs(10, 5, HERE << conn << " size: " << size << " errflag: " << errflag);
+-
+-    if (size > 0) {
+-        fd_bytes(conn->fd, size, FD_WRITE);
+-        statCounter.server.all.kbytes_out += size;
+-        statCounter.server.other.kbytes_out += size;
+-    }
+-
+-    if (!entry->isAccepting()) {
+-        debugs(10, 3, "terminating due to bad " << *entry);
+-        // TODO: Do not abuse connection for triggering cleanup.
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-    if (errflag) {
+-        ErrorState *err;
+-        err = new ErrorState(ERR_WRITE_ERROR, Http::scServiceUnavailable, gopherState->fwd->request);
+-        err->xerrno = xerrno;
+-        err->port = gopherState->fwd->request->url.port();
+-        err->url = xstrdup(entry->url());
+-        gopherState->fwd->fail(err);
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-    /*
+-     * OK. We successfully reach remote site.  Start MIME typing
+-     * stuff.  Do it anyway even though request is not HTML type.
+-     */
+-    entry->buffer();
+-
+-    gopherMimeCreate(gopherState);
+-
+-    switch (gopherState->type_id) {
+-
+-    case GOPHER_DIRECTORY:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_DIR;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    case GOPHER_INDEX:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_INDEX_RESULT;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    case GOPHER_CSO:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_CSO_RESULT;
+-        gopherState->cso_recno = 0;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    default:
+-        gopherState->conversion = GopherStateData::NORMAL;
+-        entry->flush();
+-    }
+-
+-    /* Schedule read reply. */
+-    AsyncCall::Pointer call =  commCbCall(5,5, "gopherReadReply",
+-                                          CommIoCbPtrFun(gopherReadReply, gopherState));
+-    entry->delayAwareRead(conn, gopherState->replybuf, BUFSIZ, call);
+-}
+-
+-/**
+- * This will be called when connect completes. Write request.
+- */
+-static void
+-gopherSendRequest(int, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *)data;
+-    MemBuf mb;
+-    mb.init();
+-
+-    if (gopherState->type_id == GOPHER_CSO) {
+-        const char *t = strchr(gopherState->request, '?');
+-
+-        if (t)
+-            ++t;        /* skip the ? */
+-        else
+-            t = "";
+-
+-        mb.appendf("query %s\r\nquit", t);
+-    } else {
+-        if (gopherState->type_id == GOPHER_INDEX) {
+-            if (char *t = strchr(gopherState->request, '?'))
+-                *t = '\t';
+-        }
+-        mb.append(gopherState->request, strlen(gopherState->request));
+-    }
+-    mb.append("\r\n", 2);
+-
+-    debugs(10, 5, gopherState->serverConn);
+-    AsyncCall::Pointer call = commCbCall(5,5, "gopherSendComplete",
+-                                         CommIoCbPtrFun(gopherSendComplete, gopherState));
+-    Comm::Write(gopherState->serverConn, &mb, call);
+-
+-    if (!gopherState->entry->makePublic())
+-        gopherState->entry->makePrivate(true);
+-}
+-
+-void
+-gopherStart(FwdState * fwd)
+-{
+-    GopherStateData *gopherState = new GopherStateData(fwd);
+-
+-    debugs(10, 3, gopherState->entry->url());
+-
+-    ++ statCounter.server.all.requests;
+-
+-    ++ statCounter.server.other.requests;
+-
+-    /* Parse url. */
+-    gopher_request_parse(fwd->request,
+-                         &gopherState->type_id, gopherState->request);
+-
+-    comm_add_close_handler(fwd->serverConnection()->fd, gopherStateFree, gopherState);
+-
+-    if (((gopherState->type_id == GOPHER_INDEX) || (gopherState->type_id == GOPHER_CSO))
+-            && (strchr(gopherState->request, '?') == NULL)) {
+-        /* Index URL without query word */
+-        /* We have to generate search page back to client. No need for connection */
+-        gopherMimeCreate(gopherState);
+-
+-        if (gopherState->type_id == GOPHER_INDEX) {
+-            gopherState->conversion = GopherStateData::HTML_INDEX_PAGE;
+-        } else {
+-            if (gopherState->type_id == GOPHER_CSO) {
+-                gopherState->conversion = GopherStateData::HTML_CSO_PAGE;
+-            } else {
+-                gopherState->conversion = GopherStateData::HTML_INDEX_PAGE;
+-            }
+-        }
+-
+-        gopherToHTML(gopherState, (char *) NULL, 0);
+-        fwd->complete();
+-        return;
+-    }
+-
+-    gopherState->serverConn = fwd->serverConnection();
+-    gopherSendRequest(fwd->serverConnection()->fd, gopherState);
+-    AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "gopherTimeout",
+-                                     CommTimeoutCbPtrFun(gopherTimeout, gopherState));
+-    commSetConnTimeout(fwd->serverConnection(), Config.Timeout.read, timeoutCall);
+-}
+-
+diff --git a/src/gopher.cc.CVE-2021-46784 b/src/gopher.cc.CVE-2021-46784
+deleted file mode 100644
+index 169b0e1..0000000
+--- a/src/gopher.cc.CVE-2021-46784
++++ /dev/null
+@@ -1,982 +0,0 @@
+-/*
+- * Copyright (C) 1996-2021 The Squid Software Foundation and contributors
+- *
+- * Squid software is distributed under GPLv2+ license and includes
+- * contributions from numerous individuals and organizations.
+- * Please see the COPYING and CONTRIBUTORS files for details.
+- */
+-
+-/* DEBUG: section 10    Gopher */
+-
+-#include "squid.h"
+-#include "comm.h"
+-#include "comm/Read.h"
+-#include "comm/Write.h"
+-#include "errorpage.h"
+-#include "fd.h"
+-#include "FwdState.h"
+-#include "globals.h"
+-#include "html_quote.h"
+-#include "HttpReply.h"
+-#include "HttpRequest.h"
+-#include "MemBuf.h"
+-#include "mime.h"
+-#include "parser/Tokenizer.h"
+-#include "rfc1738.h"
+-#include "SquidConfig.h"
+-#include "SquidTime.h"
+-#include "StatCounters.h"
+-#include "Store.h"
+-#include "tools.h"
+-
+-#if USE_DELAY_POOLS
+-#include "DelayPools.h"
+-#include "MemObject.h"
+-#endif
+-
+-/* gopher type code from rfc. Anawat. */
+-#define GOPHER_FILE         '0'
+-#define GOPHER_DIRECTORY    '1'
+-#define GOPHER_CSO          '2'
+-#define GOPHER_ERROR        '3'
+-#define GOPHER_MACBINHEX    '4'
+-#define GOPHER_DOSBIN       '5'
+-#define GOPHER_UUENCODED    '6'
+-#define GOPHER_INDEX        '7'
+-#define GOPHER_TELNET       '8'
+-#define GOPHER_BIN          '9'
+-#define GOPHER_REDUNT       '+'
+-#define GOPHER_3270         'T'
+-#define GOPHER_GIF          'g'
+-#define GOPHER_IMAGE        'I'
+-
+-#define GOPHER_HTML         'h'
+-#define GOPHER_INFO         'i'
+-
+-///  W3 address
+-#define GOPHER_WWW          'w'
+-#define GOPHER_SOUND        's'
+-
+-#define GOPHER_PLUS_IMAGE   ':'
+-#define GOPHER_PLUS_MOVIE   ';'
+-#define GOPHER_PLUS_SOUND   '<'
+-
+-#define GOPHER_PORT         70
+-
+-#define TAB                 '\t'
+-
+-// TODO CODE: should this be a protocol-specific thing?
+-#define TEMP_BUF_SIZE       4096
+-
+-#define MAX_CSO_RESULT      1024
+-
+-/**
+- * Gopher Gateway Internals
+- *
+- * Gopher is somewhat complex and gross because it must convert from
+- * the Gopher protocol to HTTP.
+- */
+-class GopherStateData
+-{
+-    CBDATA_CLASS(GopherStateData);
+-
+-public:
+-    GopherStateData(FwdState *aFwd) :
+-        entry(aFwd->entry),
+-        conversion(NORMAL),
+-        HTML_header_added(0),
+-        HTML_pre(0),
+-        type_id(GOPHER_FILE /* '0' */),
+-        cso_recno(0),
+-        len(0),
+-        buf(NULL),
+-        fwd(aFwd)
+-    {
+-        *request = 0;
+-        buf = (char *)memAllocate(MEM_4K_BUF);
+-        entry->lock("gopherState");
+-        *replybuf = 0;
+-    }
+-    ~GopherStateData() {if(buf) swanSong();}
+-
+-    /* AsyncJob API emulated */
+-    void deleteThis(const char *aReason);
+-    void swanSong();
+-
+-public:
+-    StoreEntry *entry;
+-    enum {
+-        NORMAL,
+-        HTML_DIR,
+-        HTML_INDEX_RESULT,
+-        HTML_CSO_RESULT,
+-        HTML_INDEX_PAGE,
+-        HTML_CSO_PAGE
+-    } conversion;
+-    int HTML_header_added;
+-    int HTML_pre;
+-    char type_id;
+-    char request[MAX_URL];
+-    int cso_recno;
+-    int len;
+-    char *buf;          /* pts to a 4k page */
+-    Comm::ConnectionPointer serverConn;
+-    FwdState::Pointer fwd;
+-    HttpReply::Pointer reply_;
+-    char replybuf[BUFSIZ];
+-};
+-
+-CBDATA_CLASS_INIT(GopherStateData);
+-
+-static CLCB gopherStateFree;
+-static void gopherMimeCreate(GopherStateData *);
+-static void gopher_request_parse(const HttpRequest * req,
+-                                 char *type_id,
+-                                 char *request);
+-static void gopherEndHTML(GopherStateData *);
+-static void gopherToHTML(GopherStateData *, char *inbuf, int len);
+-static CTCB gopherTimeout;
+-static IOCB gopherReadReply;
+-static IOCB gopherSendComplete;
+-static PF gopherSendRequest;
+-
+-static char def_gopher_bin[] = "www/unknown";
+-
+-static char def_gopher_text[] = "text/plain";
+-
+-static void
+-gopherStateFree(const CommCloseCbParams ¶ms)
+-{
+-    GopherStateData *gopherState = (GopherStateData *)params.data;
+-
+-    if (gopherState == NULL)
+-        return;
+-
+-    gopherState->deleteThis("gopherStateFree");
+-}
+-
+-void
+-GopherStateData::deleteThis(const char *)
+-{
+-    swanSong();
+-    delete this;
+-}
+-
+-void
+-GopherStateData::swanSong()
+-{
+-    if (entry)
+-        entry->unlock("gopherState");
+-
+-    if (buf) {
+-        memFree(buf, MEM_4K_BUF);
+-        buf = nullptr;
+-    }
+-}
+-
+-/**
+- * Create MIME Header for Gopher Data
+- */
+-static void
+-gopherMimeCreate(GopherStateData * gopherState)
+-{
+-    StoreEntry *entry = gopherState->entry;
+-    const char *mime_type = NULL;
+-    const char *mime_enc = NULL;
+-
+-    switch (gopherState->type_id) {
+-
+-    case GOPHER_DIRECTORY:
+-
+-    case GOPHER_INDEX:
+-
+-    case GOPHER_HTML:
+-
+-    case GOPHER_WWW:
+-
+-    case GOPHER_CSO:
+-        mime_type = "text/html";
+-        break;
+-
+-    case GOPHER_GIF:
+-
+-    case GOPHER_IMAGE:
+-
+-    case GOPHER_PLUS_IMAGE:
+-        mime_type = "image/gif";
+-        break;
+-
+-    case GOPHER_SOUND:
+-
+-    case GOPHER_PLUS_SOUND:
+-        mime_type = "audio/basic";
+-        break;
+-
+-    case GOPHER_PLUS_MOVIE:
+-        mime_type = "video/mpeg";
+-        break;
+-
+-    case GOPHER_MACBINHEX:
+-
+-    case GOPHER_DOSBIN:
+-
+-    case GOPHER_UUENCODED:
+-
+-    case GOPHER_BIN:
+-        /* Rightnow We have no idea what it is. */
+-        mime_enc = mimeGetContentEncoding(gopherState->request);
+-        mime_type = mimeGetContentType(gopherState->request);
+-        if (!mime_type)
+-            mime_type = def_gopher_bin;
+-        break;
+-
+-    case GOPHER_FILE:
+-
+-    default:
+-        mime_enc = mimeGetContentEncoding(gopherState->request);
+-        mime_type = mimeGetContentType(gopherState->request);
+-        if (!mime_type)
+-            mime_type = def_gopher_text;
+-        break;
+-    }
+-
+-    assert(entry->isEmpty());
+-
+-    HttpReply *reply = new HttpReply;
+-    entry->buffer();
+-    reply->setHeaders(Http::scOkay, "Gatewaying", mime_type, -1, -1, -2);
+-    if (mime_enc)
+-        reply->header.putStr(Http::HdrType::CONTENT_ENCODING, mime_enc);
+-
+-    entry->replaceHttpReply(reply);
+-    gopherState->reply_ = reply;
+-}
+-
+-/**
+- * Parse a gopher request into components.  By Anawat.
+- */
+-static void
+-gopher_request_parse(const HttpRequest * req, char *type_id, char *request)
+-{
+-    ::Parser::Tokenizer tok(req->url.path());
+-
+-    if (request)
+-        *request = 0;
+-
+-    tok.skip('/'); // ignore failures? path could be ab-empty
+-
+-    if (tok.atEnd()) {
+-        *type_id = GOPHER_DIRECTORY;
+-        return;
+-    }
+-
+-    static const CharacterSet anyByte("UTF-8",0x00, 0xFF);
+-
+-    SBuf typeId;
+-    (void)tok.prefix(typeId, anyByte, 1); // never fails since !atEnd()
+-    *type_id = typeId[0];
+-
+-    if (request) {
+-        SBufToCstring(request, tok.remaining().substr(0, MAX_URL-1));
+-        /* convert %xx to char */
+-        rfc1738_unescape(request);
+-    }
+-}
+-
+-/**
+- * Parse the request to determine whether it is cachable.
+- *
+- * \param req   Request data.
+- * \retval 0    Not cachable.
+- * \retval 1    Cachable.
+- */
+-int
+-gopherCachable(const HttpRequest * req)
+-{
+-    int cachable = 1;
+-    char type_id;
+-    /* parse to see type */
+-    gopher_request_parse(req,
+-                         &type_id,
+-                         NULL);
+-
+-    switch (type_id) {
+-
+-    case GOPHER_INDEX:
+-
+-    case GOPHER_CSO:
+-
+-    case GOPHER_TELNET:
+-
+-    case GOPHER_3270:
+-        cachable = 0;
+-        break;
+-
+-    default:
+-        cachable = 1;
+-    }
+-
+-    return cachable;
+-}
+-
+-static void
+-gopherHTMLHeader(StoreEntry * e, const char *title, const char *substring)
+-{
+-    storeAppendPrintf(e, "\n");
+-    storeAppendPrintf(e, "");
+-    storeAppendPrintf(e, title, substring);
+-    storeAppendPrintf(e, "");
+-    storeAppendPrintf(e, "\n");
+-    storeAppendPrintf(e, "\n
");
+-    storeAppendPrintf(e, title, substring);
+-    storeAppendPrintf(e, "
\n");
+-}
+-
+-static void
+-gopherHTMLFooter(StoreEntry * e)
+-{
+-    storeAppendPrintf(e, "
\n");
+-    storeAppendPrintf(e, "
\n");
+-    storeAppendPrintf(e, "Generated %s by %s (%s)\n",
+-                      mkrfc1123(squid_curtime),
+-                      getMyHostname(),
+-                      visible_appname_string);
+-    storeAppendPrintf(e, "
\n");
+-}
+-
+-static void
+-gopherEndHTML(GopherStateData * gopherState)
+-{
+-    StoreEntry *e = gopherState->entry;
+-
+-    if (!gopherState->HTML_header_added) {
+-        gopherHTMLHeader(e, "Server Return Nothing", NULL);
+-        storeAppendPrintf(e, "
The Gopher query resulted in a blank response
");
+-    } else if (gopherState->HTML_pre) {
+-        storeAppendPrintf(e, "
\n"); +- } +- +- gopherHTMLFooter(e); +-} +- +-/** +- * Convert Gopher to HTML. +- * +- * Borrow part of code from libwww2 came with Mosaic distribution. +- */ +-static void +-gopherToHTML(GopherStateData * gopherState, char *inbuf, int len) +-{ +- char *pos = inbuf; +- char *lpos = NULL; +- char *tline = NULL; +- LOCAL_ARRAY(char, line, TEMP_BUF_SIZE); +- LOCAL_ARRAY(char, tmpbuf, TEMP_BUF_SIZE); +- char *name = NULL; +- char *selector = NULL; +- char *host = NULL; +- char *port = NULL; +- char *escaped_selector = NULL; +- const char *icon_url = NULL; +- char gtype; +- StoreEntry *entry = NULL; +- +- memset(tmpbuf, '\0', TEMP_BUF_SIZE); +- memset(line, '\0', TEMP_BUF_SIZE); +- +- entry = gopherState->entry; +- +- if (gopherState->conversion == GopherStateData::HTML_INDEX_PAGE) { +- char *html_url = html_quote(entry->url()); +- gopherHTMLHeader(entry, "Gopher Index %s", html_url); +- storeAppendPrintf(entry, +- "

This is a searchable Gopher index. Use the search\n" +- "function of your browser to enter search terms.\n" +- "\n"); +- gopherHTMLFooter(entry); +- /* now let start sending stuff to client */ +- entry->flush(); +- gopherState->HTML_header_added = 1; +- +- return; +- } +- +- if (gopherState->conversion == GopherStateData::HTML_CSO_PAGE) { +- char *html_url = html_quote(entry->url()); +- gopherHTMLHeader(entry, "CSO Search of %s", html_url); +- storeAppendPrintf(entry, +- "

A CSO database usually contains a phonebook or\n" +- "directory. Use the search function of your browser to enter\n" +- "search terms.

\n"); +- gopherHTMLFooter(entry); +- /* now let start sending stuff to client */ +- entry->flush(); +- gopherState->HTML_header_added = 1; +- +- return; +- } +- +- String outbuf; +- +- if (!gopherState->HTML_header_added) { +- if (gopherState->conversion == GopherStateData::HTML_CSO_RESULT) +- gopherHTMLHeader(entry, "CSO Search Result", NULL); +- else +- gopherHTMLHeader(entry, "Gopher Menu", NULL); +- +- outbuf.append ("
");
+-
+-        gopherState->HTML_header_added = 1;
+-
+-        gopherState->HTML_pre = 1;
+-    }
+-
+-    while (pos < inbuf + len) {
+-        int llen;
+-        int left = len - (pos - inbuf);
+-        lpos = (char *)memchr(pos, '\n', left);
+-        if (lpos) {
+-            ++lpos;             /* Next line is after \n */
+-            llen = lpos - pos;
+-        } else {
+-            llen = left;
+-        }
+-        if (gopherState->len + llen >= TEMP_BUF_SIZE) {
+-            debugs(10, DBG_IMPORTANT, "GopherHTML: Buffer overflow. Lost some data on URL: " << entry->url()  );
+-            llen = TEMP_BUF_SIZE - gopherState->len - 1;
+-        }
+-        if (!lpos) {
+-            /* there is no complete line in inbuf */
+-            /* copy it to temp buffer */
+-            /* note: llen is adjusted above */
+-            memcpy(gopherState->buf + gopherState->len, pos, llen);
+-            gopherState->len += llen;
+-            break;
+-        }
+-        if (gopherState->len != 0) {
+-            /* there is something left from last tx. */
+-            memcpy(line, gopherState->buf, gopherState->len);
+-            memcpy(line + gopherState->len, pos, llen);
+-            llen += gopherState->len;
+-            gopherState->len = 0;
+-        } else {
+-            memcpy(line, pos, llen);
+-        }
+-        line[llen + 1] = '\0';
+-        /* move input to next line */
+-        pos = lpos;
+-
+-        /* at this point. We should have one line in buffer to process */
+-
+-        if (*line == '.') {
+-            /* skip it */
+-            memset(line, '\0', TEMP_BUF_SIZE);
+-            continue;
+-        }
+-
+-        switch (gopherState->conversion) {
+-
+-        case GopherStateData::HTML_INDEX_RESULT:
+-
+-        case GopherStateData::HTML_DIR: {
+-            tline = line;
+-            gtype = *tline;
+-            ++tline;
+-            name = tline;
+-            selector = strchr(tline, TAB);
+-
+-            if (selector) {
+-                *selector = '\0';
+-                ++selector;
+-                host = strchr(selector, TAB);
+-
+-                if (host) {
+-                    *host = '\0';
+-                    ++host;
+-                    port = strchr(host, TAB);
+-
+-                    if (port) {
+-                        char *junk;
+-                        port[0] = ':';
+-                        junk = strchr(host, TAB);
+-
+-                        if (junk)
+-                            *junk++ = 0;    /* Chop port */
+-                        else {
+-                            junk = strchr(host, '\r');
+-
+-                            if (junk)
+-                                *junk++ = 0;    /* Chop port */
+-                            else {
+-                                junk = strchr(host, '\n');
+-
+-                                if (junk)
+-                                    *junk++ = 0;    /* Chop port */
+-                            }
+-                        }
+-
+-                        if ((port[1] == '0') && (!port[2]))
+-                            port[0] = 0;    /* 0 means none */
+-                    }
+-
+-                    /* escape a selector here */
+-                    escaped_selector = xstrdup(rfc1738_escape_part(selector));
+-
+-                    switch (gtype) {
+-
+-                    case GOPHER_DIRECTORY:
+-                        icon_url = mimeGetIconURL("internal-menu");
+-                        break;
+-
+-                    case GOPHER_HTML:
+-
+-                    case GOPHER_FILE:
+-                        icon_url = mimeGetIconURL("internal-text");
+-                        break;
+-
+-                    case GOPHER_INDEX:
+-
+-                    case GOPHER_CSO:
+-                        icon_url = mimeGetIconURL("internal-index");
+-                        break;
+-
+-                    case GOPHER_IMAGE:
+-
+-                    case GOPHER_GIF:
+-
+-                    case GOPHER_PLUS_IMAGE:
+-                        icon_url = mimeGetIconURL("internal-image");
+-                        break;
+-
+-                    case GOPHER_SOUND:
+-
+-                    case GOPHER_PLUS_SOUND:
+-                        icon_url = mimeGetIconURL("internal-sound");
+-                        break;
+-
+-                    case GOPHER_PLUS_MOVIE:
+-                        icon_url = mimeGetIconURL("internal-movie");
+-                        break;
+-
+-                    case GOPHER_TELNET:
+-
+-                    case GOPHER_3270:
+-                        icon_url = mimeGetIconURL("internal-telnet");
+-                        break;
+-
+-                    case GOPHER_BIN:
+-
+-                    case GOPHER_MACBINHEX:
+-
+-                    case GOPHER_DOSBIN:
+-
+-                    case GOPHER_UUENCODED:
+-                        icon_url = mimeGetIconURL("internal-binary");
+-                        break;
+-
+-                    case GOPHER_INFO:
+-                        icon_url = NULL;
+-                        break;
+-
+-                    default:
+-                        icon_url = mimeGetIconURL("internal-unknown");
+-                        break;
+-                    }
+-
+-                    memset(tmpbuf, '\0', TEMP_BUF_SIZE);
+-
+-                    if ((gtype == GOPHER_TELNET) || (gtype == GOPHER_3270)) {
+-                        if (strlen(escaped_selector) != 0)
+-                            snprintf(tmpbuf, TEMP_BUF_SIZE, " %s\n",
+-                                     icon_url, escaped_selector, rfc1738_escape_part(host),
+-                                     *port ? ":" : "", port, html_quote(name));
+-                        else
+-                            snprintf(tmpbuf, TEMP_BUF_SIZE, " %s\n",
+-                                     icon_url, rfc1738_escape_part(host), *port ? ":" : "",
+-                                     port, html_quote(name));
+-
+-                    } else if (gtype == GOPHER_INFO) {
+-                        snprintf(tmpbuf, TEMP_BUF_SIZE, "\t%s\n", html_quote(name));
+-                    } else {
+-                        if (strncmp(selector, "GET /", 5) == 0) {
+-                            /* WWW link */
+-                            snprintf(tmpbuf, TEMP_BUF_SIZE, " %s\n",
+-                                     icon_url, host, rfc1738_escape_unescaped(selector + 5), html_quote(name));
+-                        } else {
+-                            /* Standard link */
+-                            snprintf(tmpbuf, TEMP_BUF_SIZE, " %s\n",
+-                                     icon_url, host, gtype, escaped_selector, html_quote(name));
+-                        }
+-                    }
+-
+-                    safe_free(escaped_selector);
+-                    outbuf.append(tmpbuf);
+-                } else {
+-                    memset(line, '\0', TEMP_BUF_SIZE);
+-                    continue;
+-                }
+-            } else {
+-                memset(line, '\0', TEMP_BUF_SIZE);
+-                continue;
+-            }
+-
+-            break;
+-            }           /* HTML_DIR, HTML_INDEX_RESULT */
+-
+-        case GopherStateData::HTML_CSO_RESULT: {
+-            if (line[0] == '-') {
+-                int code, recno;
+-                char *s_code, *s_recno, *result;
+-
+-                s_code = strtok(line + 1, ":\n");
+-                s_recno = strtok(NULL, ":\n");
+-                result = strtok(NULL, "\n");
+-
+-                if (!result)
+-                    break;
+-
+-                code = atoi(s_code);
+-
+-                recno = atoi(s_recno);
+-
+-                if (code != 200)
+-                    break;
+-
+-                if (gopherState->cso_recno != recno) {
+-                    snprintf(tmpbuf, TEMP_BUF_SIZE, "

Record# %d
%s

\n
", recno, html_quote(result));
+-                    gopherState->cso_recno = recno;
+-                } else {
+-                    snprintf(tmpbuf, TEMP_BUF_SIZE, "%s\n", html_quote(result));
+-                }
+-
+-                outbuf.append(tmpbuf);
+-                break;
+-            } else {
+-                int code;
+-                char *s_code, *result;
+-
+-                s_code = strtok(line, ":");
+-                result = strtok(NULL, "\n");
+-
+-                if (!result)
+-                    break;
+-
+-                code = atoi(s_code);
+-
+-                switch (code) {
+-
+-                case 200: {
+-                    /* OK */
+-                    /* Do nothing here */
+-                    break;
+-                }
+-
+-                case 102:   /* Number of matches */
+-
+-                case 501:   /* No Match */
+-
+-                case 502: { /* Too Many Matches */
+-                    /* Print the message the server returns */
+-                    snprintf(tmpbuf, TEMP_BUF_SIZE, "

%s

\n
", html_quote(result));
+-                    outbuf.append(tmpbuf);
+-                    break;
+-                }
+-
+-                }
+-            }
+-
+-            }           /* HTML_CSO_RESULT */
+-
+-        default:
+-            break;      /* do nothing */
+-
+-        }           /* switch */
+-
+-    }               /* while loop */
+-
+-    if (outbuf.size() > 0) {
+-        entry->append(outbuf.rawBuf(), outbuf.size());
+-        /* now let start sending stuff to client */
+-        entry->flush();
+-    }
+-
+-    outbuf.clean();
+-    return;
+-}
+-
+-static void
+-gopherTimeout(const CommTimeoutCbParams &io)
+-{
+-    GopherStateData *gopherState = static_cast(io.data);
+-    debugs(10, 4, HERE << io.conn << ": '" << gopherState->entry->url() << "'" );
+-
+-    gopherState->fwd->fail(new ErrorState(ERR_READ_TIMEOUT, Http::scGatewayTimeout, gopherState->fwd->request));
+-
+-    if (Comm::IsConnOpen(io.conn))
+-        io.conn->close();
+-}
+-
+-/**
+- * This will be called when data is ready to be read from fd.
+- * Read until error or connection closed.
+- */
+-static void
+-gopherReadReply(const Comm::ConnectionPointer &conn, char *buf, size_t len, Comm::Flag flag, int xerrno, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *)data;
+-    StoreEntry *entry = gopherState->entry;
+-    int clen;
+-    int bin;
+-    size_t read_sz = BUFSIZ;
+-#if USE_DELAY_POOLS
+-    DelayId delayId = entry->mem_obj->mostBytesAllowed();
+-#endif
+-
+-    /* Bail out early on Comm::ERR_CLOSING - close handlers will tidy up for us */
+-
+-    if (flag == Comm::ERR_CLOSING) {
+-        return;
+-    }
+-
+-    assert(buf == gopherState->replybuf);
+-
+-    // XXX: Should update delayId, statCounter, etc. before bailing
+-    if (!entry->isAccepting()) {
+-        debugs(10, 3, "terminating due to bad " << *entry);
+-        // TODO: Do not abuse connection for triggering cleanup.
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-#if USE_DELAY_POOLS
+-    read_sz = delayId.bytesWanted(1, read_sz);
+-#endif
+-
+-    /* leave one space for \0 in gopherToHTML */
+-
+-    if (flag == Comm::OK && len > 0) {
+-#if USE_DELAY_POOLS
+-        delayId.bytesIn(len);
+-#endif
+-
+-        statCounter.server.all.kbytes_in += len;
+-        statCounter.server.other.kbytes_in += len;
+-    }
+-
+-    debugs(10, 5, HERE << conn << " read len=" << len);
+-
+-    if (flag == Comm::OK && len > 0) {
+-        AsyncCall::Pointer nil;
+-        commSetConnTimeout(conn, Config.Timeout.read, nil);
+-        ++IOStats.Gopher.reads;
+-
+-        for (clen = len - 1, bin = 0; clen; ++bin)
+-            clen >>= 1;
+-
+-        ++IOStats.Gopher.read_hist[bin];
+-
+-        HttpRequest *req = gopherState->fwd->request;
+-        if (req->hier.bodyBytesRead < 0) {
+-            req->hier.bodyBytesRead = 0;
+-            // first bytes read, update Reply flags:
+-            gopherState->reply_->sources |= HttpMsg::srcGopher;
+-        }
+-
+-        req->hier.bodyBytesRead += len;
+-    }
+-
+-    if (flag != Comm::OK) {
+-        debugs(50, DBG_IMPORTANT, MYNAME << "error reading: " << xstrerr(xerrno));
+-
+-        if (ignoreErrno(xerrno)) {
+-            AsyncCall::Pointer call = commCbCall(5,4, "gopherReadReply",
+-                                                 CommIoCbPtrFun(gopherReadReply, gopherState));
+-            comm_read(conn, buf, read_sz, call);
+-        } else {
+-            ErrorState *err = new ErrorState(ERR_READ_ERROR, Http::scInternalServerError, gopherState->fwd->request);
+-            err->xerrno = xerrno;
+-            gopherState->fwd->fail(err);
+-            gopherState->serverConn->close();
+-        }
+-    } else if (len == 0 && entry->isEmpty()) {
+-        gopherState->fwd->fail(new ErrorState(ERR_ZERO_SIZE_OBJECT, Http::scServiceUnavailable, gopherState->fwd->request));
+-        gopherState->serverConn->close();
+-    } else if (len == 0) {
+-        /* Connection closed; retrieval done. */
+-        /* flush the rest of data in temp buf if there is one. */
+-
+-        if (gopherState->conversion != GopherStateData::NORMAL)
+-            gopherEndHTML(gopherState);
+-
+-        entry->timestampsSet();
+-        entry->flush();
+-        gopherState->fwd->complete();
+-        gopherState->serverConn->close();
+-    } else {
+-        if (gopherState->conversion != GopherStateData::NORMAL) {
+-            gopherToHTML(gopherState, buf, len);
+-        } else {
+-            entry->append(buf, len);
+-        }
+-        AsyncCall::Pointer call = commCbCall(5,4, "gopherReadReply",
+-                                             CommIoCbPtrFun(gopherReadReply, gopherState));
+-        comm_read(conn, buf, read_sz, call);
+-    }
+-}
+-
+-/**
+- * This will be called when request write is complete. Schedule read of reply.
+- */
+-static void
+-gopherSendComplete(const Comm::ConnectionPointer &conn, char *, size_t size, Comm::Flag errflag, int xerrno, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *) data;
+-    StoreEntry *entry = gopherState->entry;
+-    debugs(10, 5, HERE << conn << " size: " << size << " errflag: " << errflag);
+-
+-    if (size > 0) {
+-        fd_bytes(conn->fd, size, FD_WRITE);
+-        statCounter.server.all.kbytes_out += size;
+-        statCounter.server.other.kbytes_out += size;
+-    }
+-
+-    if (!entry->isAccepting()) {
+-        debugs(10, 3, "terminating due to bad " << *entry);
+-        // TODO: Do not abuse connection for triggering cleanup.
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-    if (errflag) {
+-        ErrorState *err;
+-        err = new ErrorState(ERR_WRITE_ERROR, Http::scServiceUnavailable, gopherState->fwd->request);
+-        err->xerrno = xerrno;
+-        err->port = gopherState->fwd->request->url.port();
+-        err->url = xstrdup(entry->url());
+-        gopherState->fwd->fail(err);
+-        gopherState->serverConn->close();
+-        return;
+-    }
+-
+-    /*
+-     * OK. We successfully reach remote site.  Start MIME typing
+-     * stuff.  Do it anyway even though request is not HTML type.
+-     */
+-    entry->buffer();
+-
+-    gopherMimeCreate(gopherState);
+-
+-    switch (gopherState->type_id) {
+-
+-    case GOPHER_DIRECTORY:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_DIR;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    case GOPHER_INDEX:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_INDEX_RESULT;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    case GOPHER_CSO:
+-        /* we got to convert it first */
+-        gopherState->conversion = GopherStateData::HTML_CSO_RESULT;
+-        gopherState->cso_recno = 0;
+-        gopherState->HTML_header_added = 0;
+-        break;
+-
+-    default:
+-        gopherState->conversion = GopherStateData::NORMAL;
+-        entry->flush();
+-    }
+-
+-    /* Schedule read reply. */
+-    AsyncCall::Pointer call =  commCbCall(5,5, "gopherReadReply",
+-                                          CommIoCbPtrFun(gopherReadReply, gopherState));
+-    entry->delayAwareRead(conn, gopherState->replybuf, BUFSIZ, call);
+-}
+-
+-/**
+- * This will be called when connect completes. Write request.
+- */
+-static void
+-gopherSendRequest(int, void *data)
+-{
+-    GopherStateData *gopherState = (GopherStateData *)data;
+-    MemBuf mb;
+-    mb.init();
+-
+-    if (gopherState->type_id == GOPHER_CSO) {
+-        const char *t = strchr(gopherState->request, '?');
+-
+-        if (t)
+-            ++t;        /* skip the ? */
+-        else
+-            t = "";
+-
+-        mb.appendf("query %s\r\nquit", t);
+-    } else {
+-        if (gopherState->type_id == GOPHER_INDEX) {
+-            if (char *t = strchr(gopherState->request, '?'))
+-                *t = '\t';
+-        }
+-        mb.append(gopherState->request, strlen(gopherState->request));
+-    }
+-    mb.append("\r\n", 2);
+-
+-    debugs(10, 5, gopherState->serverConn);
+-    AsyncCall::Pointer call = commCbCall(5,5, "gopherSendComplete",
+-                                         CommIoCbPtrFun(gopherSendComplete, gopherState));
+-    Comm::Write(gopherState->serverConn, &mb, call);
+-
+-    if (!gopherState->entry->makePublic())
+-        gopherState->entry->makePrivate(true);
+-}
+-
+-void
+-gopherStart(FwdState * fwd)
+-{
+-    GopherStateData *gopherState = new GopherStateData(fwd);
+-
+-    debugs(10, 3, gopherState->entry->url());
+-
+-    ++ statCounter.server.all.requests;
+-
+-    ++ statCounter.server.other.requests;
+-
+-    /* Parse url. */
+-    gopher_request_parse(fwd->request,
+-                         &gopherState->type_id, gopherState->request);
+-
+-    comm_add_close_handler(fwd->serverConnection()->fd, gopherStateFree, gopherState);
+-
+-    if (((gopherState->type_id == GOPHER_INDEX) || (gopherState->type_id == GOPHER_CSO))
+-            && (strchr(gopherState->request, '?') == NULL)) {
+-        /* Index URL without query word */
+-        /* We have to generate search page back to client. No need for connection */
+-        gopherMimeCreate(gopherState);
+-
+-        if (gopherState->type_id == GOPHER_INDEX) {
+-            gopherState->conversion = GopherStateData::HTML_INDEX_PAGE;
+-        } else {
+-            if (gopherState->type_id == GOPHER_CSO) {
+-                gopherState->conversion = GopherStateData::HTML_CSO_PAGE;
+-            } else {
+-                gopherState->conversion = GopherStateData::HTML_INDEX_PAGE;
+-            }
+-        }
+-
+-        gopherToHTML(gopherState, (char *) NULL, 0);
+-        fwd->complete();
+-        return;
+-    }
+-
+-    gopherState->serverConn = fwd->serverConnection();
+-    gopherSendRequest(fwd->serverConnection()->fd, gopherState);
+-    AsyncCall::Pointer timeoutCall = commCbCall(5, 4, "gopherTimeout",
+-                                     CommTimeoutCbPtrFun(gopherTimeout, gopherState));
+-    commSetConnTimeout(fwd->serverConnection(), Config.Timeout.read, timeoutCall);
+-}
+-
+diff --git a/src/gopher.h b/src/gopher.h
+deleted file mode 100644
+index 1d73bac..0000000
+--- a/src/gopher.h
++++ /dev/null
+@@ -1,29 +0,0 @@
+-/*
+- * Copyright (C) 1996-2021 The Squid Software Foundation and contributors
+- *
+- * Squid software is distributed under GPLv2+ license and includes
+- * contributions from numerous individuals and organizations.
+- * Please see the COPYING and CONTRIBUTORS files for details.
+- */
+-
+-/* DEBUG: section 10    Gopher */
+-
+-#ifndef SQUID_GOPHER_H_
+-#define SQUID_GOPHER_H_
+-
+-class FwdState;
+-class HttpRequest;
+-
+-/**
+- \defgroup ServerProtocolGopherAPI Server-Side Gopher API
+- \ingroup ServerProtocol
+- */
+-
+-/// \ingroup ServerProtocolGopherAPI
+-void gopherStart(FwdState *);
+-
+-/// \ingroup ServerProtocolGopherAPI
+-int gopherCachable(const HttpRequest *);
+-
+-#endif /* SQUID_GOPHER_H_ */
+-
+diff --git a/src/mgr/IoAction.cc b/src/mgr/IoAction.cc
+index 149f2c4..e48a2e0 100644
+--- a/src/mgr/IoAction.cc
++++ b/src/mgr/IoAction.cc
+@@ -35,9 +35,6 @@ Mgr::IoActionData::operator += (const IoActionData& stats)
+     ftp_reads += stats.ftp_reads;
+     for (int i = 0; i < IoStats::histSize; ++i)
+         ftp_read_hist[i] += stats.ftp_read_hist[i];
+-    gopher_reads += stats.gopher_reads;
+-    for (int i = 0; i < IoStats::histSize; ++i)
+-        gopher_read_hist[i] += stats.gopher_read_hist[i];
+ 
+     return *this;
+ }
+diff --git a/src/mgr/IoAction.h b/src/mgr/IoAction.h
+index 32de089..f11ade7 100644
+--- a/src/mgr/IoAction.h
++++ b/src/mgr/IoAction.h
+@@ -27,10 +27,8 @@ public:
+ public:
+     double http_reads;
+     double ftp_reads;
+-    double gopher_reads;
+     double http_read_hist[IoStats::histSize];
+     double ftp_read_hist[IoStats::histSize];
+-    double gopher_read_hist[IoStats::histSize];
+ };
+ 
+ /// implement aggregated 'io' action
+diff --git a/src/squid.8.in b/src/squid.8.in
+index 11135c3..bfffd91 100644
+--- a/src/squid.8.in
++++ b/src/squid.8.in
+@@ -25,7 +25,7 @@ command\-line
+ .PP
+ .B squid
+ is a high\-performance proxy caching server for web clients,
+-supporting FTP, gopher, ICAP, ICP, HTCP and HTTP data objects.
++supporting FTP, ICAP, ICP, HTCP and HTTP data objects.
+ Unlike traditional caching software, 
+ Squid handles all requests in a single, non-blocking process.
+ .PP
+diff --git a/src/stat.cc b/src/stat.cc
+index 8a59be4..9f2ac49 100644
+--- a/src/stat.cc
++++ b/src/stat.cc
+@@ -207,11 +207,6 @@ GetIoStats(Mgr::IoActionData& stats)
+         stats.ftp_read_hist[i] = IOStats.Ftp.read_hist[i];
+     }
+ 
+-    stats.gopher_reads = IOStats.Gopher.reads;
+-
+-    for (i = 0; i < IoStats::histSize; ++i) {
+-        stats.gopher_read_hist[i] = IOStats.Gopher.read_hist[i];
+-    }
+ }
+ 
+ void
+@@ -244,18 +239,6 @@ DumpIoStats(Mgr::IoActionData& stats, StoreEntry* sentry)
+                           Math::doublePercent(stats.ftp_read_hist[i], stats.ftp_reads));
+     }
+ 
+-    storeAppendPrintf(sentry, "\n");
+-    storeAppendPrintf(sentry, "Gopher I/O\n");
+-    storeAppendPrintf(sentry, "number of reads: %.0f\n", stats.gopher_reads);
+-    storeAppendPrintf(sentry, "Read Histogram:\n");
+-
+-    for (i = 0; i < IoStats::histSize; ++i) {
+-        storeAppendPrintf(sentry, "%5d-%5d: %9.0f %2.0f%%\n",
+-                          i ? (1 << (i - 1)) + 1 : 1,
+-                          1 << i,
+-                          stats.gopher_read_hist[i],
+-                          Math::doublePercent(stats.gopher_read_hist[i], stats.gopher_reads));
+-    }
+ 
+     storeAppendPrintf(sentry, "\n");
+ }
+diff --git a/test-suite/squidconf/regressions-3.4.0.1 b/test-suite/squidconf/regressions-3.4.0.1
+index 41a441b..85f0a64 100644
+--- a/test-suite/squidconf/regressions-3.4.0.1
++++ b/test-suite/squidconf/regressions-3.4.0.1
+@@ -44,6 +44,5 @@ refresh_pattern -i \.(gif|png|jpg|jpeg|ico)$ 40320 75% 86400
+ refresh_pattern -i \.(iso|avi|wav|mp3|mpeg|swf|flv|x-flv)$ 1440 40% 40320
+ 
+ refresh_pattern ^ftp:           1440    20%     10080
+-refresh_pattern ^gopher:        1440    0%      1440
+ refresh_pattern -i (/cgi-bin/|\?)       0       0%      0
+ refresh_pattern .       0       20%     4320
+-- 
+2.39.3
+
diff --git a/SOURCES/squid-4.15-CVE-2023-46846.patch b/SOURCES/squid-4.15-CVE-2023-46846.patch
index ff7a0ec..5738703 100644
--- a/SOURCES/squid-4.15-CVE-2023-46846.patch
+++ b/SOURCES/squid-4.15-CVE-2023-46846.patch
@@ -1,17 +1,1122 @@
-From 1b17fc9ee99b822dcca6149d1b30e3aaf7d661cf Mon Sep 17 00:00:00 2001
-From: Amos Jeffries 
-Date: Fri, 13 Oct 2023 08:44:16 +0000
-Subject: [PATCH 2/2] RFC 9112: Improve HTTP chunked encoding compliance
- (#1498)
-
----
- src/http/one/Parser.cc          |  8 +-------
- src/http/one/Parser.h           |  4 +---
- src/http/one/TeChunkedParser.cc | 23 ++++++++++++++++++-----
- src/parser/Tokenizer.cc         | 12 ++++++++++++
- src/parser/Tokenizer.h          |  7 +++++++
- 5 files changed, 39 insertions(+), 15 deletions(-)
-
+diff --git a/src/adaptation/icap/ModXact.cc b/src/adaptation/icap/ModXact.cc
+index 2db0a68..370f077 100644
+--- a/src/adaptation/icap/ModXact.cc
++++ b/src/adaptation/icap/ModXact.cc
+@@ -25,12 +25,13 @@
+ #include "comm.h"
+ #include "comm/Connection.h"
+ #include "err_detail_type.h"
+-#include "http/one/TeChunkedParser.h"
+ #include "HttpHeaderTools.h"
+ #include "HttpMsg.h"
+ #include "HttpReply.h"
+ #include "HttpRequest.h"
+ #include "MasterXaction.h"
++#include "parser/Tokenizer.h"
++#include "sbuf/Stream.h"
+ #include "SquidTime.h"
+ 
+ // flow and terminology:
+@@ -44,6 +45,8 @@ CBDATA_NAMESPACED_CLASS_INIT(Adaptation::Icap, ModXactLauncher);
+ 
+ static const size_t TheBackupLimit = BodyPipe::MaxCapacity;
+ 
++const SBuf Adaptation::Icap::ChunkExtensionValueParser::UseOriginalBodyName("use-original-body");
++
+ Adaptation::Icap::ModXact::State::State()
+ {
+     memset(this, 0, sizeof(*this));
+@@ -1108,6 +1111,7 @@ void Adaptation::Icap::ModXact::decideOnParsingBody()
+         state.parsing = State::psBody;
+         replyHttpBodySize = 0;
+         bodyParser = new Http1::TeChunkedParser;
++        bodyParser->parseExtensionValuesWith(&extensionParser);
+         makeAdaptedBodyPipe("adapted response from the ICAP server");
+         Must(state.sending == State::sendingAdapted);
+     } else {
+@@ -1142,9 +1146,8 @@ void Adaptation::Icap::ModXact::parseBody()
+     }
+ 
+     if (parsed) {
+-        if (state.readyForUob && bodyParser->useOriginBody >= 0) {
+-            prepPartialBodyEchoing(
+-                static_cast(bodyParser->useOriginBody));
++        if (state.readyForUob && extensionParser.sawUseOriginalBody()) {
++            prepPartialBodyEchoing(extensionParser.useOriginalBody());
+             stopParsing();
+             return;
+         }
+@@ -2014,3 +2017,14 @@ void Adaptation::Icap::ModXactLauncher::updateHistory(bool doStart)
+     }
+ }
+ 
++void
++Adaptation::Icap::ChunkExtensionValueParser::parse(Tokenizer &tok, const SBuf &extName)
++{
++    if (extName == UseOriginalBodyName) {
++        useOriginalBody_ = tok.udec64("use-original-body");
++        assert(useOriginalBody_ >= 0);
++    } else {
++        Ignore(tok, extName);
++    }
++}
++
+diff --git a/src/adaptation/icap/ModXact.h b/src/adaptation/icap/ModXact.h
+index f7afa69..fb4dec0 100644
+--- a/src/adaptation/icap/ModXact.h
++++ b/src/adaptation/icap/ModXact.h
+@@ -15,6 +15,7 @@
+ #include "adaptation/icap/Xaction.h"
+ #include "BodyPipe.h"
+ #include "http/one/forward.h"
++#include "http/one/TeChunkedParser.h"
+ 
+ /*
+  * ICAPModXact implements ICAP REQMOD and RESPMOD transaction using
+@@ -105,6 +106,23 @@ private:
+     enum State { stDisabled, stWriting, stIeof, stDone } theState;
+ };
+ 
++/// handles ICAP-specific chunk extensions supported by Squid
++class ChunkExtensionValueParser: public Http1::ChunkExtensionValueParser
++{
++public:
++    /* Http1::ChunkExtensionValueParser API */
++    virtual void parse(Tokenizer &tok, const SBuf &extName) override;
++
++    bool sawUseOriginalBody() const { return useOriginalBody_ >= 0; }
++    uint64_t useOriginalBody() const { assert(sawUseOriginalBody()); return static_cast(useOriginalBody_); }
++
++private:
++    static const SBuf UseOriginalBodyName;
++
++    /// the value of the parsed use-original-body chunk extension (or -1)
++    int64_t useOriginalBody_ = -1;
++};
++
+ class ModXact: public Xaction, public BodyProducer, public BodyConsumer
+ {
+     CBDATA_CLASS(ModXact);
+@@ -270,6 +288,8 @@ private:
+ 
+     int adaptHistoryId; ///< adaptation history slot reservation
+ 
++    ChunkExtensionValueParser extensionParser;
++
+     class State
+     {
+ 
+diff --git a/src/http/one/Parser.cc b/src/http/one/Parser.cc
+index 0c86733..affe0b1 100644
+--- a/src/http/one/Parser.cc
++++ b/src/http/one/Parser.cc
+@@ -7,10 +7,11 @@
+  */
+ 
+ #include "squid.h"
++#include "base/CharacterSet.h"
+ #include "Debug.h"
+ #include "http/one/Parser.h"
+-#include "http/one/Tokenizer.h"
+ #include "mime_header.h"
++#include "parser/Tokenizer.h"
+ #include "SquidConfig.h"
+ 
+ /// RFC 7230 section 2.6 - 7 magic octets
+@@ -61,20 +62,19 @@ Http::One::Parser::DelimiterCharacters()
+            RelaxedDelimiterCharacters() : CharacterSet::SP;
+ }
+ 
+-bool
+-Http::One::Parser::skipLineTerminator(Http1::Tokenizer &tok) const
++void
++Http::One::Parser::skipLineTerminator(Tokenizer &tok) const
+ {
+     if (tok.skip(Http1::CrLf()))
+-        return true;
++        return;
+ 
+     if (Config.onoff.relaxed_header_parser && tok.skipOne(CharacterSet::LF))
+-        return true;
++        return;
+ 
+     if (tok.atEnd() || (tok.remaining().length() == 1 && tok.remaining().at(0) == '\r'))
+-        return false; // need more data
++        throw InsufficientInput();
+ 
+     throw TexcHere("garbage instead of CRLF line terminator");
+-    return false; // unreachable, but make naive compilers happy
+ }
+ 
+ /// all characters except the LF line terminator
+@@ -102,7 +102,7 @@ LineCharacters()
+ void
+ Http::One::Parser::cleanMimePrefix()
+ {
+-    Http1::Tokenizer tok(mimeHeaderBlock_);
++    Tokenizer tok(mimeHeaderBlock_);
+     while (tok.skipOne(RelaxedDelimiterCharacters())) {
+         (void)tok.skipAll(LineCharacters()); // optional line content
+         // LF terminator is required.
+@@ -137,7 +137,7 @@ Http::One::Parser::cleanMimePrefix()
+ void
+ Http::One::Parser::unfoldMime()
+ {
+-    Http1::Tokenizer tok(mimeHeaderBlock_);
++    Tokenizer tok(mimeHeaderBlock_);
+     const auto szLimit = mimeHeaderBlock_.length();
+     mimeHeaderBlock_.clear();
+     // prevent the mime sender being able to make append() realloc/grow multiple times.
+@@ -228,7 +228,7 @@ Http::One::Parser::getHostHeaderField()
+     debugs(25, 5, "looking for " << name);
+ 
+     // while we can find more LF in the SBuf
+-    Http1::Tokenizer tok(mimeHeaderBlock_);
++    Tokenizer tok(mimeHeaderBlock_);
+     SBuf p;
+ 
+     while (tok.prefix(p, LineCharacters())) {
+@@ -250,7 +250,7 @@ Http::One::Parser::getHostHeaderField()
+         p.consume(namelen + 1);
+ 
+         // TODO: optimize SBuf::trim to take CharacterSet directly
+-        Http1::Tokenizer t(p);
++        Tokenizer t(p);
+         t.skipAll(CharacterSet::WSP);
+         p = t.remaining();
+ 
+@@ -278,10 +278,15 @@ Http::One::ErrorLevel()
+ }
+ 
+ // BWS = *( SP / HTAB ) ; WhitespaceCharacters() may relax this RFC 7230 rule
+-bool
+-Http::One::ParseBws(Tokenizer &tok)
++void
++Http::One::ParseBws(Parser::Tokenizer &tok)
+ {
+-    if (const auto count = tok.skipAll(Parser::WhitespaceCharacters())) {
++    const auto count = tok.skipAll(Parser::WhitespaceCharacters());
++
++    if (tok.atEnd())
++        throw InsufficientInput(); // even if count is positive
++
++    if (count) {
+         // Generating BWS is a MUST-level violation so warn about it as needed.
+         debugs(33, ErrorLevel(), "found " << count << " BWS octets");
+         // RFC 7230 says we MUST parse BWS, so we fall through even if
+@@ -289,6 +294,6 @@ Http::One::ParseBws(Tokenizer &tok)
+     }
+     // else we successfully "parsed" an empty BWS sequence
+ 
+-    return true;
++    // success: no more BWS characters expected
+ }
+ 
+diff --git a/src/http/one/Parser.h b/src/http/one/Parser.h
+index 58a5cae..40e281b 100644
+--- a/src/http/one/Parser.h
++++ b/src/http/one/Parser.h
+@@ -12,6 +12,7 @@
+ #include "anyp/ProtocolVersion.h"
+ #include "http/one/forward.h"
+ #include "http/StatusCode.h"
++#include "parser/forward.h"
+ #include "sbuf/SBuf.h"
+ 
+ namespace Http {
+@@ -40,6 +41,7 @@ class Parser : public RefCountable
+ {
+ public:
+     typedef SBuf::size_type size_type;
++    typedef ::Parser::Tokenizer Tokenizer;
+ 
+     Parser() : parseStatusCode(Http::scNone), parsingStage_(HTTP_PARSE_NONE), hackExpectsMime_(false) {}
+     virtual ~Parser() {}
+@@ -118,11 +120,11 @@ protected:
+      * detect and skip the CRLF or (if tolerant) LF line terminator
+      * consume from the tokenizer.
+      *
+-     * throws if non-terminator is detected.
++     * \throws exception on bad or InsuffientInput.
+      * \retval true only if line terminator found.
+      * \retval false incomplete or missing line terminator, need more data.
+      */
+-    bool skipLineTerminator(Http1::Tokenizer &tok) const;
++    void skipLineTerminator(Tokenizer &) const;
+ 
+     /**
+      * Scan to find the mime headers block for current message.
+@@ -159,8 +161,8 @@ private:
+ };
+ 
+ /// skips and, if needed, warns about RFC 7230 BWS ("bad" whitespace)
+-/// \returns true (always; unlike all the skip*() functions)
+-bool ParseBws(Tokenizer &tok);
++/// \throws InsufficientInput when the end of BWS cannot be confirmed
++void ParseBws(Parser::Tokenizer &);
+ 
+ /// the right debugs() level for logging HTTP violation messages
+ int ErrorLevel();
+diff --git a/src/http/one/RequestParser.cc b/src/http/one/RequestParser.cc
+index a325f7d..0f13c92 100644
+--- a/src/http/one/RequestParser.cc
++++ b/src/http/one/RequestParser.cc
+@@ -9,8 +9,8 @@
+ #include "squid.h"
+ #include "Debug.h"
+ #include "http/one/RequestParser.h"
+-#include "http/one/Tokenizer.h"
+ #include "http/ProtocolVersion.h"
++#include "parser/Tokenizer.h"
+ #include "profiler/Profiler.h"
+ #include "SquidConfig.h"
+ 
+@@ -64,7 +64,7 @@ Http::One::RequestParser::skipGarbageLines()
+  *  RFC 7230 section 2.6, 3.1 and 3.5
+  */
+ bool
+-Http::One::RequestParser::parseMethodField(Http1::Tokenizer &tok)
++Http::One::RequestParser::parseMethodField(Tokenizer &tok)
+ {
+     // method field is a sequence of TCHAR.
+     // Limit to 32 characters to prevent overly long sequences of non-HTTP
+@@ -145,7 +145,7 @@ Http::One::RequestParser::RequestTargetCharacters()
+ }
+ 
+ bool
+-Http::One::RequestParser::parseUriField(Http1::Tokenizer &tok)
++Http::One::RequestParser::parseUriField(Tokenizer &tok)
+ {
+     /* Arbitrary 64KB URI upper length limit.
+      *
+@@ -178,7 +178,7 @@ Http::One::RequestParser::parseUriField(Http1::Tokenizer &tok)
+ }
+ 
+ bool
+-Http::One::RequestParser::parseHttpVersionField(Http1::Tokenizer &tok)
++Http::One::RequestParser::parseHttpVersionField(Tokenizer &tok)
+ {
+     static const SBuf http1p0("HTTP/1.0");
+     static const SBuf http1p1("HTTP/1.1");
+@@ -253,7 +253,7 @@ Http::One::RequestParser::skipDelimiter(const size_t count, const char *where)
+ 
+ /// Parse CRs at the end of request-line, just before the terminating LF.
+ bool
+-Http::One::RequestParser::skipTrailingCrs(Http1::Tokenizer &tok)
++Http::One::RequestParser::skipTrailingCrs(Tokenizer &tok)
+ {
+     if (Config.onoff.relaxed_header_parser) {
+         (void)tok.skipAllTrailing(CharacterSet::CR); // optional; multiple OK
+@@ -289,12 +289,12 @@ Http::One::RequestParser::parseRequestFirstLine()
+     // Earlier, skipGarbageLines() took care of any leading LFs (if allowed).
+     // Now, the request line has to end at the first LF.
+     static const CharacterSet lineChars = CharacterSet::LF.complement("notLF");
+-    ::Parser::Tokenizer lineTok(buf_);
++    Tokenizer lineTok(buf_);
+     if (!lineTok.prefix(line, lineChars) || !lineTok.skip('\n')) {
+         if (buf_.length() >= Config.maxRequestHeaderSize) {
+             /* who should we blame for our failure to parse this line? */
+ 
+-            Http1::Tokenizer methodTok(buf_);
++            Tokenizer methodTok(buf_);
+             if (!parseMethodField(methodTok))
+                 return -1; // blame a bad method (or its delimiter)
+ 
+@@ -308,7 +308,7 @@ Http::One::RequestParser::parseRequestFirstLine()
+         return 0;
+     }
+ 
+-    Http1::Tokenizer tok(line);
++    Tokenizer tok(line);
+ 
+     if (!parseMethodField(tok))
+         return -1;
+diff --git a/src/http/one/RequestParser.h b/src/http/one/RequestParser.h
+index 7086548..26697cd 100644
+--- a/src/http/one/RequestParser.h
++++ b/src/http/one/RequestParser.h
+@@ -54,11 +54,11 @@ private:
+     bool doParse(const SBuf &aBuf);
+ 
+     /* all these return false and set parseStatusCode on parsing failures */
+-    bool parseMethodField(Http1::Tokenizer &);
+-    bool parseUriField(Http1::Tokenizer &);
+-    bool parseHttpVersionField(Http1::Tokenizer &);
++    bool parseMethodField(Tokenizer &);
++    bool parseUriField(Tokenizer &);
++    bool parseHttpVersionField(Tokenizer &);
+     bool skipDelimiter(const size_t count, const char *where);
+-    bool skipTrailingCrs(Http1::Tokenizer &tok);
++    bool skipTrailingCrs(Tokenizer &tok);
+ 
+     bool http0() const {return !msgProtocol_.major;}
+     static const CharacterSet &RequestTargetCharacters();
+diff --git a/src/http/one/ResponseParser.cc b/src/http/one/ResponseParser.cc
+index 24af849..65baf09 100644
+--- a/src/http/one/ResponseParser.cc
++++ b/src/http/one/ResponseParser.cc
+@@ -9,8 +9,8 @@
+ #include "squid.h"
+ #include "Debug.h"
+ #include "http/one/ResponseParser.h"
+-#include "http/one/Tokenizer.h"
+ #include "http/ProtocolVersion.h"
++#include "parser/Tokenizer.h"
+ #include "profiler/Profiler.h"
+ #include "SquidConfig.h"
+ 
+@@ -47,7 +47,7 @@ Http::One::ResponseParser::firstLineSize() const
+ // NP: we found the protocol version and consumed it already.
+ // just need the status code and reason phrase
+ int
+-Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, const CharacterSet &WspDelim)
++Http::One::ResponseParser::parseResponseStatusAndReason(Tokenizer &tok, const CharacterSet &WspDelim)
+ {
+     if (!completedStatus_) {
+         debugs(74, 9, "seek status-code in: " << tok.remaining().substr(0,10) << "...");
+@@ -87,14 +87,13 @@ Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, c
+     static const CharacterSet phraseChars = CharacterSet::WSP + CharacterSet::VCHAR + CharacterSet::OBSTEXT;
+     (void)tok.prefix(reasonPhrase_, phraseChars); // optional, no error if missing
+     try {
+-        if (skipLineTerminator(tok)) {
+-            debugs(74, DBG_DATA, "parse remaining buf={length=" << tok.remaining().length() << ", data='" << tok.remaining() << "'}");
+-            buf_ = tok.remaining(); // resume checkpoint
+-            return 1;
+-        }
++        skipLineTerminator(tok);
++        buf_ = tok.remaining(); // resume checkpoint
++        debugs(74, DBG_DATA, Raw("leftovers", buf_.rawContent(), buf_.length()));
++        return 1;
++    } catch (const InsufficientInput &) {
+         reasonPhrase_.clear();
+         return 0; // need more to be sure we have it all
+-
+     } catch (const std::exception &ex) {
+         debugs(74, 6, "invalid status-line: " << ex.what());
+     }
+@@ -119,7 +118,7 @@ Http::One::ResponseParser::parseResponseStatusAndReason(Http1::Tokenizer &tok, c
+ int
+ Http::One::ResponseParser::parseResponseFirstLine()
+ {
+-    Http1::Tokenizer tok(buf_);
++    Tokenizer tok(buf_);
+ 
+     const CharacterSet &WspDelim = DelimiterCharacters();
+ 
+diff --git a/src/http/one/ResponseParser.h b/src/http/one/ResponseParser.h
+index 15db4a0..cf13b4d 100644
+--- a/src/http/one/ResponseParser.h
++++ b/src/http/one/ResponseParser.h
+@@ -43,7 +43,7 @@ public:
+ 
+ private:
+     int parseResponseFirstLine();
+-    int parseResponseStatusAndReason(Http1::Tokenizer&, const CharacterSet &);
++    int parseResponseStatusAndReason(Tokenizer&, const CharacterSet &);
+ 
+     /// magic prefix for identifying ICY response messages
+     static const SBuf IcyMagic;
+diff --git a/src/http/one/TeChunkedParser.cc b/src/http/one/TeChunkedParser.cc
+index 754086e..6d2f8ea 100644
+--- a/src/http/one/TeChunkedParser.cc
++++ b/src/http/one/TeChunkedParser.cc
+@@ -13,10 +13,13 @@
+ #include "http/one/Tokenizer.h"
+ #include "http/ProtocolVersion.h"
+ #include "MemBuf.h"
++#include "parser/Tokenizer.h"
+ #include "Parsing.h"
++#include "sbuf/Stream.h"
+ #include "SquidConfig.h"
+ 
+-Http::One::TeChunkedParser::TeChunkedParser()
++Http::One::TeChunkedParser::TeChunkedParser():
++    customExtensionValueParser(nullptr)
+ {
+     // chunked encoding only exists in HTTP/1.1
+     Http1::Parser::msgProtocol_ = Http::ProtocolVersion(1,1);
+@@ -31,7 +34,11 @@ Http::One::TeChunkedParser::clear()
+     buf_.clear();
+     theChunkSize = theLeftBodySize = 0;
+     theOut = NULL;
+-    useOriginBody = -1;
++    // XXX: We do not reset customExtensionValueParser here. Based on the
++    // clear() API description, we must, but it makes little sense and could
++    // break method callers if they appear because some of them may forget to
++    // reset customExtensionValueParser. TODO: Remove Http1::Parser as our
++    // parent class and this unnecessary method with it.
+ }
+ 
+ bool
+@@ -49,14 +56,14 @@ Http::One::TeChunkedParser::parse(const SBuf &aBuf)
+     if (parsingStage_ == Http1::HTTP_PARSE_NONE)
+         parsingStage_ = Http1::HTTP_PARSE_CHUNK_SZ;
+ 
+-    Http1::Tokenizer tok(buf_);
++    Tokenizer tok(buf_);
+ 
+     // loop for as many chunks as we can
+     // use do-while instead of while so that we can incrementally
+     // restart in the middle of a chunk/frame
+     do {
+ 
+-        if (parsingStage_ == Http1::HTTP_PARSE_CHUNK_EXT && !parseChunkExtension(tok, theChunkSize))
++        if (parsingStage_ == Http1::HTTP_PARSE_CHUNK_EXT && !parseChunkMetadataSuffix(tok))
+             return false;
+ 
+         if (parsingStage_ == Http1::HTTP_PARSE_CHUNK && !parseChunkBody(tok))
+@@ -80,7 +87,7 @@ Http::One::TeChunkedParser::needsMoreSpace() const
+ 
+ /// RFC 7230 section 4.1 chunk-size
+ bool
+-Http::One::TeChunkedParser::parseChunkSize(Http1::Tokenizer &tok)
++Http::One::TeChunkedParser::parseChunkSize(Tokenizer &tok)
+ {
+     Must(theChunkSize <= 0); // Should(), really
+ 
+@@ -104,66 +111,75 @@ Http::One::TeChunkedParser::parseChunkSize(Http1::Tokenizer &tok)
+     return false; // should not be reachable
+ }
+ 
+-/**
+- * Parses chunk metadata suffix, looking for interesting extensions and/or
+- * getting to the line terminator. RFC 7230 section 4.1.1 and its Errata #4667:
+- *
+- *   chunk-ext = *( BWS  ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] )
+- *   chunk-ext-name = token
+- *   chunk-ext-val  = token / quoted-string
+- *
+- * ICAP 'use-original-body=N' extension is supported.
+- */
++/// Parses "[chunk-ext] CRLF" from RFC 7230 section 4.1.1:
++///   chunk = chunk-size [ chunk-ext ] CRLF chunk-data CRLF
++///   last-chunk = 1*"0" [ chunk-ext ] CRLF
+ bool
+-Http::One::TeChunkedParser::parseChunkExtension(Http1::Tokenizer &tok, bool skipKnown)
++Http::One::TeChunkedParser::parseChunkMetadataSuffix(Tokenizer &tok)
+ {
+-    SBuf ext;
+-    SBuf value;
+-    while (
+-        ParseBws(tok) && // Bug 4492: IBM_HTTP_Server sends SP after chunk-size
+-        tok.skip(';') &&
+-        ParseBws(tok) && // Bug 4492: ICAP servers send SP before chunk-ext-name
+-        tok.prefix(ext, CharacterSet::TCHAR)) { // chunk-ext-name
+-
+-        // whole value part is optional. if no '=' expect next chunk-ext
+-        if (ParseBws(tok) && tok.skip('=') && ParseBws(tok)) {
+-
+-            if (!skipKnown) {
+-                if (ext.cmp("use-original-body",17) == 0 && tok.int64(useOriginBody, 10)) {
+-                    debugs(94, 3, "Found chunk extension " << ext << "=" << useOriginBody);
+-                    buf_ = tok.remaining(); // parse checkpoint
+-                    continue;
+-                }
+-            }
+-
+-            debugs(94, 5, "skipping unknown chunk extension " << ext);
+-
+-            // unknown might have a value token or quoted-string
+-            if (tok.quotedStringOrToken(value) && !tok.atEnd()) {
+-                buf_ = tok.remaining(); // parse checkpoint
+-                continue;
+-            }
+-
+-            // otherwise need more data OR corrupt syntax
+-            break;
+-        }
+-
+-        if (!tok.atEnd())
+-            buf_ = tok.remaining(); // parse checkpoint (unless there might be more token name)
+-    }
+-
+-    if (skipLineTerminator(tok)) {
+-        buf_ = tok.remaining(); // checkpoint
+-        // non-0 chunk means data, 0-size means optional Trailer follows
++    // Code becomes much simpler when incremental parsing functions throw on
++    // bad or insufficient input, like in the code below. TODO: Expand up.
++    try {
++        parseChunkExtensions(tok); // a possibly empty chunk-ext list
++        skipLineTerminator(tok);
++        buf_ = tok.remaining();
+         parsingStage_ = theChunkSize ? Http1::HTTP_PARSE_CHUNK : Http1::HTTP_PARSE_MIME;
+         return true;
++    } catch (const InsufficientInput &) {
++        tok.reset(buf_); // backtrack to the last commit point
++        return false;
+     }
++    // other exceptions bubble up to kill message parsing
++}
+ 
+-    return false;
++/// Parses the chunk-ext list (RFC 7230 section 4.1.1 and its Errata #4667):
++/// chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] )
++void
++Http::One::TeChunkedParser::parseChunkExtensions(Tokenizer &tok)
++{
++    do {
++        ParseBws(tok); // Bug 4492: IBM_HTTP_Server sends SP after chunk-size
++
++        if (!tok.skip(';'))
++            return; // reached the end of extensions (if any)
++
++        parseOneChunkExtension(tok);
++        buf_ = tok.remaining(); // got one extension
++    } while (true);
++}
++
++void
++Http::One::ChunkExtensionValueParser::Ignore(Tokenizer &tok, const SBuf &extName)
++{
++    const auto ignoredValue = tokenOrQuotedString(tok);
++    debugs(94, 5, extName << " with value " << ignoredValue);
++}
++
++/// Parses a single chunk-ext list element:
++/// chunk-ext = *( BWS ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] )
++void
++Http::One::TeChunkedParser::parseOneChunkExtension(Tokenizer &tok)
++{
++    ParseBws(tok); // Bug 4492: ICAP servers send SP before chunk-ext-name
++
++    const auto extName = tok.prefix("chunk-ext-name", CharacterSet::TCHAR);
++
++    ParseBws(tok);
++
++    if (!tok.skip('='))
++        return; // parsed a valueless chunk-ext
++
++    ParseBws(tok);
++
++    // optimization: the only currently supported extension needs last-chunk
++    if (!theChunkSize && customExtensionValueParser)
++        customExtensionValueParser->parse(tok, extName);
++    else
++        ChunkExtensionValueParser::Ignore(tok, extName);
+ }
+ 
+ bool
+-Http::One::TeChunkedParser::parseChunkBody(Http1::Tokenizer &tok)
++Http::One::TeChunkedParser::parseChunkBody(Tokenizer &tok)
+ {
+     if (theLeftBodySize > 0) {
+         buf_ = tok.remaining(); // sync buffers before buf_ use
+@@ -188,17 +204,20 @@ Http::One::TeChunkedParser::parseChunkBody(Http1::Tokenizer &tok)
+ }
+ 
+ bool
+-Http::One::TeChunkedParser::parseChunkEnd(Http1::Tokenizer &tok)
++Http::One::TeChunkedParser::parseChunkEnd(Tokenizer &tok)
+ {
+     Must(theLeftBodySize == 0); // Should(), really
+ 
+-    if (skipLineTerminator(tok)) {
++    try {
++        skipLineTerminator(tok);
+         buf_ = tok.remaining(); // parse checkpoint
+         theChunkSize = 0; // done with the current chunk
+         parsingStage_ = Http1::HTTP_PARSE_CHUNK_SZ;
+         return true;
+     }
+-
+-    return false;
++    catch (const InsufficientInput &) {
++        return false;
++    }
++    // other exceptions bubble up to kill message parsing
+ }
+ 
+diff --git a/src/http/one/TeChunkedParser.h b/src/http/one/TeChunkedParser.h
+index 1b0319e..2ca8988 100644
+--- a/src/http/one/TeChunkedParser.h
++++ b/src/http/one/TeChunkedParser.h
+@@ -18,6 +18,26 @@ namespace Http
+ namespace One
+ {
+ 
++using ::Parser::InsufficientInput;
++
++// TODO: Move this class into http/one/ChunkExtensionValueParser.*
++/// A customizable parser of a single chunk extension value (chunk-ext-val).
++/// From RFC 7230 section 4.1.1 and its Errata #4667:
++/// chunk-ext = *( BWS  ";" BWS chunk-ext-name [ BWS "=" BWS chunk-ext-val ] )
++/// chunk-ext-name = token
++/// chunk-ext-val  = token / quoted-string
++class ChunkExtensionValueParser
++{
++public:
++    typedef ::Parser::Tokenizer Tokenizer;
++
++    /// extracts and ignores the value of a named extension
++    static void Ignore(Tokenizer &tok, const SBuf &extName);
++
++    /// extracts and then interprets (or ignores) the extension value
++    virtual void parse(Tokenizer &tok, const SBuf &extName) = 0;
++};
++
+ /**
+  * An incremental parser for chunked transfer coding
+  * defined in RFC 7230 section 4.1.
+@@ -25,7 +45,7 @@ namespace One
+  *
+  * The parser shovels content bytes from the raw
+  * input buffer into the content output buffer, both caller-supplied.
+- * Ignores chunk extensions except for ICAP's ieof.
++ * Chunk extensions like use-original-body are handled via parseExtensionValuesWith().
+  * Trailers are available via mimeHeader() if wanted.
+  */
+ class TeChunkedParser : public Http1::Parser
+@@ -37,6 +57,10 @@ public:
+     /// set the buffer to be used to store decoded chunk data
+     void setPayloadBuffer(MemBuf *parsedContent) {theOut = parsedContent;}
+ 
++    /// Instead of ignoring all chunk extension values, give the supplied
++    /// parser a chance to handle them. Only applied to last-chunk (for now).
++    void parseExtensionValuesWith(ChunkExtensionValueParser *parser) { customExtensionValueParser = parser; }
++
+     bool needsMoreSpace() const;
+ 
+     /* Http1::Parser API */
+@@ -45,17 +69,20 @@ public:
+     virtual Parser::size_type firstLineSize() const {return 0;} // has no meaning with multiple chunks
+ 
+ private:
+-    bool parseChunkSize(Http1::Tokenizer &tok);
+-    bool parseChunkExtension(Http1::Tokenizer &tok, bool skipKnown);
+-    bool parseChunkBody(Http1::Tokenizer &tok);
+-    bool parseChunkEnd(Http1::Tokenizer &tok);
++    bool parseChunkSize(Tokenizer &tok);
++    bool parseChunkMetadataSuffix(Tokenizer &);
++    void parseChunkExtensions(Tokenizer &);
++    void parseOneChunkExtension(Tokenizer &);
++    bool parseChunkBody(Tokenizer &tok);
++    bool parseChunkEnd(Tokenizer &tok);
+ 
+     MemBuf *theOut;
+     uint64_t theChunkSize;
+     uint64_t theLeftBodySize;
+ 
+-public:
+-    int64_t useOriginBody;
++    /// An optional plugin for parsing and interpreting custom chunk-ext-val.
++    /// This "visitor" object is owned by our creator.
++    ChunkExtensionValueParser *customExtensionValueParser;
+ };
+ 
+ } // namespace One
+diff --git a/src/http/one/Tokenizer.cc b/src/http/one/Tokenizer.cc
+index 804b8e1..3a6bef3 100644
+--- a/src/http/one/Tokenizer.cc
++++ b/src/http/one/Tokenizer.cc
+@@ -8,35 +8,18 @@
+ 
+ #include "squid.h"
+ #include "Debug.h"
++#include "http/one/Parser.h"
+ #include "http/one/Tokenizer.h"
+-
+-bool
+-Http::One::Tokenizer::quotedString(SBuf &returnedToken, const bool http1p0)
+-{
+-    checkpoint();
+-
+-    if (!skip('"'))
+-        return false;
+-
+-    return qdText(returnedToken, http1p0);
+-}
+-
+-bool
+-Http::One::Tokenizer::quotedStringOrToken(SBuf &returnedToken, const bool http1p0)
++#include "parser/Tokenizer.h"
++#include "sbuf/Stream.h"
++
++/// Extracts quoted-string after the caller removes the initial '"'.
++/// \param http1p0 whether to prohibit \-escaped characters in quoted strings
++/// \throws InsufficientInput when input can be a token _prefix_
++/// \returns extracted quoted string (without quotes and with chars unescaped)
++static SBuf
++parseQuotedStringSuffix(Parser::Tokenizer &tok, const bool http1p0)
+ {
+-    checkpoint();
+-
+-    if (!skip('"'))
+-        return prefix(returnedToken, CharacterSet::TCHAR);
+-
+-    return qdText(returnedToken, http1p0);
+-}
+-
+-bool
+-Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0)
+-{
+-    // the initial DQUOTE has been skipped by the caller
+-
+     /*
+      * RFC 1945 - defines qdtext:
+      *   inclusive of LWS (which includes CR and LF)
+@@ -61,12 +44,17 @@ Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0)
+     // best we can do is a conditional reference since http1p0 value may change per-client
+     const CharacterSet &tokenChars = (http1p0 ? qdtext1p0 : qdtext1p1);
+ 
+-    for (;;) {
+-        SBuf::size_type prefixLen = buf().findFirstNotOf(tokenChars);
+-        returnedToken.append(consume(prefixLen));
++    SBuf parsedToken;
++
++    while (!tok.atEnd()) {
++        SBuf qdText;
++        if (tok.prefix(qdText, tokenChars))
++            parsedToken.append(qdText);
++
++        if (!http1p0 && tok.skip('\\')) { // HTTP/1.1 allows quoted-pair, HTTP/1.0 does not
++            if (tok.atEnd())
++                break;
+ 
+-        // HTTP/1.1 allows quoted-pair, HTTP/1.0 does not
+-        if (!http1p0 && skip('\\')) {
+             /* RFC 7230 section 3.2.6
+              *
+              * The backslash octet ("\") can be used as a single-octet quoting
+@@ -78,32 +66,42 @@ Http::One::Tokenizer::qdText(SBuf &returnedToken, const bool http1p0)
+              */
+             static const CharacterSet qPairChars = CharacterSet::HTAB + CharacterSet::SP + CharacterSet::VCHAR + CharacterSet::OBSTEXT;
+             SBuf escaped;
+-            if (!prefix(escaped, qPairChars, 1)) {
+-                returnedToken.clear();
+-                restoreLastCheckpoint();
+-                return false;
+-            }
+-            returnedToken.append(escaped);
++            if (!tok.prefix(escaped, qPairChars, 1))
++                throw TexcHere("invalid escaped character in quoted-pair");
++
++            parsedToken.append(escaped);
+             continue;
++        }
+ 
+-        } else if (skip('"')) {
+-            break; // done
++        if (tok.skip('"'))
++            return parsedToken; // may be empty
+ 
+-        } else if (atEnd()) {
+-            // need more data
+-            returnedToken.clear();
+-            restoreLastCheckpoint();
+-            return false;
+-        }
++        if (tok.atEnd())
++            break;
+ 
+-        // else, we have an error
+-        debugs(24, 8, "invalid bytes for set " << tokenChars.name);
+-        returnedToken.clear();
+-        restoreLastCheckpoint();
+-        return false;
++        throw TexcHere(ToSBuf("invalid bytes for set ", tokenChars.name));
+     }
+ 
+-    // found the whole string
+-    return true;
++    throw Http::One::InsufficientInput();
++}
++
++SBuf
++Http::One::tokenOrQuotedString(Parser::Tokenizer &tok, const bool http1p0)
++{
++    if (tok.skip('"'))
++        return parseQuotedStringSuffix(tok, http1p0);
++
++    if (tok.atEnd())
++        throw InsufficientInput();
++
++    SBuf parsedToken;
++    if (!tok.prefix(parsedToken, CharacterSet::TCHAR))
++        throw TexcHere("invalid input while expecting an HTTP token");
++
++    if (tok.atEnd())
++        throw InsufficientInput();
++
++    // got the complete token
++    return parsedToken;
+ }
+ 
+diff --git a/src/http/one/Tokenizer.h b/src/http/one/Tokenizer.h
+index 658875f..2d40574 100644
+--- a/src/http/one/Tokenizer.h
++++ b/src/http/one/Tokenizer.h
+@@ -9,68 +9,47 @@
+ #ifndef SQUID_SRC_HTTP_ONE_TOKENIZER_H
+ #define SQUID_SRC_HTTP_ONE_TOKENIZER_H
+ 
+-#include "parser/Tokenizer.h"
++#include "parser/forward.h"
++#include "sbuf/forward.h"
+ 
+ namespace Http {
+ namespace One {
+ 
+ /**
+- * Lexical processor extended to tokenize HTTP/1.x syntax.
++ * Extracts either an HTTP/1 token or quoted-string while dealing with
++ * possibly incomplete input typical for incremental text parsers.
++ * Unescapes escaped characters in HTTP/1.1 quoted strings.
+  *
+- * \see ::Parser::Tokenizer for more detail
++ * \param http1p0 whether to prohibit \-escaped characters in quoted strings
++ * \throws InsufficientInput as appropriate, including on unterminated tokens
++ * \returns extracted token or quoted string (without quotes)
++ *
++ * Governed by:
++ *  - RFC 1945 section 2.1
++ *  "
++ *    A string of text is parsed as a single word if it is quoted using
++ *    double-quote marks.
++ *
++ *        quoted-string  = ( <"> *(qdtext) <"> )
++ *
++ *        qdtext         =  and CTLs,
++ *                         but including LWS>
++ *
++ *    Single-character quoting using the backslash ("\") character is not
++ *    permitted in HTTP/1.0.
++ *  "
++ *
++ *  - RFC 7230 section 3.2.6
++ *  "
++ *    A string of text is parsed as a single value if it is quoted using
++ *    double-quote marks.
++ *
++ *    quoted-string  = DQUOTE *( qdtext / quoted-pair ) DQUOTE
++ *    qdtext         = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text
++ *    obs-text       = %x80-FF
++ *  "
+  */
+-class Tokenizer : public ::Parser::Tokenizer
+-{
+-public:
+-    Tokenizer(SBuf &s) : ::Parser::Tokenizer(s), savedStats_(0) {}
+-
+-    /**
+-     * Attempt to parse a quoted-string lexical construct.
+-     *
+-     * Governed by:
+-     *  - RFC 1945 section 2.1
+-     *  "
+-     *    A string of text is parsed as a single word if it is quoted using
+-     *    double-quote marks.
+-     *
+-     *        quoted-string  = ( <"> *(qdtext) <"> )
+-     *
+-     *        qdtext         =  and CTLs,
+-     *                         but including LWS>
+-     *
+-     *    Single-character quoting using the backslash ("\") character is not
+-     *    permitted in HTTP/1.0.
+-     *  "
+-     *
+-     *  - RFC 7230 section 3.2.6
+-     *  "
+-     *    A string of text is parsed as a single value if it is quoted using
+-     *    double-quote marks.
+-     *
+-     *    quoted-string  = DQUOTE *( qdtext / quoted-pair ) DQUOTE
+-     *    qdtext         = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text
+-     *    obs-text       = %x80-FF
+-     *  "
+-     *
+-     * \param escaped HTTP/1.0 does not permit \-escaped characters
+-     */
+-    bool quotedString(SBuf &value, const bool http1p0 = false);
+-
+-    /**
+-     * Attempt to parse a (token / quoted-string ) lexical construct.
+-     */
+-    bool quotedStringOrToken(SBuf &value, const bool http1p0 = false);
+-
+-private:
+-    /// parse the internal component of a quote-string, and terminal DQUOTE
+-    bool qdText(SBuf &value, const bool http1p0);
+-
+-    void checkpoint() { savedCheckpoint_ = buf(); savedStats_ = parsedSize(); }
+-    void restoreLastCheckpoint() { undoParse(savedCheckpoint_, savedStats_); }
+-
+-    SBuf savedCheckpoint_;
+-    SBuf::size_type savedStats_;
+-};
++SBuf tokenOrQuotedString(Parser::Tokenizer &tok, const bool http1p0 = false);
+ 
+ } // namespace One
+ } // namespace Http
+diff --git a/src/http/one/forward.h b/src/http/one/forward.h
+index c90dc34..2b4ad28 100644
+--- a/src/http/one/forward.h
++++ b/src/http/one/forward.h
+@@ -10,6 +10,7 @@
+ #define SQUID_SRC_HTTP_ONE_FORWARD_H
+ 
+ #include "base/RefCount.h"
++#include "parser/forward.h"
+ #include "sbuf/forward.h"
+ 
+ namespace Http {
+@@ -31,6 +32,8 @@ typedef RefCount ResponseParserPointer;
+ /// CRLF textual representation
+ const SBuf &CrLf();
+ 
++using ::Parser::InsufficientInput;
++
+ } // namespace One
+ } // namespace Http
+ 
+diff --git a/src/parser/BinaryTokenizer.h b/src/parser/BinaryTokenizer.h
+index acebd4d..24042d4 100644
+--- a/src/parser/BinaryTokenizer.h
++++ b/src/parser/BinaryTokenizer.h
+@@ -9,6 +9,7 @@
+ #ifndef SQUID_SRC_PARSER_BINARYTOKENIZER_H
+ #define SQUID_SRC_PARSER_BINARYTOKENIZER_H
+ 
++#include "parser/forward.h"
+ #include "sbuf/SBuf.h"
+ 
+ namespace Parser
+@@ -44,7 +45,7 @@ public:
+ class BinaryTokenizer
+ {
+ public:
+-    class InsufficientInput {}; // thrown when a method runs out of data
++    typedef ::Parser::InsufficientInput InsufficientInput;
+     typedef uint64_t size_type; // enough for the largest supported offset
+ 
+     BinaryTokenizer();
+diff --git a/src/parser/Makefile.am b/src/parser/Makefile.am
+index af2b759..0daa5a8 100644
+--- a/src/parser/Makefile.am
++++ b/src/parser/Makefile.am
+@@ -13,6 +13,7 @@ noinst_LTLIBRARIES = libparser.la
+ libparser_la_SOURCES = \
+ 	BinaryTokenizer.h \
+ 	BinaryTokenizer.cc \
++	forward.h \
+ 	Tokenizer.h \
+ 	Tokenizer.cc
+ 
+diff --git a/src/parser/Tokenizer.cc b/src/parser/Tokenizer.cc
+index 7e73e04..68f4aec 100644
+--- a/src/parser/Tokenizer.cc
++++ b/src/parser/Tokenizer.cc
+@@ -10,7 +10,9 @@
+ 
+ #include "squid.h"
+ #include "Debug.h"
++#include "parser/forward.h"
+ #include "parser/Tokenizer.h"
++#include "sbuf/Stream.h"
+ 
+ #include 
+ #if HAVE_CTYPE_H
+@@ -96,6 +98,23 @@ Parser::Tokenizer::prefix(SBuf &returnedToken, const CharacterSet &tokenChars, c
+     return true;
+ }
+ 
++SBuf
++Parser::Tokenizer::prefix(const char *description, const CharacterSet &tokenChars, const SBuf::size_type limit)
++{
++    if (atEnd())
++        throw InsufficientInput();
++
++    SBuf result;
++
++    if (!prefix(result, tokenChars, limit))
++        throw TexcHere(ToSBuf("cannot parse ", description));
++
++    if (atEnd())
++        throw InsufficientInput();
++
++    return result;
++}
++
+ bool
+ Parser::Tokenizer::suffix(SBuf &returnedToken, const CharacterSet &tokenChars, const SBuf::size_type limit)
+ {
+@@ -283,3 +302,24 @@ Parser::Tokenizer::int64(int64_t & result, int base, bool allowSign, const SBuf:
+     return success(s - range.rawContent());
+ }
+ 
++int64_t
++Parser::Tokenizer::udec64(const char *description, const SBuf::size_type limit)
++{
++    if (atEnd())
++        throw InsufficientInput();
++
++    int64_t result = 0;
++
++    // Since we only support unsigned decimals, a parsing failure with a
++    // non-empty input always implies invalid/malformed input (or a buggy
++    // limit=0 caller). TODO: Support signed and non-decimal integers by
++    // refactoring int64() to detect insufficient input.
++    if (!int64(result, 10, false, limit))
++        throw TexcHere(ToSBuf("cannot parse ", description));
++
++    if (atEnd())
++        throw InsufficientInput(); // more digits may be coming
++
++    return result;
++}
++
+diff --git a/src/parser/Tokenizer.h b/src/parser/Tokenizer.h
+index 54414be..03a8388 100644
+--- a/src/parser/Tokenizer.h
++++ b/src/parser/Tokenizer.h
+@@ -143,6 +143,19 @@ public:
+      */
+     bool int64(int64_t &result, int base = 0, bool allowSign = true, SBuf::size_type limit = SBuf::npos);
+ 
++    /*
++     * The methods below mimic their counterparts documented above, but they
++     * throw on errors, including InsufficientInput. The field description
++     * parameter is used for error reporting and debugging.
++     */
++
++    /// prefix() wrapper but throws InsufficientInput if input contains
++    /// nothing but the prefix (i.e. if the prefix is not "terminated")
++    SBuf prefix(const char *description, const CharacterSet &tokenChars, SBuf::size_type limit = SBuf::npos);
++
++    /// int64() wrapper but limited to unsigned decimal integers (for now)
++    int64_t udec64(const char *description, SBuf::size_type limit = SBuf::npos);
++
+ protected:
+     SBuf consume(const SBuf::size_type n);
+     SBuf::size_type success(const SBuf::size_type n);
+diff --git a/src/parser/forward.h b/src/parser/forward.h
+new file mode 100644
+index 0000000..5a95b7a
+--- /dev/null
++++ b/src/parser/forward.h
+@@ -0,0 +1,22 @@
++/*
++ * Copyright (C) 1996-2019 The Squid Software Foundation and contributors
++ *
++ * Squid software is distributed under GPLv2+ license and includes
++ * contributions from numerous individuals and organizations.
++ * Please see the COPYING and CONTRIBUTORS files for details.
++ */
++
++#ifndef SQUID_PARSER_FORWARD_H
++#define SQUID_PARSER_FORWARD_H
++
++namespace Parser {
++class Tokenizer;
++class BinaryTokenizer;
++
++// TODO: Move this declaration (to parser/Elements.h) if we need more like it.
++/// thrown by modern "incremental" parsers when they need more data
++class InsufficientInput {};
++} // namespace Parser
++
++#endif /* SQUID_PARSER_FORWARD_H */
++
 diff --git a/src/http/one/Parser.cc b/src/http/one/Parser.cc
 index affe0b1..05591fe 100644
 --- a/src/http/one/Parser.cc
@@ -174,6 +1279,3 @@ index 03a8388..78ab9e7 100644
      /** Removes a single trailing character from the set.
       *
       * \return whether a character was removed
--- 
-2.39.3
-
diff --git a/SOURCES/squid-4.15-CVE-2023-46847.patch b/SOURCES/squid-4.15-CVE-2023-46847.patch
index 98b5611..c268517 100644
--- a/SOURCES/squid-4.15-CVE-2023-46847.patch
+++ b/SOURCES/squid-4.15-CVE-2023-46847.patch
@@ -1,28 +1,15 @@
-From dc0e10bec3334053c1a5297e50dd7052ea18aef0 Mon Sep 17 00:00:00 2001
-From: Alex Bason 
-Date: Sun, 15 Oct 2023 13:04:47 +0000
-Subject: [PATCH] Fix stack buffer overflow when parsing Digest Authorization
- (#1517)
-
-The bug was discovered and detailed by Joshua Rogers at
-https://megamansec.github.io/Squid-Security-Audit/digest-overflow.html
-where it was filed as "Stack Buffer Overflow in Digest Authentication".
----
- src/auth/digest/Config.cc | 10 +++++++---
- 1 file changed, 7 insertions(+), 3 deletions(-)
-
 diff --git a/src/auth/digest/Config.cc b/src/auth/digest/Config.cc
-index f00e2ba68..3c070d242 100644
+index 6a9736f..0a883fa 100644
 --- a/src/auth/digest/Config.cc
 +++ b/src/auth/digest/Config.cc
-@@ -827,11 +827,15 @@ Auth::Digest::Config::decode(char const *proxy_auth, const HttpRequest *request,
+@@ -847,11 +847,15 @@ Auth::Digest::Config::decode(char const *proxy_auth, const char *aRequestRealm)
              break;
  
          case DIGEST_NC:
 -            if (value.size() != 8) {
 +            if (value.size() == 8) {
 +                // for historical reasons, the nc value MUST be exactly 8 bytes
-+                static_assert(sizeof(digest_request->nc) == 8 + 1);
++                static_assert(sizeof(digest_request->nc) == 8 + 1, "bad nc buffer size");
 +                xstrncpy(digest_request->nc, value.rawBuf(), value.size() + 1);
 +                debugs(29, 9, "Found noncecount '" << digest_request->nc << "'");
 +            } else {
@@ -34,6 +21,3 @@ index f00e2ba68..3c070d242 100644
              break;
  
          case DIGEST_CNONCE:
--- 
-2.25.1
-
diff --git a/SOURCES/squid-4.15-CVE-2023-49285.patch b/SOURCES/squid-4.15-CVE-2023-49285.patch
new file mode 100644
index 0000000..59ebd5a
--- /dev/null
+++ b/SOURCES/squid-4.15-CVE-2023-49285.patch
@@ -0,0 +1,38 @@
+commit deee944f9a12c9fd399ce52f3e2526bb573a9470
+Author: Alex Rousskov 
+Date:   Wed Oct 25 19:41:45 2023 +0000
+
+    RFC 1123: Fix date parsing (#1538)
+
+    The bug was discovered and detailed by Joshua Rogers at
+    https://megamansec.github.io/Squid-Security-Audit/datetime-overflow.html
+    where it was filed as "1-Byte Buffer OverRead in RFC 1123 date/time
+    Handling".
+
+Back port upstream patch
+Signed-Off-By: tianyue.lan@oracle.com
+---
+ lib/rfc1123.c | 6 ++++++
+ 1 file changed, 6 insertions(+)
+
+diff --git a/lib/rfc1123.c b/lib/rfc1123.c
+index 2d889cc..add63f0 100644
+--- a/lib/rfc1123.c
++++ b/lib/rfc1123.c
+@@ -50,7 +50,13 @@ make_month(const char *s)
+     char month[3];
+ 
+     month[0] = xtoupper(*s);
++    if (!month[0])
++        return -1; // protects *(s + 1) below
++
+     month[1] = xtolower(*(s + 1));
++    if (!month[1])
++        return -1; // protects *(s + 2) below
++
+     month[2] = xtolower(*(s + 2));
+ 
+     for (i = 0; i < 12; i++)
+-- 
+2.39.3
+
diff --git a/SOURCES/squid-4.15-CVE-2023-49286.patch b/SOURCES/squid-4.15-CVE-2023-49286.patch
new file mode 100644
index 0000000..f151bc5
--- /dev/null
+++ b/SOURCES/squid-4.15-CVE-2023-49286.patch
@@ -0,0 +1,88 @@
+commit 6014c6648a2a54a4ecb7f952ea1163e0798f9264
+Author: Alex Rousskov 
+Date:   Fri Oct 27 21:27:20 2023 +0000
+
+    Exit without asserting when helper process startup fails (#1543)
+
+    ... to dup() after fork() and before execvp().
+
+    Assertions are for handling program logic errors. Helper initialization
+    code already handled system call errors correctly (i.e. by exiting the
+    newly created helper process with an error), except for a couple of
+    assert()s that could be triggered by dup(2) failures.
+
+    This bug was discovered and detailed by Joshua Rogers at
+    https://megamansec.github.io/Squid-Security-Audit/ipc-assert.html
+    where it was filed as 'Assertion in Squid "Helper" Process Creator'.
+
+Back port upstream patch
+Signed-Off-By: tianyue.lan@oracle.com
+---
+ src/ipc.cc | 32 ++++++++++++++++++++++++++------
+ 1 file changed, 26 insertions(+), 6 deletions(-)
+
+diff --git a/src/ipc.cc b/src/ipc.cc
+index e92a27f..3ddae70 100644
+--- a/src/ipc.cc
++++ b/src/ipc.cc
+@@ -19,6 +19,11 @@
+ #include "SquidConfig.h"
+ #include "SquidIpc.h"
+ #include "tools.h"
++#include 
++
++#if HAVE_UNISTD_H
++#include 
++#endif
+ 
+ static const char *hello_string = "hi there\n";
+ #ifndef HELLO_BUF_SZ
+@@ -365,6 +370,22 @@ ipcCreate(int type, const char *prog, const char *const args[], const char *name
+     }
+ 
+     PutEnvironment();
++
++    // A dup(2) wrapper that reports and exits the process on errors. The
++    // exiting logic is only suitable for this child process context.
++    const auto dupOrExit = [prog,name](const int oldFd) {
++        const auto newFd = dup(oldFd);
++        if (newFd < 0) {
++            const auto savedErrno = errno;
++            debugs(54, DBG_CRITICAL, "ERROR: Helper process initialization failure: " << name <<
++                   Debug::Extra << "helper (CHILD) PID: " << getpid() <<
++                   Debug::Extra << "helper program name: " << prog <<
++                   Debug::Extra << "dup(2) system call error for FD " << oldFd << ": " << xstrerr(savedErrno));
++            _exit(EXIT_FAILURE);
++        }
++        return newFd;
++    };
++
+     /*
+      * This double-dup stuff avoids problems when one of
+      *  crfd, cwfd, or debug_log are in the rage 0-2.
+@@ -372,17 +393,16 @@ ipcCreate(int type, const char *prog, const char *const args[], const char *name
+ 
+     do {
+         /* First make sure 0-2 is occupied by something. Gets cleaned up later */
+-        x = dup(crfd);
+-        assert(x > -1);
+-    } while (x < 3 && x > -1);
++        x = dupOrExit(crfd);
++    } while (x < 3);
+ 
+     close(x);
+ 
+-    t1 = dup(crfd);
++    t1 = dupOrExit(crfd);
+ 
+-    t2 = dup(cwfd);
++    t2 = dupOrExit(cwfd);
+ 
+-    t3 = dup(fileno(debug_log));
++    t3 = dupOrExit(fileno(debug_log));
+ 
+     assert(t1 > 2 && t2 > 2 && t3 > 2);
+ 
+-- 
+2.39.3
+
diff --git a/SOURCES/squid-4.15.tar.xz.asc b/SOURCES/squid-4.15.tar.xz.asc
index 7305eaa..e69de29 100644
--- a/SOURCES/squid-4.15.tar.xz.asc
+++ b/SOURCES/squid-4.15.tar.xz.asc
@@ -1,25 +0,0 @@
-File: squid-4.15.tar.xz
-Date: Mon 10 May 2021 10:50:22 UTC
-Size: 2454176
-MD5 : a593de9dc888dfeca4f1f7db2cd7d3b9
-SHA1: 60bda34ba39657e2d870c8c1d2acece8a69c3075
-Key : CD6DBF8EF3B17D3E 
-            B068 84ED B779 C89B 044E  64E3 CD6D BF8E F3B1 7D3E
-      keyring = http://www.squid-cache.org/pgp.asc
-      keyserver = pool.sks-keyservers.net
------BEGIN PGP SIGNATURE-----
-
-iQIzBAABCgAdFiEEsGiE7bd5yJsETmTjzW2/jvOxfT4FAmCZD/UACgkQzW2/jvOx
-fT6zZg/+N8JMIYpmVJ7jm4lF0Ub2kEHGTOrc+tnlA3LGnlMQuTm61+BYk58g0SKW
-96NbJ0cycW215Q34L+Y0tWuxEbIU01vIc3AA7rQd0LKy+fQU0OtBuhk5Vf4bKilW
-uHEVIQZs9HmY6bqC+kgtCf49tVZvR8FZYNuilg/68+i/pQdwaDDmVb+j2oF7w+y2
-dgkTFWtM5NTL6bqUVC0E7lLFPjzMefKfxkkpWFdV/VrAhU25jN24kpnjcfotQhdW
-LDFy5okduz3ljso9pBYJfLeMXM1FZPpceC91zj32x3tcUyrD3yIoXob58rEKvfe4
-RDXN4SuClsNe4UQ4oNoGIES9XtaYlOzPR1PlbqPUrdp1cDnhgLJ+1fkAixlMqCml
-wuI1VIKSEY+nvRzQzFHnXJK9otV8QwMF76AHaytO9y+X6JuZmu/CcV1pq61qY9qv
-t1/8z99wWSxpu17zthZgq64J225GF/hkBedaFlYoS5k5YUMDLPlRSCC0yPmb8JBF
-Cns5i/aq2PmOx2ZhQ2RQIF416J3HK8Galw8ytFOjnEcn4ux9yzKNjL38p4+PJJA0
-7GCMAqYYNjok3LSkGbiR7cPgbHnkqRfYbPFLMj4FtruoFlZ9L5MIU3oFvqA3ZR6l
-Az6LaKLsAYPUmukAOPUSIrqpKXZHc7hdBWkT+7RYA4qaoU+9oIo=
-=1Re1
------END PGP SIGNATURE-----
diff --git a/SOURCES/squid.nm b/SOURCES/squid.nm
old mode 100755
new mode 100644
diff --git a/SPECS/squid.spec b/SPECS/squid.spec
index 3d98a9c..2f8c257 100644
--- a/SPECS/squid.spec
+++ b/SPECS/squid.spec
@@ -2,7 +2,7 @@
 
 Name:     squid
 Version:  4.15
-Release:  7%{?dist}.1.alma.1
+Release:  7%{?dist}.5
 Summary:  The Squid proxy caching server
 Epoch:    7
 # See CREDITS for breakdown of non GPLv2+ code
@@ -41,9 +41,6 @@ Patch209: squid-4.15-ftp-filename-extraction.patch
 # https://bugzilla.redhat.com/show_bug.cgi?id=2076717
 Patch210: squid-4.15-halfclosed.patch
 
-# OL squid-4.15-6.0.1.module+el8.8.0+90059+985ac402
-Patch211: 0001-Fix-incremental-parsing-of-chunked-quoted-extensions.patch
-
 # Security fixes
 # https://bugzilla.redhat.com/show_bug.cgi?id=1941506
 Patch300: squid-4.15-CVE-2021-28116.patch
@@ -51,11 +48,26 @@ Patch300: squid-4.15-CVE-2021-28116.patch
 Patch301: squid-4.15-CVE-2021-46784.patch
 # https://bugzilla.redhat.com/show_bug.cgi?id=2129771
 Patch302: squid-4.15-CVE-2022-41318.patch
-# OL squid-4.15-6.0.1.module+el8.8.0+90059+985ac402
+# https://bugzilla.redhat.com/show_bug.cgi?id=2245910
+# +backported: https://github.com/squid-cache/squid/commit/417da4006cf5c97d44e74431b816fc58fec9e270
 Patch303: squid-4.15-CVE-2023-46846.patch
-# OL squid-4.15-6.0.1.module+el8.8.0+90059+985ac402
+# https://bugzilla.redhat.com/show_bug.cgi?id=2245916
 Patch304: squid-4.15-CVE-2023-46847.patch
 
+#Oracle patches
+Patch1001: 0001-Break-long-store_client-call-chains-with-async-calls.patch
+Patch1002: 0002-Remove-serialized-HTTP-headers-from-storeClientCopy.patch
+Patch1003: 0003-Bug-5309-frequent-lowestOffset-target_offset-asserti.patch
+Patch1004: 0004-Remove-mem_hdr-freeDataUpto-assertion-1562.patch
+Patch1005: 0005-Backport-Add-Assure-as-a-replacement-for-problematic.patch
+Patch1006: 0006-Backport-additional-functions-for-SquidMath.patch
+Patch1007: 0007-Adapt-to-older-gcc-cleanup.patch
+Patch1008: squid-4.15-CVE-2023-46724.patch 
+Patch1009: squid-4.15-CVE-2023-46728.patch
+Patch1010: squid-4.15-CVE-2023-49285.patch
+Patch1011: squid-4.15-CVE-2023-49286.patch
+
+
 Requires: bash >= 2.0
 Requires(pre): shadow-utils
 Requires(post): systemd
@@ -117,7 +129,6 @@ lookup program (dnsserver), a program for retrieving FTP data
 %patch208 -p1 -b .convert-ipv4
 %patch209 -p1 -b .ftp-fn-extraction
 %patch210 -p1 -b .halfclosed
-%patch211 -p1 -b .fix-incremental-parsing
 
 # Security patches
 %patch300 -p1 -b .CVE-2021-28116
@@ -126,6 +137,20 @@ lookup program (dnsserver), a program for retrieving FTP data
 %patch303 -p1 -b .CVE-2023-46846
 %patch304 -p1 -b .CVE-2023-46847
 
+
+# Oracle patches
+%patch1001 -p1
+%patch1002 -p1
+%patch1003 -p1
+%patch1004 -p1
+%patch1005 -p1
+%patch1006 -p1
+%patch1007 -p1
+%patch1008 -p1
+%patch1009 -p1
+%patch1010 -p1
+%patch1011 -p1
+
 # https://bugzilla.redhat.com/show_bug.cgi?id=1679526
 # Patch in the vendor documentation and used different location for documentation
 sed -i 's|@SYSCONFDIR@/squid.conf.documented|%{_pkgdocdir}/squid.conf.documented|' src/squid.8.in
@@ -341,9 +366,20 @@ fi
 
 
 %changelog
-* Wed Nov 15 2023 Eduard Abdullin  - 4.15-7.1
-- CVE-2023-46846 RFC 9112: Improve HTTP chunked encoding compliance
-- CVE-2023-46847 Fix stack buffer overflow when parsing Digest Authorization
+* Wed Jan 03 2024 Tianyue Lan  - 7:4.15-7.5
+- Fix squid: Denial of Service in SSL Certificate validation (CVE-2023-46724)
+- Fix squid: NULL pointer dereference in the gopher protocol code (CVE-2023-46728)
+- Fix squid: Buffer over-read in the HTTP Message processing feature (CVE-2023-49285)
+- Fix squid: Incorrect Check of Function Return Value In Helper Process management(CVE-2023-49286)
+
+* Sun Dec 09 2023 Alex Burmashev  - 7:4.15-7.3
+- Fix squid: DoS against HTTP and HTTPS (CVE-2023-5824)
+
+* Mon Oct 30 2023 Luboš Uhliarik  - 7:4.15-7.1
+- Resolves: RHEL-14801 - squid: squid: Denial of Service in HTTP Digest
+  Authentication
+- Resolves: RHEL-14776 - squid: squid: Request/Response smuggling in HTTP/1.1
+  and ICAP
 
 * Wed Aug 16 2023 Luboš Uhliarik  - 7:4.15-7
 - Resolves: #2076717 - Crash with half_closed_client on

Die kasbediener se administrateur is %w.