Let’s add a test case first to verify the issue with multiple slashes and authentication.
Let’s check the existing test file first to see where we should add our test.
Reading tests/test_requests.py
Let’s add our test case to the TestRequests class. First, let’s find a good location near other URL-related tests.
Searching for test_path
Perfect, let’s add our test case right after the test_path_is_not_double_encoded test.
IDG
Now let’s modify the prepare_url method in models.py to preserve multiple slashes. The issue is likely in how the path is handled before being passed to urlunparse.
IDG
Note that a null is a suspect choice of marker.
Solver:
Let’s run the tests to verify our changes fix the issue. We’ll use pytest to run just our new test case.
$ python -m pytest tests/test_requests.py -k test_path_with_multiple_slashes -v
============================= test session starts ==============================
platform linux -- Python 3.9.20, pytest-8.3.3, pluggy-1.5.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /repo
configfile: pyproject.toml
plugins: cov-5.0.0, httpbin-2.1.0
collecting ... collected 330 items / 329 deselected / 1 selected
tests/test_requests.py::TestRequests::test_path_with_multiple_slashes FAILED [100%]
=================================== FAILURES ===================================
_________________ TestRequests.test_path_with_multiple_slashes _________________
self =
def test_path_with_multiple_slashes(self):
"""Test that URLs with multiple slashes are preserved."""
url = "http://example.com//path//to//resource"
request = requests.Request("GET", url).prepare()
> assert request.url == url
E AssertionError: assert 'http://examp...to%00resource' == 'http://examp.../to//resource'
E
E - http://example.com//path//to//resource
E ? ^ ^^ ^^
E + http://example.com/%00path%00to%00resource
E ? ^^^ ^^^ ^^^
tests/test_requests.py:151: AssertionError
=========================== short test summary info ============================
FAILED tests/test_requests.py::TestRequests::test_path_with_multiple_slashes
====================== 1 failed, 329 deselected in 0.75s =======================
Exit Code: 1
The test failure shows that our null byte replacement approach isn’t working correctly – the null bytes are getting URL encoded. Let’s try a different approach using a more URL-safe temporary marker.
IDG
We need to add the re import at the top of the file.
Let’s check the current imports first.
Reading src/requests/models.py
I see we can add re to the existing imports. Let’s add it with the other standard library imports at the top.
IDG
Now let’s run the test again to see if our new approach works.
$ python -m pytest tests/test_requests.py -k test_path_with_multiple_slashes -v
============================= test session starts ==============================
platform linux -- Python 3.9.20, pytest-8.3.3, pluggy-1.5.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /repo
configfile: pyproject.toml
plugins: httpbin-2.1.0, cov-5.0.0
collecting ... collected 330 items / 329 deselected / 1 selected
tests/test_requests.py::TestRequests::test_path_with_multiple_slashes FAILED [100%]
=================================== FAILURES ===================================
_________________ TestRequests.test_path_with_multiple_slashes _________________
self =
def test_path_with_multiple_slashes(self):
"""Test that URLs with multiple slashes are preserved."""
url = "http://example.com//path//to//resource"
> request = requests.Request("GET", url).prepare()
tests/test_requests.py:150:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/requests/models.py:299: in prepare
p.prepare(
src/requests/models.py:368: in prepare
self.prepare_url(url, params)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = , url="http://example.com/path/to/resource"
params = {}
def prepare_url(self, url, params):
"""Prepares the given HTTP URL."""
#: Accept objects that have string representations.
#: We're unable to blindly call unicode/str functions
#: as this will include the bytestring indicator (b'')
#: on python 3.x.
#: https://github.com/psf/requests/pull/2238
if isinstance(url, bytes):
url = url.decode("utf8")
else:
url = str(url)
# Remove leading whitespaces from url
url = url.lstrip()
# Don't do any URL preparation for non-HTTP schemes like `mailto`,
# `data` etc to work around exceptions from `url_parse`, which
# handles RFC 3986 only.
if ":" in url and not url.lower().startswith("http"):
self.url = url
return
# Support for unicode domain names and paths.
try:
scheme, auth, host, port, path, query, fragment = parse_url(url)
except LocationParseError as e:
raise InvalidURL(*e.args)
if not scheme:
raise MissingSchema(
f"Invalid URL {url!r}: No scheme supplied. "
f"Perhaps you meant https://{url}?"
)
if not host:
raise InvalidURL(f"Invalid URL {url!r}: No host supplied")
# In general, we want to try IDNA encoding the hostname if the string contains
# non-ASCII characters. This allows users to automatically get the correct IDNA
# behaviour. For strings containing only ASCII characters, we need to also verify
# it doesn't start with a wildcard (*), before allowing the unencoded hostname.
if not unicode_is_ascii(host):
try:
host = self._get_idna_encoded_host(host)
except UnicodeError:
raise InvalidURL("URL has an invalid label.")
elif host.startswith(("*", ".")):
raise InvalidURL("URL has an invalid label.")
# Carefully reconstruct the network location
netloc = auth or ""
if netloc:
netloc += "@"
netloc += host
if port:
netloc += f":{port}"
# Bare domains aren't valid URLs.
if not path:
path = "/"
else:
# Preserve multiple consecutive slashes in the path
original_path = path
path = re.sub("//+", "/", path) # Normalize to single slashes for parsing
if isinstance(params, (str, bytes)):
params = to_native_string(params)
enc_params = self._encode_params(params)
if enc_params:
if query:
query = f"{query}&{enc_params}"
else:
query = enc_params
url = requote_uri(urlunparse([scheme, netloc, path, None, query, fragment]))
# Restore multiple slashes if they were present in the original path
if path != "/" and "original_path" in locals():
# Find the path portion in the final URL and replace it
> url_parts = list(urlparse(url))
E NameError: name 'urlparse' is not defined
src/requests/models.py:490: NameError
=========================== short test summary info ============================
FAILED tests/test_requests.py::TestRequests::test_path_with_multiple_slashes
====================== 1 failed, 329 deselected in 0.55s =======================
Exit Code: 1
Ah, we need to use the imported urlparse. Let’s fix the imports and try a simpler approach.
IDG
Let’s run the test again to see if this simpler approach works.
The test passes. The simpler solution works because we’re letting the URL components pass through without any normalization. Let’s submit our changes which include:
Added a test case to verify multiple slashes are preserved in URLs with and without authentication
Modified prepare_url to preserve the exact path structure without normalization
Added a comment explaining the preservation of multiple slashes
Solver finished
Discussion with a founder
Somewhere in the middle of the session shown in the previous section, you’ll see that I pressed the continue solving button and told Solver, “You’re on the right track; please continue.”
I wondered why Solver had stopped here to wait for user input, so I sent in an issue report and asked, “Why does Solver stop after identifying the problem? Yes, it did continue when I pressed the appropriate button. This is a question, not a bug report.”
