urllib2 failing with squid proxy and digest authentication #60299

Activity

1. toobaz commented on Sep 30, 2012

   

   toobazmannequin

   on Sep 30, 2012

   MannequinAuthor

   More actions

   If you run the following code:

   #! /usr/bin/python
   import urllib2
   
   MyHTTPPasswordMgr = urllib2.HTTPPasswordMgr
   proxy = urllib2.ProxyHandler({'http': 'http://proxybiblio2.si.unimib.it:8080'})
   auth = urllib2.ProxyDigestAuthHandler(MyHTTPPasswordMgr())
   auth.add_password(None, "proxybiblio2.si.unimib.it", "a", "b" )
   opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
   urllib2.install_opener(opener)
   conn = urllib2.urlopen('http://webofknowledge.com')

   an "HTTP Error 407: Proxy Authentication Required" is raised, and under the hood here's what's happening:

   • the request is made without authentication
   • the server replies it must be made with digest authentication, and gives the nonce
   • the error is raised.

   Instead, urllib2 should now try to connect with the username and password, and do so up to 5 times (as hardcoded in urllib2.http_error_auth_reqed), and then raise a "HTTP Error 401: digest auth failed". And it's indeed what it does if you replace the line "MyHTTPPasswordMgr = urllib2.HTTPPasswordMgr" with

   class MyHTTPPasswordMgr(urllib2.HTTPPasswordMgr):
       def find_user_password(self, realm, authuri):
           return "a", "b"

   So the problem is in HTTPPasswordMgr, which is apparently unable to match the authentication data with the realm. Some tests¹ suggest that this can vary according to the proxy engine and to the proxy address format (works with apache, but doesn't if then you add "http://" in front of the proxy address).

   This reminds a bit bug 680577, and in particular I noticed that (possibly unrelated) the behaviour reported in the following message:
   http://bugs.python.org/msg14444
   has not changed:

   In [1]: import urllib2

   In [2]: urllib2.HTTPPasswordMgr().is_suburi("/foo/spam", "/foo/eggs")Out[2]: True

   This affects also python 3.2, you can try the following:

   #! /usr/bin/python
   from urllib import request
   MyHTTPPasswordMgr = request.HTTPPasswordMgr
   proxy = request.ProxyHandler({'http': 'http://proxybiblio2.si.unimib.it:8080'})
   auth = request.ProxyDigestAuthHandler(MyHTTPPasswordMgr())
   auth.add_password(None, "proxybiblio2.si.unimib.it", "a", "b" )
   opener = request.build_opener(proxy, auth, request.HTTPHandler)
   request.install_opener(opener)
   conn = request.urlopen('http://webofknowledge.com')

   ¹ http://lists.python.it/pipermail/python/2012-September/013309.html (in Italian)
2. toobaz

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 30, 2012
3. orsenthil

   self-assigned this

   on Sep 30, 2012
4. BreamoreBoy commented on Jul 2, 2014

   

   BreamoreBoymannequin

   on Jul 2, 2014

   Mannequin

   More actions

   Can we have an update on this please.
5. toobaz commented on Jul 2, 2014

   

   toobazmannequin

   on Jul 2, 2014

   MannequinAuthor

   More actions

   The bug is still present in 2.7.7 and 3.4.1.

   By the way, under python 3 the workaround takes the form

   class MyHTTPPasswordMgr(urllib.request.HTTPPasswordMgr):
       def find_user_password(self, realm, authuri):
           return "a", "b"

   Finally, notice the wrong behaviour of "is_suburi()" mentioned in http://bugs.python.org/msg14444 is still present (and I still suspect it has something to do with this).
6. alexwe commented on Sep 15, 2014

   

   alexwemannequin

   on Sep 15, 2014

   Mannequin

   More actions

   So, I analyzed the error and I think I found the problem. (urllib.request - Python 3.5.0)

   It all starts with l. 1079, as the 407 error gets handled, everything ok here, in l. 1081 http_error_auth_reqed(...) gets executed.

   So next, we are in l. 939, also everything correct here, retry_http_digest_auth gets executed in l. 953. (woops, "http_digest_auth"?!, nah, let's see what comes next)

   So we are in l. 953 and follow the code to come to the get_authorization(...) call.

   Now we are in l. 981, and in that part of the code lies the problem.
   To get the username and password for the proxy, the function find_user_password(realm, req.full_url) gets executed.

   An example, if my proxy has the address abc.com:8080 and my request is for xyz.com, the function tries to find a password for the xyz.com url, instead of the abc.com:8080 url. So it can't find a password and the whole auth process stops with the 407 error.

   But if you just change the line, to use the host, normal http digest auth doesn't work anymore, I would suggest?

   So it's also obvious why the workaround of toobaz works.

   --------------------------------------------------------

   To solve the Problem, two auth handler would be needed, one for the proxy and one for normal http auth.
   Two different handlers were used in the basic auth, so I think it would be an consistent solution?
7. vadmium commented on Jun 21, 2015

   

   vadmium

   on Jun 21, 2015

   Member

   More actions

   There are two problems with the test case, and one bug in Python:

   1. HTTPPasswordMgr doesn’t handle realm=None; it has to be a string. You can use HTTPPasswordMgrWithDefaultRealm though.

   2. The password managers won’t match a proxy with a non-standard port number against a hostname without a port. So you have to include the port in the add_password() call.

   3. AbstractDigestAuthHandler.get_authorization() is using the wrong URL, as Alexander already discovered. I made proxy-digest.patch which should fix this.

   The wrong URL (final URL rather than proxy) was actually tested for in test_urllib2_localnet.ProxyAuthTests. So I fixed those tests.
8. vadmium commented on Jun 21, 2015

   

   vadmium

   on Jun 21, 2015

   Member

   More actions

   I should point out my patch also adds add_password() methods for the authentication handlers to the documentation. These were only documented by example, but everyone seems to prefer using them rather than the equivalent password manager method.
9. ezio-melotti

   transferred this issue fromon Apr 10, 2022
10. serhiy-storchaka

    added this to Urllib issueson Jul 3, 2026
11. github-project-automation

    moved this to Todo in Urllib issueson Jul 3, 2026
12. github-project-automation

    moved this from Todo to In Progress in Urllib issueson Oct 23, 2026
13. orsenthil

    added a commit that references this issue on Oct 23, 2026

    pythongh-60299: Fix ProxyDigestAuthHandler credential lookup to use p…

    ...

    194789e