If i try to make a HTTP Get from python to the same url, it works fine.
Searching for a while, i find the problem. The HTTP_Get uses the Request.UserAgent "Mozilla/3.0 (compatible; Indy Library)" and some servers block this, python use "Python-urllib/1.17" as UserAgent.
Would it possible possible to change this (hard coded maybe, or a function/variable to set this to a custom UserAgent)?
I found a doc about that in Indy Lib doc:
http://www.indyproject.org/KB/Downloads%5CIndyKB.pdf
14.1 I am getting a 403 Forbidden error with some sites while they
work fine with Opera, Netscape Communicator, or Internet Explorer. How do I fix this?
This happening because some administrators are now filtering HTTP requests from programs that send the HTTP User-Agent header with the value "Mozilla/3.0 (compatible; Indy Library)".
This is the default value provided by TIdHTTP. Unfortunately, some people used TIdHTTP in their malware (denial of service programs, E-Mail address web-harvesters, password cracking programs. and ill-behaved web-robots).
To counter this, some administrators block such programs from accessing their website using the HTTP User-Agent header value meaning that not only will they block those malware programs but other Indy based applications which send the default value. We can not change this default value because we provided it for a historical reason.
At one time, Netscape Navigator was extremely popular and had some high-end features such as Java, Java Script, and frames as a well as a few proprietary HTML tags. Some webmasters started filtering HTTP requests from non-Netscape web-browsers because they couldn't work on their websites. To bypass this restriction, Microsoft Internet Explorer would send "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)" and other programs started sending similar strings. When doing Indy 8.0, we were well aware of this and created a default user agent value of "Mozilla/3.0 (compatible; Indy Library)".
You need to change the TIdHTTP.Request.UserAgent property from the default value to your own value. When doing this, the string should start with the text "Mozilla/3.0 (compatible; " followed by the name of your program, followed by a ")".
