HTTP POST bugs in http_client

Skip to main content ↑ ↓

Notice

Please note that most of the software linked on this forum is likely to be safe to use. If you are unsure, feel free to ask in the relevant topics, or send a private message to an administrator or moderator. To help curb the problems of false positives, or in the event that you do find actual malware, you can contribute through the article linked here.

Topic: HTTP POST bugs in http_client (Read 3818 times) previous topic - next topic

0 Members and 1 Guest are viewing this topic.

HTTP POST bugs in http_client

2010-08-29 10:33:39

Trying to figure out why foo_softplaylists plugin crashes when communicating with last.fm, I've installed WireShark (a protocol analyzer) and looked into sent/received packets. It appeared that POST requests are broken. Then I've checked behavior of foo_audioscrobbler, and it behaves the same way (except it doesn't crash, showing errors in console instead). So I suppose the problem is within foobar's http_client which AFAIK is used by both plugins.

I've seen 5 POST request patterns, all seem invalid to me.
In all patterns packets are sent immediately one after another in single TCP stream, that is they seem like single HTTP request on server end.

Pattern 1
Request consists of 2 packets.
(I give here only what's inside the TCP layer.)

Packet 1:

<Some header fields>
Content-Length: <X>

<Request body (length = <X> bytes)>

Packet 2: \r\n\r\n (hex: 0d0a0d0a)

The packet 1 is normal, packet 2 is unnecessary. I have not seen any problems appearing after this pattern though. Presumably the server just ignores second packet. But this still is wrong.

Pattern 2
Request consists of 3 packets.
1: "normal" (as packet 1 from pattern 1)
2: "normal" again
3: \r\n\r\n (as packet 2 from pattern 1)

This pattern is as often as first.
The server handles this request normally, response is ok.

Pattern 3
Request consists of 4 packets.
1: "normal"
2: \r\n (hex: 0d0a)
2: "normal" again
3: \r\n\r\n

This pattern is relatively rare.
The server handles this request normally, response is ok.

Pattern 4
Request consists of 3 packets.
Packet 1:

<Some header fields>
Content-Length: <X>

So this is the header from "normal" packet, but without any data.

Packet 2: "normal"
Packet 3: \r\n\r\n

In this case server responds with something like

HTTP/1.0 400 Bad Request
Date: Sat, 28 Aug 2010 18:40:42 GMT
Server: Apache/1.3.39 (Unix)
X-Proxy-Fix-Up: headers fixed up
X-Web-Node: www195
Connection: close
Content-Type: text/xml; charset=utf-8;

<?xml version="1.0" encoding="utf-8"?>
<lfm status="failed">
<error code="3">Invalid Method - No method with that name in this package</error></lfm>

or

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/plain
Content-Length: 11
Date: Sun, 29 Aug 2010 01:21:27 GMT
Connection: close

BADSESSION

Obviously, the server treats header of packet 2 as data for incomplete packet 1, and fails to parse this "data".

Pattern 4
Request consists of 2 packets.
Packet 1: only header with no data (as in pattern 4)
Packet 2: crippled version of "normal" packet, like

Content-Length: 169

st: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded<Sequence of null bytes><Request data>\r\n\r\n

I've seen this pattern only twice. Server response was as with pattern 4.

Another problem is sometimes raised exception with message "Invalid data, received line length exceeds limit". I think it can be caused by connection aborts. Very often immediately after server response comes packet with TCP flag RST (Reset) set, which means connection abort. Although most of the time this is not related with any problems, I have not seen "Invalid data" exception without aborted connection.

All this is related to POST requests only. I haven't seen any problems with GET requests. And there are no connection aborts after GETs.

foobar 1.1, Win 7 x64.
Symptoms are the same for foobar 1.0, although I haven't tested it in such way.

HTTP POST bugs in http_client

Reply #1 – 2010-08-30 10:25:06

As TCP is a stream, there are no logical "packets" as far as an application is concerned. You shovel data in in chunks on one side of a socket and get data out in arbitrary chunks on the other side.
Any packet segmentations you see tend to be purely from the network stack splitting/coalescing individual sends according to timing from the fb2k side.

I don't suppose you happen to have Wireshark dumps of these conversations (assuming no private data happens to be in them).

HTTP POST bugs in http_client

Reply #2 – 2010-08-30 12:12:10

Well, there is that superfluous \r\n part, which is understandable (and fixed now, thank you).

But apart from that, I don't really understand what's happening there, and I have tried to reproduce it.
Could you provide one full capture of a broken HTTP transaction?
Or a sample code to trigger some of the weird behavior, for example submitting nonsense POST data against www.foobar2000.org/index.php or somethhing - the client part is important, right?

Quote from: Arkhon on 2010-08-29 10:33:39

Another problem is sometimes raised exception with message "Invalid data, received line length exceeds limit". I think it can be caused by connection aborts. Very often immediately after server response comes packet with TCP flag RST (Reset) set, which means connection abort. Although most of the time this is not related with any problems, I have not seen "Invalid data" exception without aborted connection.

Yes, this is a bug. "Received line length exceeds limit" is "failed to receive whole header" and vice versa. This has been fixed already, though sadly not before 1.1 release.

HTTP POST bugs in http_client

Reply #3 – 2010-08-30 15:14:15

Yes, packets aren't a TCP conception. But as requests I saw was always organized in recognizable packet patterns, I wrote about packets.
For your convenience I have reformulated the info here, and also give the requests in their natural form.

WireShark dumps.

Pattern 1

<normal request>
\r\n\r\n

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Pattern 2

<normal request>
<normal request>
\r\n\r\n

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFPOST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Pattern 3

<normal request>
\r\n
<normal request>
\r\n\r\n

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Pattern 4

<header from normal request>
<normal request>
\r\n\r\n

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Pattern 5

<header from normal request>
<crippled version of normal request>
\r\n\r\n

POST /2.0/ HTTP/1.1
Host: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded
Content-Length: 169

Content-Length: 169

st: ws.audioscrobbler.com
Connection: close
User-Agent: foobar2000/1.1
Accept: */*
Icy-MetaData:1
Content-Type: application/x-www-form-urlencoded.......................method=track%2Elove&track=Biscaya&artist=James+Last&api_key=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&api_sig=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF&sk=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

HTTP POST bugs in http_client

Reply #4 – 2010-08-30 22:12:00

Well, thanks.

My test program:

  try {
    static_api_ptr_t<http_client> httpClient;

    http_request_post::ptr request;
    httpClient->create_request("POST")->service_query_t(request);
    assert(request.is_valid());
      
    request->add_post_data("method", "track.love");
    request->add_post_data("track", "Biscaya");
    request->add_post_data("artist", "James Last");
    request->add_post_data("api_key", "FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF");
    request->add_post_data("api_sig", "FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF");
    request->add_post_data("sk", "FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF");
      
    pfc::string8_fast str;
    file::ptr response = request->run_ex("http://ws.audioscrobbler.com/2.0/", abort_callback_dummy());      
    response->read_string_raw(str, abort_callback_dummy());
    uDebugLog() << "Content (" << str.length() << " Bytes): " << str;
    http_reply::ptr httpResponse;
    if (response->service_query_t(httpResponse)) {
      httpResponse->get_status(str);
      uDebugLog() << "Status: " << str;
    }
  } catch(pfc::exception &ex) {
    DebugBreak();
  }

And I have never seen anything else than Pattern 1.

I am on Win7/x64 too.
What am I doing differently? Any more tips how to reproduce?
How often does that request mangling happen? Do you run any firewalls or something?

To add, all my requests were send in two packets - the first till "Icy-MetaData:1\r\n" (I think, writting this later), the second with "Content-XXX: ...\r\n\r\n<data>".