kc3-lang/libevent/http.c

Branch :

Log

Author Commit Date CI Message
4528d8e9 2020-06-25 09:08:31 Check error code of evhttp_add_header_internal() in evhttp_parse_query_impl()
7426a568 2020-05-25 11:25:18 http: Merge branch 'http-max_connections-pr-592' @jcoffland: "When the max connection limit is enabled and the limit is reached, the server will respond immediately with 503 Service Unavailable. This can be used to prevent servers from running out of file descriptors. This is better than request limiting because clients may make more than one request over a single connection. Blocking a request does not necessarily close the connection and free up a socket." * http-max_connections-pr-592: test: cover evhttp max connections Added evhttp max simultaneous connection limiting
eee26dee 2020-05-19 11:45:43 http: fix EVHTTP_CON_AUTOFREE in case of timeout (and some else) Refs: #182
083c6d54 2020-05-19 01:02:30 http: fix EVHTTP_CON_AUTOFREE in case of connection error Refs: #182
02905413 2016-02-09 18:01:00 Add callback support for error pages The existing error pages are very basic and don't allow for multi-lingual support or for conformity with other pages in a web site. The aim of the callback functionality is to allow custom error pages to be supported for calls to evhttp_send_error() by both calling applications and Libevent itself. A backward-incompatible change has been made to the title of error pages sent by evhttp_send_error(). The original version of the function used the reason argument as part of the title. That might have unforeseen side-effects if it contains HTML tags. Therefore the title has been changed to always use the standard status text. An example of the error callback can be found in this [version](https://github.com/libevent/libevent/files/123607/http-server.zip) of the 'http-server' sample. It will output error pages with very bright backgrounds, the error code using a very large font size and the reason. Closes: #323 (cherr-picked from PR)
12ad0c8f 2020-01-12 15:29:48 http: do not close connection for CONNECT
998e6834 2020-01-12 13:43:18 http: do not assume body for CONNECT
0cd536b0 2019-09-18 23:12:59 http: rename bind_socket_ai() to create_bind_socket_nonblock()
1c78451f 2019-09-16 23:24:32 http: make sure the other fields in ext_method are not changed by the callback
1c573ab3 2019-05-25 23:41:38 http: do not name variable "sun" since this breaks solaris builds -bash-3.2$ /opt/csw/bin/gcc -xc /dev/null -dM -E | grep '#define sun' #define sun 1
ad51a3c1 2019-05-12 15:19:31 http: avoid use of uninitialized value for AF_UNIX/AF_LOCAL sockaddr unixsock peer does not have sun_path initialized.
8dcb94a4 2016-01-08 13:36:20 Added http method extending User can define his own response method by calling evhttp_set_ext_method_cmp() on the struct http, or evhttp_connection_set_ext_method_cmp() on the connection. We expose a new stucture `evhttp_ext_method` which is passed to the callback if it's set. So any field can be modified, with some exceptions (in evhttp_method_): If the cmp function is set, it has the ability to modify method, and flags. Other fields will be ignored. Flags returned are OR'd with the current flags. Based on changes to the #282 from: Mark Ellzey <socket@gmail.com>
96e56beb 2019-04-02 15:43:35 http: Update allowed_methods field from 16 to 32bits.
799053db 2019-04-08 22:27:33 http: replace EVHTTP_REQ_UNKNOWN_ with 0 From the server perspective the evhttp_response_phrase_internal() should not be called with 0 before this patch, it will be called with EVHTTP_REQ_UNKNOWN_ hence this patch should not change behavior. Fixes: 68eb526d7b ("http: add WebDAV methods support") Fixes: #789 Fixes: #796 Reported-by: Thomas Bernard <miniupnp@free.fr>
68eb526d 2019-03-13 10:51:55 http: add WebDAV methods support WebDAV introduced new HTTP methods (RFC4918): PROPFIND, PROPPATCH, MKCOL, LOCK, UNLOCK, COPY, MOVE. Add support of the methods.
bf19462a 2019-03-13 00:20:25 http: suppress -Wwrite-string in evhttp_parse_query_impl()
5ee507c8 2019-03-04 06:53:42 http: implement separate timeouts for read/write/connect phase This patch allows to change timeout for next events read/write/connect separatelly, using new API: - client: evhttp_connection_set_connect_timeout_tv() -- for connect evhttp_connection_set_read_timeout_tv() -- for read evhttp_connection_set_write_timeout_tv() -- for write - server: evhttp_set_read_timeout_tv() -- for read evhttp_set_write_timeout_tv() -- for write It also changes a logic a little, before there was next fallbacks which does not handled in new API: - HTTP_CONNECT_TIMEOUT - HTTP_WRITE_TIMEOUT - HTTP_READ_TIMEOUT And introduce another internal flag (EVHTTP_CON_TIMEOUT_ADJUSTED) that will be used in evrpc, which adjust evhttp_connection timeout only if it is not default. Fixes: #692 Fixes: #715
30791ecc 2019-01-29 01:09:44 http: try to read existing data in buffer under EVHTTP_CON_READ_ON_WRITE_ERROR There are two possible ways of getting response from the server: - processing existing bufferevent buffer - reading from the socket (even after write() errored with -1, it is still possible) But we did not tried the first option, only the second one. Fixes: http/read_on_write_error (on freebsd/osx)
d3dcb5ac 2019-01-27 15:16:39 http: do not call deferred readcb if readcb is not set Otherwise evhttp_read_cb can be called with invalid connection state: http/read_on_write_error: [forking] [msg] libevent using: kqueue FAIL ../test/regress_http.c:4079: assert(req) FAIL ../test/regress_http.c:4087: assert(req)[err] evhttp_read_cb: illegal connection state 0
28d7221b 2018-12-28 04:42:20 http: Preserve socket error from listen across closesocket cleanup Closes: #738 (cherry-picked)
f3f7aa5a 2018-12-07 21:46:27 http: fix connection retries when there more then one request for connection We should not attemp to establishe the connection if there is retry timer active, since otherwise there will be a bug. Imagine next situation: con = evhttp_connection_base_new() evhttp_connection_set_retries(con, 2) req = evhttp_request_new() evhttp_make_request(con, req, ...) # failed during connecting, and timer for 2 second scheduler (retry_ev) Then another request scheduled for this evcon: evhttp_make_request(con, req, ...) # got request from server, # and now it tries to read the response from the server # (req.kind == EVHTTP_RESPONSE) # # but at this point retry_ev scheduled, # and it schedules the connect again, # and after the connect will succeeed, it will pick request with # EVHTTP_RESPONSE for sending and this is completelly wrong and will # fail in evhttp_make_header_response() since there is no # "http_server" for this evcon This was a long standing issue, that I came across few years ago firstly, bad only now I had time to dig into it (but right now it was pretty simple, by limiting amount of CPU for the process and using rr for debug to go back and forth).
b98d32d0 2018-11-14 00:20:20 http: improve error path for bufferevent_{setfd,enable,disable}() We have calls to the next functions but do not check return values, though they can be invalid and it is better to show this somehow. Also do bufferevent_setfd() first and only after it bufferevent_enable()/bufferevent_disable() since: a) it is more natural b) it will avoid extra operations c) it will not fail first bufferevent_enable() (this is the case for buffbufferevent_async at least) In this case we could add more information for issues like #709
5dc88b38 2018-11-13 21:26:12 Fix conceivable UAF of the bufferevent in evhttp_connection_free() Although this is not a problem, since bufferevent uses finalizers and will free itself only from the loop (well this is not a problem if you do not play games with various event_base in different threads) it generates questions, so rewrite it in more reliable way. Fixes: #712
26ef859a 2018-10-27 17:21:35 Add evhttp_parse_query_str_flags() And a set of flags: - EVHTTP_URI_QUERY_LAST - EVHTTP_URI_QUERY_NONCONFORMANT Fixes: #15
b94d913d 2018-10-22 23:52:46 http: allow non RFC3986 conformant during parsing request-line (http server) Reported-by: lsdyst@163.com
64ead341 2018-10-22 23:56:50 http: do not try to parse request-line if we do not have enough bytes
254fbc81 2018-10-22 23:56:19 http: allow trailing spaces (and only them) in request-line (like nginx)
6cf659b0 2018-10-22 23:25:01 http: cleanup of the request-line parsing
514dc757 2018-07-05 12:46:51 [http] fix C90 warnings
367cd9e5 2018-02-11 16:28:58 Fix evhttp_connection_get_addr() fox incomming http connections Install conn_address of the bufferevent on incomping http connections (even though this is kind of subsytem violation, so let's fix it in a simplest way and thinkg about long-term solution). Fixes: #510 Closes: #595 (pick)
cd57e38c 2018-02-13 17:43:02 http: remove message in case !Content-Length and Connection!=close Since [1] GET can have body, and hence for every incomming connection it will print this error. [1] db483e3b002b33890fc88cadd77f6fd1fccad2d2 ("Allow bodies for GET/DELETE/OPTIONS/CONNECT") Noticed-by: BotoX (irc) Refs: #408
c62f73f5 2018-01-30 15:39:41 Added evhttp max simultaneous connection limiting
61c21492 2018-01-09 21:44:57 http: fix leaks in evhttp_uriencode() Fixes: #584
727bcea1 2017-12-01 01:29:32 http: add callback to allow server to decline (and thereby close) incoming connections. This is important, as otherwise clients can easily exhaust the file descriptors available on a libevent HTTP server, which can cause problems in other code which does not handle EMFILE well: for example, see https://github.com/bitcoin/bitcoin/issues/11368 Closes: #578 (patch cherry picked)
65eb529a 2017-12-02 12:53:57 CONNECT method only takes an authority
306747e5 2017-11-04 19:13:28 Fix crashing http server when callback do not reply in place from *gencb* This is the second hunk of the first patch 5ff8eb26371c4dc56f384b2de35bea2d87814779 ("Fix crashing http server when callback do not reply in place") Fixes: #567
db483e3b 2017-10-29 22:53:41 Allow bodies for GET/DELETE/OPTIONS/CONNECT I checked with nginx, and via it's lua bindings it allows body for all this methods. Also everybody knows that some of web-servers allows body for GET even though this is not RFC conformant. Refs: #408
99d0a952 2017-09-26 19:23:29 Do not crash when evhttp_send_reply_start() is called after a timeout. This fixes the crash reported in issue #509. The "would be nice" items discussed in #509 can be addressed separately.
5ff8eb26 2017-10-23 00:13:37 Fix crashing http server when callback do not reply in place General http callback looks like: static void http_cb(struct evhttp_request *req, void *arg) { evhttp_send_reply(req, HTTP_OK, "Everything is fine", NULL); } And they will work fine becuase in this case http will write request first, and during write preparation it will disable *read callback* (in evhttp_write_buffer()), but if we don't reply immediately, for example: static void http_cb(struct evhttp_request *req, void *arg) { return; } This will leave connection in incorrect state, and if another request will be written to the same connection libevent will abort with: [err] ../http.c: illegal connection state 7 Because it thinks that read for now is not possible, since there were no write. Fix this by disabling EV_READ entirely. We couldn't just reset callbacks because this will leave EOF detection, which we don't need, since user hasn't replied to callback yet. Reported-by: Cory Fields <cory@coryfields.com>
da3f2ba2 2017-09-13 21:39:32 fix handling of close_notify (ssl) in http with openssl bufferevents Since it can arrive after we disabled events in that bufferevent and reseted fd, hence evhttp_error_cb() could be called after SSL_RECEIVED_SHUTDOWN. Closes: #557
db60ade8 2016-11-10 21:58:15 http: do not use local settings for Date header
1cbf26f6 2016-12-06 11:51:18 http: fix formatter for pritnf for req->ntoread (osx)
e9837124 2014-12-13 19:42:42 use ev_uint16_t instead of unsigned short for port Like in `sockaddr_in` structure in /usr/include/netinet/in.h @azat: convert all other users (bench, compat, ..) and tweak message Fixes: #178 Fixes: #196 Refs: 6bf1ca78 Link: https://codereview.appspot.com/156040043/#msg4
c6b1ec12 2016-08-24 17:16:32 Fix evhttp_uriencode() regression. http_uriencode_test() (in test/regress_http.c) has been failed after 72afe4c as "hello\0world" is encoded to "hello" instead of "hello%00world". This is because of a misplaced overflow check which causes the non-negative "size" specified in parameter being ignored in within-bound URI. Fixes: #392
e94250c8 2016-08-14 14:03:56 removed unused vars
72afe4c9 2016-08-14 14:00:02 pointer overflow checks for evhttp_uriencode Check to make sure pointer math is all OK.
17cc6362 2016-06-24 18:07:39 [Issue #313] set method to ASCII "NULL" if evhttp_method() returns NULL
24b52149 2016-03-25 10:21:48 evhttp_have_expect(): fix -Wlogical-not-parentheses ../http.c:589:6: warning: logical not is only applied to the left hand side of this comparison [-Wlogical-not-parentheses] if (!req->kind == EVHTTP_REQUEST || !REQ_VERSION_ATLEAST(req, 1, 1)) ^ ~~
7a4b4729 2016-03-24 13:38:05 http: set fd to -1 unconditioally, to avoid leaking of DNS requests Otherwise: http/cancel_by_host_ns_timeout_inactive_server: [msg] Nameserver 127.0.0.1:37035 has failed: request timed out. [msg] All nameservers have failed OK 1 tests ok. (0 skipped) ==26211== ==26211== FILE DESCRIPTORS: 3 open at exit. ==26211== Open file descriptor 2: /dev/pts/47 ==26211== <inherited from parent> ==26211== ==26211== Open file descriptor 1: /dev/pts/47 ==26211== <inherited from parent> ==26211== ==26211== Open file descriptor 0: /dev/pts/47 ==26211== <inherited from parent> ==26211== ==26211== ==26211== HEAP SUMMARY: ==26211== in use at exit: 1,112 bytes in 5 blocks ==26211== total heap usage: 149 allocs, 144 frees, 18,826 bytes allocated ==26211== ==26211== 40 bytes in 1 blocks are indirectly lost in loss record 1 of 5 ==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459) ==26211== by 0x498F5B: evbuffer_add_cb (buffer.c:3309) ==26211== by 0x4A0EF5: bufferevent_socket_new (bufferevent_sock.c:366) ==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375) ==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427) ==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417) ==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105) ==26211== by 0x490D5A: testcase_run_one (tinytest.c:252) ==26211== by 0x491699: tinytest_main (tinytest.c:434) ==26211== by 0x47E0E0: main (regress_main.c:461) ==26211== ==26211== 136 bytes in 1 blocks are indirectly lost in loss record 2 of 5 ==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459) ==26211== by 0x491FF0: evbuffer_new (buffer.c:365) ==26211== by 0x49A1BE: bufferevent_init_common_ (bufferevent.c:300) ==26211== by 0x4A0E44: bufferevent_socket_new (bufferevent_sock.c:353) ==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375) ==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427) ==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417) ==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105) ==26211== by 0x490D5A: testcase_run_one (tinytest.c:252) ==26211== by 0x491699: tinytest_main (tinytest.c:434) ==26211== by 0x47E0E0: main (regress_main.c:461) ==26211== ==26211== 136 bytes in 1 blocks are indirectly lost in loss record 3 of 5 ==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459) ==26211== by 0x491FF0: evbuffer_new (buffer.c:365) ==26211== by 0x49A1FB: bufferevent_init_common_ (bufferevent.c:305) ==26211== by 0x4A0E44: bufferevent_socket_new (bufferevent_sock.c:353) ==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375) ==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427) ==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417) ==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105) ==26211== by 0x490D5A: testcase_run_one (tinytest.c:252) ==26211== by 0x491699: tinytest_main (tinytest.c:434) ==26211== by 0x47E0E0: main (regress_main.c:461) ==26211== ==26211== 536 bytes in 1 blocks are indirectly lost in loss record 4 of 5 ==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459) ==26211== by 0x4A0E15: bufferevent_socket_new (bufferevent_sock.c:350) ==26211== by 0x4BFADF: evhttp_connection_base_bufferevent_new (http.c:2375) ==26211== by 0x4BFC8F: evhttp_connection_base_new (http.c:2427) ==26211== by 0x460DAA: http_cancel_test (regress_http.c:1417) ==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105) ==26211== by 0x490D5A: testcase_run_one (tinytest.c:252) ==26211== by 0x491699: tinytest_main (tinytest.c:434) ==26211== by 0x47E0E0: main (regress_main.c:461) ==26211== ==26211== 1,112 (264 direct, 848 indirect) bytes in 1 blocks are definitely lost in loss record 5 of 5 ==26211== at 0x4C2BBD5: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==26211== by 0x4AAEB2: event_mm_calloc_ (event.c:3459) ==26211== by 0x4D0564: evdns_getaddrinfo (evdns.c:4685) ==26211== by 0x4B13BA: evutil_getaddrinfo_async_ (evutil.c:1575) ==26211== by 0x4A139E: bufferevent_socket_connect_hostname (bufferevent_sock.c:524) ==26211== by 0x4C02DB: evhttp_connection_connect_ (http.c:2588) ==26211== by 0x4C04DD: evhttp_make_request (http.c:2643) ==26211== by 0x4615FF: http_cancel_test (regress_http.c:1504) ==26211== by 0x490A78: testcase_run_bare_ (tinytest.c:105) ==26211== by 0x490D5A: testcase_run_one (tinytest.c:252) ==26211== by 0x491699: tinytest_main (tinytest.c:434) ==26211== by 0x47E0E0: main (regress_main.c:461) ==26211== ==26211== LEAK SUMMARY: ==26211== definitely lost: 264 bytes in 1 blocks ==26211== indirectly lost: 848 bytes in 4 blocks ==26211== possibly lost: 0 bytes in 0 blocks ==26211== still reachable: 0 bytes in 0 blocks ==26211== suppressed: 0 bytes in 0 blocks
f0e13411 2016-03-22 23:36:19 http: avoid leaking of fd in evhttp_connection_free() Since we do close fd there if we don't have BEV_OPT_CLOSE_ON_FREE, and evcon->fd can be incorrect (non -1), so just get it from the underlying bufferevent to fix this. And after this patch the following tests report 0 instead of 2307 fd leaks: $ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel.. ==11299== FILE DESCRIPTORS: 3 open at exit. And this is stdin/stderr/stdout.
4a53c54b 2016-03-22 19:29:50 http: get fd from be layer during connection reset Since it can be non -1, and we must close it, otherwise we will have problems. And after this patch the following tests report fd 2307 instead of 2309 fd leaks: $ valgrind --leak-check=full --show-reachable=yes --track-fds=yes --error-exitcode=1 regress --no-fork http/cancel.. ==10853== FILE DESCRIPTORS: 2307 open at exit.
d4054928 2016-03-11 20:17:51 http: make fallback for EVHTTP_CON_READ_ON_WRITE_ERROR more cleaner
2ff164ab 2016-03-11 19:58:05 http: fix EVHTTP_CON_READ_ON_WRITE_ERROR when it doesn't supported by OS For example win32 doesn't accept such things (maybe via overloaded IO, I'm not sure), also I looked into curl and seems that the behaviour is the same (IOW like with EVHTTP_CON_READ_ON_WRITE_ERROR on linux/win32). Fixes: https://ci.appveyor.com/project/nmathewson/libevent/build/2.1.5.216#L499 (win32) Fixes: 680742e1665b85487f10c0ef3df021e3b8e98634 ("http: read server response even after server closed the connection") v2: v0 was just removing that flag, i.e. make it deprecated and set_flags() will return -1
7c899995 2016-03-11 19:52:32 http: do not do function calls under EVUTIL_ASSERT() to fix NDEBUG builds Fixes: 2185e639210f072f37e9d19aff7dba382db84529 ("http: assert's that evbuffer_drain() success on connection reset") Fixes: http/data_length_constraints FAIL ../test/regress_http.c:3775: assert(evhttp_request_get_response_code(req) == HTTP_ENTITYTOOLARGE): 501 vs 413
8f18a626 2016-03-11 20:40:52 http: fix leaking of response_code_line Since now evhttp_parse_response_line() can be called twice because after "HTTP/1.1 100 Continue" we can have "HTTP/1.1 200" ==29162== 9 bytes in 1 blocks are definitely lost in loss record 1 of 1 ==29162== at 0x4C29C0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==29162== by 0x5CBF0A9: strdup (in /lib/x86_64-linux-gnu/libc-2.21.so) ==29162== by 0x4AA3AC: event_mm_strdup_ (event.c:3493) ==29162== by 0x4BD843: evhttp_parse_response_line (http.c:1680) ==29162== by 0x4BE333: evhttp_parse_firstline_ (http.c:2013) ==29162== by 0x4BEA4F: evhttp_read_firstline (http.c:2243) ==29162== by 0x4BC5F8: evhttp_read_cb (http.c:1136) ==29162== by 0x4993F1: bufferevent_run_readcb_ (bufferevent.c:233) ==29162== by 0x49FBC0: bufferevent_trigger_nolock_ (bufferevent-internal.h:392) ==29162== by 0x49FF10: bufferevent_readcb (bufferevent_sock.c:208) ==29162== by 0x4A474A: event_persist_closure (event.c:1580) ==29162== by 0x4A49F5: event_process_active_single_queue (event.c:1639) Fixes: 0b46b39e95ad77951176f09782138305ba34edf3 ("http: fix "Expect: 100-continue" client side")
0b46b39e 2016-03-11 13:08:28 http: fix "Expect: 100-continue" client side Instead of sending data always at the beginning of the request wait until the server will respond with "HTTP/1.1 100 Continue". Before this patch server do send "HTTP/1.1 100 Continue" but client always send post data even without waiting server response. P.S. this patch also touches some not 100% related tab-align issues. Covered-by: http/data_length_constraints Covered-by: http/read_on_write_error
2185e639 2016-03-11 14:02:13 http: assert's that evbuffer_drain() success on connection reset Since otherwise we can have nasty bugs with part of previous *request* in current *request* and hence some parsing errors. And now we have failures: http/non_lingering_close: [forking] [err] ../http.c:1326: Assertion !evbuffer_drain(tmp, -1) failed in ../http.c
ac448a74 2016-02-15 03:26:40 http: take EVHTTP_CON_LINGERING_CLOSE into account for "Expect: 100-Continue" Also since after this patch code became more generic, we now respond with HTTP_ENTITYTOOLARGE even without "Expect: 100-Continue", which is correct by RFC. Refs: #321 v2: remove EVHTTP_CON_ABOUT_TO_CLOSE
9fde5189 2016-02-15 00:12:54 http: lingering close (like nginx have) for entity-too-large By lingering close I mean something what nginx have for this name, by this term I mean that we need to read all the body even if it's size greater then `max_body_size`, otherwise browsers on win32 (including chrome) failed read the http status - entity-too-large (while on linux chrome for instance are good), and also this includes badly written http clients. Refs: #321 v2: do this only under EVHTTP_SERVER_LINGERING_CLOSE
680742e1 2016-02-10 14:43:18 http: read server response even after server closed the connection Otherwise if we will try to write more data than server can accept (see `evhttp_set_max_body_size()` for libevent server) we will get `EPIPE` and will not try to read server's response which must contain 400 error for now (which is not strictly correct though, it must 413). ``` $ strace regress --no-fork http/data_length_constraints ... connect(10, {sa_family=AF_INET, sin_port=htons(43988), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) ... writev(10, [{"POST / HTTP/1.1\r\nHost: somehost\r"..., 60}, {"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16324}], 2) = 16384 epoll_wait(5, [{EPOLLOUT, {u32=10, u64=10}}, {EPOLLIN, {u32=11, u64=11}}], 32, 50000) = 2 writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384}], 1) = 16384 ioctl(11, FIONREAD, [32768]) = 0 readv(11, [{"POST / HTTP/1.1\r\nHost: somehost\r"..., 4096}], 1) = 4096 epoll_ctl(5, EPOLL_CTL_DEL, 11, 0x7fff09d41e50) = 0 epoll_ctl(5, EPOLL_CTL_ADD, 11, {EPOLLOUT, {u32=11, u64=11}}) = 0 epoll_wait(5, [{EPOLLOUT, {u32=10, u64=10}}, {EPOLLOUT, {u32=11, u64=11}}], 32, 50000) = 2 writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384}], 1) = 16384 writev(11, [{"HTTP/1.1 400 Bad Request\r\nConten"..., 129}, {"<HTML><HEAD>\n<TITLE>400 Bad Requ"..., 94}], 2) = 223 epoll_ctl(5, EPOLL_CTL_DEL, 11, 0x7fff09d42080) = 0 shutdown(11, SHUT_WR) = 0 close(11) = 0 epoll_wait(5, [{EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=10, u64=10}}], 32, 50000) = 1 writev(10, [{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., 16384}], 1) = -1 EPIPE (Broken pipe) --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=13954, si_uid=1000} --- epoll_ctl(5, EPOLL_CTL_DEL, 10, 0x7fff09d42010) = 0 shutdown(10, SHUT_WR) = -1 ENOTCONN (Transport endpoint is not connected) close(10) = 0 write(1, "\n FAIL ../test/regress_http.c:3"..., 37 ``` Careful reader can ask why it send error even when it didn't read `evcon->max_body_size`, and the answer will be checks for `evcon->max_body_size against `Content-Length` header, which contains ~8MB (-2 bytes). And also if we will not drain the output buffer than we will send buffer that we didn't send in previous request and instead of sending method via `evhttp_make_header()`. Fixes: http/data_length_constraints Refs: #321 v2: do this only under EVHTTP_CON_READ_ON_WRITE_ERROR flag
4dc09795 2016-02-15 02:59:40 http: fix conflicts EVHTTP_CON_AUTOFREE and EVHTTP_CON_REUSE_CONNECTED_ADDR And we can't make them continuous, since the latest is a public API, and otherwise we will break binary compatibility. Also extra check for EVHTTP_CON_PUBLIC_FLAGS_END, in case somebody forgot about this (implementer I mean). Refs: #182
365f181a 2016-02-15 03:53:25 http: coding style issue
ab3bc69f 2015-11-20 01:37:46 http: avoid epoll_ctl() on already closed fd (triggers by http/chunk_out) Before: $ strace -eepoll_ctl,close regress --verbose --no-fork +http/chunk_out close(3) = 0 close(3) = 0 close(3) = 0 close(3) = 0 close(3) = 0 close(3) = 0 close(3) = 0 close(3) = 0 http/chunk_out: epoll_ctl(5, EPOLL_CTL_ADD, 8, {EPOLLIN, {u32=8, u64=8}}) = 0 close(10) = 0 close(9) = 0 epoll_ctl(5, EPOLL_CTL_ADD, 9, {EPOLLIN, {u32=9, u64=9}}) = 0 epoll_ctl(5, EPOLL_CTL_ADD, 10, {EPOLLOUT, {u32=10, u64=10}}) = 0 epoll_ctl(5, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=11, u64=11}}) = 0 epoll_ctl(5, EPOLL_CTL_DEL, 10, 7ffffb10b870) = 0 epoll_ctl(5, EPOLL_CTL_ADD, 10, {EPOLLIN, {u32=10, u64=10}}) = 0 close(12) = 0 epoll_ctl(5, EPOLL_CTL_MOD, 11, {EPOLLIN|EPOLLOUT, {u32=11, u64=11}}) = 0 epoll_ctl(5, EPOLL_CTL_MOD, 11, {EPOLLIN, {u32=11, u64=11}}) = 0 close(11) = 0 epoll_ctl(5, EPOLL_CTL_DEL, 11, 7ffffb10b770) = -1 EBADF (Bad file descriptor) ... And trace for latest: close(11) = 0 > regress(evutil_closesocket+0x15) [0xadac4] > regress(evhttp_connection_free+0x19d) [0xbada2] > regress(evhttp_send_done+0x14d) [0xbde43] > regress(evhttp_write_cb+0x4a) [0xba1a1] > regress(bufferevent_run_writecb_+0xa2) [0x97bba] > regress(bufferevent_trigger_nolock_+0xb1) [0x9e330] > regress(bufferevent_writecb+0x2a3) [0x9e91a] > regress(event_persist_closure+0x2bb) [0xa2e09] > regress(event_process_active_single_queue+0x2a8) [0xa30b4] > regress(event_process_active+0x13f) [0xa3696] > regress(event_base_loop+0x2ab) [0xa3d97] > regress(event_base_dispatch+0x1d) [0xa371e] > regress(http_chunk_out_test_impl+0x132) [0x74940] epoll_ctl(5, EPOLL_CTL_DEL, 11, 7fff09439fe0) = -1 EBADF (Bad file descriptor) > regress(epoll_ctl+0x36) [0xb49a3] > regress(epoll_apply_one_change+0x14e) [0xb4cd4] > regress(epoll_nochangelist_del+0x87) [0xb51e4] > regress(evmap_io_del_+0x247) [0xaab04] > regress(event_del_nolock_+0x2f7) [0xa6aa8] > regress(event_callback_cancel_nolock_+0x6e) [0xa747e] > regress(event_callback_finalize_many_+0xeb) [0xa4dd6] > regress(bufferevent_decref_and_unlock_+0x21d) [0x98da7] > regress(bufferevent_writecb+0x312) [0x9e989] > regress(event_persist_closure+0x2bb) [0xa2e09] > regress(event_process_active_single_queue+0x2a8) [0xa30b4] > regress(event_process_active+0x13f) [0xa3696] > regress(event_base_loop+0x2ab) [0xa3d97] > regress(event_base_dispatch+0x1d) [0xa371e] > regress(http_chunk_out_test_impl+0x132) [0x74940] ...
040000d7 2015-11-16 00:55:30 http: install timeout for read too during connect for ssl Since during ssl handshake we can read too, and if something nasty will happens during this handshake (because of too many events in the loop of something like this) we can wait forever since there is no read timeout: (gdb) p *$2.task.connection.bufev $11 = { ... be_ops = 0x7f78c2864b00 <bufferevent_ops_openssl>, ev_read = { ... ev_ = { ev_io = { ... ev_timeout = { tv_sec = 0, tv_usec = 0 } }, ev_signal = { ... } }, ev_events = 82, ev_res = 2, ev_timeout = { tv_sec = 10889976, tv_usec = 418753 } }, ev_write = { ... ev_ = { ev_io = { ... ev_timeout = { tv_sec = 20, tv_usec = 0 } }, ev_signal = { ... } }, ev_events = 84, ev_res = 4, ev_timeout = { tv_sec = 10889977, tv_usec = 598753 } }, ... errorcb = 0x7f78c287de70 <evhttp_connection_cb>, ... timeout_read = { tv_sec = 0, tv_usec = 0 }, timeout_write = { tv_sec = 20, tv_usec = 0 }, enabled = 4 } (gdb) bt #0 0x00007f78c17c3633 in __epoll_wait_nocancel () at syscall-template.S:81 #1 0x00007f78c2aaf508 in epoll_dispatch (base=0x18f76d0, tv=<optimized out>) at epoll.c:463 ... Found-with: massive crawling Tested-with: massive crawling
c1404b56 2015-08-26 10:06:09 Include <sys/ioctl.h>, <sys/resource.h> and <sys/wait.h> optionally. Though CloudABI implements a very large part of POSIX, it does not provide these header files, for the reason that there is no raw device access, no resource limiting and no access to the global process table through wait(). It looks like these header files are not actually needed in theory. There don't seem to be any constructs in these source files that use these features, but I suspect they might still be required on some systems.
b0d3964f 2015-09-09 19:21:51 http: fix evhttp_request_own() by checking EVHTTP_USER_OWNED in more cases Suggested-by: @ellzey Fixes: http/request_own Fixes: #68
7ed02ac1 2015-09-08 16:40:55 http: fix detecting EOF without write Before this patch http server don't knows when client disconnected until it will try to write to it, IOW to detect is client still alive you need to write something to client socket, however it is not convenient since it requires to store all clients somewhere and poll them periodically, and I don't see any regressions if we will leave EV_READ always (like libevhtp do), since we already reset read callback in evhttp_write_buffer() (see http/write_during_read). Also since we don't disable EV_READ anymore we don't need some enable EV_READ, so we will reduce number of epoll_ctl() calls. Covered-by: http/terminate_chunked_oneshot Covered-by: http/write_during_read Fixes: #78
dfad1a46 2015-08-19 23:00:49 http: eliminate warning about "socklen" in evhttp_connection_connect_()
a50f5f0a 2015-01-01 06:27:31 http: reuse connected address only with EVHTTP_CON_REUSE_CONNECTED_ADDR
54c887d8 2014-11-12 20:23:46 http: use IP address that we got before (if any) during retrying Before this patch every time we are retrying our request we resolve domain, but we could optimize this (since UDP is slow) by using cached conn_address value, so do this.
8bb38425 2014-11-15 21:46:11 bufferevent: move conn_address out from http into bufferevent In http the only case when when we could store it is when we already connected, *but* if we are doing request using domain name, then we need to do request to nameserver to get IP address, and this is handled by bufferevent. So when we have IP address (from nameserver) and don't have connection to this IP address, we could already cache it to avoid extra DNS requests (since UDP is slow), and we can't do this from http layer, only from bufferevent.
22061ac1 2015-08-18 03:03:51 http: introduce evhttp_request_free_() helper
6540da38 2015-04-24 19:04:51 http: introduce evhttp_is_request_connection_close() helper
bdeec440 2015-06-17 08:24:52 Merge pull request #190 from JohnOhl/evhttp-post-fix evhttp: Fix failure to send all output data for POST/PUT requests
7fd49414 2014-11-30 19:26:20 Merge remote-tracking branch 'origin/pr/182'
24eea0da 2014-11-24 02:32:23 evhttp: Fix failure to send all output data for POST/PUT requests
10fe4ef3 2014-11-16 23:40:16 Prevent duplicate event_del on fd
2b9ec4c1 2014-10-26 01:18:10 Implement interface that provides the ability to have an outbound evhttp_connection free itself once all requests have completed
bc79cc5c 2014-09-27 21:29:45 http: reset connection before installing retry timer (fix http retries handling) This will fix some invalid read/write: ==556== Invalid read of size 8 ==556== at 0x4E4EEC6: event_queue_remove_timeout (minheap-internal.h:178) ==556== by 0x4E508AA: event_del_nolock_ (event.c:2764) ==556== by 0x4E53535: event_base_loop (event.c:3088) ==556== by 0x406FCFA: dispatch (libcrawl.c:271) ==556== by 0x402863: main (crawler.c:49) ==556== Address 0x68a3f18 is 152 bytes inside a block of size 400 free'd ==556== at 0x4C29C97: free (in /usr/local/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==556== by 0x406F140: renew (libcrawl.c:625) ==556== by 0x4E6CDE9: evhttp_connection_cb_cleanup (http.c:1331) ==556== by 0x4E6E2B2: evhttp_connection_cb (http.c:1424) ==556== by 0x4E4DF2D: bufferevent_writecb (bufferevent_sock.c:310) ==556== by 0x4E52D1D: event_process_active_single_queue (event.c:1584) ==556== by 0x4E53676: event_base_loop (event.c:1676) ==556== by 0x406FCFA: dispatch (libcrawl.c:271) ==556== by 0x402863: main (crawler.c:49) But this one because of some invalid write before (I guess). It is 100% reproduced during massive crawling (because this process has many different servers), but after spending some time for trying to reproduce this using some simple tests/utils I gave up for a few days (I have a lot of work to do), but I'm sending this patch as a reminder. Just in case, I've tried next tests: - mixing timeouts/retries - shutdown http server and return it back - slow dns server for first request - sleep before accept - hacking libevent sources to change the behaviour of http layer (so it will go into that function which I'm insterested in).
73615a37 2014-09-18 11:31:52 Merge pull request #118 from azat/http-forward-family-to-bufferevent Add evhttp_connection_set_family() to set addrinfo->family for DNS requests
12c29b0f 2014-03-21 17:32:09 Add evhttp_connection_set_family() to set addrinfo->family for DNS requests This is useful if you want to avoid extra dns requests.
e660db6d 2014-03-18 11:39:23 Catch over-large port numbers early in http Otherwise integer overflow potentially turns the port number into garbage.
da86dda9 2014-01-06 20:36:31 evhttp_request_set_on_complete_cb to be more specific about what the function actually does and usage
b083ca05 2014-01-05 20:35:46 Provide on request complete callback facility This patch provides the ability to receive a callback on the completion of a request. The callback takes place immediately before the request's resources are released.
b0bd7fe1 2013-11-18 16:06:16 Allow registering callback for parsing HTTP headers Slightly changed version of Espen Jürgensen's commit 548141e72312126fa6121f6a5f436đ251c7fb1251 for forked-daapd.
8d8decf1 2009-05-02 20:40:11 Add a variant of evhttp_send_reply_chunk() with a callback on evhttp_write_buffer() evhttp_write_buffer() used by evhttp_send_reply_chunk() can take callback executed when (part of) the buffer has been written. Using this callback to schedule the next chunk avoids buffering large amounts of data in memory.
0c7f0405 2013-10-01 19:12:13 http: implement new evhttp_connection_get_addr() api. Basically tcp final handshake looks like this: (C - client, S - server) ACK[C] - FIN/ACK[S] - FIN/ACK[S] - ACK [C] However there are servers, that didn't close connection like this, while it is still _considered_ as valid, and using libevent http layer we can do requests to such servers. Modified handshake: (C - client, S - server) ACK[C] - RST/ACK[S] - RST/ACK[S] And in this case we can't extract IP address from socket, because it is already closed, and getpeername() will return: "transport endpoint is not connected". So we need to store address that we are connecting to, after we know it, and that is what this patch do. I have reproduced it, however it have some extra packages. (I will try to fix it) https://github.com/azat/nfq-examples/blob/master/nfqnl_rst_fin.c
4464bd23 2013-10-10 16:10:50 Tweak indentation
5a5acd9a 2013-09-28 20:03:28 add a http default content type option
b04d13cd 2013-09-09 12:06:53 Merge remote-tracking branch 'origin/patches-2.0'
5eb17885 2013-09-03 14:46:47 Avoid racy bufferevent activation The evhttp_send_reply method invokes evhttp_write_buffer with a callback that may release the underlying request object and bufferevent upon completion. This cleanup callback is invoked by the underlying bufferevent's write callback. Improperly enabling write events before referencing the bufferevent could lead to use after free and memory corruption.
f22049e3 2013-08-06 19:17:08 Fix an unlikely but possible error case for http connections Found by coverity
a7f82a31 2013-07-24 20:50:05 Add evhttp_connection_get_server().
31db8a02 2013-05-23 16:29:17 Fix a double close() bug in evhttp when the underlying bufferevent uses BEV_OPT_CLOSE_ON_FREE.
7b077194 2013-03-21 13:55:40 Add new error_cb for actual reporting of HTTP request errors. It is useful to know why you callback called with NULL (i.e. it failed), for example if you set max_body with evhttp_connection_set_max_body_size() you must know that it failed because of body was longer than this size. (Commit message tweaked by Nick)
de8101a8 2013-03-01 12:00:24 Move prototype of evhttp_decode_uri_internal() to http-internal.h Make it non static, that can be called from tests
e1903e3a 2013-02-28 23:10:02 uri decode: changed the test for the existence of the next character Fix for 64b6eceaba1a4 More info here https://github.com/azat/libevent/commit/64b6eceaba1a40ab0b175fa9fd9329d3e978ce6e#commitcomment-2714685
64b6ecea 2013-02-28 17:19:44 uri decode: fix for warning "use of uninitialised value" This patch add check in evhttp_decode_uri_internal() that next 2 symbols are exists in array of chars for decoding, if don't have two next 2 symbols don't try to decode '%FF'
97094614 2013-02-14 14:13:11 Merge remote-tracking branch 'origin/patches-2.0'
b6182042 2013-02-14 09:54:56 fix #73 and fix http_connection_fail_test to catch it