summaryrefslogtreecommitdiff
path: root/compat/poll
AgeCommit message (Collapse)AuthorFilesLines
2020-02-27mingw: workaround for hangs when sending STDINLibravatar Alexandr Miloslavskiy1-28/+3
Explanation ----------- The problem here is flawed `poll()` implementation. When it tries to see if pipe can be written without blocking, it eventually calls `NtQueryInformationFile()` and tests `WriteQuotaAvailable`. However, the meaning of quota was misunderstood. The value of quota is reduced when either some data was written to a pipe, *or* there is a pending read on the pipe. Therefore, if there is a pending read of size >= than the pipe's buffer size, poll() will think that pipe is not writable and will hang forever, usually that means deadlocking both pipe users. I have studied the problem and found that Windows pipes track two values: `QuotaUsed` and `BytesInQueue`. The code in `poll()` apparently wants to know `BytesInQueue` instead of quota. Unfortunately, `BytesInQueue` can only be requested from read end of the pipe, while `poll()` receives write end. The git's implementation of `poll()` was copied from gnulib, which also contains a flawed implementation up to today. I also had a look at implementation in cygwin, which is also broken in a subtle way. It uses this code in `pipe_data_available()`: fpli.WriteQuotaAvailable = (fpli.OutboundQuota - fpli.ReadDataAvailable) However, `ReadDataAvailable` always returns 0 for the write end of the pipe, turning the code into an obfuscated version of returning pipe's total buffer size, which I guess will in turn have `poll()` always say that pipe is writable. The commit that introduced the code doesn't say anything about this change, so it could be some debugging code that slipped in. These are the typical sizes used in git: 0x2000 - default read size in `strbuf_read()` 0x1000 - default read size in CRT, used by `strbuf_getwholeline()` 0x2000 - pipe buffer size in compat\mingw.c As a consequence, as soon as child process uses `strbuf_read()`, `poll()` in parent process will hang forever, deadlocking both processes. This results in two observable behaviors: 1) If parent process begins sending STDIN quickly (and usually that's the case), then first `poll()` will succeed and first block will go through. MAX_IO_SIZE_DEFAULT is 8MB, so if STDIN exceeds 8MB, then it will deadlock. 2) If parent process waits a little bit for any reason (including OS scheduler) and child is first to issue `strbuf_read()`, then it will deadlock immediately even on small STDINs. The problem is illustrated by `git stash push`, which will currently read the entire patch into memory and then send it to `git apply` via STDIN. If patch exceeds 8MB, git hangs on Windows. Possible solutions ------------------ 1) Somehow obtain `BytesInQueue` instead of `QuotaUsed` I did a pretty thorough search and didn't find any ways to obtain the value from write end of the pipe. 2) Also give read end of the pipe to `poll()` That can be done, but it will probably invite some dirty code, because `poll()` * can accept multiple pipes at once * can accept things that are not pipes * is expected to have a well known signature. 3) Make `poll()` always reply "writable" for write end of the pipe Afterall it seems that cygwin (accidentally?) does that for years. Also, it should be noted that `pump_io_round()` writes 8MB blocks, completely ignoring the fact that pipe's buffer size is only 8KB, which means that pipe gets clogged many times during that single write. This may invite a deadlock, if child's STDERR/STDOUT gets clogged while it's trying to deal with 8MB of STDIN. Such deadlocks could be defeated with writing less than pipe's buffer size per round, and always reading everything from STDOUT/STDERR before starting next round. Therefore, making `poll()` always reply "writable" shouldn't cause any new issues or block any future solutions. 4) Increase the size of the pipe's buffer The difference between `BytesInQueue` and `QuotaUsed` is the size of pending reads. Therefore, if buffer is bigger than size of reads, `poll()` won't hang so easily. However, I found that for example `strbuf_read()` will get more and more hungry as it reads large inputs, eventually surpassing any reasonable pipe buffer size. Chosen solution --------------- Make `poll()` always reply "writable" for write end of the pipe. Hopefully one day someone will find a way to implement it properly. Reproduction ------------ printf "%8388608s" X >large_file.txt git stash push --include-untracked -- large_file.txt I have decided not to include this as test to avoid slowing down the test suite. I don't expect the specific problem to come back, and chances are that `git stash push` will be reworked to avoid sending the entire patch via STDIN. Signed-off-by: Alexandr Miloslavskiy <alexandr.miloslavskiy@syntevo.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-11Merge branch 'js/mingw-use-utf8'Libravatar Junio C Hamano1-1/+1
Windows update. * js/mingw-use-utf8: mingw: fix possible buffer overrun when calling `GetUserNameW()` mingw: use Unicode functions explicitly mingw: get pw_name in UTF-8 format
2019-06-27mingw: use Unicode functions explicitlyLibravatar Johannes Schindelin1-1/+1
Many Win32 API functions actually exist in two variants: one with the `A` suffix that takes ANSI parameters (`char *` or `const char *`) and one with the `W` suffix that takes Unicode parameters (`wchar_t *` or `const wchar_t *`). The ANSI variant assumes that the strings are encoded according to whatever is the current locale. This is not what Git wants to use on Windows: we assume that `char *` variables point to strings encoded in UTF-8. There is a pseudo UTF-8 locale on Windows, but it does not work as one might expect. In addition, if we overrode the user's locale, that would modify the behavior of programs spawned by Git (such as editors, difftools, etc), therefore we cannot use that pseudo locale. Further, it is actually highly encouraged to use the Unicode versions instead of the ANSI versions, so let's do precisely that. Note: when calling the Win32 API functions _without_ any suffix, it depends whether the `UNICODE` constant is defined before the relevant headers are #include'd. Without that constant, the ANSI variants are used. Let's be explicit and avoid that ambiguity. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-13poll (mingw): allow compiling with GCC 8 and DEVELOPER=1Libravatar Johannes Schindelin1-1/+1
The return type of the `GetProcAddress()` function is `FARPROC` which evaluates to `long long int (*)()`, i.e. it cannot be cast to the correct function signature by GCC 8. To work around that, we first cast to `void *` and go on with our merry lives. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-11-05poll: use GetTickCount64() to avoid wrap-around issuesLibravatar Steve Hoelzer1-4/+8
The value of timeout starts as an int value, and for this reason it cannot overflow unsigned long long aka ULONGLONG. The unsigned version of this initial value is available in orig_timeout. The difference (orig_timeout - elapsed) cannot wrap around because it is protected by a conditional (as can be seen in the patch text). Hence, the ULONGLONG difference can only have values that are smaller than the initial timeout value and truncation to int cannot overflow. Signed-off-by: Steve Hoelzer <shoelzer@gmail.com> [j6t: improved both implementation and log message] Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-04compat/poll: prepare for targeting Windows VistaLibravatar Johannes Schindelin2-3/+18
Windows Vista (and later) actually have a working poll(), but we still cannot use it because it only works on sockets. So let's detect when we are targeting Windows Vista and undefine those constants, and define `pollfd` so that we can declare our own pollfd struct. We also need to make sure that we override those constants *after* `winsock2.h` has been `#include`d (otherwise we would not really override those constants). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-09Replace Free Software Foundation address in license noticesLibravatar Todd Zullinger2-4/+2
The mailing address for the FSF has changed over the years. Rather than updating the address across all files, refer readers to gnu.org, as the GNU GPL documentation now suggests for license notices. The mailing address is retained in the full license files (COPYING and LGPL-2.1). The old address is still present in t/diff-lib/COPYING. This is intentional, as the file is used in tests and the contents are not expected to change. Signed-off-by: Todd Zullinger <tmz@pobox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-29poll.c: always set revents, even if to zeroLibravatar Randall S. Becker1-0/+4
Match what is done to pfd[i].revents when compute_revents() returns 0 to the upstream gnulib's commit d42461c3 ("poll: fixes for large fds", 2015-02-20). The revents field is set to 0, without incrementing the value rc to be returned from the function. The original code left the field to whatever random value the field was initialized to. This fixes occasional hangs in git-upload-pack on HPE NonStop. Signed-off-by: Randall S. Becker <randall.becker@nexbridge.ca> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-01-15mingw: avoid warnings when casting HANDLEs to intLibravatar Johannes Schindelin1-1/+1
HANDLE is defined internally as a void *, but in many cases it is actually guaranteed to be a 32-bit integer. In these cases, GCC should not warn about a cast of a pointer to an integer of a different type because we know exactly what we are doing. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-14poll: honor the timeout on Win32Libravatar Edward Thomson1-2/+14
Ensure that when passing a pipe, the gnulib poll replacement will not return 0 before the timeout has passed. Not obeying the timeout (and merely returning 0) causes pathological behavior when preparing a packfile for a repository and taking a long time to do so. If poll were to return 0 immediately, this would cause keep-alives to get sent as quickly as possible until the packfile was created. Such deviance from the standard would cause megabytes (or more) of keep-alive packets to be sent. GetTickCount is used as it is efficient, stable and monotonically increasing. (Neither GetSystemTime nor QueryPerformanceCounter have all three of these properties.) Signed-off-by: Edward Thomson <ethomson@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-04-29compat/poll: sleep 1 millisecond to avoid busy waitLibravatar Theodore Leblond1-1/+1
SwitchToThread() only gives away the rest of the current time slice to another thread in the current process. So if the thread that feeds the file decscriptor we're polling is not in the current process, we get busy-waiting. I played around with this quite a bit. After trying some more complex schemes, I found that what worked best is to just sleep 1 millisecond between iterations. Though it's a very short time, it still completely eliminates the busy wait condition, without hurting perf. There code uses SleepEx(1, TRUE) to sleep. See this page for a good discussion of why that is better than calling SwitchToThread, which is what was used previously: http://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1 Note that calling SleepEx(0, TRUE) does *not* solve the busy wait. The most striking case was when testing on a UNC share with a large repo, on a single CPU machine. Without the fix, it took 4 minutes 15 seconds, and with the fix it took just 1:08! I think it's because git-upload-pack's busy wait was eating the CPU away from the git process that's doing the real work. With multi-proc, the timing is not much different, but tons of CPU time is still wasted, which can be a killer on a server that needs to do bunch of other things. I also tested the very fast local case, and didn't see any measurable difference. On a big repo with 4500 files, the upload-pack took about 2 seconds with and without the fix. [jc: this was first accepted in msysgit tree in May 2012 via a pull request and Paolo Bonzini has also accepted the same fix to Gnulib around the same time; see $gmane/247518 for a bit more detail] Signed-off-by: Stepan Kasal <kasal@ucw.cz> Acked-by: Johannes Sixt <j6t@kdbg.org> Acked-by: Erik Faye-Lund <kusmabite@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-09-11Windows: do not redefine _WIN32_WINNTLibravatar Sebastian Schuberth1-1/+1
With MinGW runtime version 4.0 this interferes with the previous definition from sdkddkver.h. Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-04-28compat/poll/poll.c: Fix a sparse warningLibravatar Ramsay Jones1-1/+1
Sparse issues an 'Using plain integer as NULL pointer' warning when passing the constant '0' as the second parameter in the call to the WSAEventSelect() function. The function parameter has a pointer type (WSAEVENT, aka HANDLE, aka void *) so that, in order to suppress the warning, we simply pass NULL for that parameter in the function call expression. Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-17make poll() work on platforms that can't recv() on a non-socketLibravatar Joachim Schmitz1-0/+4
This way it just got added to gnulib too the other day. Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-17poll() exits too early with EFAULT if 1st arg is NULLLibravatar Joachim Schmitz1-1/+1
If poll() is used as a milli-second sleep, like in help.c, by passing a NULL in the 1st and a 0 in the 2nd arg, it exits with EFAULT. As per Paolo Bonzini, the original author, this is a bug and to be fixed Like in this commit, which is not to exit if the 2nd arg is 0. It got fixed In gnulib in the same manner the other day. Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-17fix some win32 specific dependencies in poll.cLibravatar Joachim Schmitz1-2/+6
In order for non-win32 platforms to be able to use poll.c, #ifdef the inclusion of two header files properly Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-09-17make poll available for other platforms lacking itLibravatar Joachim Schmitz2-0/+659
move poll.[ch] out of compat/win32/ into compat/poll/ and adjust Makefile with the changed paths. Adding comments to Makefile about how/when to enable it and add logic for this Signed-off-by: Joachim Schmitz <jojo@schmitz-digital.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>