summaryrefslogtreecommitdiff
path: root/t/t7519
AgeCommit message (Collapse)AuthorFilesLines
2020-01-23fsmonitor: add fsmonitor hook scripts for version 2Libravatar Kevin Willford2-0/+194
Version 2 of the fsmonitor hooks is passed the version and an update token and must pass back a last update token to use for subsequent calls to the hook. Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-13fsmonitor: handle version 2 of the hooks that will use opaque tokenLibravatar Kevin Willford2-3/+1
Some file monitors like watchman will use something other than a timestamp to keep better track of what changes happen in between calls to query the fsmonitor. The clockid in watchman is a string. Now that the index is storing an opaque token for the last update the code needs to be updated to pass that opaque token to a verion 2 of the fsmonitor hook. Because there are repos that already have version 1 of the hook and we want them to continue to work when git is updated, we need to handle both version 1 and version 2 of the hook. In order to do that a config value is being added core.fsmonitorHookVersion to force what version of the hook should be used. When this is not set it will default to -1 and then the code will attempt to call version 2 of the hook first. If that fails it will fallback to trying version 1. Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-01Merge branch 'kw/fsmonitor-watchman-fix'Libravatar Junio C Hamano1-9/+4
The watchman integration for fsmonitor was racy, which has been corrected to be more conservative. * kw/fsmonitor-watchman-fix: fsmonitor: fix watchman integration
2019-11-06fsmonitor: fix watchman integrationLibravatar Kevin Willford1-9/+4
When running Git commands quickly -- such as in a shell script or the test suite -- the Git commands frequently complete and start again during the same second. The example fsmonitor hooks to integrate with Watchman truncate the nanosecond times to seconds. In principle, this is fine, as Watchman claims to use inclusive comparisons [1]. The result should only be an over-representation of the changed paths since the last Git command. However, Watchman's own documentation claims "Using a timestamp is prone to race conditions in understanding the complete state of the file tree" [2]. All of their documented examples use a "clockspec" that looks like 'c:123:234'. Git should eventually learn how to store this type of string to provide a stronger integration, but that will be a more invasive change. When using GIT_TEST_FSMONITOR="$(pwd)/t7519/fsmonitor-watchman", scripts such as t7519-wtstatus.sh fail due to these race conditions. In fact, running any test script with GIT_TEST_FSMONITOR pointing at t/t7519/fsmonitor-wathcman will cause failures in the test_commit function. The 'git add "$indir$file"' command fails due to not enough time between the creation of '$file' and the 'git add' command. For now, subtract one second from the timestamp we pass to Watchman. This will make our window large enough to avoid these race conditions. Increasing the window causes tests like t7519-wtstatus.sh to pass. When the integration was introduced in def437671 (fsmonitor: add a sample integration script for Watchman, 2018-09-22), the query included an expression that would ignore files created and deleted in that window. The performance reason for this change was to ignore temporary files created by a build between Git commands. However, this causes failures in script scenarios where Git is creating or deleting files quickly. When using GIT_TEST_FSMONITOR as before, t2203-add-intent.sh fails due to this add-and-delete race condition. By removing the "expression" from the Watchman query, we remove this race condition. It will lead to some performance degradation in the case of users creating and deleting temporary files inside their working directory between Git commands. However, that is a cost we need to pay to be correct. [1] https://github.com/facebook/watchman/blob/master/query/since.cpp#L35-L39 [2] https://facebook.github.io/watchman/docs/clockspec.html Helped-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-12fsmonitor: don't fill bitmap with entries to be removedLibravatar William Baker1-0/+24
While doing some testing with fsmonitor enabled I found that git commands would segfault after staging and unstaging an untracked file. Looking at the crash it appeared that fsmonitor_ewah_callback was attempting to adjust bits beyond the bounds of the index cache. Digging into how this could happen it became clear that the fsmonitor extension must have been written with more bits than there were entries in the index. The root cause ended up being that fill_fsmonitor_bitmap was populating fsmonitor_dirty with bits for all entries in the index, even those that had been marked for removal. To solve this problem fill_fsmonitor_bitmap has been updated to skip entries with the the CE_REMOVE flag set. With this change the bits written for the fsmonitor extension will be consistent with the index entries written by do_write_index. Additionally, BUG checks have been added to detect if the number of bits in fsmonitor_dirty should ever exceed the number of entries in the index again. Another option that was considered was moving the call to fill_fsmonitor_bitmap closer to where the index is written (and where the fsmonitor extension itself is written). However, that did not work as the fsmonitor_dirty bitmap must be filled before the index is split during writing. Signed-off-by: William Baker <William.Baker@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-13fsmonitor: simplify determining the git worktree under WindowsLibravatar Ben Peart1-10/+3
Simplify and speed up the process of finding the git worktree when running on Windows by keeping it in perl and avoiding spawning helper processes. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-10fsmonitor: read from getcwd(), not the PWD environment variableLibravatar Alex Vandiver1-1/+2
Though the process has chdir'd to the root of the working tree, the PWD environment variable is only guaranteed to be updated accordingly if a shell is involved -- which is not guaranteed to be the case. That is, if `/usr/bin/perl` is a binary, $ENV{PWD} is unchanged from whatever spawned `git` -- if `/usr/bin/perl` is a trivial shell wrapper to the real `perl`, `$ENV{PWD}` will have been updated to the root of the working copy. Update to read from the Cwd module using the `getcwd` syscall, not the PWD environment variable. The Cygwin case is left unchanged, as it necessarily _does_ go through a shell. Signed-off-by: Alex Vandiver <alexmv@dropbox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-05fsmonitor: preserve utf8 filenames in fsmonitor-watchman logLibravatar Ben Peart1-0/+1
Update the test fsmonitor-watchman integration script to properly preserve utf8 filenames when outputting the .git/watchman-output.out log file. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-04fsmonitor: read entirety of watchman outputLibravatar Alex Vandiver1-4/+2
In Perl, setting $/ sets the string that is used as the "record separator," which sets the boundary that the `<>` construct reads to. Setting `local $/ = 0666;` evaluates the octal, getting 438, and stringifies it. Thus, the later read from `<CHLD_OUT>` stops as soon as it encounters the string "438" in the watchman output, yielding invalid JSON; repositories containing filenames with SHA1 hashes are able to trip this easily. Set `$/` to undefined, thus slurping all output from watchman. Also close STDIN which is provided to watchman, to better guarantee that we cannot deadlock with watchman while both attempting to read. Signed-off-by: Alex Vandiver <alexmv@dropbox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-04fsmonitor: MINGW support for watchman integrationLibravatar Ben Peart1-1/+1
Instead of just taking $ENV{'PWD'}, use the same logic that converts PWD to $git_work_tree on MSYS_NT in the watchman integration hook script also on MINGW. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-01fsmonitor: add test cases for fsmonitor extensionLibravatar Ben Peart3-0/+186
Test the ability to add/remove the fsmonitor index extension via update-index. Test that dirty files returned from the integration script are properly represented in the index extension and verify that ls-files correctly reports their state. Test that ensure status results are correct when using the new fsmonitor extension. Test untracked, modified, and new files by ensuring the results are identical to when not using the extension. Test that if the fsmonitor extension doesn't tell git about a change, it doesn't discover it on its own. This ensures git is honoring the extension and that we get the performance benefits desired. Three test integration scripts are provided: fsmonitor-all - marks all files as dirty fsmonitor-none - marks no files as dirty fsmonitor-watchman - integrates with Watchman with debug logging To run tests in the test suite while utilizing fsmonitor: First copy t/t7519/fsmonitor-all to a location in your path and then set GIT_FORCE_PRELOAD_TEST=true and GIT_FSMONITOR_TEST=fsmonitor-all and run your tests. Note: currently when using the test script fsmonitor-watchman on Windows, many tests fail due to a reported but not yet fixed bug in Watchman where it holds on to handles for directories and files which prevents the test directory from being cleaned up properly. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>