| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
This benchmark covers the git status time for a heavily
dirty directory - benchmarking fsmonitor's refresh
When running to compare our perl vs rs-git-fsmonitor - we see that
the perl script incurs significant overhead - further motivation
to provide a faster implementation within git.
7519.7: status (dirty) (fsmonitor=query-watchman) 10.05(7.78+1.56)
7519.20: status (dirty) (fsmonitor=rs-git-fsmonitor) 6.72(4.37+1.64)
7519.33: status (dirty) (fsmonitor=disabled) 5.62(4.24+2.03)
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Allows for simple perf comparison of different integrations. I ran it
to compare our perl script w/ rs-git-fsmonitor and found 20-30ms of
overhead on every command.
Output looks like this (extra newlines added for readability)
Test this tree
---------------------------------------------------------------------------
7519.4: status (fsmonitor=query-watchman) 0.42(0.37+0.05)
7519.5: status -uno (fsmonitor=query-watchman) 0.19(0.12+0.07)
7519.6: status -uall (fsmonitor=query-watchman) 1.36(0.73+0.62)
7519.7: diff (fsmonitor=query-watchman) 0.14(0.09+0.05)
7519.8: diff -- 0_files (fsmonitor=query-watchman) 0.14(0.11+0.03)
7519.9: diff -- 10_files (fsmonitor=query-watchman) 0.14(0.10+0.04)
7519.10: diff -- 100_files (fsmonitor=query-watchman) 0.14(0.09+0.05)
7519.11: diff -- 1000_files (fsmonitor=query-watchman) 0.14(0.08+0.06)
7519.12: diff -- 10000_files (fsmonitor=query-watchman) 0.14(0.09+0.05)
7519.13: add (fsmonitor=query-watchman) 2.04(1.32+0.66)
7519.16: status (fsmonitor=rs-git-fsmonitor) 0.39(0.32+0.08)
7519.17: status -uno (fsmonitor=rs-git-fsmonitor) 0.17(0.11+0.06)
7519.18: status -uall (fsmonitor=rs-git-fsmonitor) 1.33(0.71+0.61)
7519.19: diff (fsmonitor=rs-git-fsmonitor) 0.11(0.07+0.04)
7519.20: diff -- 0_files (fsmonitor=rs-git-fsmonitor) 0.11(0.09+0.03)
7519.21: diff -- 10_files (fsmonitor=rs-git-fsmonitor) 0.11(0.09+0.03)
7519.22: diff -- 100_files (fsmonitor=rs-git-fsmonitor) 0.11(0.07+0.04)
7519.23: diff -- 1000_files (fsmonitor=rs-git-fsmonitor) 0.11(0.06+0.06)
7519.24: diff -- 10000_files (fsmonitor=rs-git-fsmonitor) 0.11(0.06+0.06)
7519.25: add (fsmonitor=rs-git-fsmonitor) 2.03(1.28+0.69)
7519.28: status (fsmonitor=disabled) 0.77(0.59+0.99)
7519.29: status -uno (fsmonitor=disabled) 0.42(0.33+0.85)
7519.30: status -uall (fsmonitor=disabled) 1.59(1.02+1.34)
7519.31: diff (fsmonitor=disabled) 0.35(0.30+0.81)
7519.32: diff -- 0_files (fsmonitor=disabled) 0.11(0.08+0.04)
7519.33: diff -- 10_files (fsmonitor=disabled) 0.11(0.07+0.04)
7519.34: diff -- 100_files (fsmonitor=disabled) 0.11(0.08+0.03)
7519.35: diff -- 1000_files (fsmonitor=disabled) 0.11(0.10+0.02)
7519.36: diff -- 10000_files (fsmonitor=disabled) 0.12(0.07+0.06)
7519.37: add (fsmonitor=disabled) 2.24(1.48+1.44)
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Previously, the git add of the previous suiterun would
pollute the numbers in the second run
Before:
Test this tree
-----------------------------------------------------------------------------
7519.4: status (fsmonitor=fsmonitor-watchman) 0.40(0.36+0.04)
7519.5: status -uno (fsmonitor=fsmonitor-watchman) 0.19(0.12+0.07)
7519.6: status -uall (fsmonitor=fsmonitor-watchman) 1.36(0.74+0.61)
7519.7: diff (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.04)
7519.8: diff -- 0_files (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.04)
7519.9: diff -- 10_files (fsmonitor=fsmonitor-watchman) 0.14(0.09+0.05)
7519.10: diff -- 100_files (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.04)
7519.11: diff -- 1000_files (fsmonitor=fsmonitor-watchman) 0.14(0.08+0.06)
7519.12: diff -- 10000_files (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.04)
7519.13: add (fsmonitor=fsmonitor-watchman) 2.03(1.28+0.69)
7519.16: status (fsmonitor=disabled) 0.64(0.49+0.90)
7519.17: status -uno (fsmonitor=disabled) 1.15(0.92+1.00)
7519.18: status -uall (fsmonitor=disabled) 2.32(1.46+1.55)
7519.19: diff (fsmonitor=disabled) 1.44(1.12+1.76)
7519.20: diff -- 0_files (fsmonitor=disabled) 0.11(0.07+0.05)
7519.21: diff -- 10_files (fsmonitor=disabled) 0.11(0.06+0.05)
7519.22: diff -- 100_files (fsmonitor=disabled) 0.11(0.08+0.03)
7519.23: diff -- 1000_files (fsmonitor=disabled) 0.11(0.08+0.04)
7519.24: diff -- 10000_files (fsmonitor=disabled) 0.12(0.06+0.07)
7519.25: add (fsmonitor=disabled) 2.25(1.47+1.47)
After:
Test this tree
-----------------------------------------------------------------------------
7519.4: status (fsmonitor=fsmonitor-watchman) 0.41(0.33+0.09)
7519.5: status -uno (fsmonitor=fsmonitor-watchman) 0.20(0.14+0.07)
7519.6: status -uall (fsmonitor=fsmonitor-watchman) 1.37(0.78+0.58)
7519.7: diff (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.04)
7519.8: diff -- 0_files (fsmonitor=fsmonitor-watchman) 0.14(0.08+0.06)
7519.9: diff -- 10_files (fsmonitor=fsmonitor-watchman) 0.14(0.09+0.05)
7519.10: diff -- 100_files (fsmonitor=fsmonitor-watchman) 0.14(0.10+0.05)
7519.11: diff -- 1000_files (fsmonitor=fsmonitor-watchman) 0.14(0.11+0.04)
7519.12: diff -- 10000_files (fsmonitor=fsmonitor-watchman) 0.14(0.09+0.05)
7519.13: add (fsmonitor=fsmonitor-watchman) 2.04(1.27+0.71)
7519.16: status (fsmonitor=disabled) 0.78(0.59+0.99)
7519.17: status -uno (fsmonitor=disabled) 0.43(0.32+0.88)
7519.18: status -uall (fsmonitor=disabled) 1.58(0.96+1.38)
7519.19: diff (fsmonitor=disabled) 0.36(0.31+0.79)
7519.20: diff -- 0_files (fsmonitor=disabled) 0.11(0.08+0.03)
7519.21: diff -- 10_files (fsmonitor=disabled) 0.11(0.07+0.04)
7519.22: diff -- 100_files (fsmonitor=disabled) 0.11(0.08+0.04)
7519.23: diff -- 1000_files (fsmonitor=disabled) 0.11(0.07+0.05)
7519.24: diff -- 10000_files (fsmonitor=disabled) 0.12(0.08+0.05)
7519.25: add (fsmonitor=disabled) 2.25(1.48+1.47)
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
This prepares for it being called multiple times when
testing different hooks
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
It is extremely verbose, printing >10K non-useful lines
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
The full name is lengthy and makes it hard to read
Before:
7519.3: status (fsmonitor=/home/nipunn/src/server/.git/hooks/rs-git-fsmonitor) 0.02(0.01+0.00)
After
7519.3: status (fsmonitor=rs-git-fsmonitor) 0.03(0.02+0.00)
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
There was much duplication here. Prepares for making
changes to the description.
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Previously - it would silently run the perf suite w/o using
fsmonitor - fsmonitor errors are not hard failures.
Now it errors loudly.
GIT_PERF_7519_FSMONITOR="$HOME/rs-git-fsmonitorr"
./p7519-fsmonitor.sh -i -v
fatal: cannot run /home/nipunn/rs-git-fsmonitorr:
No such file or directory
not ok 2 - setup for fsmonitor
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
It is only required to be set up once. This prepares for
testing multiple hooks in one invocation.
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
In preparation for testing multiple fsmonitor hooks
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Test v2.29.0-rc1 this tree
-----------------------------------------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 1.48(0.79+0.67) 1.48(0.79+0.67) +0.0%
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.16(0.11+0.05) 0.17(0.13+0.04) +6.3%
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.36(0.77+0.58) 1.37(0.72+0.63) +0.7%
7519.5: diff (fsmonitor=.git/hooks/fsmonitor-watchman) 0.84(0.21+0.63) 0.14(0.11+0.03) -83.3%
7519.6: diff -- 0_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.07+0.05) 0.13(0.09+0.04) +8.3%
7519.7: diff -- 10_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.09+0.04) 0.13(0.07+0.06) +8.3%
7519.8: diff -- 100_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.08+0.05) 0.12(0.08+0.05) +0.0%
7519.9: diff -- 1000_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.08+0.05) 0.13(0.09+0.04) +8.3%
7519.10: diff -- 10000_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.14(0.08+0.06) 0.13(0.07+0.06) -7.1%
7519.11: add (fsmonitor=.git/hooks/fsmonitor-watchman) 2.75(1.41+1.27) 2.03(1.26+0.70) -26.2%
7519.13: status (fsmonitor=) 1.38(1.03+1.04) 1.37(1.04+1.04) -0.7%
7519.14: status -uno (fsmonitor=) 1.11(0.83+0.98) 1.10(0.89+0.90) -0.9%
7519.15: status -uall (fsmonitor=) 2.30(1.57+1.42) 2.31(1.49+1.50) +0.4%
7519.16: diff (fsmonitor=) 1.43(1.13+1.76) 1.46(1.19+1.72) +2.1%
7519.17: diff -- 0_files (fsmonitor=) 0.10(0.08+0.04) 0.11(0.08+0.04) +10.0%
7519.18: diff -- 10_files (fsmonitor=) 0.10(0.07+0.05) 0.11(0.08+0.04) +10.0%
7519.19: diff -- 100_files (fsmonitor=) 0.10(0.07+0.04) 0.11(0.07+0.05) +10.0%
7519.20: diff -- 1000_files (fsmonitor=) 0.10(0.08+0.03) 0.11(0.08+0.04) +10.0%
7519.21: diff -- 10000_files (fsmonitor=) 0.11(0.08+0.05) 0.12(0.07+0.06) +9.1%
7519.22: add (fsmonitor=) 2.26(1.46+1.49) 2.27(1.42+1.55) +0.4%
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Much of the benchmark code is redundant. This is
easier to understand and edit.
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Results for the git-diff fsmonitor optimization
in patch in the parent-rev (using a 400k file repo to test)
As you can see here - git diff with fsmonitor running is
significantly better with this patch series (80% faster on my
workload)!
GIT_PERF_LARGE_REPO=~/src/server ./run v2.29.0-rc1 . -- p7519-fsmonitor.sh
Test v2.29.0-rc1 this tree
-----------------------------------------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 1.46(0.82+0.64) 1.47(0.83+0.62) +0.7%
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.16(0.12+0.04) 0.17(0.12+0.05) +6.3%
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.36(0.73+0.62) 1.37(0.76+0.60) +0.7%
7519.5: diff (fsmonitor=.git/hooks/fsmonitor-watchman) 0.85(0.22+0.63) 0.14(0.10+0.05) -83.5%
7519.6: diff -- 0_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.08+0.05) 0.13(0.11+0.02) +8.3%
7519.7: diff -- 10_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.08+0.04) 0.13(0.09+0.04) +8.3%
7519.8: diff -- 100_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.07+0.05) 0.13(0.07+0.06) +8.3%
7519.9: diff -- 1000_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.12(0.09+0.04) 0.13(0.08+0.05) +8.3%
7519.10: diff -- 10000_files (fsmonitor=.git/hooks/fsmonitor-watchman) 0.14(0.09+0.05) 0.13(0.10+0.03) -7.1%
7519.12: status (fsmonitor=) 1.67(0.93+1.49) 1.67(0.99+1.42) +0.0%
7519.13: status -uno (fsmonitor=) 0.37(0.30+0.82) 0.37(0.33+0.79) +0.0%
7519.14: status -uall (fsmonitor=) 1.58(0.97+1.35) 1.57(0.86+1.45) -0.6%
7519.15: diff (fsmonitor=) 0.34(0.28+0.83) 0.34(0.27+0.83) +0.0%
7519.16: diff -- 0_files (fsmonitor=) 0.09(0.06+0.04) 0.09(0.08+0.02) +0.0%
7519.17: diff -- 10_files (fsmonitor=) 0.09(0.07+0.03) 0.09(0.06+0.05) +0.0%
7519.18: diff -- 100_files (fsmonitor=) 0.09(0.06+0.04) 0.09(0.06+0.04) +0.0%
7519.19: diff -- 1000_files (fsmonitor=) 0.09(0.06+0.04) 0.09(0.05+0.05) +0.0%
7519.20: diff -- 10000_files (fsmonitor=) 0.10(0.08+0.04) 0.10(0.06+0.05) +0.0%
I also added a benchmark for a tiny git diff workload w/ a pathspec.
I see an approximately .02 second overhead added w/ and w/o fsmonitor
From looking at these results, I suspected that refresh_fsmonitor
is already happening during git diff - independent of this patch
series' optimization. Confirmed that suspicion by breaking on
refresh_fsmonitor.
(gdb) bt [simplified]
0 refresh_fsmonitor at fsmonitor.c:176
1 ie_match_stat at read-cache.c:375
2 match_stat_with_submodule at diff-lib.c:237
4 builtin_diff_files at builtin/diff.c:260
5 cmd_diff at builtin/diff.c:541
6 run_builtin at git.c:450
7 handle_builtin at git.c:700
8 run_argv at git.c:767
9 cmd_main at git.c:898
10 main at common-main.c:52
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
The first git status would be inflated due to warming of
filesystem cache. This makes the results comparable.
Before
Test this tree
--------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 2.52(1.59+1.56)
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.18(0.12+0.06)
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.36(0.73+0.62)
7519.7: status (fsmonitor=) 0.69(0.52+0.90)
7519.8: status -uno (fsmonitor=) 0.37(0.28+0.81)
7519.9: status -uall (fsmonitor=) 1.53(0.93+1.32)
After
Test this tree
--------------------------------------------------------------------------------
7519.2: status (fsmonitor=.git/hooks/fsmonitor-watchman) 0.39(0.33+0.06)
7519.3: status -uno (fsmonitor=.git/hooks/fsmonitor-watchman) 0.17(0.13+0.05)
7519.4: status -uall (fsmonitor=.git/hooks/fsmonitor-watchman) 1.34(0.77+0.56)
7519.7: status (fsmonitor=) 0.70(0.53+0.90)
7519.8: status -uno (fsmonitor=) 0.37(0.32+0.78)
7519.9: status -uall (fsmonitor=) 1.55(1.01+1.25)
Signed-off-by: Nipunn Koorapati <nipunn@dropbox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
The return code of command -v with a non-existing command is 1 in bash
and 127 in dash. Use that return code directly to allow the script to
work with dash and without watchman (e.g. on Debian).
While at it stop redirecting the output. stderr is redirected to
/dev/null by test_lazy_prereq already, and stdout can actually be
useful -- the path of the found watchman executable is sent there, but
it's shown only if the script was run with --verbose.
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Acked-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
|
Add a test utility (test-drop-caches) that flushes all changes to disk
then drops file system cache on Windows, Linux, and OSX.
Add a perf test (p7519-fsmonitor.sh) for fsmonitor.
By default, the performance test will utilize the Watchman file system
monitor if it is installed. If Watchman is not installed, it will use a
dummy integration script that does not report any new or modified files.
The dummy script has very little overhead which provides optimistic results.
The performance test will also use the untracked cache feature if it is
available as fsmonitor uses it to speed up scanning for untracked files.
There are 4 environment variables that can be used to alter the default
behavior of the performance test:
GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache
GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex
GIT_PERF_7519_FSMONITOR: used to configure core.fsmonitor
GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests
The big win for using fsmonitor is the elimination of the need to scan the
working directory looking for changed and untracked files. If the file
information is all cached in RAM, the benefits are reduced.
Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
|
Nipunn Koorapati
Nguyễn Thái Ngọc Duy
René Scharfe
Ben Peart