diff options
Diffstat (limited to 'Documentation/technical')
40 files changed, 1667 insertions, 2641 deletions
diff --git a/Documentation/technical/api-allocation-growing.txt b/Documentation/technical/api-allocation-growing.txt deleted file mode 100644 index 5a59b54844..0000000000 --- a/Documentation/technical/api-allocation-growing.txt +++ /dev/null @@ -1,39 +0,0 @@ -allocation growing API -====================== - -Dynamically growing an array using realloc() is error prone and boring. - -Define your array with: - -* a pointer (`item`) that points at the array, initialized to `NULL` - (although please name the variable based on its contents, not on its - type); - -* an integer variable (`alloc`) that keeps track of how big the current - allocation is, initialized to `0`; - -* another integer variable (`nr`) to keep track of how many elements the - array currently has, initialized to `0`. - -Then before adding `n`th element to the item, call `ALLOC_GROW(item, n, -alloc)`. This ensures that the array can hold at least `n` elements by -calling `realloc(3)` and adjusting `alloc` variable. - ------------- -sometype *item; -size_t nr; -size_t alloc - -for (i = 0; i < nr; i++) - if (we like item[i] already) - return; - -/* we did not like any existing one, so add one */ -ALLOC_GROW(item, nr + 1, alloc); -item[nr++] = value you like; ------------- - -You are responsible for updating the `nr` variable. - -If you need to specify the number of elements to allocate explicitly -then use the macro `REALLOC_ARRAY(item, alloc)` instead of `ALLOC_GROW`. diff --git a/Documentation/technical/api-argv-array.txt b/Documentation/technical/api-argv-array.txt deleted file mode 100644 index 870c8edbfb..0000000000 --- a/Documentation/technical/api-argv-array.txt +++ /dev/null @@ -1,65 +0,0 @@ -argv-array API -============== - -The argv-array API allows one to dynamically build and store -NULL-terminated lists. An argv-array maintains the invariant that the -`argv` member always points to a non-NULL array, and that the array is -always NULL-terminated at the element pointed to by `argv[argc]`. This -makes the result suitable for passing to functions expecting to receive -argv from main(), or the link:api-run-command.html[run-command API]. - -The string-list API (documented in string-list.h) is similar, but cannot be -used for these purposes; instead of storing a straight string pointer, -it contains an item structure with a `util` field that is not compatible -with the traditional argv interface. - -Each `argv_array` manages its own memory. Any strings pushed into the -array are duplicated, and all memory is freed by argv_array_clear(). - -Data Structures ---------------- - -`struct argv_array`:: - - A single array. This should be initialized by assignment from - `ARGV_ARRAY_INIT`, or by calling `argv_array_init`. The `argv` - member contains the actual array; the `argc` member contains the - number of elements in the array, not including the terminating - NULL. - -Functions ---------- - -`argv_array_init`:: - Initialize an array. This is no different than assigning from - `ARGV_ARRAY_INIT`. - -`argv_array_push`:: - Push a copy of a string onto the end of the array. - -`argv_array_pushl`:: - Push a list of strings onto the end of the array. The arguments - should be a list of `const char *` strings, terminated by a NULL - argument. - -`argv_array_pushf`:: - Format a string and push it onto the end of the array. This is a - convenience wrapper combining `strbuf_addf` and `argv_array_push`. - -`argv_array_pushv`:: - Push a null-terminated array of strings onto the end of the array. - -`argv_array_pop`:: - Remove the final element from the array. If there are no - elements in the array, do nothing. - -`argv_array_clear`:: - Free all memory associated with the array and return it to the - initial, empty state. - -`argv_array_detach`:: - Disconnect the `argv` member from the `argv_array` struct and - return it. The caller is responsible for freeing the memory used - by the array, and by the strings it references. After detaching, - the `argv_array` is in a reinitialized state and can be pushed - into again. diff --git a/Documentation/technical/api-config.txt b/Documentation/technical/api-config.txt deleted file mode 100644 index fa39ac9d71..0000000000 --- a/Documentation/technical/api-config.txt +++ /dev/null @@ -1,319 +0,0 @@ -config API -========== - -The config API gives callers a way to access Git configuration files -(and files which have the same syntax). See linkgit:git-config[1] for a -discussion of the config file syntax. - -General Usage -------------- - -Config files are parsed linearly, and each variable found is passed to a -caller-provided callback function. The callback function is responsible -for any actions to be taken on the config option, and is free to ignore -some options. It is not uncommon for the configuration to be parsed -several times during the run of a Git program, with different callbacks -picking out different variables useful to themselves. - -A config callback function takes three parameters: - -- the name of the parsed variable. This is in canonical "flat" form: the - section, subsection, and variable segments will be separated by dots, - and the section and variable segments will be all lowercase. E.g., - `core.ignorecase`, `diff.SomeType.textconv`. - -- the value of the found variable, as a string. If the variable had no - value specified, the value will be NULL (typically this means it - should be interpreted as boolean true). - -- a void pointer passed in by the caller of the config API; this can - contain callback-specific data - -A config callback should return 0 for success, or -1 if the variable -could not be parsed properly. - -Basic Config Querying ---------------------- - -Most programs will simply want to look up variables in all config files -that Git knows about, using the normal precedence rules. To do this, -call `git_config` with a callback function and void data pointer. - -`git_config` will read all config sources in order of increasing -priority. Thus a callback should typically overwrite previously-seen -entries with new ones (e.g., if both the user-wide `~/.gitconfig` and -repo-specific `.git/config` contain `color.ui`, the config machinery -will first feed the user-wide one to the callback, and then the -repo-specific one; by overwriting, the higher-priority repo-specific -value is left at the end). - -The `config_with_options` function lets the caller examine config -while adjusting some of the default behavior of `git_config`. It should -almost never be used by "regular" Git code that is looking up -configuration variables. It is intended for advanced callers like -`git-config`, which are intentionally tweaking the normal config-lookup -process. It takes two extra parameters: - -`config_source`:: -If this parameter is non-NULL, it specifies the source to parse for -configuration, rather than looking in the usual files. See `struct -git_config_source` in `config.h` for details. Regular `git_config` defaults -to `NULL`. - -`opts`:: -Specify options to adjust the behavior of parsing config files. See `struct -config_options` in `config.h` for details. As an example: regular `git_config` -sets `opts.respect_includes` to `1` by default. - -Reading Specific Files ----------------------- - -To read a specific file in git-config format, use -`git_config_from_file`. This takes the same callback and data parameters -as `git_config`. - -Querying For Specific Variables -------------------------------- - -For programs wanting to query for specific variables in a non-callback -manner, the config API provides two functions `git_config_get_value` -and `git_config_get_value_multi`. They both read values from an internal -cache generated previously from reading the config files. - -`int git_config_get_value(const char *key, const char **value)`:: - - Finds the highest-priority value for the configuration variable `key`, - stores the pointer to it in `value` and returns 0. When the - configuration variable `key` is not found, returns 1 without touching - `value`. The caller should not free or modify `value`, as it is owned - by the cache. - -`const struct string_list *git_config_get_value_multi(const char *key)`:: - - Finds and returns the value list, sorted in order of increasing priority - for the configuration variable `key`. When the configuration variable - `key` is not found, returns NULL. The caller should not free or modify - the returned pointer, as it is owned by the cache. - -`void git_config_clear(void)`:: - - Resets and invalidates the config cache. - -The config API also provides type specific API functions which do conversion -as well as retrieval for the queried variable, including: - -`int git_config_get_int(const char *key, int *dest)`:: - - Finds and parses the value to an integer for the configuration variable - `key`. Dies on error; otherwise, stores the value of the parsed integer in - `dest` and returns 0. When the configuration variable `key` is not found, - returns 1 without touching `dest`. - -`int git_config_get_ulong(const char *key, unsigned long *dest)`:: - - Similar to `git_config_get_int` but for unsigned longs. - -`int git_config_get_bool(const char *key, int *dest)`:: - - Finds and parses the value into a boolean value, for the configuration - variable `key` respecting keywords like "true" and "false". Integer - values are converted into true/false values (when they are non-zero or - zero, respectively). Other values cause a die(). If parsing is successful, - stores the value of the parsed result in `dest` and returns 0. When the - configuration variable `key` is not found, returns 1 without touching - `dest`. - -`int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest)`:: - - Similar to `git_config_get_bool`, except that integers are copied as-is, - and `is_bool` flag is unset. - -`int git_config_get_maybe_bool(const char *key, int *dest)`:: - - Similar to `git_config_get_bool`, except that it returns -1 on error - rather than dying. - -`int git_config_get_string_const(const char *key, const char **dest)`:: - - Allocates and copies the retrieved string into the `dest` parameter for - the configuration variable `key`; if NULL string is given, prints an - error message and returns -1. When the configuration variable `key` is - not found, returns 1 without touching `dest`. - -`int git_config_get_string(const char *key, char **dest)`:: - - Similar to `git_config_get_string_const`, except that retrieved value - copied into the `dest` parameter is a mutable string. - -`int git_config_get_pathname(const char *key, const char **dest)`:: - - Similar to `git_config_get_string`, but expands `~` or `~user` into - the user's home directory when found at the beginning of the path. - -`git_die_config(const char *key, const char *err, ...)`:: - - First prints the error message specified by the caller in `err` and then - dies printing the line number and the file name of the highest priority - value for the configuration variable `key`. - -`void git_die_config_linenr(const char *key, const char *filename, int linenr)`:: - - Helper function which formats the die error message according to the - parameters entered. Used by `git_die_config()`. It can be used by callers - handling `git_config_get_value_multi()` to print the correct error message - for the desired value. - -See test-config.c for usage examples. - -Value Parsing Helpers ---------------------- - -To aid in parsing string values, the config API provides callbacks with -a number of helper functions, including: - -`git_config_int`:: -Parse the string to an integer, including unit factors. Dies on error; -otherwise, returns the parsed result. - -`git_config_ulong`:: -Identical to `git_config_int`, but for unsigned longs. - -`git_config_bool`:: -Parse a string into a boolean value, respecting keywords like "true" and -"false". Integer values are converted into true/false values (when they -are non-zero or zero, respectively). Other values cause a die(). If -parsing is successful, the return value is the result. - -`git_config_bool_or_int`:: -Same as `git_config_bool`, except that integers are returned as-is, and -an `is_bool` flag is unset. - -`git_parse_maybe_bool`:: -Same as `git_config_bool`, except that it returns -1 on error rather -than dying. - -`git_config_string`:: -Allocates and copies the value string into the `dest` parameter; if no -string is given, prints an error message and returns -1. - -`git_config_pathname`:: -Similar to `git_config_string`, but expands `~` or `~user` into the -user's home directory when found at the beginning of the path. - -Include Directives ------------------- - -By default, the config parser does not respect include directives. -However, a caller can use the special `git_config_include` wrapper -callback to support them. To do so, you simply wrap your "real" callback -function and data pointer in a `struct config_include_data`, and pass -the wrapper to the regular config-reading functions. For example: - -------------------------------------------- -int read_file_with_include(const char *file, config_fn_t fn, void *data) -{ - struct config_include_data inc = CONFIG_INCLUDE_INIT; - inc.fn = fn; - inc.data = data; - return git_config_from_file(git_config_include, file, &inc); -} -------------------------------------------- - -`git_config` respects includes automatically. The lower-level -`git_config_from_file` does not. - -Custom Configsets ------------------ - -A `config_set` can be used to construct an in-memory cache for -config-like files that the caller specifies (i.e., files like `.gitmodules`, -`~/.gitconfig` etc.). For example, - ---------------------------------------- -struct config_set gm_config; -git_configset_init(&gm_config); -int b; -/* we add config files to the config_set */ -git_configset_add_file(&gm_config, ".gitmodules"); -git_configset_add_file(&gm_config, ".gitmodules_alt"); - -if (!git_configset_get_bool(gm_config, "submodule.frotz.ignore", &b)) { - /* hack hack hack */ -} - -/* when we are done with the configset */ -git_configset_clear(&gm_config); ----------------------------------------- - -Configset API provides functions for the above mentioned work flow, including: - -`void git_configset_init(struct config_set *cs)`:: - - Initializes the config_set `cs`. - -`int git_configset_add_file(struct config_set *cs, const char *filename)`:: - - Parses the file and adds the variable-value pairs to the `config_set`, - dies if there is an error in parsing the file. Returns 0 on success, or - -1 if the file does not exist or is inaccessible. The user has to decide - if he wants to free the incomplete configset or continue using it when - the function returns -1. - -`int git_configset_get_value(struct config_set *cs, const char *key, const char **value)`:: - - Finds the highest-priority value for the configuration variable `key` - and config set `cs`, stores the pointer to it in `value` and returns 0. - When the configuration variable `key` is not found, returns 1 without - touching `value`. The caller should not free or modify `value`, as it - is owned by the cache. - -`const struct string_list *git_configset_get_value_multi(struct config_set *cs, const char *key)`:: - - Finds and returns the value list, sorted in order of increasing priority - for the configuration variable `key` and config set `cs`. When the - configuration variable `key` is not found, returns NULL. The caller - should not free or modify the returned pointer, as it is owned by the cache. - -`void git_configset_clear(struct config_set *cs)`:: - - Clears `config_set` structure, removes all saved variable-value pairs. - -In addition to above functions, the `config_set` API provides type specific -functions in the vein of `git_config_get_int` and family but with an extra -parameter, pointer to struct `config_set`. -They all behave similarly to the `git_config_get*()` family described in -"Querying For Specific Variables" above. - -Writing Config Files --------------------- - -Git gives multiple entry points in the Config API to write config values to -files namely `git_config_set_in_file` and `git_config_set`, which write to -a specific config file or to `.git/config` respectively. They both take a -key/value pair as parameter. -In the end they both call `git_config_set_multivar_in_file` which takes four -parameters: - -- the name of the file, as a string, to which key/value pairs will be written. - -- the name of key, as a string. This is in canonical "flat" form: the section, - subsection, and variable segments will be separated by dots, and the section - and variable segments will be all lowercase. - E.g., `core.ignorecase`, `diff.SomeType.textconv`. - -- the value of the variable, as a string. If value is equal to NULL, it will - remove the matching key from the config file. - -- the value regex, as a string. It will disregard key/value pairs where value - does not match. - -- a multi_replace value, as an int. If value is equal to zero, nothing or only - one matching key/value is replaced, else all matching key/values (regardless - how many) are removed, before the new pair is written. - -It returns 0 on success. - -Also, there are functions `git_config_rename_section` and -`git_config_rename_section_in_file` with parameters `old_name` and `new_name` -for renaming or removing sections in the config files. If NULL is passed -through `new_name` parameter, the section will be removed from the config file. diff --git a/Documentation/technical/api-credentials.txt b/Documentation/technical/api-credentials.txt deleted file mode 100644 index 75368f26ca..0000000000 --- a/Documentation/technical/api-credentials.txt +++ /dev/null @@ -1,271 +0,0 @@ -credentials API -=============== - -The credentials API provides an abstracted way of gathering username and -password credentials from the user (even though credentials in the wider -world can take many forms, in this document the word "credential" always -refers to a username and password pair). - -This document describes two interfaces: the C API that the credential -subsystem provides to the rest of Git, and the protocol that Git uses to -communicate with system-specific "credential helpers". If you are -writing Git code that wants to look up or prompt for credentials, see -the section "C API" below. If you want to write your own helper, see -the section on "Credential Helpers" below. - -Typical setup -------------- - ------------- -+-----------------------+ -| Git code (C) |--- to server requiring ---> -| | authentication -|.......................| -| C credential API |--- prompt ---> User -+-----------------------+ - ^ | - | pipe | - | v -+-----------------------+ -| Git credential helper | -+-----------------------+ ------------- - -The Git code (typically a remote-helper) will call the C API to obtain -credential data like a login/password pair (credential_fill). The -API will itself call a remote helper (e.g. "git credential-cache" or -"git credential-store") that may retrieve credential data from a -store. If the credential helper cannot find the information, the C API -will prompt the user. Then, the caller of the API takes care of -contacting the server, and does the actual authentication. - -C API ------ - -The credential C API is meant to be called by Git code which needs to -acquire or store a credential. It is centered around an object -representing a single credential and provides three basic operations: -fill (acquire credentials by calling helpers and/or prompting the user), -approve (mark a credential as successfully used so that it can be stored -for later use), and reject (mark a credential as unsuccessful so that it -can be erased from any persistent storage). - -Data Structures -~~~~~~~~~~~~~~~ - -`struct credential`:: - - This struct represents a single username/password combination - along with any associated context. All string fields should be - heap-allocated (or NULL if they are not known or not applicable). - The meaning of the individual context fields is the same as - their counterparts in the helper protocol; see the section below - for a description of each field. -+ -The `helpers` member of the struct is a `string_list` of helpers. Each -string specifies an external helper which will be run, in order, to -either acquire or store credentials. See the section on credential -helpers below. This list is filled-in by the API functions -according to the corresponding configuration variables before -consulting helpers, so there usually is no need for a caller to -modify the helpers field at all. -+ -This struct should always be initialized with `CREDENTIAL_INIT` or -`credential_init`. - - -Functions -~~~~~~~~~ - -`credential_init`:: - - Initialize a credential structure, setting all fields to empty. - -`credential_clear`:: - - Free any resources associated with the credential structure, - returning it to a pristine initialized state. - -`credential_fill`:: - - Instruct the credential subsystem to fill the username and - password fields of the passed credential struct by first - consulting helpers, then asking the user. After this function - returns, the username and password fields of the credential are - guaranteed to be non-NULL. If an error occurs, the function will - die(). - -`credential_reject`:: - - Inform the credential subsystem that the provided credentials - have been rejected. This will cause the credential subsystem to - notify any helpers of the rejection (which allows them, for - example, to purge the invalid credentials from storage). It - will also free() the username and password fields of the - credential and set them to NULL (readying the credential for - another call to `credential_fill`). Any errors from helpers are - ignored. - -`credential_approve`:: - - Inform the credential subsystem that the provided credentials - were successfully used for authentication. This will cause the - credential subsystem to notify any helpers of the approval, so - that they may store the result to be used again. Any errors - from helpers are ignored. - -`credential_from_url`:: - - Parse a URL into broken-down credential fields. - -Example -~~~~~~~ - -The example below shows how the functions of the credential API could be -used to login to a fictitious "foo" service on a remote host: - ------------------------------------------------------------------------ -int foo_login(struct foo_connection *f) -{ - int status; - /* - * Create a credential with some context; we don't yet know the - * username or password. - */ - - struct credential c = CREDENTIAL_INIT; - c.protocol = xstrdup("foo"); - c.host = xstrdup(f->hostname); - - /* - * Fill in the username and password fields by contacting - * helpers and/or asking the user. The function will die if it - * fails. - */ - credential_fill(&c); - - /* - * Otherwise, we have a username and password. Try to use it. - */ - status = send_foo_login(f, c.username, c.password); - switch (status) { - case FOO_OK: - /* It worked. Store the credential for later use. */ - credential_accept(&c); - break; - case FOO_BAD_LOGIN: - /* Erase the credential from storage so we don't try it - * again. */ - credential_reject(&c); - break; - default: - /* - * Some other error occurred. We don't know if the - * credential is good or bad, so report nothing to the - * credential subsystem. - */ - } - - /* Free any associated resources. */ - credential_clear(&c); - - return status; -} ------------------------------------------------------------------------ - - -Credential Helpers ------------------- - -Credential helpers are programs executed by Git to fetch or save -credentials from and to long-term storage (where "long-term" is simply -longer than a single Git process; e.g., credentials may be stored -in-memory for a few minutes, or indefinitely on disk). - -Each helper is specified by a single string in the configuration -variable `credential.helper` (and others, see linkgit:git-config[1]). -The string is transformed by Git into a command to be executed using -these rules: - - 1. If the helper string begins with "!", it is considered a shell - snippet, and everything after the "!" becomes the command. - - 2. Otherwise, if the helper string begins with an absolute path, the - verbatim helper string becomes the command. - - 3. Otherwise, the string "git credential-" is prepended to the helper - string, and the result becomes the command. - -The resulting command then has an "operation" argument appended to it -(see below for details), and the result is executed by the shell. - -Here are some example specifications: - ----------------------------------------------------- -# run "git credential-foo" -foo - -# same as above, but pass an argument to the helper -foo --bar=baz - -# the arguments are parsed by the shell, so use shell -# quoting if necessary -foo --bar="whitespace arg" - -# you can also use an absolute path, which will not use the git wrapper -/path/to/my/helper --with-arguments - -# or you can specify your own shell snippet -!f() { echo "password=`cat $HOME/.secret`"; }; f ----------------------------------------------------- - -Generally speaking, rule (3) above is the simplest for users to specify. -Authors of credential helpers should make an effort to assist their -users by naming their program "git-credential-$NAME", and putting it in -the $PATH or $GIT_EXEC_PATH during installation, which will allow a user -to enable it with `git config credential.helper $NAME`. - -When a helper is executed, it will have one "operation" argument -appended to its command line, which is one of: - -`get`:: - - Return a matching credential, if any exists. - -`store`:: - - Store the credential, if applicable to the helper. - -`erase`:: - - Remove a matching credential, if any, from the helper's storage. - -The details of the credential will be provided on the helper's stdin -stream. The exact format is the same as the input/output format of the -`git credential` plumbing command (see the section `INPUT/OUTPUT -FORMAT` in linkgit:git-credential[1] for a detailed specification). - -For a `get` operation, the helper should produce a list of attributes -on stdout in the same format. A helper is free to produce a subset, or -even no values at all if it has nothing useful to provide. Any provided -attributes will overwrite those already known about by Git. If a helper -outputs a `quit` attribute with a value of `true` or `1`, no further -helpers will be consulted, nor will the user be prompted (if no -credential has been provided, the operation will then fail). - -For a `store` or `erase` operation, the helper's output is ignored. -If it fails to perform the requested operation, it may complain to -stderr to inform the user. If it does not support the requested -operation (e.g., a read-only store), it should silently ignore the -request. - -If a helper receives any other operation, it should silently ignore the -request. This leaves room for future operations to be added (older -helpers will just ignore the new requests). - -See also --------- - -linkgit:gitcredentials[7] - -linkgit:git-config[1] (See configuration variables `credential.*`) diff --git a/Documentation/technical/api-diff.txt b/Documentation/technical/api-diff.txt deleted file mode 100644 index 30fc0e9c93..0000000000 --- a/Documentation/technical/api-diff.txt +++ /dev/null @@ -1,174 +0,0 @@ -diff API -======== - -The diff API is for programs that compare two sets of files (e.g. two -trees, one tree and the index) and present the found difference in -various ways. The calling program is responsible for feeding the API -pairs of files, one from the "old" set and the corresponding one from -"new" set, that are different. The library called through this API is -called diffcore, and is responsible for two things. - -* finding total rewrites (`-B`), renames (`-M`) and copies (`-C`), and - changes that touch a string (`-S`), as specified by the caller. - -* outputting the differences in various formats, as specified by the - caller. - -Calling sequence ----------------- - -* Prepare `struct diff_options` to record the set of diff options, and - then call `repo_diff_setup()` to initialize this structure. This - sets up the vanilla default. - -* Fill in the options structure to specify desired output format, rename - detection, etc. `diff_opt_parse()` can be used to parse options given - from the command line in a way consistent with existing git-diff - family of programs. - -* Call `diff_setup_done()`; this inspects the options set up so far for - internal consistency and make necessary tweaking to it (e.g. if - textual patch output was asked, recursive behaviour is turned on); - the callback set_default in diff_options can be used to tweak this more. - -* As you find different pairs of files, call `diff_change()` to feed - modified files, `diff_addremove()` to feed created or deleted files, - or `diff_unmerge()` to feed a file whose state is 'unmerged' to the - API. These are thin wrappers to a lower-level `diff_queue()` function - that is flexible enough to record any of these kinds of changes. - -* Once you finish feeding the pairs of files, call `diffcore_std()`. - This will tell the diffcore library to go ahead and do its work. - -* Calling `diff_flush()` will produce the output. - - -Data structures ---------------- - -* `struct diff_filespec` - -This is the internal representation for a single file (blob). It -records the blob object name (if known -- for a work tree file it -typically is a NUL SHA-1), filemode and pathname. This is what the -`diff_addremove()`, `diff_change()` and `diff_unmerge()` synthesize and -feed `diff_queue()` function with. - -* `struct diff_filepair` - -This records a pair of `struct diff_filespec`; the filespec for a file -in the "old" set (i.e. preimage) is called `one`, and the filespec for a -file in the "new" set (i.e. postimage) is called `two`. A change that -represents file creation has NULL in `one`, and file deletion has NULL -in `two`. - -A `filepair` starts pointing at `one` and `two` that are from the same -filename, but `diffcore_std()` can break pairs and match component -filespecs with other filespecs from a different filepair to form new -filepair. This is called 'rename detection'. - -* `struct diff_queue` - -This is a collection of filepairs. Notable members are: - -`queue`:: - - An array of pointers to `struct diff_filepair`. This - dynamically grows as you add filepairs; - -`alloc`:: - - The allocated size of the `queue` array; - -`nr`:: - - The number of elements in the `queue` array. - - -* `struct diff_options` - -This describes the set of options the calling program wants to affect -the operation of diffcore library with. - -Notable members are: - -`output_format`:: - The output format used when `diff_flush()` is run. - -`context`:: - Number of context lines to generate in patch output. - -`break_opt`, `detect_rename`, `rename-score`, `rename_limit`:: - Affects the way detection logic for complete rewrites, renames - and copies. - -`abbrev`:: - Number of hexdigits to abbreviate raw format output to. - -`pickaxe`:: - A constant string (can and typically does contain newlines to - look for a block of text, not just a single line) to filter out - the filepairs that do not change the number of strings contained - in its preimage and postimage of the diff_queue. - -`flags`:: - This is mostly a collection of boolean options that affects the - operation, but some do not have anything to do with the diffcore - library. - -`touched_flags`:: - Records whether a flag has been changed due to user request - (rather than just set/unset by default). - -`set_default`:: - Callback which allows tweaking the options in diff_setup_done(). - -BINARY, TEXT;; - Affects the way how a file that is seemingly binary is treated. - -FULL_INDEX;; - Tells the patch output format not to use abbreviated object - names on the "index" lines. - -FIND_COPIES_HARDER;; - Tells the diffcore library that the caller is feeding unchanged - filepairs to allow copies from unmodified files be detected. - -COLOR_DIFF;; - Output should be colored. - -COLOR_DIFF_WORDS;; - Output is a colored word-diff. - -NO_INDEX;; - Tells diff-files that the input is not tracked files but files - in random locations on the filesystem. - -ALLOW_EXTERNAL;; - Tells output routine that it is Ok to call user specified patch - output routine. Plumbing disables this to ensure stable output. - -QUIET;; - Do not show any output. - -REVERSE_DIFF;; - Tells the library that the calling program is feeding the - filepairs reversed; `one` is two, and `two` is one. - -EXIT_WITH_STATUS;; - For communication between the calling program and the options - parser; tell the calling program to signal the presence of - difference using program exit code. - -HAS_CHANGES;; - Internal; used for optimization to see if there is any change. - -SILENT_ON_REMOVE;; - Affects if diff-files shows removed files. - -RECURSIVE, TREE_IN_RECURSIVE;; - Tells if tree traversal done by tree-diff should recursively - descend into a tree object pair that are different in preimage - and postimage set. - -(JC) diff --git a/Documentation/technical/api-directory-listing.txt b/Documentation/technical/api-directory-listing.txt deleted file mode 100644 index 5abb8e8b1f..0000000000 --- a/Documentation/technical/api-directory-listing.txt +++ /dev/null @@ -1,130 +0,0 @@ -directory listing API -===================== - -The directory listing API is used to enumerate paths in the work tree, -optionally taking `.git/info/exclude` and `.gitignore` files per -directory into account. - -Data structure --------------- - -`struct dir_struct` structure is used to pass directory traversal -options to the library and to record the paths discovered. A single -`struct dir_struct` is used regardless of whether or not the traversal -recursively descends into subdirectories. - -The notable options are: - -`exclude_per_dir`:: - - The name of the file to be read in each directory for excluded - files (typically `.gitignore`). - -`flags`:: - - A bit-field of options: - -`DIR_SHOW_IGNORED`::: - - Return just ignored files in `entries[]`, not untracked - files. This flag is mutually exclusive with - `DIR_SHOW_IGNORED_TOO`. - -`DIR_SHOW_IGNORED_TOO`::: - - Similar to `DIR_SHOW_IGNORED`, but return ignored files in - `ignored[]` in addition to untracked files in - `entries[]`. This flag is mutually exclusive with - `DIR_SHOW_IGNORED`. - -`DIR_KEEP_UNTRACKED_CONTENTS`::: - - Only has meaning if `DIR_SHOW_IGNORED_TOO` is also set; if this is set, the - untracked contents of untracked directories are also returned in - `entries[]`. - -`DIR_SHOW_IGNORED_TOO_MODE_MATCHING`::: - - Only has meaning if `DIR_SHOW_IGNORED_TOO` is also set; if - this is set, returns ignored files and directories that match - an exclude pattern. If a directory matches an exclude pattern, - then the directory is returned and the contained paths are - not. A directory that does not match an exclude pattern will - not be returned even if all of its contents are ignored. In - this case, the contents are returned as individual entries. -+ -If this is set, files and directories that explicitly match an ignore -pattern are reported. Implicitly ignored directories (directories that -do not match an ignore pattern, but whose contents are all ignored) -are not reported, instead all of the contents are reported. - -`DIR_COLLECT_IGNORED`::: - - Special mode for git-add. Return ignored files in `ignored[]` and - untracked files in `entries[]`. Only returns ignored files that match - pathspec exactly (no wildcards). Does not recurse into ignored - directories. - -`DIR_SHOW_OTHER_DIRECTORIES`::: - - Include a directory that is not tracked. - -`DIR_HIDE_EMPTY_DIRECTORIES`::: - - Do not include a directory that is not tracked and is empty. - -`DIR_NO_GITLINKS`::: - - If set, recurse into a directory that looks like a Git - directory. Otherwise it is shown as a directory. - -The result of the enumeration is left in these fields: - -`entries[]`:: - - An array of `struct dir_entry`, each element of which describes - a path. - -`nr`:: - - The number of members in `entries[]` array. - -`alloc`:: - - Internal use; keeps track of allocation of `entries[]` array. - -`ignored[]`:: - - An array of `struct dir_entry`, used for ignored paths with the - `DIR_SHOW_IGNORED_TOO` and `DIR_COLLECT_IGNORED` flags. - -`ignored_nr`:: - - The number of members in `ignored[]` array. - -Calling sequence ----------------- - -Note: index may be looked at for .gitignore files that are CE_SKIP_WORKTREE -marked. If you to exclude files, make sure you have loaded index first. - -* Prepare `struct dir_struct dir` and clear it with `memset(&dir, 0, - sizeof(dir))`. - -* To add single exclude pattern, call `add_exclude_list()` and then - `add_exclude()`. - -* To add patterns from a file (e.g. `.git/info/exclude`), call - `add_excludes_from_file()` , and/or set `dir.exclude_per_dir`. A - short-hand function `setup_standard_excludes()` can be used to set - up the standard set of exclude settings. - -* Set options described in the Data Structure section above. - -* Call `read_directory()`. - -* Use `dir.entries[]`. - -* Call `clear_directory()` when none of the contained elements are no longer in use. - -(JC) diff --git a/Documentation/technical/api-gitattributes.txt b/Documentation/technical/api-gitattributes.txt deleted file mode 100644 index 45f0df600f..0000000000 --- a/Documentation/technical/api-gitattributes.txt +++ /dev/null @@ -1,154 +0,0 @@ -gitattributes API -================= - -gitattributes mechanism gives a uniform way to associate various -attributes to set of paths. - - -Data Structure --------------- - -`struct git_attr`:: - - An attribute is an opaque object that is identified by its name. - Pass the name to `git_attr()` function to obtain the object of - this type. The internal representation of this structure is - of no interest to the calling programs. The name of the - attribute can be retrieved by calling `git_attr_name()`. - -`struct attr_check_item`:: - - This structure represents one attribute and its value. - -`struct attr_check`:: - - This structure represents a collection of `attr_check_item`. - It is passed to `git_check_attr()` function, specifying the - attributes to check, and receives their values. - - -Attribute Values ----------------- - -An attribute for a path can be in one of four states: Set, Unset, -Unspecified or set to a string, and `.value` member of `struct -attr_check_item` records it. There are three macros to check these: - -`ATTR_TRUE()`:: - - Returns true if the attribute is Set for the path. - -`ATTR_FALSE()`:: - - Returns true if the attribute is Unset for the path. - -`ATTR_UNSET()`:: - - Returns true if the attribute is Unspecified for the path. - -If none of the above returns true, `.value` member points at a string -value of the attribute for the path. - - -Querying Specific Attributes ----------------------------- - -* Prepare `struct attr_check` using attr_check_initl() - function, enumerating the names of attributes whose values you are - interested in, terminated with a NULL pointer. Alternatively, an - empty `struct attr_check` can be prepared by calling - `attr_check_alloc()` function and then attributes you want to - ask about can be added to it with `attr_check_append()` - function. - -* Call `git_check_attr()` to check the attributes for the path. - -* Inspect `attr_check` structure to see how each of the - attribute in the array is defined for the path. - - -Example -------- - -To see how attributes "crlf" and "ident" are set for different paths. - -. Prepare a `struct attr_check` with two elements (because - we are checking two attributes): - ------------- -static struct attr_check *check; -static void setup_check(void) -{ - if (check) - return; /* already done */ - check = attr_check_initl("crlf", "ident", NULL); -} ------------- - -. Call `git_check_attr()` with the prepared `struct attr_check`: - ------------- - const char *path; - - setup_check(); - git_check_attr(path, check); ------------- - -. Act on `.value` member of the result, left in `check->items[]`: - ------------- - const char *value = check->items[0].value; - - if (ATTR_TRUE(value)) { - The attribute is Set, by listing only the name of the - attribute in the gitattributes file for the path. - } else if (ATTR_FALSE(value)) { - The attribute is Unset, by listing the name of the - attribute prefixed with a dash - for the path. - } else if (ATTR_UNSET(value)) { - The attribute is neither set nor unset for the path. - } else if (!strcmp(value, "input")) { - If none of ATTR_TRUE(), ATTR_FALSE(), or ATTR_UNSET() is - true, the value is a string set in the gitattributes - file for the path by saying "attr=value". - } else if (... other check using value as string ...) { - ... - } ------------- - -To see how attributes in argv[] are set for different paths, only -the first step in the above would be different. - ------------- -static struct attr_check *check; -static void setup_check(const char **argv) -{ - check = attr_check_alloc(); - while (*argv) { - struct git_attr *attr = git_attr(*argv); - attr_check_append(check, attr); - argv++; - } -} ------------- - - -Querying All Attributes ------------------------ - -To get the values of all attributes associated with a file: - -* Prepare an empty `attr_check` structure by calling - `attr_check_alloc()`. - -* Call `git_all_attrs()`, which populates the `attr_check` - with the attributes attached to the path. - -* Iterate over the `attr_check.items[]` array to examine - the attribute names and values. The name of the attribute - described by an `attr_check.items[]` object can be retrieved via - `git_attr_name(check->items[i].attr)`. (Please note that no items - will be returned for unset attributes, so `ATTR_UNSET()` will return - false for all returned `attr_check.items[]` objects.) - -* Free the `attr_check` struct by calling `attr_check_free()`. diff --git a/Documentation/technical/api-grep.txt b/Documentation/technical/api-grep.txt deleted file mode 100644 index a69cc8964d..0000000000 --- a/Documentation/technical/api-grep.txt +++ /dev/null @@ -1,8 +0,0 @@ -grep API -======== - -Talk about <grep.h>, things like: - -* grep_buffer() - -(JC) diff --git a/Documentation/technical/api-history-graph.txt b/Documentation/technical/api-history-graph.txt deleted file mode 100644 index d0d1707c8c..0000000000 --- a/Documentation/technical/api-history-graph.txt +++ /dev/null @@ -1,173 +0,0 @@ -history graph API -================= - -The graph API is used to draw a text-based representation of the commit -history. The API generates the graph in a line-by-line fashion. - -Functions ---------- - -Core functions: - -* `graph_init()` creates a new `struct git_graph` - -* `graph_update()` moves the graph to a new commit. - -* `graph_next_line()` outputs the next line of the graph into a strbuf. It - does not add a terminating newline. - -* `graph_padding_line()` outputs a line of vertical padding in the graph. It - is similar to `graph_next_line()`, but is guaranteed to never print the line - containing the current commit. Where `graph_next_line()` would print the - commit line next, `graph_padding_line()` prints a line that simply extends - all branch lines downwards one row, leaving their positions unchanged. - -* `graph_is_commit_finished()` determines if the graph has output all lines - necessary for the current commit. If `graph_update()` is called before all - lines for the current commit have been printed, the next call to - `graph_next_line()` will output an ellipsis, to indicate that a portion of - the graph was omitted. - -The following utility functions are wrappers around `graph_next_line()` and -`graph_is_commit_finished()`. They always print the output to stdout. -They can all be called with a NULL graph argument, in which case no graph -output will be printed. - -* `graph_show_commit()` calls `graph_next_line()` and - `graph_is_commit_finished()` until one of them return non-zero. This prints - all graph lines up to, and including, the line containing this commit. - Output is printed to stdout. The last line printed does not contain a - terminating newline. - -* `graph_show_oneline()` calls `graph_next_line()` and prints the result to - stdout. The line printed does not contain a terminating newline. - -* `graph_show_padding()` calls `graph_padding_line()` and prints the result to - stdout. The line printed does not contain a terminating newline. - -* `graph_show_remainder()` calls `graph_next_line()` until - `graph_is_commit_finished()` returns non-zero. Output is printed to stdout. - The last line printed does not contain a terminating newline. Returns 1 if - output was printed, and 0 if no output was necessary. - -* `graph_show_strbuf()` prints the specified strbuf to stdout, prefixing all - lines but the first with a graph line. The caller is responsible for - ensuring graph output for the first line has already been printed to stdout. - (This can be done with `graph_show_commit()` or `graph_show_oneline()`.) If - a NULL graph is supplied, the strbuf is printed as-is. - -* `graph_show_commit_msg()` is similar to `graph_show_strbuf()`, but it also - prints the remainder of the graph, if more lines are needed after the strbuf - ends. It is better than directly calling `graph_show_strbuf()` followed by - `graph_show_remainder()` since it properly handles buffers that do not end in - a terminating newline. The output printed by `graph_show_commit_msg()` will - end in a newline if and only if the strbuf ends in a newline. - -Data structure --------------- -`struct git_graph` is an opaque data type used to store the current graph -state. - -Calling sequence ----------------- - -* Create a `struct git_graph` by calling `graph_init()`. When using the - revision walking API, this is done automatically by `setup_revisions()` if - the '--graph' option is supplied. - -* Use the revision walking API to walk through a group of contiguous commits. - The `get_revision()` function automatically calls `graph_update()` each time - it is invoked. - -* For each commit, call `graph_next_line()` repeatedly, until - `graph_is_commit_finished()` returns non-zero. Each call to - `graph_next_line()` will output a single line of the graph. The resulting - lines will not contain any newlines. `graph_next_line()` returns 1 if the - resulting line contains the current commit, or 0 if this is merely a line - needed to adjust the graph before or after the current commit. This return - value can be used to determine where to print the commit summary information - alongside the graph output. - -Limitations ------------ - -* `graph_update()` must be called with commits in topological order. It should - not be called on a commit if it has already been invoked with an ancestor of - that commit, or the graph output will be incorrect. - -* `graph_update()` must be called on a contiguous group of commits. If - `graph_update()` is called on a particular commit, it should later be called - on all parents of that commit. Parents must not be skipped, or the graph - output will appear incorrect. -+ -`graph_update()` may be used on a pruned set of commits only if the parent list -has been rewritten so as to include only ancestors from the pruned set. - -* The graph API does not currently support reverse commit ordering. In - order to implement reverse ordering, the graphing API needs an - (efficient) mechanism to find the children of a commit. - -Sample usage ------------- - ------------- -struct commit *commit; -struct git_graph *graph = graph_init(opts); - -while ((commit = get_revision(opts)) != NULL) { - while (!graph_is_commit_finished(graph)) - { - struct strbuf sb; - int is_commit_line; - - strbuf_init(&sb, 0); - is_commit_line = graph_next_line(graph, &sb); - fputs(sb.buf, stdout); - - if (is_commit_line) - log_tree_commit(opts, commit); - else - putchar(opts->diffopt.line_termination); - } -} ------------- - -Sample output -------------- - -The following is an example of the output from the graph API. This output does -not include any commit summary information--callers are responsible for -outputting that information, if desired. - ------------- -* -* -* -|\ -* | -| | * -| \ \ -| \ \ -*-. \ \ -|\ \ \ \ -| | * | | -| | | | | * -| | | | | * -| | | | | * -| | | | | |\ -| | | | | | * -| * | | | | | -| | | | | * \ -| | | | | |\ | -| | | | * | | | -| | | | * | | | -* | | | | | | | -| |/ / / / / / -|/| / / / / / -* | | | | | | -|/ / / / / / -* | | | | | -| | | | | * -| | | | |/ -| | | | * ------------- diff --git a/Documentation/technical/api-merge.txt b/Documentation/technical/api-merge.txt index 9dc1bed768..487d4d83ff 100644 --- a/Documentation/technical/api-merge.txt +++ b/Documentation/technical/api-merge.txt @@ -28,77 +28,9 @@ and `diff.c` for examples. * `struct ll_merge_options` -This describes the set of options the calling program wants to affect -the operation of a low-level (single file) merge. Some options: - -`virtual_ancestor`:: - Behave as though this were part of a merge between common - ancestors in a recursive merge. - If a helper program is specified by the - `[merge "<driver>"] recursive` configuration, it will - be used (see linkgit:gitattributes[5]). - -`variant`:: - Resolve local conflicts automatically in favor - of one side or the other (as in 'git merge-file' - `--ours`/`--theirs`/`--union`). Can be `0`, - `XDL_MERGE_FAVOR_OURS`, `XDL_MERGE_FAVOR_THEIRS`, or - `XDL_MERGE_FAVOR_UNION`. - -`renormalize`:: - Resmudge and clean the "base", "theirs" and "ours" files - before merging. Use this when the merge is likely to have - overlapped with a change in smudge/clean or end-of-line - normalization rules. +Check ll-merge.h for details. Low-level (single file) merge ----------------------------- -`ll_merge`:: - - Perform a three-way single-file merge in core. This is - a thin wrapper around `xdl_merge` that takes the path and - any merge backend specified in `.gitattributes` or - `.git/info/attributes` into account. Returns 0 for a - clean merge. - -Calling sequence: - -* Prepare a `struct ll_merge_options` to record options. - If you have no special requests, skip this and pass `NULL` - as the `opts` parameter to use the default options. - -* Allocate an mmbuffer_t variable for the result. - -* Allocate and fill variables with the file's original content - and two modified versions (using `read_mmfile`, for example). - -* Call `ll_merge()`. - -* Read the merged content from `result_buf.ptr` and `result_buf.size`. - -* Release buffers when finished. A simple - `free(ancestor.ptr); free(ours.ptr); free(theirs.ptr); - free(result_buf.ptr);` will do. - -If the modifications do not merge cleanly, `ll_merge` will return a -nonzero value and `result_buf` will generally include a description of -the conflict bracketed by markers such as the traditional `<<<<<<<` -and `>>>>>>>`. - -The `ancestor_label`, `our_label`, and `their_label` parameters are -used to label the different sides of a conflict if the merge driver -supports this. - -Everything else ---------------- - -Talk about <merge-recursive.h> and merge_file(): - - - merge_trees() to merge with rename detection - - merge_recursive() for ancestor consolidation - - try_merge_command() for other strategies - - conflict format - - merge options - -(Daniel, Miklos, Stephan, JC) +Check ll-merge.h for details. diff --git a/Documentation/technical/api-object-access.txt b/Documentation/technical/api-object-access.txt deleted file mode 100644 index 5b29622d00..0000000000 --- a/Documentation/technical/api-object-access.txt +++ /dev/null @@ -1,15 +0,0 @@ -object access API -================= - -Talk about <sha1-file.c> and <object.h> family, things like - -* read_sha1_file() -* read_object_with_reference() -* has_sha1_file() -* write_sha1_file() -* pretend_object_file() -* lookup_{object,commit,tag,blob,tree} -* parse_{object,commit,tag,blob,tree} -* Use of object flags - -(JC, Shawn, Daniel, Dscho, Linus) diff --git a/Documentation/technical/api-oid-array.txt b/Documentation/technical/api-oid-array.txt deleted file mode 100644 index 9febfb1d52..0000000000 --- a/Documentation/technical/api-oid-array.txt +++ /dev/null @@ -1,85 +0,0 @@ -oid-array API -============== - -The oid-array API provides storage and manipulation of sets of object -identifiers. The emphasis is on storage and processing efficiency, -making them suitable for large lists. Note that the ordering of items is -not preserved over some operations. - -Data Structures ---------------- - -`struct oid_array`:: - - A single array of object IDs. This should be initialized by - assignment from `OID_ARRAY_INIT`. The `oid` member contains - the actual data. The `nr` member contains the number of items in - the set. The `alloc` and `sorted` members are used internally, - and should not be needed by API callers. - -Functions ---------- - -`oid_array_append`:: - Add an item to the set. The object ID will be placed at the end of - the array (but note that some operations below may lose this - ordering). - -`oid_array_lookup`:: - Perform a binary search of the array for a specific object ID. - If found, returns the offset (in number of elements) of the - object ID. If not found, returns a negative integer. If the array - is not sorted, this function has the side effect of sorting it. - -`oid_array_clear`:: - Free all memory associated with the array and return it to the - initial, empty state. - -`oid_array_for_each`:: - Iterate over each element of the list, executing the callback - function for each one. Does not sort the list, so any custom - hash order is retained. If the callback returns a non-zero - value, the iteration ends immediately and the callback's - return is propagated; otherwise, 0 is returned. - -`oid_array_for_each_unique`:: - Iterate over each unique element of the list in sorted order, - but otherwise behave like `oid_array_for_each`. If the array - is not sorted, this function has the side effect of sorting - it. - -Examples --------- - ------------------------------------------ -int print_callback(const struct object_id *oid, - void *data) -{ - printf("%s\n", oid_to_hex(oid)); - return 0; /* always continue */ -} - -void some_func(void) -{ - struct sha1_array hashes = OID_ARRAY_INIT; - struct object_id oid; - - /* Read objects into our set */ - while (read_object_from_stdin(oid.hash)) - oid_array_append(&hashes, &oid); - - /* Check if some objects are in our set */ - while (read_object_from_stdin(oid.hash)) { - if (oid_array_lookup(&hashes, &oid) >= 0) - printf("it's in there!\n"); - - /* - * Print the unique set of objects. We could also have - * avoided adding duplicate objects in the first place, - * but we would end up re-sorting the array repeatedly. - * Instead, this will sort once and then skip duplicates - * in linear time. - */ - oid_array_for_each_unique(&hashes, print_callback, NULL); -} ------------------------------------------ diff --git a/Documentation/technical/api-parse-options.txt b/Documentation/technical/api-parse-options.txt index 2b036d7838..2e2e7c10c6 100644 --- a/Documentation/technical/api-parse-options.txt +++ b/Documentation/technical/api-parse-options.txt @@ -198,8 +198,10 @@ There are some macros to easily define options: The filename will be prefixed by passing the filename along with the prefix argument of `parse_options()` to `prefix_filename()`. -`OPT_ARGUMENT(long, description)`:: +`OPT_ARGUMENT(long, &int_var, description)`:: Introduce a long-option argument that will be kept in `argv[]`. + If this option was seen, `int_var` will be set to one (except + if a `NULL` pointer was passed). `OPT_NUMBER_CALLBACK(&var, description, func_ptr)`:: Recognize numerical options like -123 and feed the integer as diff --git a/Documentation/technical/api-quote.txt b/Documentation/technical/api-quote.txt deleted file mode 100644 index e8a1bce94e..0000000000 --- a/Documentation/technical/api-quote.txt +++ /dev/null @@ -1,10 +0,0 @@ -quote API -========= - -Talk about <quote.h>, things like - -* sq_quote and unquote -* c_style quote and unquote -* quoting for foreign languages - -(JC) diff --git a/Documentation/technical/api-ref-iteration.txt b/Documentation/technical/api-ref-iteration.txt deleted file mode 100644 index 46c3d5c355..0000000000 --- a/Documentation/technical/api-ref-iteration.txt +++ /dev/null @@ -1,78 +0,0 @@ -ref iteration API -================= - - -Iteration of refs is done by using an iterate function which will call a -callback function for every ref. The callback function has this -signature: - - int handle_one_ref(const char *refname, const struct object_id *oid, - int flags, void *cb_data); - -There are different kinds of iterate functions which all take a -callback of this type. The callback is then called for each found ref -until the callback returns nonzero. The returned value is then also -returned by the iterate function. - -Iteration functions -------------------- - -* `head_ref()` just iterates the head ref. - -* `for_each_ref()` iterates all refs. - -* `for_each_ref_in()` iterates all refs which have a defined prefix and - strips that prefix from the passed variable refname. - -* `for_each_tag_ref()`, `for_each_branch_ref()`, `for_each_remote_ref()`, - `for_each_replace_ref()` iterate refs from the respective area. - -* `for_each_glob_ref()` iterates all refs that match the specified glob - pattern. - -* `for_each_glob_ref_in()` the previous and `for_each_ref_in()` combined. - -* Use `refs_` API for accessing submodules. The submodule ref store could - be obtained with `get_submodule_ref_store()`. - -* `for_each_rawref()` can be used to learn about broken ref and symref. - -* `for_each_reflog()` iterates each reflog file. - -Submodules ----------- - -If you want to iterate the refs of a submodule you first need to add the -submodules object database. You can do this by a code-snippet like -this: - - const char *path = "path/to/submodule" - if (add_submodule_odb(path)) - die("Error submodule '%s' not populated.", path); - -`add_submodule_odb()` will return zero on success. If you -do not do this you will get an error for each ref that it does not point -to a valid object. - -Note: As a side-effect of this you can not safely assume that all -objects you lookup are available in superproject. All submodule objects -will be available the same way as the superprojects objects. - -Example: --------- - ----- -static int handle_remote_ref(const char *refname, - const unsigned char *sha1, int flags, void *cb_data) -{ - struct strbuf *output = cb_data; - strbuf_addf(output, "%s\n", refname); - return 0; -} - -... - - struct strbuf output = STRBUF_INIT; - for_each_remote_ref(handle_remote_ref, &output); - printf("%s", output.buf); ----- diff --git a/Documentation/technical/api-remote.txt b/Documentation/technical/api-remote.txt deleted file mode 100644 index f10941b2e8..0000000000 --- a/Documentation/technical/api-remote.txt +++ /dev/null @@ -1,127 +0,0 @@ -Remotes configuration API -========================= - -The API in remote.h gives access to the configuration related to -remotes. It handles all three configuration mechanisms historically -and currently used by Git, and presents the information in a uniform -fashion. Note that the code also handles plain URLs without any -configuration, giving them just the default information. - -struct remote -------------- - -`name`:: - - The user's nickname for the remote - -`url`:: - - An array of all of the url_nr URLs configured for the remote - -`pushurl`:: - - An array of all of the pushurl_nr push URLs configured for the remote - -`push`:: - - An array of refspecs configured for pushing, with - push_refspec being the literal strings, and push_refspec_nr - being the quantity. - -`fetch`:: - - An array of refspecs configured for fetching, with - fetch_refspec being the literal strings, and fetch_refspec_nr - being the quantity. - -`fetch_tags`:: - - The setting for whether to fetch tags (as a separate rule from - the configured refspecs); -1 means never to fetch tags, 0 - means to auto-follow tags based on the default heuristic, 1 - means to always auto-follow tags, and 2 means to fetch all - tags. - -`receivepack`, `uploadpack`:: - - The configured helper programs to run on the remote side, for - Git-native protocols. - -`http_proxy`:: - - The proxy to use for curl (http, https, ftp, etc.) URLs. - -`http_proxy_authmethod`:: - - The method used for authenticating against `http_proxy`. - -struct remotes can be found by name with remote_get(), and iterated -through with for_each_remote(). remote_get(NULL) will return the -default remote, given the current branch and configuration. - -struct refspec --------------- - -A struct refspec holds the parsed interpretation of a refspec. If it -will force updates (starts with a '+'), force is true. If it is a -pattern (sides end with '*') pattern is true. src and dest are the -two sides (including '*' characters if present); if there is only one -side, it is src, and dst is NULL; if sides exist but are empty (i.e., -the refspec either starts or ends with ':'), the corresponding side is -"". - -An array of strings can be parsed into an array of struct refspecs -using parse_fetch_refspec() or parse_push_refspec(). - -remote_find_tracking(), given a remote and a struct refspec with -either src or dst filled out, will fill out the other such that the -result is in the "fetch" specification for the remote (note that this -evaluates patterns and returns a single result). - -struct branch -------------- - -Note that this may end up moving to branch.h - -struct branch holds the configuration for a branch. It can be looked -up with branch_get(name) for "refs/heads/{name}", or with -branch_get(NULL) for HEAD. - -It contains: - -`name`:: - - The short name of the branch. - -`refname`:: - - The full path for the branch ref. - -`remote_name`:: - - The name of the remote listed in the configuration. - -`merge_name`:: - - An array of the "merge" lines in the configuration. - -`merge`:: - - An array of the struct refspecs used for the merge lines. That - is, merge[i]->dst is a local tracking ref which should be - merged into this branch by default. - -`merge_nr`:: - - The number of merge configurations - -branch_has_merge_config() returns true if the given branch has merge -configuration given. - -Other stuff ------------ - -There is other stuff in remote.h that is related, in general, to the -process of interacting with remotes. - -(Daniel Barkalow) diff --git a/Documentation/technical/api-revision-walking.txt b/Documentation/technical/api-revision-walking.txt deleted file mode 100644 index 03f9ea6ac4..0000000000 --- a/Documentation/technical/api-revision-walking.txt +++ /dev/null @@ -1,72 +0,0 @@ -revision walking API -==================== - -The revision walking API offers functions to build a list of revisions -and then iterate over that list. - -Calling sequence ----------------- - -The walking API has a given calling sequence: first you need to -initialize a rev_info structure, then add revisions to control what kind -of revision list do you want to get, finally you can iterate over the -revision list. - -Functions ---------- - -`repo_init_revisions`:: - - Initialize a rev_info structure with default values. The third - parameter may be NULL or can be prefix path, and then the `.prefix` - variable will be set to it. This is typically the first function you - want to call when you want to deal with a revision list. After calling - this function, you are free to customize options, like set - `.ignore_merges` to 0 if you don't want to ignore merges, and so on. See - `revision.h` for a complete list of available options. - -`add_pending_object`:: - - This function can be used if you want to add commit objects as revision - information. You can use the `UNINTERESTING` object flag to indicate if - you want to include or exclude the given commit (and commits reachable - from the given commit) from the revision list. -+ -NOTE: If you have the commits as a string list then you probably want to -use setup_revisions(), instead of parsing each string and using this -function. - -`setup_revisions`:: - - Parse revision information, filling in the `rev_info` structure, and - removing the used arguments from the argument list. Returns the number - of arguments left that weren't recognized, which are also moved to the - head of the argument list. The last parameter is used in case no - parameter given by the first two arguments. - -`prepare_revision_walk`:: - - Prepares the rev_info structure for a walk. You should check if it - returns any error (non-zero return code) and if it does not, you can - start using get_revision() to do the iteration. - -`get_revision`:: - - Takes a pointer to a `rev_info` structure and iterates over it, - returning a `struct commit *` each time you call it. The end of the - revision list is indicated by returning a NULL pointer. - -`reset_revision_walk`:: - - Reset the flags used by the revision walking api. You can use - this to do multiple sequential revision walks. - -Data structures ---------------- - -Talk about <revision.h>, things like: - -* two diff_options, one for path limiting, another for output; -* remaining functions; - -(Linus, JC, Dscho) diff --git a/Documentation/technical/api-run-command.txt b/Documentation/technical/api-run-command.txt deleted file mode 100644 index 8bf3e37f53..0000000000 --- a/Documentation/technical/api-run-command.txt +++ /dev/null @@ -1,264 +0,0 @@ -run-command API -=============== - -The run-command API offers a versatile tool to run sub-processes with -redirected input and output as well as with a modified environment -and an alternate current directory. - -A similar API offers the capability to run a function asynchronously, -which is primarily used to capture the output that the function -produces in the caller in order to process it. - - -Functions ---------- - -`child_process_init`:: - - Initialize a struct child_process variable. - -`start_command`:: - - Start a sub-process. Takes a pointer to a `struct child_process` - that specifies the details and returns pipe FDs (if requested). - See below for details. - -`finish_command`:: - - Wait for the completion of a sub-process that was started with - start_command(). - -`run_command`:: - - A convenience function that encapsulates a sequence of - start_command() followed by finish_command(). Takes a pointer - to a `struct child_process` that specifies the details. - -`run_command_v_opt`, `run_command_v_opt_cd_env`:: - - Convenience functions that encapsulate a sequence of - start_command() followed by finish_command(). The argument argv - specifies the program and its arguments. The argument opt is zero - or more of the flags `RUN_COMMAND_NO_STDIN`, `RUN_GIT_CMD`, - `RUN_COMMAND_STDOUT_TO_STDERR`, or `RUN_SILENT_EXEC_FAILURE` - that correspond to the members .no_stdin, .git_cmd, - .stdout_to_stderr, .silent_exec_failure of `struct child_process`. - The argument dir corresponds the member .dir. The argument env - corresponds to the member .env. - -`child_process_clear`:: - - Release the memory associated with the struct child_process. - Most users of the run-command API don't need to call this - function explicitly because `start_command` invokes it on - failure and `finish_command` calls it automatically already. - -The functions above do the following: - -. If a system call failed, errno is set and -1 is returned. A diagnostic - is printed. - -. If the program was not found, then -1 is returned and errno is set to - ENOENT; a diagnostic is printed only if .silent_exec_failure is 0. - -. Otherwise, the program is run. If it terminates regularly, its exit - code is returned. No diagnostic is printed, even if the exit code is - non-zero. - -. If the program terminated due to a signal, then the return value is the - signal number + 128, ie. the same value that a POSIX shell's $? would - report. A diagnostic is printed. - - -`start_async`:: - - Run a function asynchronously. Takes a pointer to a `struct - async` that specifies the details and returns a set of pipe FDs - for communication with the function. See below for details. - -`finish_async`:: - - Wait for the completion of an asynchronous function that was - started with start_async(). - -`run_hook`:: - - Run a hook. - The first argument is a pathname to an index file, or NULL - if the hook uses the default index file or no index is needed. - The second argument is the name of the hook. - The further arguments correspond to the hook arguments. - The last argument has to be NULL to terminate the arguments list. - If the hook does not exist or is not executable, the return - value will be zero. - If it is executable, the hook will be executed and the exit - status of the hook is returned. - On execution, .stdout_to_stderr and .no_stdin will be set. - (See below.) - - -Data structures ---------------- - -* `struct child_process` - -This describes the arguments, redirections, and environment of a -command to run in a sub-process. - -The caller: - -1. allocates and clears (using child_process_init() or - CHILD_PROCESS_INIT) a struct child_process variable; -2. initializes the members; -3. calls start_command(); -4. processes the data; -5. closes file descriptors (if necessary; see below); -6. calls finish_command(). - -The .argv member is set up as an array of string pointers (NULL -terminated), of which .argv[0] is the program name to run (usually -without a path). If the command to run is a git command, set argv[0] to -the command name without the 'git-' prefix and set .git_cmd = 1. - -Note that the ownership of the memory pointed to by .argv stays with the -caller, but it should survive until `finish_command` completes. If the -.argv member is NULL, `start_command` will point it at the .args -`argv_array` (so you may use one or the other, but you must use exactly -one). The memory in .args will be cleaned up automatically during -`finish_command` (or during `start_command` when it is unsuccessful). - -The members .in, .out, .err are used to redirect stdin, stdout, -stderr as follows: - -. Specify 0 to request no special redirection. No new file descriptor - is allocated. The child process simply inherits the channel from the - parent. - -. Specify -1 to have a pipe allocated; start_command() replaces -1 - by the pipe FD in the following way: - - .in: Returns the writable pipe end into which the caller writes; - the readable end of the pipe becomes the child's stdin. - - .out, .err: Returns the readable pipe end from which the caller - reads; the writable end of the pipe end becomes child's - stdout/stderr. - - The caller of start_command() must close the so returned FDs - after it has completed reading from/writing to it! - -. Specify a file descriptor > 0 to be used by the child: - - .in: The FD must be readable; it becomes child's stdin. - .out: The FD must be writable; it becomes child's stdout. - .err: The FD must be writable; it becomes child's stderr. - - The specified FD is closed by start_command(), even if it fails to - run the sub-process! - -. Special forms of redirection are available by setting these members - to 1: - - .no_stdin, .no_stdout, .no_stderr: The respective channel is - redirected to /dev/null. - - .stdout_to_stderr: stdout of the child is redirected to its - stderr. This happens after stderr is itself redirected. - So stdout will follow stderr to wherever it is - redirected. - -To modify the environment of the sub-process, specify an array of -string pointers (NULL terminated) in .env: - -. If the string is of the form "VAR=value", i.e. it contains '=' - the variable is added to the child process's environment. - -. If the string does not contain '=', it names an environment - variable that will be removed from the child process's environment. - -If the .env member is NULL, `start_command` will point it at the -.env_array `argv_array` (so you may use one or the other, but not both). -The memory in .env_array will be cleaned up automatically during -`finish_command` (or during `start_command` when it is unsuccessful). - -To specify a new initial working directory for the sub-process, -specify it in the .dir member. - -If the program cannot be found, the functions return -1 and set -errno to ENOENT. Normally, an error message is printed, but if -.silent_exec_failure is set to 1, no message is printed for this -special error condition. - - -* `struct async` - -This describes a function to run asynchronously, whose purpose is -to produce output that the caller reads. - -The caller: - -1. allocates and clears (memset(&asy, 0, sizeof(asy));) a - struct async variable; -2. initializes .proc and .data; -3. calls start_async(); -4. processes communicates with proc through .in and .out; -5. closes .in and .out; -6. calls finish_async(). - -The members .in, .out are used to provide a set of fd's for -communication between the caller and the callee as follows: - -. Specify 0 to have no file descriptor passed. The callee will - receive -1 in the corresponding argument. - -. Specify < 0 to have a pipe allocated; start_async() replaces - with the pipe FD in the following way: - - .in: Returns the writable pipe end into which the caller - writes; the readable end of the pipe becomes the function's - in argument. - - .out: Returns the readable pipe end from which the caller - reads; the writable end of the pipe becomes the function's - out argument. - - The caller of start_async() must close the returned FDs after it - has completed reading from/writing from them. - -. Specify a file descriptor > 0 to be used by the function: - - .in: The FD must be readable; it becomes the function's in. - .out: The FD must be writable; it becomes the function's out. - - The specified FD is closed by start_async(), even if it fails to - run the function. - -The function pointer in .proc has the following signature: - - int proc(int in, int out, void *data); - -. in, out specifies a set of file descriptors to which the function - must read/write the data that it needs/produces. The function - *must* close these descriptors before it returns. A descriptor - may be -1 if the caller did not configure a descriptor for that - direction. - -. data is the value that the caller has specified in the .data member - of struct async. - -. The return value of the function is 0 on success and non-zero - on failure. If the function indicates failure, finish_async() will - report failure as well. - - -There are serious restrictions on what the asynchronous function can do -because this facility is implemented by a thread in the same address -space on most platforms (when pthreads is available), but by a pipe to -a forked process otherwise: - -. It cannot change the program's state (global variables, environment, - etc.) in a way that the caller notices; in other words, .in and .out - are the only communication channels to the caller. - -. It must not change the program's state that the caller of the - facility also uses. diff --git a/Documentation/technical/api-setup.txt b/Documentation/technical/api-setup.txt deleted file mode 100644 index eb1fa9853e..0000000000 --- a/Documentation/technical/api-setup.txt +++ /dev/null @@ -1,47 +0,0 @@ -setup API -========= - -Talk about - -* setup_git_directory() -* setup_git_directory_gently() -* is_inside_git_dir() -* is_inside_work_tree() -* setup_work_tree() - -(Dscho) - -Pathspec --------- - -See glossary-context.txt for the syntax of pathspec. In memory, a -pathspec set is represented by "struct pathspec" and is prepared by -parse_pathspec(). This function takes several arguments: - -- magic_mask specifies what features that are NOT supported by the - following code. If a user attempts to use such a feature, - parse_pathspec() can reject it early. - -- flags specifies other things that the caller wants parse_pathspec to - perform. - -- prefix and args come from cmd_* functions - -parse_pathspec() helps catch unsupported features and reject them -politely. At a lower level, different pathspec-related functions may -not support the same set of features. Such pathspec-sensitive -functions are guarded with GUARD_PATHSPEC(), which will die in an -unfriendly way when an unsupported feature is requested. - -The command designers are supposed to make sure that GUARD_PATHSPEC() -never dies. They have to make sure all unsupported features are caught -by parse_pathspec(), not by GUARD_PATHSPEC. grepping GUARD_PATHSPEC() -should give the designers all pathspec-sensitive codepaths and what -features they support. - -A similar process is applied when a new pathspec magic is added. The -designer lifts the GUARD_PATHSPEC restriction in the functions that -support the new magic. At the same time (s)he has to make sure this -new feature will be caught at parse_pathspec() in commands that cannot -handle the new magic in some cases. grepping parse_pathspec() should -help. diff --git a/Documentation/technical/api-sigchain.txt b/Documentation/technical/api-sigchain.txt deleted file mode 100644 index 9e1189ef01..0000000000 --- a/Documentation/technical/api-sigchain.txt +++ /dev/null @@ -1,41 +0,0 @@ -sigchain API -============ - -Code often wants to set a signal handler to clean up temporary files or -other work-in-progress when we die unexpectedly. For multiple pieces of -code to do this without conflicting, each piece of code must remember -the old value of the handler and restore it either when: - - 1. The work-in-progress is finished, and the handler is no longer - necessary. The handler should revert to the original behavior - (either another handler, SIG_DFL, or SIG_IGN). - - 2. The signal is received. We should then do our cleanup, then chain - to the next handler (or die if it is SIG_DFL). - -Sigchain is a tiny library for keeping a stack of handlers. Your handler -and installation code should look something like: - ------------------------------------------- - void clean_foo_on_signal(int sig) - { - clean_foo(); - sigchain_pop(sig); - raise(sig); - } - - void other_func() - { - sigchain_push_common(clean_foo_on_signal); - mess_up_foo(); - clean_foo(); - } ------------------------------------------- - -Handlers are given the typedef of sigchain_fun. This is the same type -that is given to signal() or sigaction(). It is perfectly reasonable to -push SIG_DFL or SIG_IGN onto the stack. - -You can sigchain_push and sigchain_pop individual signals. For -convenience, sigchain_push_common will push the handler onto the stack -for many common signals. diff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt deleted file mode 100644 index fb06089393..0000000000 --- a/Documentation/technical/api-submodule-config.txt +++ /dev/null @@ -1,66 +0,0 @@ -submodule config cache API -========================== - -The submodule config cache API allows to read submodule -configurations/information from specified revisions. Internally -information is lazily read into a cache that is used to avoid -unnecessary parsing of the same .gitmodules files. Lookups can be done by -submodule path or name. - -Usage ------ - -To initialize the cache with configurations from the worktree the caller -typically first calls `gitmodules_config()` to read values from the -worktree .gitmodules and then to overlay the local git config values -`parse_submodule_config_option()` from the config parsing -infrastructure. - -The caller can look up information about submodules by using the -`submodule_from_path()` or `submodule_from_name()` functions. They return -a `struct submodule` which contains the values. The API automatically -initializes and allocates the needed infrastructure on-demand. If the -caller does only want to lookup values from revisions the initialization -can be skipped. - -If the internal cache might grow too big or when the caller is done with -the API, all internally cached values can be freed with submodule_free(). - -Data Structures ---------------- - -`struct submodule`:: - - This structure is used to return the information about one - submodule for a certain revision. It is returned by the lookup - functions. - -Functions ---------- - -`void submodule_free(struct repository *r)`:: - - Use these to free the internally cached values. - -`int parse_submodule_config_option(const char *var, const char *value)`:: - - Can be passed to the config parsing infrastructure to parse - local (worktree) submodule configurations. - -`const struct submodule *submodule_from_path(const unsigned char *treeish_name, const char *path)`:: - - Given a tree-ish in the superproject and a path, return the - submodule that is bound at the path in the named tree. - -`const struct submodule *submodule_from_name(const unsigned char *treeish_name, const char *name)`:: - - The same as above but lookup by name. - -Whenever a submodule configuration is parsed in `parse_submodule_config_option` -via e.g. `gitmodules_config()`, it will overwrite the null_sha1 entry. -So in the normal case, when HEAD:.gitmodules is parsed first and then overlayed -with the repository configuration, the null_sha1 entry contains the local -configuration of a submodule (e.g. consolidated values from local git -configuration and the .gitmodules file in the worktree). - -For an example usage see test-submodule-config.c. diff --git a/Documentation/technical/api-trace.txt b/Documentation/technical/api-trace.txt deleted file mode 100644 index fadb5979c4..0000000000 --- a/Documentation/technical/api-trace.txt +++ /dev/null @@ -1,140 +0,0 @@ -trace API -========= - -The trace API can be used to print debug messages to stderr or a file. Trace -code is inactive unless explicitly enabled by setting `GIT_TRACE*` environment -variables. - -The trace implementation automatically adds `timestamp file:line ... \n` to -all trace messages. E.g.: - ------------- -23:59:59.123456 git.c:312 trace: built-in: git 'foo' -00:00:00.000001 builtin/foo.c:99 foo: some message ------------- - -Data Structures ---------------- - -`struct trace_key`:: - - Defines a trace key (or category). The default (for API functions that - don't take a key) is `GIT_TRACE`. -+ -E.g. to define a trace key controlled by environment variable `GIT_TRACE_FOO`: -+ ------------- -static struct trace_key trace_foo = TRACE_KEY_INIT(FOO); - -static void trace_print_foo(const char *message) -{ - trace_printf_key(&trace_foo, "%s", message); -} ------------- -+ -Note: don't use `const` as the trace implementation stores internal state in -the `trace_key` structure. - -Functions ---------- - -`int trace_want(struct trace_key *key)`:: - - Checks whether the trace key is enabled. Used to prevent expensive - string formatting before calling one of the printing APIs. - -`void trace_disable(struct trace_key *key)`:: - - Disables tracing for the specified key, even if the environment - variable was set. - -`void trace_printf(const char *format, ...)`:: -`void trace_printf_key(struct trace_key *key, const char *format, ...)`:: - - Prints a formatted message, similar to printf. - -`void trace_argv_printf(const char **argv, const char *format, ...)``:: - - Prints a formatted message, followed by a quoted list of arguments. - -`void trace_strbuf(struct trace_key *key, const struct strbuf *data)`:: - - Prints the strbuf, without additional formatting (i.e. doesn't - choke on `%` or even `\0`). - -`uint64_t getnanotime(void)`:: - - Returns nanoseconds since the epoch (01/01/1970), typically used - for performance measurements. -+ -Currently there are high precision timer implementations for Linux (using -`clock_gettime(CLOCK_MONOTONIC)`) and Windows (`QueryPerformanceCounter`). -Other platforms use `gettimeofday` as time source. - -`void trace_performance(uint64_t nanos, const char *format, ...)`:: -`void trace_performance_since(uint64_t start, const char *format, ...)`:: - - Prints the elapsed time (in nanoseconds), or elapsed time since - `start`, followed by a formatted message. Enabled via environment - variable `GIT_TRACE_PERFORMANCE`. Used for manual profiling, e.g.: -+ ------------- -uint64_t start = getnanotime(); -/* code section to measure */ -trace_performance_since(start, "foobar"); ------------- -+ ------------- -uint64_t t = 0; -for (;;) { - /* ignore */ - t -= getnanotime(); - /* code section to measure */ - t += getnanotime(); - /* ignore */ -} -trace_performance(t, "frotz"); ------------- - -Bugs & Caveats --------------- - -GIT_TRACE_* environment variables can be used to tell Git to show -trace output to its standard error stream. Git can often spawn a pager -internally to run its subcommand and send its standard output and -standard error to it. - -Because GIT_TRACE_PERFORMANCE trace is generated only at the very end -of the program with atexit(), which happens after the pager exits, it -would not work well if you send its log to the standard error output -and let Git spawn the pager at the same time. - -As a work around, you can for example use '--no-pager', or set -GIT_TRACE_PERFORMANCE to another file descriptor which is redirected -to stderr, or set GIT_TRACE_PERFORMANCE to a file specified by its -absolute path. - -For example instead of the following command which by default may not -print any performance information: - ------------- -GIT_TRACE_PERFORMANCE=2 git log -1 ------------- - -you may want to use: - ------------- -GIT_TRACE_PERFORMANCE=2 git --no-pager log -1 ------------- - -or: - ------------- -GIT_TRACE_PERFORMANCE=3 3>&2 git log -1 ------------- - -or: - ------------- -GIT_TRACE_PERFORMANCE=/path/to/log/file git log -1 ------------- diff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt new file mode 100644 index 0000000000..6b6085585d --- /dev/null +++ b/Documentation/technical/api-trace2.txt @@ -0,0 +1,1171 @@ += Trace2 API + +The Trace2 API can be used to print debug, performance, and telemetry +information to stderr or a file. The Trace2 feature is inactive unless +explicitly enabled by enabling one or more Trace2 Targets. + +The Trace2 API is intended to replace the existing (Trace1) +printf-style tracing provided by the existing `GIT_TRACE` and +`GIT_TRACE_PERFORMANCE` facilities. During initial implementation, +Trace2 and Trace1 may operate in parallel. + +The Trace2 API defines a set of high-level messages with known fields, +such as (`start`: `argv`) and (`exit`: {`exit-code`, `elapsed-time`}). + +Trace2 instrumentation throughout the Git code base sends Trace2 +messages to the enabled Trace2 Targets. Targets transform these +messages content into purpose-specific formats and write events to +their data streams. In this manner, the Trace2 API can drive +many different types of analysis. + +Targets are defined using a VTable allowing easy extension to other +formats in the future. This might be used to define a binary format, +for example. + +Trace2 is controlled using `trace2.*` config values in the system and +global config files and `GIT_TRACE2*` environment variables. Trace2 does +not read from repo local or worktree config files or respect `-c` +command line config settings. + +== Trace2 Targets + +Trace2 defines the following set of Trace2 Targets. +Format details are given in a later section. + +=== The Normal Format Target + +The normal format target is a tradition printf format and similar +to GIT_TRACE format. This format is enabled with the `GIT_TRACE2` +environment variable or the `trace2.normalTarget` system or global +config setting. + +For example + +------------ +$ export GIT_TRACE2=~/log.normal +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.normalTarget ~/log.normal +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.normal +12:28:42.620009 common-main.c:38 version 2.20.1.155.g426c96fcdb +12:28:42.620989 common-main.c:39 start git version +12:28:42.621101 git.c:432 cmd_name version (version) +12:28:42.621215 git.c:662 exit elapsed:0.001227 code:0 +12:28:42.621250 trace2/tr2_tgt_normal.c:124 atexit elapsed:0.001265 code:0 +------------ + +=== The Performance Format Target + +The performance format target (PERF) is a column-based format to +replace GIT_TRACE_PERFORMANCE and is suitable for development and +testing, possibly to complement tools like gprof. This format is +enabled with the `GIT_TRACE2_PERF` environment variable or the +`trace2.perfTarget` system or global config setting. + +For example + +------------ +$ export GIT_TRACE2_PERF=~/log.perf +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.perfTarget ~/log.perf +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.perf +12:28:42.620675 common-main.c:38 | d0 | main | version | | | | | 2.20.1.155.g426c96fcdb +12:28:42.621001 common-main.c:39 | d0 | main | start | | 0.001173 | | | git version +12:28:42.621111 git.c:432 | d0 | main | cmd_name | | | | | version (version) +12:28:42.621225 git.c:662 | d0 | main | exit | | 0.001227 | | | code:0 +12:28:42.621259 trace2/tr2_tgt_perf.c:211 | d0 | main | atexit | | 0.001265 | | | code:0 +------------ + +=== The Event Format Target + +The event format target is a JSON-based format of event data suitable +for telemetry analysis. This format is enabled with the `GIT_TRACE2_EVENT` +environment variable or the `trace2.eventTarget` system or global config +setting. + +For example + +------------ +$ export GIT_TRACE2_EVENT=~/log.event +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +or + +------------ +$ git config --global trace2.eventTarget ~/log.event +$ git version +git version 2.20.1.155.g426c96fcdb +------------ + +yields + +------------ +$ cat ~/log.event +{"event":"version","sid":"sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.620713Z","file":"common-main.c","line":38,"evt":"2","exe":"2.20.1.155.g426c96fcdb"} +{"event":"start","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621027Z","file":"common-main.c","line":39,"t_abs":0.001173,"argv":["git","version"]} +{"event":"cmd_name","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621122Z","file":"git.c","line":432,"name":"version","hierarchy":"version"} +{"event":"exit","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621236Z","file":"git.c","line":662,"t_abs":0.001227,"code":0} +{"event":"atexit","sid":"20190408T191610.507018Z-H9b68c35f-P000059a8","thread":"main","time":"2019-01-16T17:28:42.621268Z","file":"trace2/tr2_tgt_event.c","line":163,"t_abs":0.001265,"code":0} +------------ + +=== Enabling a Target + +To enable a target, set the corresponding environment variable or +system or global config value to one of the following: + +include::../trace2-target-values.txt[] + +When trace files are written to a target directory, they will be named according +to the last component of the SID (optionally followed by a counter to avoid +filename collisions). + +== Trace2 API + +All public Trace2 functions and macros are defined in `trace2.h` and +`trace2.c`. All public symbols are prefixed with `trace2_`. + +There are no public Trace2 data structures. + +The Trace2 code also defines a set of private functions and data types +in the `trace2/` directory. These symbols are prefixed with `tr2_` +and should only be used by functions in `trace2.c`. + +== Conventions for Public Functions and Macros + +The functions defined by the Trace2 API are declared and documented +in `trace2.h`. It defines the API functions and wrapper macros for +Trace2. + +Some functions have a `_fl()` suffix to indicate that they take `file` +and `line-number` arguments. + +Some functions have a `_va_fl()` suffix to indicate that they also +take a `va_list` argument. + +Some functions have a `_printf_fl()` suffix to indicate that they also +take a varargs argument. + +There are CPP wrapper macros and ifdefs to hide most of these details. +See `trace2.h` for more details. The following discussion will only +describe the simplified forms. + +== Public API + +All Trace2 API functions send a message to all of the active +Trace2 Targets. This section describes the set of available +messages. + +It helps to divide these functions into groups for discussion +purposes. + +=== Basic Command Messages + +These are concerned with the lifetime of the overall git process. +e.g: `void trace2_initialize_clock()`, `void trace2_initialize()`, +`int trace2_is_enabled()`, `void trace2_cmd_start(int argc, const char **argv)`. + +=== Command Detail Messages + +These are concerned with describing the specific Git command +after the command line, config, and environment are inspected. +e.g: `void trace2_cmd_name(const char *name)`, +`void trace2_cmd_mode(const char *mode)`. + +=== Child Process Messages + +These are concerned with the various spawned child processes, +including shell scripts, git commands, editors, pagers, and hooks. + +e.g: `void trace2_child_start(struct child_process *cmd)`. + +=== Git Thread Messages + +These messages are concerned with Git thread usage. + +e.g: `void trace2_thread_start(const char *thread_name)`. + +=== Region and Data Messages + +These are concerned with recording performance data +over regions or spans of code. e.g: +`void trace2_region_enter(const char *category, const char *label, const struct repository *repo)`. + +Refer to trace2.h for details about all trace2 functions. + +== Trace2 Target Formats + +=== NORMAL Format + +Events are written as lines of the form: + +------------ +[<time> SP <filename>:<line> SP+] <event-name> [[SP] <event-message>] LF +------------ + +`<event-name>`:: + + is the event name. + +`<event-message>`:: + is a free-form printf message intended for human consumption. ++ +Note that this may contain embedded LF or CRLF characters that are +not escaped, so the event may spill across multiple lines. + +If `GIT_TRACE2_BRIEF` or `trace2.normalBrief` is true, the `time`, `filename`, +and `line` fields are omitted. + +This target is intended to be more of a summary (like GIT_TRACE) and +less detailed than the other targets. It ignores thread, region, and +data messages, for example. + +=== PERF Format + +Events are written as lines of the form: + +------------ +[<time> SP <filename>:<line> SP+ + BAR SP] d<depth> SP + BAR SP <thread-name> SP+ + BAR SP <event-name> SP+ + BAR SP [r<repo-id>] SP+ + BAR SP [<t_abs>] SP+ + BAR SP [<t_rel>] SP+ + BAR SP [<category>] SP+ + BAR SP DOTS* <perf-event-message> + LF +------------ + +`<depth>`:: + is the git process depth. This is the number of parent + git processes. A top-level git command has depth value "d0". + A child of it has depth value "d1". A second level child + has depth value "d2" and so on. + +`<thread-name>`:: + is a unique name for the thread. The primary thread + is called "main". Other thread names are of the form "th%d:%s" + and include a unique number and the name of the thread-proc. + +`<event-name>`:: + is the event name. + +`<repo-id>`:: + when present, is a number indicating the repository + in use. A `def_repo` event is emitted when a repository is + opened. This defines the repo-id and associated worktree. + Subsequent repo-specific events will reference this repo-id. ++ +Currently, this is always "r1" for the main repository. +This field is in anticipation of in-proc submodules in the future. + +`<t_abs>`:: + when present, is the absolute time in seconds since the + program started. + +`<t_rel>`:: + when present, is time in seconds relative to the start of + the current region. For a thread-exit event, it is the elapsed + time of the thread. + +`<category>`:: + is present on region and data events and is used to + indicate a broad category, such as "index" or "status". + +`<perf-event-message>`:: + is a free-form printf message intended for human consumption. + +------------ +15:33:33.532712 wt-status.c:2310 | d0 | main | region_enter | r1 | 0.126064 | | status | label:print +15:33:33.532712 wt-status.c:2331 | d0 | main | region_leave | r1 | 0.127568 | 0.001504 | status | label:print +------------ + +If `GIT_TRACE2_PERF_BRIEF` or `trace2.perfBrief` is true, the `time`, `file`, +and `line` fields are omitted. + +------------ +d0 | main | region_leave | r1 | 0.011717 | 0.009122 | index | label:preload +------------ + +The PERF target is intended for interactive performance analysis +during development and is quite noisy. + +=== EVENT Format + +Each event is a JSON-object containing multiple key/value pairs +written as a single line and followed by a LF. + +------------ +'{' <key> ':' <value> [',' <key> ':' <value>]* '}' LF +------------ + +Some key/value pairs are common to all events and some are +event-specific. + +==== Common Key/Value Pairs + +The following key/value pairs are common to all events: + +------------ +{ + "event":"version", + "sid":"20190408T191827.272759Z-H9b68c35f-P00003510", + "thread":"main", + "time":"2019-04-08T19:18:27.282761Z", + "file":"common-main.c", + "line":42, + ... +} +------------ + +`"event":<event>`:: + is the event name. + +`"sid":<sid>`:: + is the session-id. This is a unique string to identify the + process instance to allow all events emitted by a process to + be identified. A session-id is used instead of a PID because + PIDs are recycled by the OS. For child git processes, the + session-id is prepended with the session-id of the parent git + process to allow parent-child relationships to be identified + during post-processing. + +`"thread":<thread>`:: + is the thread name. + +`"time":<time>`:: + is the UTC time of the event. + +`"file":<filename>`:: + is source file generating the event. + +`"line":<line-number>`:: + is the integer source line number generating the event. + +`"repo":<repo-id>`:: + when present, is the integer repo-id as described previously. + +If `GIT_TRACE2_EVENT_BRIEF` or `trace2.eventBrief` is true, the `file` +and `line` fields are omitted from all events and the `time` field is +only present on the "start" and "atexit" events. + +==== Event-Specific Key/Value Pairs + +`"version"`:: + This event gives the version of the executable and the EVENT format. It + should always be the first event in a trace session. The EVENT format + version will be incremented if new event types are added, if existing + fields are removed, or if there are significant changes in + interpretation of existing events or fields. Smaller changes, such as + adding a new field to an existing event, will not require an increment + to the EVENT format version. ++ +------------ +{ + "event":"version", + ... + "evt":"2", # EVENT format version + "exe":"2.20.1.155.g426c96fcdb" # git version +} +------------ + +`"discard"`:: + This event is written to the git-trace2-discard sentinel file if there + are too many files in the target trace directory (see the + trace2.maxFiles config option). ++ +------------ +{ + "event":"discard", + ... +} +------------ + +`"start"`:: + This event contains the complete argv received by main(). ++ +------------ +{ + "event":"start", + ... + "t_abs":0.001227, # elapsed time in seconds + "argv":["git","version"] +} +------------ + +`"exit"`:: + This event is emitted when git calls `exit()`. ++ +------------ +{ + "event":"exit", + ... + "t_abs":0.001227, # elapsed time in seconds + "code":0 # exit code +} +------------ + +`"atexit"`:: + This event is emitted by the Trace2 `atexit` routine during + final shutdown. It should be the last event emitted by the + process. ++ +(The elapsed time reported here is greater than the time reported in +the "exit" event because it runs after all other atexit tasks have +completed.) ++ +------------ +{ + "event":"atexit", + ... + "t_abs":0.001227, # elapsed time in seconds + "code":0 # exit code +} +------------ + +`"signal"`:: + This event is emitted when the program is terminated by a user + signal. Depending on the platform, the signal event may + prevent the "atexit" event from being generated. ++ +------------ +{ + "event":"signal", + ... + "t_abs":0.001227, # elapsed time in seconds + "signo":13 # SIGTERM, SIGINT, etc. +} +------------ + +`"error"`:: + This event is emitted when one of the `error()`, `die()`, + or `usage()` functions are called. ++ +------------ +{ + "event":"error", + ... + "msg":"invalid option: --cahced", # formatted error message + "fmt":"invalid option: %s" # error format string +} +------------ ++ +The error event may be emitted more than once. The format string +allows post-processors to group errors by type without worrying +about specific error arguments. + +`"cmd_path"`:: + This event contains the discovered full path of the git + executable (on platforms that are configured to resolve it). ++ +------------ +{ + "event":"cmd_path", + ... + "path":"C:/work/gfw/git.exe" +} +------------ + +`"cmd_name"`:: + This event contains the command name for this git process + and the hierarchy of commands from parent git processes. ++ +------------ +{ + "event":"cmd_name", + ... + "name":"pack-objects", + "hierarchy":"push/pack-objects" +} +------------ ++ +Normally, the "name" field contains the canonical name of the +command. When a canonical name is not available, one of +these special values are used: ++ +------------ +"_query_" # "git --html-path" +"_run_dashed_" # when "git foo" tries to run "git-foo" +"_run_shell_alias_" # alias expansion to a shell command +"_run_git_alias_" # alias expansion to a git command +"_usage_" # usage error +------------ + +`"cmd_mode"`:: + This event, when present, describes the command variant This + event may be emitted more than once. ++ +------------ +{ + "event":"cmd_mode", + ... + "name":"branch" +} +------------ ++ +The "name" field is an arbitrary string to describe the command mode. +For example, checkout can checkout a branch or an individual file. +And these variations typically have different performance +characteristics that are not comparable. + +`"alias"`:: + This event is present when an alias is expanded. ++ +------------ +{ + "event":"alias", + ... + "alias":"l", # registered alias + "argv":["log","--graph"] # alias expansion +} +------------ + +`"child_start"`:: + This event describes a child process that is about to be + spawned. ++ +------------ +{ + "event":"child_start", + ... + "child_id":2, + "child_class":"?", + "use_shell":false, + "argv":["git","rev-list","--objects","--stdin","--not","--all","--quiet"] + + "hook_name":"<hook_name>" # present when child_class is "hook" + "cd":"<path>" # present when cd is required +} +------------ ++ +The "child_id" field can be used to match this child_start with the +corresponding child_exit event. ++ +The "child_class" field is a rough classification, such as "editor", +"pager", "transport/*", and "hook". Unclassified children are classified +with "?". + +`"child_exit"`:: + This event is generated after the current process has returned + from the waitpid() and collected the exit information from the + child. ++ +------------ +{ + "event":"child_exit", + ... + "child_id":2, + "pid":14708, # child PID + "code":0, # child exit-code + "t_rel":0.110605 # observed run-time of child process +} +------------ ++ +Note that the session-id of the child process is not available to +the current/spawning process, so the child's PID is reported here as +a hint for post-processing. (But it is only a hint because the child +process may be a shell script which doesn't have a session-id.) ++ +Note that the `t_rel` field contains the observed run time in seconds +for the child process (starting before the fork/exec/spawn and +stopping after the waitpid() and includes OS process creation overhead). +So this time will be slightly larger than the atexit time reported by +the child process itself. + +`"exec"`:: + This event is generated before git attempts to `exec()` + another command rather than starting a child process. ++ +------------ +{ + "event":"exec", + ... + "exec_id":0, + "exe":"git", + "argv":["foo", "bar"] +} +------------ ++ +The "exec_id" field is a command-unique id and is only useful if the +`exec()` fails and a corresponding exec_result event is generated. + +`"exec_result"`:: + This event is generated if the `exec()` fails and control + returns to the current git command. ++ +------------ +{ + "event":"exec_result", + ... + "exec_id":0, + "code":1 # error code (errno) from exec() +} +------------ + +`"thread_start"`:: + This event is generated when a thread is started. It is + generated from *within* the new thread's thread-proc (for TLS + reasons). ++ +------------ +{ + "event":"thread_start", + ... + "thread":"th02:preload_thread" # thread name +} +------------ + +`"thread_exit"`:: + This event is generated when a thread exits. It is generated + from *within* the thread's thread-proc (for TLS reasons). ++ +------------ +{ + "event":"thread_exit", + ... + "thread":"th02:preload_thread", # thread name + "t_rel":0.007328 # thread elapsed time +} +------------ + +`"def_param"`:: + This event is generated to log a global parameter, such as a config + setting, command-line flag, or environment variable. ++ +------------ +{ + "event":"def_param", + ... + "param":"core.abbrev", + "value":"7" +} +------------ + +`"def_repo"`:: + This event defines a repo-id and associates it with the root + of the worktree. ++ +------------ +{ + "event":"def_repo", + ... + "repo":1, + "worktree":"/Users/jeffhost/work/gfw" +} +------------ ++ +As stated earlier, the repo-id is currently always 1, so there will +only be one def_repo event. Later, if in-proc submodules are +supported, a def_repo event should be emitted for each submodule +visited. + +`"region_enter"`:: + This event is generated when entering a region. ++ +------------ +{ + "event":"region_enter", + ... + "repo":1, # optional + "nesting":1, # current region stack depth + "category":"index", # optional + "label":"do_read_index", # optional + "msg":".git/index" # optional +} +------------ ++ +The `category` field may be used in a future enhancement to +do category-based filtering. ++ +`GIT_TRACE2_EVENT_NESTING` or `trace2.eventNesting` can be used to +filter deeply nested regions and data events. It defaults to "2". + +`"region_leave"`:: + This event is generated when leaving a region. ++ +------------ +{ + "event":"region_leave", + ... + "repo":1, # optional + "t_rel":0.002876, # time spent in region in seconds + "nesting":1, # region stack depth + "category":"index", # optional + "label":"do_read_index", # optional + "msg":".git/index" # optional +} +------------ + +`"data"`:: + This event is generated to log a thread- and region-local + key/value pair. ++ +------------ +{ + "event":"data", + ... + "repo":1, # optional + "t_abs":0.024107, # absolute elapsed time + "t_rel":0.001031, # elapsed time in region/thread + "nesting":2, # region stack depth + "category":"index", + "key":"read/cache_nr", + "value":"3552" +} +------------ ++ +The "value" field may be an integer or a string. + +`"data-json"`:: + This event is generated to log a pre-formatted JSON string + containing structured data. ++ +------------ +{ + "event":"data_json", + ... + "repo":1, # optional + "t_abs":0.015905, + "t_rel":0.015905, + "nesting":1, + "category":"process", + "key":"windows/ancestry", + "value":["bash.exe","bash.exe"] +} +------------ + +== Example Trace2 API Usage + +Here is a hypothetical usage of the Trace2 API showing the intended +usage (without worrying about the actual Git details). + +Initialization:: + + Initialization happens in `main()`. Behind the scenes, an + `atexit` and `signal` handler are registered. ++ +---------------- +int main(int argc, const char **argv) +{ + int exit_code; + + trace2_initialize(); + trace2_cmd_start(argv); + + exit_code = cmd_main(argc, argv); + + trace2_cmd_exit(exit_code); + + return exit_code; +} +---------------- + +Command Details:: + + After the basics are established, additional command + information can be sent to Trace2 as it is discovered. ++ +---------------- +int cmd_checkout(int argc, const char **argv) +{ + trace2_cmd_name("checkout"); + trace2_cmd_mode("branch"); + trace2_def_repo(the_repository); + + // emit "def_param" messages for "interesting" config settings. + trace2_cmd_list_config(); + + if (do_something()) + trace2_cmd_error("Path '%s': cannot do something", path); + + return 0; +} +---------------- + +Child Processes:: + + Wrap code spawning child processes. ++ +---------------- +void run_child(...) +{ + int child_exit_code; + struct child_process cmd = CHILD_PROCESS_INIT; + ... + cmd.trace2_child_class = "editor"; + + trace2_child_start(&cmd); + child_exit_code = spawn_child_and_wait_for_it(); + trace2_child_exit(&cmd, child_exit_code); +} +---------------- ++ +For example, the following fetch command spawned ssh, index-pack, +rev-list, and gc. This example also shows that fetch took +5.199 seconds and of that 4.932 was in ssh. ++ +---------------- +$ export GIT_TRACE2_BRIEF=1 +$ export GIT_TRACE2=~/log.normal +$ git fetch origin +... +---------------- ++ +---------------- +$ cat ~/log.normal +version 2.20.1.vfs.1.1.47.g534dbe1ad1 +start git fetch origin +worktree /Users/jeffhost/work/gfw +cmd_name fetch (fetch) +child_start[0] ssh git@github.com ... +child_start[1] git index-pack ... +... (Trace2 events from child processes omitted) +child_exit[1] pid:14707 code:0 elapsed:0.076353 +child_exit[0] pid:14706 code:0 elapsed:4.931869 +child_start[2] git rev-list ... +... (Trace2 events from child process omitted) +child_exit[2] pid:14708 code:0 elapsed:0.110605 +child_start[3] git gc --auto +... (Trace2 events from child process omitted) +child_exit[3] pid:14709 code:0 elapsed:0.006240 +exit elapsed:5.198503 code:0 +atexit elapsed:5.198541 code:0 +---------------- ++ +When a git process is a (direct or indirect) child of another +git process, it inherits Trace2 context information. This +allows the child to print the command hierarchy. This example +shows gc as child[3] of fetch. When the gc process reports +its name as "gc", it also reports the hierarchy as "fetch/gc". +(In this example, trace2 messages from the child process is +indented for clarity.) ++ +---------------- +$ export GIT_TRACE2_BRIEF=1 +$ export GIT_TRACE2=~/log.normal +$ git fetch origin +... +---------------- ++ +---------------- +$ cat ~/log.normal +version 2.20.1.160.g5676107ecd.dirty +start git fetch official +worktree /Users/jeffhost/work/gfw +cmd_name fetch (fetch) +... +child_start[3] git gc --auto + version 2.20.1.160.g5676107ecd.dirty + start /Users/jeffhost/work/gfw/git gc --auto + worktree /Users/jeffhost/work/gfw + cmd_name gc (fetch/gc) + exit elapsed:0.001959 code:0 + atexit elapsed:0.001997 code:0 +child_exit[3] pid:20303 code:0 elapsed:0.007564 +exit elapsed:3.868938 code:0 +atexit elapsed:3.868970 code:0 +---------------- + +Regions:: + + Regions can be use to time an interesting section of code. ++ +---------------- +void wt_status_collect(struct wt_status *s) +{ + trace2_region_enter("status", "worktrees", s->repo); + wt_status_collect_changes_worktree(s); + trace2_region_leave("status", "worktrees", s->repo); + + trace2_region_enter("status", "index", s->repo); + wt_status_collect_changes_index(s); + trace2_region_leave("status", "index", s->repo); + + trace2_region_enter("status", "untracked", s->repo); + wt_status_collect_untracked(s); + trace2_region_leave("status", "untracked", s->repo); +} + +void wt_status_print(struct wt_status *s) +{ + trace2_region_enter("status", "print", s->repo); + switch (s->status_format) { + ... + } + trace2_region_leave("status", "print", s->repo); +} +---------------- ++ +In this example, scanning for untracked files ran from +0.012568 to ++0.027149 (since the process started) and took 0.014581 seconds. ++ +---------------- +$ export GIT_TRACE2_PERF_BRIEF=1 +$ export GIT_TRACE2_PERF=~/log.perf +$ git status +... + +$ cat ~/log.perf +d0 | main | version | | | | | 2.20.1.160.g5676107ecd.dirty +d0 | main | start | | 0.001173 | | | git status +d0 | main | def_repo | r1 | | | | worktree:/Users/jeffhost/work/gfw +d0 | main | cmd_name | | | | | status (status) +... +d0 | main | region_enter | r1 | 0.010988 | | status | label:worktrees +d0 | main | region_leave | r1 | 0.011236 | 0.000248 | status | label:worktrees +d0 | main | region_enter | r1 | 0.011260 | | status | label:index +d0 | main | region_leave | r1 | 0.012542 | 0.001282 | status | label:index +d0 | main | region_enter | r1 | 0.012568 | | status | label:untracked +d0 | main | region_leave | r1 | 0.027149 | 0.014581 | status | label:untracked +d0 | main | region_enter | r1 | 0.027411 | | status | label:print +d0 | main | region_leave | r1 | 0.028741 | 0.001330 | status | label:print +d0 | main | exit | | 0.028778 | | | code:0 +d0 | main | atexit | | 0.028809 | | | code:0 +---------------- ++ +Regions may be nested. This causes messages to be indented in the +PERF target, for example. +Elapsed times are relative to the start of the corresponding nesting +level as expected. For example, if we add region message to: ++ +---------------- +static enum path_treatment read_directory_recursive(struct dir_struct *dir, + struct index_state *istate, const char *base, int baselen, + struct untracked_cache_dir *untracked, int check_only, + int stop_at_first_file, const struct pathspec *pathspec) +{ + enum path_treatment state, subdir_state, dir_state = path_none; + + trace2_region_enter_printf("dir", "read_recursive", NULL, "%.*s", baselen, base); + ... + trace2_region_leave_printf("dir", "read_recursive", NULL, "%.*s", baselen, base); + return dir_state; +} +---------------- ++ +We can further investigate the time spent scanning for untracked files. ++ +---------------- +$ export GIT_TRACE2_PERF_BRIEF=1 +$ export GIT_TRACE2_PERF=~/log.perf +$ git status +... +$ cat ~/log.perf +d0 | main | version | | | | | 2.20.1.162.gb4ccea44db.dirty +d0 | main | start | | 0.001173 | | | git status +d0 | main | def_repo | r1 | | | | worktree:/Users/jeffhost/work/gfw +d0 | main | cmd_name | | | | | status (status) +... +d0 | main | region_enter | r1 | 0.015047 | | status | label:untracked +d0 | main | region_enter | | 0.015132 | | dir | ..label:read_recursive +d0 | main | region_enter | | 0.016341 | | dir | ....label:read_recursive vcs-svn/ +d0 | main | region_leave | | 0.016422 | 0.000081 | dir | ....label:read_recursive vcs-svn/ +d0 | main | region_enter | | 0.016446 | | dir | ....label:read_recursive xdiff/ +d0 | main | region_leave | | 0.016522 | 0.000076 | dir | ....label:read_recursive xdiff/ +d0 | main | region_enter | | 0.016612 | | dir | ....label:read_recursive git-gui/ +d0 | main | region_enter | | 0.016698 | | dir | ......label:read_recursive git-gui/po/ +d0 | main | region_enter | | 0.016810 | | dir | ........label:read_recursive git-gui/po/glossary/ +d0 | main | region_leave | | 0.016863 | 0.000053 | dir | ........label:read_recursive git-gui/po/glossary/ +... +d0 | main | region_enter | | 0.031876 | | dir | ....label:read_recursive builtin/ +d0 | main | region_leave | | 0.032270 | 0.000394 | dir | ....label:read_recursive builtin/ +d0 | main | region_leave | | 0.032414 | 0.017282 | dir | ..label:read_recursive +d0 | main | region_leave | r1 | 0.032454 | 0.017407 | status | label:untracked +... +d0 | main | exit | | 0.034279 | | | code:0 +d0 | main | atexit | | 0.034322 | | | code:0 +---------------- ++ +Trace2 regions are similar to the existing trace_performance_enter() +and trace_performance_leave() routines, but are thread safe and +maintain per-thread stacks of timers. + +Data Messages:: + + Data messages added to a region. ++ +---------------- +int read_index_from(struct index_state *istate, const char *path, + const char *gitdir) +{ + trace2_region_enter_printf("index", "do_read_index", the_repository, "%s", path); + + ... + + trace2_data_intmax("index", the_repository, "read/version", istate->version); + trace2_data_intmax("index", the_repository, "read/cache_nr", istate->cache_nr); + + trace2_region_leave_printf("index", "do_read_index", the_repository, "%s", path); +} +---------------- ++ +This example shows that the index contained 3552 entries. ++ +---------------- +$ export GIT_TRACE2_PERF_BRIEF=1 +$ export GIT_TRACE2_PERF=~/log.perf +$ git status +... +$ cat ~/log.perf +d0 | main | version | | | | | 2.20.1.156.gf9916ae094.dirty +d0 | main | start | | 0.001173 | | | git status +d0 | main | def_repo | r1 | | | | worktree:/Users/jeffhost/work/gfw +d0 | main | cmd_name | | | | | status (status) +d0 | main | region_enter | r1 | 0.001791 | | index | label:do_read_index .git/index +d0 | main | data | r1 | 0.002494 | 0.000703 | index | ..read/version:2 +d0 | main | data | r1 | 0.002520 | 0.000729 | index | ..read/cache_nr:3552 +d0 | main | region_leave | r1 | 0.002539 | 0.000748 | index | label:do_read_index .git/index +... +---------------- + +Thread Events:: + + Thread messages added to a thread-proc. ++ +For example, the multithreaded preload-index code can be +instrumented with a region around the thread pool and then +per-thread start and exit events within the threadproc. ++ +---------------- +static void *preload_thread(void *_data) +{ + // start the per-thread clock and emit a message. + trace2_thread_start("preload_thread"); + + // report which chunk of the array this thread was assigned. + trace2_data_intmax("index", the_repository, "offset", p->offset); + trace2_data_intmax("index", the_repository, "count", nr); + + do { + ... + } while (--nr > 0); + ... + + // report elapsed time taken by this thread. + trace2_thread_exit(); + return NULL; +} + +void preload_index(struct index_state *index, + const struct pathspec *pathspec, + unsigned int refresh_flags) +{ + trace2_region_enter("index", "preload", the_repository); + + for (i = 0; i < threads; i++) { + ... /* create thread */ + } + + for (i = 0; i < threads; i++) { + ... /* join thread */ + } + + trace2_region_leave("index", "preload", the_repository); +} +---------------- ++ +In this example preload_index() was executed by the `main` thread +and started the `preload` region. Seven threads, named +`th01:preload_thread` through `th07:preload_thread`, were started. +Events from each thread are atomically appended to the shared target +stream as they occur so they may appear in random order with respect +other threads. Finally, the main thread waits for the threads to +finish and leaves the region. ++ +Data events are tagged with the active thread name. They are used +to report the per-thread parameters. ++ +---------------- +$ export GIT_TRACE2_PERF_BRIEF=1 +$ export GIT_TRACE2_PERF=~/log.perf +$ git status +... +$ cat ~/log.perf +... +d0 | main | region_enter | r1 | 0.002595 | | index | label:preload +d0 | th01:preload_thread | thread_start | | 0.002699 | | | +d0 | th02:preload_thread | thread_start | | 0.002721 | | | +d0 | th01:preload_thread | data | r1 | 0.002736 | 0.000037 | index | offset:0 +d0 | th02:preload_thread | data | r1 | 0.002751 | 0.000030 | index | offset:2032 +d0 | th03:preload_thread | thread_start | | 0.002711 | | | +d0 | th06:preload_thread | thread_start | | 0.002739 | | | +d0 | th01:preload_thread | data | r1 | 0.002766 | 0.000067 | index | count:508 +d0 | th06:preload_thread | data | r1 | 0.002856 | 0.000117 | index | offset:2540 +d0 | th03:preload_thread | data | r1 | 0.002824 | 0.000113 | index | offset:1016 +d0 | th04:preload_thread | thread_start | | 0.002710 | | | +d0 | th02:preload_thread | data | r1 | 0.002779 | 0.000058 | index | count:508 +d0 | th06:preload_thread | data | r1 | 0.002966 | 0.000227 | index | count:508 +d0 | th07:preload_thread | thread_start | | 0.002741 | | | +d0 | th07:preload_thread | data | r1 | 0.003017 | 0.000276 | index | offset:3048 +d0 | th05:preload_thread | thread_start | | 0.002712 | | | +d0 | th05:preload_thread | data | r1 | 0.003067 | 0.000355 | index | offset:1524 +d0 | th05:preload_thread | data | r1 | 0.003090 | 0.000378 | index | count:508 +d0 | th07:preload_thread | data | r1 | 0.003037 | 0.000296 | index | count:504 +d0 | th03:preload_thread | data | r1 | 0.002971 | 0.000260 | index | count:508 +d0 | th04:preload_thread | data | r1 | 0.002983 | 0.000273 | index | offset:508 +d0 | th04:preload_thread | data | r1 | 0.007311 | 0.004601 | index | count:508 +d0 | th05:preload_thread | thread_exit | | 0.008781 | 0.006069 | | +d0 | th01:preload_thread | thread_exit | | 0.009561 | 0.006862 | | +d0 | th03:preload_thread | thread_exit | | 0.009742 | 0.007031 | | +d0 | th06:preload_thread | thread_exit | | 0.009820 | 0.007081 | | +d0 | th02:preload_thread | thread_exit | | 0.010274 | 0.007553 | | +d0 | th07:preload_thread | thread_exit | | 0.010477 | 0.007736 | | +d0 | th04:preload_thread | thread_exit | | 0.011657 | 0.008947 | | +d0 | main | region_leave | r1 | 0.011717 | 0.009122 | index | label:preload +... +d0 | main | exit | | 0.029996 | | | code:0 +d0 | main | atexit | | 0.030027 | | | code:0 +---------------- ++ +In this example, the preload region took 0.009122 seconds. The 7 threads +took between 0.006069 and 0.008947 seconds to work on their portion of +the index. Thread "th01" worked on 508 items at offset 0. Thread "th02" +worked on 508 items at offset 2032. Thread "th04" worked on 508 items +at offset 508. ++ +This example also shows that thread names are assigned in a racy manner +as each thread starts and allocates TLS storage. + +== Future Work + +=== Relationship to the Existing Trace Api (api-trace.txt) + +There are a few issues to resolve before we can completely +switch to Trace2. + +* Updating existing tests that assume GIT_TRACE format messages. + +* How to best handle custom GIT_TRACE_<key> messages? + +** The GIT_TRACE_<key> mechanism allows each <key> to write to a +different file (in addition to just stderr). + +** Do we want to maintain that ability or simply write to the existing +Trace2 targets (and convert <key> to a "category"). diff --git a/Documentation/technical/api-tree-walking.txt b/Documentation/technical/api-tree-walking.txt deleted file mode 100644 index bde18622a8..0000000000 --- a/Documentation/technical/api-tree-walking.txt +++ /dev/null @@ -1,147 +0,0 @@ -tree walking API -================ - -The tree walking API is used to traverse and inspect trees. - -Data Structures ---------------- - -`struct name_entry`:: - - An entry in a tree. Each entry has a sha1 identifier, pathname, and - mode. - -`struct tree_desc`:: - - A semi-opaque data structure used to maintain the current state of the - walk. -+ -* `buffer` is a pointer into the memory representation of the tree. It always -points at the current entry being visited. - -* `size` counts the number of bytes left in the `buffer`. - -* `entry` points to the current entry being visited. - -`struct traverse_info`:: - - A structure used to maintain the state of a traversal. -+ -* `prev` points to the traverse_info which was used to descend into the -current tree. If this is the top-level tree `prev` will point to -a dummy traverse_info. - -* `name` is the entry for the current tree (if the tree is a subtree). - -* `pathlen` is the length of the full path for the current tree. - -* `conflicts` can be used by callbacks to maintain directory-file conflicts. - -* `fn` is a callback called for each entry in the tree. See Traversing for more -information. - -* `data` can be anything the `fn` callback would want to use. - -* `show_all_errors` tells whether to stop at the first error or not. - -Initializing ------------- - -`init_tree_desc`:: - - Initialize a `tree_desc` and decode its first entry. The buffer and - size parameters are assumed to be the same as the buffer and size - members of `struct tree`. - -`fill_tree_descriptor`:: - - Initialize a `tree_desc` and decode its first entry given the - object ID of a tree. Returns the `buffer` member if the latter - is a valid tree identifier and NULL otherwise. - -`setup_traverse_info`:: - - Initialize a `traverse_info` given the pathname of the tree to start - traversing from. The `base` argument is assumed to be the `path` - member of the `name_entry` being recursed into unless the tree is a - top-level tree in which case the empty string ("") is used. - -Walking -------- - -`tree_entry`:: - - Visit the next entry in a tree. Returns 1 when there are more entries - left to visit and 0 when all entries have been visited. This is - commonly used in the test of a while loop. - -`tree_entry_len`:: - - Calculate the length of a tree entry's pathname. This utilizes the - memory structure of a tree entry to avoid the overhead of using a - generic strlen(). - -`update_tree_entry`:: - - Walk to the next entry in a tree. This is commonly used in conjunction - with `tree_entry_extract` to inspect the current entry. - -`tree_entry_extract`:: - - Decode the entry currently being visited (the one pointed to by - `tree_desc's` `entry` member) and return the sha1 of the entry. The - `pathp` and `modep` arguments are set to the entry's pathname and mode - respectively. - -`get_tree_entry`:: - - Find an entry in a tree given a pathname and the sha1 of a tree to - search. Returns 0 if the entry is found and -1 otherwise. The third - and fourth parameters are set to the entry's sha1 and mode - respectively. - -Traversing ----------- - -`traverse_trees`:: - - Traverse `n` number of trees in parallel. The `fn` callback member of - `traverse_info` is called once for each tree entry. - -`traverse_callback_t`:: - The arguments passed to the traverse callback are as follows: -+ -* `n` counts the number of trees being traversed. - -* `mask` has its nth bit set if something exists in the nth entry. - -* `dirmask` has its nth bit set if the nth tree's entry is a directory. - -* `entry` is an array of size `n` where the nth entry is from the nth tree. - -* `info` maintains the state of the traversal. - -+ -Returning a negative value will terminate the traversal. Otherwise the -return value is treated as an update mask. If the nth bit is set the nth tree -will be updated and if the bit is not set the nth tree entry will be the -same in the next callback invocation. - -`make_traverse_path`:: - - Generate the full pathname of a tree entry based from the root of the - traversal. For example, if the traversal has recursed into another - tree named "bar" the pathname of an entry "baz" in the "bar" - tree would be "bar/baz". - -`traverse_path_len`:: - - Calculate the length of a pathname returned by `make_traverse_path`. - This utilizes the memory structure of a tree entry to avoid the - overhead of using a generic strlen(). - -Authors -------- - -Written by Junio C Hamano <gitster@pobox.com> and Linus Torvalds -<torvalds@linux-foundation.org> diff --git a/Documentation/technical/api-xdiff-interface.txt b/Documentation/technical/api-xdiff-interface.txt deleted file mode 100644 index 6296ecad1d..0000000000 --- a/Documentation/technical/api-xdiff-interface.txt +++ /dev/null @@ -1,7 +0,0 @@ -xdiff interface API -=================== - -Talk about our calling convention to xdiff library, including -xdiff_emit_consume_fn. - -(Dscho, JC) diff --git a/Documentation/technical/bundle-format.txt b/Documentation/technical/bundle-format.txt new file mode 100644 index 0000000000..0e828151a5 --- /dev/null +++ b/Documentation/technical/bundle-format.txt @@ -0,0 +1,48 @@ += Git bundle v2 format + +The Git bundle format is a format that represents both refs and Git objects. + +== Format + +We will use ABNF notation to define the Git bundle format. See +protocol-common.txt for the details. + +---- +bundle = signature *prerequisite *reference LF pack +signature = "# v2 git bundle" LF + +prerequisite = "-" obj-id SP comment LF +comment = *CHAR +reference = obj-id SP refname LF + +pack = ... ; packfile +---- + +== Semantics + +A Git bundle consists of three parts. + +* "Prerequisites" lists the objects that are NOT included in the bundle and the + reader of the bundle MUST already have, in order to use the data in the + bundle. The objects stored in the bundle may refer to prerequisite objects and + anything reachable from them (e.g. a tree object in the bundle can reference + a blob that is reachable from a prerequisite) and/or expressed as a delta + against prerequisite objects. + +* "References" record the tips of the history graph, iow, what the reader of the + bundle CAN "git fetch" from it. + +* "Pack" is the pack data stream "git fetch" would send, if you fetch from a + repository that has the references recorded in the "References" above into a + repository that has references pointing at the objects listed in + "Prerequisites" above. + +In the bundle format, there can be a comment following a prerequisite obj-id. +This is a comment and it has no specific meaning. The writer of the bundle MAY +put any string here. The reader of the bundle MUST ignore the comment. + +=== Note on the shallow clone and a Git bundle + +Note that the prerequisites does not represent a shallow-clone boundary. The +semantics of the prerequisites and the shallow-clone boundaries are different, +and the Git bundle v2 format cannot represent a shallow clone repository. diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt index cc0474ba3e..1beef17182 100644 --- a/Documentation/technical/commit-graph-format.txt +++ b/Documentation/technical/commit-graph-format.txt @@ -17,6 +17,9 @@ metadata, including: - The parents of the commit, stored using positional references within the graph file. +- The Bloom filter of the commit carrying the paths that were changed between + the commit and its first parent, if requested. + These positional references are stored as unsigned 32-bit integers corresponding to the array position within the list of commit OIDs. Due to some special constants we use to track parents, we can store at most @@ -44,8 +47,9 @@ HEADER: 1-byte number (C) of "chunks" - 1-byte (reserved for later use) - Current clients should ignore this value. + 1-byte number (B) of base commit-graphs + We infer the length (H*B) of the Base Graphs chunk + from this value. CHUNK LOOKUP: @@ -76,7 +80,7 @@ CHUNK DATA: of the ith commit. Stores value 0x7000000 if no parent in that position. If there are more than two parents, the second value has its most-significant bit on and the other bits store an array - position into the Large Edge List chunk. + position into the Extra Edge List chunk. * The next 8 bytes store the generation number of the commit and the commit time in seconds since EPOCH. The generation number uses the higher 30 bits of the first 4 bytes, while the commit @@ -84,7 +88,7 @@ CHUNK DATA: 2 bits of the lowest byte, storing the 33rd and 34th bit of the commit time. - Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional] + Extra Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional] This list of 4-byte values store the second through nth parents for all octopus merges. The second parent value in the commit data stores an array position within this list along with the most-significant bit @@ -92,6 +96,39 @@ CHUNK DATA: positions for the parents until reaching a value with the most-significant bit on. The other bits correspond to the position of the last parent. + Bloom Filter Index (ID: {'B', 'I', 'D', 'X'}) (N * 4 bytes) [Optional] + * The ith entry, BIDX[i], stores the number of bytes in all Bloom filters + from commit 0 to commit i (inclusive) in lexicographic order. The Bloom + filter for the i-th commit spans from BIDX[i-1] to BIDX[i] (plus header + length), where BIDX[-1] is 0. + * The BIDX chunk is ignored if the BDAT chunk is not present. + + Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional] + * It starts with header consisting of three unsigned 32-bit integers: + - Version of the hash algorithm being used. We currently only support + value 1 which corresponds to the 32-bit version of the murmur3 hash + implemented exactly as described in + https://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double + hashing technique using seed values 0x293ae76f and 0x7e646e2 as + described in https://doi.org/10.1007/978-3-540-30494-4_26 "Bloom Filters + in Probabilistic Verification" + - The number of times a path is hashed and hence the number of bit positions + that cumulatively determine whether a file is present in the commit. + - The minimum number of bits 'b' per entry in the Bloom filter. If the filter + contains 'n' entries, then the filter size is the minimum number of 64-bit + words that contain n*b bits. + * The rest of the chunk is the concatenation of all the computed Bloom + filters for the commits in lexicographic order. + * Note: Commits with no changes or more than 512 changes have Bloom filters + of length zero. + * The BDAT chunk is present if and only if BIDX is present. + + Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional] + This list of H-byte hashes describe a set of B commit-graph files that + form a commit-graph chain. The graph position for the ith commit in this + file's OID Lookup chunk is equal to i plus the number of commits in all + base graphs. If B is non-zero, this chunk must exist. + TRAILER: H-byte HASH-checksum of all of the above. diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt index 7805b0968c..808fa30b99 100644 --- a/Documentation/technical/commit-graph.txt +++ b/Documentation/technical/commit-graph.txt @@ -22,11 +22,11 @@ as "commit-graph" either in the .git/objects/info directory or in the info directory of an alternate. The commit-graph file stores the commit graph structure along with some -extra metadata to speed up graph walks. By listing commit OIDs in lexi- -cographic order, we can identify an integer position for each commit and -refer to the parents of a commit using those integer positions. We use -binary search to find initial commits and then use the integer positions -for fast lookups during the walk. +extra metadata to speed up graph walks. By listing commit OIDs in +lexicographic order, we can identify an integer position for each commit +and refer to the parents of a commit using those integer positions. We +use binary search to find initial commits and then use the integer +positions for fast lookups during the walk. A consumer may load the following info for a commit from the graph: @@ -85,7 +85,7 @@ have generation number represented by the macro GENERATION_NUMBER_ZERO = 0. Since the commit-graph file is closed under reachability, we can guarantee the following weaker condition on all commits: - If A and B are commits with generation numbers N amd M, respectively, + If A and B are commits with generation numbers N and M, respectively, and N < M, then A cannot reach B. Note how the strict inequality differs from the inequality when we have @@ -127,36 +127,210 @@ Design Details helpful for these clones, anyway. The commit-graph will not be read or written when shallow commits are present. -Future Work ------------ - -- After computing and storing generation numbers, we must make graph - walks aware of generation numbers to gain the performance benefits they - enable. This will mostly be accomplished by swapping a commit-date-ordered - priority queue with one ordered by generation number. The following - operations are important candidates: - - - 'log --topo-order' - - 'tag --merged' - -- A server could provide a commit-graph file as part of the network protocol - to avoid extra calculations by clients. This feature is only of benefit if - the user is willing to trust the file, because verifying the file is correct - is as hard as computing it from scratch. +Commit Graphs Chains +-------------------- + +Typically, repos grow with near-constant velocity (commits per day). Over time, +the number of commits added by a fetch operation is much smaller than the +number of commits in the full history. By creating a "chain" of commit-graphs, +we enable fast writes of new commit data without rewriting the entire commit +history -- at least, most of the time. + +## File Layout + +A commit-graph chain uses multiple files, and we use a fixed naming convention +to organize these files. Each commit-graph file has a name +`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex- +valued hash stored in the footer of that file (which is a hash of the file's +contents before that hash). For a chain of commit-graph files, a plain-text +file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the +hashes for the files in order from "lowest" to "highest". + +For example, if the `commit-graph-chain` file contains the lines + +``` + {hash0} + {hash1} + {hash2} +``` + +then the commit-graph chain looks like the following diagram: + + +-----------------------+ + | graph-{hash2}.graph | + +-----------------------+ + | + +-----------------------+ + | | + | graph-{hash1}.graph | + | | + +-----------------------+ + | + +-----------------------+ + | | + | | + | | + | graph-{hash0}.graph | + | | + | | + | | + +-----------------------+ + +Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of +commits in `graph-{hash1}.graph`, and X2 be the number of commits in +`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`, +then we interpret this as being the commit in position (X0 + X1 + i), and that +will be used as its "graph position". The commits in `graph-{hash2}.graph` use these +positions to refer to their parents, which may be in `graph-{hash1}.graph` or +`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking +its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 + +X2). + +Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data +specifying the hashes of all files in the lower layers. In the above example, +`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains +`{hash0}` and `{hash1}`. + +## Merging commit-graph files + +If we only added a new commit-graph file on every write, we would run into a +linear search problem through many commit-graph files. Instead, we use a merge +strategy to decide when the stack should collapse some number of levels. + +The diagram below shows such a collapse. As a set of new commits are added, it +is determined by the merge strategy that the files should collapse to +`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and +the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}` +file. + + +---------------------+ + | | + | (new commits) | + | | + +---------------------+ + | | + +-----------------------+ +---------------------+ + | graph-{hash2} |->| | + +-----------------------+ +---------------------+ + | | | + +-----------------------+ +---------------------+ + | | | | + | graph-{hash1} |->| | + | | | | + +-----------------------+ +---------------------+ + | tmp_graphXXX + +-----------------------+ + | | + | | + | | + | graph-{hash0} | + | | + | | + | | + +-----------------------+ + +During this process, the commits to write are combined, sorted and we write the +contents to a temporary file, all while holding a `commit-graph-chain.lock` +lock-file. When the file is flushed, we rename it to `graph-{hash3}` +according to the computed `{hash3}`. Finally, we write the new chain data to +`commit-graph-chain.lock`: + +``` + {hash3} + {hash0} +``` + +We then close the lock-file. + +## Merge Strategy + +When writing a set of commits that do not exist in the commit-graph stack of +height N, we default to creating a new file at level N + 1. We then decide to +merge with the Nth level if one of two conditions hold: + + 1. `--size-multiple=<X>` is specified or X = 2, and the number of commits in + level N is less than X times the number of commits in level N + 1. + + 2. `--max-commits=<C>` is specified with non-zero C and the number of commits + in level N + 1 is more than C commits. + +This decision cascades down the levels: when we merge a level we create a new +set of commits that then compares to the next level. + +The first condition bounds the number of levels to be logarithmic in the total +number of commits. The second condition bounds the total number of commits in +a `graph-{hashN}` file and not in the `commit-graph` file, preventing +significant performance issues when the stack merges and another process only +partially reads the previous stack. + +The merge strategy values (2 for the size multiple, 64,000 for the maximum +number of commits) could be extracted into config settings for full +flexibility. + +## Deleting graph-{hash} files + +After a new tip file is written, some `graph-{hash}` files may no longer +be part of a chain. It is important to remove these files from disk, eventually. +The main reason to delay removal is that another process could read the +`commit-graph-chain` file before it is rewritten, but then look for the +`graph-{hash}` files after they are deleted. + +To allow holding old split commit-graphs for a while after they are unreferenced, +we update the modified times of the files when they become unreferenced. Then, +we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}` +files whose modified times are older than a given expiry window. This window +defaults to zero, but can be changed using command-line arguments or a config +setting. + +## Chains across multiple object directories + +In a repo with alternates, we look for the `commit-graph-chain` file starting +in the local object directory and then in each alternate. The first file that +exists defines our chain. As we look for the `graph-{hash}` files for +each `{hash}` in the chain file, we follow the same pattern for the host +directories. + +This allows commit-graphs to be split across multiple forks in a fork network. +The typical case is a large "base" repo with many smaller forks. + +As the base repo advances, it will likely update and merge its commit-graph +chain more frequently than the forks. If a fork updates their commit-graph after +the base repo, then it should "reparent" the commit-graph chain onto the new +chain in the base repo. When reading each `graph-{hash}` file, we track +the object directory containing it. During a write of a new commit-graph file, +we check for any changes in the source object directory and read the +`commit-graph-chain` file for that source and create a new file based on those +files. During this "reparent" operation, we necessarily need to collapse all +levels in the fork, as all of the files are invalid against the new base file. + +It is crucial to be careful when cleaning up "unreferenced" `graph-{hash}.graph` +files in this scenario. It falls to the user to define the proper settings for +their custom environment: + + 1. When merging levels in the base repo, the unreferenced files may still be + referenced by chains from fork repos. + + 2. The expiry time should be set to a length of time such that every fork has + time to recompute their commit-graph chain to "reparent" onto the new base + file(s). + + 3. If the commit-graph chain is updated in the base, the fork will not have + access to the new chain until its chain is updated to reference those files. + (This may change in the future [5].) Related Links ------------- [0] https://bugs.chromium.org/p/git/issues/detail?id=8 Chromium work item for: Serialized Commit Graph -[1] https://public-inbox.org/git/20110713070517.GC18566@sigill.intra.peff.net/ +[1] https://lore.kernel.org/git/20110713070517.GC18566@sigill.intra.peff.net/ An abandoned patch that introduced generation numbers. -[2] https://public-inbox.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/ +[2] https://lore.kernel.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/ Discussion about generation numbers on commits and how they interact with fsck. -[3] https://public-inbox.org/git/20170908034739.4op3w4f2ma5s65ku@sigill.intra.peff.net/ +[3] https://lore.kernel.org/git/20170908034739.4op3w4f2ma5s65ku@sigill.intra.peff.net/ More discussion about generation numbers and not storing them inside commit objects. A valuable quote: @@ -168,5 +342,9 @@ Related Links commit objects (i.e., packv4 or something like the "metapacks" I proposed a few years ago)." -[4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u +[4] https://lore.kernel.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u A patch to remove the ahead-behind calculation from 'status'. + +[5] https://lore.kernel.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/ + A discussion of a "two-dimensional graph position" that can allow reading + multiple commit-graph chains at the same time. diff --git a/Documentation/technical/directory-rename-detection.txt b/Documentation/technical/directory-rename-detection.txt index 1c0086e287..844629c8c4 100644 --- a/Documentation/technical/directory-rename-detection.txt +++ b/Documentation/technical/directory-rename-detection.txt @@ -20,8 +20,8 @@ More interesting possibilities exist, though, such as: * one side of history renames x -> z, and the other renames some file to x/e, causing the need for the merge to do a transitive rename. - * one side of history renames x -> z, but also renames all files within - x. For example, x/a -> z/alpha, x/b -> z/bravo, etc. + * one side of history renames x -> z, but also renames all files within x. + For example, x/a -> z/alpha, x/b -> z/bravo, etc. * both 'x' and 'y' being merged into a single directory 'z', with a directory rename being detected for both x->z and y->z. diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index bc2ace2a6e..5b2db3be1e 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -456,7 +456,7 @@ packfile marked as UNREACHABLE_GARBAGE (using the PSRC field; see below). To avoid the race when writing new objects referring to an about-to-be-deleted object, code paths that write new objects will need to copy any objects from UNREACHABLE_GARBAGE packs that they -refer to to new, non-UNREACHABLE_GARBAGE packs (or loose objects). +refer to new, non-UNREACHABLE_GARBAGE packs (or loose objects). UNREACHABLE_GARBAGE are then safe to delete if their creation time (as indicated by the file's mtime) is long enough ago. @@ -531,7 +531,7 @@ Until Git protocol gains SHA-256 support, using SHA-256 based storage on public-facing Git servers is strongly discouraged. Once Git protocol gains SHA-256 support, SHA-256 based servers are likely not to support SHA-1 compatibility, to avoid what may be a very expensive -hash reencode during clone and to encourage peers to modernize. +hash re-encode during clone and to encourage peers to modernize. The design described here allows fetches by SHA-1 clients of a personal SHA-256 repository because it's not much more difficult than @@ -602,7 +602,7 @@ git --output-format=sha1 log abac87a^{sha1}..f787cac^{sha256} Choice of Hash -------------- -In early 2005, around the time that Git was written, Xiaoyun Wang, +In early 2005, around the time that Git was written, Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu announced an attack finding SHA-1 collisions in 2^69 operations. In August they published details. Luckily, no practical demonstrations of a collision in full SHA-1 were @@ -730,7 +730,7 @@ adoption. Using hash functions in parallel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -(e.g. https://public-inbox.org/git/22708.8913.864049.452252@chiark.greenend.org.uk/ ) +(e.g. https://lore.kernel.org/git/22708.8913.864049.452252@chiark.greenend.org.uk/ ) Objects newly created would be addressed by the new hash, but inside such an object (e.g. commit) it is still possible to address objects using the old hash function. @@ -783,7 +783,7 @@ bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com, sbeller@google.com Initial version sent to -http://public-inbox.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com +http://lore.kernel.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com 2017-03-03 jrnieder@gmail.com Incorporated suggestions from jonathantanmy and sbeller: @@ -820,8 +820,8 @@ Later history: edits. This document history is no longer being maintained as it would now be superfluous to the commit log -[1] http://public-inbox.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ -[2] http://public-inbox.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ -[3] http://public-inbox.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ -[4] http://public-inbox.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net -[5] https://public-inbox.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/ +[1] http://lore.kernel.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ +[2] http://lore.kernel.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ +[3] http://lore.kernel.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ +[4] http://lore.kernel.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net +[5] https://lore.kernel.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/ diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt index 9c5b6f0fac..51a79e63de 100644 --- a/Documentation/technical/http-protocol.txt +++ b/Documentation/technical/http-protocol.txt @@ -216,7 +216,7 @@ smart server reply: S: 001e# service=git-upload-pack\n S: 0000 S: 004895dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint\0multi_ack\n - S: 0042d049f6c27a2244e12041955e262a404c7faba355 refs/heads/master\n + S: 003fd049f6c27a2244e12041955e262a404c7faba355 refs/heads/master\n S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\n S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\n S: 0000 diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt index 7c4d67aa6a..faa25c5c52 100644 --- a/Documentation/technical/index-format.txt +++ b/Documentation/technical/index-format.txt @@ -318,7 +318,7 @@ The remaining data of each directory block is grouped by type: == End of Index Entry The End of Index Entry (EOIE) is used to locate the end of the variable - length index entries and the begining of the extensions. Code can take + length index entries and the beginning of the extensions. Code can take advantage of this to quickly locate the index extensions without having to parse through all of the index entries. @@ -351,7 +351,7 @@ The remaining data of each directory block is grouped by type: - A number of index offset entries each consisting of: - - 32-bit offset from the begining of the file to the first cache entry + - 32-bit offset from the beginning of the file to the first cache entry in this block of entries. - 32-bit count of cache entries in this block diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt index d7e57639f7..4e7631437a 100644 --- a/Documentation/technical/multi-pack-index.txt +++ b/Documentation/technical/multi-pack-index.txt @@ -36,7 +36,7 @@ Design Details directory of an alternate. It refers only to packfiles in that same directory. -- The pack.multiIndex config setting must be on to consume MIDX files. +- The core.multiPackIndex config setting must be on to consume MIDX files. - The file format includes parameters for the object ID hash function, so a future change of hash algorithm does not require @@ -102,8 +102,8 @@ Related Links [0] https://bugs.chromium.org/p/git/issues/detail?id=6 Chromium work item for: Multi-Pack Index (MIDX) -[1] https://public-inbox.org/git/20180107181459.222909-1-dstolee@microsoft.com/ +[1] https://lore.kernel.org/git/20180107181459.222909-1-dstolee@microsoft.com/ An earlier RFC for the multi-pack-index feature -[2] https://public-inbox.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/ +[2] https://lore.kernel.org/git/alpine.DEB.2.20.1803091557510.23109@alexmv-linux/ Git Merge 2018 Contributor's summit notes (includes discussion of MIDX) diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt index cab5bdd2ff..d3a142c652 100644 --- a/Documentation/technical/pack-format.txt +++ b/Documentation/technical/pack-format.txt @@ -315,10 +315,11 @@ CHUNK DATA: Stores two 4-byte values for every object. 1: The pack-int-id for the pack storing this object. 2: The offset within the pack. - If all offsets are less than 2^31, then the large offset chunk + If all offsets are less than 2^32, then the large offset chunk will not exist and offsets are stored as in IDX v1. If there is at least one offset value larger than 2^32-1, then - the large offset chunk must exist. If the large offset chunk + the large offset chunk must exist, and offsets larger than + 2^31-1 must be stored in it instead. If the large offset chunk exists and the 31st bit is on, then removing that bit reveals the row in the large offsets containing the 8-byte offset of this object. diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt index 6ac774d5f6..a4573d12ce 100644 --- a/Documentation/technical/pack-protocol.txt +++ b/Documentation/technical/pack-protocol.txt @@ -22,6 +22,16 @@ protocol-common.txt. When the grammar indicate `PKT-LINE(...)`, unless otherwise noted the usual pkt-line LF rules apply: the sender SHOULD include a LF, but the receiver MUST NOT complain if it is not present. +An error packet is a special pkt-line that contains an error string. + +---- + error-line = PKT-LINE("ERR" SP explanation-text) +---- + +Throughout the protocol, where `PKT-LINE(...)` is expected, an error packet MAY +be sent. Once this packet is sent by a client or a server, the data transfer +process defined in this protocol is terminated. + Transports ---------- There are three transports over which the packfile protocol is @@ -86,16 +96,9 @@ Basically what the Git client is doing to connect to an 'upload-pack' process on the server side over the Git protocol is this: $ echo -e -n \ - "0039git-upload-pack /schacon/gitbook.git\0host=example.com\0" | + "003agit-upload-pack /schacon/gitbook.git\0host=example.com\0" | nc -v example.com 9418 -If the server refuses the request for some reasons, it could abort -gracefully with an error message. - ----- - error-line = PKT-LINE("ERR" SP explanation-text) ----- - SSH Transport ------------- @@ -168,9 +171,9 @@ with a version number (if "version=1" is sent as an Extra Parameter), and a listing of each reference it has (all branches and tags) along with the object name that each reference currently points to. - $ echo -e -n "0044git-upload-pack /schacon/gitbook.git\0host=example.com\0\0version=1\0" | + $ echo -e -n "0045git-upload-pack /schacon/gitbook.git\0host=example.com\0\0version=1\0" | nc -v example.com 9418 - 000aversion 1 + 000eversion 1 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\0multi_ack thin-pack side-band side-band-64k ofs-delta shallow no-progress include-tag 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration @@ -398,12 +401,11 @@ from the client). Then the server will start sending its packfile data. ---- - server-response = *ack_multi ack / nak / error-line + server-response = *ack_multi ack / nak ack_multi = PKT-LINE("ACK" SP obj-id ack_status) ack_status = "continue" / "common" / "ready" ack = PKT-LINE("ACK" SP obj-id) nak = PKT-LINE("NAK") - error-line = PKT-LINE("ERR" SP explanation-text) ---- A simple clone may look like this (with no 'have' lines): @@ -642,7 +644,7 @@ update was successful, or 'ng [refname] [error]' if the update was not. command-ok = PKT-LINE("ok" SP refname) command-fail = PKT-LINE("ng" SP refname SP error-msg) - error-msg = 1*(OCTECT) ; where not "ok" + error-msg = 1*(OCTET) ; where not "ok" ---- Updates can be unsuccessful for a number of reasons. The reference can have @@ -655,14 +657,14 @@ can be rejected. An example client/server communication might look like this: ---- - S: 007c74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/local\0report-status delete-refs ofs-delta\n + S: 006274730d410fcb6603ace96f1dc55ea6196122532d refs/heads/local\0report-status delete-refs ofs-delta\n S: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe refs/heads/debug\n S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/master\n - S: 003f74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/team\n + S: 003d74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/team\n S: 0000 - C: 003e7d1665144a3a975c05f1f43902ddaf084e784dbe 74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/debug\n - C: 003e74730d410fcb6603ace96f1dc55ea6196122532d 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/master\n + C: 00677d1665144a3a975c05f1f43902ddaf084e784dbe 74730d410fcb6603ace96f1dc55ea6196122532d refs/heads/debug\n + C: 006874730d410fcb6603ace96f1dc55ea6196122532d 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a refs/heads/master\n C: 0000 C: [PACKDATA] diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index 1ef66bd788..b9e17e7a28 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -30,12 +30,20 @@ advance* during clone and fetch operations and thereby reduce download times and disk usage. Missing objects can later be "demand fetched" if/when needed. +A remote that can later provide the missing objects is called a +promisor remote, as it promises to send the objects when +requested. Initially Git supported only one promisor remote, the origin +remote from which the user cloned and that was configured in the +"extensions.partialClone" config option. Later support for more than +one promisor remote has been implemented. + Use of partial clone requires that the user be online and the origin -remote be available for on-demand fetching of missing objects. This may -or may not be problematic for the user. For example, if the user can -stay within the pre-selected subset of the source tree, they may not -encounter any missing objects. Alternatively, the user could try to -pre-fetch various objects if they know that they are going offline. +remote or other promisor remotes be available for on-demand fetching +of missing objects. This may or may not be problematic for the user. +For example, if the user can stay within the pre-selected subset of +the source tree, they may not encounter any missing objects. +Alternatively, the user could try to pre-fetch various objects if they +know that they are going offline. Non-Goals @@ -100,21 +108,21 @@ or commits that reference missing trees. Handling Missing Objects ------------------------ -- An object may be missing due to a partial clone or fetch, or missing due - to repository corruption. To differentiate these cases, the local - repository specially indicates such filtered packfiles obtained from the - promisor remote as "promisor packfiles". +- An object may be missing due to a partial clone or fetch, or missing + due to repository corruption. To differentiate these cases, the + local repository specially indicates such filtered packfiles + obtained from promisor remotes as "promisor packfiles". + These promisor packfiles consist of a "<name>.promisor" file with arbitrary contents (like the "<name>.keep" files), in addition to their "<name>.pack" and "<name>.idx" files. - The local repository considers a "promisor object" to be an object that - it knows (to the best of its ability) that the promisor remote has promised - that it has, either because the local repository has that object in one of + it knows (to the best of its ability) that promisor remotes have promised + that they have, either because the local repository has that object in one of its promisor packfiles, or because another promisor object refers to it. + -When Git encounters a missing object, Git can see if it a promisor object +When Git encounters a missing object, Git can see if it is a promisor object and handle it appropriately. If not, Git can report a corruption. + This means that there is no need for the client to explicitly maintain an @@ -123,12 +131,12 @@ expensive-to-modify list of missing objects.[a] - Since almost all Git code currently expects any referenced object to be present locally and because we do not want to force every command to do a dry-run first, a fallback mechanism is added to allow Git to attempt - to dynamically fetch missing objects from the promisor remote. + to dynamically fetch missing objects from promisor remotes. + When the normal object lookup fails to find an object, Git invokes -fetch-object to try to get the object from the server and then retry -the object lookup. This allows objects to be "faulted in" without -complicated prediction algorithms. +promisor_remote_get_direct() to try to get the object from a promisor +remote and then retry the object lookup. This allows objects to be +"faulted in" without complicated prediction algorithms. + For efficiency reasons, no check as to whether the missing object is actually a promisor object is performed. @@ -157,8 +165,7 @@ and prefetch those objects in bulk. + We are not happy with this global variable and would like to remove it, but that requires significant refactoring of the object code to pass an -additional flag. We hope that concurrent efforts to add an ODB API can -encompass this. +additional flag. Fetching Missing Objects @@ -182,21 +189,63 @@ has been updated to not use any object flags when the corresponding argument though they are not necessary. +Using many promisor remotes +--------------------------- + +Many promisor remotes can be configured and used. + +This allows for example a user to have multiple geographically-close +cache servers for fetching missing blobs while continuing to do +filtered `git-fetch` commands from the central server. + +When fetching objects, promisor remotes are tried one after the other +until all the objects have been fetched. + +Remotes that are considered "promisor" remotes are those specified by +the following configuration variables: + +- `extensions.partialClone = <name>` + +- `remote.<name>.promisor = true` + +- `remote.<name>.partialCloneFilter = ...` + +Only one promisor remote can be configured using the +`extensions.partialClone` config variable. This promisor remote will +be the last one tried when fetching objects. + +We decided to make it the last one we try, because it is likely that +someone using many promisor remotes is doing so because the other +promisor remotes are better for some reason (maybe they are closer or +faster for some kind of objects) than the origin, and the origin is +likely to be the remote specified by extensions.partialClone. + +This justification is not very strong, but one choice had to be made, +and anyway the long term plan should be to make the order somehow +fully configurable. + +For now though the other promisor remotes will be tried in the order +they appear in the config file. + Current Limitations ------------------- -- The remote used for a partial clone (or the first partial fetch - following a regular clone) is marked as the "promisor remote". +- It is not possible to specify the order in which the promisor + remotes are tried in other ways than the order in which they appear + in the config file. + -We are currently limited to a single promisor remote and only that -remote may be used for subsequent partial fetches. +It is also not possible to specify an order to be used when fetching +from one remote and a different order when fetching from another +remote. + +- It is not possible to push only specific objects to a promisor + remote. + -We accept this limitation because we believe initial users of this -feature will be using it on repositories with a strong single central -server. +It is not possible to push at the same time to multiple promisor +remote in a specific order. -- Dynamic object fetching will only ask the promisor remote for missing - objects. We assume that the promisor remote has a complete view of the +- Dynamic object fetching will only ask promisor remotes for missing + objects. We assume that promisor remotes have a complete view of the repository and can satisfy all such requests. - Repack essentially treats promisor and non-promisor packfiles as 2 @@ -218,15 +267,17 @@ server. Future Work ----------- -- Allow more than one promisor remote and define a strategy for fetching - missing objects from specific promisor remotes or of iterating over the - set of promisor remotes until a missing object is found. +- Improve the way to specify the order in which promisor remotes are + tried. + -A user might want to have multiple geographically-close cache servers -for fetching missing blobs while continuing to do filtered `git-fetch` -commands from the central server, for example. +For example this could allow to specify explicitly something like: +"When fetching from this remote, I want to use these promisor remotes +in this order, though, when pushing or fetching to that remote, I want +to use those promisor remotes in that order." + +- Allow pushing to promisor remotes. + -Or the user might want to work in a triangular work flow with multiple +The user might want to work in a triangular work flow with multiple promisor remotes that each have an incomplete view of the repository. - Allow repack to work on promisor packfiles (while keeping them distinct @@ -299,26 +350,26 @@ Related Links [0] https://crbug.com/git/2 Bug#2: Partial Clone -[1] https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com/ + +[1] https://lore.kernel.org/git/20170113155253.1644-1-benpeart@microsoft.com/ + Subject: [RFC] Add support for downloading blobs on demand + Date: Fri, 13 Jan 2017 10:52:53 -0500 -[2] https://public-inbox.org/git/cover.1506714999.git.jonathantanmy@google.com/ + +[2] https://lore.kernel.org/git/cover.1506714999.git.jonathantanmy@google.com/ + Subject: [PATCH 00/18] Partial clone (from clone to lazy fetch in 18 patches) + Date: Fri, 29 Sep 2017 13:11:36 -0700 -[3] https://public-inbox.org/git/20170426221346.25337-1-jonathantanmy@google.com/ + +[3] https://lore.kernel.org/git/20170426221346.25337-1-jonathantanmy@google.com/ + Subject: Proposal for missing blob support in Git repos + Date: Wed, 26 Apr 2017 15:13:46 -0700 -[4] https://public-inbox.org/git/1488999039-37631-1-git-send-email-git@jeffhostetler.com/ + +[4] https://lore.kernel.org/git/1488999039-37631-1-git-send-email-git@jeffhostetler.com/ + Subject: [PATCH 00/10] RFC Partial Clone and Fetch + Date: Wed, 8 Mar 2017 18:50:29 +0000 -[5] https://public-inbox.org/git/20170505152802.6724-1-benpeart@microsoft.com/ + +[5] https://lore.kernel.org/git/20170505152802.6724-1-benpeart@microsoft.com/ + Subject: [PATCH v7 00/10] refactor the filter process code into a reusable module + Date: Fri, 5 May 2017 11:27:52 -0400 -[6] https://public-inbox.org/git/20170714132651.170708-1-benpeart@microsoft.com/ + +[6] https://lore.kernel.org/git/20170714132651.170708-1-benpeart@microsoft.com/ + Subject: [RFC/PATCH v2 0/1] Add support for downloading blobs on demand + Date: Fri, 14 Jul 2017 09:26:50 -0400 diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt index 332d209b58..2b267c0da6 100644 --- a/Documentation/technical/protocol-capabilities.txt +++ b/Documentation/technical/protocol-capabilities.txt @@ -1,6 +1,10 @@ Git Protocol Capabilities ========================= +NOTE: this document describes capabilities for versions 0 and 1 of the pack +protocol. For version 2, please refer to the link:protocol-v2.html[protocol-v2] +doc. + Servers SHOULD support all capabilities defined in this document. On the very first line of the initial server response of either @@ -172,6 +176,20 @@ agent strings are purely informative for statistics and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features. +symref +------ + +This parameterized capability is used to inform the receiver which symbolic ref +points to which ref; for example, "symref=HEAD:refs/heads/master" tells the +receiver that HEAD points to master. This capability can be repeated to +represent multiple symrefs. + +Servers SHOULD include this capability for the HEAD symref if it is one of the +refs being sent. + +Clients MAY use the parameters from this capability to select the proper initial +branch when cloning a repository. + shallow ------- diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 09e4e0273f..3996d70891 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -1,5 +1,5 @@ - Git Wire Protocol, Version 2 -============================== +Git Wire Protocol, Version 2 +============================ This document presents a specification for a version 2 of Git's wire protocol. Protocol v2 will improve upon v1 in the following ways: @@ -22,8 +22,8 @@ will be commands which a client can request be executed. Once a command has completed, a client can reuse the connection and request that other commands be executed. - Packet-Line Framing ---------------------- +Packet-Line Framing +------------------- All communication is done using packet-line framing, just as in v1. See `Documentation/technical/pack-protocol.txt` and @@ -33,9 +33,11 @@ In protocol v2 these special packets will have the following semantics: * '0000' Flush Packet (flush-pkt) - indicates the end of a message * '0001' Delimiter Packet (delim-pkt) - separates sections of a message + * '0002' Message Packet (response-end-pkt) - indicates the end of a response + for stateless connections - Initial Client Request ------------------------- +Initial Client Request +---------------------- In general a client can request to speak protocol v2 by sending `version=2` through the respective side-channel for the transport being @@ -43,22 +45,22 @@ used which inevitably sets `GIT_PROTOCOL`. More information can be found in `pack-protocol.txt` and `http-protocol.txt`. In all cases the response from the server is the capability advertisement. - Git Transport -~~~~~~~~~~~~~~~ +Git Transport +~~~~~~~~~~~~~ When using the git:// transport, you can request to use protocol v2 by sending "version=2" as an extra parameter: 003egit-upload-pack /project.git\0host=myserver.com\0\0version=2\0 - SSH and File Transport -~~~~~~~~~~~~~~~~~~~~~~~~ +SSH and File Transport +~~~~~~~~~~~~~~~~~~~~~~ When using either the ssh:// or file:// transport, the GIT_PROTOCOL environment variable must be set explicitly to include "version=2". - HTTP Transport -~~~~~~~~~~~~~~~~ +HTTP Transport +~~~~~~~~~~~~~~ When using the http:// or https:// transport a client makes a "smart" info/refs request as described in `http-protocol.txt` and requests that @@ -79,8 +81,8 @@ A v2 server would reply: Subsequent requests are then made directly to the service `$GIT_URL/git-upload-pack`. (This works the same for git-receive-pack). - Capability Advertisement --------------------------- +Capability Advertisement +------------------------ A server which decides to communicate (based on a request from a client) using protocol version 2, notifies the client by sending a version string @@ -101,8 +103,8 @@ to be executed by the client. key = 1*(ALPHA | DIGIT | "-_") value = 1*(ALPHA | DIGIT | " -_.,?\/{}[]()<>!@#$%^&*+=:;") - Command Request ------------------ +Command Request +--------------- After receiving the capability advertisement, a client can then issue a request to select the command it wants with any particular capabilities @@ -137,11 +139,11 @@ command be executed or can terminate the connection. A client may optionally send an empty request consisting of just a flush-pkt to indicate that no more requests will be made. - Capabilities --------------- +Capabilities +------------ There are two different types of capabilities: normal capabilities, -which can be used to to convey information or alter the behavior of a +which can be used to convey information or alter the behavior of a request, and commands, which are the core actions that a client wants to perform (fetch, push, etc). @@ -153,8 +155,8 @@ management on the server side in order to function correctly. This permits simple round-robin load-balancing on the server side, without needing to worry about state management. - agent -~~~~~~~ +agent +~~~~~ The server can advertise the `agent` capability with a value `X` (in the form `agent=X`) to notify the client that the server is running version @@ -168,8 +170,8 @@ printable ASCII characters except space (i.e., the byte range 32 < x < and debugging purposes, and MUST NOT be used to programmatically assume the presence or absence of particular features. - ls-refs -~~~~~~~~~ +ls-refs +~~~~~~~ `ls-refs` is the command used to request a reference advertisement in v2. Unlike the current reference advertisement, ls-refs takes in arguments @@ -199,8 +201,8 @@ The output of ls-refs is as follows: symref = "symref-target:" symref-target peeled = "peeled:" obj-id - fetch -~~~~~~~ +fetch +~~~~~ `fetch` is the command used to fetch a packfile in v2. It can be looked at as a modified version of the v1 fetch where the ref-advertisement is @@ -252,7 +254,7 @@ A `fetch` request can take the following arguments: ofs-delta Indicate that the client understands PACKv2 with delta referring to its base by position in pack rather than by an oid. That is, - they can read OBJ_OFS_DELTA (ake type 6) in a packfile. + they can read OBJ_OFS_DELTA (aka type 6) in a packfile. If the 'shallow' feature is advertised the following arguments can be included in the clients request as well as the potential addition of the @@ -296,7 +298,13 @@ included in the client's request: Request that various objects from the packfile be omitted using one of several filtering techniques. These are intended for use with partial clone and partial fetch operations. See - `rev-list` for possible "filter-spec" values. + `rev-list` for possible "filter-spec" values. When communicating + with other processes, senders SHOULD translate scaled integers + (e.g. "1k") into a fully-expanded form (e.g. "1024") to aid + interoperability with older receivers that may not understand + newly-invented scaling suffixes. However, receivers SHOULD + accept the following suffixes: 'k', 'm', and 'g' for 1024, + 1048576, and 1073741824, respectively. If the 'ref-in-want' feature is advertised, the following argument can be included in the client's request as well as the potential addition of @@ -307,6 +315,16 @@ the 'wanted-refs' section in the server's response as explained below. particular ref, where <ref> is the full name of a ref on the server. +If the 'sideband-all' feature is advertised, the following argument can be +included in the client's request: + + sideband-all + Instruct the server to send the whole response multiplexed, not just + the packfile section. All non-flush and non-delim PKT-LINE in the + response (not only in the packfile section) will then start with a byte + indicating its sideband (1, 2, or 3), and the server may send "0005\2" + (a PKT-LINE of sideband 2 with no payload) as a keepalive packet. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. @@ -428,8 +446,8 @@ header. 2 - progress messages 3 - fatal error message just before stream aborts - server-option -~~~~~~~~~~~~~~~ +server-option +~~~~~~~~~~~~~ If advertised, indicates that any number of server specific options can be included in a request. This is done by sending each option as a diff --git a/Documentation/technical/racy-git.txt b/Documentation/technical/racy-git.txt index 4a8be4d144..ceda4bbfda 100644 --- a/Documentation/technical/racy-git.txt +++ b/Documentation/technical/racy-git.txt @@ -51,7 +51,7 @@ of git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git only fixes the issue for file systems with exactly 1 ns or 1 s resolution. Other file systems are still broken in current Linux kernels (e.g. CEPH, CIFS, NTFS, UDF), see -https://lkml.org/lkml/2015/6/9/714 +https://lore.kernel.org/lkml/5577240D.7020309@gmail.com/ Racy Git -------- diff --git a/Documentation/technical/rerere.txt b/Documentation/technical/rerere.txt index aa22d7ace8..af5f9fc24f 100644 --- a/Documentation/technical/rerere.txt +++ b/Documentation/technical/rerere.txt @@ -117,7 +117,7 @@ early A became C or B, a late X became Y or Z". We can see there are 4 combinations of ("B or C", "C or B") x ("X or Y", "Y or X"). By sorting, the conflict is given its canonical name, namely, "an -early part became B or C, a late part becames X or Y", and whenever +early part became B or C, a late part became X or Y", and whenever any of these four patterns appear, and we can get to the same conflict and resolution that we saw earlier. |