diff options
Diffstat (limited to 'Documentation/technical')
| -rw-r--r-- | Documentation/technical/api-argv-array.txt | 8 | ||||
| -rw-r--r-- | Documentation/technical/api-builtin.txt | 17 | ||||
| -rw-r--r-- | Documentation/technical/api-gitattributes.txt | 2 | ||||
| -rw-r--r-- | Documentation/technical/api-hash.txt | 52 | ||||
| -rw-r--r-- | Documentation/technical/api-hashmap.txt | 234 | ||||
| -rw-r--r-- | Documentation/technical/api-parse-options.txt | 10 | ||||
| -rw-r--r-- | Documentation/technical/api-remote.txt | 20 | ||||
| -rw-r--r-- | Documentation/technical/api-run-command.txt | 7 | ||||
| -rw-r--r-- | Documentation/technical/api-strbuf.txt | 28 | ||||
| -rw-r--r-- | Documentation/technical/api-string-list.txt | 2 | ||||
| -rw-r--r-- | Documentation/technical/bitmap-format.txt | 164 | ||||
| -rw-r--r-- | Documentation/technical/http-protocol.txt | 231 | ||||
| -rw-r--r-- | Documentation/technical/index-format.txt | 35 | ||||
| -rw-r--r-- | Documentation/technical/pack-protocol.txt | 18 | ||||
| -rw-r--r-- | Documentation/technical/protocol-capabilities.txt | 49 | ||||
| -rw-r--r-- | Documentation/technical/protocol-common.txt | 2 |
16 files changed, 660 insertions, 219 deletions
diff --git a/Documentation/technical/api-argv-array.txt b/Documentation/technical/api-argv-array.txt index a6b7d83a8e..1a797812fb 100644 --- a/Documentation/technical/api-argv-array.txt +++ b/Documentation/technical/api-argv-array.txt @@ -53,11 +53,3 @@ Functions `argv_array_clear`:: Free all memory associated with the array and return it to the initial, empty state. - -`argv_array_detach`:: - Detach the argv array from the `struct argv_array`, transferring - ownership of the allocated array and strings. - -`argv_array_free_detached`:: - Free the memory allocated by a `struct argv_array` that was later - detached and is now no longer needed. diff --git a/Documentation/technical/api-builtin.txt b/Documentation/technical/api-builtin.txt index f3c1357b7c..22a39b9299 100644 --- a/Documentation/technical/api-builtin.txt +++ b/Documentation/technical/api-builtin.txt @@ -14,19 +14,22 @@ Git: . Add the external declaration for the function to `builtin.h`. -. Add the command to `commands[]` table in `handle_internal_command()`, - defined in `git.c`. The entry should look like: +. Add the command to the `commands[]` table defined in `git.c`. + The entry should look like: { "foo", cmd_foo, <options> }, + where options is the bitwise-or of: `RUN_SETUP`:: - - Make sure there is a Git directory to work on, and if there is a - work tree, chdir to the top of it if the command was invoked - in a subdirectory. If there is no work tree, no chdir() is - done. + If there is not a Git directory to work on, abort. If there + is a work tree, chdir to the top of it if the command was + invoked in a subdirectory. If there is no work tree, no + chdir() is done. + +`RUN_SETUP_GENTLY`:: + If there is a Git directory, chdir as per RUN_SETUP, otherwise, + don't chdir anywhere. `USE_PAGER`:: diff --git a/Documentation/technical/api-gitattributes.txt b/Documentation/technical/api-gitattributes.txt index ce363b6305..2602668677 100644 --- a/Documentation/technical/api-gitattributes.txt +++ b/Documentation/technical/api-gitattributes.txt @@ -99,7 +99,7 @@ static void setup_check(void) The attribute is Unset, by listing the name of the attribute prefixed with a dash - for the path. } else if (ATTR_UNSET(value)) { - The attribute is not set nor unset for the path. + The attribute is neither set nor unset for the path. } else if (!strcmp(value, "input")) { If none of ATTR_TRUE(), ATTR_FALSE(), or ATTR_UNSET() is true, the value is a string set in the gitattributes diff --git a/Documentation/technical/api-hash.txt b/Documentation/technical/api-hash.txt deleted file mode 100644 index e5061e0677..0000000000 --- a/Documentation/technical/api-hash.txt +++ /dev/null @@ -1,52 +0,0 @@ -hash API -======== - -The hash API is a collection of simple hash table functions. Users are expected -to implement their own hashing. - -Data Structures ---------------- - -`struct hash_table`:: - - The hash table structure. The `array` member points to the hash table - entries. The `size` member counts the total number of valid and invalid - entries in the table. The `nr` member keeps track of the number of - valid entries. - -`struct hash_table_entry`:: - - An opaque structure representing an entry in the hash table. The `hash` - member is the entry's hash key and the `ptr` member is the entry's - value. - -Functions ---------- - -`init_hash`:: - - Initialize the hash table. - -`free_hash`:: - - Release memory associated with the hash table. - -`insert_hash`:: - - Insert a pointer into the hash table. If an entry with that hash - already exists, a pointer to the existing entry's value is returned. - Otherwise NULL is returned. This allows callers to implement - chaining, etc. - -`lookup_hash`:: - - Lookup an entry in the hash table. If an entry with that hash exists - the entry's value is returned. Otherwise NULL is returned. - -`for_each_hash`:: - - Call a function for each entry in the hash table. The function is - expected to take the entry's value as its only argument and return an - int. If the function returns a negative int the loop is aborted - immediately. Otherwise, the return value is accumulated and the sum - returned upon completion of the loop. diff --git a/Documentation/technical/api-hashmap.txt b/Documentation/technical/api-hashmap.txt new file mode 100644 index 0000000000..b977ae8bbb --- /dev/null +++ b/Documentation/technical/api-hashmap.txt @@ -0,0 +1,234 @@ +hashmap API +=========== + +The hashmap API is a generic implementation of hash-based key-value mappings. + +Data Structures +--------------- + +`struct hashmap`:: + + The hash table structure. ++ +The `size` member keeps track of the total number of entries. The `cmpfn` +member is a function used to compare two entries for equality. The `table` and +`tablesize` members store the hash table and its size, respectively. + +`struct hashmap_entry`:: + + An opaque structure representing an entry in the hash table, which must + be used as first member of user data structures. Ideally it should be + followed by an int-sized member to prevent unused memory on 64-bit + systems due to alignment. ++ +The `hash` member is the entry's hash code and the `next` member points to the +next entry in case of collisions (i.e. if multiple entries map to the same +bucket). + +`struct hashmap_iter`:: + + An iterator structure, to be used with hashmap_iter_* functions. + +Types +----- + +`int (*hashmap_cmp_fn)(const void *entry, const void *entry_or_key, const void *keydata)`:: + + User-supplied function to test two hashmap entries for equality. Shall + return 0 if the entries are equal. ++ +This function is always called with non-NULL `entry` / `entry_or_key` +parameters that have the same hash code. When looking up an entry, the `key` +and `keydata` parameters to hashmap_get and hashmap_remove are always passed +as second and third argument, respectively. Otherwise, `keydata` is NULL. + +Functions +--------- + +`unsigned int strhash(const char *buf)`:: +`unsigned int strihash(const char *buf)`:: +`unsigned int memhash(const void *buf, size_t len)`:: +`unsigned int memihash(const void *buf, size_t len)`:: + + Ready-to-use hash functions for strings, using the FNV-1 algorithm (see + http://www.isthe.com/chongo/tech/comp/fnv). ++ +`strhash` and `strihash` take 0-terminated strings, while `memhash` and +`memihash` operate on arbitrary-length memory. ++ +`strihash` and `memihash` are case insensitive versions. + +`void hashmap_init(struct hashmap *map, hashmap_cmp_fn equals_function, size_t initial_size)`:: + + Initializes a hashmap structure. ++ +`map` is the hashmap to initialize. ++ +The `equals_function` can be specified to compare two entries for equality. +If NULL, entries are considered equal if their hash codes are equal. ++ +If the total number of entries is known in advance, the `initial_size` +parameter may be used to preallocate a sufficiently large table and thus +prevent expensive resizing. If 0, the table is dynamically resized. + +`void hashmap_free(struct hashmap *map, int free_entries)`:: + + Frees a hashmap structure and allocated memory. ++ +`map` is the hashmap to free. ++ +If `free_entries` is true, each hashmap_entry in the map is freed as well +(using stdlib's free()). + +`void hashmap_entry_init(void *entry, unsigned int hash)`:: + + Initializes a hashmap_entry structure. ++ +`entry` points to the entry to initialize. ++ +`hash` is the hash code of the entry. + +`void *hashmap_get(const struct hashmap *map, const void *key, const void *keydata)`:: + + Returns the hashmap entry for the specified key, or NULL if not found. ++ +`map` is the hashmap structure. ++ +`key` is a hashmap_entry structure (or user data structure that starts with +hashmap_entry) that has at least been initialized with the proper hash code +(via `hashmap_entry_init`). ++ +If an entry with matching hash code is found, `key` and `keydata` are passed +to `hashmap_cmp_fn` to decide whether the entry matches the key. + +`void *hashmap_get_next(const struct hashmap *map, const void *entry)`:: + + Returns the next equal hashmap entry, or NULL if not found. This can be + used to iterate over duplicate entries (see `hashmap_add`). ++ +`map` is the hashmap structure. ++ +`entry` is the hashmap_entry to start the search from, obtained via a previous +call to `hashmap_get` or `hashmap_get_next`. + +`void hashmap_add(struct hashmap *map, void *entry)`:: + + Adds a hashmap entry. This allows to add duplicate entries (i.e. + separate values with the same key according to hashmap_cmp_fn). ++ +`map` is the hashmap structure. ++ +`entry` is the entry to add. + +`void *hashmap_put(struct hashmap *map, void *entry)`:: + + Adds or replaces a hashmap entry. If the hashmap contains duplicate + entries equal to the specified entry, only one of them will be replaced. ++ +`map` is the hashmap structure. ++ +`entry` is the entry to add or replace. ++ +Returns the replaced entry, or NULL if not found (i.e. the entry was added). + +`void *hashmap_remove(struct hashmap *map, const void *key, const void *keydata)`:: + + Removes a hashmap entry matching the specified key. If the hashmap + contains duplicate entries equal to the specified key, only one of + them will be removed. ++ +`map` is the hashmap structure. ++ +`key` is a hashmap_entry structure (or user data structure that starts with +hashmap_entry) that has at least been initialized with the proper hash code +(via `hashmap_entry_init`). ++ +If an entry with matching hash code is found, `key` and `keydata` are +passed to `hashmap_cmp_fn` to decide whether the entry matches the key. ++ +Returns the removed entry, or NULL if not found. + +`void hashmap_iter_init(struct hashmap *map, struct hashmap_iter *iter)`:: +`void *hashmap_iter_next(struct hashmap_iter *iter)`:: +`void *hashmap_iter_first(struct hashmap *map, struct hashmap_iter *iter)`:: + + Used to iterate over all entries of a hashmap. ++ +`hashmap_iter_init` initializes a `hashmap_iter` structure. ++ +`hashmap_iter_next` returns the next hashmap_entry, or NULL if there are no +more entries. ++ +`hashmap_iter_first` is a combination of both (i.e. initializes the iterator +and returns the first entry, if any). + +Usage example +------------- + +Here's a simple usage example that maps long keys to double values. +------------ +struct hashmap map; + +struct long2double { + struct hashmap_entry ent; /* must be the first member! */ + long key; + double value; +}; + +static int long2double_cmp(const struct long2double *e1, const struct long2double *e2, const void *unused) +{ + return !(e1->key == e2->key); +} + +void long2double_init(void) +{ + hashmap_init(&map, (hashmap_cmp_fn) long2double_cmp, 0); +} + +void long2double_free(void) +{ + hashmap_free(&map, 1); +} + +static struct long2double *find_entry(long key) +{ + struct long2double k; + hashmap_entry_init(&k, memhash(&key, sizeof(long))); + k.key = key; + return hashmap_get(&map, &k, NULL); +} + +double get_value(long key) +{ + struct long2double *e = find_entry(key); + return e ? e->value : 0; +} + +void set_value(long key, double value) +{ + struct long2double *e = find_entry(key); + if (!e) { + e = malloc(sizeof(struct long2double)); + hashmap_entry_init(e, memhash(&key, sizeof(long))); + e->key = key; + hashmap_add(&map, e); + } + e->value = value; +} +------------ + +Using variable-sized keys +------------------------- + +The `hashmap_entry_get` and `hashmap_entry_remove` functions expect an ordinary +`hashmap_entry` structure as key to find the correct entry. If the key data is +variable-sized (e.g. a FLEX_ARRAY string) or quite large, it is undesirable +to create a full-fledged entry structure on the heap and copy all the key data +into the structure. + +In this case, the `keydata` parameter can be used to pass +variable-sized key data directly to the comparison function, and the `key` +parameter can be a stripped-down, fixed size entry structure allocated on the +stack. + +See test-hashmap.c for an example using arbitrary-length strings as keys. diff --git a/Documentation/technical/api-parse-options.txt b/Documentation/technical/api-parse-options.txt index 0be2b5159f..1f2db31312 100644 --- a/Documentation/technical/api-parse-options.txt +++ b/Documentation/technical/api-parse-options.txt @@ -29,9 +29,9 @@ that allow to change the behavior of a command. The parse-options API allows: -* 'sticked' and 'separate form' of options with arguments. - `-oArg` is sticked, `-o Arg` is separate form. - `--option=Arg` is sticked, `--option Arg` is separate form. +* 'stuck' and 'separate form' of options with arguments. + `-oArg` is stuck, `-o Arg` is separate form. + `--option=Arg` is stuck, `--option Arg` is separate form. * Long options may be 'abbreviated', as long as the abbreviation is unambiguous. @@ -160,10 +160,6 @@ There are some macros to easily define options: `int_var` is set to `integer` with `--option`, and reset to zero with `--no-option`. -`OPT_SET_PTR(short, long, &ptr_var, description, ptr)`:: - Introduce a boolean option. - If used, set `ptr_var` to `ptr`. - `OPT_STRING(short, long, &str_var, arg_str, description)`:: Introduce an option with string argument. The string argument is put into `str_var`. diff --git a/Documentation/technical/api-remote.txt b/Documentation/technical/api-remote.txt index 4be87768f6..5d245aa9d1 100644 --- a/Documentation/technical/api-remote.txt +++ b/Documentation/technical/api-remote.txt @@ -58,16 +58,16 @@ default remote, given the current branch and configuration. struct refspec -------------- -A struct refspec holds the parsed interpretation of a refspec. If it -will force updates (starts with a '+'), force is true. If it is a -pattern (sides end with '*') pattern is true. src and dest are the two -sides (if a pattern, only the part outside of the wildcards); if there -is only one side, it is src, and dst is NULL; if sides exist but are -empty (i.e., the refspec either starts or ends with ':'), the -corresponding side is "". - -This parsing can be done to an array of strings to give an array of -struct refpsecs with parse_ref_spec(). +A struct refspec holds the parsed interpretation of a refspec. If it +will force updates (starts with a '+'), force is true. If it is a +pattern (sides end with '*') pattern is true. src and dest are the +two sides (including '*' characters if present); if there is only one +side, it is src, and dst is NULL; if sides exist but are empty (i.e., +the refspec either starts or ends with ':'), the corresponding side is +"". + +An array of strings can be parsed into an array of struct refspecs +using parse_fetch_refspec() or parse_push_refspec(). remote_find_tracking(), given a remote and a struct refspec with either src or dst filled out, will fill out the other such that the diff --git a/Documentation/technical/api-run-command.txt b/Documentation/technical/api-run-command.txt index 5d7d7f2d32..69510ae57a 100644 --- a/Documentation/technical/api-run-command.txt +++ b/Documentation/technical/api-run-command.txt @@ -109,6 +109,13 @@ terminated), of which .argv[0] is the program name to run (usually without a path). If the command to run is a git command, set argv[0] to the command name without the 'git-' prefix and set .git_cmd = 1. +Note that the ownership of the memory pointed to by .argv stays with the +caller, but it should survive until `finish_command` completes. If the +.argv member is NULL, `start_command` will point it at the .args +`argv_array` (so you may use one or the other, but you must use exactly +one). The memory in .args will be cleaned up automatically during +`finish_command` (or during `start_command` when it is unsuccessful). + The members .in, .out, .err are used to redirect stdin, stdout, stderr as follows: diff --git a/Documentation/technical/api-strbuf.txt b/Documentation/technical/api-strbuf.txt index 3350d97dda..f9c06a7573 100644 --- a/Documentation/technical/api-strbuf.txt +++ b/Documentation/technical/api-strbuf.txt @@ -7,10 +7,10 @@ use the mem* functions than a str* one (memchr vs. strchr e.g.). Though, one has to be careful about the fact that str* functions often stop on NULs and that strbufs may have embedded NULs. -An strbuf is NUL terminated for convenience, but no function in the +A strbuf is NUL terminated for convenience, but no function in the strbuf API actually relies on the string being free of NULs. -strbufs has some invariants that are very important to keep in mind: +strbufs have some invariants that are very important to keep in mind: . The `buf` member is never NULL, so it can be used in any usual C string operations safely. strbuf's _have_ to be initialized either by @@ -56,8 +56,8 @@ Data structures * `struct strbuf` This is the string buffer structure. The `len` member can be used to -determine the current length of the string, and `buf` member provides access to -the string itself. +determine the current length of the string, and `buf` member provides +access to the string itself. Functions --------- @@ -121,10 +121,28 @@ Functions * Related to the contents of the buffer +`strbuf_trim`:: + + Strip whitespace from the beginning and end of a string. + Equivalent to performing `strbuf_rtrim()` followed by `strbuf_ltrim()`. + `strbuf_rtrim`:: Strip whitespace from the end of a string. +`strbuf_ltrim`:: + + Strip whitespace from the beginning of a string. + +`strbuf_reencode`:: + + Replace the contents of the strbuf with a reencoded form. Returns -1 + on error, 0 on success. + +`strbuf_tolower`:: + + Lowercase each character in the buffer using `tolower`. + `strbuf_cmp`:: Compare two buffers. Returns an integer less than, equal to, or greater @@ -184,7 +202,7 @@ strbuf_addstr(sb, "immediate string"); `strbuf_addbuf`:: - Copy the contents of an other buffer at the end of the current one. + Copy the contents of another buffer at the end of the current one. `strbuf_adddup`:: diff --git a/Documentation/technical/api-string-list.txt b/Documentation/technical/api-string-list.txt index 20be348834..f1add51efe 100644 --- a/Documentation/technical/api-string-list.txt +++ b/Documentation/technical/api-string-list.txt @@ -200,3 +200,5 @@ Represents the list itself. You should not tamper with it. . Setting the `strdup_strings` member to 1 will strdup() the strings before adding them, see above. +. The `compare_strings_fn` member is used to specify a custom compare + function, otherwise `strcmp()` is used as the default function. diff --git a/Documentation/technical/bitmap-format.txt b/Documentation/technical/bitmap-format.txt new file mode 100644 index 0000000000..f8c18a0f7a --- /dev/null +++ b/Documentation/technical/bitmap-format.txt @@ -0,0 +1,164 @@ +GIT bitmap v1 format +==================== + + - A header appears at the beginning: + + 4-byte signature: {'B', 'I', 'T', 'M'} + + 2-byte version number (network byte order) + The current implementation only supports version 1 + of the bitmap index (the same one as JGit). + + 2-byte flags (network byte order) + + The following flags are supported: + + - BITMAP_OPT_FULL_DAG (0x1) REQUIRED + This flag must always be present. It implies that the bitmap + index has been generated for a packfile with full closure + (i.e. where every single object in the packfile can find + its parent links inside the same packfile). This is a + requirement for the bitmap index format, also present in JGit, + that greatly reduces the complexity of the implementation. + + - BITMAP_OPT_HASH_CACHE (0x4) + If present, the end of the bitmap file contains + `N` 32-bit name-hash values, one per object in the + pack. The format and meaning of the name-hash is + described below. + + 4-byte entry count (network byte order) + + The total count of entries (bitmapped commits) in this bitmap index. + + 20-byte checksum + + The SHA1 checksum of the pack this bitmap index belongs to. + + - 4 EWAH bitmaps that act as type indexes + + Type indexes are serialized after the hash cache in the shape + of four EWAH bitmaps stored consecutively (see Appendix A for + the serialization format of an EWAH bitmap). + + There is a bitmap for each Git object type, stored in the following + order: + + - Commits + - Trees + - Blobs + - Tags + + In each bitmap, the `n`th bit is set to true if the `n`th object + in the packfile is of that type. + + The obvious consequence is that the OR of all 4 bitmaps will result + in a full set (all bits set), and the AND of all 4 bitmaps will + result in an empty bitmap (no bits set). + + - N entries with compressed bitmaps, one for each indexed commit + + Where `N` is the total amount of entries in this bitmap index. + Each entry contains the following: + + - 4-byte object position (network byte order) + The position **in the index for the packfile** where the + bitmap for this commit is found. + + - 1-byte XOR-offset + The xor offset used to compress this bitmap. For an entry + in position `x`, a XOR offset of `y` means that the actual + bitmap representing this commit is composed by XORing the + bitmap for this entry with the bitmap in entry `x-y` (i.e. + the bitmap `y` entries before this one). + + Note that this compression can be recursive. In order to + XOR this entry with a previous one, the previous entry needs + to be decompressed first, and so on. + + The hard-limit for this offset is 160 (an entry can only be + xor'ed against one of the 160 entries preceding it). This + number is always positive, and hence entries are always xor'ed + with **previous** bitmaps, not bitmaps that will come afterwards + in the index. + + - 1-byte flags for this bitmap + At the moment the only available flag is `0x1`, which hints + that this bitmap can be re-used when rebuilding bitmap indexes + for the repository. + + - The compressed bitmap itself, see Appendix A. + +== Appendix A: Serialization format for an EWAH bitmap + +Ewah bitmaps are serialized in the same protocol as the JAVAEWAH +library, making them backwards compatible with the JGit +implementation: + + - 4-byte number of bits of the resulting UNCOMPRESSED bitmap + + - 4-byte number of words of the COMPRESSED bitmap, when stored + + - N x 8-byte words, as specified by the previous field + + This is the actual content of the compressed bitmap. + + - 4-byte position of the current RLW for the compressed + bitmap + +All words are stored in network byte order for their corresponding +sizes. + +The compressed bitmap is stored in a form of run-length encoding, as +follows. It consists of a concatenation of an arbitrary number of +chunks. Each chunk consists of one or more 64-bit words + + H L_1 L_2 L_3 .... L_M + +H is called RLW (run length word). It consists of (from lower to higher +order bits): + + - 1 bit: the repeated bit B + + - 32 bits: repetition count K (unsigned) + + - 31 bits: literal word count M (unsigned) + +The bitstream represented by the above chunk is then: + + - K repetitions of B + + - The bits stored in `L_1` through `L_M`. Within a word, bits at + lower order come earlier in the stream than those at higher + order. + +The next word after `L_M` (if any) must again be a RLW, for the next +chunk. For efficient appending to the bitstream, the EWAH stores a +pointer to the last RLW in the stream. + + +== Appendix B: Optional Bitmap Sections + +These sections may or may not be present in the `.bitmap` file; their +presence is indicated by the header flags section described above. + +Name-hash cache +--------------- + +If the BITMAP_OPT_HASH_CACHE flag is set, the end of the bitmap contains +a cache of 32-bit values, one per object in the pack. The value at +position `i` is the hash of the pathname at which the `i`th object +(counting in index order) in the pack can be found. This can be fed +into the delta heuristics to compare objects with similar pathnames. + +The hash algorithm used is: + + hash = 0; + while ((c = *name++)) + if (!isspace(c)) + hash = (hash >> 2) + (c << 24); + +Note that this hashing scheme is tied to the BITMAP_OPT_HASH_CACHE flag. +If implementations want to choose a different hashing scheme, they are +free to do so, but MUST allocate a new header flag (because comparing +hashes made under two different schemes would be pointless). diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt index d21d77d1de..229f845dfa 100644 --- a/Documentation/technical/http-protocol.txt +++ b/Documentation/technical/http-protocol.txt @@ -20,13 +20,13 @@ URL syntax documented by RFC 1738, so they are of the form: http://<host>:<port>/<path>?<searchpart> -Within this documentation the placeholder $GIT_URL will stand for +Within this documentation the placeholder `$GIT_URL` will stand for the http:// repository URL entered by the end-user. -Servers SHOULD handle all requests to locations matching $GIT_URL, as +Servers SHOULD handle all requests to locations matching `$GIT_URL`, as both the "smart" and "dumb" HTTP protocols used by Git operate by appending additional path components onto the end of the user -supplied $GIT_URL string. +supplied `$GIT_URL` string. An example of a dumb client requesting for a loose object: @@ -43,10 +43,10 @@ An example of a request to a submodule: $GIT_URL: http://example.com/git/repo.git/path/submodule.git URL request: http://example.com/git/repo.git/path/submodule.git/info/refs -Clients MUST strip a trailing '/', if present, from the user supplied -$GIT_URL string to prevent empty path tokens ('//') from appearing +Clients MUST strip a trailing `/`, if present, from the user supplied +`$GIT_URL` string to prevent empty path tokens (`//`) from appearing in any URL sent to a server. Compatible clients MUST expand -'$GIT_URL/info/refs' as 'foo/info/refs' and not 'foo//info/refs'. +`$GIT_URL/info/refs` as `foo/info/refs` and not `foo//info/refs`. Authentication @@ -60,7 +60,7 @@ Because Git repositories are accessed by standard path components server administrators MAY use directory based permissions within their HTTP server to control repository access. -Clients SHOULD support Basic authentication as described by RFC 2616. +Clients SHOULD support Basic authentication as described by RFC 2617. Servers SHOULD support Basic authentication by relying upon the HTTP server placed in front of the Git server software. @@ -103,14 +103,14 @@ Except where noted, all standard HTTP behavior SHOULD be assumed by both client and server. This includes (but is not necessarily limited to): -If there is no repository at $GIT_URL, or the resource pointed to by a -location matching $GIT_URL does not exist, the server MUST NOT respond -with '200 OK' response. A server SHOULD respond with -'404 Not Found', '410 Gone', or any other suitable HTTP status code +If there is no repository at `$GIT_URL`, or the resource pointed to by a +location matching `$GIT_URL` does not exist, the server MUST NOT respond +with `200 OK` response. A server SHOULD respond with +`404 Not Found`, `410 Gone`, or any other suitable HTTP status code which does not imply the resource exists as requested. -If there is a repository at $GIT_URL, but access is not currently -permitted, the server MUST respond with the '403 Forbidden' HTTP +If there is a repository at `$GIT_URL`, but access is not currently +permitted, the server MUST respond with the `403 Forbidden` HTTP status code. Servers SHOULD support both HTTP 1.0 and HTTP 1.1. @@ -126,9 +126,9 @@ Servers MAY return ETag and/or Last-Modified headers. Clients MAY revalidate cached entities by including If-Modified-Since and/or If-None-Match request headers. -Servers MAY return '304 Not Modified' if the relevant headers appear +Servers MAY return `304 Not Modified` if the relevant headers appear in the request and the entity has not changed. Clients MUST treat -'304 Not Modified' identical to '200 OK' by reusing the cached entity. +`304 Not Modified` identical to `200 OK` by reusing the cached entity. Clients MAY reuse a cached entity without revalidation if the Cache-Control and/or Expires header permits caching. Clients and @@ -148,7 +148,7 @@ HTTP clients that only support the "dumb" protocol MUST discover references by making a request for the special info/refs file of the repository. -Dumb HTTP clients MUST make a GET request to $GIT_URL/info/refs, +Dumb HTTP clients MUST make a `GET` request to `$GIT_URL/info/refs`, without any search/query parameters. C: GET $GIT_URL/info/refs HTTP/1.0 @@ -161,21 +161,21 @@ without any search/query parameters. S: a3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{} The Content-Type of the returned info/refs entity SHOULD be -"text/plain; charset=utf-8", but MAY be any content type. +`text/plain; charset=utf-8`, but MAY be any content type. Clients MUST NOT attempt to validate the returned Content-Type. Dumb servers MUST NOT return a return type starting with -"application/x-git-". +`application/x-git-`. Cache-Control headers MAY be returned to disable caching of the returned entity. When examining the response clients SHOULD only examine the HTTP -status code. Valid responses are '200 OK', or '304 Not Modified'. +status code. Valid responses are `200 OK`, or `304 Not Modified`. The returned content is a UNIX formatted text file describing each ref and its known value. The file SHOULD be sorted by name according to the C locale ordering. The file SHOULD NOT include -the default ref named 'HEAD'. +the default ref named `HEAD`. info_refs = *( ref_record ) ref_record = any_ref / peeled_ref @@ -192,13 +192,14 @@ HTTP clients that support the "smart" protocol (or both the a parameterized request for the info/refs file of the repository. The request MUST contain exactly one query parameter, -'service=$servicename', where $servicename MUST be the service +`service=$servicename`, where `$servicename` MUST be the service name the client wishes to contact to complete the operation. The request MUST NOT contain additional query parameters. C: GET $GIT_URL/info/refs?service=git-upload-pack HTTP/1.0 - dumb server reply: +dumb server reply: + S: 200 OK S: S: 95dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint @@ -206,7 +207,8 @@ The request MUST NOT contain additional query parameters. S: 2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0 S: a3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{} - smart server reply: +smart server reply: + S: 200 OK S: Content-Type: application/x-git-upload-pack-advertisement S: Cache-Control: no-cache @@ -228,7 +230,7 @@ Smart Server Response ^^^^^^^^^^^^^^^^^^^^^ If the server does not recognize the requested service name, or the requested service name has been disabled by the server administrator, -the server MUST respond with the '403 Forbidden' HTTP status code. +the server MUST respond with the `403 Forbidden` HTTP status code. Otherwise, smart servers MUST respond with the smart server reply format for the requested service name. @@ -236,35 +238,35 @@ format for the requested service name. Cache-Control headers SHOULD be used to disable caching of the returned entity. -The Content-Type MUST be 'application/x-$servicename-advertisement'. +The Content-Type MUST be `application/x-$servicename-advertisement`. Clients SHOULD fall back to the dumb protocol if another content type is returned. When falling back to the dumb protocol clients -SHOULD NOT make an additional request to $GIT_URL/info/refs, but +SHOULD NOT make an additional request to `$GIT_URL/info/refs`, but instead SHOULD use the response already in hand. Clients MUST NOT continue if they do not support the dumb protocol. -Clients MUST validate the status code is either '200 OK' or -'304 Not Modified'. +Clients MUST validate the status code is either `200 OK` or +`304 Not Modified`. Clients MUST validate the first five bytes of the response entity -matches the regex "^[0-9a-f]{4}#". If this test fails, clients +matches the regex `^[0-9a-f]{4}#`. If this test fails, clients MUST NOT continue. Clients MUST parse the entire response as a sequence of pkt-line records. -Clients MUST verify the first pkt-line is "# service=$servicename". +Clients MUST verify the first pkt-line is `# service=$servicename`. Servers MUST set $servicename to be the request parameter value. Servers SHOULD include an LF at the end of this line. Clients MUST ignore an LF at the end of the line. -Servers MUST terminate the response with the magic "0000" end +Servers MUST terminate the response with the magic `0000` end pkt-line marker. The returned response is a pkt-line stream describing each ref and its known value. The stream SHOULD be sorted by name according to the C locale ordering. The stream SHOULD include the default ref -named 'HEAD' as the first ref. The stream MUST include capability +named `HEAD` as the first ref. The stream MUST include capability declarations behind a NUL on the first ref. smart_reply = PKT-LINE("# service=$servicename" LF) @@ -286,12 +288,13 @@ declarations behind a NUL on the first ref. peeled_ref = PKT-LINE(obj-id SP name LF) PKT-LINE(obj-id SP name "^{}" LF + Smart Service git-upload-pack ------------------------------ -This service reads from the repository pointed to by $GIT_URL. +This service reads from the repository pointed to by `$GIT_URL`. Clients MUST first perform ref discovery with -'$GIT_URL/info/refs?service=git-upload-pack'. +`$GIT_URL/info/refs?service=git-upload-pack`. C: POST $GIT_URL/git-upload-pack HTTP/1.0 C: Content-Type: application/x-git-upload-pack-request @@ -313,10 +316,10 @@ to prevent caching of the response. Servers SHOULD support all capabilities defined here. -Clients MUST send at least one 'want' command in the request body. -Clients MUST NOT reference an id in a 'want' command which did not +Clients MUST send at least one "want" command in the request body. +Clients MUST NOT reference an id in a "want" command which did not appear in the response obtained through ref discovery unless the -server advertises capability "allow-tip-sha1-in-want". +server advertises capability `allow-tip-sha1-in-want`. compute_request = want_list have_list @@ -332,128 +335,128 @@ server advertises capability "allow-tip-sha1-in-want". have_list = *PKT-LINE("have" SP id LF) TODO: Document this further. -TODO: Don't use uppercase for variable names below. The Negotiation Algorithm ~~~~~~~~~~~~~~~~~~~~~~~~~ The computation to select the minimal pack proceeds as follows -(c = client, s = server): +(C = client, S = server): + +'init step:' + +C: Use ref discovery to obtain the advertised refs. + +C: Place any object seen into set `advertised`. - init step: - (c) Use ref discovery to obtain the advertised refs. - (c) Place any object seen into set ADVERTISED. +C: Build an empty set, `common`, to hold the objects that are later + determined to be on both ends. - (c) Build an empty set, COMMON, to hold the objects that are later - determined to be on both ends. - (c) Build a set, WANT, of the objects from ADVERTISED the client - wants to fetch, based on what it saw during ref discovery. +C: Build a set, `want`, of the objects from `advertised` the client + wants to fetch, based on what it saw during ref discovery. - (c) Start a queue, C_PENDING, ordered by commit time (popping newest - first). Add all client refs. When a commit is popped from - the queue its parents SHOULD be automatically inserted back. - Commits MUST only enter the queue once. +C: Start a queue, `c_pending`, ordered by commit time (popping newest + first). Add all client refs. When a commit is popped from + the queue its parents SHOULD be automatically inserted back. + Commits MUST only enter the queue once. - one compute step: - (c) Send one $GIT_URL/git-upload-pack request: +'one compute step:' - C: 0032want <WANT #1>............................... - C: 0032want <WANT #2>............................... +C: Send one `$GIT_URL/git-upload-pack` request: + + C: 0032want <want #1>............................... + C: 0032want <want #2>............................... .... - C: 0032have <COMMON #1>............................. - C: 0032have <COMMON #2>............................. + C: 0032have <common #1>............................. + C: 0032have <common #2>............................. .... - C: 0032have <HAVE #1>............................... - C: 0032have <HAVE #2>............................... + C: 0032have <have #1>............................... + C: 0032have <have #2>............................... .... C: 0000 - The stream is organized into "commands", with each command - appearing by itself in a pkt-line. Within a command line - the text leading up to the first space is the command name, - and the remainder of the line to the first LF is the value. - Command lines are terminated with an LF as the last byte of - the pkt-line value. +The stream is organized into "commands", with each command +appearing by itself in a pkt-line. Within a command line, +the text leading up to the first space is the command name, +and the remainder of the line to the first LF is the value. +Command lines are terminated with an LF as the last byte of +the pkt-line value. - Commands MUST appear in the following order, if they appear - at all in the request stream: +Commands MUST appear in the following order, if they appear +at all in the request stream: - * want - * have +* "want" +* "have" - The stream is terminated by a pkt-line flush ("0000"). +The stream is terminated by a pkt-line flush (`0000`). - A single "want" or "have" command MUST have one hex formatted - SHA-1 as its value. Multiple SHA-1s MUST be sent by sending - multiple commands. +A single "want" or "have" command MUST have one hex formatted +SHA-1 as its value. Multiple SHA-1s MUST be sent by sending +multiple commands. - The HAVE list is created by popping the first 32 commits - from C_PENDING. Less can be supplied if C_PENDING empties. +The `have` list is created by popping the first 32 commits +from `c_pending`. Less can be supplied if `c_pending` empties. - If the client has sent 256 HAVE commits and has not yet - received one of those back from S_COMMON, or the client has - emptied C_PENDING it SHOULD include a "done" command to let - the server know it won't proceed: +If the client has sent 256 "have" commits and has not yet +received one of those back from `s_common`, or the client has +emptied `c_pending` it SHOULD include a "done" command to let +the server know it won't proceed: C: 0009done - (s) Parse the git-upload-pack request: - - Verify all objects in WANT are directly reachable from refs. - - The server MAY walk backwards through history or through - the reflog to permit slightly stale requests. +S: Parse the git-upload-pack request: - If no WANT objects are received, send an error: +Verify all objects in `want` are directly reachable from refs. -TODO: Define error if no want lines are requested. +The server MAY walk backwards through history or through +the reflog to permit slightly stale requests. - If any WANT object is not reachable, send an error: +If no "want" objects are received, send an error: +TODO: Define error if no "want" lines are requested. -TODO: Define error if an invalid want is requested. +If any "want" object is not reachable, send an error: +TODO: Define error if an invalid "want" is requested. - Create an empty list, S_COMMON. +Create an empty list, `s_common`. - If 'have' was sent: +If "have" was sent: - Loop through the objects in the order supplied by the client. - For each object, if the server has the object reachable from - a ref, add it to S_COMMON. If a commit is added to S_COMMON, - do not add any ancestors, even if they also appear in HAVE. +Loop through the objects in the order supplied by the client. - (s) Send the git-upload-pack response: +For each object, if the server has the object reachable from +a ref, add it to `s_common`. If a commit is added to `s_common`, +do not add any ancestors, even if they also appear in `have`. - If the server has found a closed set of objects to pack or the - request ends with "done", it replies with the pack. +S: Send the git-upload-pack response: +If the server has found a closed set of objects to pack or the +request ends with "done", it replies with the pack. TODO: Document the pack based response - S: PACK... - The returned stream is the side-band-64k protocol supported - by the git-upload-pack service, and the pack is embedded into - stream 1. Progress messages from the server side MAY appear - in stream 2. + S: PACK... - Here a "closed set of objects" is defined to have at least - one path from every WANT to at least one COMMON object. +The returned stream is the side-band-64k protocol supported +by the git-upload-pack service, and the pack is embedded into +stream 1. Progress messages from the server side MAY appear +in stream 2. - If the server needs more information, it replies with a - status continue response: +Here a "closed set of objects" is defined to have at least +one path from every "want" to at least one "common" object. +If the server needs more information, it replies with a +status continue response: TODO: Document the non-pack response - (c) Parse the upload-pack response: - -TODO: Document parsing response +C: Parse the upload-pack response: + TODO: Document parsing response - Do another compute step. +'Do another compute step.' Smart Service git-receive-pack ------------------------------ -This service reads from the repository pointed to by $GIT_URL. +This service reads from the repository pointed to by `$GIT_URL`. Clients MUST first perform ref discovery with -'$GIT_URL/info/refs?service=git-receive-pack'. +`$GIT_URL/info/refs?service=git-receive-pack`. C: POST $GIT_URL/git-receive-pack HTTP/1.0 C: Content-Type: application/x-git-receive-pack-request @@ -497,7 +500,7 @@ TODO: Document this further. References ---------- -link:http://www.ietf.org/rfc/rfc1738.txt[RFC 1738: Uniform Resource Locators (URL)] -link:http://www.ietf.org/rfc/rfc2616.txt[RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1] +http://www.ietf.org/rfc/rfc1738.txt[RFC 1738: Uniform Resource Locators (URL)] +http://www.ietf.org/rfc/rfc2616.txt[RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1] link:technical/pack-protocol.html link:technical/protocol-capabilities.html diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt index f352a9b22e..fe6f31667d 100644 --- a/Documentation/technical/index-format.txt +++ b/Documentation/technical/index-format.txt @@ -129,6 +129,9 @@ Git index format (Version 4) In version 4, the padding after the pathname does not exist. + Interpretation of index entries in split index mode is completely + different. See below for details. + == Extensions === Cached tree @@ -198,3 +201,35 @@ Git index format - At most three 160-bit object names of the entry in stages from 1 to 3 (nothing is written for a missing stage). +=== Split index + + In split index mode, the majority of index entries could be stored + in a separate file. This extension records the changes to be made on + top of that to produce the final index. + + The signature for this extension is { 'l', 'i, 'n', 'k' }. + + The extension consists of: + + - 160-bit SHA-1 of the shared index file. The shared index file path + is $GIT_DIR/sharedindex.<SHA-1>. If all 160 bits are zero, the + index does not require a shared index file. + + - An ewah-encoded delete bitmap, each bit represents an entry in the + shared index. If a bit is set, its corresponding entry in the + shared index will be removed from the final index. Note, because + a delete operation changes index entry positions, but we do need + original positions in replace phase, it's best to just mark + entries for removal, then do a mass deletion after replacement. + + - An ewah-encoded replace bitmap, each bit represents an entry in + the shared index. If a bit is set, its corresponding entry in the + shared index will be replaced with an entry in this index + file. All replaced entries are stored in sorted order in this + index. The first "1" bit in the replace bitmap corresponds to the + first index entry, the second "1" bit to the second entry and so + on. Replaced entries may have empty path names to save space. + + The remaining index entries after replaced ones will be added to the + final index. These added entries are also sorted by entry namme then + stage. diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt index b898e97988..18dea8d15f 100644 --- a/Documentation/technical/pack-protocol.txt +++ b/Documentation/technical/pack-protocol.txt @@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag. ---- advertised-refs = (no-refs / list-of-refs) + *shallow flush-pkt no-refs = PKT-LINE(zero-id SP "capabilities^{}" @@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag. other-tip = obj-id SP refname LF other-peeled = obj-id SP refname "^{}" LF + shallow = PKT-LINE("shallow" SP obj-id) + capability-list = capability *(SP capability) capability = 1*(LC_ALPHA / DIGIT / "-" / "_") LC_ALPHA = %x61-7A @@ -234,10 +237,10 @@ The client now sends the maximum commit history depth it wants for this transaction, which is the number of commits it wants from the tip of the history, if any, as a 'deepen' line. A depth of 0 is the same as not making a depth request. The client does not want to receive -any commits beyond this depth, nor objects needed only to complete -those commits. Commits whose parents are not received as a result are -defined as shallow and marked as such in the server. This information -is sent back to the client in the next step. +any commits beyond this depth, nor does it want objects needed only to +complete those commits. Commits whose parents are not received as a +result are defined as shallow and marked as such in the server. This +information is sent back to the client in the next step. Once all the 'want's and 'shallow's (and optional 'deepen') are transferred, clients MUST send a flush-pkt, to tell the server side @@ -335,7 +338,8 @@ during a prior round. This helps to ensure that at least one common ancestor is found before we give up entirely. Once the 'done' line is read from the client, the server will either -send a final 'ACK obj-id' or it will send a 'NAK'. The server only sends +send a final 'ACK obj-id' or it will send a 'NAK'. 'obj-id' is the object +name of the last commit determined to be common. The server only sends ACK after 'done' if there is at least one common base and multi_ack or multi_ack_detailed is enabled. The server always sends NAK after 'done' if there is no common base found. @@ -461,7 +465,9 @@ contain all the objects that the server will need to complete the new references. ---- - update-request = command-list [pack-file] + update-request = *shallow command-list [pack-file] + + shallow = PKT-LINE("shallow" SP obj-id) command-list = PKT-LINE(command NUL capability-list LF) *PKT-LINE(command LF) diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt index fd8ffa5df3..e174343847 100644 --- a/Documentation/technical/protocol-capabilities.txt +++ b/Documentation/technical/protocol-capabilities.txt @@ -69,17 +69,50 @@ ends. Without multi_ack the client would have sent that c-b-a chain anyway, interleaved with S-R-Q. +multi_ack_detailed +------------------ +This is an extension of multi_ack that permits client to better +understand the server's in-memory state. See pack-protocol.txt, +section "Packfile Negotiation" for more information. + +no-done +------- +This capability should only be used with the smart HTTP protocol. If +multi_ack_detailed and no-done are both present, then the sender is +free to immediately send a pack following its first "ACK obj-id ready" +message. + +Without no-done in the smart HTTP protocol, the server session would +end and the client has to make another trip to send "done" before +the server can send the pack. no-done removes the last round and +thus slightly reduces latency. + thin-pack --------- -This capability means that the server can send a 'thin' pack, a pack -which does not contain base objects; if those base objects are available -on client side. Client requests 'thin-pack' capability when it -understands how to "thicken" it by adding required delta bases making -it self-contained. - -Client MUST NOT request 'thin-pack' capability if it cannot turn a thin -pack into a self-contained pack. +A thin pack is one with deltas which reference base objects not +contained within the pack (but are known to exist at the receiving +end). This can reduce the network traffic significantly, but it +requires the receiving end to know how to "thicken" these packs by +adding the missing bases to the pack. + +The upload-pack server advertises 'thin-pack' when it can generate +and send a thin pack. A client requests the 'thin-pack' capability +when it understands how to "thicken" it, notifying the server that +it can receive such a pack. A client MUST NOT request the +'thin-pack' capability if it cannot turn a thin pack into a +self-contained pack. + +Receive-pack, on the other hand, is assumed by default to be able to +handle thin packs, but can ask the client not to use the feature by +advertising the 'no-thin' capability. A client MUST NOT send a thin +pack if the server advertises the 'no-thin' capability. + +The reasons for this asymmetry are historical. The receive-pack +program did not exist until after the invention of thin packs, so +historically the reference implementation of receive-pack always +understood thin packs. Adding 'no-thin' later allowed receive-pack +to disable the feature in a backwards-compatible manner. side-band, side-band-64k diff --git a/Documentation/technical/protocol-common.txt b/Documentation/technical/protocol-common.txt index fb7ff084f8..889985f707 100644 --- a/Documentation/technical/protocol-common.txt +++ b/Documentation/technical/protocol-common.txt @@ -39,7 +39,7 @@ More specifically, they: caret `^`, colon `:`, question-mark `?`, asterisk `*`, or open bracket `[` anywhere. -. They cannot end with a slash `/` nor a dot `.`. +. They cannot end with a slash `/` or a dot `.`. . They cannot end with the sequence `.lock`. |