diff options
Diffstat (limited to 'vendor/github.com/tetratelabs/wazero/RATIONALE.md')
| -rw-r--r-- | vendor/github.com/tetratelabs/wazero/RATIONALE.md | 1587 |
1 files changed, 0 insertions, 1587 deletions
diff --git a/vendor/github.com/tetratelabs/wazero/RATIONALE.md b/vendor/github.com/tetratelabs/wazero/RATIONALE.md deleted file mode 100644 index ffb1a299e..000000000 --- a/vendor/github.com/tetratelabs/wazero/RATIONALE.md +++ /dev/null @@ -1,1587 +0,0 @@ -# Notable rationale of wazero - -## Zero dependencies - -Wazero has zero dependencies to differentiate itself from other runtimes which -have heavy impact usually due to CGO. By avoiding CGO, wazero avoids -prerequisites such as shared libraries or libc, and lets users keep features -like cross compilation. - -Avoiding go.mod dependencies reduces interference on Go version support, and -size of a statically compiled binary. However, doing so brings some -responsibility into the project. - -Go's native platform support is good: We don't need platform-specific code to -get monotonic time, nor do we need much work to implement certain features -needed by our compiler such as `mmap`. That said, Go does not support all -common operating systems to the same degree. For example, Go 1.18 includes -`Mprotect` on Linux and Darwin, but not FreeBSD. - -The general tradeoff the project takes from a zero dependency policy is more -explicit support of platforms (in the compiler runtime), as well a larger and -more technically difficult codebase. - -At some point, we may allow extensions to supply their own platform-specific -hooks. Until then, one end user impact/tradeoff is some glitches trying -untested platforms (with the Compiler runtime). - -### Why do we use CGO to implement system calls on darwin? - -wazero is dependency and CGO free by design. In some cases, we have code that -can optionally use CGO, but retain a fallback for when that's disabled. The only -operating system (`GOOS`) we use CGO by default in is `darwin`. - -Unlike other operating systems, regardless of `CGO_ENABLED`, Go always uses -"CGO" mechanisms in the runtime layer of `darwin`. This is explained in -[Statically linked binaries on Mac OS X](https://developer.apple.com/library/archive/qa/qa1118/_index.html#//apple_ref/doc/uid/DTS10001666): - -> Apple does not support statically linked binaries on Mac OS X. A statically -> linked binary assumes binary compatibility at the kernel system call -> interface, and we do not make any guarantees on that front. Rather, we strive -> to ensure binary compatibility in each dynamically linked system library and -> framework. - -This plays to our advantage for system calls that aren't yet exposed in the Go -standard library, notably `futimens` for nanosecond-precision timestamp -manipulation. - -### Why not x/sys - -Going beyond Go's SDK limitations can be accomplished with their [x/sys library](https://pkg.go.dev/golang.org/x/sys/unix). -For example, this includes `zsyscall_freebsd_amd64.go` missing from the Go SDK. - -However, like all dependencies, x/sys is a source of conflict. For example, -x/sys had to be in order to upgrade to Go 1.18. - -If we depended on x/sys, we could get more precise functionality needed for -features such as clocks or more platform support for the compiler runtime. - -That said, formally supporting an operating system may still require testing as -even use of x/sys can require platform-specifics. For example, [mmap-go](https://github.com/edsrzf/mmap-go) -uses x/sys, but also mentions limitations, some not surmountable with x/sys -alone. - -Regardless, we may at some point introduce a separate go.mod for users to use -x/sys as a platform plugin without forcing all users to maintain that -dependency. - -## Project structure - -wazero uses internal packages extensively to balance API compatibility desires for end users with the need to safely -share internals between compilers. - -End-user packages include `wazero`, with `Config` structs, `api`, with shared types, and the built-in `wasi` library. -Everything else is internal. - -We put the main program for wazero into a directory of the same name to match conventions used in `go install`, -notably the name of the folder becomes the binary name. We chose to use `cmd/wazero` as it is common practice -and less surprising than `wazero/wazero`. - -### Internal packages - -Most code in wazero is internal, and it is acknowledged that this prevents external implementation of facets such as -compilers or decoding. It also prevents splitting this code into separate repositories, resulting in a larger monorepo. -This also adds work as more code needs to be centrally reviewed. - -However, the alternative is neither secure nor viable. To allow external implementation would require exporting symbols -public, such as the `CodeSection`, which can easily create bugs. Moreover, there's a high drift risk for any attempt at -external implementations, compounded not just by wazero's code organization, but also the fast moving Wasm and WASI -specifications. - -For example, implementing a compiler correctly requires expertise in Wasm, Golang and assembly. This requires deep -insight into how internals are meant to be structured and the various tiers of testing required for `wazero` to result -in a high quality experience. Even if someone had these skills, supporting external code would introduce variables which -are constants in the central one. Supporting an external codebase is harder on the project team, and could starve time -from the already large burden on the central codebase. - -The tradeoffs of internal packages are a larger codebase and responsibility to implement all standard features. It also -implies thinking about extension more as forking is not viable for reasons above also. The primary mitigation of these -realities are friendly OSS licensing, high rigor and a collaborative spirit which aim to make contribution in the shared -codebase productive. - -### Avoiding cyclic dependencies - -wazero shares constants and interfaces with internal code by a sharing pattern described below: -* shared interfaces and constants go in one package under root: `api`. -* user APIs and structs depend on `api` and go into the root package `wazero`. - * e.g. `InstantiateModule` -> `/wasm.go` depends on the type `api.Module`. -* implementation code can also depend on `api` in a corresponding package under `/internal`. - * Ex package `wasm` -> `/internal/wasm/*.go` and can depend on the type `api.Module`. - -The above guarantees no cyclic dependencies at the cost of having to re-define symbols that exist in both packages. -For example, if `wasm.Store` is a type the user needs access to, it is narrowed by a cover type in the `wazero`: - -```go -type runtime struct { - s *wasm.Store -} -``` - -This is not as bad as it sounds as mutations are only available via configuration. This means exported functions are -limited to only a few functions. - -### Avoiding security bugs - -In order to avoid security flaws such as code insertion, nothing in the public API is permitted to write directly to any -mutable symbol in the internal package. For example, the package `api` is shared with internal code. To ensure -immutability, the `api` package cannot contain any mutable public symbol, such as a slice or a struct with an exported -field. - -In practice, this means shared functionality like memory mutation need to be implemented by interfaces. - -Here are some examples: -* `api.Memory` protects access by exposing functions like `WriteFloat64Le` instead of exporting a buffer (`[]byte`). -* There is no exported symbol for the `[]byte` representing the `CodeSection` - -Besides security, this practice prevents other bugs and allows centralization of validation logic such as decoding Wasm. - -## API Design - -### Why is `context.Context` inconsistent? - -It may seem strange that only certain API have an initial `context.Context` -parameter. We originally had a `context.Context` for anything that might be -traced, but it turned out to be only useful for lifecycle and host functions. - -For instruction-scoped aspects like memory updates, a context parameter is too -fine-grained and also invisible in practice. For example, most users will use -the compiler engine, and its memory, global or table access will never use go's -context. - -### Why does `api.ValueType` map to uint64? - -WebAssembly allows functions to be defined either by the guest or the host, -with signatures expressed as WebAssembly types. For example, `i32` is a 32-bit -type which might be interpreted as signed. Function signatures can have zero or -more parameters or results even if WebAssembly 1.0 allows up to one result. - -The guest can export functions, so that the host can call it. In the case of -wazero, the host is Go and an exported function can be called via -`api.Function`. `api.Function` allows users to supply parameters and read -results as a slice of uint64. For example, if there are no results, an empty -slice is returned. The user can learn the signature via `FunctionDescription`, -which returns the `api.ValueType` corresponding to each parameter or result. -`api.ValueType` defines the mapping of WebAssembly types to `uint64` values for -reason described in this section. The special case of `v128` is also mentioned -below. - -wazero maps each value type to a uint64 values because it holds the largest -type in WebAssembly 1.0 (i64). A slice allows you to express empty (e.g. a -nullary signature), for example a start function. - -Here's an example of calling a function, noting this syntax works for both a -signature `(param i32 i32) (result i32)` and `(param i64 i64) (result i64)` -```go -x, y := uint64(1), uint64(2) -results, err := mod.ExportedFunction("add").Call(ctx, x, y) -if err != nil { - log.Panicln(err) -} -fmt.Printf("%d + %d = %d\n", x, y, results[0]) -``` - -WebAssembly does not define an encoding strategy for host defined parameters or -results. This means the encoding rules above are defined by wazero instead. To -address this, we clarified mapping both in `api.ValueType` and added helper -functions like `api.EncodeF64`. This allows users conversions typical in Go -programming, and utilities to avoid ambiguity and edge cases around casting. - -Alternatively, we could have defined a byte buffer based approach and a binary -encoding of value types in and out. For example, an empty byte slice would mean -no values, while a non-empty could use a binary encoding for supported values. -This could work, but it is more difficult for the normal case of i32 and i64. -It also shares a struggle with the current approach, which is that value types -were added after WebAssembly 1.0 and not all of them have an encoding. More on -this below. - -In summary, wazero chose an approach for signature mapping because there was -none, and the one we chose biases towards simplicity with integers and handles -the rest with documentation and utilities. - -#### Post 1.0 value types - -Value types added after WebAssembly 1.0 stressed the current model, as some -have no encoding or are larger than 64 bits. While problematic, these value -types are not commonly used in exported (extern) functions. However, some -decisions were made and detailed below. - -For example `externref` has no guest representation. wazero chose to map -references to uint64 as that's the largest value needed to encode a pointer on -supported platforms. While there are two reference types, `externref` and -`functype`, the latter is an internal detail of function tables, and the former -is rarely if ever used in function signatures as of the end of 2022. - -The only value larger than 64 bits is used for SIMD (`v128`). Vectorizing via -host functions is not used as of the end of 2022. Even if it were, it would be -inefficient vs guest vectorization due to host function overhead. In other -words, the `v128` value type is unlikely to be in an exported function -signature. That it requires two uint64 values to encode is an internal detail -and not worth changing the exported function interface `api.Function`, as doing -so would break all users. - -### Interfaces, not structs - -All exported types in public packages, regardless of configuration vs runtime, are interfaces. The primary benefits are -internal flexibility and avoiding people accidentally mis-initializing by instantiating the types on their own vs using -the `NewXxx` constructor functions. In other words, there's less support load when things can't be done incorrectly. - -Here's an example: -```go -rt := &RuntimeConfig{} // not initialized properly (fields are nil which shouldn't be) -rt := RuntimeConfig{} // not initialized properly (should be a pointer) -rt := wazero.NewRuntimeConfig() // initialized properly -``` - -There are a few drawbacks to this, notably some work for maintainers. -* Interfaces are decoupled from the structs implementing them, which means the signature has to be repeated twice. -* Interfaces have to be documented and guarded at time of use, that 3rd party implementations aren't supported. -* As of Golang 1.21, interfaces are still [not well supported](https://github.com/golang/go/issues/5860) in godoc. - -## Config - -wazero configures scopes such as Runtime and Module using `XxxConfig` types. For example, `RuntimeConfig` configures -`Runtime` and `ModuleConfig` configure `Module` (instantiation). In all cases, config types begin defaults and can be -customized by a user, e.g., selecting features or a module name override. - -### Why don't we make each configuration setting return an error? -No config types create resources that would need to be closed, nor do they return errors on use. This helps reduce -resource leaks, and makes chaining easier. It makes it possible to parse configuration (ex by parsing yaml) independent -of validating it. - -Instead of: -``` -cfg, err = cfg.WithFS(fs) -if err != nil { - return err -} -cfg, err = cfg.WithName(name) -if err != nil { - return err -} -mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg) -if err != nil { - return err -} -``` - -There's only one call site to handle errors: -``` -cfg = cfg.WithFS(fs).WithName(name) -mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg) -if err != nil { - return err -} -``` - -This allows users one place to look for errors, and also the benefit that if anything internally opens a resource, but -errs, there's nothing they need to close. In other words, users don't need to track which resources need closing on -partial error, as that is handled internally by the only code that can read configuration fields. - -### Why are configuration immutable? -While it seems certain scopes like `Runtime` won't repeat within a process, they do, possibly in different goroutines. -For example, some users create a new runtime for each module, and some re-use the same base module configuration with -only small updates (ex the name) for each instantiation. Making configuration immutable allows them to be safely used in -any goroutine. - -Since config are immutable, changes apply via return val, similar to `append` in a slice. - -For example, both of these are the same sort of error: -```go -append(slice, element) // bug as only the return value has the updated slice. -cfg.WithName(next) // bug as only the return value has the updated name. -``` - -Here's an example of correct use: re-assigning explicitly or via chaining. -```go -cfg = cfg.WithName(name) // explicit - -mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg.WithName(name)) // implicit -if err != nil { - return err -} -``` - -### Why aren't configuration assigned with option types? -The option pattern is a familiar one in Go. For example, someone defines a type `func (x X) err` and uses it to update -the target. For example, you could imagine wazero could choose to make `ModuleConfig` from options vs chaining fields. - -Ex instead of: -```go -type ModuleConfig interface { - WithName(string) ModuleConfig - WithFS(fs.FS) ModuleConfig -} - -struct moduleConfig { - name string - fs fs.FS -} - -func (c *moduleConfig) WithName(name string) ModuleConfig { - ret := *c // copy - ret.name = name - return &ret -} - -func (c *moduleConfig) WithFS(fs fs.FS) ModuleConfig { - ret := *c // copy - ret.setFS("/", fs) - return &ret -} - -config := r.NewModuleConfig().WithFS(fs) -configDerived := config.WithName("name") -``` - -An option function could be defined, then refactor each config method into an name prefixed option function: -```go -type ModuleConfig interface { -} -struct moduleConfig { - name string - fs fs.FS -} - -type ModuleConfigOption func(c *moduleConfig) - -func ModuleConfigName(name string) ModuleConfigOption { - return func(c *moduleConfig) { - c.name = name - } -} - -func ModuleConfigFS(fs fs.FS) ModuleConfigOption { - return func(c *moduleConfig) { - c.fs = fs - } -} - -func (r *runtime) NewModuleConfig(opts ...ModuleConfigOption) ModuleConfig { - ret := newModuleConfig() // defaults - for _, opt := range opts { - opt(&ret.config) - } - return ret -} - -func (c *moduleConfig) WithOptions(opts ...ModuleConfigOption) ModuleConfig { - ret := *c // copy base config - for _, opt := range opts { - opt(&ret.config) - } - return ret -} - -config := r.NewModuleConfig(ModuleConfigFS(fs)) -configDerived := config.WithOptions(ModuleConfigName("name")) -``` - -wazero took the path of the former design primarily due to: -* interfaces provide natural namespaces for their methods, which is more direct than functions with name prefixes. -* parsing config into function callbacks is more direct vs parsing config into a slice of functions to do the same. -* in either case derived config is needed and the options pattern is more awkward to achieve that. - -There are other reasons such as test and debug being simpler without options: the above list is constrained to conserve -space. It is accepted that the options pattern is common in Go, which is the main reason for documenting this decision. - -### Why aren't config types deeply structured? -wazero's configuration types cover the two main scopes of WebAssembly use: -* `RuntimeConfig`: This is the broadest scope, so applies also to compilation - and instantiation. e.g. This controls the WebAssembly Specification Version. -* `ModuleConfig`: This affects modules instantiated after compilation and what - resources are allowed. e.g. This defines how or if STDOUT is captured. This - also allows sub-configuration of `FSConfig`. - -These default to a flat definition each, with lazy sub-configuration only after -proven to be necessary. A flat structure is easier to work with and is also -easy to discover. Unlike the option pattern described earlier, more -configuration in the interface doesn't taint the package namespace, only -`ModuleConfig`. - -We default to a flat structure to encourage simplicity. If we eagerly broke out -all possible configurations into sub-types (e.g. ClockConfig), it would be hard -to notice configuration sprawl. By keeping the config flat, it is easy to see -the cognitive load we may be adding to our users. - -In other words, discomfort adding more configuration is a feature, not a bug. -We should only add new configuration rarely, and before doing so, ensure it -will be used. In fact, this is why we support using context fields for -experimental configuration. By letting users practice, we can find out if a -configuration was a good idea or not before committing to it, and potentially -sprawling our types. - -In reflection, this approach worked well for the nearly 1.5 year period leading -to version 1.0. We've only had to create a single sub-configuration, `FSConfig`, -and it was well understood why when it occurred. - -## Why does `ModuleConfig.WithStartFunctions` default to `_start`? - -We formerly had functions like `StartWASICommand` that would verify -preconditions and start WASI's `_start` command. However, this caused confusion -because both many languages compiled a WASI dependency, and many did so -inconsistently. - -The conflict is that exported functions need to use features the language -runtime provides, such as garbage collection. There's a "chicken-egg problem" -where `_start` needs to complete in order for exported behavior to work. - -For example, unlike `GOOS=wasip1` in Go 1.21, TinyGo's "wasi" target supports -function exports. So, the only way to use FFI style is via the "wasi" target. -Not explicitly calling `_start` before an ABI such as wapc-go, would crash, due -to setup not happening (e.g. to implement `panic`). Other embedders such as -Envoy also called `_start` for the same reason. To avoid a common problem for -users unaware of WASI, and also to simplify normal use of WASI (e.g. `main`), -we added `_start` to `ModuleConfig.WithStartFunctions`. - -In cases of multiple initializers, such as in wapc-go, users can override this -to add the others *after* `_start`. Users who want to explicitly control -`_start`, such as some of our unit tests, can clear the start functions and -remove it. - -This decision was made in 2022, and holds true in 2023, even with the -introduction of "wasix". It holds because "wasix" is backwards compatible with -"wasip1". In the future, there will be other ways to start applications, and -may not be backwards compatible with "wasip1". - -Most notably WASI "Preview 2" is not implemented in a way compatible with -wasip1. Its start function is likely to be different, and defined in the -wasi-cli "world". When the design settles, and it is implemented by compilers, -wazero will attempt to support "wasip2". However, it won't do so in a way that -breaks existing compilers. - -In other words, we won't remove `_start` if "wasip2" continues a path of an -alternate function name. If we did, we'd break existing users despite our -compatibility promise saying we don't. The most likely case is that when we -build-in something incompatible with "wasip1", that start function will be -added to the start functions list in addition to `_start`. - -See http://wasix.org -See https://github.com/WebAssembly/wasi-cli - -## Runtime == Engine+Store -wazero defines a single user-type which combines the specification concept of `Store` with the unspecified `Engine` -which manages them. - -### Why not multi-store? -Multi-store isn't supported as the extra tier complicates lifecycle and locking. Moreover, in practice it is unusual for -there to be an engine that has multiple stores which have multiple modules. More often, it is the case that there is -either 1 engine with 1 store and multiple modules, or 1 engine with many stores, each having 1 non-host module. In worst -case, a user can use multiple runtimes until "multi-store" is better understood. - -If later, we have demand for multiple stores, that can be accomplished by overload. e.g. `Runtime.InstantiateInStore` or -`Runtime.Store(name) Store`. - -## Exit - -### Why do we only return a `sys.ExitError` on a non-zero exit code? - -It is reasonable to think an exit error should be returned, even if the code is -success (zero). Even on success, the module is no longer functional. For -example, function exports would error later. However, wazero does not. The only -time `sys.ExitError` is on error (non-zero). - -This decision was to improve performance and ergonomics for guests that both -use WASI (have a `_start` function), and also allow custom exports. -Specifically, Rust, TinyGo and normal wasi-libc, don't exit the module during -`_start`. If they did, it would invalidate their function exports. This means -it is unlikely most compilers will change this behavior. - -`GOOS=waspi1` from Go 1.21 does exit during `_start`. However, it doesn't -support other exports besides `_start`, and `_start` is not defined to be -called multiple times anyway. - -Since `sys.ExitError` is not always returned, we added `Module.IsClosed` for -defensive checks. This helps integrators avoid calling functions which will -always fail. - -### Why panic with `sys.ExitError` after a host function exits? - -Currently, the only portable way to stop processing code is via panic. For -example, WebAssembly "trap" instructions, such as divide by zero, are -implemented via panic. This ensures code isn't executed after it. - -When code reaches the WASI `proc_exit` instruction, we need to stop processing. -Regardless of the exit code, any code invoked after exit would be in an -inconsistent state. This is likely why unreachable instructions are sometimes -inserted after exit: https://github.com/emscripten-core/emscripten/issues/12322 - -## WASI - -Unfortunately, [WASI Snapshot Preview 1](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) is not formally defined enough, and has APIs with ambiguous semantics. -This section describes how Wazero interprets and implements the semantics of several WASI APIs that may be interpreted differently by different wasm runtimes. -Those APIs may affect the portability of a WASI application. - -### Why don't we attempt to pass wasi-testsuite on user-defined `fs.FS`? - -While most cases work fine on an `os.File` based implementation, we won't -promise wasi-testsuite compatibility on user defined wrappers of `os.DirFS`. -The only option for real systems is to use our `sysfs.FS`. - -There are a lot of areas where windows behaves differently, despite the -`os.File` abstraction. This goes well beyond file locking concerns (e.g. -`EBUSY` errors on open files). For example, errors like `ACCESS_DENIED` aren't -properly mapped to `EPERM`. There are trickier parts too. `FileInfo.Sys()` -doesn't return enough information to build inodes needed for WASI. To rebuild -them requires the full path to the underlying file, not just its directory -name, and there's no way for us to get that information. At one point we tried, -but in practice things became tangled and functionality such as read-only -wrappers became untenable. Finally, there are version-specific behaviors which -are difficult to maintain even in our own code. For example, go 1.20 opens -files in a different way than versions before it. - -### Why aren't WASI rules enforced? - -The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has a -number of rules for a "command module", but only the memory export rule is enforced. If a "_start" function exists, it -is enforced to be the correct signature and succeed, but the export itself isn't enforced. It follows that this means -exports are not required to be contained to a "_start" function invocation. Finally, the "__indirect_function_table" -export is also not enforced. - -The reason for the exceptions are that implementations aren't following the rules. For example, TinyGo doesn't export -"__indirect_function_table", so crashing on this would make wazero unable to run TinyGo modules. Similarly, modules -loaded by wapc-go don't always define a "_start" function. Since "snapshot-01" is not a proper version, and certainly -not a W3C recommendation, there's no sense in breaking users over matters like this. - -### Why is I/O configuration not coupled to WASI? - -WebAssembly System Interfaces (WASI) is a formalization of a practice that can be done anyway: Define a host function to -access a system interface, such as writing to STDOUT. WASI stalled at snapshot-01 and as of early 2023, is being -rewritten entirely. - -This instability implies a need to transition between WASI specs, which places wazero in a position that requires -decoupling. For example, if code uses two different functions to call `fd_write`, the underlying configuration must be -centralized and decoupled. Otherwise, calls using the same file descriptor number will end up writing to different -places. - -In short, wazero defined system configuration in `ModuleConfig`, not a WASI type. This allows end-users to switch from -one spec to another with minimal impact. This has other helpful benefits, as centralized resources are simpler to close -coherently (ex via `Module.Close`). - -In reflection, this worked well as more ABI became usable in wazero. - -### Background on `ModuleConfig` design - -WebAssembly 1.0 (20191205) specifies some aspects to control isolation between modules ([sandboxing](https://en.wikipedia.org/wiki/Sandbox_(computer_security))). -For example, `wasm.Memory` has size constraints and each instance of it is isolated from each other. While `wasm.Memory` -can be shared, by exporting it, it is not exported by default. In fact a WebAssembly Module (Wasm) has no memory by -default. - -While memory is defined in WebAssembly 1.0 (20191205), many aspects are not. Let's use an example of `exec.Cmd` as for -example, a WebAssembly System Interfaces (WASI) command is implemented as a module with a `_start` function, and in many -ways acts similar to a process with a `main` function. - -To capture "hello world" written to the console (stdout a.k.a. file descriptor 1) in `exec.Cmd`, you would set the -`Stdout` field accordingly, perhaps to a buffer. In WebAssembly 1.0 (20191205), the only way to perform something like -this is via a host function (ex `HostModuleFunctionBuilder`) and internally copy memory corresponding to that string -to a buffer. - -WASI implements system interfaces with host functions. Concretely, to write to console, a WASI command `Module` imports -"fd_write" from "wasi_snapshot_preview1" and calls it with the `fd` parameter set to 1 (STDOUT). - -The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has no -means to declare configuration, although its function definitions imply configuration for example if fd 1 should exist, -and if so where should it write. Moreover, snapshot-01 was last updated in late 2020 and the specification is being -completely rewritten as of early 2022. This means WASI as defined by "snapshot-01" will not clarify aspects like which -file descriptors are required. While it is possible a subsequent version may, it is too early to tell as no version of -WASI has reached a stage near W3C recommendation. Even if it did, module authors are not required to only use WASI to -write to console, as they can define their own host functions, such as they did before WASI existed. - -wazero aims to serve Go developers as a primary function, and help them transition between WASI specifications. In -order to do this, we have to allow top-level configuration. To ensure isolation by default, `ModuleConfig` has WithXXX -that override defaults to no-op or empty. One `ModuleConfig` instance is used regardless of how many times the same WASI -functions are imported. The nil defaults allow safe concurrency in these situations, as well lower the cost when they -are never used. Finally, a one-to-one mapping with `Module` allows the module to close the `ModuleConfig` instead of -confusing users with another API to close. - -Naming, defaults and validation rules of aspects like `STDIN` and `Environ` are intentionally similar to other Go -libraries such as `exec.Cmd` or `syscall.SetEnv`, and differences called out where helpful. For example, there's no goal -to emulate any operating system primitive specific to Windows (such as a 'c:\' drive). Moreover, certain defaults -working with real system calls are neither relevant nor safe to inherit: For example, `exec.Cmd` defaults to read STDIN -from a real file descriptor ("/dev/null"). Defaulting to this, vs reading `io.EOF`, would be unsafe as it can exhaust -file descriptors if resources aren't managed properly. In other words, blind copying of defaults isn't wise as it can -violate isolation or endanger the embedding process. In summary, we try to be similar to normal Go code, but often need -act differently and document `ModuleConfig` is more about emulating, not necessarily performing real system calls. - -## File systems - -### Motivation on `sys.FS` - -The `sys.FS` abstraction in wazero was created because of limitations in -`fs.FS`, and `fs.File` in Go. Compilers targeting `wasip1` may access -functionality that writes new files. The ability to overcome this was requested -even before wazero was named this, via issue #21 in March 2021. - -A month later, golang/go#45757 was raised by someone else on the same topic. As -of July 2023, this has not resolved to a writeable file system abstraction. - -Over the next year more use cases accumulated, consolidated in March 2022 into -#390. This closed in January 2023 with a milestone of providing more -functionality, limited to users giving a real directory. This didn't yet expose -a file abstraction for general purpose use. Internally, this used `os.File`. -However, a wasm module instance is a virtual machine. Only supporting `os.File` -breaks sand-boxing use cases. Moreover, `os.File` is not an interface. Even -though this abstracts functionality, it does allow interception use cases. - -Hence, a few days later in January 2023, we had more issues asking to expose an -abstraction, #1013 and later #1532, on use cases like masking access to files. -In other words, the use case requests never stopped, and aren't solved by -exposing only real files. - -In summary, the primary motivation for exposing a replacement for `fs.FS` and -`fs.File` was around repetitive use case requests for years, around -interception and the ability to create new files, both virtual and real files. -While some use cases are solved with real files, not all are. Regardless, an -interface approach is necessary to ensure users can intercept I/O operations. - -### Why doesn't `sys.File` have a `Fd()` method? - -There are many features we could expose. We could make File expose underlying -file descriptors in case they are supported, for integration of system calls -that accept multiple ones, namely `poll` for multiplexing. This special case is -described in a subsequent section. - -As noted above, users have been asking for a file abstraction for over two -years, and a common answer was to wait. Making users wait is a problem, -especially so long. Good reasons to make people wait are stabilization. Edge -case features are not a great reason to hold abstractions from users. - -Another reason is implementation difficulty. Go did not attempt to abstract -file descriptors. For example, unlike `fs.ReadFile` there is no `fs.FdFile` -interface. Most likely, this is because file descriptors are an implementation -detail of common features. Programming languages, including Go, do not require -end users to know about file descriptors. Types such as `fs.File` can be used -without any knowledge of them. Implementations may or may not have file -descriptors. For example, in Go, `os.DirFS` has underlying file descriptors -while `embed.FS` does not. - -Despite this, some may want to expose a non-standard interface because -`os.File` has `Fd() uintptr` to return a file descriptor. Mainly, this is -handy to integrate with `syscall` package functions (on `GOOS` values that -declare them). Notice, though that `uintptr` is unsafe and not an abstraction. -Close inspection will find some `os.File` types internally use `poll.FD` -instead, yet this is not possible to use abstractly because that type is not -exposed. For example, `plan9` uses a different type than `poll.FD`. In other -words, even in real files, `Fd()` is not wholly portable, despite it being -useful on many operating systems with the `syscall` package. - -The reasons above, why Go doesn't abstract `FdFile` interface are a subset of -reasons why `sys.File` does not. If we exposed `File.Fd()` we not only would -have to declare all the edge cases that Go describes including impact of -finalizers, we would have to describe these in terms of virtualized files. -Then, we would have to reason with this value vs our existing virtualized -`sys.FileTable`, mapping whatever type we return to keys in that table, also -in consideration of garbage collection impact. The combination of issues like -this could lead down a path of not implementing a file system abstraction at -all, and instead a weak key mapped abstraction of the `syscall` package. Once -we finished with all the edge cases, we would have lost context of the original -reason why we started.. simply to allow file write access! - -When wazero attempts to do more than what the Go programming language team, it -has to be carefully evaluated, to: -* Be possible to implement at least for `os.File` backed files -* Not be confusing or cognitively hard for virtual file systems and normal use. -* Affordable: custom code is solely the responsible by the core team, a much - smaller group of individuals than who maintain the Go programming language. - -Due to problems well known in Go, consideration of the end users who constantly -ask for basic file system functionality, and the difficulty virtualizing file -descriptors at multiple levels, we don't expose `Fd()` and likely won't ever -expose `Fd()` on `sys.File`. - -### Why does `sys.File` have a `Poll()` method, while `sys.FS` does not? - -wazero exposes `File.Poll` which allows one-at-a-time poll use cases, -requested by multiple users. This not only includes abstract tests such as -Go 1.21 `GOOS=wasip1`, but real use cases including python and container2wasm -repls, as well listen sockets. The main use cases is non-blocking poll on a -single file. Being a single file, this has no risk of problems such as -head-of-line blocking, even when emulated. - -The main use case of multi-poll are bidirectional network services, something -not used in `GOOS=wasip1` standard libraries, but could be in the future. -Moving forward without a multi-poller allows wazero to expose its file system -abstraction instead of continuing to hold back it back for edge cases. We'll -continue discussion below regardless, as rationale was requested. - -You can loop through multiple `sys.File`, using `File.Poll` to see if an event -is ready, but there is a head-of-line blocking problem. If a long timeout is -used, bad luck could have a file that has nothing to read or write before one -that does. This could cause more blocking than necessary, even if you could -poll the others just after with a zero timeout. What's worse than this is if -unlimited blocking was used (`timeout=-1`). The host implementations could use -goroutines to avoid this, but interrupting a "forever" poll is problematic. All -of these are reasons to consider a multi-poll API, but do not require exporting -`File.Fd()`. - -Should multi-poll becomes critical, `sys.FS` could expose a `Poll` function -like below, despite it being the non-portable, complicated if possible to -implement on all platforms and virtual file systems. -```go -ready, errno := fs.Poll([]sys.PollFile{{f1, sys.POLLIN}, {f2, sys.POLLOUT}}, timeoutMillis) -``` - -A real filesystem could handle this by using an approach like the internal -`unix.Poll` function in Go, passing file descriptors on unix platforms, or -returning `sys.ENOSYS` for unsupported operating systems. Implementation for -virtual files could have a strategy around timeout to avoid the worst case of -head-of-line blocking (unlimited timeout). - -Let's remember that when designing abstractions, it is not best to add an -interface for everything. Certainly, Go doesn't, as evidenced by them not -exposing `poll.FD` in `os.File`! Such a multi-poll could be limited to -built-in filesystems in the wazero repository, avoiding complexity of trying to -support and test this abstractly. This would still permit multiplexing for CLI -users, and also permit single file polling as exists now. - -### Why doesn't wazero implement the working directory? - -An early design of wazero's API included a `WithWorkDirFS` which allowed -control over which file a relative path such as "./config.yml" resolved to, -independent of the root file system. This intended to help separate concerns -like mutability of files, but it didn't work and was removed. - -Compilers that target wasm act differently with regard to the working -directory. For example, wasi-libc, used by TinyGo, -tracks working directory changes in compiled wasm instead: initially "/" until -code calls `chdir`. Zig assumes the first pre-opened file descriptor is the -working directory. - -The only place wazero can standardize a layered concern is via a host function. -Since WASI doesn't use host functions to track the working directory, we can't -standardize the storage and initial value of it. - -Meanwhile, code may be able to affect the working directory by compiling -`chdir` into their main function, using an argument or ENV for the initial -value (possibly `PWD`). Those unable to control the compiled code should only -use absolute paths in configuration. - -See -* https://github.com/golang/go/blob/go1.20/src/syscall/fs_js.go#L324 -* https://github.com/WebAssembly/wasi-libc/pull/214#issue-673090117 -* https://github.com/ziglang/zig/blob/53a9ee699a35a3d245ab6d1dac1f0687a4dcb42c/src/main.zig#L32 - -### Why ignore the error returned by io.Reader when n > 1? - -Per https://pkg.go.dev/io#Reader, if we receive an error, any bytes read should -be processed first. At the syscall abstraction (`fd_read`), the caller is the -processor, so we can't process the bytes inline and also return the error (as -`EIO`). - -Let's assume we want to return the bytes read on error to the caller. This -implies we at least temporarily ignore the error alongside them. The choice -remaining is whether to persist the error returned with the read until a -possible next call, or ignore the error. - -If we persist an error returned, it would be coupled to a file descriptor, but -effectively it is boolean as this case coerces to `EIO`. If we track a "last -error" on a file descriptor, it could be complicated for a couple reasons -including whether the error is transient or permanent, or if the error would -apply to any FD operation, or just read. Finally, there may never be a -subsequent read as perhaps the bytes leading up to the error are enough to -satisfy the processor. - -This decision boils down to whether or not to track an error bit per file -descriptor or not. If not, the assumption is that a subsequent operation would -also error, this time without reading any bytes. - -The current opinion is to go with the simplest path, which is to return the -bytes read and ignore the error the there were any. Assume a subsequent -operation will err if it needs to. This helps reduce the complexity of the code -in wazero and also accommodates the scenario where the bytes read are enough to -satisfy its processor. - -### File descriptor allocation strategy - -File descriptor allocation currently uses a strategy similar the one implemented -by unix systems: when opening a file, the lowest unused number is picked. - -The WASI standard documents that programs cannot expect that file descriptor -numbers will be allocated with a lowest-first strategy, and they should instead -assume the values will be random. Since _random_ is a very imprecise concept in -computers, we technically satisfying the implementation with the descriptor -allocation strategy we use in Wazero. We could imagine adding more _randomness_ -to the descriptor selection process, however this should never be used as a -security measure to prevent applications from guessing the next file number so -there are no strong incentives to complicate the logic. - -### Why does `FSConfig.WithDirMount` not match behaviour with `os.DirFS`? - -It may seem that we should require any feature that seems like a standard -library in Go, to behave the same way as the standard library. Doing so would -present least surprise to Go developers. In the case of how we handle -filesystems, we break from that as it is incompatible with the expectations of -WASI, the most commonly implemented filesystem ABI. - -The main reason is that `os.DirFS` is a virtual filesystem abstraction while -WASI is an abstraction over syscalls. For example, the signature of `fs.Open` -does not permit use of flags. This creates conflict on what default behaviors -to take when Go implemented `os.DirFS`. On the other hand, `path_open` can pass -flags, and in fact tests require them to be honored in specific ways. - -This conflict requires us to choose what to be more compatible with, and which -type of user to surprise the least. We assume there will be more developers -compiling code to wasm than developers of custom filesystem plugins, and those -compiling code to wasm will be better served if we are compatible with WASI. -Hence on conflict, we prefer WASI behavior vs the behavior of `os.DirFS`. - -See https://github.com/WebAssembly/wasi-testsuite -See https://github.com/golang/go/issues/58141 - -## Why is our `Readdir` function more like Go's `os.File` than POSIX `readdir`? - -At one point we attempted to move from a bulk `Readdir` function to something -more like the POSIX `DIR` struct, exposing functions like `telldir`, `seekdir` -and `readdir`. However, we chose the design more like `os.File.Readdir`, -because it performs and fits wasip1 better. - -### wasip1/wasix - -`fd_readdir` in wasip1 (and so also wasix) is like `getdents` in Linux, not -`readdir` in POSIX. `getdents` is more like Go's `os.File.Readdir`. - -We currently have an internal type `sys.DirentCache` which only is used by -wasip1 or wasix. When `HostModuleBuilder` adds support for instantiation state, -we could move this to the `wasi_snapshot_preview1` package. Meanwhile, all -filesystem code is internal anyway, so this special-case is acceptable. - -### wasip2 - -`directory-entry-stream` in wasi-filesystem preview2 is defined in component -model, not an ABI, but in wasmtime it is a consuming iterator. A consuming -iterator is easy to support with anything (like `Readdir(1)`), even if it is -inefficient as you can neither bulk read nor skip. The implementation of the -preview1 adapter (uses preview2) confirms this. They use a dirent cache similar -in some ways to our `sysfs.DirentCache`. As there is no seek concept in -preview2, they interpret the cookie as numeric and read on repeat entries when -a cache wasn't available. Note: we currently do not skip-read like this as it -risks buffering large directories, and no user has requested entries before the -cache, yet. - -Regardless, wasip2 is not complete until the end of 2023. We can defer design -discussion until after it is stable and after the reference impl wasmtime -implements it. - -See - * https://github.com/WebAssembly/wasi-filesystem/blob/ef9fc87c07323a6827632edeb6a7388b31266c8e/example-world.md#directory_entry_stream - * https://github.com/bytecodealliance/wasmtime/blob/b741f7c79d72492d17ab8a29c8ffe4687715938e/crates/wasi/src/preview2/preview2/filesystem.rs#L286-L296 - * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L2131-L2137 - * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L936 - -### wasip3 - -`directory-entry-stream` is documented to change significantly in wasip3 moving -from synchronous to synchronous streams. This is dramatically different than -POSIX `readdir` which is synchronous. - -Regardless, wasip3 is not complete until after wasip2, which means 2024 or -later. We can defer design discussion until after it is stable and after the -reference impl wasmtime implements it. - -See - * https://github.com/WebAssembly/WASI/blob/ddfe3d1dda5d1473f37ecebc552ae20ce5fd319a/docs/WitInWasi.md#Streams - * https://docs.google.com/presentation/d/1MNVOZ8hdofO3tI0szg_i-Yoy0N2QPU2C--LzVuoGSlE/edit#slide=id.g1270ef7d5b6_0_662 - -### How do we implement `Pread` with an `fs.File`? - -`ReadAt` is the Go equivalent to `pread`: it does not affect, and is not -affected by, the underlying file offset. Unfortunately, `io.ReaderAt` is not -implemented by all `fs.File`. For example, as of Go 1.19, `embed.openFile` does -not. - -The initial implementation of `fd_pread` instead used `Seek`. To avoid a -regression, we fall back to `io.Seeker` when `io.ReaderAt` is not supported. - -This requires obtaining the initial file offset, seeking to the intended read -offset, and resetting the file offset the initial state. If this final seek -fails, the file offset is left in an undefined state. This is not thread-safe. - -While seeking per read seems expensive, the common case of `embed.openFile` is -only accessing a single int64 field, which is cheap. - -### Pre-opened files - -WASI includes `fd_prestat_get` and `fd_prestat_dir_name` functions used to -learn any directory paths for file descriptors open at initialization time. - -For example, `__wasilibc_register_preopened_fd` scans any file descriptors past -STDERR (1) and invokes `fd_prestat_dir_name` to learn any path prefixes they -correspond to. Zig's `preopensAlloc` does similar. These pre-open functions are -not used again after initialization. - -wazero supports stdio pre-opens followed by any mounts e.g `.:/`. The guest -path is a directory and its name, e.g. "/" is returned by `fd_prestat_dir_name` -for file descriptor 3 (STDERR+1). The first longest match wins on multiple -pre-opens, which allows a path like "/tmp" to match regardless of order vs "/". - -See - * https://github.com/WebAssembly/wasi-libc/blob/a02298043ff551ce1157bc2ee7ab74c3bffe7144/libc-bottom-half/sources/preopens.c - * https://github.com/ziglang/zig/blob/9cb06f3b8bf9ea6b5e5307711bc97328762d6a1d/lib/std/fs/wasi.zig#L50-L53 - -### fd_prestat_dir_name - -`fd_prestat_dir_name` is a WASI function to return the path of the pre-opened -directory of a file descriptor. It has the following three parameters, and the -third `path_len` has ambiguous semantics. - -* `fd`: a file descriptor -* `path`: the offset for the result path -* `path_len`: In wazero, `FdPrestatDirName` writes the result path string to - `path` offset for the exact length of `path_len`. - -Wasmer considers `path_len` to be the maximum length instead of the exact -length that should be written. -See https://github.com/wasmerio/wasmer/blob/3463c51268ed551933392a4063bd4f8e7498b0f6/lib/wasi/src/syscalls/mod.rs#L764 - -The semantics in wazero follows that of wasmtime. -See https://github.com/bytecodealliance/wasmtime/blob/2ca01ae9478f199337cf743a6ab543e8c3f3b238/crates/wasi-common/src/snapshots/preview_1.rs#L578-L582 - -Their semantics match when `path_len` == the length of `path`, so in practice -this difference won't matter match. - -## fd_readdir - -### Why does "wasi_snapshot_preview1" require dot entries when POSIX does not? - -In October 2019, WASI project knew requiring dot entries ("." and "..") was not -documented in preview1, not required by POSIX and problematic to synthesize. -For example, Windows runtimes backed by `FindNextFileW` could not return these. -A year later, the tag representing WASI preview 1 (`snapshot-01`) was made. -This did not include the requested change of making dot entries optional. - -The `phases/snapshot/docs.md` document was altered in subsequent years in -significant ways, often in lock-step with wasmtime or wasi-libc. In January -2022, `sock_accept` was added to `phases/snapshot/docs.md`, a document later -renamed to later renamed to `legacy/preview1/docs.md`. - -As a result, the ABI and behavior remained unstable: The `snapshot-01` tag was -not an effective basis of portability. A test suite was requested well before -this tag, in April 2019. Meanwhile, compliance had no meaning. Developers had -to track changes to the latest doc, while clarifying with wasi-libc or wasmtime -behavior. This lack of stability could have permitted a fix to the dot entries -problem, just as it permitted changes desired by other users. - -In November 2022, the wasi-testsuite project began and started solidifying -expectations. This quickly led to changes in runtimes and the spec doc. WASI -began importing tests from wasmtime as required behaviors for all runtimes. -Some changes implied changes to wasi-libc. For example, `readdir` began to -imply inode fan-outs, which caused performance regressions. Most notably a -test merged in January required dot entries. Tests were merged without running -against any runtime, and even when run ad-hoc only against Linux. Hence, -portability issues mentioned over three years earlier did not trigger any -failure until wazero (which tests Windows) noticed. - -In the same month, wazero requested to revert this change primarily because -Go does not return them from `os.ReadDir`, and materializing them is -complicated due to tests also requiring inodes. Moreover, they are discarded by -not just Go, but other common programming languages. This was rejected by the -WASI lead for preview1, but considered for the completely different ABI named -preview2. - -In February 2023, the WASI chair declared that new rule requiring preview1 to -return dot entries "was decided by the subgroup as a whole", citing meeting -notes. According to these notes, the WASI lead stated incorrectly that POSIX -conformance required returning dot entries, something it explicitly says are -optional. In other words, he said filtering them out would make Preview1 -non-conforming, and asked if anyone objects to this. The co-chair was noted to -say "Because there are existing P1 programs, we shouldn’t make changes like -this." No other were recorded to say anything. - -In summary, preview1 was changed retrospectively to require dot entries and -preview2 was changed to require their absence. This rule was reverse engineered -from wasmtime tests, and affirmed on two false premises: - -* POSIX compliance requires dot entries - * POSIX literally says these are optional -* WASI cannot make changes because there are existing P1 programs. - * Changes to Preview 1 happened before and after this topic. - -As of June 2023, wasi-testsuite still only runs on Linux, so compliance of this -rule on Windows is left to runtimes to decide to validate. The preview2 adapter -uses fake cookies zero and one to refer to dot dirents, uses a real inode for -the dot(".") entry and zero inode for dot-dot(".."). - -See https://github.com/WebAssembly/wasi-filesystem/issues/3 -See https://github.com/WebAssembly/WASI/tree/snapshot-01 -See https://github.com/WebAssembly/WASI/issues/9 -See https://github.com/WebAssembly/WASI/pull/458 -See https://github.com/WebAssembly/wasi-testsuite/pull/32 -See https://github.com/WebAssembly/wasi-libc/pull/345 -See https://github.com/WebAssembly/wasi-testsuite/issues/52 -See https://github.com/WebAssembly/WASI/pull/516 -See https://github.com/WebAssembly/meetings/blob/main/wasi/2023/WASI-02-09.md#should-preview1-fd_readdir-filter-out--and- -See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041 - -### Why are dot (".") and dot-dot ("..") entries problematic? - -When reading a directory, dot (".") and dot-dot ("..") entries are problematic. -For example, Go does not return them from `os.ReadDir`, and materializing them -is complicated (at least dot-dot is). - -A directory entry has stat information in it. The stat information includes -inode which is used for comparing file equivalence. In the simple case of dot, -we could materialize a special entry to expose the same info as stat on the fd -would return. However, doing this and not doing dot-dot would cause confusion, -and dot-dot is far more tricky. To back-fill inode information about a parent -directory would be costly and subtle. For example, the pre-open (mount) of the -directory may be different than its logical parent. This is easy to understand -when considering the common case of mounting "/" and "/tmp" as pre-opens. To -implement ".." from "/tmp" requires information from a separate pre-open, this -includes state to even know the difference. There are easier edge cases as -well, such as the decision to not return ".." from a root path. In any case, -this should start to explain that faking entries when underlying stdlib doesn't -return them is tricky and requires quite a lot of state. - -Another issue is around the `Dirent.Off` value of a directory entry, sometimes -called a "cookie" in Linux man pagers. When the host operating system or -library function does not return dot entries, to support functions such as -`seekdir`, you still need a value for `Dirent.Off`. Naively, you can synthesize -these by choosing sequential offsets zero and one. However, POSIX strictly says -offsets should be treated opaquely. The backing filesystem could use these to -represent real entries. For example, a directory with one entry could use zero -as the `Dirent.Off` value. If you also used zero for the "." dirent, there -would be a clash. This means if you synthesize `Dirent.Off` for any entry, you -need to synthesize this value for all entries. In practice, the simplest way is -using an incrementing number, such as done in the WASI preview2 adapter. - -Working around these issues causes expense to all users of wazero, so we'd -then look to see if that would be justified or not. However, the most common -compilers involved in end user questions, as of early 2023 are TinyGo, Rust and -Zig. All of these compile code which ignores dot and dot-dot entries. In other -words, faking these entries would not only cost our codebase with complexity, -but it would also add unnecessary overhead as the values aren't commonly used. - -The final reason why we might do this, is an end users or a specification -requiring us to. As of early 2023, no end user has raised concern over Go and -by extension wazero not returning dot and dot-dot. The snapshot-01 spec of WASI -does not mention anything on this point. Also, POSIX has the following to say, -which summarizes to "these are optional" - -> The readdir() function shall not return directory entries containing empty names. If entries for dot or dot-dot exist, one entry shall be returned for dot and one entry shall be returned for dot-dot; otherwise, they shall not be returned. - -Unfortunately, as described above, the WASI project decided in early 2023 to -require dot entries in both the spec and the wasi-testsuite. For only this -reason, wazero adds overhead to synthesize dot entries despite it being -unnecessary for most users. - -See https://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html -See https://github.com/golang/go/blob/go1.20/src/os/dir_unix.go#L108-L110 -See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041 - -### Why don't we pre-populate an inode for the dot-dot ("..") entry? - -We only populate an inode for dot (".") because wasi-testsuite requires it, and -we likely already have it (because we cache it). We could attempt to populate -one for dot-dot (".."), but chose not to. - -Firstly, wasi-testsuite does not require the inode of dot-dot, possibly because -the wasip2 adapter doesn't populate it (but we don't really know why). - -The only other reason to populate it would be to avoid wasi-libc's stat fanout -when it is missing. However, wasi-libc explicitly doesn't fan-out to lstat on -the ".." entry on a zero ino. - -Fetching dot-dot's inode despite the above not only doesn't help wasi-libc, but -it also hurts languages that don't use it, such as Go. These languages would -pay a stat syscall penalty even if they don't need the inode. In fact, Go -discards both dot entries! - -In summary, there are no significant upsides in attempting to pre-fetch -dot-dot's inode, and there are downsides to doing it anyway. - -See - * https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94 - * https://github.com/WebAssembly/wasi-testsuite/blob/main/tests/rust/src/bin/fd_readdir.rs#L108 - * https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1037 - -### Why don't we require inodes to be non-zero? - -We don't require a non-zero value for `Dirent.Ino` because doing so can prevent -a real one from resolving later via `Stat_t.Ino`. - -We define `Ino` like `d_ino` in POSIX which doesn't special-case zero. It can -be zero for a few reasons: - -* The file is not a regular file or directory. -* The underlying filesystem does not support inodes. e.g. embed:fs -* A directory doesn't include inodes, but a later stat can. e.g. Windows -* The backend is based on wasi-filesystem (a.k.a wasip2), which has - `directory_entry.inode` optional, and might remove it entirely. - -There are other downsides to returning a zero inode in widely used compilers: - -* File equivalence utilities, like `os.SameFile` will not work. -* wasi-libc's `wasip1` mode will call `lstat` and attempt to retrieve a - non-zero value (unless the entry is named ".."). - -A new compiler may accidentally skip a `Dirent` with a zero `Ino` if emulating -a non-POSIX function and re-using `Dirent.Ino` for `d_fileno`. - -* Linux `getdents` doesn't define `d_fileno` must be non-zero -* BSD `getdirentries` is implementation specific. For example, OpenBSD will - return dirents with a zero `d_fileno`, but Darwin will skip them. - -The above shouldn't be a problem, even in the case of BSD, because `wasip1` is -defined more in terms of `getdents` than `getdirentries`. The bottom half of -either should treat `wasip1` (or any similar ABI such as wasix or wasip2) as a -different operating system and either use different logic that doesn't skip, or -synthesize a fake non-zero `d_fileno` when `d_ino` is zero. - -However, this has been a problem. Go's `syscall.ParseDirent` utility is shared -for all `GOOS=unix`. For simplicity, this abstracts `direntIno` with data from -`d_fileno` or `d_ino`, and drops if either are zero, even if `d_fileno` is the -only field with zero explicitly defined. This led to a change to special case -`GOOS=wasip1` as otherwise virtual files would be unconditionally skipped. - -In practice, this problem is rather unique due to so many compilers relying on -wasi-libc, which tolerates a zero inode. For example, while issues were -reported about the performance regression when wasi-libc began doing a fan-out -on zero `Dirent.Ino`, no issues were reported about dirents being dropped as a -result. - -In summary, rather than complicating implementation and forcing non-zero inodes -for a rare case, we permit zero. We instead document this topic thoroughly, so -that emerging compilers can re-use the research and reference it on conflict. -We also document that `Ino` should be non-zero, so that users implementing that -field will attempt to get it. - -See - * https://github.com/WebAssembly/wasi-filesystem/pull/81 - * https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94 - * https://linux.die.net/man/3/getdents - * https://www.unix.com/man-page/osx/2/getdirentries/ - * https://man.openbsd.org/OpenBSD-5.4/getdirentries.2 - * https://github.com/golang/go/blob/go1.20/src/syscall/dirent.go#L60-L102 - * https://go-review.googlesource.com/c/go/+/507915 - -## sys.Walltime and Nanotime - -The `sys` package has two function types, `Walltime` and `Nanotime` for real -and monotonic clock exports. The naming matches conventions used in Go. - -```go -func time_now() (sec int64, nsec int32, mono int64) { - sec, nsec = walltime() - return sec, nsec, nanotime() -} -``` - -Splitting functions for wall and clock time allow implementations to choose -whether to implement the clock once (as in Go), or split them out. - -Each can be configured with a `ClockResolution`, although is it usually -incorrect as detailed in a sub-heading below. The only reason for exposing this -is to satisfy WASI: - -See https://github.com/WebAssembly/wasi-clocks - -### Why default to fake time? - -WebAssembly has an implicit design pattern of capabilities based security. By -defaulting to a fake time, we reduce the chance of timing attacks, at the cost -of requiring configuration to opt-into real clocks. - -See https://gruss.cc/files/fantastictimers.pdf for an example attacks. - -### Why does fake time increase on reading? - -Both the fake nanotime and walltime increase by 1ms on reading. Particularly in -the case of nanotime, this prevents spinning. - -### Why not `time.Clock`? - -wazero can't use `time.Clock` as a plugin for clock implementation as it is -only substitutable with build flags (`faketime`) and conflates wall and -monotonic time in the same call. - -Go's `time.Clock` was added monotonic time after the fact. For portability with -prior APIs, a decision was made to combine readings into the same API call. - -See https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md - -WebAssembly time imports do not have the same concern. In fact even Go's -imports for clocks split walltime from nanotime readings. - -See https://github.com/golang/go/blob/go1.20/misc/wasm/wasm_exec.js#L243-L255 - -Finally, Go's clock is not an interface. WebAssembly users who want determinism -or security need to be able to substitute an alternative clock implementation -from the host process one. - -### `ClockResolution` - -A clock's resolution is hardware and OS dependent so requires a system call to retrieve an accurate value. -Go does not provide a function for getting resolution, so without CGO we don't have an easy way to get an actual -value. For now, we return fixed values of 1us for realtime and 1ns for monotonic, assuming that realtime clocks are -often lower precision than monotonic clocks. In the future, this could be improved by having OS+arch specific assembly -to make syscalls. - -For example, Go implements time.Now for linux-amd64 with this [assembly](https://github.com/golang/go/blob/go1.20/src/runtime/time_linux_amd64.s). -Because retrieving resolution is not generally called often, unlike getting time, it could be appropriate to only -implement the fallback logic that does not use VDSO (executing syscalls in user mode). The syscall for clock_getres -is 229 and should be usable. https://pkg.go.dev/syscall#pkg-constants. - -If implementing similar for Windows, [mingw](https://github.com/mirror/mingw-w64/blob/6a0e9165008f731bccadfc41a59719cf7c8efc02/mingw-w64-libraries/winpthreads/src/clock.c#L77 -) is often a good source to find the Windows API calls that correspond -to a POSIX method. - -Writing assembly would allow making syscalls without CGO, but comes with the cost that it will require implementations -across many combinations of OS and architecture. - -## sys.Nanosleep - -All major programming languages have a `sleep` mechanism to block for a -duration. Sleep is typically implemented by a WASI `poll_oneoff` relative clock -subscription. - -For example, the below ends up calling `wasi_snapshot_preview1.poll_oneoff`: - -```zig -const std = @import("std"); -pub fn main() !void { - std.time.sleep(std.time.ns_per_s * 5); -} -``` - -Besides Zig, this is also the case with TinyGo (`-target=wasi`) and Rust -(`--target wasm32-wasi`). - -We decided to expose `sys.Nanosleep` to allow overriding the implementation -used in the common case, even if it isn't used by Go, because this gives an -easy and efficient closure over a common program function. We also documented -`sys.Nanotime` to warn users that some compilers don't optimize sleep. - -## sys.Osyield - -We expose `sys.Osyield`, to allow users to control the behavior of WASI's -`sched_yield` without a new build of wazero. This is mainly for parity with -all other related features which we allow users to implement, including -`sys.Nanosleep`. Unlike others, we don't provide an out-of-box implementation -primarily because it will cause performance problems when accessed. - -For example, the below implementation uses CGO, which might result in a 1us -delay per invocation depending on the platform. - -See https://github.com/golang/go/issues/19409#issuecomment-284788196 -```go -//go:noescape -//go:linkname osyield runtime.osyield -func osyield() -``` - -In practice, a request to customize this is unlikely to happen until other -thread based functions are implemented. That said, as of early 2023, there are -a few signs of implementation interest and cross-referencing: - -See https://github.com/WebAssembly/stack-switching/discussions/38 -See https://github.com/WebAssembly/wasi-threads#what-can-be-skipped -See https://slinkydeveloper.com/Kubernetes-controllers-A-New-Hope/ - -## sys.Stat_t - -We expose `stat` information as `sys.Stat_t`, like `syscall.Stat_t` except -defined without build constraints. For example, you can use `sys.Stat_t` on -`GOOS=windows` which doesn't define `syscall.Stat_t`. - -The first use case of this is to return inodes from `fs.FileInfo` without -relying on platform-specifics. For example, a user could return `*sys.Stat_t` -from `info.Sys()` and define a non-zero inode for a virtual file, or map a -real inode to a virtual one. - -Notable choices per field are listed below, where `sys.Stat_t` is unlike -`syscall.Stat_t` on `GOOS=linux`, or needs clarification. One common issue -not repeated below is that numeric fields are 64-bit when at least one platform -defines it that large. Also, zero values are equivalent to nil or absent. - -* `Dev` and `Ino` (`Inode`) are both defined unsigned as they are defined - opaque, and most `syscall.Stat_t` also defined them unsigned. There are - separate sections in this document discussing the impact of zero in `Ino`. -* `Mode` is defined as a `fs.FileMode` even though that is not defined in POSIX - and will not map to all possible values. This is because the current use is - WASI, which doesn't define any types or features not already supported. By - using `fs.FileMode`, we can re-use routine experience in Go. -* `NLink` is unsigned because it is defined that way in `syscall.Stat_t`: there - can never be less than zero links to a file. We suggest defaulting to 1 in - conversions when information is not knowable because at least that many links - exist. -* `Size` is signed because it is defined that way in `syscall.Stat_t`: while - regular files and directories will always be non-negative, irregular files - are possibly negative or not defined. Notably sparse files are known to - return negative values. -* `Atim`, `Mtim` and `Ctim` are signed because they are defined that way in - `syscall.Stat_t`: Negative values are time before 1970. The resolution is - nanosecond because that's the maximum resolution currently supported in Go. - -### Why do we use `sys.EpochNanos` instead of `time.Time` or similar? - -To simplify documentation, we defined a type alias `sys.EpochNanos` for int64. -`time.Time` is a data structure, and we could have used this for -`syscall.Stat_t` time values. The most important reason we do not is conversion -penalty deriving time from common types. - -The most common ABI used in `wasip2`. This, and compatible ABI such as `wasix`, -encode timestamps in memory as a 64-bit number. If we used `time.Time`, we -would have to convert an underlying type like `syscall.Timespec` to `time.Time` -only to later have to call `.UnixNano()` to convert it back to a 64-bit number. - -In the future, the component model module "wasi-filesystem" may represent stat -timestamps with a type shared with "wasi-clocks", abstractly structured similar -to `time.Time`. However, component model intentionally does not define an ABI. -It is likely that the canonical ABI for timestamp will be in two parts, but it -is not required for it to be intermediately represented this way. A utility -like `syscall.NsecToTimespec` could split an int64 so that it could be written -to memory as 96 bytes (int64, int32), without allocating a struct. - -Finally, some may confuse epoch nanoseconds with 32-bit epoch seconds. While -32-bit epoch seconds has "The year 2038" problem, epoch nanoseconds has -"The Year 2262" problem, which is even less concerning for this library. If -the Go programming language and wazero exist in the 2200's, we can make a major -version increment to adjust the `sys.EpochNanos` approach. Meanwhile, we have -faster code. - -## poll_oneoff - -`poll_oneoff` is a WASI API for waiting for I/O events on multiple handles. -It is conceptually similar to the POSIX `poll(2)` syscall. -The name is not `poll`, because it references [“the fact that this function is not efficient -when used repeatedly with the same large set of handles”][poll_oneoff]. - -We chose to support this API in a handful of cases that work for regular files -and standard input. We currently do not support other types of file descriptors such -as socket handles. - -### Clock Subscriptions - -As detailed above in [sys.Nanosleep](#sysnanosleep), `poll_oneoff` handles -relative clock subscriptions. In our implementation we use `sys.Nanosleep()` -for this purpose in most cases, except when polling for interactive input -from `os.Stdin` (see more details below). - -### FdRead and FdWrite Subscriptions - -When subscribing a file descriptor (except `Stdin`) for reads or writes, -the implementation will generally return immediately with success, unless -the file descriptor is unknown. The file descriptor is not checked further -for new incoming data. Any timeout is cancelled, and the API call is able -to return, unless there are subscriptions to `Stdin`: these are handled -separately. - -### FdRead and FdWrite Subscription to Stdin - -Subscribing `Stdin` for reads (writes make no sense and cause an error), -requires extra care: wazero allows to configure a custom reader for `Stdin`. - -In general, if a custom reader is found, the behavior will be the same -as for regular file descriptors: data is assumed to be present and -a success is written back to the result buffer. - -However, if the reader is detected to read from `os.Stdin`, -a special code path is followed, invoking `sysfs.poll()`. - -`sysfs.poll()` is a wrapper for `poll(2)` on POSIX systems, -and it is emulated on Windows. - -### Poll on POSIX - -On POSIX systems, `poll(2)` allows to wait for incoming data on a file -descriptor, and block until either data becomes available or the timeout -expires. - -Usage of `syfs.poll()` is currently only reserved for standard input, because - -1. it is really only necessary to handle interactive input: otherwise, - there is no way in Go to peek from Standard Input without actually - reading (and thus consuming) from it; - -2. if `Stdin` is connected to a pipe, it is ok in most cases to return - with success immediately; - -3. `syfs.poll()` is currently a blocking call, irrespective of goroutines, - because the underlying syscall is; thus, it is better to limit its usage. - -So, if the subscription is for `os.Stdin` and the handle is detected -to correspond to an interactive session, then `sysfs.poll()` will be -invoked with a the `Stdin` handle *and* the timeout. - -This also means that in this specific case, the timeout is uninterruptible, -unless data becomes available on `Stdin` itself. - -### Select on Windows - -On Windows `sysfs.poll()` cannot be delegated to a single -syscall, because there is no single syscall to handle sockets, -pipes and regular files. - -Instead, we emulate its behavior for the cases that are currently -of interest. - -- For regular files, we _always_ report them as ready, as -[most operating systems do anyway][async-io-windows]. - -- For pipes, we invoke [`PeekNamedPipe`][peeknamedpipe] -for each file handle we detect is a pipe open for reading. -We currently ignore pipes open for writing. - -- Notably, we include also support for sockets using the [WinSock -implementation of `poll`][wsapoll], but instead -of relying on the timeout argument of the `WSAPoll` function, -we set a 0-duration timeout so that it behaves like a peek. - -This way, we can check for regular files all at once, -at the beginning of the function, then we poll pipes and -sockets periodically using a cancellable `time.Tick`, -which plays nicely with the rest of the Go runtime. - -### Impact of blocking - -Because this is a blocking syscall, it will also block the carrier thread of -the goroutine, preventing any means to support context cancellation directly. - -There are ways to obviate this issue. We outline here one idea, that is however -not currently implemented. A common approach to support context cancellation is -to add a signal file descriptor to the set, e.g. the read-end of a pipe or an -eventfd on Linux. When the context is canceled, we may unblock a Select call by -writing to the fd, causing it to return immediately. This however requires to -do a bit of housekeeping to hide the "special" FD from the end-user. - -[poll_oneoff]: https://github.com/WebAssembly/wasi-poll#why-is-the-function-called-poll_oneoff -[async-io-windows]: https://tinyclouds.org/iocp_links -[peeknamedpipe]: https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-peeknamedpipe -[wsapoll]: https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsapoll - -## Signed encoding of integer global constant initializers - -wazero treats integer global constant initializers signed as their interpretation is not known at declaration time. For -example, there is no signed integer [value type](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#value-types%E2%91%A0). - -To get at the problem, let's use an example. -``` -(global (export "start_epoch") i64 (i64.const 1620216263544)) -``` - -In both signed and unsigned LEB128 encoding, this value is the same bit pattern. The problem is that some numbers are -not. For example, 16256 is `807f` encoded as unsigned, but `80ff00` encoded as signed. - -While the specification mentions uninterpreted integers are in abstract [unsigned values](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A0), -the binary encoding is clear that they are encoded [signed](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A4). - -For consistency, we go with signed encoding in the special case of global constant initializers. - -## Implementation limitations - -WebAssembly 1.0 (20191205) specification allows runtimes to [limit certain aspects of Wasm module or execution](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#a2-implementation-limitations). - -wazero limitations are imposed pragmatically and described below. - -### Number of functions in a module - -The possible number of function instances in [a module](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#module-instances%E2%91%A0) is not specified in the WebAssembly specifications since [`funcaddr`](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#syntax-funcaddr) corresponding to a function instance in a store can be arbitrary number. -wazero limits the maximum function instances to 2^27 as even that number would occupy 1GB in function pointers. - -That is because not only we _believe_ that all use cases are fine with the limitation, but also we have no way to test wazero runtimes under these unusual circumstances. - -### Number of function types in a store - -There's no limitation on the number of function types in [a store](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#store%E2%91%A0) according to the spec. In wazero implementation, we assign each function type to a unique ID, and choose to use `uint32` to represent the IDs. -Therefore the maximum number of function types a store can have is limited to 2^27 as even that number would occupy 512MB just to reference the function types. - -This is due to the same reason for the limitation on the number of functions above. - -### Number of values on the stack in a function - -While the the spec does not clarify a limitation of function stack values, wazero limits this to 2^27 = 134,217,728. -The reason is that we internally represent all the values as 64-bit integers regardless of its types (including f32, f64), and 2^27 values means -1 GiB = (2^30). 1 GiB is the reasonable for most applications [as we see a Goroutine has 250 MB as a limit on the stack for 32-bit arch](https://github.com/golang/go/blob/go1.20/src/runtime/proc.go#L152-L159), considering that WebAssembly is (currently) 32-bit environment. - -All the functions are statically analyzed at module instantiation phase, and if a function can potentially reach this limit, an error is returned. - -### Number of globals in a module - -Theoretically, a module can declare globals (including imports) up to 2^32 times. However, wazero limits this to 2^27(134,217,728) per module. -That is because internally we store globals in a slice with pointer types (meaning 8 bytes on 64-bit platforms), and therefore 2^27 globals -means that we have 1 GiB size of slice which seems large enough for most applications. - -### Number of tables in a module - -While the the spec says that a module can have up to 2^32 tables, wazero limits this to 2^27 = 134,217,728. -One of the reasons is even that number would occupy 1GB in the pointers tables alone. Not only that, we access tables slice by -table index by using 32-bit signed offset in the compiler implementation, which means that the table index of 2^27 can reach 2^27 * 8 (pointer size on 64-bit machines) = 2^30 offsets in bytes. - -We _believe_ that all use cases are fine with the limitation, but also note that we have no way to test wazero runtimes under these unusual circumstances. - -If a module reaches this limit, an error is returned at the compilation phase. - -## Compiler engine implementation - -### Why it's safe to execute runtime-generated machine codes against async Goroutine preemption - -Goroutine preemption is the mechanism of the Go runtime to switch goroutines contexts on an OS thread. -There are two types of preemption: cooperative preemption and async preemption. The former happens, for example, -when making a function call, and it is not an issue for our runtime-generated functions as they do not make -direct function calls to Go-implemented functions. On the other hand, the latter, async preemption, can be problematic -since it tries to interrupt the execution of Goroutine at any point of function, and manipulates CPU register states. - -Fortunately, our runtime-generated machine codes do not need to take the async preemption into account. -All the assembly codes are entered via the trampoline implemented as Go Assembler Function (e.g. [arch_amd64.s](./arch_amd64.s)), -and as of Go 1.20, these assembler functions are considered as _unsafe_ for async preemption: -- https://github.com/golang/go/blob/go1.20rc1/src/runtime/preempt.go#L406-L407 -- https://github.com/golang/go/blob/9f0234214473dfb785a5ad84a8fc62a6a395cbc3/src/runtime/traceback.go#L227 - -From the Go runtime point of view, the execution of runtime-generated machine codes is considered as a part of -that trampoline function. Therefore, runtime-generated machine code is also correctly considered unsafe for async preemption. - -## Why context cancellation is handled in Go code rather than native code - -Since [wazero v1.0.0-pre.9](https://github.com/tetratelabs/wazero/releases/tag/v1.0.0-pre.9), the runtime -supports integration with Go contexts to interrupt execution after a timeout, or in response to explicit cancellation. -This support is internally implemented as a special opcode `builtinFunctionCheckExitCode` that triggers the execution of -a Go function (`ModuleInstance.FailIfClosed`) that atomically checks a sentinel value at strategic points in the code. - -[It _is indeed_ possible to check the sentinel value directly, without leaving the native world][native_check], thus sparing some cycles; -however, because native code never preempts (see section above), this may lead to a state where the other goroutines -never get the chance to run, and thus never get the chance to set the sentinel value; effectively preventing -cancellation from taking place. - -[native_check]: https://github.com/tetratelabs/wazero/issues/1409 - -## Golang patterns - -### Hammer tests -Code that uses concurrency primitives, such as locks or atomics, should include "hammer tests", which run large loops -inside a bounded amount of goroutines, run by half that many `GOMAXPROCS`. These are named consistently "hammer", so -they are easy to find. The name inherits from some existing tests in [golang/go](https://github.com/golang/go/search?q=hammer&type=code). - -Here is an annotated description of the key pieces of a hammer test: -1. `P` declares the count of goroutines to use, defaulting to 8 or 4 if `testing.Short`. - * Half this amount are the cores used, and 4 is less than a modern laptop's CPU. This allows multiple "hammer" tests to run in parallel. -2. `N` declares the scale of work (loop) per goroutine, defaulting to value that finishes in ~0.1s on a modern laptop. - * When in doubt, try 1000 or 100 if `testing.Short` - * Remember, there are multiple hammer tests and CI nodes are slow. Slower tests hurt feedback loops. -3. `defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(P/2))` makes goroutines switch cores, testing visibility of shared data. -4. To ensure goroutines execute at the same time, block them with `sync.WaitGroup`, initialized to `Add(P)`. - * `sync.WaitGroup` internally uses `runtime_Semacquire` not available in any other library. - * `sync.WaitGroup.Add` with a negative value can unblock many goroutines at the same time, e.g. without a for loop. -5. Track goroutines progress via `finished := make(chan int)` where each goroutine in `P` defers `finished <- 1`. - 1. Tests use `require.XXX`, so `recover()` into `t.Fail` in a `defer` function before `finished <- 1`. - * This makes it easier to spot larger concurrency problems as you see each failure, not just the first. - 2. After the `defer` function, await unblocked, then run the stateful function `N` times in a normal loop. - * This loop should trigger shared state problems as locks or atomics are contended by `P` goroutines. -6. After all `P` goroutines launch, atomically release all of them with `WaitGroup.Add(-P)`. -7. Block the runner on goroutine completion, by (`<-finished`) for each `P`. -8. When all goroutines complete, `return` if `t.Failed()`, otherwise perform follow-up state checks. - -This is implemented in wazero in [hammer.go](internal/testing/hammer/hammer.go) - -### Lock-free, cross-goroutine observations of updates - -How to achieve cross-goroutine reads of a variable are not explicitly defined in https://go.dev/ref/mem. wazero uses -atomics to implement this following unofficial practice. For example, a `Close` operation can be guarded to happen only -once via compare-and-swap (CAS) against a zero value. When we use this pattern, we consistently use atomics to both -read and update the same numeric field. - -In lieu of formal documentation, we infer this pattern works from other sources (besides tests): - * `sync.WaitGroup` by definition must support calling `Add` from other goroutines. Internally, it uses atomics. - * rsc in golang/go#5045 writes "atomics guarantee sequential consistency among the atomic variables". - -See https://github.com/golang/go/blob/go1.20/src/sync/waitgroup.go#L64 -See https://github.com/golang/go/issues/5045#issuecomment-252730563 -See https://www.youtube.com/watch?v=VmrEG-3bWyM |