change pipe-level none handling and empty subquery

book/src/super-sql/expressions/match.md

@@ -19,7 +19,8 @@ match expression.  A value matches a type by the same rules as a
 its underlying value and [named types](../types/named.md) are distinct.
 
 If no case matches, the result is `<else-result>` or, without an
-`else` clause, `none`, which disappears from a record, array, or map
+`else` clause, `none`, which disappears from a record, array, or map,
+is dropped from the output of the [values](../operators/values.md) operator,
 and is typically handled with the `??` operator.
 An [error](../types/error.md) or `missing` value is passed through
 unless a case has its type.

book/src/super-sql/expressions/subqueries.md

@@ -19,6 +19,9 @@ results in the value `{s:"HELLO, WORLD"}`.
 Except for subqueries appearing as the right-hand side of
 an [in](containment.md) operator, the result of a subquery must be a single value.
 When multiple values are generated, an [error](../types/error.md) is produced.
+When no values are generated, the result is `none`, which disappears from a
+record, array, or map and is dropped from the output of the
+[values](../operators/values.md) operator, or, in a SQL expression, `null`.
 
 For the [in](containment.md) operator, any subquery on the right-hand side is
 always treated as an [array subquery](#array-subqueries), thus
@@ -278,3 +281,20 @@ values walk(this, &addOne)
 [2,3,4]
 [{x:[2,"foo"]},{y:3}]
 ```
+
+---
+
+_A subquery that produces no values is none_
+
+```mdtest-spq
+# spq
+values {s:(unnest a | where this > 1)}, (unnest a | where this > 1) ?? 0
+# input
+{a:[1,2]}
+{a:[1]}
+# expected output
+{s:2}
+2
+{}
+0
+```

book/src/super-sql/expressions/type-assertions.md

@@ -7,7 +7,8 @@ A type assertion has the form
 where `<expr>` is any [expression](intro.md) and `<type>` is any
 [type](../types/intro.md).  Its value is the value of `<expr>` when that
 value has type `<type>` and is otherwise `none`,
-which disappears from a record, array, or map and is typically handled with
+which disappears from a record, array, or map, is dropped from the output of the
+[values](../operators/values.md) operator, and is typically handled with
 the `??` operator, e.g.,
 ```
 v.(string) ?? "default"

book/src/super-sql/functions/generics/ok.md

@@ -16,8 +16,9 @@ unmodified.
 
 This function is useful when errors are expected and are to be explicitly managed
 with `none` operators.  The `none` disappears from a record expression, an
-array, or a map and otherwise must be handled with `??` since a query may not
-output a `none`.
+array, or a map and is dropped from the output of the
+[values](../../operators/values.md) operator, so `??` is needed only to replace
+it with another value.
 
 ## Examples
 

book/src/super-sql/operators/aggregate.md

@@ -55,6 +55,10 @@ The output values are records formed from the
 [field assignments](intro.md#field-assignment)
 first from the grouping expressions then from the aggregate functions
 in left-to-right order.
+As in a record expression, a grouping key whose value
+is a plain `none` (e.g., `ok(x)` when `x` is missing) disappears from its
+output record, so the values whose key is `none` form a group whose record
+has no field for that key.
 
 When the result of `aggregate` is a single value (e.g., a single aggregate
 function without grouping expressions or a single grouping expression without aggregates)

book/src/super-sql/operators/debug.md

@@ -16,6 +16,8 @@ each value in the input, evaluates the optional
 and transmits each result to a _debugging output_.
 If `<expr>` is omitted, then each input value is passed unmodified to the
 debugging output.
+As for the [values](values.md) operator, a result that is a plain `none`
+(e.g., from [ok](../functions/generics/ok.md) or `?.`) is dropped.
 
 An optional filter may be applied to the values sent to the debugging output,
 which is specified with Boolean-valued expression `<pred>`.

book/src/super-sql/operators/unnest.md

@@ -41,7 +41,8 @@ produces
 ```
 
 The value of `<expr>` must be an array (or a set), whose elements form the
-derived sequence.
+derived sequence.  A `null` or a plain `none` value (e.g., from
+[ok](../functions/generics/ok.md) or `?.`) produces no values.
 
 To carry other values along with each unnested element, give the unnested
 value a name with `:=` or list the values in a `with` clause.  Then each

book/src/super-sql/operators/values.md

@@ -19,6 +19,13 @@ in left-to-right order.  Each `<expr>` may be any valid
 The input order convolved with left-to-right evaluation order is preserved
 at the output.
 
+A result that is a plain `none`, e.g., from [ok](../functions/generics/ok.md),
+`?.`, a [type assertion](../expressions/type-assertions.md) that fails, or a
+[match](../expressions/match.md) with no matching case and no `else`,
+is dropped, so a `none` is never passed to a downstream operator.
+To output another value in its place, use the `??` operator.
+An option's `none` (e.g., `none::int64`) is a value and is output as is.
+
 The `values` operator name is optional since it can be used as a
 [shortcut](intro.md#shortcuts).  When used as a shortcut, only one expression
 may be present.
@@ -127,3 +134,18 @@ values [a,b],[b,a] | collect(this)
 # expected output
 [[1,2],[2,1]]
 ```
+
+---
+
+_A plain none is dropped unless it's replaced with ??_
+```mdtest-spq
+# spq
+values ok(a), ok(a) ?? 0
+# input
+{a:1}
+{b:2}
+# expected output
+1
+1
+0
+```

compiler/dag/expr.go

@@ -146,7 +146,11 @@ type (
 	SubqueryExpr struct {
 		Kind       string `json:"kind" unpack:""`
 		Correlated bool   `json:"correlated"`
-		Body       Seq    `json:"body"`
+		// SQL is true for a subquery in a SQL expression, whose value is
+		// null when the subquery produces no values.  Otherwise, the value
+		// is none.
+		SQL  bool `json:"sql,omitempty"`
+		Body Seq  `json:"body"`
 	}
 	ThisExpr struct {
 		Kind  string      `json:"kind" unpack:""`

compiler/dag/op.go

@@ -370,7 +370,7 @@ func NewFilterOp(e Expr) *FilterOp {
 }
 
 func NewValuesOp(exprs ...Expr) *ValuesOp {
-	return &ValuesOp{"ValuesOp", exprs}
+	return &ValuesOp{Kind: "ValuesOp", Exprs: exprs}
 }
 
 func CopySeq(seq Seq) Seq {

compiler/rungen/vexpr.go

@@ -313,11 +313,11 @@ func (b *Builder) compileVamSubquery(query *dag.SubqueryExpr) (expr.Evaluator, e
 			return nil, err
 		}
 		body := b.combineVam(exits)
-		return op.NewCachedSubquery(b.sctx(), body), nil
+		return op.NewCachedSubquery(b.sctx(), body, query.SQL), nil
 	}
 	var create func() *op.Subquery
 	create = func() *op.Subquery {
-		subquery := op.NewSubquery(b.rctx.Context, b.sctx(), func() *op.Subquery {
+		subquery := op.NewSubquery(b.rctx.Context, b.sctx(), query.SQL, func() *op.Subquery {
 			b.mu.Lock()
 			defer b.mu.Unlock()
 			return create()

compiler/rungen/vop.go

@@ -317,7 +317,7 @@ func (b *Builder) compileVamLeaf(o dag.Op, parent vio.Puller) (vio.Puller, error
 		return vio.NewPuller(vector.NewNull(1)), nil
 	case *dag.OutputOp:
 		if parent != nil {
-			parent = op.NewOutput(b.sctx(), parent)
+			parent = op.NewOutput(parent)
 		}
 		b.channels[o.Name] = append(b.channels[o.Name], parent)
 		return parent, nil

compiler/semantic/analyzer.go

@@ -79,9 +79,6 @@ type translator struct {
 	sctx     *super.Context
 	types    *sup.Analyzer
 	defs     *super.Context
-	// subqueries is the depth of expression subqueries being translated.
-	// Their results may be none (see checker.output).
-	subqueries int
 }
 
 func newTranslator(ctx context.Context, r reporter, env *exec.Environment) *translator {

compiler/semantic/checker.go

@@ -162,7 +162,7 @@ func (c *checker) op(typ super.Type, op sem.Op) super.Type {
 	case *sem.UnnestOp:
 		return c.seq(c.unnest(op.Expr, c.expr(typ, op.Expr), op.Field), op.Body)
 	case *sem.ValuesOp:
-		return c.fuse(c.exprs(typ, op.Exprs))
+		return c.values(c.exprs(typ, op.Exprs))
 	default:
 		panic(op)
 	}
@@ -213,13 +213,21 @@ func (c *checker) unnest(loc ast.Node, typ super.Type, field string) super.Type
 	if !ok {
 		c.keepErrs(errs)
 	}
+	if typ == nil {
+		// The value is always a plain none, so there are no values.
+		return c.unknown
+	}
 	return typ
 }
 
 func (c *checker) unnestCheck(loc ast.Node, typ super.Type) (super.Type, bool) {
 	if isUnknown(typ) {
 		return c.unknown, true
 	}
+	if typ == super.TypeNone {
+		// A plain none produces no values.
+		return nil, true
+	}
 	switch typ := under(typ).(type) {
 	case *super.TypeError:
 		c.error(loc, errors.New("unnested record cannot be an error"))
@@ -230,7 +238,9 @@ func (c *checker) unnestCheck(loc ast.Node, typ super.Type) (super.Type, bool) {
 		for _, t := range typ.Types {
 			typ, tok := c.unnestCheck(loc, t)
 			if tok {
-				types = append(types, typ)
+				if typ != nil {
+					types = append(types, typ)
+				}
 				ok = true
 			}
 		}
@@ -258,6 +268,9 @@ func (c *checker) unnestField(loc ast.Node, typ super.Type, field string) (super
 		return c.unknown, true
 	}
 	elemType, ok := c.unnestCheck(loc, rec.Fields[k].Type)
+	if elemType == nil {
+		return nil, ok
+	}
 	fields := slices.Clone(rec.Fields)
 	fields[k].Type = elemType
 	out, err := c.t.sctx.LookupTypeRecord(fields)
@@ -1734,23 +1747,40 @@ func (c *checker) result(loc ast.Node, typ super.Type) {
 	}
 }
 
-// output reports an error at loc if typ, the type of an output value, may
-// be a plain none.  (An option none or a none of a named type is a value
-// and is allowed.)
-func (c *checker) output(loc ast.Node, typ super.Type) {
-	typ = defuse(typ)
-	if typ != nil && !hasUnknown(typ) && hasPlainNone(typ) {
-		c.error(loc, errNone)
+// values returns the type of the output of a values operator whose
+// expressions have types.  The operator drops plain nones, so none is removed
+// from each type and an expression that's always a plain none produces no
+// values.  (An option none or a none of a named type is a value and is kept.)
+func (c *checker) values(types []super.Type) super.Type {
+	var out []super.Type
+	for _, typ := range types {
+		if typ, ok := c.withoutPlainNone(typ); ok {
+			out = append(out, typ)
+		}
 	}
+	return c.fuse(out)
 }
 
-// hasPlainNone reports whether typ is none or a union with none as one of
-// its types.  Unlike hasNone, a named type whose type is none isn't none.
-func hasPlainNone(typ super.Type) bool {
-	if u, ok := typ.(*super.TypeUnion); ok {
-		return slices.ContainsFunc(u.Types, hasPlainNone)
+// withoutPlainNone returns typ with plain none removed from it (from its
+// members if it's a union).  It returns false if nothing is left.
+func (c *checker) withoutPlainNone(typ super.Type) (super.Type, bool) {
+	if typ == nil || hasUnknown(typ) {
+		return typ, true
+	}
+	fusion, isFusion := typ.(*super.TypeFusion)
+	out, ok := c.withoutTypes(defuse(typ), func(typ super.Type) bool {
+		return typ == super.TypeNone
+	})
+	if !ok {
+		return nil, false
+	}
+	if isFusion && out != fusion.Type {
+		if _, isUnion := out.(*super.TypeUnion); isUnion {
+			// The values are still those of a fusion.
+			return c.t.sctx.MustLookupTypeFusion(out), true
+		}
 	}
-	return typ == super.TypeNone
+	return out, true
 }
 
 // notMissing reports an error at loc if typ may be missing.

compiler/semantic/compare.go

@@ -281,7 +281,7 @@ func eqExpr(aexpr, bexpr sem.Expr) bool {
 		return ok && eqExpr(a.Expr, b.Expr) && eqExpr(a.From, b.From) && eqExpr(a.To, b.To)
 	case *sem.SubqueryExpr:
 		b, ok := bexpr.(*sem.SubqueryExpr)
-		return ok && a.Correlated == b.Correlated && a.Array == b.Array && eqSeq(a.Body, b.Body)
+		return ok && a.Correlated == b.Correlated && a.Array == b.Array && a.SQL == b.SQL && eqSeq(a.Body, b.Body)
 	case *sem.ThisExpr:
 		b, ok := bexpr.(*sem.ThisExpr)
 		if !ok || len(a.Chain) != len(b.Chain) {

compiler/semantic/dagen.go

@@ -587,6 +587,7 @@ func (d *dagen) subquery(e *sem.SubqueryExpr) *dag.SubqueryExpr {
 	subquery := &dag.SubqueryExpr{
 		Kind:       "SubqueryExpr",
 		Correlated: e.Correlated,
+		SQL:        e.SQL,
 		Body:       d.seq(e.Body),
 	}
 	if e.Array {

compiler/semantic/expr.go

@@ -730,6 +730,7 @@ func (t *translator) maybeSubquery(n ast.Node, name string, inType super.Type) (
 		return &sem.SubqueryExpr{
 			Node:       n,
 			Correlated: isCorrelated(seq),
+			SQL:        t.scope.sql != nil,
 			Body:       seq,
 		}, typ
 	}
@@ -884,6 +885,7 @@ func (t *translator) maybeSubqueryCall(call *ast.CallExpr, name string, inType s
 		Node:       call,
 		Array:      false,
 		Correlated: isCorrelated(userOp),
+		SQL:        t.scope.sql != nil,
 		Body:       userOp,
 	}, typ
 }
@@ -1327,14 +1329,13 @@ func (t *translator) arraySubquery(elems []sem.ArrayElem) *sem.SubqueryExpr {
 func (t *translator) subqueryExpr(astExpr ast.Expr, array bool, body ast.Seq, inType super.Type) (*sem.SubqueryExpr, super.Type) {
 	// We pass inType in whether or not it's correlated since an uncorrelated
 	// subquery will just ignore it.
-	t.subqueries++
 	seq, outType := t.seq(body, inType)
-	t.subqueries--
 	correlated := isCorrelated(seq)
 	e := &sem.SubqueryExpr{
 		Node:       astExpr,
 		Array:      array,
 		Correlated: correlated,
+		SQL:        t.scope.sql != nil,
 		Body:       seq,
 	}
 	// Add a nullscan only for uncorrelated queries that don't have a source.

compiler/semantic/op.go

@@ -1117,9 +1117,8 @@ func (t *translator) semOp(o ast.Op, seq sem.Seq, inType super.Type) (sem.Seq, s
 		exprs, types := t.exprs(o.Exprs, inType)
 		for k, typ := range types {
 			t.checker.result(o.Exprs[k], typ)
-			t.output(o.Exprs[k], typ)
 		}
-		return append(seq, sem.NewValues(o, exprs...)), t.checker.fuse(types)
+		return append(seq, sem.NewValues(o, exprs...)), t.checker.values(types)
 	case *ast.Script:
 		t.error(o, errors.New("a script of multiple statements cannot be used here"))
 		return append(seq, badOp), badType
@@ -1655,10 +1654,10 @@ func (t *translator) exprOp(e ast.Expr, seq sem.Seq, inType super.Type) (sem.Seq
 	if t.isBool(out) {
 		return append(seq, &sem.FilterOp{Node: e, Expr: out}), inType
 	}
-	// As with the values operator, the value must not be missing or none.
+	// As with the values operator, the value must not be missing and a
+	// plain none is dropped.
 	t.checker.result(e, typ)
-	t.output(e, typ)
-	return append(seq, sem.NewValues(e, out)), typ
+	return append(seq, sem.NewValues(e, out)), t.checker.values([]super.Type{typ})
 }
 
 func (t *translator) isBool(e sem.Expr) bool {
@@ -1962,21 +1961,6 @@ func (t *translator) maybeEval(e sem.Expr) (super.Value, bool) {
 	return newEvaluator(t, t.resolver.funcs).maybeEval(t.sctx, e)
 }
 
-// output reports an error at loc if typ, the type of a value output by the
-// values operator, may be a plain none (e.g., from ok() or ?.).  A plain none
-// disappears from a record, array, or map but otherwise must be handled with
-// ?? before it leaves the operator that produced it, i.e., a plain none is
-// never passed to a downstream operator, even one that would handle it.
-// (A plain none that reaches the query output at runtime is an error.)  The
-// result of an expression subquery may be none since it's an expression
-// value.
-func (t *translator) output(loc ast.Node, typ super.Type) {
-	if t.subqueries > 0 {
-		return
-	}
-	t.checker.output(loc, typ)
-}
-
 // unnestExpr translates the expression of an unnest operator and returns it,
 // its type, and, if the unnest is named ("unnest name:=expr") or has a with
 // clause ("with assignments"), the name of the field to unnest.  In that