Introduction to TableParser

RFC 4180 compliant. From version 2.0.0, TableParser reads and writes CSV exactly as defined by RFC 4180, the standard for CSV files. Every rule of the RFC is tested in Rfc4180Spec, and a table that TableParser writes is read back exactly (except that a line break inside a field is always read as LF). In particular:

Fields are separated by commas, and spaces are part of a field (so a, b has the fields a and b).

A field may be quoted, in which case it may contain commas, line breaks and quotes (which are doubled: "say ""hi""").

When writing, a field is quoted only when it has to be (when it contains a comma, a quote or a line break, or starts or ends with a space).

Records end with CRLF when written; CRLF or LF are both read.

To read data whose fields are separated by ", " (for example, written by 1.x with CsvAttributes(", ")), use RowConfig.trimmingRowConfig, which ignores spaces next to commas, as 1.x did. To write LF line endings instead of CRLF, use CsvAttributes(",", "\"", "\n").

Full Scaladoc is available via javadoc.io, for Scala 3 (and, for the 1.x line, Scala 2.13).

A functional parser of tables, implemented in Scala. Typically, the input is in the form of a "CSV" (comma-separated-values) file. However, it is perfectly possible to parse other formats. Indeed, as of 1.3.0, it is possible to parse Parquet files. Please see the TableParser Parquet Design design document for more information.

TableParser aims to make it as simple as possible to ingest a fully typed tabular dataset. The principal mechanism for this is the use of case classes to specify the types of fields in the dataset. All conversions from strings to standard types are performed automatically, and the parsers (and renderers) of case classes are derived by the compiler, with no limit on the number of fields. For non-standard types, it suffices simply to provide a converter of the form String=>T.

It is possible to parse sequences of String (one per row)-- the typical situation for a CSV file--or sequences of String (where the table corresponds to a matrix of cells).

This library makes extensive use of type classes (given instances, and derivation from a case class's Mirror). Indeed, it is implemented very similarly to JSON readers. There is a row-parser configuration (RowConfig) that allows the programmer to vary the delimiter, the quote character, and how lists within a cell are written.

In addition to parsing, TableParser provides a mechanism for rendering a table in hierarchical form (for example for XML or HTML). An output structure which is itself tabular or sequence-oriented can be generated quite easily using the rows of the table, together with something like, for instance, a JSON writer.

Package Structure

As of version 2.0.0, the code is made up of six modules (each published as tableparser-):

core: parsing and rendering tables. Most of the remainder of this README file refers to the core module.
cats: for use with Cats Effect, particularly IO (see IOTable).
zio: for use with ZIO (see ZIOTable).
crypto: encryption of tables (e.g., of the rows of a CSV file), using tsec and Cats Effect (new in 2.0.0: in 1.x, this was part of cats).
json: reading and writing tables as JSON, using circe (new in 2.0.0: in 1.x, JSON support, using spray-json, was part of core).
parquet: for use with Apache Parquet (beginning with 1.3.0).

The spark module (for use with Apache Spark) is on the 1.x line only, because Spark is not published for Scala 3.

Quick Intro

The simplest way to get an introduction to TableParser is to consult the movie.sc. and airbnb.sc worksheets (the latter is in the cats package). These give detailed descriptions of each stage of the process.

Another way to see how it works is to look at this application Pairings which takes a CSV file, parses it, transforms the data, and outputs a JSON file. This way of parsing is a little different from what is shown in the worksheets. But both are effective. The minimum code necessary to parse the CSV file as a table of "Player"s, using as many defaults as possible is:

case class Player(first: String, last: String)

object Player extends TableParserHelper[Player]() {
  def cellParser: CellParser[Player] = CellParser.derived[Player]
}

val pty: Try[Table[Player]] = Table.parseFile("players.csv")

The TableParserHelper used here is an abstract subclass of CellParsers, and is customized for the row type (in this case, Player). In particular, it defines an implicit TableParser[Table[X]], where X is the row type (Player in this example). This assumes that the source input file ("players.csv") contains a header row which includes column names corresponding to the parameters of the case class Player (in this case "first" and "last"). If, for example, your CSV file does not have a header row, then you make a minor change to the line object Player...

The input file looks something like this (the first and last columns are required, others are ignored):

Id,First,Last,
1,Adam,Sullivan,
2,Amy,Avagadro,
3,Anna,Peterson,

etc...

Note that columns not needed for the Player case class are simply ignored. Also, note that a column whose name matches exactly is preferred; only if there is none is the case of the column names ignored.

For another simple use case TableParser, please see my blog at: https://scalaprof.blogspot.com/2019/04/new-projects.html

User Guide

This version of the README.md file refers to version: 2.1.0.

From version 2.0.0, TableParser is for Scala 3 only (it is built with Scala 3.3 LTS, so it can be used from any later Scala 3 version), and it needs Java 21 or later. For Scala 2.13 (or Java 17), and for the spark module, use the 1.x line (the latest is 1.7.2), which is on the 1.x branch.

To use it, add the modules you need to your build.sbt:

libraryDependencies ++= Seq(
    "com.phasmidsoftware" %% "tableparser-core" % "2.1.0"
  , "com.phasmidsoftware" %% "tableparser-cats" % "2.1.0"    // cats-effect IO support
  , "com.phasmidsoftware" %% "tableparser-zio" % "2.1.0"     // ZIO support
  , "com.phasmidsoftware" %% "tableparser-crypto" % "2.1.0"  // encryption
  , "com.phasmidsoftware" %% "tableparser-json" % "2.1.0"    // JSON (circe)
  , "com.phasmidsoftware" %% "tableparser-parquet" % "2.1.0" // Parquet files
)

If you are moving from 1.x, see Migrating from 1.x below. See the release notes (at the end) for history.

Migrating from 1.x

2.0.0 is for Scala 3, and makes the most of it: almost everything that was written out by hand for each case class in 1.x is now derived by the compiler. In the table below, "1.x" is what you wrote for 1.7, and "2.0" is what replaces it.

Topic	1.x	2.0
Scala and Java	Scala 2.13 or 3; Java 17	Scala 3 only (3.3 LTS or later); Java 21
Parsing a case class	`cellParser2(apply)` ... `cellParser13(apply)` (at most 13 fields)	`CellParser.derived[T]` (or `derives CellParser`), with no limit on the number of fields
Rendering a case class as CSV	`renderer2(apply)`, `generator2(apply)` ... (at most 19 fields)	`CsvRenderer.derived[T]`, `CsvGenerator.derived[T]`, or both at once with `derives CsvProduct`
Hierarchical rendering of a case class	`renderer2("tr")(apply)` etc.	`HierarchicalRenderer.derived[T]("tr")`
Parquet: grouped case classes	`converter2(apply)` ... `converter8(apply)`, and `converterMap`/`groupedHelpers`	derived automatically (`ParquetCellConverter.derived`): each case class needs only its `ColumnHelper`
Field names	found by reflection (`Reflection.extractFieldNames`, with a `ClassTag`)	from the case class's `Mirror`, at compile time (the `FieldNames[T]` type class); `Reflection` is gone
Implicits	`implicit val`, `implicitly`	`given`, `summon` (an `implicit val` still works, of course). Note that a wildcard import doesn't bring givens: use `import X.{given, *}`
Configuring how rows are read	`implicit object C extends DefaultRowConfig { override val delimiter = """\t""".r; override val string = ... }`	`given RowConfig = RowConfig(delimiter = '\t')`: `RowConfig` is a case class, whose regexes are derived from its characters (`withDelimiter` etc. change one property). `DefaultRowConfig` is gone
CSV format	spaces next to a comma were ignored; LF line endings	RFC 4180: spaces are part of a field (use `RowConfig.trimmingRowConfig` for 1.x behavior); CRLF when writing (`CsvAttributes(",", "\"", "\n")` for LF)
Reading and writing in the same format	`RowConfig` and `CsvAttributes` were unrelated	`rowConfig.csvAttributes` and `csvAttributes.rowConfig`
Column names when writing	the field names (with dots for nested case classes)	the names the parser looks up (via `ColumnHelper`), so a table that is written can be read back
Row order	`Content` was a `ParIterable` (rows could come back in any order)	`Content` is a `Vector`, in the order the rows were read; `content.par` for parallelism
Dates	Joda-Time (`org.joda.time.LocalDate`)	`java.time.LocalDate`
JSON	spray-json, in core (`TableJsonFormat`)	circe, in the new tableparser-json module (`TableCodecs`)
Encryption	in tableparser-cats	in the new tableparser-crypto module (package `com.phasmidsoftware.tableparser.crypto`)
`MaybeStatistics`	a sealed trait	an enum
`Sequential`, `Sequence`, the examples, `table.Main`, `EvaluateZIO`	deprecated in 1.7.0	removed (the examples are now in the tests)
A blank `Option[URL]` cell	a failure	`None`, as for every other type
`StringConverter`'s instances, `Header.multiply`/`prepend`/`alphabet`	public	removed (unused)
Names ending in `$` (e.g. `BooleanCellParser$`)	as in 1.x	the `$` is dropped (e.g. `BooleanCellParser`)

Parsing

Parsing Row-oriented Tables

The Table trait expresses the result of parsing from a serialized representation of a table. Each row is represented by a parametric type Row. Typically, this Row type is a case class with one parameter corresponding to one column in the table file. However, some table files will have too many columns to be practical for this correspondence. In such a situation, you have two choices: (1) parsing each row as a list of String (also known as a "raw" row); (2) parsing each row as a hierarchical arrangement of case classes (or tuples). Typically, especially if the dataset is new to you, you will start with (1) and run an analysis on the columns to help you design the classes for option (2).

For the first option, you will do something like the following (see the AnalysisSpec unit tests):

private val sampleFile = "2023-01-metropolitan-street-sample.csv"
private val triedSampleResource: Try[URL] = FP.resource[Analysis](sampleFile)
val fraction = 4
val parser = RawTableParser().setPredicate(TableParser.sampler(fraction))
val ui = IOUsing(for (u <- triedSampleResource) yield Source.fromURL(u)) {
    s => parser.doParse(s) map (rawTable => println(Analysis(rawTable)))
}
ui.unsafeRunSync()

This analysis will give you a list of columns, each showing its name, whether it is optional (i.e., contains nulls), and (if it's a numerical column), its range, mean, and standard deviation.

Incidentally, this raw parser has three signatures, one for resources, one for files, and one for a sequence of Strings. And the default for raw row parsing is to allow quoted strings to span multiple lines.

But, if not parsing as raw rows, you will need to design a class hierarchy to model the columns of the table. TableParser will take care of any depth of case classes. There is no limit on the number of parameters of a case class (see UnstructuredMovie, in the tests, which has a field for each of the 28 columns of the movie database).

The names of the parameters of a case class do not necessarily have to be the same as the column from which the value derives. The ColumnHelper class is available to manage the mapping between parameters and columns.

The result of parsing a table file (CSV, etc.) will be a Table[Row], wrapped in Try. There are object methods to parse most forms of text: File, Resource, InputStream, URL, Seq[String], etc. (see Table below).

The parser responsible for parsing the contents of a cell is called CellParser[T], where T is the type of the value in the cell in question. T is covariant so that if you have alternative parsers which generate different subclasses of trait, for instance, this can be done.

In order for TableParser to know how to construct a case class from a set of values, an instance of CellParser[T] must be in scope. It is derived from the case class (using its Mirror):

given CellParser[T] = CellParser.derived[T]

or, equivalently, case class T(...) derives CellParser. The parsers of the fields come from their own types (each must have a CellParser), and a field which is itself a case class (with no CellParser of its own) is derived in turn. Each field is read from the column whose name is given by the ColumnHelper[T] for that field's name.

NOTE: for a case class which is defined locally (inside a method), don't derive its CellParser inside a companion object that you write for it: initializing that object would refer to itself (a StackOverflowError). Use derives CellParser, or define the given alongside the case class.

See the section on CellParsers below.

Parsing Column-oriented Tables

Parquet parsing

The mechanisms described above are primarily designed for parsing row-oriented tables such as CSV. However, as of version 1.3.0, it is possible to parse Parquet files. Please see the TableParser Parquet Design design document for more information.

Much of the core functionality is applicable to column-oriented tables as well. The most significant differences are

that the header is always known and is defined by the Parquet file itself.
the concept of the RowParser does not translate to column-oriented tables.

For an example of how to parse a Parquet file, see the YellowTaxiTrip example in **YellowTaxiTrip. This is from the NYC Taxi dataset.

Table

The Table class, which extends Iterable[Row], also has several methods for manipulation:

query methods

def content: Content[Row]
def maybeHeader: Option[Header]
def toCSV(using renderer: CsvRenderer[Row], generator: CsvGenerator[Row], csvAttributes: CsvAttributes): Try[String]
def maybeColumnNames: Option[Seq[String]]
def column(name: String): Iterator[Option[String]]
def writeCSVFile(path: Path)(using renderer: CsvRenderer[Row], generator: CsvGenerator[Row], csvAttributes: CsvAttributes): Unit

To write a table encrypted, see TableCrypt.writeCSVFileEncrypted in the crypto module.

transformation methods

def filter(p: Row => Boolean): Table[Row]
def filterNot(p: Row => Boolean): Table[Row]
def filterValid(using rv: Validity[Row]): Table[Row]
def map[S](f: Row => S): Table[S]
def flatMap[U](f: Row => Iterable[U]): Table[U]
def mapOptional[S](f: Row => Option[S]): Table[S]
def unit[S](sc: Content[S], maybeHeader: Option[Header]): Table[S]
def unit[S](rows: Iterable[S], maybeHeader: Option[Header]): Table[S]
def ++[U >: Row](table: Table[U]): Table[U]
def zip[R](rt: Table[R]): Table[(Row, R)]
def processRows[S](f: Iterable[Row] => Iterable[S]): Table[S]
def processRows[R, S](f: (Iterable[Row], Iterable[R]) => Iterable[S])(other: Table[R]): Table[S]
def sort[S >: Row : Ordering]: Table[S]
def select(range: Range): Table[Row]
def select(n: Int): Table[Row]
def drop(n: Int): Table[Row]
def dropWhile(p: Row => Boolean): Table[Row]
def take(n: Int): Table[Row]
def takeWhile(p: Row => Boolean): Table[Row]
def sample(n: Int)(using random: Random): Table[Row]
def slice(from: Int, until: Int): Table[Row]
lazy val shuffle: Table[Row]

It is to be expected that join methods will be added later (based upon the second signature of processRows).

The following object methods are available for parsing text:

def parse[T: TableParser](ws: Iterable[String]): Try[T]
def parse[T: TableParser](ws: Iterator[String]): Try[T]
def parseSource[T: TableParser](x: => Source): Try[T]
def parse[T: TableParser](sy: => Try[Source]): Try[T]
def parse[T: TableParser](u: => URI)(using codec: Codec): Try[T]
def parse[T: TableParser](u: => URI, enc: String): Try[T]
def parseInputStream[T: TableParser](i: => InputStream)(using codec: Codec): Try[T]
def parseInputStream[T: TableParser](si: Try[InputStream], enc: String): Try[T]
def parseFile[T: TableParser](path: Path)(using codec: Codec): Try[T]
def parseFile[T: TableParser](path: Path, enc: String): Try[T]
def parseResource[T: TableParser](w: String, clazz: Class[?] = getClass)(using codec: Codec): Try[T]
def parseResource[T: TableParser](u: => URL)(using codec: Codec): Try[T]
def parseResource[T: TableParser](u: => URL, enc: String): Try[T]
def parseSequence[T: TableParser](wss: Iterator[Seq[String]]): Try[T]

and, for raw tables (of RawRow): parseRaw, parsePathRaw, parseFileRaw and parseResourceRaw. IOTable (in the cats module) and ZIOTable (in the zio module) have the same methods, giving an IO or a Task.

Please note that, in the case of a parameter being an Auto-closeable object such as InputStream or Source, it is the caller's responsibility to close it after parsing. However, if the parameter is a Path, File, or filename, or URL/URI, then any Source object that is instantiated within the parse method will be closed. This applies also to the parseInputStream methods: the internally defined Source will be closed (but not the stream).

Additionally, there is an extension method parse (defined in the TableParser companion object) which allows for expressions such as:

parser parse source

This is the recommended way to parse because it is the simplest. It also allows chaining of copy (or "lens") methods to configure the parser, for example:

val parser = RawTableParser().setPredicate(TableParser.sampler(2)).setMultiline(true)

Streaming

Parsing into a Table reads the whole table into memory. For a large file, the cats module can instead stream the rows, as they are read, with fs2:

import com.phasmidsoftware.tableparser.cats.table.IOTable

val movies: fs2.Stream[IO, Movie] = IOTable.streamFile(MovieTableParser)(Paths.get("movie_metadata.csv"))
val count: IO[Long] = movies.filter(_.production.country == "USA").compile.count

The header is read first; then each row is parsed only when the stream pulls it, so the whole table is never in memory (a stream can even read endless input, with take). The file is opened when the stream is run, and closed when it ends, however it ends. The parser's predicate selects the rows; if the parser is forgiving, a row which can't be parsed is logged and left out, otherwise the stream fails at that row. The parser is passed as a value (not a given), so that the type of the rows is known: here, Movie. There are also stream (from any Source), streamResource and streamLines.

To write a stream of rows as CSV, use the pipes IOTable.csvLines (the header, then a line for each row) and IOTable.writeCSVFile:

movies.through(IOTable.writeCSVFile[Movie](Paths.get("movies.csv"))).compile.drain

The zio module has the same methods, with ZStream (from zio-streams): ZIOTable.streamFile(MovieTableParser)(path) is a ZStream[Any, Throwable, Movie]; ZIOTable.csvLines is a ZPipeline and ZIOTable.writeCSVFile a ZSink:

movies.run(ZIOTable.writeCSVFile[Movie](Paths.get("movies.csv")))

These are built on StringTableParser.rows (in core), which parses lazily: it gives the header and an Iterator of the rows.

Selecting rows, columns and cells (RFC 7111)

RFC 7111 defines fragment identifiers for CSV: the part of a URI after "#" can select rows (data.csv#row=5-7), columns (#col=2) or cells (#cell=4,1-6,2), and several at once (#row=3;6). As in the RFC, rows and columns are numbered from 1, the header line is row 1 (so, with one header line, row 2 is the first row of data), and "*" means the last row or column. A fragment identifier with a syntax error is ignored; so is a specification which selects backwards (5-4) or beyond the table.

Table.parse of a URI with a fragment which selects rows parses only those rows (as do IOTable.streamURI and ZIOTable.streamURI, which read no further than they need to):

val berkeley: Try[Table[Reading]] = Table.parse[Table[Reading]](URI.create("file:///data/weather.csv#row=5-7"))

CsvFragment.parse parses a fragment identifier (giving None for a syntax error), as a RowSelection, ColumnSelection or CellSelection, each of which can select from a table. Rows can be selected from any table; columns and cells only from a raw table (a case class can't lose its fields). A selection keeps the header. Since a cell selection may be of several disjoint rectangles, it gives a table for each.

TableParser

TableParser is also the name of a trait that takes a parametric type called "Table" in its definition. This is NOT the same at the Table type (described above). TableParser is defined thus:

trait TableParser[Table] {
  type Row
  type Input
  protected val maybeHeader: Option[Header] = None
  val headerRowsToRead: Int = 1
  def forgiving: Boolean = false
  def multiline: Boolean = false
  val predicate: Try[Row] => Boolean = includeAll
  def rowParser: RowParser[Row, Input]
  def builder(rows: Iterator[Row]): Table
  def parse(xs: Iterator[Input], n: Int = headerRowsToRead): Try[Table]

}

The type Row defines the specific row type of the resulting Table (for example, Movie, in the example below). The type Input defines the input type, typically String, but there are also alternatives such as Seq[String]. maybeHeader is an optional parameter that, if defined, specifies the (fixed) header to be used for parsing. In such a case, the system will not look for a header row(s) in the input. headerRowsToRead is the number of header rows to be read, as sometimes it extends beyond one line. forgiving, which defaults to false, can be set to true if you expect that some rows will not parse, but where this will not invalidate your dataset as a whole. multiline is used to allow (or disallow when false) quoted strings to span multiple lines.

In forgiving mode, any exceptions thrown in the parsing of a row are collected and then logged. rowParser is the specific parser for the Row type (see below). builder is used by the parse method. parse is the main method of TableParser and takes an Iterator[Input] and yields a Try[Table]. You can also specify the number of header rows to be read, but this defaults to headerRowsToRead.

The predicate is used to filter rows (which are the results of parsing). By default, all rows are included. TableParser also provides a method (sampler) to create a random sampling function. Note, however, that a significant part of the time for building a table from a large file is just reading and parsing the file. Sampling will not reduce this portion of the time.

Associated with TableParser is an abstract class called TableParserHelper, whose purpose is to make your coding job easier. TableParserHelper is designed to be extended i.e., subclassed, by the companion object of the case class that you wish to parse from a row of your input. Doing it this way makes it easier for the implicit TableParser instance to be found. You can also set up your application along the lines of the examples below, such as the Movie example.

The constructor for TableParserHelper takes two parameters, both of which can be defaulted:

sourceHasHeaderRow: Boolean = true
forgiving: Boolean = false

RowParser

RowParser is a trait that defines how a line of text is to be parsed as a Row. Row is a parametric type that, in subtypes of RowParser, is context-bound to CellParser. A second parametric type Input is defined: this will take on values of String or Seq[String], according to the form of input. Typically, the StandardRowParser is used, which takes as its constructor parameter a LineParser.

The methods of RowParser are:

def parse(header: Header)(w: String): Try[Row]

def parseIndexed(header: Header)(indexedRow: (Input, Int)): Try[Row]

def parseHeader(w: String): Try[Header]

The parseIndexed method is useful when we care about the sequential aspect of the input. This is particularly important if strings are allowed to spread over newlines (as in the Airbnb dataset).

LineParser

The LineParser takes five parameters: two regexes, a String and two Chars. These define, respectively, the delimiter regex, the string regex, list enclosures, the list separator, and the quote character. Rather than invoke the constructor directly, it is easier to invoke the companion object's apply method, which takes a single using parameter: a RowConfig.

RowConfig is a case class of characters, from which the regexes are derived:

case class RowConfig(delimiter: Char = ',', quote: Char = '"', listSep: Char = '|', listEnclosure: String = "{}",
                     trim: Boolean = false, maybeString: Option[Regex] = None)

The default (RowConfig.default) reads RFC 4180. To read, for example, tab-separated values, define given RowConfig = RowConfig(delimiter = '\t') (or RowConfig.default withDelimiter '\t'): the regex for an unquoted field changes with the delimiter. If trim is true (as in RowConfig.trimmingRowConfig), spaces next to a delimiter are ignored, as in 1.x. rowConfig.csvAttributes gives the CsvAttributes which write tables in the same format (and csvAttributes.rowConfig the reverse). Two consecutive quote characters, within a quoted string, will be parsed as a single quote character. The LineParser constructor will perform some basic checks that its parameters are consistent.

StringsParser

StringsParser is a trait which defines an alternative mechanism for converting a line of text to a Row. As with the RowParser, Row is a parametric type which is context-bound to CellParser. StringsParser is useful when the individual columns have already been split into elements of a sequence. Typically, the StandardStringsParser is used.

The methods of StringsParser are:

def parse(ws: Seq[String])(header: Header): IO[Row]

def parseHeader(ws: Seq[String]): IO[Header]

CellParsers

There are a number of methods which return an instance of CellParser for various situations:

def rawRowCellParser: CellParser[RawRow]
def cellParserRepetition[P: CellParser](start: Int = 1): CellParser[Seq[P]]
def cellParserSeq[P: CellParser]: CellParser[Seq[P]]
def cellParserOption[P: CellParser]: CellParser[Option[P]]
lazy val cellParserOptionNonEmptyString: CellParser[Option[String]]
def cellParser[P: CellParser, T](construct: P => T): CellParser[T]
inline def cellParser2Conditional[K, P, T](construct: (K, P) => T, parsers: Map[K, CellParser[P]]): CellParser[T]
def columnHelper[T](maybePrefix: Option[String], aliases: (String, String)*): ColumnHelper[T]
etc. including other ways to instantiate a ColumnHelper[T].

The parser of a case class itself is derived: see CellParser.derived above (it replaces 1.x's cellParser1 ... cellParser13).

There is one additional method to handle the situation where you want to vary the parser for a set of cells according to the value in another (key) column: cellParser2Conditional. In this case, you must supply a Map, which specifies which parser is to be used for each possible value of the key column. If the value in that column is not one of the keys of the map, an exception will be thrown. For an example of this, please see the example in CellParsersSpec ("conditionally parse").

Implicits

Keep in mind when using given (implicit) values that the best practice is to define a given involving a type T, for example, CellParser[T], in the companion object of T. This will tend to eliminate any ambiguously defined implicits, and it also tends to avoid any problems with initialization. If you still run into initialization problems, try defining the troublemaker as lazy. It also relieves you from having to make up names for the implicit values (which the compiler more or less ignores, anyway). Just ensure that the name is valid, doesn't invoke a recursion, and is not in conflict with another name. If you look in the example of Principal (below), you will see that this is also the place to define optional parsers, sequential parsers, etc.

Content

The rows of a Table are represented by a case class called Content:

case class Content[+Row](private val xs: Vector[Row])

The rows are kept in the order they were read. (In 1.x, they were held in a ParIterable, which shuffled them.) For parallel processing, content.par gives the rows as a parallel collection.

Caveats

A case class which represents a row (or part of a row) of the table you want to create from parsing, or which you want to render must abide by certain rules:

There should not be any fields defined in the body of the case class. So, no val, var or lazy val. Instead, any behavior you want to add to the class, beyond the parameters (fields) of the class, must be defined using def.

Example: Movie

In this example, we parse the IMDB Movie dataset from Kaggle. The basic structure of the application code will look something like this:

    import Movie._
    val x: Try[Table[Movie]] = Table.parseResource[Table[Movie]]("movie_metadata.csv", classOf[Movie])

In this example, the row type is Movie, a case class with eleven parameters. The data can be found in a local resource (relative to this class) called movie_metadata.csv. All the (given) details that characterize this particular table input are provided in the Movie companion object.

The Movie class looks like this:

case class Movie(title: String, format: Format, production: Production, reviews: Reviews, director: Principal, actor1: Principal, actor2: Principal, actor3: Option[Principal], genres: AttributeSet, plotKeywords: AttributeSet, imdb: String)

Note that we make actor3 optional because some movies don't specify an "actor3".

In order to parse a Movie, we will need to declare some implicit values in the companion object. The following is the required code:

object Movie extends CellParsers with CsvGenerators with CsvRenderers {
    val missing: Movie = apply("", Format.none, Production.none, Reviews.none, Principal.nemo, Principal.nemo, Principal.nemo, None, AttributeSet.none, AttributeSet.none, "")
    val header = "color,director_name,num_critic_for_reviews,duration,director_facebook_likes,actor_3_facebook_likes,actor_2_name,actor_1_facebook_likes,gross,genres,actor_1_name,movie_title,num_voted_users,cast_total_facebook_likes,actor_3_name,facenumber_in_poster,plot_keywords,movie_imdb_link,num_user_for_reviews,language,country,content_rating,budget,title_year,actor_2_facebook_likes,imdb_score,aspect_ratio,movie_facebook_likes"
    implicit val helper: ColumnHelper[Movie] = columnHelper(camelToSnakeCaseColumnNameMapperLower,
        "title" -> "movie_title",
        "imdb" -> "movie_imdb_link")
    implicit val parser: CellParser[Movie] = CellParser.derived[Movie]
    implicit val renderer: CsvRenderer[Movie] = CsvRenderer.derived[Movie]
    implicit val generator: CsvGenerator[Movie] = CsvGenerator.derived[Movie]

    // Additional code shown below for parsing tables or processing rows.
}

We define a missing object because that is sometimes convenient to use with Spark. Similarly, the header object is the standard header strings, which can be used when reading a CSV file without a header. The (implicit) helper is used to map the names of columns appropriately. The (implicit) parser is the required CellParser for Movie. The (implicit) renderer is used to render a Movie as a CSV file. The (implicit) generator is used for outputting a Movie in other format(s).

Each of the case classes referenced in the declaration of Movie, will also need a similar companion object. For example, the Principal, whose case class is defined thus:

case class Principal(name: Name, facebookLikes: Int)

requires a companion object that looks like this:

object Principal extends CellParsers with CsvGenerators with CsvRenderers {
    val nemo: Principal = Principal(Name.nemo, 0)
    implicit val helper: ColumnHelper[Principal] = columnHelper(camelToSnakeCaseColumnNameMapperLower, Some("$x_$c"))
    implicit val parser: CellParser[Principal] = CellParser.derived[Principal]
    implicit val parserOpt: CellParser[Option[Principal]] = cellParserOption
    implicit val renderer: CsvRenderer[Principal] = CsvRenderer.derived[Principal]
    implicit val generator: CsvGenerator[Principal] = CsvGenerator.derived[Principal]
    // An optional Principal (e.g. actor3) has the same columns as a Principal: when it is missing, they are empty.
    implicit val rendererOpt: CsvRenderer[Option[Principal]] = optionProductRenderer[Principal]
    implicit val generatorOpt: CsvGenerator[Option[Principal]] = optionProductGenerator[Principal]
}

Like Movie, it has a default value (nemo), as well as a helper to get the column names correct. We need to define parsers, renderers, etc. for optional Principals. Unlike with primitive values such as Int, Double, we do have to add additional implicit definitions to accomplish this. Note that optionProductRenderer (and optionProductGenerator) are for an optional case class, which has several columns: None is written as the right number of empty cells. For an optional value with one column, use optionRenderer, which takes an optional parameter that defines a String to be used when the value is missing.

The other case classes look like this:

case class Format(color: String, language: String, aspectRatio: Double, duration: Int)
case class Production(country: String, budget: Option[Int], gross: Int, titleYear: Int)
case class Reviews(imdbScore: Double, facebookLikes: Int, contentRating: Rating, numUsersReview: Int, numUsersVoted: Int, numCriticReviews: Int, totalFacebookLikes: Int)
case class Name(first: String, middle: Option[String], last: String, suffix: Option[String])
case class Rating(code: String, age: Option[Int])

Consult the actual code in Movie.scala for the details of what is required in the corresponding companion objects.

The Movie object has additional code like this:

object Movie ... {
    // Required code as shown above

    // First, we need a StringParser[Movie]...
    implicit val movieConfig: RowConfig = RowConfig(listEnclosure = "")
    implicit val stringParser: StringParser[Movie] = StandardRowParser.create[Movie]

    // Next, if we want to be able to parse rows into a Table[Movie], we will need a TableParser[Movie]...
    trait MovieTableParser extends HeadedCSVTableParser[Movie] {
        override val forgiving: Boolean = true
    }
    implicit object MovieTableParser extends MovieTableParser
}

We use the forgiving mode for MovieTableParser because we expect that there will be many rows which cannot be parsed. In this code, helper, and the other columnHelpers, specify parameter-column mappings. Note that helper for Principal has an extra parameter at the start of the parameter list:

Some("$x_$c")

which is an (optional) formatter for the purpose of prefixing a string to column names. That's because there are several "Principal" parameters in a Movie, and each one has its own set of attributes. In this format parameter, "$x" is substituted by the prefix (the optional value passed into the lookup method) while $c represents the translated column name.

A couple of parameters of Movie are actually attribute sets (AttributeSet). These are basically lists of String within one column value. Such lists are parsed as lists as they are parsed from the original strings and then returned as strings in the form "{" element "," element ... "}" The parsing from the original string obeys the RowConfig parameters of listSep and listEnclosure.

In this case, the row config is defined as movieConfig, whose lists are not enclosed (in 1.x, they were enclosed in "{}" by default). A RowConfig also defines the delimiter character which separates the columns, and the quote character which can be used to include cell values which contain the delimiter.

A parameter can be optional, for example, in the Movie example, the Production class is defined thus:

case class Production(country: String, budget: Option[Int], gross: Int, title_year: Int)

In this example, some movies do not have a budget provided. All you have to do is declare it optional in the case class, and TableParser will specify it as Some(x) if valid, else None.

Note that there is a default given RowConfig (RowConfig.default) defined in the object RowConfig.

If, instead of building a Table[Movie], you prefer to process rows into an Iterator[Movie], then you should define the following instead of the MovieTableParser:

object Movie ... {
    // Required code as shown above...

    // Finally, an optional row processor--this is useful when you simply want to end up with an _Iterator[Movie]_
    // rather than a Table[Movie]. Typically, we will either use this OR MovieTableParser.
    trait MovieRowProcessor extends StringRowProcessor[Movie] {
        override val forgiving: Boolean = true
    }
    implicit object MovieRowProcessor extends MovieRowProcessor
}

Example: Submissions

This example has two variations on the earlier theme of the Movies example: (1) each row (a Submission) has an unknown number of Question parameters; (2) instead of reading each row from a single String, we read each row from a sequence of Strings, each corresponding to a cell.

The example comes from a report on the submissions to a Scala exam. Only one question is included in this example.

case class Submission(username: String, lastName: String, firstName: String, questions: Seq[Question])

object Submission extends CellParsers {
    implicit val submissionColumnHelper: ColumnHelper[Submission] = columnHelper(ColumnHelper.camelCaseColumnNameMapperSpace, Some("$c $x"))
    implicit val submissionParser: CellParser[Submission] = CellParser.derived[Submission]
    implicit val parser: StandardStringsParser[Submission] = StandardStringsParser[Submission]()

    implicit object TableParser extends StringsTableParser[Table[Submission]] {
      type Row = Submission

      protected def builder(rows: Iterable[Row], header: Header): Table[Row] = HeadedTable(rows, header)

      override val forgiving: Boolean = false

      val rowParser: RowParser[Row, Seq[String]] = summon[RowParser[Row, Seq[String]]]
    }
}

case class Question(questionId: String, question: String, answer: Option[String], possiblePoints: Int, autoScore: Option[Double], manualScore: Option[Double])

object Question extends CellParsers {
    private val mapper: String => String = _.replaceAll("(_)", " ")
    implicit val helper: ColumnHelper[Question] = columnHelper(mapper, Some("$c $x"), "questionId" -> "question_ID")
    implicit val optParserString: CellParser[Option[String]] = cellParserOption
    implicit val parser: CellParser[Question] = CellParser.derived[Question]
    implicit val seqParser: CellParser[Seq[Question]] = cellParserRepetition[Question]()
}

To test this example, we run a unit test as follows (using scalatest):

behavior of "TableParser"
it should "parse Submission" in {
    val rows: Seq[Seq[String]] = Seq(
      Seq("Username", "Last Name", "First Name", "Question ID 1", "Question 1", "Answer 1", "Possible Points 1", "Auto Score 1", "Manual Score 1"),
      Seq("001234567s", "Mr.", "Nobody", "Question ID 1", "The following are all good reasons to learn Scala -- except for one.", "Scala is the only functional language available on the Java Virtual Machine", "4", "4", "")
    )
    import Submission.TableParser
    matchTry(Table.parseSequence(rows.iterator)) {
      case rt@HeadedTable(_, _) =>
        println(rt.head)
        rt.size shouldBe 1
    }
}

Note the use of cellParserRepetition. The parameter allows the programmer to define the start value of the sequence number for the columns. In this case, we use the default value: 1 and so don't have to explicitly specify it. Also, note that the instance of ColumnHelper defined here has the formatter defined as "$c $x", which is in the opposite order from the Movie example.

Rendering

TableParser provides a general mechanism for rendering (serializing to text) tables. Indeed, the tables built by the parsers extend TableRenderable[Row], which supports the render(using ev: Renderer[Table[Row], O]) method. There are two distinct mechanisms for rendering a table:

one to a straight serialized output, for example, when rendering a table as a CSV file.
the other to a hierarchical (i.e., tree-structured) output, such as an HTML file.

Non-hierarchical output

For this type of output, the application programmer must provide an instance of Writer[O] which is, for example, a StringBuilder, BufferedOutput, or perhaps an I/O Monad.

The non-hierarchical output does not support the same customization of renderings as does the hierarchical output. It's intended more as a straight, quick-and-dirty output mechanism to a CSV file.

Here, for example, is an appropriate definition.

given StringBuilderWriteable: Writable[StringBuilder] with {
	def writeRaw(o: StringBuilder)(x: CharSequence): StringBuilder = o.append(x.toString)
	def unit: StringBuilder = new StringBuilder
	override def delimiter: CharSequence = "|"
}

The default delimiter is ", ". You can override the newline and quote methods too if you don't want the defaults.

And then, following this, you will write something like the following code:

print(table.render.toString)

The Writable object will take care of inserting the delimiter and quotes as appropriate. Columns will appear in the same order as the parameters of Row type (which must be either a Product, such as a case class, or an Array or a Seq). If you need to change the order of the rows, you will need to override the writeRow method of Writable.

Hierarchical rendering

A type class called TreeWriter is the main type for hierarchical rendering. One of the instance methods of Table[Row] is a method as follows:

def renderHierarchical[U: TreeWriter](style: String, attributes: Map[String, String] = Map())(using rr: HierarchicalRenderer[Row]): U

Providing that you have defined a given of type TreeWriter[U] and a HierarchicalRenderer[Row], then the renderHierarchical method will produce an instance of U which will be a tree containing all the rows of this table.

What sort of type is U? An XML node would be appropriate. The specifications use a type called HTML which is provided in package core.render.tag more as an exemplar rather than something definitive.

case class HTML(name: String, attributes: Seq[Attribute], content: Option[String], tags: Seq[Tag])

The example TreeWriter for this type (from HierarchicalRendererSpec) is reproduced here:

trait TreeWriterHTML extends TreeWriter[HTML] {
  def evaluate(node: Node): HTML = HTML(node.style, node.attributes.toSeq.map(kv => Attribute(kv)), node.content, node.children.toSeq map evaluate)
}

implicit object treeWriterHTML extends TreeWriterHTML

If we have a row type as, for example:

case class Complex(r: Double, i: Double)

Then, we should define appropriate renderers along the following lines:

implicit val valueRenderer: HierarchicalRenderer[Double] = renderer[Double]("td")
implicit val complexRenderer: HierarchicalRenderer[Complex] = HierarchicalRenderer.derived[Complex]("tr")

We can then write something like:

val table = HeadedTable(Seq(Complex(0, 1), Complex(-1, 0)), Header.create("r", "i"))
val h = table.renderHierarchical("table", Map("border" -> "1"))

The result of this will be an HTML tree that can be written out thus as a string:

 <table border="1">
 <tr>
 <td name="r">0.0</td>
 <td name="i">1.0</td></tr>
 <tr>
 <td name="r">-1.0</td>
 <td name="i">0.0</td></tr></table>

As with the parsing methods, the conversion between instances of types (especially case classes) and Strings is hierarchical (recursive).

If you need to set HTML attributes for a specific type, for example, a row in the above example, then an attribute map can be passed to HierarchicalRenderer.derived (its second parameter).

CSV Rendering

If you simply need to write a table to CSV (comma-separated value) format as a String, then use the toCSV method of Table[T]. Note that there is also an object method of Table called toCSVRow, which can be used for instances of Table[Row]. More control can be gained by using CsvTableStringRenderer[T] or CsvTableFileRenderer[T] for a particular type T.

These require customizable (implicit) evidence parameters and are defined as follows:

case class CsvTableStringRenderer[T]()(using CsvRenderer[T], CsvGenerator[T], CsvAttributes) extends CsvTableRenderer[T, StringBuilder] ...
case class CsvTableFileRenderer[T](path: Path)(using CsvRenderer[T], CsvGenerator[T], CsvAttributes) extends CsvTableRenderer[T, FileWriter] ...
abstract class CsvTableRenderer[T, O]()(using CsvRenderer[T], CsvGenerator[T], Writable[O], CsvAttributes) extends Renderer[Table[T], O] {...}

CsvRenderer[T] determines the layout of the rows, while CsvGenerator[T] determines the header. CsvAttributes specify the delimiter and quote characters for the output. For a case class, these are derived: CsvRenderer.derived[T] and CsvGenerator.derived[T], or derives CsvProduct for both at once (they replace 1.x's renderer1 ... renderer19 and generator1 ... generator19). The column names come from the same ColumnHelper that parsing uses, so a table that is written can be read back. Other instances can be created using methods in CsvRenderers and CsvGenerators respectively:

sequenceRenderer, optionRenderer, optionProductRenderer.
sequenceGenerator, optionGenerator, optionProductGenerator.

In some situations, you will want to omit values (and corresponding header columns) when outputting a CSV file. You may use the following methods (from the same types as above):

def skipRenderer[T](alignment: Int = 1)(using ca: CsvAttributes): CsvRenderer[T]
def skipGenerator[T](using ca: CsvAttributes): CsvGenerator[T]

Note that, when rendering a CSV row, you may want to simply render some number of delimiters (this would be in the case where you have a fixed header). You can use the alignment parameter of skipRenderer to ensure the alignment is correct.

As usual, the standard types are pre-defined for both CsvRenderer[T] and CsvGenerator[T] (for Int, Double, etc.).

The methods mentioned above render tables in the form of CSV Strings. However, there are also methods available to render tables as a File: writeCSVFile and writeCSVFileRow. These utilize the type CsvTableFileRenderer[T] mentioned above.

If you wish to output only a subset of rows, then you should use one of the methods defined in Table such as take.

Other String Rendering

Apart from CSV, there is currently only one implementation of String rendering, and that is Json rendering. Although JSON is indeed a hierarchical serialization format, the manner of creating a JSON string masks the hierarchical aspects.

JSON support is in its own module, tableparser-json, which uses circe, so that tableparser-core doesn't depend on a JSON library. (1.x used spray-json, in core.)

Although this section is concerned with rendering, it is also true, of course, to say that tables can be read from Json strings. TableCodecs provides a circe Encoder and Decoder for Table[T], given an Encoder (or Decoder) for T. A table is a JSON object with a "rows" array and, if the table has a header, a "header" array of column names.

The following example, from TableCodecsSpec.scala (in the json module), reads the movie database, writes it as JSON, and reads the JSON back. UnstructuredMovie (in core's tests) is a case class with a field for each of the 28 columns, many of them optional:

import com.phasmidsoftware.tableparser.json.render.JsonTableRenderer
import com.phasmidsoftware.tableparser.json.table.TableCodecs.given
import io.circe.Codec
import io.circe.parser.decode

given Codec.AsObject[UnstructuredMovie] = Codec.AsObject.derived

val movies: Table[UnstructuredMovie] = Table.parseResource[Table[UnstructuredMovie]]("movie_metadata.csv", classOf[UnstructuredMovie]).get
val json: String = new JsonTableRenderer[UnstructuredMovie].render(movies)
val decoded: Either[io.circe.Error, Table[UnstructuredMovie]] = decode[Table[UnstructuredMovie]](json)
decoded.map(_.toList) shouldBe Right(movies.toList)

The JSON looks like this (abbreviated):

{
  "rows" : [
    {
      "color" : "Color",
      "directorName" : "James Cameron",
      "numCriticForReviews" : 723,
      ...
      "gross" : 760505847,
      ...
    },
    ...
  ],
  "header" : [
    "color",
    "director_name",
    "num_critic_for_reviews",
    ...
  ]
}

Note that the header and the keys of the rows differ. The header is the table's own header: here, the column names of the CSV file. The keys of each row are whatever its Encoder writes: for a derived Codec, the field names. A missing (None) value is written as null (and a missing or null value is read as None). Reading JSON which is malformed, or which doesn't match the row type, gives a Left. For more examples (numbers, dates, nested case classes, and bad input), see TableCodecsSpec.scala and JsonTableRendererSpec.scala.

Release Notes

V2.0.3 -> V2.1.0

Streaming (see Streaming): the rows of a table can be read as they are parsed, rather than all at once:
- cats: IOTable.stream (and streamFile, streamResource, streamLines, streamURI) give an fs2 Stream[IO, Row]; the pipes csvLines and writeCSVFile write one. The cats module now depends on fs2 (fs2-core, fs2-io).
- zio: the same methods give a ZStream; csvLines is a ZPipeline and writeCSVFile a ZSink. The zio module now depends on zio-streams.
- core: StringTableParser.rows parses lazily (the header, then an Iterator of rows).
RFC 7111 (see Selecting rows, columns and cells): fragment identifiers such as data.csv#row=5-7. Table.parse of a URI (and streamURI) selects the rows of its fragment; CsvFragment parses fragments, and selects rows (from any table), or columns and cells (from raw tables). Up to 2.0.3, a URI with a fragment could not be parsed at all.

V2.0.2 -> V2.0.3

Bug fix: Table.writeCSVFile (and so Table.writeCSVFileRow) wrote an empty file: it never closed the file, so what it wrote was never flushed. It also ignored any failure. It now closes the file, and throws a TableException on failure.
Encryption: new TableCrypt.writeCSVFileEncryptedIO, which writes an encrypted table in IO (the rows are encrypted in parallel, and the file is always closed). writeCSVFileEncrypted runs it. Every line now ends with the CsvAttributes newline (the rows used to end with LF, even though the header ended with CRLF).
Tests: none is ignored any more (each of the five was fixed, or rewritten to test what is right); the zio tests which use matchZIO now run (it returned a Task which was never run, so about 35 of them checked nothing). 573 tests (and 20 integration tests); statement coverage is 90.4% (branch coverage 84.9%).

V2.0.1 -> V2.0.2

Encrypted tables (the crypto module) can be written and read again (both had been broken since the modules were split):
- TableCrypt.writeCSVFileEncrypted takes a key function (from the key of a row to its encryption key), as EncryptedHeadedStringTableParser does, so the same key map writes and reads. It returns a Try[Unit]: up to 2.0.1, it failed without saying so, and wrote nothing.
- Table.parse (and parseFile, parseResource, etc.) decrypts with an EncryptedHeadedStringTableParser: up to 2.0.1, it gave the encrypted rows. Its parseIO no longer drops the first encrypted row.
- New: HexEncryption.encryptHex (the inverse of decryptHex) and WritableCrypt.fileWritableCrypt.
A table whose rows are _List_s (or other _Seq_s) is rendered correctly: a row was written as, e.g., "1,List(2)".
LineParser's construction check finds the delimiter of RowConfig's regexes (it took it to be "Q" or "s"), and no longer makes the constructor fail for a delimiter which has no single character.
The code uses Scala 3's indentation syntax (a layout-only change).
Tests: 542 (and 18 integration tests); statement coverage is 87.5% (branch coverage 81.3%).

V2.0.0 -> V2.0.1

The cats and zio modules now behave as effects should:
- IOTable does its work when the IO is run (and each time it is run), not when the IO is made: writeCSVFileIO, for example, wrote the file even if the IO was never run.
- ZIOTable closes the files (and streams) that it reads: it used to leave them open.
- ZIOTable reports a file that can't be opened as a failure (which catchAll etc. can handle), not as a defect.
- Reading and writing files runs on the blocking thread pool (IO.blocking, ZIO.attemptBlocking).
- ZIOTable now has toCSV, writeCSVFileIO and writeCSVFileRow, and both modules have parseSequence (they were commented out).

V1.7.x -> V2.0.0

Scala 3 only (built with Scala 3.3 LTS; tested on 3.9 too), and Java 21. For Scala 2.13, Java 17 or Spark, use the 1.x line. See Migrating from 1.x for what changes in your code.
Derivation: the parsers and renderers of case classes (CellParser, CsvRenderer, CsvGenerator, HierarchicalRenderer, and parquet's ParquetCellConverter) are derived from the case class's Mirror. This replaces the cellParserN, rendererN, generatorN and converterN families, removes their limits on the number of fields, and replaces reflection (Reflection is gone).
RFC 4180: CSV is read and written as the RFC defines it, by default (see the top of this file).
Round trip: a table that is written can be read back: the column names written are those that the parser looks up.
RowConfig is a case class, whose regexes are derived from its delimiter and quote, and which converts to and from CsvAttributes (issues #48, #38, #32). Trimming (RowConfig.trimmingRowConfig) now ignores spaces before a delimiter as well as after it.
Content keeps the rows in order (it is a Vector; par gives a parallel collection).
New modules: tableparser-crypto (encryption, from cats) and tableparser-json (circe, replacing spray-json in core).
java.time instead of Joda-Time (nscala-time is no longer a dependency).
Scala 3 idioms throughout: given/using, extension methods, enums, top-level definitions, infix methods.
Header (issue #12): the leftovers of an old approach to letter names are gone, and ++ joins extra header lines pairwise.
Bug fixes:
- Table.parse(uri, enc) and parseInputStream(..., enc) (and the cats and zio versions) ignored the encoding.
- parseRaw (and parseFileRaw, etc.), given a fixed header, dropped the first row of data.
- IOTable and ZIOTable's parse (of strings or a Source) failed for a parser which reads its header from the input.
- IOTable.toCSVRow and writeCSVFileRow threw, rather than failing in IO, for a table with no header.
- HeadedStringTableParser ignored its fixed header.
- hexStringToBytes lost leading zero bytes, so about 1 in 256 encrypted rows couldn't be decrypted (also fixed in 1.7.2).
A blank Option[URL] cell is now None (it was a failure).
Tests: about 500, now including every method of IOTable and ZIOTable; statement coverage is 82%.

V1.7.1 -> V1.7.2

Bug fix, back-ported from 2.0: decrypting a Hex string failed whenever the ciphertext began with a zero byte (HexEncryption.hexStringToBytes dropped leading zero bytes), so about 1 in 256 encrypted rows couldn't be decrypted.

V1.7.0 -> V1.7.1

Bug fixes, back-ported from 2.0:
- When writing CSV, a String (or a column name) is quoted when it has to be, as in RFC 4180: when it contains the delimiter, a quote or a line break, or starts or ends with whitespace (quotes inside it are doubled). Up to 1.7.0, a String containing a comma or a quote was written as a broken row.
- In multiline mode, the line break inside a quoted field is kept (it used to be dropped).
- A column whose name matches exactly is preferred to one which differs only in case.
New: CsvRenderers.optionProductRenderer and CsvGenerators.optionProductGenerator, for an optional field with several columns (e.g. a case class): None is written as the right number of empty cells. V1.6.0 -> V1.7.0
The last release planned for Scala 2.13. 2.0.0 will be for Scala 3 only (see the plan); the spark module stays on the 1.x line.
Logging: the library no longer depends on logback or ships a logback.xml. It logs through SLF4J, so add the SLF4J backend of your choice (for example, logback-classic) to your own project if you want to see its logs.
Deprecated, to leave the library in 2.0.0: the examples (Movie and the crime example), table.Main, Sequential (and Sequence, NonSequential, OrderingWithZero), and the zio module's EvaluateZIO.
The zio module no longer depends on zio-http, which it didn't use.
Dependency updates: Scala 2.13.18, cats-effect 3.7.1 (with tsec 0.6.0), zio 2.1.27, Parquet 1.18.1, Spark 4.0.4, and others.
The integration tests (against full-size data files) run again, separately: sbt +integration/test. V1.5.2 -> V1.6.0
Cross-built for Scala 2.13 and Scala 3 (except for the spark module, which is 2.13 only). No API changes, except that a few context bounds became explicit implicit parameter lists (so that callers may pass them explicitly on Scala 3.6+) and Main (in core/table/Analysis.scala) no longer extends App.
The unused dependency on flog has been removed. V1.5.1 -> V1.5.2 (not published separately: its changes are first released in 1.6.0)
Merged the old Indexed branch (2023) with 1.5.1.
New Validity trait, and Table.filterValid to keep only the rows that are valid.
New Table.sample and Content.sample to take a random sample of rows.
New Sequential module (Sequential, Sequence, OrderingWithZero) for tables whose rows carry a sequence number; it is not yet used elsewhere in the library.
The crime example was reworked (Crime and CrimeLocation). V1.5.0 -> V1.5.1
Removed debug print statements. V1.4.1 -> V1.5.0 Implemented column-grouping of flat Parquet files (13. Grouped Case Class Mapping) See TableParserParquetDesign V1.4.0 -> V1.4.1
Refactoring, internal improvements such as extending csv renderers to 19-tuples. V1.3.0 -> V1.4.0
New Analyzer trait with deprecated methods for the old Analysis class and object.
The old Analysis class has been renamed ColumnStatistics. V1.2.6 -> V1.3.0
Now supports parsing of Parquet files; Path is the primary type for accessing files. V1.2.5 -> V1.2.6
Non-functional change to TableParser. The only difference is that CircleCI is not using Java 17 V1.2.3 -> V1.2.5
Dependency updates, e.g., use Spark 4.0.1 (not 4.1.1) and Spark is now "provided." V1.2.1 -> V1.2.3
Minor changes, mostly in the build.sbt file. V1.2.0 -> V1.2.1
Added support for ZIO. V1.1.4 -> V1.2.0
Significant changes including the completion of the split into three packages with...
Functioning spark package.
Now supports Iterator to Iterator processing.

V1.1.3 -> V1.1.4

Split into three modules: core, cats and spark.
Minor changes functionally speaking.

V1.1.2 -> V1.1.3

Use of Cats IO
Table contents are now parallelized
Option of having sequential rows of user type
Improved Analysis by allowing Histogram

V1.1.1 -> V1.1.2

Make RawRow a type (not just a type alias)

V1.1.0 -> V1.1.1

Enable cryptographic capabilities
Uses TSEC-JCA and Cats IO.
Many relatively minor fixes/improvements.

V1.0.15 -> V1.1.0

Enable CSV-rendering and selection of table rows.

V1.0.14 -> V1.0.15

Minor changes

V1.0.13 -> V1.0.14

Enabled multi-line quoted strings: if a quoted string spans more than one line, this is acceptable.
Implemented analysis of raw-row tables.
Implemented sampling of input.
Provided a new mechanism for configuring and using parsers (see the worksheets).
Implemented Table.parseResourceRaw and Table.parseFileRaw for those situations where you just want to parse an input file into a Table[Seq[String]].

V1.0.12 -> V1.0.13

mostly concerned with publishing TableParser in Maven Central

V1.0.11 -> V1.0.12

mostly internal refactoring: restored Renderable interface (though different from before)

V1.0.10 -> V1.0.11

introduction of logging;
introduction of JSON (spray) for read/write of Table;
Table now supports Iterable=>Iterable methods.
renaming of Renderer to HierarchicalRenderer and introduction of StringRenderer
introduction of TableParserHelper;
renamed TableWithoutHeader as UnheadedTable and TableWithHeader as HeadedTable;
added various methods, inc. replaceHeader, to Table.
Table parsing is now based on Iterator rather than Iterable.
Table rows are now based on Vector (at least for the standard TableWithHeader)

V1.0.9 -> V1.0.10

build.sbt: changed scalaVersion to 2.13.3
added StringTableParserWithHeader;
now column names are found by case-independent comparison.

V1.0.8 -> V1.0.9

build.sbt: changed scalaVersion to 2.12.10

V1.0.7 -> V1.0.8

build.sbt: changed scalaVersion to 2.12.9
refactored the concept of tables with/without headers in TableParser;
enabled program-defined headers that match Excel-style numbers or letters.

V1.0.6 -> V1.0.7

build.sbt: changed scalaVersion to 2.12.8
CellParser: parametric type T is now covariant;
CellParsers: added new method cellParserOptionNonEmptyString; then for each of the cellParserN methods, the signature has had a defaultable fields parameter to allow explicit field naming;
Reflection: changed the message to refer to the cellParserN signatures;
README.md: fixed some issues with the doc regarding the MovieTableParser; added new features above.

V1.0.5 -> V1.0.6

Added a standard implicit value of ColumnHelper for situations that don't need extra help.

V1.0.4 -> V1.0.5

Added a convenient way of rendering a table as a non-hierarchical structure. In other words, serialization to a CSV file.

V1.0.3 -> V1.0.4

Added the ability to add header row and header column for tables (NOTE: not finalized yet, but functional).

V1.0.2 -> V1.0.3

Added no implicit warnings
Created mechanism for rendering the result of parsing in a hierarchical structure.

V1.0.1 -> V1.0.2

Added self-checking of LineParser;
Able to parse two quote-chars together in a quotation as one quote char;
Added enc and codec params as appropriate to Table.parse methods.
Added stringCellParser;
Now, properly closes source in Table.parse methods.

V1.0.0 -> V.1.0.1

Fixed Issue #1;
Added parsing of Seq[Seq[String]];
Added cellParserRepetition;
Implemented closing of Source in Table.parse methods;
Added encoding parameters to Table.parse methods.

Name		Name	Last commit message	Last commit date
.circleci		.circleci
cats/src		cats/src
core/src		core/src
crypto/src		crypto/src
docs		docs
json/src		json/src
parquet/src		parquet/src
project		project
tmp		tmp
zio/src		zio/src
.gitignore		.gitignore
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md
build.sbt		build.sbt
info6205.spring2023.teamproject.csv		info6205.spring2023.teamproject.csv
publish.sbt		publish.sbt
sonar-project.properties		sonar-project.properties
yellowtaxitrip.csv		yellowtaxitrip.csv

Navigation Menu

Repository navigation

About

Repository files navigation

Introduction to TableParser

Package Structure

Quick Intro

User Guide

Migrating from 1.x

Parsing

Parsing Row-oriented Tables

Parsing Column-oriented Tables

Parquet parsing

Table

query methods

transformation methods

Streaming

Selecting rows, columns and cells (RFC 7111)

TableParser

RowParser

LineParser

StringsParser

CellParsers

Implicits

Content

Caveats

Example: Movie

Example: Submissions

Rendering

Non-hierarchical output

Hierarchical rendering

CSV Rendering

Other String Rendering

Release Notes

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Introduction to TableParser

Package Structure

Quick Intro

User Guide

Migrating from 1.x

Parsing

Parsing Row-oriented Tables

Parsing Column-oriented Tables

Parquet parsing

Table

query methods

transformation methods

Streaming

Selecting rows, columns and cells (RFC 7111)

TableParser

RowParser

LineParser

StringsParser

CellParsers

Implicits

Content

Caveats

Example: Movie

Example: Submissions

Rendering

Non-hierarchical output

Hierarchical rendering

CSV Rendering

Other String Rendering

Release Notes

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages