Conceptual Overview

Basic terminology

Graphics engine is the part of an application which is responsible for the GPU graphics. LambdaCube 3D is a DSL for implementing graphics engines.

From a purely functional point of view, the graphics engine is a pure function. We call this function the pipeline.

A frame is the output of the pipeline. A frame is something which can be put on the screen.

Rendering is the computation of one frame given the pipeline and its input.

Pipelines can be assembled from smaller parts.

Multi-pass rendering is rendering with a pipeline which is a composition of several pipelines where the output of a pipeline is part of the input of another pipeline. LambdaCube 3D supports multi-pass rendering.

A dynamic pipeline is a composed pipeline in which the composition depends on the pipeline input.
Currently LambdaCube 3D does not support dynamic pipelines. This limitation is planned to be lifted. Until then a workaround is to use several pipelines and the application logic chooses at every rendering the right pipeline depending on the input. (If a predefined set of pipelines is not enough, it is possible to create pipelines on the fly.)

Pipeline inputs

There are limitations on possible pipeline inputs, which reflects the fact that GPUs are specialised hardware. The current limitations are inherited from OpenGL 3.3.

The input of a pipeline consists of zero or more vertex streams, zero or more uniforms and zero or more textures.

A uniform is a piece of data with a statically known size. Supported uniform types are Bool, Int, Float, Vec 3 Bool, Mat 4 4 Float, etc. For example, a camera position can be part of the input as a 4 dimensional Float matrix uniform. The terminology ‘uniform’ comes from the fact that uniforms are constant values during the rendering.

A texture is a uniform with type Texture. A Texture is an image with sampling configurations (discussed later). Currently textures are treated specially in LambdaCube 3D but this is planned to be changed.

A stream is a vector. In LambdaCube 3D there are two kind of streams: primitive streams and fragment streams.

A primitive is either a point, a line, or a triangle.

A point is a vertex.
A line is a pair of vertices.
A triangle is a triplet of vertices.

A vertex is a tuple of attributes.

An attribute is a piece of data with a statically known size. Supported attribute types are for example Float, Bool or Vec 3 Float.
Typical attributes are the spatial position and the color of a vertex.

For example, the primitive stream of a tetrahedron with one position attribute looks something like

Triangle (V3 0 0 0) (V3 1 0 0) (V3 0 1 0)   -- 1st triangle
Triangle (V3 1 0 0) (V3 0 1 0) (V3 0 0 1)   -- 2nd triangle
Triangle (V3 0 1 0) (V3 0 0 1) (V3 0 0 0)   -- 3rd triangle
Triangle (V3 0 0 1) (V3 0 0 0) (V3 1 0 0)   -- 4th triangle

The primitive stream of a tetrahedron with a position and an additional Boolean attribute looks something like

Triangle (V3 0 0 0, True ) (V3 1 0 0, True ) (V3 0 1 0, True )   -- 1st triangle
Triangle (V3 1 0 0, True ) (V3 0 1 0, True ) (V3 0 0 1, False)   -- 2nd triangle
Triangle (V3 0 1 0, False) (V3 0 0 1, True ) (V3 0 0 0, True )   -- 3rd triangle
Triangle (V3 0 0 1, False) (V3 0 0 0, True ) (V3 1 0 0, True )   -- 4th triangle

Note that the representation of streams and primitives are hidden. Usually the primitive stream is stored as a flattened stream of vertices by the application.

tetrahedron = map (\v -> V4 v%x v%y v%z 1)
    [ V3 0 0 0, V3 1 0 0, V3 0 1 0
    , V3 1 0 0, V3 0 1 0, V3 0 0 1
    , V3 0 1 0, V3 0 0 1, V3 0 0 0
    , V3 0 0 1, V3 0 0 0, V3 1 0 0
    ]

tetrahedronStream = fetchArrays @Triangle tetrahedron

Frames and images

A frame consists of several images of the same dimension.

An image is a two dimensional array of values.

There are different kind of images depending on the role the image plays.

A color image is to store color values. Supported color values are Float, and Float vectors with dimension at most 4. For example, a color image of Float values can be used to store a grayscale image, but the Float values can be interpreted in any color space, or they can be used to a totally different purpose too.

The emptyColorImage function creates an empty color image given a color value:

emptyColorImage navy

A depth image is to store depth values. A depth value has type Float and it is usually used to store the distance between the viewpoint and the object shown on a color image. In LambdaCube 3D, a depth image has different type than a color image with Float values because graphics hardware has dedicated operations for depth images.

The emptyDepthImage function creates an empty depth image given a depth value:

emptyDepthImage 1

A stencil image is to used for masking (discussed later).

Creating frames from images

The imageFrame function creates a frame given a tuple of images:

imageFrame (emptyDepthImage 1, emptyColorImage navy)

There are limitations about how many images a frame can contain. A frame can contain at most one depth image and at most one stencil image.

Pipelines

Ideally, a pipeline is an arbitrary computable function which produces a frame given an input. This is not the case. There are limitations on possible pipelines, which reflects the fact that GPUs are specialised hardware. The current limitations are inherited from OpenGL 3.3.

Here is a step-by-step overview about how to construct pipelines.

Constant frames

The simplest pipeline has no input and outputs a constant frame:

makeFrame = imageFrame (emptyDepthImage 1, emptyColorImage navy)

From primitive streams to frames

A frame can be constructed from a vertex stream in several phases. We go step-by-step through the phases. Again, if these steps seem ad-hoc, think about the specialised hardware. (We believe that some of these limitiation can be lifted by the compiler and the libraries. We plan to gradually improve the current interface to be more and more higher-level.)

Primitive stream transformations

Usually the first phase is the transfromation of the vertices in the primitive stream. Typically the vertex coordinates are projected onto the viewport and the depth (the distance of the vertex from the viewer) is calculated. If we use homogeneous coordinates, the projection and the depth can be calculated at the same time by multiplying the vertex with a projection matrix.

The vertices usually have other attributes next to the coordinates, and these attributes can be transformed also in the first phase. The restriction is that at least the vertex coordinates used in the rasterization phase should be produced.

The mapPrimitives function transforms a primitive stream into another primitive stream.

The following example we project and scale the vertices, and we also store the original coordinates of the vertices, because we would like to use them later:

mapPrimitives (\((x)) -> (scale 0.5 (projmat *. x), x))

Here we suppose that the vertices are Vec 4 Float values and projmat has type Mat 4 4 Float. The ((x)) is the special syntax for tuples with one element.

The projmat is embodies the camera and prespective projection transformations. It is a sequential composition of simpler transformations, using rotMatrixY, lookat, perspective functions from prelude.

    projmat = perspective 0.1 100.0 (30 * pi / 180) 1.0
          .*. lookat (V3 3.0 1.3 0.3) (V3 0.0 0.0 0.0) (V3 0.0 1.0 0.0)
          .*. rotMatrixY (pi / 24.0 * time)

Rasterization

In the rasterization phase, each primitive is turned into a set of disjunct unshaded (not-yet-colored) fragments. A fragment has viewport coordinates and custom attributes. It is similar to a pixel.

The rasterization step is not programmable, but configrable by a rasterization context.

An example rasterization context for rasterizing triangles:

TriangleCtx CullNone PolygonFill NoOffset LastVertex

Some explanation (do not mind if you don’t understands these now):

CullNone tells that both side of the triangles are visible.
PolygonFill tells that not only the edges of the trianges are visible.
NoOffset tells that rasterization will not modify depth values.
LastVertex tells that in case of no interpolation which vertex attribute to use in fragments.

The rasterizePrimitives creates a functions primitive stream to a fragment stream, given a rasterization context and a tuple of Smooth and Flat values:

rasterizePrimitives (TriangleCtx CullNone PolygonFill NoOffset LastVertex) ((Smooth))

Here the tuple contains only one Smooth value, so we use the special syntax. The rule is that the tuple size should be equal to the number of vertex attributes minus the position attribute used by the rasterisation phase. Each element of this tuple tells how to interpolate the individual attribute values.

Filtering fragment streams

It is possible to filter fragment streams. Filtering is used for example to make fragments look transparent. If we do not want filtering, we can write

filterFragments (\_ -> True)

or we can skip this phase.

Shading

In the shading phase we transform the fragment stream usually to give color to fragments.

This phase is programmable, the program is given by the mapFragments function:

mapFragments (\_ -> ((V4 0 1 0 1))) -- RGBA
mapFragments (\_ -> ((green)))

We can paint the fragment green or some time varying color:

mapFragments (\((x)) -> (( (rotMatrixZ time *. rotMatrixY time *. x) *! f time )) )

f is here a helper function:

f x = (x + sin x + sin (1.1 * x)) `mod` 4 * 2

Accumulation

In the accumulation phase sets of overlapping fragments are accumulated into pixels. Accumulation means that we combine the color values of overlapping fragments to get the color of the pixel.

Accumulation is not programmable, but configurable by an accumulation context. The accumulation context tells how to combine previous values on the frame with new incoming values.

An example accumulation context:

(DepthOp Less True, ColorOp NoBlending (V4 True True True True))

DepthOp tells how to accumulate depth values.
Less tells that depth test succeed if the fragment’s depth values is less (it is closer).
True tells that the fragments are opaque, and should we written on the frame if the depth test succeeds.
ColorOp tells how to accumulate (the first) color values.
NoBlending tells that we don’t want to blend (mix) values, the new value is written.
(V4 True True True True) gives the write masks for individual color channels.

We can pair a shaded fragment stream with an accumulation context with accumulateWith:

accumulateWith (DepthOp Less True, ColorOp NoBlending (V4 True True True True))

The actual accumulation is done with overlay (used as operator here):

...
              imageFrame (emptyDepthImage 1, emptyColorImage navy)
    `overlay` accumulablefragments
  where
    accumulablefragments
        = vertexstream
        & mapPrimitives (\((x)) -> (scale 0.5 (projmat *. x), x))
        & rasterizePrimitives (TriangleCtx CullNone PolygonFill NoOffset LastVertex) ((Smooth))
        & filterFragments (\((x)) -> True)
        & mapFragments (\((x)) -> (((rotMatrixZ time *. rotMatrixY time *. x) *! f time)))
        & accumulateWith (DepthOp Less True, ColorOp NoBlending (V4 True True True True))

Here (&) is the flipped function application, and vertexstream is a vertex stream of (Vec 4 Float) attributes.

Putting all together

The pure function which creates the final frame, given the time, the projection matrix and a triangle vertex stream:

f x = (x + sin x + sin (1.1 * x)) `mod` 4 * 2

makeFrame (time :: Float)
          (vertexstream :: PrimitiveStream Triangle ((Vec 4 Float)))

    =         imageFrame (emptyDepthImage 1, emptyColorImage navy)
    `overlay` accumulablefragments
  where
    projmat = perspective 0.1 100.0 (30 * pi / 180) 1.0
          .*. lookat (V3 3.0 1.3 0.3) (V3 0.0 0.0 0.0) (V3 0.0 1.0 0.0)
          .*. rotMatrixY (pi / 24.0 * time)

    accumulablefragments
        = vertexstream
        & mapPrimitives (\((x)) -> (scale 0.5 (projmat *. x), x))
        & rasterizePrimitives (TriangleCtx CullNone PolygonFill NoOffset LastVertex) ((Smooth))
        & filterFragments (\((x)) -> True)
        & mapFragments (\((x)) -> (((rotMatrixZ time *. rotMatrixY time *. x) *! f time)))
        & accumulateWith (DepthOp Less True, ColorOp NoBlending (V4 True True True True))

Interfacing pipelines with the outer world

We can connect the inputs of the previous pipeline to our application by naming the input uniforms and input streams. The renderFrame function turns a frame into an action of producing it on the GPU.

main = renderFrame $
   makeFrame (Uniform "Time")
             (fetch "stream4" ((Attribute "position4")))

Or we can use our tetrahedron, which we defined before.

tetrahedron = map (\v -> V4 v%x v%y v%z 1)
    [ V3 0 0 0, V3 1 0 0, V3 0 1 0
    , V3 1 0 0, V3 0 1 0, V3 0 0 1
    , V3 0 1 0, V3 0 0 1, V3 0 0 0
    , V3 0 0 1, V3 0 0 0, V3 1 0 0
    ]

tetrahedronStream = fetchArrays Triangle ((tetrahedron))

main = renderFrame $
   makeFrame (Uniform "Time")
             tetrahedronStream