Tui front-end framework

Introduction

Tui is our front-end framework. First introduced in Totara 13 it brings an entirely new way of producing a user experience for our products, and with it many new technologies and ways of developing.

For the release of Totara 13, Totara Learn's front end remains largely as it was in Totara 12, we have not converted much of the user experience. We have applied Totara 13's new Ventura theme look and feel to Learn, via the Legacy theme, but the underlying technology stack for those user interfaces (UIs) remains in place. 

Use of Tui within Totara 13 is entirely optional, due to our complete separation of client-side logic and server-side logic. The existing methods of producing a user experience for Totara Learn all continue to operate exactly as they did previously.

Totara Perform and Totara Engage however use Tui consistently for nearly their entire user experience (UX), only using the old methods when they were unavoidable.

Background

Understanding the problems of our previous approach

The following problem statements capture recurring themes in what we hear from Totara Partners, their clients, and our own team members' experiences. We can refer to these following problem statements as they help to maintain focus on underlying problems we are solving.

“User navigation through our system component pages is unintuitive and complex”
“User interfaces in many locations throughout the platform are inconsistent”
“At scale, users are presented with a lot of information that is not highly relevant”
“Supporting legacy browsers either limits or increases effort with adoption of new technology options”
“Modern User Interface trends are lacking or take a long time to enter the product”
“Custom theming can be time-consuming and requires less-than-ideal approaches to customisation”
“There are examples of duplicated component and page layout implementations throughout the codebase”
“It has always been difficult removing front-end dependencies”
“It is increasingly difficult to employ front-end developers to work with our technology stack”

In 2017 the Design team put out a quick survey for some Totara Partners' front-end developers, to comment on what they felt our theme shortcomings were. The results were short and sharp from 15 respondents:

• Consistent markup required
• Simpler approach to CSS overrides required
• Overriding rendered page elements desirable

In 2019 we did a more comprehensive follow-up survey, the results were similar but with more attached insights from 52 respondents:

• Consistent markup required
• Simpler approach to CSS overrides required
• More/better documentation required
• Acknowledgement that a new framework would be beneficial (while Bootstrap is also preference for some)
• Customisations are often done because of a lack of consistently applied appearance settings in the UI itself.

The intention is to continue on this trend and continue to ask questions that help us to understand our partner theming pain points. As part of Totara's ongoing commitment to improving our product UX, our CX team have, and will continue to pitch browser support questions at partners, so that we can find a balance between realistic browser support and innovation.

Focusing on a target

Our aim is to modernise our product suite's front end.

To deliver an experience for our users that removes friction from their need to uptake content and interact, and their admins’ needs to create, maintain and report. It also needs to pave the way for our partners to build and maintain on top of a technology offering that is fit-for-purpose.

Being able to adapt our products to meet changing business needs is important, from offering modern learning experiences to creating whole new products, and everything in between.

Over time, as a result of our origins, we have built our product up from an extensive set of needs. Sometimes the features we've created are very specific, sometimes they're widely used and sometimes they're questionable as to their existence.

Because of our origins and our need to adapt, we're responding by taking a hard look at our 'front end'. We use quotes there because the front end includes potentially any part of our products that a human interacts with, rather than a collection of files with code inside.

Defining fit for purpose

Being able to implement innovations that will serve us well in the future, without becoming (or adding to) burdensome maintenance. To be able to effectively handle feature requests which are increasingly complicated in their requirements-driven design and complex in their construction.

Listening to our partners, we've heard that our products need greater visual consistency, easier and more intuitive user flows, extensibility and consistency in our technical components, and preferably delivered immediately.

We can do this by looking at our front end from three directions:

1. Design - ensuring that our growing suite of products are cohesive, user-centric and modern in how they look and behave.
2. Development - create a robust and consistent set of base components that can be put together quickly and reliably into full-featured user interfaces.
3. Awareness - engage with our community to help them better understand our thinking, direction and ultimately how they can adopt or extend our product.

Goals

There are three immediate goals, relating to the directions set out above.

Iterate our 'design library'

• Converting our incomplete set of design components in a static Sketch file (current state) into a small series of public docs that articulate our approach to visual and UX design and an enhanced catalogue of implemented components directly within our core platform. Our 'design library' evolves into a 'design system' worthy of an enterprise platform.
• Commit to constant engagement with our design system. Refer to it during workflow and discussion, iterate it where it falls short or requires adaptation. Challenge new concepts where they can either be something that exists and is familiar to users, or introduce new concepts holistically.
• Capture design best practice in our design system, so that we can articulate it across our mixed-discipline teams, and to our partners. This also introduces efficiencies at some levels when working through new features or improvements.
• Deliver a new modern theme for Engage and Perform, and Learn where possible and appropriate.

Rebuild our component architecture

• Updating our technology stack. Technology has evolved significantly over the last ten years during Totara's growth. There's a lot going on in the open source world we can leverage, and our users are increasingly exposed to technically advanced features in other platforms. We need to implement features on a technical level that matches our maturing design level.
• Revise and streamline how partner themers customise our product. The theming process itself has become cumbersome, making use of modern development workflows and tools has some clear advantages over our existing offering.
• Support decoupling of how we send and receive data from the server with our new GraphQL approach, and rendering page elements that consume and change that data.

Support the Academy and Marketing teams

• Supply latest mock-ups, prototypes and information as early as possible to ensure timely delivery of marketing and learning materials for our partners and the community.
• Leverage the in-house experience of the Academy team to test our own internal innovations, as well as listening to our partner requirements.
• Identify high-impact improvements or features and pass the details to our Marketing and Academy teams so they can shine a spotlight on our collective efforts. We should be proud of the work we do to achieve our goals, because making software is hard.

Licensing

Tui is provided under a proprietary licence and has been fully separated and isolated from the Totara Core product to ensure its separation from the GPL licence used within Totara Core.

For more information on these changes in Totara 13 please see our Totara 13 front-end architecture, licensing and distribution changes document as well as the Totara 13 code reorganisation developer documentation.

When working with Tui all integrations with Totara Core must be through the existing mechanisms. Tui can not intimately integrate with Totara Core otherwise it would contravene the GPL. It is is a standalone front-end framework that has been integrated with Totara Core through channels that any other front-end framework could use.

If including third-party libraries and code within a Tui bundle then it is important that the licence applied to that code is compatible for use when bundled and distributed by with code using our proprietary licence.

All Totara work on Tui will be licensed using our proprietary licence. Third-party developers are free to choose a licence that best suits their interests, however it MUST not contravene our proprietary licence.

Technology stack

Primary technologies

Vue.js

Vue.js is a progressive JavaScript framework used to build user interfaces. It is designed to be incrementally adoptable, and easy to integrate with.

While it is readily capable of simple experiences it is also an excellent choice when constructing rich, interaction heavy experiences such as full or even partial Single Page Applications (SPAs). There is also no shortage of tool and IDE integration available to aid with development.

Vue.js allows us to create isolated, reusable UI components and greatly simplifies interaction with the DOM by allowing us to treat the UI as a function of our state.

When writing Tui bundles we are creating single component Vue files that can contain markup, JavaScript, SCSS and Langstrings.

To ensure consistent and organised design, and flexibility when styling Tui, we use CSS variables and an industry standard CSS selector naming convention.

The official Vue.js website provides excellent documentation on getting started with and learning Vue. Any knowledge you or your team have of Vue is directly applicable to developing with Tui.

GraphQL

Tui predominantly uses GraphQL when interacting with Totara Core (the server).

GraphQL is a self documenting, flexible language for querying data from a server and writing back information via mutations. It supports types, interfaces, object types, and many other language structures that enable rich description and validation of the server side APIs.

The official GraphQL website provides an abundance of documentation to help you learn about GraphQL and how to work with it.

For information on the server side implementation of GraphQL see our Integration with Totara Core developer documentation. 

SCSS

We use the SCSS superset of CSS syntax to remove repetition in our CSS code, which includes use of variables, functions, mixins and nested rules. SCSS is compiled to CSS as part of our build process, and also by the PHP SCSS parser on the server as required. Official documentation site for SCSS can be found at https://sass-lang.com/documentation, and the PHP SCSS parser at https://github.com/sabberworm/PHP-CSS-Parser.

CSS Variables (aka Custom Properties)

Tui makes use of CSS Variables to provide easier runtime value overrides within our UIs. In Totara 16 and earlier, in order to support IE11 we resolve these to actual values as part of our build process.

Supporting technologies

Vue Apollo and Apollo Client

We use the Vue Apollo and the Apollo Client libraries in order to facilitate easy integration between Tui and GraphQL.

Apollo Client provides organisation and structure to the way in which data from GraphQL is made available to Tui. Apollo Client is an entirely independent state management library, and we rely upon the Vue Apollo library to integrate it with Tui and GraphQL. Presently we use only a small fraction of what the Apollo Client library provides. See the official Apollo Client documentation for more information on the features of that library.

Vue Apollo specifically integrates GraphQL into Tui via Apollo Client. The official Vue Apollo documentation provides information on how that is achieved.

Apollo allows us to have a single source of truth for data fetched from the server via GraphQL queries and maintain consistency throughout the UI when something changes.

Webpack

Webpack is used to compile Tui source files into the build files that are used on the client.

It is a static module bundler for modern JavaScript applications, which Tui is. It provides bundling, dependency management and resolution, output translations, separate processing for modes (e.g. production and development) and integrates easily with modern development tools and libraries, such as those used for code linting, unit testing, and style checking.

See the official Webpack website for more information about Webpack.

Jest

Tui makes use of Jest as a unit test framework. All core Tui components should be accompanied by unit tests, in the same way we require all backend changes that can be tested to include automated tests.

Jest is a clean, integrated JS unit testing framework that provides all the things we need to write good unit tests. More specifically, vue-test-utils provides helpers for testing Vue components with Jest.

For more information on Jest see the official website jestjs.io.

eslint and stylelint

These two libraries are used to ensure that our code is consistent and meets basic essential quality standards.

The eslint library finds, prompts and fixes problems in JavaScript code. The stylelint library on the other hand finds, prompts and fixes problems in your SCSS and CSS.

Terminology

Tui component

High-level 'components' as the term relates to Totara codebase 'system components', for example 'theme_ventura', 'engage_article' and 'mod_perform'. Each high-level Tui component encapsulates its own dependencies, as well as referencing available core dependencies, for example utility Javascript, unit tests and static assets such as images.

Tui Single File Components (SFCs)

Low-level 'components' as the term relates to small units of composable UI, for example 'Grid.vue', 'Button.vue' and 'Uniform.vue'. Each low-level Tui SFC encapsulates its HTML, Javascript, SCSS and Langstring dependencies, as well as referencing available core dependencies, for example utility Javascript, CSS Variables and GraphQL imports. These are also known as 'Vue components' as they are built with Vue.

Bundles

Files created by Webpack that 'bundle' Javascript and other asset dependencies that are then served to a web browser. The Tui framework's configuration and extension of Webpack bundles each component separately and then pages request the bundles it needs from either the browser cache or the server.

Build

The process by which raw source files are manipulated and output into build files, ready for serving in a web browser. Moving from a raw state into a built state requires a build process to be run using the Command Line Interface (CLI). When the build process is run, source files are modified for example by transpiling Javascript code into a format suitable for legacy web browsers, or compiling SCSS into CSS, or compressing files.

Architectural goals

Design by composition over creation

Over many years of development, the origin of our front end in Totara 12 and earlier encouraged duplication of code between different plugins and core. The net result of this has been inconsistency between UIs throughout Totara learn, sometimes the differences are small - sometimes they are significantly different. Our strong preference going forward is to promote and enable use of our Tui framework so that UIs can be constructed more quickly, using reliable and road-tested components that have consistent behaviour and follow predictable UX flows.

Standalone separation versus intentional coupling

We have created reusable abstractions in a way that facilitates widespread adoption, and these abstractions happen in some different ways:

• Low-level uncoupled SFCs within the core Tui framework. They typically have a single purpose, for example they would not be an aggregate of other small components or request data from a GraphQL service, for example. A 'Card' component for example.
• Low-level coupled SFCs within the core Tui framework. Similar to the uncoupled components in that they still typically have a single purpose, but different because they are intentionally coupled with one or more components. A 'Grid' component coupled with its 'GridItem' component for example.
• Mid-level SFCs within the core Tui framework. These components are aggregated smaller components and are usually smarter than the previous two types as they may have a number of 'props' that conditionally control rendering logic. A 'DataTable' component for example. 

We sometimes refer to the first two types as 'dumb' components, and the second as 'smart' components. There are exceptions to the above rules, and where making an exception makes sense for reusability, we've allowed that.

Consistency of approach in components

Throughout Totara Engage and Totara Perform, you will find that their particular implementations make use of our core components. The result of doing this has been improved predictability for low-level components, and also time savings from not implementing multiple versions or the same components between multiple teams. When we identify a UX or visual design improvement, we implement that improvement within our core component(s) and see the results reflected across all usages.

Implementing smart UIs

Industry trends with front-end technology lean heavily toward Single Page Applications (SPAs), and while in the right circumstances these types of application have their benefits, they also come with drawbacks. You can expect some of these benefits to be harnessed within the Tui framework, in that we will implement UIs that do more without impeding users, and naturally that means combining multiple page loads into a single experience. We anticipate that this will not be at a whole application level, but instead within targeted areas of the platform, where condensing previously elaborate workflows into succinct steps is the right thing to do. Rather than SPAs, the concept is more likened to mini-SPAs within entry point pages.

Building with accessibility in mind

We involved accessibility in the design phase of each component, meaning that accessibility has been an integral part of each of our SFCs. We've made accessibility-related props of the components required where practical and possible, and widely supported and available where not. Leaning on our composition over creation, in addition to additional checks, means that our page-level accessiblity is of a high standard.

Enabling extendability

When you build on top of the Tui framework you will find that overriding different aspects of the core code, as well as implementing your own innovations, are all still possible. This makes the Tui framework adaptable to your needs. Within your own custom themes you can limit customisation to simply overriding core CSS Variable definitions, or do more by either overriding entire components or individual technology blocks within each component. You are able to do this within plugins also.

As well as component overrides, you can also extend by using per-component Webpack configurations and also create your own custom code blocks inside your SFCs, as we have done with Langstrings.

Product bound versioning and dependency management

Unlike our older technology stack which uses the independently versioned Bootstrap library, the Tui framework will be versioned in lockstep with our product release versions.

By creating reusable components, we see the benefits of consistent changes across usages of those components, and we are also presented with the option to deprecate a given component and replace it more easily than if we needed to replace multiple implementations that may need to cater for different use cases, and may have different internal dependencies or are authored differently.

Our preference is to release cohesive, related improvements or fixes within major releases, however inevitably some changes may need to be made in a minor release. Each Tui component within /client/component/ has its its own bundle produced by Webpack, which means that you won't need to worry about re-compiling your own code every time we make a change - so long as the API and assumptions of a given component are the same or non-breaking, interoperability should be maintained.

Implementation overview

HTML

Tui components contain HTML that is structured semantically, using the HTML5 standard and provides containing wrappers enough to allow for a wide range of theming needs. Much of our HTML now, or in the future, needs to serve more than one purpose on a given web page, once it is rendered. Because of this, we manipulate HTML by embedding logic within the markup, to control application of CSS selectors, ARIA attributes, conditional component trees and more.

Read more about the Tui framework's approach to using semantic markup.

CSS

With the styling power that CSS provides as a base technology, it is very easy and tempting to write styles that apply in one situation that should not in another. In line with our architectural goal of composition, we now target component styles using the Block Element Modifier (BEM) naming convention. While this is not foolproof because all styles still reside within the global namespace within a web browser, the BEM convention drastically reduces the likelihood of a style regression caused by naming clashes, specificity battles and the Cascade. We manipulate basic CSS by using SCSS, a superset of CSS that offers features that allow succinct authoring techniques while outputting selectors that appropriately target markup while also allowing extensibility by other developers.

Read more about the Tui framework's CSS conventions.

Javascript

As with our use of HTML, rendered markup needs to serve more than one purpose, and we do this using Javascript logic. Web browsers are complex, as are web applications that run within them. The Tui framework needs to operate in a number of different user environments while at the same time providing reliable and safe mechanisms for bridging HTML, CSS and Javascript. The Vue front-end framework has been adopted to facilitate this bridging, for example by abstracting raw Document Object Model (DOM) manipulation to a virtual paradigm, where keeping track of changes of state becomes more reliable.

Read more about the Tui framework's usage of Javascript.

SVG

Like most web technologies there are pros and cons in a given usage. One of the changes the Tui framework brings is an implementation of SVG for icons and images. We don't manipulate these much in our core code, but other developers can. We do wrap each of our components in a Vue template that gets bundled for optimal cached deliver to the client.

Read more about the Tui framework's approach to rendering SVG images and SVG icons with Totara Perform, Totara Engage, and Totara Learn.

String internationalisation

Totara 13 remains translatable into different languages, with UIs created in Vue making use of a localisation API.

Read more about the Tui framework's string internationalisation.

Bringing it all together

Each of the above technologies work in unison to create UIs, the authoring of the above technologies happens within Vue SFCs. Triggering the manipulation of each given technology happens on the CLI during our build process, which is a mix of NPM scripted Tasks and Webpack Loader transformations and finally bundling. The result of each transformation is the addition of language features, file optimisations or compatibility enhancements. Each of the technologies in the Tui framework benefits from being manipulated to serve as many use cases as required.

Read more about how NPM scripts and Webpack produce files ready for the client during the build process.

Component structure

Tui components are organised differently to previous front-end code. We used to create Mustache templates for logicless HTML, LESS files for compiling to CSS, and AMD for encapsulating JavaScript logic. This provided us with a 'separation of concerns', a way of dividing code into somewhat manageable chunks at a technology level (HTML/JS/CSS). We still want this separation of concerns, but now we do this at a component level.

Read more about the Tui framework's guidelines on component structure.

Accessibility

We involved accessibility in the design phase of each component, meaning that accessibility has been an integral part of each of our SFCs. We've made accessibility-related props of the components required where practical and possible, and widely supported and available where not. Leaning on our composition over creation, in addition to additional checks, means that our page-level accessibility is of a high standard.

Read more about the Tui framework's approach to accessibility.

Rendering pipeline

UIs are typically composed in two ways - first render and dynamic render.

First render components are those that appears on a web page during the page load process, the Tui framework does this synchronously by initialising the required dependencies injected by PHP, ready for runtime. There are a number of techniques to optimise this first rendering of components, including server-side rendering, and while the Tui framework cannot assume that Node is available to provide this type of optimisation, it would be possible for implementing developers to adopt this.

Read more about the Tui framework's rendering pipeline.

Client-side state management

Using the Vue.js library within the Tui framework provides additional benefits beyond virtualising DOM manipulation. One of these benefits is the ability to understand a standard flow of data. Components will often receive data in the form of properties (props) from another parent component, or from the PHP output renderer that injected one or more root Vue components. As well, components will usually contain their own data to keep track of internal state that it should be concerned about, or to make use of Vue's data reactivity mechanism. Following Vue's own documented guidelines, changes to state or interactions are often 'propagated upward' using Events to a parent component that needs to be concerned about one or more of its child components' state.

Read more about the Tui framework's approach to client-side state management.

Mediation and caching

The different mix of technology outputs between Totara 13 and previous versions of Totara has meant looking at how files are mediated by the server and cached by the client. As per usual, there is a need for two modes - development mode and production mode.

Development mode assumes that file loading performance is not important, in essence every time a page is reloaded new files are retrieved from the server. Due to the different implementations between Tui and Legacy, the mediation and caching has a different collection of settings:

$CFG->forced_plugin_settings['totara_tui'] = ['cache_js' => false, 'cache_scss' => false, 'development_mode' => true];

These settings are also available within Totara 13's UI by navigating to /admin/settings.php?section=totara_tui_settings. This page describes each of the settings. Also as per usual, it is important to understand the difference between development and production mode. With any/all of the development settings enabled, end users will experience slow download and page load times.

Legacy Theme

In order to implement the Tui framework as our new technology stack alongside Totara Learn's existing technology stack, the Legacy Theme has been created. This theme is intended as a stop-gap solution to bridge the differences between technology stacks.

Read more about the Tui framework's dependency on and implementation of the Legacy theme.

Creating Custom themes

As with previous versions of of Totara, Totara 13 allows for the creation of custom themes, including inheritance of parent themes.

Read more about creating Custom themes with the Tui framework.

Theme UI overrides

With the Tui framework's adoption of CSS variables, the process of overriding variable values has become less of an overhead in terms of processing power required to deliver a result to the client. With previous versions of Totara, string replacements were performed on files by PHP as well as invoking the SCSS parser during runtime, after an end user makes changes within the Theme Appearance Settings UI. Now, changes are made in the UI and overriding values are output into a file at second-to-last position in the Cascade. The overriding values are CSS variable overrides, which means that the newly output definitions take effect instead of previously defined values.

Customisations non-inclusive, there is now only one file that follows the above output overriding, and that is the Custom CSS input box in the Theme Appearance Settings UI. This input box can accept further CSS variable re-definitions which then take effect instead of all previously defined values. Because this file is now last in the Cascade, it also means that any other CSS selector overrides come last.

Integration with Totara Core

Tui is designed to be standalone, in that while it is an essential part of our products it is also interacting with them using arms-length methods, methods that any front-end framework or application could use. Primarily this involves communicating with the server using GraphQL queries and mutations. Where back-end integration is required JSON maps are used to generically explain structure from one side to the other.

Because Tui is not directly web accessible it also relies upon Totara Core to mediate required resources from the server to client. This is taken care of by a Totara Core plugin, totara_tui, which when given a component name can predict the file location for the build bundles. Holistically the totara_tui plugins primary function is to act like a webserver.

For detailed information on the integration please see Integration with Totara Core.

Development process

The development process does not need to change to support the development of Tui as a front-end framework, nor implement within our new products. However it requires that our team be upskilled. Both the front-end development team, and the cross-disciplinary team members.

This includes upskilling those back end developers with an interest in front-end development, the design team members on the ideals for component creation and the tight tie in this requires with the design system, and those who participate in solution design as Tui offers a greater set of solution possibilities.

What will change is the requirement for front-end improvements to be covered by unit tests.

More information

The following pages provide more information on Tui: