Question Analysis in WaxLRS

| Comments

We’ve had Question Analysis in WaxLRS for a while now, but haven’t talked much about it. I’m not going to go into too much detail here, but I thought I’d give a brief overview of what it is and why it’s useful.

In short, Question Analysis helps find questions that do not measure the same things as are measured by overall score on the assessment they are part of. The calculation is independent of question difficulty, except insofar as excessively easy or hard questions will sometimes be indicated, and there are many possible explanations, from trick questions to infrequently answered questions to badly phrased questions to questions with typos. All we’re trying to do is point out questions that are worth taking a harder look at, and are better candidates for reworking or elimination than other questions in the assessment.

Nine Awesome Reads About the Experience API, and One Cartoon

| Comments

At Saltbox we really enjoy reading about Experience API (blog posts, journal articles, tweets, use cases, etc). We especially love it when people grasp how the standard can potentially impact their corner of the learning world as practitioners, developers or observers. So we decided to compile a list of our favorites! We hope you enjoy this curated list from some amazing people, especially if you’re just getting your feet wet with the Experience API (just released in v1.0! fist pump):

A Big Day for the Future of Learning

| Comments

April 26th marked the official launch of the new Experience API 1.0 (xAPI) learning technology standard and big step forward in the evolution of learning & development.

The buzz continues to grow about the formal, informal, social, and mobile learning opportunities the xAPI affords. We are excited about the advanced data analysis we can provide learning professionals in order to improve training, and ultimately employee performance.

One of the most important things about the xAPI is that it changes the learning conversation from formal training and tracking of traditional metrics (e.g. completions, test scores, number of classes) to answering questions like “which questions least correlate to success (and can be eliminated)?” or “which learning path is most effective for new hire training?”.

Data at the Point of Experience

| Comments

One of the biggest promises of the Experience API is delivering contextual information at the moment of need, such as the distribution of the answers so far after completing a question. Unfortunately, such aggregate data are expensive to compute in the client; it might take several seconds to retrieve the necessary information for even a few thousand answers.

One option is for systems providing contextual information to expose additional APIs for accessing the aggregated data. That may be reasonable for some scenarios, but for answer distribution and many others the approach poses overriding interoperability and portability issues.

Instead, I recommend that LRSs and reporting systems providing summary information intended for use at the point of experience materialize those summaries periodically into summary statements with a sketched form such as ‘LRS summarized the answers for Question 7 as ________’.

How Tin Can Can Help: Save Out Experience Data

| Comments

This is the second in a series of posts (and somewhat belated, we’re pretty busy here at Saltbox) answering designer questions about how the Experience API, Tin Can, can help in different scenarios. For more introduction, take a look at the first post.

In this post I’m tackling a question posed by Julie Dirksen. Due to the nature of the question and answer, this post will be less about data analysis and more about Tin Can technology.

I’d like a learner to be able to save the data out of a learning experience that they think is relevant – to a virtual notebook, or a file they can print or whatever. How can Tin Can help do that?

How Tin Can Can Help: Vehicle Fleet

| Comments

There’s been some great discussion around a post by Koreen Olbrish. One part of that discussion has been questions from designers around the idea of “How can Tin Can help?” Kevin Thorn and Julie Dirksen both posted a variety of scenarios, and I’m hoping we see a lot more.

I’m going to join the discussion and put on a second hat that doesn’t come up too often. My main hat nowadays is CTO of Saltbox – developer, ops, chicken sacrifices, and any other technical stuff, and that’s the hat I’m usually wearing when I participate in the Tin Can API standards discussion. This second hat is something that used to be a big part of what I did: social science researcher doing data analysis.

A Web of String: PKI Signatures and the Tin Can API

| Comments

DISCLAIMER: None of the new URIs in this post should be considered final.

  "actor": {
    "mbox": "",
    "name": "Russell Duhon"
  "verb": {
    "id": "",
    "display": {
      "en": "created"
  "object": {
    "id": "",
    "definition": {
      "name": {
        "en": "A Web of String: PKI Signatures and the Tin Can API"
  "context": {
    "contextActivities": {
      "parent": {
        "id": ""
    "extensions": {
      "": {
        "created": {
          "signature": "-----BEGIN PGP SIGNATURE-----
          Version: GnuPG v1.4.11 (GNU/Linux)

          -----END PGP SIGNATURE-----",
          "accepted": {
            "signature": "-----BEGIN PGP SIGNATURE-----
            Version: GnuPG v1.4.11 (GNU/Linux)

            -----END PGP SIGNATURE-----",
            "lrs": "",
            "trust": "marginal"

I’ve been thinking about options for using public key infrastructure (PKI) and web of trust (WOT) concepts in the Tin Can API/Experience API to support a number of important scenarios. Spurred by a conversation with @ellenm1 I’ve put together this blog post.

Making Statements: Tin Can API Best Practices

| Comments

As more and more tools create Tin Can API/Experience API statements, best practices are starting to come together. In particular, I’ve seen certain approaches prove themselves effective while working with customers of our Wax Learning Record Store. I’ve compiled several of those best practices here, for reference.

Client-side statement ID

All statements sent to an LRS have an ID. If no ID is provided in the submitted statement, an ID is automatically generated for the statement. This ID is a UUID, so even systems that do not coordinate can reliably assign unique identifiers. Early Tin Can adopters are taking advantage of the flexibility of the standard, and often sending statements to multiple Learning Record Stores. Those Learning Record Stores might later communicate back and forth. If the ID of the statement is generated by the LRS, then each statement will have different ones. Instead, clients should generate statement IDs, so that all LRSs agree on the ID of the same statement. While this is a best practice, it is only a best practice if your tool generates a real UUID, using one of the standard algorithms. The best way to do that is by using a UUID library (for instance, one is included in the Python standard library). If you do not have a UUID library available, please send each statement only to one LRS first and allow it to generate the UUID (it will then return the generated UUID for you to use in the future).

Tin Can API: Go From Zero to Production in 2 Days

| Comments

We are pleased to announce that we are offering the first Tin Can API technical workshop! This workshop is designed to take any developer or instructional technologist from having a basic understanding of the Tin Can API to enabling an existing or new application to generate Tin Can statements!

Thank you Mojocat Creative Services & Surf Incubator for producing and powering this exciting workhop! We hope to see you there.

Learn more about this workshop!

Matching/Filtering for Tin Can Statements

| Comments

Tin Can API statements flexibly describe learning-related events. Like many event-focused systems, use cases frequently involve ‘watching’, ‘filtering’, and generally reacting to events having certain characteristics. While there are many extremely capable systems and libraries for performing those operations, there’s still a need for a portable, flexible way of defining filters that match or don’t match individual statements, for use as part of those larger systems.

After hearing from several people about scenarios where I saw such filters providing part of the solution, I started brainstorming. The results are below.

Generally speaking, a statement is of interest if a particular part of the statement has a value meeting certain criteria, or if combination of parts-meeting-criteria using and, or, or not apply. Other important considerations for the solution are ease of implementation, readability, and concision.

The solution I’ve come up with looks like this:

    "or": {
        "$.result.success": true,
        "$.result.score.scaled": {
            ">": 0.5
    "$": ""

That filter would match any statement with the ‘completed’ verb and either a successful result or a scaled score over 0.5.

The keys that start with a $ are JSONPath expressions. JSONPath is a standard, similar to XPath, for selectively extracting data from JSON documents.

The result of the JSONPath expression is compared to the value. If the value is not an object, that part of the filter matches if and only if the result of the JSONPath expression exactly matches the value. If the value is an object, the key(s) in that object define how the value must match, such as “>” above (in addition to the obvious mathematical operators, options will probably include intersection, set membership, subset, nonexistence, and so forth).

Inside value objects and in the filter as a whole, multiple filters are “anded” together, and combined/derived filters may be created by nesting filters inside “and”, “or”, and “not”.

Note: to ensure portability and security, no ‘script expressions’ are allowed in JSONPath in a filter.

If you like what you’re reading, comment or otherwise get in touch. I hope to see this work eventually lead to a community standard that can be applied in many scenarios and across many platforms where statements flow.