Integrating School IDs

Well, it's been a long time since I've made an update and this is my attempt at getting back at my projects. Unfortunately, my life is a bit chaotic right now. I have a wedding that I'm planning and the US situation is not the best and keeping a lot of people (like me) anxious. The other major thing that has happened is that I've been diagnosed with ADHD. I'm not sure if it was something that triggered an adult version, or it was just something I've always had but it suddenly became worse, but I've been struggling with keeping focus. I started taking meds and hope that this will let me get back to this project.

I will also be changing the format of this blog a little. Before, I would always do the code, then write the post. For this blog, and possibly for the future, I plan to write the blog as I develop. This is because one of the biggest struggles that I've had coming back to code was about how to approach viewing permissions for classes. While I was trying to write the code for Learning Demonstrations, I kept coming back to how the classes look like. What I found is that the only way to view classes was top be a student or a teacher. I needed to expand that for parents and other administrators. This is where the descent into madness begun, because I cannot seem to get a good handle on how to determine what to show to who, and how to determine what role the viewer is.  Since I couldn't find an answer, I tried to only concentrate on the Learning Demonstrations themselves, so that I could at least begin working on this, but I could not even get started without going down different directions. This could be the ADHD or burnout (not from this, rather from work), or something else. Whatever it is it has made it very hard to continue coding.

I did manage to get something built: the school ID system.

School IDs

School ID's are  way for school members to be identified quickly. In the past students had an ID that they had to memorize or, if your school had the money, the would get a lanyard or a card. While that still happens in a lot of school, digital IDs is the way to go nowadays. This should include a way to get the ID into their digital wallet and to provide a barcode that we can scan. This will allow us to tie their ID into other services that may extend to the physical world, such as checking in, attendance, etc.

Before we can make ID's though we need to actually have real id-like numbers. Before, we simply used the database id as the main id. This number will not really work since, and specially with new database, most of the numbers will be in the single to triple digit format. This is not really good for IDs; they should have a set length that is equal to all, as well some kind of randomness that makes guessing a person's id hard. They should also be unique without making SQL inserts complicated.

Enter Hashids, a “small PHP library to generate YouTube-like ids from numbers” that allows me to encode all the database ids into a unique hash id of a predetermined length. I decided to make the predetermined length a config variable (as opposed to a variable that can be changed in the database through an interface) to sort of force the user to make that decision during install. Once set, it should really not change, as doing so would require a re-encoding of all ids in the database.

The id is not being encoded real-time, rather it is saved as a field in the database named school_id. This id is a required field that is created after the model is created in the database.  I do this by hooking into the created and creating model hooks:

        static::creating(function (Person $person)
        {
            $person->school_id = time();
        });
        static::created(function (Person $person)
        {
            $hashids = new Hashids('FabLMS', config('lms.school_id_length'), '0123456789cfhistu');
            $person->school_id = $hashids->encode($person->id);
            $person->save();
        });

Now, the last important thing is that the id field of the person is still the primary key, meaning that all find operations will use the database id to search. However, by adding this directive:

    public function getRouteKeyName(): string
    {
        return 'school_id';
    }

I now make sure that all requests use this school id field to load the Person record, thus sanitizing my routes.

Real School Ids

I now have an ID system to identify my users. But what should I do with it first? Other than show it in the profile, of course. Well, users need ids, so I decided to create the school id system.  This system is in charge of creating and displaying id's for every user in the system. This led to my first problem: should all ids be the same?

The answer turned out to be, maybe? In some cases yes, in some cases no. I could foresee cases where one would be all that's needed, or another where campus-specific ids might be needed, such as when you're trying to determine whether a student is in the correct area based on which campus they belong to.  Different cards through roles are also useful, if anything so you can differentiate between staff and parents, but making them for every role turns into an unmanageable nightmare and issues arise for people with multiple roles. After consideration, I decided on four options that can be set by the administrator and created a school setting category for it:

As can be seen, this system allows for the number of id's to be in the range of 1 - (#of campuses * 3), making it as complicated as you want, while still enforcing manageable inputs.

The next step was to be able to create the IDs. I decided that an ID that is completely HTML-based was the way to go. Images would have been nice, but it would then need an image manipulation tool, which is not something I want to get into.  Converting HTML into an image was another nightmare that I researched. The only way that I could find was to run a headless chrome browser in the server that would render the image, which you can then take a screenshot in memory of the page, thus getting an image.  Instead, going HTML would ensure that it can be seen on any web-capable device( which could then take a picture) and that it could be “resized” using html flex containers.

Once that was decided, I wanted to make an editor that was user-friendly, or at least as user friendly as someone like me could make. Livewire to rescue! Using Livewire I made a component that is able to create an id. I think it looks ok, but it could definetly use some touch ups. With it, I linked the setting page so that all the ID's could be created from there and added a “My School Id" link in the user's profile.

Untangling Learning Demonstrations

After completing the ID system, I decided to tackle the Learning Demonstrations, since that will be the central part of this project. The issue was that, the more and more I think about the things I need to do, the more I realize how much of the basic foundation I was missing.  My plan from now on is to write about my development process as I code, so that I can better plan out what I'm working on, and I can concentrate better on next part of the project without getting lost in too many other tangents.

A Learning Demonstration (LD) is my version of “homework” or “assignments”. Thew work flow of this Learning Demonstration is as follows:

1. Teacher selects what Skills they hope to teach with this LD and select them.
2. Teacher enters the details of the LD, including description, classes to post it to, grading strategy and supporting documents.
3. Teacher posts LD and a message is sent to students (maybe parents?) about the LD creation.
4. Students work on the LD in a “workspace” area where they can submit work.
5. Students complete LD (either in-person or online).
6. Teachers grade the LD
7. The grade is reported back to the student.

So far so good. It's not until we start figuring out what the model should look like when we start having a problem. Specifically, the issue is with the “supporting documents” section, as this could mean a variety of things from URLS to actual documents that the teacher uploads to a link to a google document. As most LDs will have some sort of supporting document, such as a worksheet, or reading material we need a way for teachers to “link" document and document data to the LD easily. We haven't yet gotten to the part that it needs a name, description, etc. 

If we zoom out a bit, we can also see that this “need” to upload or link files becomes more prevalent in almost everything that the teacher or student would like to do. Want to post an announcement? You might need a file or a URL. Want to post a resource? same. A lot of the things we would like to do require some sort of file transfer, manipulation, and just simply accessing information contained in that file with it's own format constrictions.  In fact, looking at my previous system, we can see that we can set the background image as a link, but NOT as an upload, which might be legitimately what the user wants to do.

Uploading vs. Linking

So what should be done is to allow the users to have a way to upload or link files in the system so that it can be posted where it needs to be posted. But should we create our own repository of documents or rely on a 3rd party for those needs? Should we do both?

Uploading would probably be the easiest on users for non-text files.  Files like PDF's or special application files are not readily available in documents sites like Google Docs without uploading it first there. So, in a way, it's much easier to simply provide a drag-and-drop area for users to upload the files and the system to link it to where it needs to go.  Simple, right? Well, not because we now need to build something to manage and curate the files that we uploaded. Realistically, users do not want to upload the same file over and over. They need a way to re-use the files that they have previously uploaded, which means they need a way to manage, search, and organize the files that they have previously uploaded so that they can be re-used. This also applies to text documents, if we need people to produce them in our system, then we need a text editor, sharing permissions, etc.  A “simple” upload becomes a file explorer project and a cheap re-creation of Google Docs, and it doesn't even touch on massive amount of storage space that you'll need for your users. Currently, at my school for about 10 years worth of data for a 500-student school is about 30Tb. This requires some heavy server and data center access.

We can use linking instead. Linking relies on a separate, 3rd party product that gives you all the document needs, be that editing, creating, uploading, etc.  Once done in this 3rd party app, you can use that file by “linking” it to the system. For example, using Google Docs, you would select a file from it, click link and the system would add an entry to the database with the information needed to access that file (usually a URL). No file data resides on the system, only information about how to reach that file.

The benefits are obvious; we get a fully developed document manager for people to work as they want and we still get the data we need linked to the correct places. However, then we run into other issues that are endemic to this method. What happens if the user deletes a file? Well, there is no way to mark that in the system; we also don't really want that file deleted, as it may contain information that we need to look at in the next few years. The ideal for a school database is that the work that students create during their time here should be preserved so that we know what happened in the past, and so that we can use artifacts of the past to facilitate learning in the present.  There is also a permission issue. With our own upload, we determine who can see or not see a specific file. With a 3rd party system you need to make sure that your files have the correct permissions. A lot of times I have seen a file being posted up on an LMS through Google, but the permissions were not assigned correctly, so no one could open the document other than the original owner.

Finally, there is one more major issue with 3rd parties and that is, which one should we use? Google? Office? Dropbox? something else?  Each of these services require a special connector developed to work with the system. We simply can't use the same code for Google and Microsoft, because the configuration, set up and interface are radically different from each other, not to mention that keys and passwords are done differently in every system.  So how many should we support? 

Integrators for Integrating Integrations

What we need is a way to integrate 3rd party tools into our system in a way that is easy for the user to configure globally or personally, and to include that integration everywhere that is appropriate.  This does not inherently solve the problem of linking to a missing file, but it does allow us to create a way for us to create reusable code that can be used to quickly create a component to interact with this service. In this system I call that each of those components integrators.

But we're still thinking too small! Integrators should actually integrate into more aspects of the LMS. There should be an integrator for login, in case the user wants to use different logins for people. It should be possible, for example, to force all students to authenticate using Google (for their Workspace account), all staff use LDAP and all parents pick between email, google, microsoft, etc. Also, if you take a look at the ID system, one of the things that I mentioned in my previous post was the ability for users to add their id to their google or apple wallet. That feature is not currently built, not because of it's difficulty (I have the code written and working for my work) but rather because when I wrote it for myself, I hard coded a lot of specific values, such as uploading the certificates (with private keys!) to the server and allowing my app to access it while denying access to others. I also needed to sign up to Apple developers so that I could use their wallet feature, and create a Google service account to allow wallet access, as well as sign up for the wallet access itself. I considered building cod just for those two, but then I thought, what if the users don't want that option? or something different?  There should be a way to integrate other services into the id services.

Thus, so far we have at least 3 use cases that can use integrators:

• File access/linking
• Login systems
• ID Cards

But we can easily thing of more:

• Mail Chimp-sort of services for mass mailing
• Linking to an external calendaring system
• Different ways to scan ID's depending on technology

In a way, integrators is a sort of “plugin” system that allows for other developers to create plugins that will integrate into different parts of the system. These integrators should also try to consolidate services into a single integrator, so that if we would like to use an integration for Google Wallet and Google Docs, there should only be a single integrator, not 2, and the configuration for both these integrators should be combined into a single control panel. We should also be able, as a user, to install new integration via some sort of management system, ideally distributable online or through zip files that can be uploaded into the LMS.

Integrators are not user toys, they're for developers to create and for users to install.  This means that integrators are complicated pieces of code that needs to adhere to an interface. The code will the use this interface to display elements on appropriate html containers, to link files, or do anything that we expect these services to do. So let's make some definitions:

Integrator - This is a package that contains all the code to run integration to a single provider (i.e. Google, Microsoft, etc). It will be installable and configurable by users.

Integrator Service - An integrator will have at least one of these and can have multiple. Each of these will be usable by a feature in the LMS. For example, the Google Integrator might have a Integrator service for the file system providing file access to Google Docs, and a service for the wallet, providing the ability to add the user id to the Google Wallet.

Integrator Service Interface - Each integrator service will be an implementation of this interface. Each kind of service will have its own unique interface that will be provided by the LMS.

Integrator Configuration - This is a way for the integrator itself to be configured. At this point and due to the large amount of livewire components that I'm already using, I'm heavily leaning for this configuration editor to be a livewire component that is provided by the integrator.

Integrator Namespace - Each integrator will have a unique name space that they will use, which should follow the dot notation.  All settings, folders and keys should use this namespace as a prefix.

Integrator Storage - Integrators will have access to three different kinds of storage that they can use to persist their data: Settings, Payload and File System.

Settings Storage

This storage space is for settings. It persist information to the database with this structure:

• name: varchar, dotted notation, uses namespace as prefix.
• value: the setting, stored as JSON

This can be accessed by the App\Models\Utilities\SystemSetting model, which uses the setting name to return an array and it can take an array to store the settings in the database.

Payload Storage

This storage is for internal reference. it can take an array of data (same as settings) as the payload, but it will use a UUID as a storage id. You can manipulate this payload:

use App\Models\Utilities\DataPayload;

//create a payload
$data = ['foo' => 'bar'];
$payload = DataPayload::createPayload($data);

//save the id!!
$this->payload_id = $payload->id;

//now you can do whatever
$data = DataPayload::find($this->payload_id);
$payload = $data->payload;
$payload['other'] = $some_other_data;
//save the new payload
$data->payload = $payload;
$data->save();

//delete it
$data->delete();

File System Storage

Finally, the integrator will have access to the file system in order to save data. Specifically, it will have access to two disks that can be used to store file data. This data can be accessed using Laravel's File System API.

public: The public disk is for public data.

private: This disk is for accessing server data that is NOT accessible by public.

The public disk is mainly for assets, while the private disk is mainly for secret storage, such as a google key file or a private key certificate.

Continuing the Posts

I wrote the last few sections without writing any code (yet), although it helped me plan out a bit my direction. I've decided to stop this post her and publish what I have because this posts is already too long, and because I want to force myself to publish something. 

As I mentioned in the beginning, as a way to help myself code more and concentrate more, I'll be writing the blog post while I write the code, as opposed to after. At this point, I'm not sure what that will look look like; whether I will edit it after or not or what the format will be. I plan to experiment a bit. Regardless, my goal is to get back to this project and to get my mind right again.

Will update when I have more.