Intelligent Integrations

Integrators will be the main topic of this post, although I'm also tweaking some stuff for my first AI integration. I have some ideas that may be overkill, so we will see.  But integrators first and there are two kinds.

System Integrators integrate into the system directly. They're not using user credentials, although they may use other kinds of credentials. These credentials need to be secure, so they'll not be persisted in the database, instead they'll be saved to the private access storage as a JSON file. These integrators will register themselves in all the required services and will only be configurable by admins with the settings.integrators permission. These integrators will still have an entry in the integration instances table, where it will maintain the needed information for the integration between the system and the service.

Personal Integrators are integrators that connect a service to a Person in the system. They must be initiated by a user in the system and can be applied to specific roles, as determined by the integration admin. Users will be using their own credentials, but FABLMS will not be storing these and will utilize tokens instead, preferring systems like oAuth. These connections will be stored in the integration instances table every time a user makes a connection to a service with the needed information to establish and renew the connection to the service.

So, both of these Integrators will actually be derived from a single Integrator class that will be a database model of entries in the integrators table.  This table will keep all the different integrators available in the system and might eventually get its own interface to add third-party integrators, which is nice to plan for even if it will never be used. The second table will maintain the integration instances that will link different aspects of the integrator to either a Person or the System, It should be noted that each integrator will be responsible for providing multiple integration services to either the system or a user, which will then establish a service connection between them, which we call an integration instance.  Let's define some of these terms.

An integration service is a service provided by the integrator such as authentication (personal service), or system storage (system service), or an online pass-service like Google/Apple wallet (both system and personal service). An integrator is responsible for integrating all these services with their service provider, such as Google, Microsoft, Apple, etc. A single integrator may have a lot of different integration services available, which will be registered by the main integrator class. The number of services will grow with more service hooks I make available, so I'll be writing the main logic for the registration that will be able to use all future services that I will build later.

A service connection is a connection between an integration service and either the system or an individual user. This connection is something that must be established by a party, either the user by clicking a button and entering credentials to make the connection, or by an admin adding credentials to the system. Either way, some external authentication has to take place and it is individual. Some services might use the same kind of connections as well, such as accessing your Google Drive and Google Calendar will only need a single service connection to google by a user, which could even then be integrated in the authentication connection if we're forcing the user to login using that integration.

Once a service connection is established, an integration instance is created to keep all the necessary information about that connection so it can be used, renewed, etc. The instance data is saved in a database table that will contain all these integration instances. Each integration instance can be used to re-establish the service connection and interact with the service the user (or system) is trying to use.

Now let's get some details. We will need an integrators table, which should look something like this:

table: integrators
id: int
name: string                 # The name of the integrator
className: string            # The class name of the main integrator class, 
description: nullable string # An optional description of the integration
data: json                   # A JSON-column that can be used by the integrator class to save information
version: string              # The version of the integrator running. Useful for updates/upgrades and features.
enabled: bool                # Whether this integrator is enabled.

Each integrator will offer different services, which will be registered in the system through entries in the next table, the integration_services table.

table: integration_services
id: int
integrator_id: id    # Link to the parent integrator
name: string         # The name of the service
className: string    # The class name for this service.
description: string  # Optional description for this service
service_type: string # A system-reserved tag that where in the code this service an be used.
data: json           # A JSON-column that can be used by this service to save data.
enabled: bool        # Whether this service is enabled.

Finally, when a connection is established, an entry will be added to the integration_instances table:

table: integration_instances
service_id: int         # Link to the integration service, part of the primary key
person_id: nullable int # The id of the person or null if its a system connection. Second part of the prymary key.
data: json              # A JSON-column that can be used by this instance to save data.
enabled: bool           # Whether this connection is still enabled.

But that is only the persistence mechanism, the important part is the underlying code logic. Central to the whole system will be the IntegrationsManager class. This class will re the access point to do everything that requires integrations. Through this class, we can access all the integrators in the system, as well as all the available ones. It can also get specific services and will be responsible for actually registering all the integration services. I'm not yet sure if will handle the connections, but I'm leaning towards no.

Because of the importance of this class, it will be a singleton that will be loaded to through the service container using a new service provider IntegrationServiceProvider. We will also have models for the integrators table, Integrator and the integrator_services table, IntegratorService. However the table integration_instances will actually be a Pivot class IntegrationInstance that will be retrieved when accessing a connection between the user and the integration.

Since I have to start somewhere building the foundation, I will start by moving the existing auth code login into the integrator. This will allow me to start building how the integrator will play with the other components. I will start by building out the LocalIntegrator, which will handle all local authentication and local document storage. Once that is built and working, I'll concentrate on the Google one.

Integrator Cycle

Let's talk about how the integrator works as a package. It is assumed that someone (in this case, me) built an integrator with a bunch of services. The integrator code is installed in the system somehow, either through a zip file, or through a composer command. This makes the code live on the system, but does not add the integrator to the system. To actually add the integrator, a command has to happen. For now, I will build an artisan command

php artisan fablms:register-integrator {className : The name of the main Integrator class to register} {--force : This flag will force re-registration}

This will first check to make sure the Integrator is a valid class (extending LmsIntegrator), then insert a row into the integrators table and enable it. Finally, it will pass the IntegrationsManager instance to the Integrator's registerServices() method that will register all the services that is has by calling the IntegrationsManager::registerService() method that will insert a row for every valid service entered and enable that service. There will be an option to force re-registration, which will allow the integrator to overwrite the default data.

The system is set up in two-halves that complement each other. On the one side is the Integrator Model, which refers to an entry in the integrators table. The advantage of this model is that it can persist information to the database, it can be accessed quickly by a simple key or an element in the table structure, and it can be easily linked to people and other objects in my relational database. The downside is that, other than some data reserved for persistence, there is no custom code.

The other half is completely supplied by a “third party” (aka me) that has little knowledge about the LMS itself, but can understand how to follow API specifications. This class, LmsIntegrator is an extension of the Integrator model, thus the extended class has the ability to manipulate the model as well. The LmsIntegrator class is actually an abstract class that will be extended by the third party to create their own integrator. The integrator itself will contain mostly metadata that will tell the system how the integrator will fit in. 

The main purpose of the integrator is to be able to provide a list of IntegrationServices that this Integrator has. an IntegrationService is a model that tells us information about a specific service. Since the hook for the service must be created before the actual integration service, the only integration services that a third party is allowed to define are the services contained in the IntegratorServiceTypes enum, which currently looks like this:

enum IntegratorServiceTypes: string
{
    case AUTHENTICATION = 'auth';
	case DOCUMENTS = 'documents';
	case WORK = 'work';
	case AI = 'ai';
	case SMS = 'sms';
}

There can only be one kind of service per integrator. Meaning that you can't register two google authentication services, or two google documents services, which are used to retrieve the user's documents.  The service and integrator will also use the HasRoles trait so that it can be assigned to roles in the system. The permissions will be inherited though, so the user will only have to place permissions on the integrator and the services will default to the same permissions. However you can also turn off inheritance and set your own permissions on the service itself.

The IntegrationService model is also then extended to the LmsIntegrationModel abstract class, which defines all the functions that third parties must define in order to build a service. Each service must be contained in it's own class, and each class can only define one service.  The service itself will have metadata that will allow the system to know whether a user or the system can connect (i.e. integrate).  It will also be responsible to connect either a person or a system , thus establishing a connection that they can use to access the integration, or register, which will allow a registration to happen so that the user or the system can enter information about how to connect.

The last part in this system is the IntegrationConnection model. Each IntegrationService model will have a HasMany-BelongsTo relationship with multiple instances of Connection classes. Each connection is define in the database as a connection between the system or a user and the service. Each connection must also have it's own class, which will be filled out when a connection is established. The IntegrationConnection class is extended bty a large number of Connection classes that each describe a different type of connection. For example, we will have AuthenticationConnection class that only deals with authentiucating users (or the system) to a service, or the DocumentFilesConnection class, which ties a user to an external repository of documents.  Currently, I wrote 5 connection classes:

• AiConnection: A connection for a user to link an AI to their account.
• AiSystemConnection: A connection for the system to use a global AI.
• AuthConnection: A connection for the users to authenticate to a service.
• DocumentFilesConnection: A connection for users to an external file repository that they can access.
• WorkFilesConnection: A system connection that allows the system to store files.

A third party would then extend each of these classes with their own logic to allow connectivity and access. The Integration System diagram for the classes looks like this:

To give an example, here is what the interface for the AuthConnection class looks like

abstract class AuthConnection extends LmsIntegrationConnection
{
	/**
	 * @return bool If set to true, then the login system will get a password from the user to pass
	 * to this authenticator
	 */
	abstract public static function requiresPassword(): bool;
	
	/**
	 * @return string A html-string of the button for the user to login with. This will only
	 * really be done when the user chooses their login method. It should not be a link, as it will be
	 * wrapped by one.
	 */
	abstract static public function loginButton(): string;
	
	/**
	 * @return bool If set to true, the Login system will redirect the user to another page that will
	 * handle the authentication.
	 */
	abstract public static function requiresRedirection(): bool;
	
	/**************************************************************
	 * PASSWORD FUNCTIONS
	 */
	
	/**
	 * @return bool Whether this authenticator can be assigned a password by an
	 * administrator
	 */
	abstract public function canChangePassword(): bool;
	
	/**
	 * @return bool Whether the user can reset their own password.
	 */
	abstract public function canResetPassword(): bool;
	
	/**
	 * This function will attempt to change a password for this authenticator.
	 * @param string $password THe new password
	 * @return bool Whether the operation was successful.
	 */
	abstract public function setPassword(string $password): bool;
	
	/**
	 * This function will attempt to authenticate the user with a provided
	 *   password.
	 * @param string $password The password from the login form
	 * @param bool $rememberMe Whether the user should be remembered
	 * @param bool $autoLogin Whether the user should be logged in automatically. Defaults to true.
	 * @return bool Whether the user was successfully logged in.
	 */
	abstract function attemptLogin(string $password, bool $rememberMe, bool $autoLogin = true): bool;
	
	/**
	 * @param string $password The password to test
	 * @return bool Whether the current password matches the one provided.
	 */
	abstract public function verifyPassword(string $password): bool;
	
	/**
	 * @return bool Whether this authenticator can set if user must change their password.
	 */
	abstract public function canSetMustChangePassword(): bool;
	
	/**
	 * Sets whether the user
	 * @param bool $mustChangePassword Optional, whether the password must be changed. Defaults to true
	 * @return void
	 */
	abstract public function setMustChangePassword(bool $mustChangePassword = true): void;
	
	/**
	 * @return bool Returns whether the user must change their password.
	 */
	abstract public function mustChangePassword(): bool;
		
}

This is a copy and paste of my previous class, so the refactoring was quite painless.  I have also defined the the spec files for all the available integrations: AuthConnection for authentication, DocumentFilesConnection for user documents, and WorkFilesConnection for System file storage.  Once that was done I removed the old storage code and preplaced all the calls with the correct connections calls and everything worked the way it did before, only with a centralized integration connections and settings that you can change.

Speaking of settings, Integrators and IntegrationServices now have two more functions: isConfigurable() which is used to denote whether a service or an integrator has a configuration screen and configPath() which returns the path needed to access that service or integrator config screen. The integrator can also now publish their own routes, thus allowing for their own settings system with their own controllers, etc. This will make it easier for third parties (again, me) to create their own setting pages and logic independent of my system.  It also mean that I had to build an integration screen and config area. The Integration screen now also allows me to re-register the integrators, thus moving he functionality of the artisan command to the actual web.

All the configuration screens have also been relegated to their own page, which will list them and allow the user to access them all at once.

Google Integration

So now I have a centralized integration system, it's time to expand it to the next step, which is creating an integration with Google. In the last release I created the integration between Google and auth and documents. Each one of these services should be a straight copy, with one major distinction: I need to provide the ability to connect to Google independently of whether we're using the Google authentication. In the original code, the only way to get access to the Google suite was to set your login to Google and let the driver authenticate you. Now, however, the idea is that we can enable integration without having to force a login component. Instead the system will provide a Connections page for the user that will allow them to establish their own integration connection to any of the service that are available to them.  This will, of course be role-oriented, which means that we will need a way to attach roles to services (not connections) to “give permissions” to these integration services.

Next, we will need to provide a way for an authenticator to determine whether the user will get their data through login or through an independent connection. If an independent connection is the case, then a page will be built for all the possible services that a user is able to connect to, and a mechanism for the user to connect to that service that is independent of our system.  There will have to be a way to alert the user if the integration settings are lost and need to be reconnected.

The next decision that I made was to sperate the code for the Google integration completely from the main code. I would like to have the ability for 3rd parties to add integrators, which means that they will be publishing their own packages and putting them in the database. I would like to give an example of what this looks like by actually making the google integration its own composer package and be able to pull it in with a composer require command. Thus, I will be developing this package in a subfolder of my LMS code that will not be submitted to Git. When I release, I will also create a new project for this integration and publish it to packagist.  This will eventually allow me to set up a system that I can query composer to see if there are updates to the Integrators and allow system managers the ability to update them with the click of a button.

Doing the Google integration was actually simpler than I imagined, since the code was already written in the last update, so it was mostly a copy-paste job. Along with this, I also opens up the ability for users to register services on their own.   Users can now go their own profile and they will see this widget:

This means that I gave permissions to a role the ability to integrate with Google. They can click on the red button (it expands into a connect button) and attempt to connect. This will lead them to a Google prompt authenticating. Once they're authenticated, the system will try to auto connect to all the available service. Thus it will then display all the services connected to it, and will also show services that are not connected, but can be registered to so we can enter information to connect later.

AI Integration

In the picture above, you will see the next integration that I added: AI. I already had some AI code built, back when I created the Rubrics code, so I felt it was an easy step to create and bring in the AI capabilities into the actual system.  I decided to stick with Google, although I may do something with OpenAI as well. With this new service, I also introduced the idea of a Secure Vault. This vault will take important values and write them to a safe file in a private space, thus avoiding having to encrypt the values in the Database. However, I felt that this would blow up too much if it was applied to users, so I did end up adding support for encrypted values in the database. Thus, system AI settings are stored in an external file, which personal AI settings are stored encrypted in the Database.

Note that I coded the above with the possibility of allowing users access to the system AI if desired. This is not a requirement, simply a way I chose to develop this so users can share AI. In practice, there could be licensing issues that might prevent this from happening, but I wanted to showcase the ability.  Once the integration happened, I figure I would add a hook to be able to create rubrics through AI. So now, when creating rubrics, you will see this button:

 Clicking it will expand to the area where you can run the AI query:

A neat little add on, is the ability to edit a prompt for each object, either as a system prompt (for system users) or as a personal prompt for teachers to run. Clicking on the edit for the prompt will take the user to the edit prompt system, which allows the user to directly change the prompt being sent to the AI to match their desires. From here, they can also set the AI temperature, and even upload  documents for the AI to use.

The last piece to this was making sure that I kept the code to the Google Integrator separate. I did this by creating a packages folder in my root project directory and adding my file structure there.  I made those classes available by adding this to my composer.json

"autoload-dev": {
        "psr-4": {
            ...
            "halestar\\FabLmsGoogleIntegrator\\": "packages/halestar/FabLmsGoogleIntegrator/src"
        }
    },

Once I completed developing, I created GitHub Project and released v. 0.1. I then added the repo to packagist, when I noticed that I had forgotten my composer.json file in my root plugin project. So I added it, but my version is now 0.1.1.  This add on can now be added by executing

composer require halestar/fablms-google-integrator

File Types

The last extra I added was the ability to define system files types that are allowed to be uploaded into the system. The idea is to create a central place where all the files allowed were defined. This is done through the MimeType manage in the storage section of the settings. This is automatically sent to the any of the document storages places, restricting the files that can come into the system.

This has been a really large update. I've been dying to release it for a few weeks now, but there's always something more that I wanted to build. I'm forcing myself to stop so I can move on to the next feature: Learning Demonstrations. 

In the next update I plan to start defining what a Learning Demonstration is, as well as the process of building one. I will be leveraging the new integration system so that the LD can be fully fleshed out, with Document upload, AI assistance and the ability to post the first LD to teacher's classes. I'm still not ready to tackle the Class View Component, but I'm toying with the idea of creating integrations with Google Classroom and use that until I can build one for myself.

I will write more when done.