How Feed Processing Works in Concourse

Feed processing involves populating and synchronizing data that is maintained by an external system. Typical examples include user, course, and registration data, often found in the student information system. It is also possible to use feed processing to migrate data from one system to another, such as inserting course descriptions or outcomes directly on to syllabi.

 

Syllabus_Geek_Note_Smallest.png*Note: Before continuing, it may be helpful to get a better understanding of how courses, sections, and templates interact.

 

 

Construction

At the end of this article you will find the Concourse feed specifications, samples for each of the feed files, and a debugging guide. They detail everything you should need to know to construct and process feeds in the system.

Though many of the definitions and parameters should be straightforward, we call out a few of the more important and/or complicated parameters below.

Further, while there are many feeds that can be created and run, it is only necessary to construct the ones that are required to support the current scope of your implementation. Likewise, we suggest you begin experimenting with feeds in your sandbox that contain only a few rows to make sure that they are constructed properly before moving ahead to fully populated files in your production system.

 

All Feeds

  • Source: It is highly recommended that the data provided by these feeds be derived directly from their authoritative, often the Student Information System (SIS). This is critical to easily integrate Concourse with other campus systems.
  • Separators: All feed files need to be a pipe "|" separated text file. Since much of this feed data should be obtained from a report or script within your SIS, IT should have no issue producing this for you. However, if you are to generate the data manually, note that this is different than a comma separated value (CSV) worksheet that is typical in Excel. You can learn how to work with PSV files here.
  • Updates: Making changes to your feed file and processing them again will update all the data with some exceptions, the unique identifier(s) for the feed row. They are the USER_IDENTIFIER, COURSE_IDENTIFIER, and SECTION_IDENTIFIER columns. If a new user, course, or section identifier is entered, it will create a new entry and not update the existing one. Feed specific exceptions are noted in the feed specification.
  • Relationship: These feed files require a degree of coordination and sequencing in order to properly work. The user feeds identifies people. It is normally the first feed to be processed. Second comes the course feed. Next is the section feed, which relates courses to course sections. It does not actually create individual sectional syllabi from course syllabi. Instead it takes the courses already processed and applies one or more sections to them, all of which will share the same syllabus. You also then have the option to process syllabus item feeds which will insert content directly onto a course syllabus. Last comes the registration feed, which ties together users with courses and their sections.

 

System Data Feeds 

  • User Feed
    • USER_IDENTIFIER: This is the unique identifier for the user that is often used for common authentication (e.g. smithb). 
    • INTERNAL_USER: This defines whether the user will be logging in using credentials stored in Concourse (=1) or those from a central campus system (=0), such as an LMS or portal.
  • Course Feed
    • COURSE_IDENTIFIER: This is the unique identifier for the course and should be a sequence already used in the SIS (e.g. 2012_SP_ECON_1100_2A). Also it is typical to use this to uniquely define the template set, which should be constructed in such a way that an SIS script can easily concatenate the parameters into a unique identifier (e.g. ACCT_4010_TMPL).
    • CAMPUS_IDENTIFIER: This needs to be set to one of the external identifiers provided on the domain administration page. Also, if "Campus" is relabeled as something else (e.g. Program) to match your organizational structure, the feed still needs to use the "Campus" column title and data.
    • DEPARTMENT_IDENTIFIER: This needs to be set to one of the external identifiers provided on the domain administration page. Also, if "Department" is relabeled as something else (e.g. Division) to match your organizational structure, the feed still needs to use the "Department" column title and data.
    • START_DATE and END_DATE: These are required for every course, even if they are not appropriate for something like a template. If that is the case, simply fill in a reasonable date range (e.g. the start of the academic year to the end). When templates are cloned, these dates are replaced with the then appropriate dates.
    • CLONE_FROM_IDENTIFIER: Probably the trickiest of all the parameters, this will determine whether or not the course to be created should be from scratch or from another course using a course identifier already in the system. It is used most often when replicating a template for a given course section. This is why it is so important to create course identifiers for templates that can be scripted from the SIS (e.g. ACCT_4010_TMPL).
    • IS_STRUCTURED: This will determine if the created course will be based on the Concourse structure (=1), that is using distinct fields for description, objectives, course policies, etc., or as simply a place to store uploaded syllabi (=0).
    • IS_TEMPLATE: This is a flag to determine if the created course is to be a template (=1). If used in combination with the CLONE_FROM_IDENTIFIER and the course to be cloned is also a template, a linked template is then created, a very powerful but advanced concept.
  • Section Feed
    • Overview: This feed is used to "insert" instances of sections within the courses created from the course feed. The reason this feed is separate from the course feed is to allow for multiple sections to participate in a single "master" syllabus. However, most institutions simplify their processes by creating one section per syllabus. Further, if you do not run this feed following the course feed, no sections will be available for users to be registered with.
    • SECTION_IDENTIFIER: As noted above, many institutions will create one section per syllabus, which means this identifier and the COURSE_IDENTIFIER will look very similar. In this case, the course identifier may be 2012_SP_ECON_1100_2A (i.e. a course identifier really being used as a unique section identifier) whereas the section identifier is then 2012_SP_ECON_1100_2A_SEC (i.e. a dummy section identifier).
  • Registration Feed
    • GROUP_NAME: This will determine into which group the user is placed when they are registered for a course. The seven available groups include: Managers, Developers, Instructors, Assistants, Students, Guests, and Public.

 

Data Removal Feeds

  • Disable User Feed
    • DISABLED: This determines whether a users' account is disabled (=1) or enabled (=0). A user must be disabled to later be completely removed from the system.
  • Mark Course For Deletion Feed
    • MARK_FOR_DELETION: This determines whether a course is marked for deletion (=1) or not (=0). A course must be marked for deletion to later be completely removed from the system.
  • Drop Registrant Feed
    • DROP: This determines whether a user should be dropped from a all course sections (=1) or not (=0). If a user is inadvertently dropped, they must be re-added using the registration feed.

 

Syllabus Content Feeds

  • Item Feed
    • Overview: It is important to understand the downstream effects of inserting syllabus content via a feed, especially when being placed on templates. If the item already exists, its data will be replaced entirely (all fields). If the item category does not exist, it will be inserted and inherit the group permissions assigned to the syllabus as a whole. Updated content in an item feed will replace existing content as a whole. In other words blank columns will end up removing content, not leave the fields alone. Further, item feeds will interact with linked templates the same as one manually editing syllabi would. In other words the addition/editing of content via a feed will cascade down to children syllabi, potentially removing content that was already present.
    • NOTES: This is the HTML that will be directly inserted in to the syllabus. For that reason tags should be used in order to style the data as you'd like it to appear. In most cases this is simply the use of surrounding <p> tags.
    • IS_LOCKED: This determines whether the syllabus content is locked (=1) or editable (=0) from within the system by those granted such editing permission. It is typical that synchronized data be locked and migrated (one-time) data be editable.
    • DESCRIPTION, REQUISITES, OBJECTIVES, and OUTCOMES: Only a part of certain feed category types, these fields operate much like the NOTES field but data is inserted into more specifically labeled parts of the syllabus.

 

Delete Locked Items Feeds

  • Item Feed
    • Overview: It is important to understand the downstream effects of removing syllabus content via a feed, especially when being drawn from templates. The deletion of an item will remove it and any children from a syllabus. If the item is linked to other syllabi through templates, these will also be removing. Only locked items can be deleted with a feed.
    • DELETE: This determines whether item content should be deleted from a syllabus (=1) or not (=0). If content is removed, it is not recoverable.

 

Process

The feed processing area is located at Admin > Feeds. It is a feature that is only available to system administrators. Once you've created a feed, all you need to do is select the Type of feed you are going to run, Browse for the file on your computer, and click Process. The results of the processing and/or errors will then be shown below. Here's an example from running a course feed:

Processing Output
Created: 2013_F1_ACCT_2025_1A (line 1)
Created: 2013_FA_BIOL_4100_3C (line 2)
Created: Cloned 2013_FA_BIOL_4100_3D from 2013_FA_BIOL_4100_3C (line 3)
Updated: INST_TMPL (line 4)
ERROR: Bad row at line 5: Bad department identifier

A full list of processing errors, possible causes, and suggested resolutions is found in the PDF at the end of this article.

 

Syllabus_Geek_Note_Smallest.png*Note: Feed processing is highly dependent on the size of the feed to be processed. Files with less than a hundred lines should process in short order, in less than a minute. However files exceeding a thousand records can take considerably longer.

 

Updates and Exceptions

In most cases data can be updated by simply re-running the feed with the new data. However there are a few instances where this is not possible.

 

All Feeds 

  • Identifiers: As the unique identifier for a feed, these cannot be updated by inserting another one. If a new identifier is used, a new entry in the database may be made. This covers the USER_IDENTIFIER, COURSE_IDENTIFIER, and SECTION_IDENTIFIER parameters.
  • Rows: Feed rows are processed as a whole. Therefore the processing of a row will update all the columns for that row and not individual fields.

 

System Data Feeds 

  • Course Feed
    • CLONE_FROM_IDENTIFIER: This field cannot be updated. Cloning happens upon initial creation only (i.e. the first time the row is processed).
    • IS_STRUCTURED: Though this field can be updated, it may cause unexpected behavior. Switching a course from structured to unstructured or reverse will end up making syllabus data that was previously uploaded/entered inaccessible.
    • IS_TEMPLATE: Once a template is cloned it is either linked to the new template or not. Updates to this field will not turn a regular course into a linked template or break the links of a previously linked template.

 

Data Removal Feeds

  • Drop Registrant Feed
    • DROP:  If a user is dropped, entry of a 0 will not place the user back in the course sections. They must be re-added using the registration feed.

 

Delete Locked Items Feeds

  • Item Feed
    • DELETE: If syllabus content is deleted, entry of a 0 will not place the content back on the syllabus. It must be re-added using a syllabus item feed.

 

Automation

Once you've got the process of constructing and manually uploading feed files, you can go the extra mile to automate the processing of feeds on a regular basis. Learn about automated processing here.

 

Syllabus_Geek_Note_Smallest.png  *Note:  The Syllabus Geeks understand that some institutions may be using a syllabus system from which they would like to migrate data to Concourse.  For information on things to consider regarding data migration, click here.

 

------------------- Feed Documentation -------------------

 

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

1 Comments

Powered by Zendesk